Oracle, שרון עוזיאל

1 Copyright © 2011, Oracle and/or its affiliates. All rights

reserved.

Insert Information Protection Policy Classification from Slide 8


reserved.


The following is intended to outline our general product

direction. It is intended for information purposes only, and may

not be incorporated into any contract. It is not a commitment to

deliver any material, code, or functionality, and should not be

relied upon in making purchasing decisions.

The development, release, and timing of any features or

functionality described for Oracle’s products remain at the sole

discretion of Oracle.


reserved.


Oracle Big Data Next Generation Data Management

Sharon Uziel, Oracle Consulting Infrastructure Manager


reserved.



reserved.


Explosive Data Growth Harnessing insight from Big Data provides an opportunity to gain

competitive advantage

STRUCTURED DATA UNSTRUCTURED DATA Content Provided By Cloudera.

2005 2015 2010

More than 90% is

unstructured data

Approx. 500

quadrillion files

Quantity doubles

every 2 years

1.8 trillion gigabytes of data

was created in 2011…

10,000

5,000

0

Requires capability

for rapid:

Assimilation

Interpretation

Response/Action

GIG

AB

YT

ES

OF

DA

TA

) C

RE

AT

ED

(I

N B

ILL

ION

S)


reserved.


Understanding the Scope of Big Data Big Data enables including all types of data in decision making

models

Successful

companies: Leverage existing

frameworks

Develop new

models

Move quickly and

adapt

• Structured &

Unstructured

• Internal &

External

• Transactional

& Data

Warehouse


reserved.


Oracle’s Vision for Analysis on ALL Data Provide a complete solution allowing you to manage your

business, not complex information technology configurations

Stream Acquire Organize/Discover

Analyze Visualize/Decide


reserved.


Oracle Big Data Platform Accelerate time to market and reduce risk with end-to-end solution

Endeca Information

Discovery

Stream Acquire Organize /Discover

Analyze Visualize /Decide

Oracle is the industry leader in database and information management. With largest global customer base, the power of Oracle provides all the components you

need to get results from your Big Data initiatives broadly and quickly


reserved.


Industry Big Data Use Cases Potential Benefits

Banking & Finance

• Analysis of data sets across lines of business (loans, insurance, on-line banking, card products) for market assessment

• Risk analysis & revenue lift for new & existing products • Analysis of stock portfolio trends & risk

• Increased share of customer • Increased customer loyalty • Increased overall revenue • Decreased financial risk

Healthcare • Analysis of unexpected health condition associations using electronic health records and visualization

• Improved quality of care • Reduced cost of care

On-Line Services &

Social Media

• Advertising performance / optimization • Feature popularity & consumer ratings • People & career matching • Search optimization • Security threat analysis • Troubleshooting

• On-line service loyalty • Better social community experience • More secure and predictable services

Automotive • Analysis of auto sensors reporting location, parts and

component problems • Increased customer safety & loyalty •Minimize warranty claims • Optimize manufacturing processes

Sample of Big Data Use Cases Today Companies across industries are using Big Data insights to grow

their business


reserved.


An example in Banking

Registered Customer using

Internet banking

• Not a Credit Card Customer

• Reviewing Features/benefits

Server Logs show

• Frequent visits to CC pages

• Considerable time spent

BI/DWH identifies …

• this Customer as an HNI

• credit worthiness for offer

• Seek info on preferred channel

• Make an offer

Unstructured data in Server Logs is VALUABLE!!


reserved.


Cross-Industry Collaboration

Retail + Telco

112 113 114 115 117 118

123 122

119 116 120 121

126 125 124 127

Customer enters shopping mall (Telco

captures “high volume” location data from Cell Phone)

Customer Profile: 30-35 Female 2 kids < 5yrs Singed up for coupons

Send Coupon: Proximity to store < 200meters 10% discount if used within next 15 minutes

“Impulsive” Buying Behavior • Coupon used • Increased spend Revenue share with Telco, a Win-Win!

Layout of a Shopping Mall


reserved.


Information

Architectures Today:

Decisions based on

database data

Big Data augments traditional data ..

Driving data-based business decisions

Big Data:

Decisions

based on all

your data

Video and

Images

Machine-

Generated Data

Social

Data

Documents Tapping into

diverse data sets

Finding & monetizing

hidden relationships

Transactions


reserved.


How New, Big Data adds Value?

“I think” “I want”

Retail Decisions

Stores

Web

Search Social

Networks

Catalog/ Call

Center

“I found it”

Looking back “PAST”

Looking ahead “FUTURE”


reserved.


Big Data Opportunity

Through 2015, more than 85

percent of Fortune 500

organizations will fail to effectively

exploit big data for competitive

advantage.

Source: Gartner BI Summit, “Extreme Data: Challenges and Opportunities

for Large-Scale Data Warehousing, BI and Analytics” (May 2011)


reserved.


Oracle Approach


reserved.


Exadata Big Data Appliance Exalytics

Oracle Integrated Solution Stack for Big Data

ACQUIRE ORGANIZE DECIDE ANALYZE

In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)

Oracle Big data Connectors

Oracle Data Integrator

Analytic

Applications,

OBIEE,

Hyperion

HDFS

Enterprise

Applications

Oracle NoSQL

Database

Exadata Exalytics Big Data

Appliance


reserved.


Information Management Optimization

InfiniBand Oracle

Exalytics

Traditional Sources

ELT Platform (ODI for bulk data loads +

Potentially Goldengate for

CDC)

Comprehensive Analytics & Visualisation

Platform

(Retail Analytics, ADI, Exalytics ,OBIEE, BI

Apps)

Database Consolidation

Platform

(Any application on 11.2 databases)

1 / 10 GbE

InfiniBand Oracle

Big Data

Appliance

Big Data Integration Platform • Big Data Connectors

Oracle

Exadata


reserved.



Oracle Big Data Appliance


In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)



Analytic

Applications,

OBIEE,

Hyperion

HDFS

Oracle NoSQL

Database


Appliance


reserved.


Big Data Appliance

• 18 Sun X4270 M2 Servers per Rack

– 864 GB memory

– 216 cores

– 648 TB storage

• 40 Gb/s InfiniBand Fabric

– Inter-rack Connectivity

– Inter-node Connectivity

• 10 Gb/s Ethernet Connectivity

– Data center connectivity

What is Oracle Approach?


reserved.





In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)



Analytic

Applications,

OBIEE,

Hyperion

HDFS

Enterprise

Applications

Oracle NoSQL

Database


Appliance


reserved.


What is Hadoop? • Scalable fault-tolerant distributed system for data storage and processing

– Open source under Apache license

• Enables analysis of Big Data

– Can store huge volumes of unstructured data, e.g.,weblogs, transaction data, social media data

– Enables massive data aggregation

– Highly scalable and robust

– Problems move from processor bound (small data, complex computations) to data bound (huge

data, often simple computations)

• Consists of two key services

1. Hadoop Distributed File System (HDFS)

2. Map-Reduce

• Other Projects based on core Hadoop

– Hive, Pig, HBase, Flume, Sqoop, and others

• Originally sponsored by Yahoo! Apache project Cloudera

• Based on Google's GFS and Big Table whitepaper


reserved.


Hadoop in action

SHUFFLE /SORT

SHUFFLE /SORT

MAP

MAP

MAP

MAP

SHUFFLE /SORT

REDUCE

REDUCE

INPUT 2

INPUT 1

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

OUTPUT 1


reserved.


Data & Processing Flow

SHUFFLE /SORT

SHUFFLE /SORT

MAP

MAP

MAP

MAP

SHUFFLE /SORT

REDUCE

REDUCE

SHUFFLE /SORT

SHUFFLE /SORT

REDUCE

REDUCE

REDUCE

INPUT 2

INPUT 1

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

REDUCE

ORACLE LOADER FOR HADOOP ORACLE BIG DATA APPLIANCE EXADATA


reserved.


Apache Hadoop Apache Sqoop

Apache Hive Apache Mahout

Apache Pig Apache Whirr

Apache HBase Apache Oozie

Apache Zookeeper Fuse-DFS

Apache Flume Hue

Cloudera Hadoop Distribution What is Oracle Approach?

Latest details at: http://www.cloudera.com/hadoop-details/


reserved.


Cloudera Manager

http://www.cloudera.com/wp-content/uploads/2011/12/Cloudera-Manager-DS-3.7-FNL2.pdf























reserved.





In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)



Analytic

Applications,

OBIEE,

Hyperion

HDFS

Enterprise

Applications

Oracle NoSQL

Database


Appliance


reserved.


Oracle NoSQL

• Simple data storage, typically non-SQL or Not-only-SQL for Solution

categories such as

• Online interactive processing

• Social Networks

• Email

• Shopping Cart

• Large data repositories without a fixed schema

• Extract Transform Load batch processing (Hadoop)

• Distributed (Cloud) storage

• Large amounts of data (Terabyte – Petabyte range)

What is the functional need?


reserved.


Oracle NoSQL

• Simple Data Model

– Key-value pair with major+minor-key paradigm

– Read/insert/update/delete

• Scalability

– Dynamic data partitioning and distribution

– Optimized data access via intelligent driver

• High availability

– One or more replicas

– Resilient to partition master failures

– No single point of failure

– Disaster recovery through location of replicas

• Transparent load balancing

– Reads from master or replicas

– Driver is network topology & latency aware


Storage Nodes Data Center A

Storage Nodes Data Center B

NoSQL DB Driver

Application

NoSQL DB Driver

Application


reserved.


Selection Option for Use Case

Hadoop Distributed File System (HDFS)

Oracle NoSQL Database

File System Database

Parallel scanning Indexed storage

No inherent structure Simple data structure

High volume writes High volume random reads and writes


reserved.





In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)



Analytic

Applications,

OBIEE,

Hyperion

HDFS

Enterprise

Applications

Oracle NoSQL

Database


Appliance


reserved.


Oracle Big Data Connectors

1. Connect HDFS to traditional RDBMS

2. Provide ability to access HDFS directly from RDBMS

3. Provide ability to integrate from source file to Hadoop

Cluster to Oracle database visually using a wizard

based approach

4. Allow advanced analytics users to leverage a Hadoop

Cluster with HDFS and MapReduce from the R

environment

What is the functional need?


reserved.



1. Oracle Loader for Hadoop (OLH) – A map/reduce utility for optimized load of data into Oracle Database

– Pre-partition, sort, and transform data into an Oracle ready format on Hadoop and load into

Oracle Database

2. Oracle Direct Connector for Hadoop Distributed File

System – Directly access to data files on HDFS

• Create an external table pointing to file location on HDFS

• Query data from database using SQL

• Load data into database when required



reserved.



3. Oracle Data Integrator Application Adapter for Hadoop – The knowledge modules simplify processing of unstructured and structured data

on Hadoop

– Data Validation and transformation in Hadoop.

– Exporting Hadoop data-sets to Oracle.

4. Oracle R Connector for Hadoop – Allows R users to leverage a Hadoop Cluster with HDFS and MapReduce from

the R environment

– Provides transparent access to Hadoop Cluster: MapReduce and HDFS-resident

data



reserved.


Oracle Loader for Hadoop: Offline/Online Option

SHUFFLE /SORT

SHUFFLE /SORT

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

ORACLE LOADER FOR HADOOP

Read target table metadata from the database

Perform partitioning, sorting, and data conversion

Write from reducer nodes to Oracle Data Pump files

Import into the database in parallel using external table mechanism

DATA

DATA

DATA

DATA

DATA

Copy files from HDFS to a location where database can access them


reserved.


Oracle Direct Connector for HDFS (ODCH)

SHUFFLE /SORT

SHUFFLE /SORT

REDUCE

REDUCE

REDUCE

MAP

MAP

MAP

MAP

MAP

MAP

REDUCE

REDUCE

Directly access data files on HDFS from external tables

DATA

DATA

DATA

DATA

DATA

ANY MAPREDUCE JOB External Table

SQL QUERY

ODCH

DATA

DATA

DATA

DATA

DATA

DATA

DATA

DATA

DATA

ODCH

Directly access data files on HDFS from external tables

• Raw data in delimited text file format

• Data Pump files created by Oracle

Loader for Hadoop (OLH)


reserved.


Selection Option for Use Case

Oracle Loader for Hadoop Output Option Use Case Characteristics

Online load with JDBC The simplest use case for non partitioned

tables

Online load with Direct Path Fast online load for partitioned tables

Offline load with datapump files Fastest load method for external tables

Direct HDFS

Oracle Direct Connector for HDFS Leave data on HDFS

Load into database when needed


reserved.


Oracle R Enterprise What is the functional need?

Open source language and environment Used for statistical computing and graphics Strength in easily producing publication-quality plots Highly extensible with open source community R packages


reserved.


What Are ’s Challenges?

1. R is memory constrained

–R processing is single threaded - does not exploit available

compute infrastructure

–R lacks industrial strength for enterprise use cases

2. R has lacked mindshare in Enterprise market

–R is still met with caution by the long established SAS and

IBM/SPSS statistical community

• However, major university (e.g. Yale ) Statistics courses now taught in R

• The FDA has recently shown indications for approval of new drugs for which

the submission’s data analysis was performed using R

http://www.r-project.org/index.html


reserved.


Oracle R Connector for Hadoop Architecture

*optional

ORE*

Client Host

R Engine

Hadoop

Cluster

Software

R Engine

MapReduce

Nodes

HDFS

Oracle Big Data

Appliance

Oracle Exadata

R Engine ORE*

ORHC ORHC

ORE*


reserved.





In-D

ata

base

An

aly

tics

Oracle

Database +

Options (Oracle R

Enterprise,

OLAP, Spatial,

Partitioning,

RAC, etc.)

Hadoop (MapReduce)



Analytic

Applications,

OBIEE,

Hyperion

HDFS

Enterprise

Applications

Oracle NoSQL

Database


Appliance


reserved.



Knowledge module for Hadoop

© 2011 Oracle Corporation – Proprietary and Confidential

Can we do the integration from source file to Hadoop Cluster to Oracle database visually using a wizard based approach? ( Not too keen to write the map reduce code)


reserved.


Oracle Big Data Appliance

Transforms

Via MapReduce

Loads

Activates

Oracle

Loader for

Hadoop

Oracle Data

Integrator

Oracle Exadata

Oracle Data Integration for Big Data

ODI Hadoop Integration

• New ODI Technology for Hive

• New ODI KM’s for Hive • Reverse from Hive Tables

• File to Hive

• Hive Control Append

• Hive Transform

• Hive to Oracle (OLH)

• Hive is used within KM’s to generate

SQL like calls which are transformed

into Map Reduce statements

Oracle Approach: Improving Productivity and Efficiency for Big Data

http://hadoop.apache.org/


reserved.


Oracle Big Data

Analysis Approach


reserved.


Approach 1: Discovery led Analytics

Data

1. Un-Modeled Data

2. External Data (Low control

on format and access, Low

Quality)

3. Non-Structured Data

4. ..and structured , internal

data

Hadoop, Oracle Connectors and R

Analysis

1. Fast exploration of new and

un anticipated questions.

2. Non structured navigation

paths

3. Analysis on all possible

dimensions (nothing is left

out)


reserved.


Approach 2: Discovery led Analytics

Structured

Semi-Structured

Unstructured

Diverse and changing

information

Automatically unified and

enriched in Endeca Server

– no predefined model

required

Drag-and-drop application

composition

Interactive search,

navigation and analytics

for exploration and

analysis

Endeca Information Discovery

Endeca Information

Discovery


reserved.


Endeca Dashboard

Typical Search …

+

+

1. Auto Indexing

2. All Dimensions across all data

3. Intuitive summaries – refinement counts

4. Tag Clouds (Image and text)

5. No need for pre specified navigation paths

1. Search across structured and Non-STRUCTURED data

2. With minimum effort required for Schema design

3. No summarization .. data up to atomic level of detail is available for analysis.

4. Sentiment Analysis

5. Intuitive business use focused dashboard

6. Visualize anything on a map

+


reserved.


Familiar Methodology, Enhanced Results

Supply

Chain

Merch

andise ERP Web Intra

net Prod Merch ERP

Roster SCM Store CMS WCM

Data Warehouse

Data Marts

ETL

Semantic Layer

Report Layer

Portal Layer

Endeca Server

ITL

Traditional Delivery Endeca Agility Benefits

1. Gather

Requirements,

Define Scope

2. Model, Create,

Load, and

Configure Data

Repository

3. Define Semantics,

Create Reports,

Build Portal

Presentation 4. Administer and

Manage System

1. Incorporate More

Sources, Satisfy

More Users

2. More Flexible Data

Repository

3. Streamlined

Application

Development

4. Low maintenance

and overhead


reserved.


Enterprise Systems

& Data Sources

Data Warehousing

And Data Marts

Next Gen Information Management Platform

Information

Integration

Data Marts

Visualization MicroStrategy

Reports Multi-

Dimensional

Analysis

Data

Warehouse

Traditional BI Reports, Charts

OLAP Cubes

Stores, Merchandise, supply chain

Fly Buy Custom

Applications

ETL Systems (Data Stage / OWB)

Endeca Information Delivery Information

Delivery &

Decision-Making

Non-structured Data Access/Transformation

Log files File System Content Mgt

Systems

Endeca

Server

Exalytics (BI Foundation, In Memory times ten & Essbase)

Goldengate

Pre Built Oracle Analytical Applications, ADI

EXADATA +

Oracle R Enterprise

Big Data Connectors

Big Data Appliance

ODI

Data Quality + MDM (Site,Product,Customer,Supplier)

Web


reserved.


Oracle Integrated Solution Stack Oracle Engineered Systems

ACQUIRE ORGANIZE ANALYZE DECIDE


reserved.


Q&A


reserved.



reserved.


Oracle, שרון עוזיאל

Technology

oracle andor

location data

types of data

big data insights

big data initiatives

unstructured data assimilation

power of oracle

scope of big databig