A new platform for a new era emc

A NEW PLATFORM FOR A NEW ERA

O

Shahaf Azriely

Sr. Field Engineer, Israel Pre-Sale Manager SEMEA

© Copyright 2013 Pivotal. All rights reserved.© Copyright 2013 Pivotal. All rights reserved.

Who is Pivotal?


Introducing Pivotal

Led by CEO, Paul Maritz, former CEO of VMware

Redefining Enterprise Platform-as-a-Service

Enabling a new class of applications, leveraging big & fast data, with the power of cloud independence

© Copyright 2013 Pivotal. All rights reserved.

Integrating EMC and VMWare Assets

Cloud Storage

Virtualization

Pivotal DataFabric

Pivotal CloudFabric

Data-DrivenApplication

Development

Pivotal Data Science Labs

...ETC


Pivotal Data Fabric

Cloud Storage

Virtualization

Pivotal CloudFabric

Data-DrivenApplication

Development

Pivotal Data Science Labs

...ETC

Pivotal DataFabric


Enterprise Data Architecture

AnalyticData Marts

MPP Database

OperationalIntelligence

In-Memory DB

Run-TimeApplications

In-Memory Object

Enterprise Data WarehouseRDBMS

Data StagingPlatformData

IngestionSystem

Stream/CEP


AnalyticData Marts

OperationalIntelligence

Run-TimeApplications

Enterprise Data Warehouse

Data StagingPlatformData Ingestion

System

Pivotal Data Portfolio Today


Multi-Target Deployment Model

depl

oy

Portable

Elastic

Promotable

HW abstracted

Manageable

Public Cloud

Private Cloud

On Premise


PIVOTAL HDThe Foundation for Change


Our Big Bets for the Future

1. HDFS becomes the data substrate for the next generation of data infrastructures

2. A set of integrated, enterprise-scale services will evolve on top of HDFS – stream ingestion, analytical processing, and transactional serving

3. Provisioning flexibility and elasticity become critical capabilities for this data infrastructure


Did You Know?

Our HD distribution has been scale-tested on our unique, 1,000-node Analytics Work Bench

Our distribution is the first to bundle VMWare’s Hadoop Virtualization Extensions (HVE)

We are backed by EMC’s global, 24x7 support infrastructure

Available as a software-only or appliance-based solution


Hadoop Pain Points

• No Integrated Hadoop Stack• Hadoop, Pig, Hive, HBase, Zookeeper, Oozie, Mahout…Integrated Product Suite

• No Industry standard ETL and BI Stack Integration• Informatica, Microstrategy, Business Objects …Interoperability

• Poor Job and Application Monitoring Solution• Non-existent Performance MonitoringMonitoring

• Complex System Configuration and Manageability• No Data Format Interoperability & Storage Abstractions

Operability and Manageability

• Poor Dimensional Lookup Performance• Very poor Random Access and Serving PerformancePerformance


The Pivotal Position on Hadoop

Hadoop fits Pivotal’s strategy based on open source innovation for Big Data analytics

– Hadoop and Pivotal are complementary technologies

Hadoop needs to become mission-critical and easier to use and manage for enterprise customers

– Lacks operational interfaces and high-level tooling for big data analysis

– Pivotal HD addresses these challenges offering robust operational tools and with Advanced Database Services powered by HAWQ

– HAWQ is the first true SQL processing engine that runs on Hadoop

Why Hadoop?


Pivotal HD Enterprise1.0

Commercially supported distribution of Apache Hadoop 2.0 – HDFS, MapReduce 2.0, YARN, Pig, Hive, HBase,

Mahout, Zookeeper, Flume, Sqoop, Hadoop Virtualization Extensions (HVE)

– Spring Hadoop integrates the Spring Framework into Hadoop

▪ Create and run Hadoop MapReduce, Hive and Pig jobs▪ Work with HDFS and HBase

Open Source Apache Stack


Pivotal HD Open Source Components

•Hadoop Distributed File System HDFS•Processing framework for writing scalable data applicationsMapReduce•Procedural language that abstracts lower level MapReducePig•Highly reliable distributed coordinationZookeeper•System for querying data on top of HDFS (SQL-like query)Hive•Database for random, real time read/write accessHBase•Scalable machine learning librariesMahout


Pivotal HD Components

•Cluster installation, upgrade and expansion tools ICM

•Visual interface to monitor jobs, cluster health, system metricsCommand Center

•Supports virtual node awareness HVE

•Virtual resource partitioning and performance monitoringMore-VRP

•Enterprise grade NAS-based storage option for HadoopIsilon Integration

•SQL query processor based on GPDB running on HDFSHAWQ

•Extension Framework component of HAWQ to create external tablesGPXF


Command

Center

&

More-VRP

ICM Deployment&

Configuration

DataLoader

XtensionFramework

CatalogServices

QueryPlanner

Dynamic Pipelining

HAWQ

HDFSHadoop Virtualization

HBase

Pig, Hive & Mahout

Map Reduce

Sqoop Flume

Resource Management & Workflow

Yarn

Zookeeper

Chorus

Partner Tools and Applications

Spring

Spring Data Framework

ANSI SQL + Analytics

Collaboration & Orchestration

Applications

Apache Pivotal HD Added Value Pivotal Partners

Pivotal HD

Cetas

MoreVRP

Pivotal HD Architecture


Powerful Partner Ecosystem


Powerful Partner Ecosystem


Use Cases

Pivotal HD & HAWQ GA will come available by end of 05/13.

1. Retail – leavreging for an enterprise data lake. All data will flow into PivHD HDFS. Some will be loaded into HAWQ.

2. Telco – petabytes of data with network/cell phone tower data will be stored in PivHD and HAWQ for faster analytics.

3. Financial – migration from GPDB to leverage GPXF to allow interconnection to Hbase.

More information Under NDA


L E A R N M O R E

goPivotal.com F O L L O W U S

@gopivotal

Shahaf [email protected]

mailto:[email protected]

A new platform for a new era emc

Technology

pivotal position

pivotal hd enterprise1

pivotal hdthe foundation

big data analysis pivotal

integrated hadoop stack

querying data

data substrate

hadoop mapreduce