Introduction to the Hortonworks YARN Ready Program

Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Introduction to YARN Ready

John Kreisa – VP Strategic MarketingAjay Singh - Director, Partner Solutions


YARN Ready Program OverviewJohn Kreisa


Why YARN Ready? -> Apache Hadoop YARN

• YARN is the architecture center of Apache Hadoop 2 enabling more efficient cluster utilization

• More and more customers are asking about tools and applications that are integrated into YARN to maximize the value of their Hadoop cluster

• YARN Ready is a program offered by Hortonworks the leading developers of YARN

– YARN was conceived of by Arun Murthy Hortonworks founder– He submitted JIRA MapReduce 279 in January 2008 and has been working on it since– Hortonworks has written 80% of the YARN code

Page 3


The Two YARN Ready Program Goals

Expand the Hadoop ecosystem by enabling ISVs to build & validate YARN-based applications, using Apache technologies

Allow enterprises to know which technologies maximize Hadoop resources in an ecosystem-friendly way


Two Achievements Available

HDP Certified

YARN Ready

Badge Description

HDP (Certification) Integrates with HDP 2.x

YARN Ready (Validation) Uses the YARN resource scheduler


YARN Technical Overview and ValueAjay Singh


Hadoop 1.0: Batch

HADOOP 1.0Built for Web-Scale Batch Apps

Single App

BATCH

HDFS

Single App

INTERACTIVE

Single App

BATCH

HDFS

• All other usage patterns must leverage that same infrastructure

• Forces the creation of silos for managing mixed workloads

Single App

BATCH

HDFS

Single App

ONLINE


Hadoop 2.0: Multi Workload

HADOOP 1.0

HDFS(redundant, reliable storage)

MapReduce(cluster resource management

& data processing)

HDFS2(redundant, reliable storage)

YARN(cluster resource management)

MapReduce(data processing)

Others(data processing)

HADOOP 2.0

Single Use SystemBatch Apps

Multi Purpose PlatformBatch, Interactive, Online, Streaming, …


Key Driver Of Hadoop Adoption: Enterprise Data Lake

FlexibleEnables other purpose-built data processing models beyond MapReduce (batch), such as interactive and streaming

EfficientDouble processing IN Hadoop on the same hardware while providing predictable performance & quality of service

SharedProvides a stable, reliable, secure foundation and shared operational services across multiple workloads

Data Processing Engines Run Natively IN HadoopBATCH

MapReduceINTERACTIVE

TezSTREAMING

StormIN-MEMORY

SparkGRAPHGiraph

ONLINEHBase, Accumulo

OTHERS

HDFS: Redundant, Reliable Storage

YARN: Cluster Resource Management


YARN Development Framework

YARN : Data Operating System

°1 ° ° ° ° ° ° ° °

° ° ° ° ° ° ° ° °

°

°°

° ° ° ° ° ° °

° ° ° ° ° ° N

HDFS (Hadoop Distributed File System)

System

BatchMapReduce

InteractiveTez

Engine Long RunningSlider

Direct

ISV Apps

Scripting

Pig

SQL

Hive

Cascading

JavaScala

NoSQL

HBaseAccumulo

Stream

StormAPIISV

AppsISVAps

Applications

Others

Spark ISV Apps

ISVApps


Tez – Next Generation Data Processing Framework

YARN ApplicationMaster to run DAG of Tez Tasks

Task with pluggable Input, Processor and Output

Tez Task - <Input, Processor, Output>

Task

ProcessorInput Output


Slider – Onboarding Long Running Services To YARN

A YARN application to deploy existing distributed applications, monitor them and make them larger or smaller as desired - even while the cluster is running

Has a plug-in provider architecture to support different applications

Automatic Deploy, Manage, Monitor via Ambari– Standard management APIs for Ambari to use

– Standard ways to collect metrics

Clusters can be stopped, "frozen" and restarted, "thawed" later– Distribution of the deployed application across the YARN cluster is persisted.


# of Apps / Ease of Integration

Con

trol

Integration Option Tradeoffs

YARN Native Apps

Tez Apps

Services on YARN (Slider)

MapReduce Apps

Hive


HADOOP

YARN

HDFS

SQL

Datanode

HDFS

Visual Data & Analytics Workflow

Actian Analytics Platform – YARN Native Integration ExampleTransform Hadoop into a High Performance Analytics Platform

Datanode

HDFS

Datanode

HDFS

Datanode

HDFS

X100X100X100

ReadLoad

Actian VectorBlend &Enrich

Data Science & Analytics

Datanode

HDFS

X100

HDFS

Vector

• Original file format• Standard block

replication

• Column-based blocks

• Binary• Compressed• Partitioned

• Faster Loading• Faster SQL• Standard SQL• Better Scaling

NamenodeHigh Performance, Industrialized SQL

Database

High Performance, Data Science &

Analytics


Partners with YARN Ready Apps


YARN Ready Program DetailsJohn Kreisa


Partner Benefits - Marketing

Recognition Recognition as Hortonworks Certified Technology Partner on our website along with the YARN Ready

badge

Hortonworks quote for reference in partner marketing assets

Promotion to customers in Hortonworks led events and activities

Joint Marketing & Field Alignment

Invitation to Modern Data Architecture Webinar Series

Hortonworks Sales Briefing and alignment with Field BD

Automatic participation in digital marketing initiatives

LinkedIn and FB campaigns

Twitter and blog activity


Partner Benefits - Technical Tech Expertise YARN implementation guide and sample code

Integration best practice guidance

Access to the architects and builders of YARN

Roadmap Input YARN, Tez and Stinger roadmap visibility

Input into the direction of YARN and the YARN engines

Access to pre-release builds to speed integration with new features


Process for Partners

Become a Hortonworks partner Sign up at http://hortonworks.com/partners/become-a-partner/

Become Certified Download the certification kit (a .zip archive) from the partner portal

Validate with HDP (Sandbox OK)

Complete the provided templates (samples are included) for

Solution Overview

Reference Architecture

Certification Report

Submit the three documents to [email protected]. SLA: 10 working days for Hortonworks response.

Collaborate with Hortonworks on joint marketing – via [email protected]


Certification Kit

A .zip file available to partners

Contents:

Program Guide

Solution Overview template (DOC)

Reference Architecture template (PPT)

Certification Report template & sample (DOC)

Sample YARN App code*

Sample Slider provider code*

YARN app test recommendations*

Logos (available after certification)

PR Template (available after certification)

README file (text)

*YARN Ready, Coming – target July


Calls to Action

Engage us on your YARN initiative

Become HDP Certified and YARN Ready

Develop a joint Sandbox Tutorial

Collaborate on “Art of Possible” Demonstration

Co-Market & Co-Sell


Questions?Join us for the next series of webinars on integrating to YARN:YARN native July 24 @ 9am PTSlider integration August 7 @ 9am PTTez integration August 21 @ 9am PT

Page 22

https://hortonworks.webex.com/hortonworks/onstage/g.php?t=a&d=625466603




Thank You

Introduction to the Hortonworks YARN Ready Program

Technology

apache hadoop yarn yarn

yarn ready apps

hadoop cluster yarn

batch hadoop

rights reserved tez

yarnbased applications

yarn resource scheduler

data processing hadoop