Hadoop MapReduce

SEMINAR ONSEMINAR ONAndroid App DevelopmentAndroid App Development

Trained by-Trained by-Hewlett-Packard Education Services, Hewlett-Packard Education Services,

MumbaiMumbai

Presented to-Mr. R.K. Banyal By-Mr. Hukum Chand Saini Urvashi Kataria

About HPES:About HPES:• American global IT company headquartered in Palo-

Alto, California, US.• Provider of products, soft wares, technologies,

solutions and services to individual as well as small & medium sized business.

• Major operations include- HP Software, HP Financial Services & Corporate Investments

• Provides practical training in fields like Big Data, Android App Dev, Embedded Systems etc.

An android application that allows you to enjoy your as well as your dear ones birthday.

Save the days, get reminded of them, capture moments on the day itself, get greeted by the app, and celebrate!!

About Birthday Bash:About Birthday Bash:

The home screen:The home screen:

Calculating age and Calculating age and further:further:

Saving name for specified Saving name for specified date:date:

Happy Birthday!Happy Birthday!

Hadoop Map Reduce

(Map + reduce)

Presentation on:Presentation on:

Why MapReduce?Why MapReduce?• Large scale data processing was difficult!

Managing hundreds or thousands of processors

Managing parallelization and distribution

Reliable execution with easy data access

MapReduce provides all of these, easily!

What is Hadoop MapReduce?What is Hadoop MapReduce?

Hadoop ClusterHadoop Cluster HDFS (Physical) HDFS (Physical) StorageStorage

MapReduce ObjectsMapReduce Objects

How Map and Reduce Work How Map and Reduce Work TogetherTogether

Hadoop MapReduce: A Closer Hadoop MapReduce: A Closer LookLook

file

file

InputFormat

Split Split Split

RR RR RR

Map Map Map

Input (K, V) pairs

PartitionerIntermediate (K, V) pairs

Sort

Reduce

OutputFormat

Files loaded from local HDFS store

RecordReaders

Final (K, V) pairs

Writeback to local HDFS store

file

file

InputFormat

Split Split Split

RR RR RR

Map Map Map

Input (K, V) pairs

PartitionerIntermediate (K, V) pairs

Sort

Reduce

OutputFormat

Files loaded from local HDFS store

RecordReaders

Final (K, V) pairs

Writeback to local HDFS store

Node 1 Node 2

Shuffling Process

Intermediate (K,V) pairs

exchanged by all nodes

AlgorithmAlgorithmmap(key, value):// key: document name; value: text of document

for each word w in value:emit(w, 1)

reduce(key, values):// key: a word; values: an iterator over counts

result = 0for each count v in values:result += vemit(key,result)

map(key=url, val=contents): for each word w in contents:

emit (w, “1”)reduce(key=word, values=uniq_counts)://Sum all “1”s in values list

emit result “(word, sum)”

The very famous:The very famous:Word Count ExampleWord Count Example

Ways to MapReduceWays to MapReduce

Libraries Languages

Note: Java is most common, but other languages can be used

Common Data Sources Common Data Sources for MapReduce Jobsfor MapReduce Jobs

Service ProvidersService Providers• Open Source

o Apache

• Commercialo Clouderao Hortonworkso MapRo AWS MapReduceo Microsoft HDInsight (Beta)

Advancements:Advancements:MRV1 & MRV2MRV1 & MRV2

MRV2 (MAPREDUCE VERSION 2)•Splits the existing JobTracker’s roles

o Resource managemento Job lifecycle management

•MapReduce 2.0 provides many benefits over the existing MapReduce framework:

o Better scalability o Through distributed job lifecycle management o Support for multiple Hadoop MapReduce API versions in a

single cluster

Better MapReduce - Better MapReduce - OptimizationsOptimizations

Advantages of MapReduceAdvantages of MapReduce

• Distributed data and computation.• Tasks are independent. Entire nodes can fail and restart.• Linear scaling in the idle case. It’s used to design cheap

commodity, hardware.• Simple programming model. The end-user programmer

only writes map reduce task.

Disadvantages/ Cases where Disadvantages/ Cases where MR isn’t a suitable choice:MR isn’t a suitable choice:

• Real time processing• It is not always very easy to implement each and every

thing as a map reduce program• When your intermediate processes need to talk to each

other • When your processing requires lot of data to be shuffled

over the network• When you need to handle streaming data. MR is best suited

to batch process huge amount of data which you already have

Limitations of Limitations of MapReduceMapReduce

RDBMS vs. RDBMS vs. HadoopHadoop

Traditional RDBMS Hadoop / MapReduce

Data Size Gigabytes (Terabytes) Petabytes (Hexabytes)

Access Interactive and Batch Batch – NOT Interactive

Updates Read / Write many times

Write once, Read many times

Structure Static Schema Dynamic Schema

Integrity High (ACID) Low

Scaling Nonlinear Linear

Query Response Time

Can be near immediate Has latency (due to batch processing)

ReferencesReferences• J. Dean and S. Ghemawat. “MapReduce: Simplified Data

Processing on Large Clusters.” Proceedings of the 6th Symposium on Operating System Design and Implementation (OSDI 2004), pages 137-150. 2004.

• S. Ghemawat, H. Gobioff, and S.-T. Leung. “The Google File System.” OSDI 200?

• http://hadoop.apache.org/common/docs/current/mapred_tutorial.html. “Map/Reduce Tutorial”. Fetched January 21, 2010.

• Tom White. Hadoop: The Definitive Guide. O'Reilly Media. June 5, 2009

• http://developer.yahoo.com/hadoop/tutorial/module4.html• J. Lin and C. Dyer. Data-Intensive Text Processing with

MapReduce, Book Draft. February 7, 2010.

Thank You!!Thank You!!