A Comparison of Join Algorithms for Log Processing in MapReduce Spyros Blanas, Jignesh M. Patel, Vuk Ercegovac, Jun Rao, Eugene J. Shekita, Yuanyuan Tian.

A Comparison of Join Algorithms for Log Processing in

MapReduce

Spyros Blanas, Jignesh M. Patel,Vuk Ercegovac, Jun Rao,

Eugene J. Shekita, Yuanyuan Tian

By Santosh Kumar Nukavarapu

Contents• Introduction

• Requirement

• Log Processing and MapReduce

• Join Algorithms in Map Reduce i) Overview of Repartition Join Algorithm ii)Outlook of Broadcast Join, Semi-Join, Per-Split Semi-Join

• Experimental Evaluation

• Results

• Conclusion and Future Work

Introduction

• Map Reduce is very popular in analysis of large datasets. Positives

• Hide’s the parallelization, fault tolerance and load balancing details through it’s framework.

Negatives ):

• Ignores many concepts of Parallel RDBMs.

• Lack of declarative language, solid schema and indexes.

• Facebook,Yahoo,Google and many Web 2.0 companies are highly interested in Map Reduce.

Why ?

• log processing is very important data analysis that is required by these companies.

• Map Reduce absolutely suit’s their Requirement.

Requirement

Log Processing And Map Reduce

What is Log Processing ?

• Log of events such as click-stream,phone call records or sequence of transactions are collected and are stored in flat files.

•Then these files are processed to compute various statistics to derive some business insights.

Reasons to use Map Reduce for Log Processing :

1.Extremely large amount of Data involved.

Company Data stored per day

China Mobile 5-8 TB

Facebook 6TB

2. Log records do not always follow the same schema.

3. Third, all the log records within a time period are typically analyzed together, making simple scans preferable to index scans.

4. Important to keep the job analysis going even in the event of failures

Parallel RDBMS Map Reduce

Solid Schema not suitable for developers/analysts. ):

Lack of a solid schema

Formatting and Loading this huge amount of data is a challenge. ):

Easily achieved here

Log Records within a time period are analyzed together, so we need simple scans rather than indexes.

Does not support indexing and simple scans can be easily achieved

Failures and processing cannot take place in parallel.

We achieve it here

Problem Specification Problem Required Solution Solutions achieved here

1. Log needs to be joined with Reference data like information of users

So we need to go for joins yes

2. Map Reduce Framework is cumbersome for joins

We need to have some join algorithms for Map Reduce

yes

3. MapReduce programmers use inefficient algorithms

Guidance about correct algorithm at correct time

Yes, but also some work is left for future!

Assumptions made for our JOIN ALGORITHMS IN MAPREDUCE

• We consider an equi-join between a log table L and a reference table R on a single column.

• L,R and the Join Result is stored in DFS.

• Scans are used to access L and R.

• Each map or reduce task can optionally implement two additional functions: init() and close() .

• These functions can be called before or after each map or reduce task.

L L.k=R.k R, with |L| |⊲⊳ ≫R|

Algorithm1 : Repartition Join

Map Phase :

1.Each map task works on a split of either R or L.

2.Each map task tags the record with its originating table.

3.Outputs the extracted join key and the tagged record as a (key, value) pair.

4.The outputs are then partitioned, sorted and merged by the framework.

Reducer Phase :

1. All the records for each join key are grouped together and eventually fed to a reducer.

2. For each join key, the reduce function first separates and buffers the input records into two sets according to the table tag.

3. Performs a cross-product between records in the above sets.

Problem with this version of Algorithm :

• All the records for a given join key from both L and R have to be buffered.• So, we can be out of memory ):

Improvement to Re-partition joinPhase /Function Improvement

Map Function output key is changed to a compositeof the join key and the table tag.

partitioning function Hashcode is computed from just the join key part of the composite key

Grouping function records are grouped on just thejoin key

Algorithm Problem or Case Key Method/Point for achieving solution

Broadcast Join If the reference table R is much smallerthan the log table L, i.e. |R| |L|≪

Broadcast the smaller table R, as it avoids sorting on both tables and more importantly avoids the network overhead for moving the larger table L.

Semi-Join R is large, many records in R may not be actually referenced by any records in table L

In the map function, a main-memory hash table is used to determine the set of unique join keys in a split of L.

Per-Split Semi-Join One problem with semi-join is that not every record in the filtered version of R will join with a particular split Li of L.

Moves just therecords in R that will join with each split of L.

Experimental Evaluation

Hardware/software/procedure Configuration /version

Cluster 100 node

Each node 2.4GHz Intel Core 2 Duo processor4GB of DRAMtwo SATA disks

Operating System Red Hat EnterpriseServer 5.2 running Linux 2.6.18

2 Racks own gigabit Ethernet switch

rack level bandwidth 32Gb/s

Hadoop version 0.19.0configured it to runup to two map and two reduce tasks concurrently per node

block size 128 Mb

Replication Each HDFs block was replicated 3 times

Results

Picture taken from : A Comparison of Join Algorithms for Log Processing in MapReduce by Spyros Blanas, Jignesh M. Patel, Vuk Ercegovac, Jun Rao,Eugene J. Shekita, Yuanyuan Tian.

Picture taken from : A Comparison of Join Algorithms for Log Processing in MapReduce by Spyros Blanas, Jignesh M. Patel, Vuk Ercegovac, Jun Rao,Eugene J. Shekita, Yuanyuan Tian.

Conclusion

•Joining log data with all kinds of reference data in MapReduce has emerged as an important part of analytic operations for : 1. Enterprise customers 2. Web 2.0 companies

•Evaluated the join methods on a 100-node system.•Shown Unique tradeoffs of these join algorithms in the context of MapReduce.•Study can help an optimizer select the appropriate algorithm based on data.

Future Work

•Evaluating methods for multi-way joins.•Exploring indexing methods to speedup join queries, •Designing an optimization module that can automatically select the appropriate join algorithms.

References

• Google labs

• A Comparison of Join Algorithms for Log Processing in MapReduce by Spyros Blanas, Jignesh M. Patel, Vuk Ercegovac, Jun Rao,Eugene J. Shekita, Yuanyuan Tian.

• Wikipedia

• Ibm.com