Advanced topics on Mapreduce with Hadoop
Jiaheng Lu
Department of Computer Science
Renmin University of Chinawww.jiahenglu.net
Outline
Brief Review Chaining MapReduce Jobs Join in MapReduce Bloom Filter
Brief Review
A parallel programming framework Divide and merge
split0
split1
split2
Input data
Map task
Mappers
Map task
Map task
Shuffle
Reduce task
Reducers
Reduce task
Output data
output0
output1
Chaining MapReduce jobs
Chaining in a sequence Chaining with complex dependency Chaining preprocessing and postprocessing
steps
Chaining in a sequence
Simple and straightforward [MAP | REDUCE]+; MAP+ | REDUCE | MAP* Output of last is the input to the next Similar to pipes
Configuration conf = getConf();
JobConf job = new JobConf(conf);
job.setJobName("ChainJob");
job.setInputFormat(TextInputFormat.class);
job.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(job, in);
FileOutputFormat.setOutputPath(job, out);
JobConf map1Conf = new JobConf(false);
ChainMapper.addMapper(job, Map1.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf);
Chaining with complex dependency
Jobs are not chained in a linear fashion
Use addDependingJob() method to add dependency information:
x.addDependingJob(y)
Chaining preprocessing and postprocessing steps
Example: remove stop word in IR Approaches:
Separate: inefficient Chaining those steps into a single job
Use ChainMapper.addMapper() and ChainReducer.setReducer
Map+ | Reduce | Map*
Join in MapReduce
Reduce-side join Broadcast join Map-side filtering and Reduce-side join
A given key A range from dataset(broadcast) a Bloom filter
Reduce-side join
Map output <key, value> key>>join key, value>>tagged with data source
Reduce do a full cross-product of values output the combination results
Example
a b
1 ab
1 cd
4 ef
a c
1 b
2 d
4 c
table x
table y
map()
map()
1
4
key
x ab
x cd
x ef
value
1
2
4
key
y b
y d
y c
valuetag
join key
shuffle()
1
key
x ab
x cd
y b
valuelist
2 y d
4x ef
y c
reduce()
a b c
1 ab b
1 cd b
4 ef c
output
1
Broadcast join (replicated join)
Broadcast the smaller table Do join in Map()
Using distributed cache
DistributedCache.addCacheFile()
Map-side filtering and Reduce-side join
Join key: student IDs from info generate IDs file from info broadcast join
What if the IDs file can’t be stored in memory? a Bloom Filter
A Bloom Filter
Introduction Implementation of bloom filter Use in MapReduce join
Introduction to Bloom Filter
space-efficient data structure, constant size, test elements, add(), contains()
no false negatives and a small probability of false positives
Implementation of bloom filter
Apply a bit array Add elements
generate k indexes set the k bits to 1
Test elements generate k indexes all k bits are 1 >> true, not all are 1 >> false
Example
0
0
0
0
0
0
0
0
0
0
0
1
2
3
4
5
6
7
8
9
1
0
1
0
0
0
1
0
0
0
0
1
2
3
4
5
6
7
8
9
add x(0,2,6)
1
0
1
1
0
0
1
0
0
1
0
1
2
3
4
5
6
7
8
9
add y(0,3,9)
1
0
1
1
0
0
1
0
0
1
0
1
2
3
4
5
6
7
8
9
contain m(1,3,9)
1
0
1
1
0
0
1
0
0
1
0
1
2
3
4
5
6
7
8
9
contain n(0,2,9)initial state
① ② ③ ④ ⑤
× √false positives
Use in MapReduce join
A separate subjob to create a Bloom Filter
Broadcast the Bloom Filter and use in Map() of join job
drop the useless record, and do join in reduce
References
Chunk Lam, “Hadoop in action” Jairam Chandar, “Join Algorithms using
Map/Reduce”
THANK YOU
Hadoop
