Jan 26, 2015
1
A Guide to Python Frameworks for HadoopUri [email protected] March 2014
2
Goals for today
1. Easy to jump into Hadoop with Python2. Describe 5 ways to use Python with Hadoop, batch
and interactive3. Guidelines for choosing Python framework
3
Code:https://github.com/laserson/rock-health-python
Blog post:http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/
Slides:http://www.slideshare.net/urilaserson/
4
About the speaker
• Joined Cloudera late 2012• Focus on life sciences/medical
• PhD in BME/computational biology at MIT/Harvard (2005-2012)
• Focused on genomics• Cofounded Good Start Genetics (2007-)
• Applying next-gen DNA sequencing to genetic carrier screening
5
About the speaker
• No formal training in computer science• Never touched Java• Almost all work using Python
6
7
Python frameworks for Hadoop
• Hadoop Streaming• mrjob (Yelp)• dumbo• Luigi (Spotify)• hadoopy• pydoop• PySpark• happy• Disco• octopy• Mortar Data• Pig UDF/Jython• hipy• Impala + Numba
8
Goals for Python framework
1. “Pseudocodiness”/simplicity2. Flexibility/generality3. Ease of use/installation4. Performance
9
Python frameworks for Hadoop
• Hadoop Streaming• mrjob (Yelp)• dumbo• Luigi (Spotify)• hadoopy• pydoop• PySpark• happy• Disco• octopy• Mortar Data• Pig UDF/Jython• hipy• Impala + Numba
10
Python frameworks for Hadoop
• Hadoop Streaming ✓• mrjob (Yelp) ✓• dumbo• Luigi (Spotify) ✓• hadoopy• pydoop• PySpark✓• happy abandoned? Jython-based• Disco not Hadoop• octopy not serious/not Hadoop• Mortar Data HaaS; support numpy, scipy, nltk, pip-installable in UDF• Pig UDF/Jython Pig is another talk; Jython limited• hipy Python syntactic sugar to construct Hive queries• Impala + Numba✓
11
An n-gram is a tuple of n words.
Problem: aggregating the Google n-gram datahttp://books.google.com/ngrams
12
An n-gram is a tuple of n words.
Problem: aggregating the Google n-gram datahttp://books.google.com/ngrams
1 2 3 4 5 6 7 8
( )
8-gram
13
"A partial differential equation is an equation that contains partial derivatives."
14
A partial differential equation is an equation that contains partial derivatives.
A 1partial 2differential 1equation 2is 1an 1that 1contains 1derivatives. 1
1-grams
15
A partial differential equation is an equation that contains partial derivatives.
A partial 1partial differential 1differential equation 1equation is 1is an 1an equation 1equation that 1that contains 1contains partial 1partial derivatives. 1
2-grams
16
A partial differential equation is an equation that contains partial derivatives.
A partial differential equation is 1partial differential equation is an 1differential equation is an equation 1equation is an equation that 1is an equation that contains 1an equation that contains partial 1equation that contains partial derivatives. 1
5-grams
17
18
goto code
19
flourished in 1993 2 2 2flourished in 1998 2 2 1flourished in 1999 6 6 4flourished in 2000 5 5 5flourished in 2001 1 1 1flourished in 2002 7 7 3flourished in 2003 9 9 4flourished in 2004 22 21 13flourished in 2005 37 37 22flourished in 2006 55 55 38flourished in 2007 99 98 76flourished in 2008 220 215 118fluid of 1899 2 2 1fluid of 2000 3 3 1fluid of 2002 2 1 1fluid of 2003 3 3 1fluid of 2004 3 3 3
2-gram year matches pages volumes
20
Compute how often two words are near each other in a given year.
Two words are “near” if they are both present in a 2-, 3-, 4-, or 5-gram.
21
...2-grams...(cat, the) 1999 14(the, cat) 1999 7002
...3-grams...(the, cheshire, cat) 1999 563
...4-grams...
...5-grams...(the, cat, in, the, hat) 1999 1023(the, dog, chased, the, cat) 1999 403(cat, is, one, of, the) 1999 24
(cat, the) 1999 8006(hat, the) 1999 1023
raw data
aggregated results
lexicographicordering
internal n-grams counted by smaller n-grams:• avoids double-counting• increases sensitivity (observed at least 40 times)
22
What is Hadoop?
• Ecosystem of tools• Core is the HDFS file system• Downloadable set of jars that can be run on any
machine
23
HDFS design assumptions
• Based on Google File System• Files are large (GBs to TBs)• Failures are common
• Massive scale means failures very likely• Disk, node, or network failures
• Accesses are large and sequential• Files are append-only
24
HDFS properties
• Fault-tolerant• Gracefully responds to node/disk/network failures
• Horizontally scalable• Low marginal cost
• High-bandwidth
1
2
3
4
5
2
4
5
1
2
5
1
3
4
2
3
5
1
3
4
Input File
HDFS storage distributionNode A Node B Node C Node D Node E
25
MapReduce computation
26
MapReduce computation
• Structured as1. Embarrassingly parallel “map stage”2. Cluster-wide distributed sort (“shuffle”)3. Aggregation “reduce stage”
• Data-locality: process the data where it is stored• Fault-tolerance: failed tasks automatically detected
and restarted• Schema-on-read: data must not be stored conforming
to rigid schema
27
Pseudocode for MapReduce
def map(record): (ngram, year, count) = unpack(record) // ensure word1 has the lexicographically first word: (word1, word2) = sorted(ngram[first], ngram[last]) key = (word1, word2, year) emit(key, count)
def reduce(key, values): emit(key, sum(values))
All source code available on GitHub:https://github.com/laserson/rock-health-python
28
Native Java
import org.apache.hadoop.conf.Configured;import org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.mapreduce.Job;import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;import org.apache.hadoop.util.Tool;import org.apache.hadoop.util.ToolRunner;
public class NgramsDriver extends Configured implements Tool {
public int run(String[] args) throws Exception { Job job = new Job(getConf()); job.setJarByClass(getClass()); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.setMapperClass(NgramsMapper.class); job.setCombinerClass(NgramsReducer.class); job.setReducerClass(NgramsReducer.class); job.setOutputKeyClass(TextTriple.class); job.setOutputValueClass(IntWritable.class); job.setNumReduceTasks(10); return job.waitForCompletion(true) ? 0 : 1; } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new NgramsDriver(), args); System.exit(exitCode); }}
import java.io.IOException;import java.util.ArrayList;import java.util.Collections;import java.util.List;import java.util.regex.Matcher;import java.util.regex.Pattern;
import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.io.LongWritable;import org.apache.hadoop.io.Text;import org.apache.hadoop.mapreduce.Mapper;import org.apache.hadoop.mapreduce.lib.input.FileSplit;import org.apache.log4j.Logger;
public class NgramsMapper extends Mapper<LongWritable, Text, TextTriple, IntWritable> { private Logger LOG = Logger.getLogger(getClass()); private int expectedTokens; @Override protected void setup(Context context) throws IOException, InterruptedException { String inputFile = ((FileSplit) context.getInputSplit()).getPath().getName(); LOG.info("inputFile: " + inputFile); Pattern c = Pattern.compile("([\\d]+)gram"); Matcher m = c.matcher(inputFile); m.find(); expectedTokens = Integer.parseInt(m.group(1)); return; } @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] data = value.toString().split("\\t"); if (data.length < 3) { return; } String[] ngram = data[0].split("\\s+"); String year = data[1]; IntWritable count = new IntWritable(Integer.parseInt(data[2])); if (ngram.length != this.expectedTokens) { return; } // build keyOut List<String> triple = new ArrayList<String>(3); triple.add(ngram[0]); triple.add(ngram[expectedTokens - 1]); Collections.sort(triple); triple.add(year); TextTriple keyOut = new TextTriple(triple); context.write(keyOut, count); }}
import java.io.IOException;
import org.apache.hadoop.io.IntWritable;import org.apache.hadoop.mapreduce.Reducer;
public class NgramsReducer extends Reducer<TextTriple, IntWritable, TextTriple, IntWritable> { @Override protected void reduce(TextTriple key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable value : values) { sum += value.get(); } context.write(key, new IntWritable(sum)); }}
import java.io.DataInput;import java.io.DataOutput;import java.io.IOException;import java.util.List;
import org.apache.hadoop.io.Text;import org.apache.hadoop.io.WritableComparable;
public class TextTriple implements WritableComparable<TextTriple> { private Text first; private Text second; private Text third; public TextTriple() { set(new Text(), new Text(), new Text()); } public TextTriple(List<String> list) { set(new Text(list.get(0)), new Text(list.get(1)), new Text(list.get(2))); } public void set(Text first, Text second, Text third) { this.first = first; this.second = second; this.third = third; } public void write(DataOutput out) throws IOException { first.write(out); second.write(out); third.write(out); }
public void readFields(DataInput in) throws IOException { first.readFields(in); second.readFields(in); third.readFields(in); }
@Override public int hashCode() { return first.hashCode() * 163 + second.hashCode() * 31 + third.hashCode(); } @Override public boolean equals(Object obj) { if (obj instanceof TextTriple) { TextTriple tt = (TextTriple) obj; return first.equals(tt.first) && second.equals(tt.second) && third.equals(tt.third); } return false; } @Override public String toString() { return first + "\t" + second + "\t" + third; }
public int compareTo(TextTriple other) { int comp = first.compareTo(other.first); if (comp != 0) { return comp; } comp = second.compareTo(other.second); if (comp != 0) { return comp; } return third.compareTo(other.third); } }
29
Native Java
• Maximum flexibility• Fastest performance• Native to Hadoop• Most difficult to write
30
Hadoop Streaming
hadoop jar hadoop-streaming-*-.jar \ -input path/to/input -output path/to/output -mapper “grep WARN”
31
Hadoop Streaming: features
• Canonical method for using any executable as mapper/reducer
• Includes shell commands, like grep• Transparent communication with Hadoop though
stdin/stdout• Key boundaries manually detected in reducer• Built-in with Hadoop: should require no additional
framework installation• Developer must decide how to encode more
complicated objects (e.g., JSON) or binary data
32
Hadoop Streaming
goto code
33
mrjob
class NgramNeighbors(MRJob): # specify input/intermed/output serialization # default output protocol is JSON; here we set it to text OUTPUT_PROTOCOL = RawProtocol
def mapper(self, key, line): pass def combiner(self, key, counts): pass def reducer(self, key, counts): pass
if __name__ == '__main__': # sets up a runner, based on command line options NgramNeighbors.run()
34
mrjob: features
• Abstracted MapReduce interface• Handles complex Python objects• Multi-step MapReduce workflows• Extremely tight AWS integration• Easily choose to run locally, on Hadoop cluster, or on
EMR• Actively developed; great documentation
35
mrjob
goto code
36
mrjob: serialization
class MyMRJob(mrjob.job.MRJob): INPUT_PROTOCOL = mrjob.protocol.RawValueProtocol INTERNAL_PROTOCOL = mrjob.protocol.JSONProtocol OUTPUT_PROTOCOL = mrjob.protocol.JSONProtocol
Defaults
RawProtocol / RawValueProtocolJSONProtocol / JSONValueProtocolPickleProtocol / PickleValueProtocolReprProtocol / ReprValueProtocol
Available
Custom protocols can be written.No current support for binary serialization schemes.
37
luigi
• Full-fledged workflow management, task scheduling, dependency resolution tool in Python (similar to Apache Oozie)
• Built-in support for Hadoop by wrapping Streaming• Not as fully-featured as mrjob for Hadoop, but easily
customizable• Internal serialization through repr/eval• Actively developed at Spotify• README is good but documentation is lacking
38
luigi
goto code
39
The cluster used for benchmarking
• 5 virtual machines• 4 CPUs• 10 GB RAM• 100 GB disk• CentOS 6.2
• CDH4 (Hadoop 2)• 20 map tasks• 10 reduce tasks
• Python 2.6
40
(Unscientific) performance comparison
41
(Unscientific) performance comparison
Streaming has lowest overhead
42
(Unscientific) performance comparison
JSON SerDe
43
Feature comparison
44
Feature comparison
45
Questions?
46
Cloudera Hadoop Stack
Unified Scale-Out StorageFor Any Type of Data
Batch Processing
Workload Management
System
Managem
entData
Managem
ent
47
Cloudera Hadoop Stack
Unified Scale-Out StorageFor Any Type of Data
Batch Processing
Workload Management
Online NoSQL
Analytic SQL Search
Machine Learning
and Streaming
3rd PartyApps
System
Managem
entData
Managem
ent
48
What is Spark?
• Started in 2009 as academic project from Amplab at UCBerkeley; now ASF and >100 contributors
• In-memory distributed execution engine• Operates on Resilient Distributed Datasets (RDDs)• Provides richer distributed computing primitives for
various problems• Can support SQL, stream processing, ML, graph
computation• Supports Scala, Java, and Python
Spark uses a general DAG scheduler
• Application aware scheduler• Uses locality for both disk
and memory• Partitioning-aware
to avoid shuffles• Can rewrite and optimize
graph based on analysis
join
union
groupBy
map
Stage 3
Stage 1
Stage 2
A: B:
C: D:
E:
F:
G:
= cached data partition
50
Operations on RDDs
Zaharia 2011
51
Apache Spark
file = spark.textFile("hdfs://...")errors = file.filter(lambda line: "ERROR” in line)# Count all the errorserrors.count()# Count errors mentioning MySQLerrors.filter(lambda line: "MySQL” in line).count()# Fetch the MySQL errors as an array of stringserrors.filter(lambda line: "MySQL” in line).collect()
val points = spark.textFile(...).map(parsePoint).cache()var w = Vector.random(D) // current separating planefor (i <- 1 to ITERATIONS) { val gradient = points.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce(_ + _) w -= gradient}println("Final separating plane: " + w)
Log
filte
ring
(Pyt
hon)
Logi
stic
regr
essi
on(S
cala
)
52
Apache Spark
goto code
53
What’s Impala?
• Interactive SQL• Typically 4-65x faster than the latest Hive (observed up to 100x faster)• Responses in seconds instead of minutes (sometimes sub-second)
• ANSI-92 standard SQL queries with HiveQL• Compatible SQL interface for existing Hadoop/CDH applications• Based on industry standard SQL
• Natively on Hadoop/HBase storage and metadata• Flexibility, scale, and cost advantages of Hadoop• No duplication/synchronization of data and metadata• Local processing to avoid network bottlenecks
• Separate runtime from batch processing• Hive, Pig, MapReduce are designed and great for batch• Impala is purpose-built for low-latency SQL queries on Hadoop
Cloudera Confidential. ©2013 Cloudera, Inc. All Rights Reserved.
54
Cloudera Impala
SELECT cosmic as snp_id, vcf_chrom as chr, vcf_pos as pos, sample_id as sample, vcf_call_gt as genotype, sample_affection as phenotypeFROM hg19_parquet_snappy_join_cached_partitionedWHERE COSMIC IS NOT NULL AND dbSNP IS NULL AND sample_study = ”breast_cancer" AND VCF_CHROM = "16";
55
Impala Architecture: Planner
• Example: query with join and aggregationSELECT state, SUM(revenue)FROM HdfsTbl h JOIN HbaseTbl b ON (...)GROUP BY 1 ORDER BY 2 desc LIMIT 10
HbaseScan
HashJoin
HdfsScan Exch
TopN
Agg
Exch
at coordinator at DataNodes at region servers
AggTopN
Agg
HashJoin
HdfsScan
HbaseScan
Cloudera Confidential. ©2013 Cloudera, Inc. All Rights Reserved.
56
Impala User-defined Functions (UDFs)
• Tuple => Scalar value• Substring• sin, cos, pow, …• Machine-learning models
• Supports Hive UDFs (Java)• Highly unpleasurable
• Impala (native) UDFs• C++ interface designed for efficiency• Similar to Postgres UDFs• Runs any LLVM-compiled code
57
LLVM compiler infrastructure
58
LLVM: C++ example
bool StringEq(FunctionContext* context, const StringVal& arg1, const StringVal& arg2) { if (arg1.is_null != arg2.is_null) return false; if (arg1.is_null) return true; if (arg1.len != arg2.len) return false; return (arg1.ptr == arg2.ptr) || memcmp(arg1.ptr, arg2.ptr, arg1.len) == 0;}
59
LLVM: IR output; ModuleID = '<stdin>'target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-n8:16:32:64-S128"target triple = "x86_64-apple-macosx10.7.0"
%"class.impala_udf::FunctionContext" = type { %"class.impala::FunctionContextImpl"* }%"class.impala::FunctionContextImpl" = type opaque%"struct.impala_udf::StringVal" = type { %"struct.impala_udf::AnyVal", i32, i8* }%"struct.impala_udf::AnyVal" = type { i8 }
; Function Attrs: nounwind readonly ssp uwtabledefine zeroext i1 @_Z8StringEqPN10impala_udf15FunctionContextERKNS_9StringValES4_(%"class.impala_udf::FunctionContext"* nocapture %context, %"struct.impala_udf::StringVal"* nocapture %arg1, %"struct.impala_udf::StringVal"* nocapture %arg2) #0 {entry: %is_null = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg1, i64 0, i32 0, i32 0 %0 = load i8* %is_null, align 1, !tbaa !0, !range !3 %is_null1 = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg2, i64 0, i32 0, i32 0 %1 = load i8* %is_null1, align 1, !tbaa !0, !range !3 %cmp = icmp eq i8 %0, %1 br i1 %cmp, label %if.end, label %return
if.end: ; preds = %entry %tobool = icmp eq i8 %0, 0 br i1 %tobool, label %if.end7, label %return
if.end7: ; preds = %if.end %len = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg1, i64 0, i32 1 %2 = load i32* %len, align 4, !tbaa !4 %len8 = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg2, i64 0, i32 1 %3 = load i32* %len8, align 4, !tbaa !4 %cmp9 = icmp eq i32 %2, %3 br i1 %cmp9, label %if.end11, label %return
if.end11: ; preds = %if.end7 %ptr = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg1, i64 0, i32 2 %4 = load i8** %ptr, align 8, !tbaa !5 %ptr12 = getelementptr inbounds %"struct.impala_udf::StringVal"* %arg2, i64 0, i32 2 %5 = load i8** %ptr12, align 8, !tbaa !5 %cmp13 = icmp eq i8* %4, %5 br i1 %cmp13, label %return, label %lor.rhs
lor.rhs: ; preds = %if.end11 %conv17 = sext i32 %2 to i64 %call = tail call i32 @memcmp(i8* %4, i8* %5, i64 %conv17) %cmp18 = icmp eq i32 %call, 0 br label %return
return: ; preds = %lor.rhs, %if.end11, %if.end7, %if.end, %entry %retval.0 = phi i1 [ false, %entry ], [ true, %if.end ], [ false, %if.end7 ], [ true, %if.end11 ], [ %cmp18, %lor.rhs ] ret i1 %retval.0}
; Function Attrs: nounwind readonlydeclare i32 @memcmp(i8* nocapture, i8* nocapture, i64) #1
attributes #0 = { nounwind readonly ssp uwtable "less-precise-fpmad"="false" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf"="true" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false" "use-soft-float"="false" }attributes #1 = { nounwind readonly "less-precise-fpmad"="false" "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf"="true" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "unsafe-fp-math"="false" "use-soft-float"="false" }
!0 = metadata !{metadata !"bool", metadata !1}!1 = metadata !{metadata !"omnipotent char", metadata !2}!2 = metadata !{metadata !"Simple C/C++ TBAA"}!3 = metadata !{i8 0, i8 2}!4 = metadata !{metadata !"int", metadata !1}!5 = metadata !{metadata !"any pointer", metadata !1}
60
LLVM compiler infrastructure
NumbaPython
61
Iris data and BigMLdef predict_species_orig(sepal_width=None, petal_length=None, petal_width=None): """ Predictor for species from model/52952081035d07727e01d836
Predictive model by BigML - Machine Learning Made Easy """ if (petal_width is None): return u'Iris-virginica' if (petal_width > 0.8): if (petal_width <= 1.75): if (petal_length is None): return u'Iris-versicolor' if (petal_length > 4.95): if (petal_width <= 1.55): return u'Iris-virginica' if (petal_width > 1.55): if (petal_length > 5.45): return u'Iris-virginica' if (petal_length <= 5.45): return u'Iris-versicolor' if (petal_length <= 4.95): if (petal_width <= 1.65): return u'Iris-versicolor' if (petal_width > 1.65): return u'Iris-virginica' if (petal_width > 1.75): if (petal_length is None): return u'Iris-virginica' if (petal_length > 4.85): return u'Iris-virginica' if (petal_length <= 4.85): if (sepal_width is None): return u'Iris-virginica' if (sepal_width <= 3.1): return u'Iris-virginica' if (sepal_width > 3.1): return u'Iris-versicolor' if (petal_width <= 0.8): return u'Iris-setosa'
62
Impala + Numba
goto code
63
Impala + Numba
• Still pre-alpha• Significantly faster execution thanks to native LLVM• Significantly easier to write UDFs
64
Conclusions
65
If you have access to a Hadoop cluster and you want a one-off quick-and-dirty job…
Hadoop Streaming
66
If you want an expressive Pythonic interface to build complex, regular ETL workflows…
Luigi
67
If you want to integrate Hadoop with other regular processes…
Luigi
68
If you don’t have access to Hadoop and want to try stuff out…
mrjob
69
If you’re heavily using AWS…
mrjob
70
If you want to work interactively…
PySpark
71
If you want to do in-memory analytics…
PySpark
72
If you want to do anything…*
PySpark
73
If you want ease of Python with high performance
Impala + Numba
74
If you want to write Python UDFs for SQL queries…
Impala + Numba
75
Code:https://github.com/laserson/rock-health-python
Blog post:http://blog.cloudera.com/blog/2013/01/a-guide-to-python-frameworks-for-hadoop/
Slides:http://www.slideshare.net/urilaserson/
76