Embracing the Data Deluge: Data-Intensive Computing for the Masses Jimmy Lin University of Maryland Tuesday, July 13, 2010 This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United See http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details
47
Embed
Embracing the Data Deluge: Data-Intensive Computing for the Masses
Embracing the Data Deluge: Data-Intensive Computing for the Masses. Jimmy Lin University of Maryland Tuesday, July 13, 2010. - PowerPoint PPT Presentation
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Embracing the Data Deluge:Data-Intensive Computing for the Masses
Jimmy LinUniversity of Maryland
Tuesday, July 13, 2010
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United StatesSee http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details
Introduction We live in a world of large data…
Staying relevant requires embracing it! In text processing…
Emergence and dominance of empirical, data-driven research Constant danger: uninteresting conclusions on “toy” datasets
(or, experiments taking forever) In the natural sciences…
Emergence of the 4th Paradigm: data-intensive eScience Difficult computer science problems!
How do we practically scale to large datasets? Case study in text processing: statistical machine translation Case study in bioinformatics: DNA sequence alignment
How much data? Google processes 20 PB a day (2008) Wayback Machine has 3 PB + 100 TB/month (3/2009) eBay has 6.5 PB of user data + 50 TB/day (5/2009) Facebook has 36 PB of user data + 80-90 TB/day (6/2010) CERN’s LHC: 15 PB a year (any day now) LSST: 6-10 PB a year (~2015)
640K ought to be enough for anybody.
No data like more data!
(Banko and Brill, ACL 2001)(Brants et al., EMNLP 2007)
The reality: programmer shoulders the burden of managing concurrency…(I want my students developing new machine learning algorithms, not debugging race conditions)
Independent per-record processing in parallel Aggregation of intermediate results to generate final output
Programmers specify two functions:map (k, v) → <k’, v’>*reduce (k’, v’) → <k’, v’>* All values with the same key are sent to the same reducer
The execution framework handles everything else… Handles scheduling Handles data management, transport, etc. Handles synchronization Handles errors and faults
mapmap map map
Shuffle and Sort: aggregate values by keys
reduce reduce reduce
k1 k2 k3 k4 k5 k6v1 v2 v3 v4 v5 v6
ba 1 2 c c3 6 a c5 2 b c7 8
a 1 5 b 2 7 c 2 3 6 8
r1 s1 r2 s2 r3 s3
split 0split 1split 2split 3split 4
worker
worker
worker
worker
worker
UserProgram
outputfile 0
outputfile 1
(1) submit
(2) schedule map (2) schedule reduce
(3) read(4) local write
(5) remote read(6) write
Inputfiles
Mapphase
Intermediate files(on local disk)
Reducephase
Outputfiles
Adapted from (Dean and Ghemawat, OSDI 2004)
Master
(I want my students developing new machine learning algorithms, not debugging race conditions)
MapReduce Implementations Google has a proprietary implementation in C++
Bindings in Java, Python Hadoop is an open-source implementation in Java
Development led by Yahoo, used in production Now an Apache project Rapidly expanding software ecosystem
Lots of custom research implementations For GPUs, cell processors, etc.
Case Study #1Statistical Machine Translation
Chris Dyer (Linguistics Ph.D., 2010)
Translation Model
LanguageModel
Decoder
Foreign Input Sentence
maria no daba una bofetada a la bruja verde
English Output Sentencemary did not slap the green witch
Word Alignment
Statistical Machine Translation
(vi, i saw)(la mesa pequeña, the small table)…
Phrase Extraction
i saw the small tablevi la mesa pequeña
Parallel Sentences
he sat at the tablethe service was good
Target-Language Text
Training Data
Maria no dio una bofetada a la bruja verde
Mary not
did not
no
did not give
give a slap to the witch green
slap
a slap
to the
to
the
green witch
the witch
by
slap
Translation as a Tiling Problem
Mary
did not
slap
the
green witch
The Data Bottleneck
“Every time I fire a linguist, the performance of our … system goes up.”- Fred Jelinek
Translation Model
LanguageModel
Decoder
Foreign Input Sentence
maria no daba una bofetada a la bruja verde
English Output Sentencemary did not slap the green witch
Word Alignment
Statistical Machine Translation
(vi, i saw)(la mesa pequeña, the small table)…
Phrase Extraction
i saw the small tablevi la mesa pequeña
Parallel Sentences
he sat at the tablethe service was good
Target-Language Text
Training Data
We’ve built MapReduce implementations of these two components!
HMM Alignment: Giza
Single-core commodity server
HMM Alignment: MapReduce
Single-core commodity server
38 processor cluster
HMM Alignment: MapReduce
38 processor cluster
1/38 Single-core commodity server
What’s the point? The optimally-parallelized version doesn’t exist! MapReduce occupies a sweet spot in the design space for
a large class of problems: Fast… in terms of running time + scaling characteristics Easy… in terms of programming effort Cheap… in terms of hardware costs
Chris Dyer, Aaron Cordova, Alex Mont, and Jimmy Lin. Fast, Easy, and Cheap: Construction of Statistical Machine Translation Models with MapReduce. Proceedings of the Third Workshop on Statistical Machine Translation at ACL 2008
Case Study #2DNA Sequence Alignment
Michael Schatz(Computer Science Ph.D., 2010)
Strangely-Formatted Manuscript Dickens: A Tale of Two Cities
Text written on a long spool
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
… With Duplicates Dickens: A Tale of Two Cities
“Backup” on four more copies
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
Shredded Book Reconstruction Dickens accidently shreds the manuscript
How can he reconstruct the text? 5 copies x 138,656 words / 5 words per fragment = 138k
fragments The short fragments from every copy are mixed together Some fragments are identical
It was the best of of times, it was thetimes, it was the worst age of wisdom, it was the age of foolishness, …
It was the best worst of times, it wasof times, it was the the age of wisdom, it was the age of foolishness,
It was the the worst of times, it best of times, it was was the age of wisdom, it was the age of foolishness, …
It was was the worst of times,the best of times, it it was the age of wisdom, it was the age of foolishness, …
It it was the worst ofwas the best of times, times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of of times, it was thetimes, it was the worst age of wisdom, it was the age of foolishness, …
It was the best worst of times, it wasof times, it was the the age of wisdom, it was the age of foolishness,
It was the the worst of times, it best of times, it was was the age of wisdom, it was the age of foolishness, …
It was was the worst of times,the best of times, it it was the age of wisdom, it was the age of foolishness, …
It it was the worst ofwas the best of times, times, it was the age of wisdom, it was the age of foolishness, …
Greedy AssemblyIt was the best of
of times, it was the
best of times, it was
times, it was the worst
was the best of times,
the best of times, it
of times, it was the
times, it was the age
It was the best of
of times, it was the
best of times, it was
times, it was the worst
was the best of times,
the best of times, it
it was the worst of
of times, it was the
times, it was the age
it was the age of
was the age of wisdom,
the age of wisdom, it
age of wisdom, it was
of wisdom, it was the
it was the age of
was the age of foolishness,
the worst of times, it
The repeated sequence make the correct reconstruction ambiguous!
Alternative: model sequence reconstruction as a graph problem…
de Bruijn Graph Construction Dk = (V,E)
V = All length-k subfragments (k < l) E = Directed edges between consecutive subfragments
(Nodes overlap by k-1 words)
Locally constructed graph reveals the global structure Overlaps between sequences implicitly computed
It was the best of
Original Fragment
It was the best was the best of
Directed Edge
de Bruijn, 1946Idury and Waterman, 1995Pevzner, Tang, Waterman, 2001
de Bruijn Graph Assembly
the age of foolishness
It was the best
best of times, it
was the best of
the best of times,
of times, it was
times, it was the
it was the worst
was the worst of
worst of times, it
the worst of times,
it was the age
was the age ofthe age of wisdom,
age of wisdom, it
of wisdom, it was
wisdom, it was the
A unique Eulerian tour of the graph reconstructs the
original text
If a unique tour does not exist, try to simplify the
graph as much as possible
de Bruijn Graph Assembly
the age of foolishness
It was the best of times, it
of times, it was the
it was the worst of times, it
it was the age ofthe age of wisdom, it was theA unique Eulerian tour of the
graph reconstructs the original text
If a unique tour does not exist, try to simplify the
Assembly of Large Genomes with Cloud Computing.Schatz MC, Sommer D, Kelley D, Pop M, et al. In Preparation.
Source: flickr (fatboyke/2918399820)
Source: flickr (60in3/2338247189)
Best thing since sliced bread? Distributed programming models:
MapReduce is the first Definitely not the only And probably not even the best Alternatives: Pig, Dryad/DryadLINQ, Pregel, etc.
It’s all about the right level of abstraction The von Neumann architecture won’t cut it anymore
Separating the what from how Developer specifies the computation that needs to be performed Execution framework handles actual execution Framework hides system-level details from the developers
Source: NY Times (6/14/2006)
The datacenter is the computer!What are the appropriate abstractions for the datacenter computer?
Source: flickr (infidelic/3008675635)
Source: Wikipedia (Tide)
Commoditization of large-data processing capabilities allows us to ride the rising tide!