Big Data big problems.
Big Databig problems.
What is Big Data? Volume Velocity Variety
VolumeBillions of Things:
Posts, Tweets and Likes Web Transactions Sensor Readings
VelocityStreaming Data:
Twitter: 500,000,000 TPD Walmart: 20,000,000 TPD Hopper: 750,000,000 TPD
Variety Integrating Many Sources of Data:
Unstructured Web Content Semi-structured Logs Relational Databases Images, Video, Audio
So What’s Changed?
Mobile devices Social Web Sensors, Metrics Digitization of
everything
Open Source Tools•Hadoop: distributed processing
•R: predictive analytics for big data
•Hive, Pig: ad-hoc analytics for Hadoop
•Mahout: machine learning for Hadoop
•HBase, Cassandra: distributed databases
•ElasticSearch: distributed search engine
•Storm: distributed processing for data streams
"The best minds of my generation are thinking about how to make people click ads"
- Jeff Hammerbacher (Facebook, Accel, Cloudera)
Big Minds + Big Data
Aggregate, Summarize Detect Patterns Model, Simulate Forecast, Predict
Open Data
Reports Request/Response APIs Small Data
TextText
Hack/reduce
Open Hackspace in Boston Home for Pre-seed projects, Community events
Not-for-profit sponsored by local industry and government
Hack/reduce Cluster
240-core cluster sponsored by GoGrid, a cloud computing company.
Available for use at today’s Open Data Day.
What do you with a 240-core Cluster?
Use the power of many machines to analyze Big Data sets.
How do you get computers to work together like that??That’s what Hadoop is for.
An Example
Daily Hansard: transcript of Canadian parliament since 1994
Swearwords.txt (http://www.bannedwordlist.com)
Who are the most foul-mouthed Federal MPs?
Results
•20 years of House of Commons statements
•511,341 Statements analyzed
•121,985,310 Words spoken
•3,839 Swearwords spoken
•1 in 133 statements has a swearword
Top 5 Swearers (absolute)
Pat Martin NDP 98
Randy White Conservative 88
Alexa McDonough
NDP 52
Jim Silye Conservative 50
Yvan Loubier Bloc Quebecois 49
Top 5 Swearers (relative)
Randy WhiteConservativ
e0.037
%88 299,114
Dennis Mills Liberal0.023
%14 62,221
Gerry RitzConservativ
e0.022
%22 99,037
John McCallum
Conservative
0.017%
38 226,155
John McKay Liberal0.016
%44 268,188
Top 5 Words Spoken
Paul Szabo 1,482,106
Pat Martin 1,053,365
Don Boudria 867,204
Yvan Loubier 861,888
Peter McKay 844,130
Prime Ministers
Jean Chrétien 11 604,431
Paul Martin 6 485,990
Stephen Harper 22 620,999
"The best minds of my generation are thinking about how to make people click ads"
- Jeff Hammerbacher (Facebook, Accel, Cloudera)