Stay Curious Accelerated Innovation
The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]
Apr 14, 2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
![Page 1: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/1.jpg)
Stay Curious
Accelerated Innovation
![Page 2: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/2.jpg)
Working With Over 200 Customers
2
![Page 3: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/3.jpg)
Gordon Moore
3
![Page 4: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/4.jpg)
Moore's Law
4
10^4
10^6
10^8
10^10
10^12
10^14
10^16
10^2
0
1950
1955
1960
1965
1970
1975
1980
1985
1990
1995
2000
2005
2010
2015
2020
2025
Dec PDP 1
Altair 8800
Pentium
Core i7 Quad
Core 2 Duo
Compaq Deskpro 386
Apple 2
Calculations / Sec / $1000
![Page 5: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/5.jpg)
Ray Kurzweil
5
![Page 6: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/6.jpg)
Accelerated Innovation
6
10^4
10^6
10^8
10^10
10^12
10^14
10^16
10^2
0
Ston
eAg
e
1950
1955
1960
1965
1970
1975
1980
1985
1990
1995
2000
2005
2010
2015
2020
2025
Dec PDP 1
Altair 8800
Pentium
Core i7 Quad
Core 2 Duo
Compaq Deskpro 386
Apple 2
Calculations / Sec / $1000
![Page 7: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/7.jpg)
Acceleration of Innovation in Big Data
StormParquetSentrySparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
SparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
FlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
HivePig
MahoutHBase
ZooKeeperCore Hadoop
HBaseZooKeeper
Core Hadoop
FlinkDrill
RangerAmbariIgniteStorm
ParquetSentrySparkImpalaSolr
KafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore HadoopCore Hadoop
HDFS, MR
SamzaKudu
SamsaraAtlasApexNiFiFlinkDrill
RangerAmbariIgniteStorm
ParquetSentrySparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
2006 2008 2009 2010 2011 2012 2013 2014 Present
![Page 8: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/8.jpg)
Acceleration of Innovation in Big Data
StormParquetSentrySparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
SparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
FlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
HivePig
MahoutHBase
ZooKeeperCore Hadoop
HBaseZooKeeper
Core Hadoop
FlinkDrill
RangerAmbariIgniteStorm
ParquetSentrySparkImpalaSolr
KafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore HadoopCore Hadoop
HDFS, MR
SamzaKudu
SamsaraAtlasApexNiFiFlinkDrill
RangerAmbariIgniteStorm
ParquetSentrySparkImpala
SolrKafkaFlumeBigtopOozie
MRUnitHCatalog
SqoopWhirrAvroHivePig
MahoutHBase
ZooKeeperCore Hadoop
2006 2008 2009 2010 2011 2012 2013 2014 Present
Acceleration of Complexity
![Page 9: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/9.jpg)
Hadoop - Build for NoSQL
9
DBETL Reporting
HadoopRaw Load View
Schema on Write
Schema on Read
![Page 10: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/10.jpg)
Hadoop - Disruptive
0 s
7,500 s
15,000 s
22,500 s
30,000 s
25x40GB 50x20GB 200x10GB
Hadoop DBMS-X
https://wiki.umiacs.umd.edu/ccc/images/8/8c/CLuE-Madden.pdf
![Page 11: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/11.jpg)
Spark - Disruptive?
0 min
15 min
30 min
45 min
60 min
Join Machine Learning
Map Reduce Tez Spark
Datameer Benchmark
![Page 12: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/12.jpg)
Flink - Already Faster
http://www.slideshare.net/FlinkForward/dongwon-kim-a-comparative-performance-evaluation-of-flink
0 s
1250 s
2500 s
3750 s
5000 s
10GB/Node 20GB/Node 40GB/Node 80GB/Node 160GB/Node
Flink Spark
![Page 13: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/13.jpg)
Stack
Hardware
Data Center OS (e.g. Mesosphere)
Storage
Compute
Mor
e Diffi
cult
to C
hang
e
Hadoop Killer!
![Page 14: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/14.jpg)
Yarn vs. Mesosphere
Yarn Mesosphere
Unix Process Linux Containers
Resources Requested Resources Offered
Batch Centric Flexible on Job
Internal Scheduling Client Scheduling
VS
![Page 15: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/15.jpg)
Data Scientist
80% Data Preparation
20% Machine Learning
Feature Selection
![Page 16: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/16.jpg)
Deep Learning
Unsupervised Learning of Features
Data Scientist Replacement?
![Page 17: The Acceleration of Innovation in Big Data // Stefan Groschupf, Datameer [FirstMark's Data Driven]](https://reader043.cupdf.com/reader043/viewer/2022030310/58f05d7e1a28ab86378b458b/html5/page/17.jpg)
@StefanGroschupf
Related Documents
![Combining Machine Learning With Expert Human Judgement // Eric Colson, Stitch Fix [FirstMark's Data Driven]](https://static.cupdf.com/doc/110x72/58eccc8e1a28ab985c8b4615/combining-machine-learning-with-expert-human-judgement-eric-colson-stitch.jpg)
![A Virtual Distributed Storage System // Haoyuan Li, Alluxio [FirstMark's Data Driven]](https://static.cupdf.com/doc/110x72/58ee29bf1a28ab53738b4689/a-virtual-distributed-storage-system-haoyuan-li-alluxio-firstmarks.jpg)
![Paul Ford, aka @ftrain: Talent is What Annoys You [FirstMark's Design Driven]](https://static.cupdf.com/doc/110x72/5870705b1a28ab48378b7709/paul-ford-aka-ftrain-talent-is-what-annoys-you-firstmarks-design.jpg)
![The Benefits of Fast Business Intelligence // Amir Orad, Sisense [FirstMark's Data Driven]](https://static.cupdf.com/doc/110x72/5875084f1a28ab29208b68f7/the-benefits-of-fast-business-intelligence-amir-orad-sisense-firstmarks.jpg)
