Top Banner
Collecting and Analyzing sensor data Bigdata with Hadoop or other NoSQL databases
19

Collecting and analyzing sensor data with hadoop or other no sql databases

Dec 01, 2014

Download

Technology

Matteo Redaelli

Scouting howto collecting and analyzing sensor data with hadoop or other no sql databases
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Collecting and analyzing sensor data with hadoop or other no sql databases

Collecting and Analyzing

sensor data

Bigdata with Hadoop or other NoSQL databases

Page 2: Collecting and analyzing sensor data with hadoop or other no sql databases

Who am I

I am an Open Source enthusiast!

matteo DOT redaelli AT gmail DOT com

http://www.redaelli.org/matteo/

Page 3: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop ecosystem (1 of 2)

● HDFS is the distribuited file system of Hadoop: data are usually stored as text/csv files (rows are distribuited in the cluster)

● HIVE is the datawarehouse of Hadoop

Page 5: Collecting and analyzing sensor data with hadoop or other no sql databases

Collecting

Apache flume from Cloudera

Page 9: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop evolution (1 of 2)

Page 10: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop evolution (2 of 2)

Page 12: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop top distributions: Hortonworks

Page 13: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop top distributions: MapR

Page 16: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop alternatives: Riak

Riakhttp://docs.basho.com/riak/1.2.1

/cookbooks/use-cases/sensor-data/

Page 17: Collecting and analyzing sensor data with hadoop or other no sql databases

Hadoop alternatives: Kafka + Storm

Apache Kafka (from Linkedin) for aggregating

Apache Storm (from Twitter) for realtime computing

Page 18: Collecting and analyzing sensor data with hadoop or other no sql databases

Alternatives: timeseries databases

OpenTSDB Hadoop Hbase

Influxdb

Kairosdb Cassandra, Hadoop Hbase