INTRODUCTION TO APACHE STORM Sapienza University of Rome Data Mining Class A.Y. 2016-2017
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
INTRODUCTION TO APACHE STORM
Sapienza University of Rome Data Mining ClassA.Y. 2016-2017
Team
2
Riccardo Di Stefano
Roberto Gaudenzi
Davide Mazza
Lorenzo Rutigliano
Sara Veterini
Federico Croce
Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
https://it.linkedin.com/in/lorenzo-rutigliano-00a007135/it
https://it.linkedin.com/in/sara-veterini-667684116
https://it.linkedin.com/in/roberto-gaudenzi-4b0422116
https://it.linkedin.com/in/federico-croce-921a19134/it
https://it.linkedin.com/in/riccardo-di-stefano-439a11134
https://it.linkedin.com/in/davide-mazza-33a9b291
Contacts and Links
3
https://github.com/davidemazza/ApacheStorm
http://www.slideshare.net/DavideMazza6/apache-storm-tutorial
3Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
IntroductionApache Storm is a free and open source distributed fault-tolerant realtime computation system that make easy to process unbounded streams of data.
> use-cases: financial applications, network monitoring, social network analysis, online machine learning, ecc..
> different from traditional batch systems (store and process) .
4Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Companies
5Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Unbounded Sequence of Tuples
Tuple: Core unit of data, is a named list of values
6Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
TopologiesAn application is defined in Storm through a Topology that describes its logic as a DAG of operators and streams.
Spouts: are the sources of data streams. Usually read data from external sources (e.g. Twitter API) or from disk and emit them in the topology.
Bolts: processes input streams and (eventually) produce output streams. They represent the application logic.
7Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
ArchitectureTwo kinds of nodes in a Storm cluster:
➢ The Master node runs a daemon called “Nimbus” to which topologies are submitted. It is responsible for scheduling, job orchestration, and monitoring for failures.
➢ Each Worker (Slave) node runs a daemon called “Supervisor”, that can run one or more worker process in which applications are executed.
The coordination between this two entities is done through Zookeeper. It is mainly used to maintain state, because Nimbus and Supervisors are stateless.
8Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
ArchitectureThree entities are involved in running a topology:
➢ Worker Process: one or more per cluster, each one is related to only one topology (for design reasons related to fault-tolerance and isolation).
➢ Executor: thread in the Worker process. It runs one or more tasks for the same component (spout or bolt).
➢ Task: a component replica.
Therefore Workers provide inter-topology parallelism, Executors intra-topology and Tasks intra-component.
Worker process
Executor Task
TaskTask
Task
9Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Simple Example
10Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
ExampleWe will show how to compute the average of the grades using a simple Storm topology.
We will use:
➢ one spout;➢ two bolts that work in parallel;➢ another bolt in which the previous two converge
11Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Spout
This represents the spout.
Its job is to read a stream of numbers.
Our stream represents the grades, so they are within 18 and 30
12Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Bolt
This represents the bolt.
We can distinguish three different bolts in our example:
1. SummationBolt: computes the sum of the numbers;2. CounterBolt: counts the numbers;3. AverageBolt: computes the average.
13Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Topology
14Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
15Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
16Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
17Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
18Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
19Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream
Topology
20Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Stream Output
Trident
21Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Trident
➢ A high-level abstraction on top of Storm
➢ Uses Spout and Bolt auto-generated by Trident before execution
➢ Trident has functions, filters, joins, grouping, and aggregation
➢ Process streams as a series of batches
22Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Topology
➢ Receives input stream from spout
➢ Do ordered sequence of operation (filter, aggregation, grouping, etc.,) on the stream
23Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Tuples & Spout
➢ TridentTuple is a named list of values.
➢ TridentTuple interface is the data model of a Trident topology
➢ TridentSpout is similar to Storm spout, with additional options
➢ TridentSpout has many sample implementation of trident spout
24Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Example of Spout
25Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations
➢ Filter
➢ Function
➢ Aggregation
➢ Grouping
➢ Merging and Joining
26Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: Filter
➢ Object used to perform the task of input validation.
➢ Gets a subset of trident tuple fields as input
➢ Returns either true or false
➢ True → tuple is kept in the output stream
➢ False → the tuple is removed from the stream
27Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: Function➢ Object used to perform a simple operation on a single trident tuple.
➢ Takes a subset of trident tuple fields
➢ Emits zero or more new trident tuple fields.
28Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: AggregationObject used to perform aggregation operations on an input batch or partition or stream.
➢ Aggregate → Aggregates each batch of trident tuple in isolation
➢ PartitionAggregate → Aggregates each partition instead of the entire batch of trident tuple.
➢ PersistentAggregate → Aggregates on all trident tuple across all batch.
29Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: Aggregation
30Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: Grouping
➢ Inbuilt operation and can be called by the groupBy method
➢ Repartitions the stream by doing a partitionBy on the specified fields
➢ Groups tuples together whose group fields are equal
31Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Operations: Merging and Joining
➢ Merging combines one or more streams
➢ Joining uses trident tuple field from both sides to check and join two streams.
32Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
State Maintenance
➢ State information can be stored in the topology itself
➢ if any tuple fails during processing, then the failed tuple is retried.
➢ If the tuple has failed before updating the state → retrying the tuple will make the state
stable.
➢ if the tuple has failed after updating the state → then retrying the same tuple will make
the state unstable
33Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
When to use Trident?
It will be difficult to achieve exactly once processing in the case of Storm
34Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Trident will be useful for those use-cases where you require exactly once processing.
Trident Example
35Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Trident Demo: Twitter LanguagesWhich are the most used languages in Twitter?
The code is built on top of Trident and gets a stream of tweets using twitter4J library
For each tweet the language is extracted
A hashmap of counters is maintained and periodically published on a tweet by the code itself
36Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Trident example setupTo setup your twitter application:
● go to https://apps.twitter.com/ and create a new app● fill the form, leaving callback url empty● after creating the app, go to keys and access tokens● pick consumer secret and consumer key info● select create my access tokens if no tokens are present, then pick access
token and access token secret● open project TwitterTridentExample in Eclipse, open file twitter4j.properties
in the project, and copy your info
Now you are ready!
37Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Homework
38Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Homework
39Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
https://github.com/davidemazza/ApacheStorm
Folder “Homework”
Thanks! 40Apache Storm - Sapienza University of Rome - Data Mining Class - A.Y. 2016-2017
Related Documents
