Comparing processing frameworks v7

Testing Processing Frameworks

Streamingand

Gabriela Choy

Spark / StormSpark Storm

Implemented in Scala Clojure, Java

Delivery Semantics Exactly once At least once. Exactly once with Trident

APIPython, Java, Scala Java, Scala, Clojure,

Python, etc. Trident: Java, Scala, Clojure.

Processing ModelBatch. Micro-batches with Spark Streaming. ~ 500ms

Record at a time/ Trident allows for micro-batches.

Latency 1 - 2 seconds sub-seconds

Pipeline

Streaming

MySQL

MySQL

Producer/s

Producer/s

Each pipeline is run independently

Node Specifications

Spark Streaming Storm

4 AWS nodes m3.medium

Zookeeper 3.4.6Kafka 0.8.2.1

Spark (streaming) 1.3

4 AWS nodes m3.medium

Zookeeper 3.4.6Kafka 0.8.2.1Storm 0.9.5

Spark Streaming: 1 master node, 3 workers

Cluster Configuration

Master node

Worker 1

Worker 2

Worker 3

Storm : 1 nimbus, 3 Supervisors

Cluster Configuration

Nimbus

Supervisor 1

Supervisor 2

Supervisor 3

Metric

Throughput: amount of data that is being processed.

● By changing batch size

● By changing load (i.e. Scaling up)

● Programs used for benchmarking will be wordcount.

# Producers Batch Interval

1 1s, 2s, 3s, 4s, 6s

41s, 2s, 3s, 4s, 6s

8 1s, 2s, 3s, 4s, 6s

Tests for Spark Streaming

Throughput for 1 producer with 95% CI

Throughput for 1 producer

Throughput for 4 producers with 95% CI

Throughput for 4 producers

Throughput for 8 producers

Throughput for 8 producers

# Producers Tuples Emitted-Acked

1 10 min

4 10 min

8 10 min

Tests for Storm

Preliminary results for storm

Tuples Emitted per Second

Tuples Acked per Second

Spout Latency

Takeaways

● Setting the batch interval in spark streaming should be done by monitoring processing times and load size

● For Storm as numbers of producers increase so does throughput and spout latency.

Would like to add:

● Increase number of producers. Use real data.

● Add a graph as a second use case.

● Dashboard to monitor live streaming.

Gabriela Choy

Bsc. in Chem. Engineering. ULA, Vnzla.Msc in Statistics. UT Dallas

Previously: Worked in Device Reliability Engineering at View, Inc.

About Me