Building production spark streaming applications

© Rocana, Inc. All Rights Reserved. | 1

Joey Echeverria, Platform Technical Lead - @fwiffo

Data Day Texas 2017

Building Production Spark Streaming Applications


Joey• Where I work: Rocana – Platform Technical Lead

• Where I used to work: Cloudera (’11-’15), NSA

• Distributed systems, security, data processing, big data



Context• We built a system for large scale realtime collection, processing, and

analysis of event-oriented machine data

• On prem or in the cloud, but not SaaS

• Supportability is a big deal for us• Predictability of performance under load and failures• Ease of configuration and operation• Behavior in wacky environments


Apache Spark Streaming


Spark streaming overview• Stream processing API built on top of the Spark execution engine

• Micro-batching• Every n-milliseconds fetch records from data source• Execute Spark jobs on each input batch

• DStream API• Wrapper around the RDD API• Lets the developer think in terms of transformations on a stream of events


Input BatchSpark Batch

Engine Output Batch


Structured streaming• New streaming API for Spark

• Re-use DataFrames API for streaming

• API was too new when we started• First release was an alpha• No Kafka support at the time

• Details won't apply, but the overall approach should be in the ballpark


Other notes• Our experience is with Spark 1.6.2

• 2.0.0 was released after we started our Spark integration

• We use the Apache release of Spark• Supports both CDH and HDP without recompiling• We run Spark on YARN, so we're decoupled from other users on the cluster


Use CaseReal-time alerting on IT operational data


Our typical customer use cases• >100K events / sec (8.6B events / day), sub-second end to end latency,

full fidelity retention, critical use cases

• Quality of service - “are credit card transactions happening fast enough?”

• Fraud detection - “detect, investigate, prosecute, and learn from fraud.”

• Forensic diagnostics - “what really caused the outage last friday?”

• Security - “who’s doing what, where, when, why, and how, and is that ok?”

• User behavior - ”capture and correlate user behavior with system performance, then feed it to downstream systems in realtime.”


Overall architectureweirdo formats

transformation 1weirdo format -> event

avro events

transformation 2event -> storage-specific

storage-specific representation of events


Real-time alerting• Define aggregations, conditions, and actions

• Use cases:• Send me an e-mail when the number of failed login events from a user is > 3

within an hour• Create a ServiceNow ticket when CPU utilization spikes to > 95% for 10 minutes


UI


Architecture


Packaging, Deployment, and Execution


Packaging• Application classes and dependencies

• Two options• Shade all dependencies into an uber jar

• Make sure Hadoop and Spark dependencies are marked provided• Submit application jars and dependent jars when submitting


Deployment modes• Standalone

• Manually start up head and worker services• Resource control depends on options selected when launching daemons• Difficult to mix versions

• Apache Mesos• Coarse grained run mode, launch executors as Mesos tasks• Can use dynamic allocation to launch executors on demand

• Apache Hadoop YARN• Best choice if your cluster is already running YARN


Spark on YARN• Client mode versus cluster mode

• Client mode == Spark Driver on local server• Cluster mode == Spark Driver in YARN AM

• Spark executors run in YARN containers (one JVM per executor)• spark.executor.instances

• Each executor core uses one YARN vCore• spark.executor.cores


Job submission• Most documentation covers spark-submit

• OK for testing, but not great for production

• We use spark submitter APIs• Built easier to use wrapper API• Hide some of the details of configuration

• Some configuration parameters aren't respected when using submitter API• spark.executor.cores, spark.executor.memory• spark.driver.cores, spark.driver.memory


Job monitoring• Streaming applications are always on

• Need to monitor the job for failures• Restart the job on recoverable failures• Notify an admin on fatal failures (e.g. misconfiguration)

• Validate as much up front as possible• Our application runs rules through a type checker and query planner before

saving


Instrumentation, Metrics, and Monitoring


Instrumentation

You can't fix whatyou don't measure


Instrumentation APIs• Spark supports Dropwizard (née CodaHale) metrics

• Collect both application and framework metrics• Supports most popular metric types

• Counters• Gauges• Histograms• Timers• etc.

• Use your own APIs• Best option if you have your existing metric collection infrastructure


Custom metrics• Implement the org.apache.spark.metrics.source.Source interface

• Register your source with sparkEnv.metricsSystem().registerSource()• If you're measuring something during execution, you need to register the metric

on the executors• Register executor metrics in a static block• You can't register a metrics source until the SparkEnv has been initialized

SparkEnv sparkEnv = SparkEnv.get();if (sparkEnv != null) { // create and register source}


Metrics collection• Configure $SPARK_HOME/conf/metrics.properties

• Built-in sinks• ConsoleSink• CVSSink• JmxSink• MetricsServlet• GraphiteSink• Slf4jSink• GangliaSink

• Or build your own


Build your own• Implement the org.apache.spark.metrics.sink.Sink interface

• We built a KafkaEventSink that sends the metrics to a Kafka topic formatted as Osso* events

• Our system has a metrics collector• Aggregates metrics in a Parquet table• Query and visualize metrics using SQL

• *http://www.osso-project.org

http://www.osso-project.org/

http://www.osso-project.org/


Report and visualize


Gotcha• Due to the order of metrics subsystem initialization, your collection plugin

must be on the system classpath, not application classpath• https://issues.apache.org/jira/browse/SPARK-18115

• Options:• Deploy library on cluster nodes (e.g. add to HADOOP_CLASSPATH)• Build a custom Spark assembly jar

https://issues.apache.org/jira/browse/SPARK-18115






Custom spark assembly• Maven shade plugin

• Merge upstream Spark assembly JAR with your library and dependencies• Shade/rename library packages

• Might break configuration parameters as well • *.sink.kafka.com_rocana_assembly_shaded_kafka_brokers

• Mark any dependencies already in the assembly as provided• Ask me about our akka.version fiasco


Configuration and Tuning


Architecture


Predicting CPU/task resources• Each output operation creates a separate batch job when processing a

micro-batch• number of jobs = number of output ops

• Each data shuffle/re-partitioning creates a separate stage• number of stages per job = number of shuffles + 1

• Each partition in a stage creates a separate task• number of tasks per job = number of stages * number of partitions


Resources for alerting• Each rule has a single output operation (write to Kafka)

• Each rule has 3 stages1. Read from Kafka, project, filter and group data for aggregation2. Aggregate values, filter (conditions) and group data for triggers3. Aggregate trigger results and send trigger events to Kafka

• First stage partitions = number of Kafka partitions

• Stage 2 and 3 use spark.default.parallelism partitions


Example• 100 rules, Kafka partitions = 50, spark.default.parallelism = 50

• number of jobs = 100

• number of stages per job = 3

• number of tasks per job = 3 * 50 = 150

• total number of tasks = 100 * 150 = 15,000


Task slots• number of task slots = spark.executor.instances * spark.executor.cores

• Example• 50 instances * 8 cores = 400 task slots


Waves• The jobs processing the micro-batches will run in waves based on

available task slots

• Number of waves = total number of tasks / number of task slots

• Example• Number of waves = 15,000 / 400 = 38 waves


Max time per wave• maximum time per wave = micro-batch duration / number of waves

• Example:• 15 second micro-batch duration• maximum time per wave = 15,000 ms / 38 waves = 394 ms per wave

• If the average task time > 394 ms, then Spark streaming will fall behind


Monitoring batch processing time


Delay scheduling• A technique of delaying scheduling of tasks to get better data locality

• Works great for long running batch tasks• Not ideal for low-latency stream processing tasks

• Tip• Set spark.locality.wait = 0ms• Results

• Running job with 800 tasks on a very small (2 task slot) cluster, 300 event micro-batch• With default setting: 402 seconds• With 0ms setting: 26 seconds (15.5x faster)


Model memory requirements• Persistent memory used by stateful operators

• reduceByWindow, reduceByKeyAndWindow• countByWindow, countByValueAndWindow• mapWithState, updateStateByKey

• Model retention time• Built-in time-based retention (e.g. reduceByWindow)• Explicit state management (e.g. org.apache.spark.streaming.State#remove())


Example• Use reduceByKeyAndWindow to sum integers with a 30 second window

and 10 second slide over 10,000 keys

• active windows = window length / window slide• 30s / 10s = 3

• estimated memory = active windows * num keys * (state size + key size)• 3 *10,000 * (16 bytes + 80 bytes) = 2.75 MB


Monitor Memory


Putting it altogether• Pick your packaging and deployment model based on operational needs,

not developer convenience

• Use Spark submitter APIs whenever possible

• Measure and report operational metrics

• Focus configuration and tuning on the expected behavior of your application• Model, configure, monitor


Questions?@fwiffo | [email protected]

Building production spark streaming applications

Technology