Stream Processing with Kafka and Samza Diego Pacheco @diego_pacheco Principal Software Architect
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Stream Processing with Kafka and Samza
Diego Pacheco @diego_pacheco Principal Software Architect
●LinkedIN 2011●Implemented with Scala and Java●Motivation: Real-time data feeds●Goals:–Low Latency–High Throughtput
●Kafka at LinkedIN(2014):–300+ brokers–18k topics–140k partitions–220B messages per day–40TB inboud–160TB outbound–Peak Load: 3.25M messages/second
●Use case: Activity Stream, Offline log processing
NO JMS
● LinkedIN 2013
● Stream Processing with Save Points.
● Multi-tenancy: 1 Thread per container
● State is simple
– You handle logging and restoring
– Single threaded programing
● Works with YARN
● Works well with Kafka
● Simple API – Record-like.
● Stream Processing
● Low Latency
● Async Processing
● Local State● Stores data localy on DISK● SAME machine where container runs
– Awesome FIT for Statefull processing
● Tight Integration with Kafka
● Strong Model For Streams: Ordered, Highly Avaliable, Partitioned and Durable(Kafka).
● Full feature Set of Kafka
● Client Side Join
Stream Processing with Kafka and Samza
Diego Pacheco @diego_pacheco Principal Software Architect
Thank You!Obrigado !
Related Documents
