Librecon 2016 bilbao: kappa architecture IoT of the cars

Kappa Architecture: IoT of the cars

Juantomás García

Who am I• Data Solutions Manager at Open Sistemas

• Co-author of the first spanish book about free software “La pastilla roja”

• Former president of Hispalinux (Spanish Linux User Group).

• Co-organizer of Machine Learning Spain Meetup

What’s Kappa ArchitectureJuly 2, 2014 Jay Kreps coined the term Kappa Architecture in an article for O’reilly Radar

Who is Jay KrepsJay has been involved in lots of projects:

Author of the essay:

The Log: What every software engineer should know about real-time data's unifying abstraction (12/16/2013)

https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying

Who is Jay KrepsAuthor of the book: I ♥ Logs

Who is Jay KrepsInvolved with projects as:

Apache Kafka

Apache Samza

Voldemort

Azkaban

Ex-Linkedin

Now co-founder and CEO of Confluent

Usual data flow

Kappa Architecture Way

Tools we use to implement Kappa Architecture

Tools we use to implement Kappa Architecture

Tools we use to implement Kappa Architecture

• Our favorite kafka features.

• Easy to use

• Robust & Resilient

• Kafka connectors

• Kafka streams

• Compact logs

• Very fast writes and reads

Tools we use to implement Kappa Architecture

Some favorite Spark features • If you have an schema spark SQL is

perfect.

• Spark streaming work very fine with spark and almost all streaming sources.

• Structured queries will be a huge advance.

• We love scala and scala is the spirit of spark.

Some favorite Spark features • We love code like this:

A real case• One of our clients want to monitor all the

cars informations via OBD II

• OBD II is a car interface with the car electronics.

• Client develop an app to read all the car information throw a ODB II with bluetooth

A real case

First Problems• We need to scale the rest interfaces too

many requests.

• MySQL don’t scale

• Client wants to make realtime expensive queries.

Some Numbers

Architecture V2

Architecture KappaV3

Now we’re more flexible• We can have queries like:

• What are the drivers that are not client of the X gas brand, has a few gas and are near of gas station of the brand X and if true, send a notification with a discount coupon and a link with the route.

Takeaways• Kappa architecture is not a silver bullet but

helps in a lot of solutions.

• Kafka + spark streaming is our favorite tools

• There a lots of improvements:

• OLAP like Apache Druid

• Graph databases like neo4j

• Kafka streams and compacts logs

¿Questions?• My contact info:

• Juantomás García

• [email protected]

• @juantomas