Building Kafka-powered Activity Stream

BUILDING KAFKA-POWERED MESSAGE BUS

Activity Stream and beyond

CHALLENGES

➤ Needed a way to aggregate project-related activity information generated by various services in one place

➤ Needed a messaging backbone to carry messages between services

SOLUTIONS

➤ Activity Stream on the product side

➤ Message Bus on the tech side

“ A Message Bus is a combination of a common data model, a common command set, and a messaging infrastructure to allow different systems to communicate through a shared set of interfaces.

- "Enterprise Integration Patterns"

Apache Kafka

is publish-subscribe messaging

rethought as a distributed commit log.

by

WHY KAFKA?

KAFKA IS FAST

A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients.

KAFKA IS SCALABLE

➤ Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization.

➤ It can be elastically and transparently expanded without downtime.

➤ Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers.

KAFKA IS DURABLE

➤ Messages are persisted on disk and replicated within the cluster to prevent data loss.

➤ Each broker can handle terabytes of messages without performance impact.

KAFKA IS DISTRIBUTED BY DESIGN

Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.

GENERAL SETUP

READING / WRITING TO A TWO-PARTITION TOPIC

CONSUMER GROUPS

MESSAGE BUS GEM

A simple Ruby interface to Apache Kafka.

https://github.com/crealytics/message_bus

Camato Shopping

… service

Mesosphere

GewgawConsumer

Bishop

OUR SETUP

MessageBus

Producer

MessageBus

Producer

… service

MessageBus

Producer

Kafka cluster

Message Bus

Consumer

TrafficDirector run

DBActivity Stream

… service

Message Bus

Consumer

event message

event message

persistence

ACTIVITY STREAM

NEXT STEPS

➤ Keep adding messages (and message types) to Activity Stream

➤ Keep improving MessageBus gem

➤ Ensure proper production-ready Kafka setup

➤ Use Message Bus for decoupling service APIs and inter-service communication

➤ Asynchronous only, needs more research

➤ Input from Martin Mauch: use CrealyticsLogger to push JSON messages to Kafka (?) — under consideration

DEMO

DANKE!