Voxxed Days Vienna 2017 - Microservices with Kafka

albertlaszlorobert #VoxxedVienna

Microservices with Kafka

László-Róbert ALBERT


Róbert


FAULT

TOLERANCE

PE

RF

OR

MA

NC

E

RELIABILITY SECURITY

AVAILABILITY

HIGH

SCALABILITY

REPLICATION

CH

AN

GE SELF HEALING





Account

Manager


Core Banking

Account

Manager


Core Banking

Transfer

Account

Manager


Core Banking

Transfer

Account

Manager


Source: http://www.slideshare.net/dberkholz/

how-microservices-are-redefining-

modern-application-architecture


Source: http://www.slideshare.net/dberkholz/

how-microservices-are-redefining-

modern-application-architecture



500 million messages / day


500 million messages / day 700 billion messages / day


500 million messages / day 700 billion messages / day 1.4 trillion messages / day


500 million messages / day 700 billion messages / day 1.4 trillion messages / day


Messaging Brokers Features Kafka ActiveMQ RabbitMQ


Messaging Brokers Features

Support for multiple protocols

• Advanced Message Queueing Protocol

• Streaming Text Oriented Messaging Protocol

• Message Queuing Telemetry Transport

• Etc

Kafka ActiveMQ RabbitMQ







• Etc


Java Messaging System







• Etc



Java Authentication and Authorization Service

Simple Authentication and Security Layer







• Etc





Transactions







• Etc





Transactions

Filtering







• Etc





Transactions

Filtering

Persistence

albertlaszlorobert #VoxxedVienna 2

6

Features

Perf

orm

an

ce


Messaging Brokers Performance

Message producing

2

7

Message consumption

LinkedIn – Kafka: a Distributed Messaging System for Log Processing


Features

Perf

orm

an

ce





System Availability & Performance


System Availability & Performance


Scalability

3

4


Scalability

3

5

albertlaszlorobert #VoxxedVienna 3

6


Producer

© ING 2016

// build a 'default' factory based on your-implicit-kafka-producer.properties

// - the properties file contains the list of brokers to connect to

ProducerFactory factory = ProducerFactory.builder().build();

// create a message producer with a Payload that sends the messages to topic ‘test’

Producer<Message<Payload>> producer =

factory.createMessageProducer("test", Payload.class);

// send a Message { header: Map<String, String>; payload: Payload }

producer.send(

Message.<Payload> builder()

.header("applicationId", "sample-producer")

.header("timestamp", now())

.payload(new Payload("/some/file/path"))

.build()

);


Producer

your-implicit-producer.properties

your-security.properties


KafkaAppender


Consumer // build a 'default' factory – your-implicit-kafka-consumer.properties

// - the properties contains the list of brokers from which it will receive messages

ConsumerFactory factory = ConsumerFactory.builder().build();

// obtain a consumer instance that can process Message<Payload> messages

Consumer<Message<Payload>> consumer = factory.createMessageConsumer(Payload.class);

// prepare the message (Message { header: Map<String, String>; payload: Payload } ) handler

ReceiverHandler<Message<Payload>> messageReceiver =

new ReceiverHandler<Message<Payload>>() {

@Override

public void onReceive(Message<Payload> message) {

System.out.printf("message received: %s%n", message);

}

};

// register the handler to receive messages from the topic named “test”

consumer.registerHandler("test", messageReceiver);

© ING 2016


Consumer

your-implicit-consumer.properties

your-security.properties


Serialize


Filter


Transformer


Forward & Dispatch


Replay


Features

Perf

orm

an

ce


Features

FIL

TE

RIN

G

DIS

PA

TC

H

Perf

orm

an

ce


Features F

ILT

ER

ING

D

ISP

AT

CH

Perf

orm

an

ce


Use microservices with Kafka

… code ahead of the curve

Vienna • 16 March 2017

László-Róbert ALBERT https://www.linkedin.com/in/albertlaszlorobert

http://www.linkedin.com/company/ing

Voxxed Days Vienna 2017 - Microservices with Kafka

Software