Building event-drivenMicroservices withKafka Ecosystem · Building event-driven Microservices with Kafka Ecosystem •service autonomy is key in a Microservices Architecture! •not

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH

Building event-driven Microservices with Kafka EcosystemGuido SchmutzLondon, 30.5.2018

@gschmutz guidoschmutz.wordpress.com

Guido Schmutz

Working at Trivadis for more than 21 years

Oracle ACE Director for Fusion Middleware and SOA

Consultant, Trainer Software Architect for Java, Oracle, SOA and

Big Data / Fast Data

Head of Trivadis Architecture Board

Technology Manager @ Trivadis

More than 30 years of software development experience

Contact: [email protected]

Blog: http://guidoschmutz.wordpress.com

Slideshare: http://www.slideshare.net/gschmutz

Twitter: gschmutz

Building event-driven Microservices with Kafka Ecosystem

mailto:[email protected]

http://guidoschmutz.wordpress.com/

http://www.slideshare.net/gschmutz

http://twitter.com/

Agenda

1. Where do we come from?

2. What are Microservices?

3. Why not Event Driven?

4. What about streaming sources?

5. What about integrating legacy applications?

6. What about (historical) data analytics?

7. Why Kafka for Event-Driven Microservices?

8. Summary



Where do we come from?

Shop Rich UI

Shop Backend Application

Traditional Approach

Search Facade

Customer DAO

Order DAO

Order Facade

Shop UI

Product DAO

UI Logic

DataBusiness

GUI

Customer Fat Client App

Customer BOCustomer UIDataGUI

Data Storage

Shared Database


sync request/response

Shop UI App

Business Activity Service

SOA Approach


Contract-first Web Services

Technical layers offer their own interfaces

Reuse on each level

Lower layer often wraps legacy code

Search BAS

Customer DAO

Order DAO

Order BAS

Shop UI

Product DAO

UI LogicGUI

Business Entity ServiceShop Web App

Shop UI UI LogicGUI

Data Storage

CustomerDatabase

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order andProduct DBSOAP

SOAP

SOAP SOAP

SOAP

SOAP

SOAP

Shop UI App

Business Activity Service

Virtualized SOA Approach

Search BASCustomer DAO

Order DAO

Order BAS

Shop UI UI LogicGUI

Business Entity Service

Shop Web App

Shop UI UI LogicGUI

Data Storage

CustomerDatabase

Customer BES

Payment BES

Product BES

Order BES

Custer BAS

Order andProduct DB

Service Virtualization Layer

Service Bus

SOAP SOAP

SOAP

SOAP

SOAP

SOAP

SOAP



What are Microservices?

What are Microservices?


Tightly Scoped behind clear interfaces

Responsible for managing their own data (not necessarily the infrastructure)

Should be highly decoupled

Independently deployable, self-contained and autonomous

SOA done right ?!

Microservice Approach


Tightly Scoped

Responsible for managing their own data

Highly decoupled

Independently deployable, self-contained and autonomous

{ }Customer API

CustomerCustomer Logic

Order Microservice

{ }Order API OrderOrder Logic

Product Microservice

{ }Product API ProductProduct Logic

Stock Microservice

{ }Stock API StockStock Logic

Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

Microservice Approachwith API Gateway

Customer Microservice

{ }Customer API


Order Microservice




Stock Microservice


Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

API Gateway


Synchronous World of Request-Response leads to tight, point-to-point couplings


problem in lower end of chain have a ripple effect on other service• crash of service• overloaded service / slow response time• change of interface

Service 2Service 1

{ }

APILogic

{ }

API Logic

StateState

Service 3

{ }

API Logic

State

Service 4

{ }

API Logic

State

Service 5

{ }

API Logic

State

Service 7

{ }

API Logic

State

Service 6

{ }

API Logic

State


Why not Event-Driven?

3 mechanisms through which services interact

Request-Driven (Imperative) Event Driven (Functional)

Service

Logic

State

Event Consume

OrderEvent{iPad}

Command”Order IPad”boolean order(IPad)

EventPublish

OrderValidatedEvent{iPad}

Query”Retrieve my OrdersList<Orders> getAllOrders(for)


Event-Driven (Async)Microservice Approach


{ }Customer API


Order Microservice




Stock Microservice


Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

API Gateway


Event Store

sync request/response

async request/response

async, event pub/sub

What about streaming sources?


How to work with streaming data sources


{ }Customer API


Order Microservice




Stock Microservice


Shop Web App

Shop UI UI LogicGUI

REST

REST

REST

REST

Event Store

Location

Social

Clickstream

Sensor Data

MobileApps

WeatherData

Event Stream


Hadoop ClusterdHadoop Cluster

Stream Processing Cluster

Streaming Analytics ArchitectureBI Tools

SQL

Search / Explore

Online & Mobile Apps

Search

ServiceEvent Stream

ResultsStream Analytics

Reference /Models

Dashboard

Location

Social

Clickstream

Sensor Data

MobileApps

WeatherData

Microservice Cluster

Microservice State

{ }

API

EventStream

EventStream

Event Store

Service


What about integrating legacy applications?


Data Store

Integrate existing systems through CDC

Customer

Event Store

Integration

Microservice

StateLogic

CDCCDC Connector

Customer Fat Client App

Customer BOCustomer UI

Stream Processing

ResultsStream Processor

Reference /Models

Dashboard

Capture changes directly on database

Change Data Capture (CDC) => think like a global database trigger

Transform existing systems to event producer




Integrate existing systems through CDCBI Tools

SQL

Search / Explore


Search

Service

Event Stream

ResultsStream Processor

Reference /Models

DashboardLocation

Social

Clickstream

Sensor Data

MobileApps

WeatherData Microservice Cluster

Microservice State

{ }

API

EventStream

EventStream

Event Store

Service

Billing &Ordering

CRM / Profile

MarketingCampaigns

Change DataCapture


What about (historical) data analytics?


Streaming & (Big) Data Analytics Architecture

Event Stream

Hadoop ClusterdHadoop ClusterBig Data Cluster

Data Fl

ow

Parallel Processing

Storage

Storage

Raw

Ref

ined

Results

Microservice Cluster

Microservice State

{ }

API


StreamProcessor

State

{ }

API

EventStream

EventStream

SQL

Search

Service

BI Tools

Enterprise Data Warehouse

Search / Explore


SQL Export

SearchEvent Store

Service

Location

Social

Clickstream

Sensor Data

MobileApps

WeatherData

Billing &Ordering

CRM / Profile

MarketingCampaigns

Change DataCapture

File Import / SQL Import


Why Kafka for Event-Driven Microservices?


Apache Kafka – A Streaming Platform


High-Level Architecture

Distributed Log at the Core

Scale-Out Architecture

Logs do not (necessarily) forget

Apache Kafka – Schema Registry


Producer

Kafka Broker

Consumer

Schema Registry

Kafka Kafka

avroschema

avroavro

avroschema

Apache Kafka – scalable message processing and more!


Source Connector

trucking_driver

Kafka BrokerSink

Connector

Stream Processing

Schema Registry

Kafka Kafka

RESTful RESTful

Legacy Microservice

Change Data Capture (CDC)


RDBMS cdc-source trucking_driver

Customer Topicelasticsearc

h-sink NoSQL

Demo



{ }Customer API


Order Microservice


REST

REST

Kafka

Customer

Hold Data for Long-Term – Data Retention

Producer 1

Broker 1

Broker 2

Broker 3

1. Never

2. Time based (TTL) log.retention.{ms | minutes | hours}

3. Size based log.retention.bytes

4. Log compaction based (entries with same key are removed):

kafka-topics.sh --zookeeper zk:2181 \--create --topic customers \--replication-factor 1 \--partitions 1 \--config cleanup.policy=compact


Topic Viewed as Event Stream or State Stream (Change Log)

Event Stream State Stream (Change Log Stream)

2015-10-02 11,Peter,Muster,3010,Berne

2016-10-04 12,Paul,Steffen,8001,Zurich

2016-12-02 21,Lisa,Meier,3043,Ittigen

2017-05-03 11,Peter,Muster,3015,Berne2017-05-03 21,Lisa,Steffen,8001,Zurich2017-07-03 11,Peter,Muster,3052,Zollikofen


2015-10-02 11,Peter,Muster,3010,Berne

2016-10-04 12,Paul,Steffen,8001,Zurich

2016-12-02 21,Lisa,Meier,3043,Ittigen

2017-05-03 11,Peter,Muster,3015,Berne2017-05-03 21,Lisa,Steffen,8001,Zurich

2017-07-03 11,Peter,Muster,3052,Zollikofen

Keep Topics in Compacted Form

0 1 2 3 4 5 6 7 8 9 10 11

K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2

V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11

OffsetKeyValue

3 4 6 8 9 10

K1 K3 K4 K5 K2 K6

V4 V5 V7 V9 V10 V11

OffsetKeyValue

Compaction


V1 V2

V3

V4 V5

V6

V7

V8

V9V10 V11

K1 K3 K4 K5K2 K6

Keep Topics both in Original and Compacted Form


Kafka BrokerA B C B A E K A E A C K

B E A C KProducer

Kafka BrokerA B C B A E K A E A C K

B E A C K

Producer

Consume and Produce

OR

TX

Enrich Stream with Static Data with Kafka Streams


Order Topic Consume and Produce

CustomerTable

Customer Topic Customer Handler

Join

Customer Local State

Order & Customer Topic

Summary



Big Data

Summary

Billing &Ordering

CRM / Profile

MarketingCampaigns

SQL

Search

Service

BI Tools

Enterprise Data Warehouse

Search / Explore


File Import / SQL Import

Event Hub

Data

Flow

Data

Flow

Change Data

Capture

Parallel Processing

Storage

Storage

Raw

Refined

Results

SQL Export

Microservice State

{ }

API

StreamProcessor

State

{ }

API

EventStream

EventStream

Search

Service

Location

Social

Clickstream

Sensor Data

MobileApps

WeatherData

Stream Processing

Microservices

Summary


• service autonomy is key in a Microservices Architecture!

• not all communication need to be synchronous => separate into• commands

• events

• queries

• Kafka is well suited as an event broker / event store• brings many more interesting features beyond just “message passing”

References


Microservices Blog Series, Ben Stopford, Confluent:

• https://www.confluent.io/blog/tag/microservices

Apache Kafka for Microservices: A Confluent Online Talk Series:

• https://www.confluent.io/landing-page/microservices-online-talk-series/

Turning the database inside-out with Apache Samza, Martin Kleppmann, Con

• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:

• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/

Immutability Changes Everything, Pat Helland, Salesforce:

• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

Commander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:

• https://www.youtube.com/watch?v=B1-gS0oEtYc

https://www.confluent.io/blog/tag/microservices

https://www.confluent.io/landing-page/microservices-online-talk-series/

https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/

http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

https://www.youtube.com/watch?v=B1-gS0oEtYc

Technology on its own won't help you.You need to know how to use it properly.


Building event-drivenMicroservices withKafka Ecosystem · Building event-driven Microservices with Kafka Ecosystem •service autonomy is key in a Microservices Architecture! •not

Documents