Developing event-driven microservices with event sourcing and CQRS (Shanghai)

@crichardson

Event-driven microservices with event sourcing and CQRS

Chris Richardson

Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action

@crichardson [email protected] http://eventuate.io July 2016

@crichardson

Presentation goal

Show how Event Sourcing and Command Query Responsibility

Segregation (CQRS) are a great way to implement

microservices

@crichardson

About Chris

@crichardson

About Chris

Consultant and trainer focusing on modern

application architectures including microservices

(http://www.chrisrichardson.net/)

@crichardson

About Chris

Founder of a startup that is creating a platform that makes it easier for developers to write transactional

microservices (http://eventuate.io)

@crichardson

For more information

http://learnmicroservices.io

@crichardson

Agenda

Monolith vs. microservices

Event-driven microservices

Overview of event sourcing

Implementing queries in an event sourced application

@crichardson

Let’s imagine you are building a large, complex application,

e.g. an online store

@crichardson

Successful software development

Architecture

Process OrganizationAgile Continuous delivery …

Small, autonomous, teams

@crichardson

?Architecture

@crichardson

@crichardson

The monolithic architecture

Tomcat

Browser

WAR

SQL database

HTML

REST/JSON

Client App

Simple to ….

Develop Test

Deploy Scale

Catalog Module

Reviews Module

Orders Module

StoreFront UI Module

@crichardson

But successful

applications keep

growing ….

@crichardson

Monolithic architecture


Small, autonomous, teams

@crichardson

Apply functional decomposition

X axis - horizontal duplication

Z axis

- data

partit

ioning

Y axis - functional

decomposition

Scale b

y split

ting s

imilar

thing

s

Scale by splitting

different things

@crichardson

Microservice architecture

Browser

Mobile Device

Store Front UI

API Gateway

Catalog Service

Review Service

Order Service

… Service

Catalog Database

Review Database

Order Database

… Database

HTML

REST

REST

REST

@crichardson

Microservice architecture


Small, autonomous, teams✔ ✔

@crichardson

Drawbacks

Complexity

@crichardson

DrawbacksComplexity of developing a distributed system

Implementing inter-process communication

Handling partial failures

Complexity of implementing business transactions that span multiple databases (without 2PC)

Complexity of testing a distributed system

Complexity of deploying and operating a distributed system

Managing the development and deployment of features that span multiple services

Fortunately solutions exists

@crichardson

The benefits typically outweigh the drawbacks

for large, complex applications

@crichardson

Issues to address

How to deploy the services?

How do the services communicate?

How do clients of the application communicate with the services?

How to partition the system into services?

How to deal with distributed data management problems?

….

@crichardson

Microservices pattern language

http://microservices.io/

@crichardson

Agenda





Data management patterns

@crichardson

The Database

Shared database

Order Service Customer Service … Service

Order table Customer table …

orderTotal creditLimit

Tight coupling Simple and

ACID

@crichardson

Database per service

Order Service Customer Service

Order Database Customer Database

Order table Customer table

orderTotal creditLimit

Loose coupling 😀 but more complex 😓 and….

@crichardson

2PC (aka. distributed transactions)

is not viable choice for most modern applications

@crichardson

Customer management

How to maintain data consistency without 2PC?

Order management

Order Service

placeOrder()

Customer Service

updateCreditLimit()

Customer

creditLimit ...

has ordersbelongs toOrder

total

Invariant: sum(open order.total) <= customer.creditLimit

?

@crichardson

Event-driven architecture

@crichardson

Use event-driven, eventually consistent order processing

Order Service

Customer Service

Order created

Credit Reserved

Credit Check Failed

Place Order

OR

@crichardson

How atomically update database and publish an event

Order Service

Order Database

Message Broker

insert Order

publish OrderCreatedEvent

dual write problem

?

@crichardson

Failure = inconsistent system

Order Service

Order Database

Message Broker

insert Order

publish OrderCreatedEvent

X

@crichardson

2PC is not an option

@crichardson

Reliably publish events when state changes

@crichardson

Agenda





@crichardson

Just publish events

Application

Database

Message Broker

update

publish

X

@crichardson

Event sourcingFor each domain object (i.e. DDD aggregate):

Identify (state changing) domain events, e.g. use Event Storming

Define Event classes

For example,

ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent

Order: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped

@crichardson

Persists events NOT current state

Order

status ….

101 ACCEPTED

Order tableX

@crichardson

Persists events NOT current state

Event table

Entity type Event id

Entity id

Event data

Order 902101 …OrderApproved

Order 903101 …OrderShipped

Event type

Order 901101 …OrderCreated

@crichardson

Replay events to recreate state

Order

state

OrderCreated(…) OrderAccepted(…) OrderShipped(…)

Events

Periodically snapshot to avoid loading all events

@crichardson

The present is a fold over history

currentState = foldl(applyEvent, initialState, events)

@crichardson

Familiar concepts restructured

class Customer {

public void reserveCredit( orderId : String, amount : Money) {

// verify

// update state this.xyz = … }

public List<Event> process( ReserveCreditCommand cmd) { // verify … return singletonList( new CreditReservedEvent(…); ) }

public void apply( CreditReservedEvent event) { // update state this.xyz = event.xyz }

@crichardson

Event

Aggregates: Command ⇒ Events

AggregateCommand Event

@crichardson

Aggregates: Event ⇒ Updated aggregate

AggregateEvent Updated Aggregate

@crichardson

Request handling in an event sourced application

HTTP Handler

Event Store

pastEvents = findEvents(entityId)

Order

new()

applyEvents(pastEvents)

newEvents = processCmd(someCmd)

saveEvents(newEvents) (optimistic locking)

Order Service

applyEvents(newEvents)

@crichardson

Event Store publishes events consumed by other services

Event Store

Event Subscriber

subscribe(EventTypes)

publish(event)

publish(event)

Customer

update()

Customer Service

@crichardson

Event Store publishes events consumed by other services

Event Store

Event Subscriber

subscribe(EventTypes)

publish(event)

publish(event)

CQRS View

update()

Service Xyz

send notifications

…

Event store = database + message broker

Hybrid database and message broker

Implementations:

Home grown/DIY

geteventstore.com by Greg Young

http://eventuate.io (mine)

Event Store

Save aggregate

events

Get aggregate

events

Subscribe to events

@crichardson

Benefits of event sourcingSolves data consistency issues in a Microservice/NoSQL based architecture

Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…

Eliminates O/R mapping problem (mostly)

Reifies state changes:

Built in, reliable audit log

temporal queries

Preserved history ⇒ More easily implement future requirements

@crichardson

Drawbacks of event sourcingRequires application rewrite

Weird and unfamiliar style of programming

Events = a historical record of your bad design decisions

Must detect and ignore duplicate events

Idempotent event handlers

Track most recent event and ignore older ones

…

Querying the event store can be challenging

Some queries might be complex/inefficient, e.g. accounts with a balance > X

Event store might only support lookup of events by entity id

Must use Command Query Responsibility Segregation (CQRS) to handle queries ⇒ application must handle eventually consistent data

@crichardson

Agenda





@crichardson

Find recent, valuable customers

SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ?

Customer Service

Order Service

What if event sourcing is

used?…. is no longer easy

@crichardson

Use Command Query Responsibility Segregation

(CQRS)

@crichardson

Command Query Responsibility Segregation (CQRS)

Application logic

Commands Queries

X

@crichardson

Command Query Responsibility Segregation (CQRS)

Command side

Commands

Aggregate

Event Store

Events

Query side

Queries

(Materialized) View

Events

@crichardson

Query side design

Event Store

Updater

View Updater Service

Events

Reader

HTTP GET Request

View Query Service

View Store

e.g. MongoDB

ElasticSearch Neo4J

update query

@crichardson

Persisting a customer and order history in MongoDB

{ "_id" : "0000014f9a45004b 0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } }

Denormalized = efficient lookup

@crichardson

Persisting customers and order info using Spring Data for MongoDB...

@crichardson

Persisting customers and order using Spring Data for MongoDB...

Benefits and drawbacks of CQRS

Benefits

Necessary in an event sourced architecture

Separation of concerns = simpler command and query models

Supports multiple denormalized views

Improved scalability and performance

Drawbacks

Complexity

Potential code duplication

Replication lag/eventually consistent views

@crichardson

Summary

Use microservices to accelerate development

Use an event-driven architecture to maintain data consistency

Implement an event-driven architecture using event sourcing

Use CQRS to implement materialized views for queries

@crichardson

@crichardson [email protected]

http://learnmicroservices.io

Questions?

Developing event-driven microservices with event sourcing and CQRS (Shanghai)

Software