Primer on CQRS and Event Sourcing

SEPARATING & EVENTING.Introducing CQRS and Event Sourcing.

PAUL DE RAAIJ.Developer @ Coolblue

• Primer on CQRS & Event Sourcing

• Guided by Coolblue systems and challenges

THE GAME PLAN.

INTRODUCE COOLBLUE SYSTEMS

FRONTENDBACKEND

COMPONENTS OVERVIEWApplications and data stores in play

Vanessa CMS

Oracle DB MySQL

Website

BI Reporting CRM Service

REPLICATION

MESSAGING

READ CAPACITY VS WRITE CAPACITYWe do far more read operations then we do write operations

FRONTENDBACKEND

EASY, JUST ADD MORE HARDWAREOh, and stay clear from the CFO

Vanessa CMS

Oracle DB MySQL

Website


REPLICATION

MESSAGING

MySQLMySQL

Oracle DBOracle DB

SPLIT THE CONCERNSSeparate read operations from write operations

FRONTENDBACKEND

SCALE READS BY ADDING SLAVESSeparate write operations from read operations

Vanessa CMS

Oracle DBMySQL

<master>

Website


REPLICATION

MESSAGING

MySQL<slave>

REPLICATION

WRITING READING

MySQL<slave>

INCREASED COMPLEXITYCode mixes concerns of reading and writing

FRONTEND

WRITING

BACKEND

SEPARATE CONCERNSA layer responsible for reading, a layer responsible for writing

Vanessa CMS

Oracle DB

Website


REPLICATION

MESSAGING

READING

CMS

Website

BI Reporting

MySQL<master>

MySQL<slave>

REPLICATION

WRITING READING

MySQL<slave>

CQRS TO THE RESCUE

FIRST A LITTLE HISTORYMeet Bertrand Meyer.

• Designer of Eiffel programming language

• Coined ‘design by contract’

• Coined ‘open/closed principle’

• Introduced command query separation (CQS)

BERTRAND MEYER.

• Separate methods that change and read state

• Command methods for changing state

• Query methods to read state

• Most likely from the same datastore & object model

COMMAND QUERY SEPARATION.

CQS BOILS DOWN TO THIS.

Database

COMMANDS QUERIES

Controller

Business Layer

class Product{ protected $active; /** * Command method activating a product; Changes state */ public function activate() { $this->active = true; } /** * Command method deactivating a product; Changes state */ public function deactivate() { $this->active = false; } /** * Query method which returns the active state of the product; Read state */ public function isActive() { return $this->active; } }

CQS Example - 1/1

• CQS extended

• Not separation by methods, but by classes

• Command objects change state

• Query objects read state

• Brings separation of concerns

COMMAND QUERY RESPONSIBILITY SEPARATION.

FRONTEND

WRITING

BACKEND

SEPARATED CONCERNSUse of commands and queries separates write and read concerns

Vanessa CMS

Oracle DB

Website


REPLICATION

MESSAGING

READING

CMS

Website

BI Reporting

MySQL<master>

MySQL<slave>

REPLICATION

COMMANDS QUERYING

MySQL<slave>

COMMANDSTell to do something

• Command is the message to change state

• Message handled by a CommandHandler

• Returns void (so nothing)

• Describes business intent

• Immutable

COMMAND CHARACTERISTICS.

QUERYFetch state from data store

• Read state from data store

• Does not change state

• Has a return value

QUERY CHARACTERISTICS.

CQRS IN A PICTURE

FRONTEND

WRITING

BACKEND

SEPARATED READ AND WRITE CONCERNSLet those millions of visitors do their thing

Vanessa CMS

Oracle DB

Website


REPLICATION

MESSAGING

READING

CMS

Website

BI Reporting

MySQL<master>

MySQL<slave>

REPLICATION

COMMANDS QUERIES

MySQL<slave>

DIFFERENT VIEWS ON DATAEach component has a specific view on the data

FRONTEND

WRITING

BACKEND

COMPONENTS WITH DIFFERENT NEEDSBut are using the same data model

Vanessa CMS

Oracle DB

Website


REPLICATION

MESSAGING

READING

CMS

Website

BI Reporting

MySQL<master>

MySQL<slave>

REPLICATION

WRITING READING

MySQL<slave>

CQRS TO THE RESCUE

ZOOMING IN ON THE PRODUCT ENTITYEach component has its own definition of a product

USAGE PRODUCT ENTITY.

Vanessa BI Reporting Website

Goal: provide CRUD oriented interface for manipulating product information

Goal: Create a price report for our entire catalog which we can use for comparison against our competitors

Goal: Display product information to website visitors in the least amount of time

Product: - Product ID - Product Name - Published

ProductAvailability: - Availability ID - Stock Location - Stock amount

ProductSalesPrice: - ProductSalesPrice ID - Price excluding VAT - Price including VAT

Product: - Product ID - Product Name - Price excluding VAT - Price including VAT

Product: - Product ID - Product Name - Published - Stock amount - Price excluding VAT - Price including VAT

FRONTEND

WRITING

BACKEND

PROVIDE MULTIPLE DATA STORESUsing different data sets tailored on the component usage

Vanessa CMS

Oracle DB

Website


MESSAGING

READING

CMS

Website

BI Reporting

Data store (representation)

Data store (reporting)

Magic replication process

WRITING

MySQL<master>

REPLICATION

FRONTEND

WRITING

BACKEND

OPTIMIZE MODEL FOR QUERYINGRead models used for query results

Vanessa CMS

Oracle DB

Website


MESSAGING

READING

CMS

Website

BI Reporting

Datastore (representation)

Datastore (reporting)

READMODEL

WRITING


WRITING

MySQL<master>

REPLICATION

• Model specialized for queries

• Data Transfer Object

• Can be used

• Dynamically

• Flattened persistence

READ MODEL.

FRONTEND

WRITING

BACKEND

DYNAMIC USE FOR READ MODELQuery prepares data to populate read model on run time

Vanessa CMS

Oracle DB

Website


MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

WRITING


WRITING

MySQL<master>

REPLICATION

• Ideal for queries that aren’t heavily used

• Saves persisting flattened representation of model

• Queries do most of the work

DYNAMIC USE OF READ MODEL.

FRONTEND

WRITING

BACKEND

FLATTENED READ MODELPersist flattened representation in datastores

Vanessa CMS

Oracle DB

Website


MESSAGING

READING

CMS

Website

BI Reporting



READ MODEL

WRITING


WRITING

MySQL<master>

REPLICATION

• Ideal for queries that are heavily used

• Hard work done during persisting

• Querying will be as simple as `select * from TABLE`

FLATTENED READ MODEL.

FRONTEND

WRITING

BACKEND

BEST OF BOTH WORLDSHybrid model which uses flattened and dynamic model

Vanessa CMS

Oracle DB

Website


MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

WRITING


WRITING

MySQL<master>

REPLICATION

READ MODEL

REPLICATING INFORMATIONSynchronizing the read data stores with the transactional data store

• CQRS does not enforce any mechanism

• Possible options:

• Database replication

• Messaging

• Eventing

REPLICATION: IT’S UP TO YOU.

SYNCHRONIZE VIA MESSAGINGUse message bus for updating all data stores

CMSData store

(representation)

Data store

Message bus

CreateProductCommandName: ‘Product NEW’

CommandHandler

CommandHandler


CommandHandler

FRONTEND

WRITING

BACKEND

DEFINING THE MAGICAL REPLICATIONAdding command bus, commands and command handlers

Vanessa CMS

Oracle DB

Website

BI Reporting

CRM Service

MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

MySQL<master>

Command bus

Command Handlers Command Handlers

Command Handlers COMMANDSCOMMANDS

CONSISTENCY

• Data that has been written should be available immediately

• For example

• Adding a product to the shopping basket

• Decreasing the stock of a product

IMMEDIATE CONSISTENCY.

• Data that has been written should be available whenever possible

• For example

• Adding a review for a product

• Altering product content

EVENTUAL CONSISTENCY.

• Eventually consistency impacts workflow

• Can the business handle this workflow

SYNCHRONOUS VS ASYNCHRONOUS WORKFLOW.

HANDLING THE UNEXPECTEDThe business will ask questions you didn’t expect upfront

• List all users that have accessed order ‘X’ in 2014

• List all products that were removed from a shopping basket for shop ‘Z’ in period ‘Q’ to ‘R’

BUSINESS REQUESTS.

EVENT SOURCING

NOT ABOUT PERSISTING STATEAbout capturing a sequence of events to change state

YOU MIGHT KNOW THISGIT is event sourced

• Relates to most domains in the real world

• Account at the bank

• Lifecycle of a product

• No coupling between representation of state and storage

• Events can be replayed

• Append-only model scales easier

• Extra bonus: auditing trail

WHY USE EVENT SOURCING.

• Events describe something that happened in the past

• Represents action

• Record user intent

WHAT ARE EVENTS.

EVENTS LEAD TO OBJECT STATE.

ProductCreatedEvent Name: ‘Product X’

ProductPublishedEventActive: true

ProductRenamedEvent Name: ‘Product NEW’

ProductPricedEventPrice: EUR 600,-

Product:Name: Product NEWPrice: EUR 600,-Active: true

YOUR DATABASE WILL LOOK LIKE THIS.

UUID Sequence Event Payload Date Issuerdb75c660-ce28-11e4-83f2-0002a5d5c51b 1 CreatedProductEvent {

"name": "test" }

2015-03-19 12:45:03 User A

db75c660-ce28-11e4-83f2-0002a5d5c51b 2 RenamedProductEvent { "name": “Product X" }

2015-03-19 13:01:23 User B

82669760-ce29-11e4-8ae9-0002a5d5c51b 8 DeletedProductEvent {} 2015-03-21 18:03:19 User B

SYNCHRONIZE VIA EVENTINGUse events for updating data stores

CMS

Data store (representation) Event Store

Command bus

CreateProductCommandName: ‘Product NEW’

CommandHandler Event Listener Event ListenerCreatedProductEventName: ‘Product NEW’

Event bus


FRONTEND

WRITING

BACKEND

EVENT SOURCING OUR DATAStop storing state, but persist intentions and events

Vanessa CMS

Event store

Website

BI Reporting

CRM Service

MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

Event store

Command bus


Event listeners COMMANDSCOMMANDS

Event bus

EVENTS

PROJECTIONSDerive state from a stream of events

• Aggregating events into structural representation

• Replays events applying business requirements

PROJECTIONS.

FRONTEND

WRITING

BACKEND

PROJECT EVENTS INTO DATA REPRESENTATIONAggregate events to structural representations

Vanessa CMS

Event store

Website

BI Reporting

CRM Service

MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

Event store

Command bus


Event listeners

COMMANDSCOMMANDS

Event bus

EVENTS

Projections

DATA REQUIREMENTS OF COMPONENT.

Vanessa BI Reporting Website

Goal: provide CRUD oriented interface for manipulating product information

Goal: Create a price report for our entire catalog which we can use for comparison against our competitors

Goal: Display product information to website visitors in the least amount of time

Product: - Product ID - Product Name - Published

ProductAvailability: - Availability ID - Stock Location - Stock amount

ProductSalesPrice: - ProductSalesPrice ID - Price excluding VAT - Price including VAT

Product: - Product ID - Product Name - Price excluding VAT - Price including VAT

Product: - Product ID - Product Name - Published - Stock amount - Price excluding VAT - Price including VAT

PROJECTING PRODUCTS TO OUR DATA STORES.

ProductCreatedEventName: ‘Product X’

ProductPublishedEventActive: true

ProductRenamedEventName: ‘Product NEW’


Product:Name: Product NEWPrice: EUR 600,-Active: true

REPRESENTATION PROJECTION

ProductCreatedEventName: ‘Product X’

ProductRenamedEventName: ‘Product NEW’


Product:Name: Product NEWPrice: EUR 600,-

REPORTING PROJECTION

• List all users that accessed order ‘X’ in 2015

• List the times that product ‘Y’ has been removed from a shopping basket and been replaced by product ‘Z’

• List all changes between ‘Q’ and ‘R’ executed by user ‘Z’

COOLBLUE PROJECTIONS.

FRONTEND

WRITING

BACKEND

Vanessa CMS

Event store

Website

BI Reporting

CRM Service

MESSAGING

READING

CMS

Website

BI Reporting



READMODEL

Event store

Command bus


Event listeners

COMMANDSCOMMANDS

Event bus

EVENTS

Projections

OVERVIEW AFTER APPLYING PATTERNSCQRS and Event Sourcing helped to tackle our challenges

• Scale read operations differently from write operations

• Simplicity in domain model by separating read and writes

• Added auditing trail

• Added ability to capture intent over state

ACHIEVEMENTS.

REFLECTION

• Powerful patterns, but do they apply to your needs?

• CQRS probably influences your UI and thus the business workflow (Synchronous workflow vs Asynchronous workflow)

REFLECTING.

• CQRS is not easy

• Event sourcing is not mandatory for CQRS

• Commands are not fire and forget

• CQRS does not eliminate thinking about concurrency

BUST SOME CQRS MYTHS.

• Look at Broadway framework

• Experiment with your own implementation

• Do not take things for granted, not even this presentation

FOOD FOR THOUGHT.

QUESTIONS?

Paul de RaaijE-mail: [email protected]: @pderaaijSlideshare: http://www.slideshare.net/pderaaij

THANK YOU!

Primer on CQRS and Event Sourcing

Engineering

active state

state command methods

state message

queryfetch state

state query objects

product changes state

read operations

changing state query