Page 1
@crichardson
Events on the outside, on the inside and at the core
Chris Richardson
Founder of the original CloudFoundry.com Author of POJOs in Action
@crichardson [email protected] http://plainoldobjects.com http://microservices.io http://eventuate.io
Page 2
@crichardson
Presentation goal
How event sourcing enables the event-driven
enterprise
Page 3
@crichardson
About Chris
Page 4
@crichardson
About Chris
Consultant and trainer focusing on microservices (http://www.chrisrichardson.net/)
Page 5
@crichardson
About Chris
Founder of a startup that is creating a platform that makes it easy for
application developers write microservices
(http://bit.ly/trialeventuate)
Page 6
@crichardson
For more information
https://github.com/cer/event-sourcing-examples
http://microservices.io
http://plainoldobjects.com/
https://twitter.com/crichardson
http://eventuate.io/
Page 7
@crichardson
Agenda
Events on the outside
Events on the inside
Events at the core with event sourcing
Designing event-centric domain model
Page 8
@crichardson
What’s an event?
http://www.merriam-webster.com/dictionary/event
Page 9
@crichardson
Examples of events
Page 10
@crichardson
Instacart: event-driven grocery shopping
Page 11
Who wants to be notified?Humans via
Email
SMS
Mobile push
…
Applications
Applications that implement other parts of the business process, e.g. order fulfillment system
Analytics dashboards, monitoring, …
Applications that deliver notifications to humans: Twilio, SendGrid, …
…
Page 12
@crichardson
How: Polling for events
HTTP
Periodically poll for events
AtomPub
Based on HTTP
Head is constantly changing
Tail is immutable and can be efficiently cached
High-latency, inefficient
Page 13
@crichardson
Using WebSockets
Browser
Web Socket
STOMP
Client application
Service
Web Socket
STOMP
ApplicationSUBSCRIBE
MESSAGEMESSAGE
Low latency, more efficient, but what about past events?
Page 14
@crichardson
Webhooks = user-defined HTTP callback
Client Service
register(events, callbackUrl)
POST callbackUrl
POST callbackUrl
…
https://en.wikipedia.org/wiki/WebhookLow latency, more efficient, but what
about past events?
Page 15
@crichardson
Github webhookshttps://developer.github.com/webhooks/
Installed on an organization or repository
e.g. POST /repos/:owner/:repo/hooks
Available events:
push - push to a repository
fork - repository is forked
pull_request - assigned, unassigned, …
push - push to a repository
…
Page 16
Twilio - Telephony and SMS as a service
REST API
Allocate phone numbers
Make and receive phone calls
Send and receive SMS messages
Pay per use:
Phone calls - per-minute
SMS – per SMS sent or received
Phone number – per month
Examples
OpenVBX is a web-based, open source phone system
StubHub – notifies sellers of a pending sale via phone
SurveyMonkey – interactive polling
Salesforce – SMS-based voting for 19,000 conference attendees
Page 17
@crichardson
Using Twilio
Twilio Your Application
TwiML doc
HTTP GET/POST
REST API
Manage resourcesSend SMS
Initiate voice calls
Webhooks handle incoming SMS and voice calls
VoiceSMS
Phone number ⇒ SMS URL + VOICE URL
Page 18
@crichardson
Integration hubs - Zapier, IFTTTApplication abstraction:
Triggers - events published by application: polling or Webhooks
Action - operation supported by application, e.g. REST API end points
Page 19
@crichardson
WebHooks =
web friendly publish/subscribe
Page 20
@crichardson
Enterprise integration patterns
http://www.enterpriseintegrationpatterns.com/patterns/messaging/
Page 21
@crichardson
messaging system
Messaging-based IPC
Sender RecipientChannelmessage
Page 22
@crichardson
Example messaging systems
Page 23
@crichardson
The event-driven enterprise
App A App BApp X
App Y
Firewall
Message Broker
webhooks
WebSockets
AtomPub
App Z App C
MsgBrokerClient Msg
BrokerClient
MsgBrokerClient
App D
MsgBrokerClient
Inside the firewallOutside the firewall
Page 24
@crichardson
Agenda
Events on the outside
Events on the inside
Events at the core with event sourcing
Designing event-centric domain model
Page 25
@crichardson
Traditional monolithic architecture
Simple to develop
test deploy scale
Shopping Cart
CustomersOrders
Mobile client RESTAPI
Browser WebUI
MessageBroker
Adapter
MySQLAdapter
RDBMS
…
Message Broker
Page 26
@crichardson
But that leads* to monolithic hell
For large and/or complex applications…
Page 27
@crichardson
Today: use a microservice, polyglot architecture
Orders
Customers
…Shopping UI
Mobile Client
Browser
API Gateway
Order management
RESTAPI
CustomerManagement
RESTAPI
….
RESTAPI
OrderDatabase
(MongoDB)MongoDBAdapter
CustomerDatabase(Sharded MySQL)
MySQLAdapter
Page 28
@crichardson
But now we have distributed data management
problems
Page 29
@crichardson
Example: placing an order
Order Service Customer Service
Order Database
Customer Database
Order #1 Customer #1
No 2PC
No ACID
NoSQL SQL
Page 30
@crichardson
Customer management
How to maintain invariants?
Order management
Order Service
placeOrder()
Customer Service
updateCreditLimit()
Customer
creditLimit ...
has ordersbelongs toOrder
total
Invariant: sum(open order.total) <= customer.creditLimit
?
Page 31
@crichardson
Use an event-driven architecture
Services publish events when something important happens, e.g. state changes
Services subscribe to events and update their state
Maintain eventual consistency across multiple aggregates (in multiple datastores)
Synchronize replicated data
Page 32
@crichardson
Event-driven application architectureApplication
Service A Service B
Message Broker
Service C
MsgBrokerClient
MsgBrokerClient
MsgBrokerClient
Service D
MsgBrokerClient
Page 33
@crichardson
Order ManagementOrder
id : 4567 total: 343 state = CREATED
Customer Management
Customer creditLimit : 12000 creditReservations: {}
Customer creditLimit : 12000 creditReservations: { 4567 -> 343}
Order id : 4567 total: 343 state = OPEN
Eventually consistent credit checking
Message Bus
createOrder()
Publishes:Subscribes to:
Subscribes to:
publishes:
OrderCreatedEvent
CreditReservedEvent
OrderCreatedEvent CreditReservedEvent
Page 34
@crichardson
Now there are two problems to solve….
Page 35
@crichardson
Problem #1: How to design eventually consistent business logic?
More on that later….
Page 36
@crichardson
Problem #2: How to atomically update database and publish an event
Order Service
Order Database
Message Broker
insert Order
publish OrderCreatedEvent
dual write problem
?
Page 37
@crichardson
Update and publish using 2PC
Guaranteed atomicity BUT
Need a distributed transaction manager
Database and message broker must support 2PC
Impacts reliability
Not fashionable
2PC is best avoided
Page 38
@crichardson
Transaction log tailingHow:
Read the database “transaction log” = single source of truth
Publish events to message broker
LinkedIn databus https://github.com/linkedin/databus
Supports Oracle and MySQL
Publish as events
AWS DynamoDB streams
Ordered sequence of creates, updates, deletes made to a DynamoDB table
Last 24 hours
Subscribe to get changes
MongoDB
Read the oplog
Page 39
Transaction log tailing: benefits and drawbacks
Benefits
No 2PC
No application changes required
Guaranteed to be accurate
Drawbacks
Immature
Database specific solutions
Low-level DB changes rather business level events = need to reverse engineer domain events
Page 40
@crichardson
Use database triggers
Track changes to tables
Insert events into an event table
Use datastore as a message queue
Pull events from event table and write to message broker
Page 41
Database triggers: benefits and drawbacks
Benefits
No 2PC
No application changes required
Drawbacks
Requires the database to support them
Database specific solutions
Low-level DB changes rather business level events = need to reverse engineer domain events
Error-prone, e.g. missing trigger
Page 42
@crichardson
Application created events
Use datastore as a message queue
Txn #1: Update database: new entity state & event
Txn #2: Consume event
Txn #3: Mark event as consumed
Eventually consistent mechanism (used by eBay)
See BASE: An Acid Alternative, http://bit.ly/ebaybase
Page 43
Application created eventsBenefits
High-level domain events
No 2PC
Drawbacks
Requires changes to the application
Only works for SQL and some NoSQL databases
Error-prone
Page 44
@crichardson
Agenda
Events on the outside
Events on the inside
Events at the core with event sourcing
Designing event-centric domain model
Page 45
@crichardson
Just publish events
Application
Database
Message Broker
update
publish
X
Page 46
@crichardson
Event sourcingFor each aggregate (business entity):
Identify (state-changing) domain events
Define Event classes
For example,
ShoppingCart: ItemAddedEvent, ItemRemovedEvent, OrderPlacedEvent
Order: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
Page 47
@crichardson
Persists events NOT current state
Order
status ….
Event table
101 ACCEPTED
Order tableXOrderCreated101 901 …
OrderApproved101 902 …
OrderShipped101 903 …
Page 48
@crichardson
Replay events to recreate state
Order
state
OrderCreated(…) OrderAccepted(…) OrderShipped(…)
Events
Periodically snapshot to avoid loading all events
Page 49
@crichardson
The present is a fold over history
currentState = foldl(applyEvent, initialState, events)
Page 50
@crichardson
Domain logic = event-driven aggregates
CustomerAggregateOrder
Aggregate
Order CreatedEvent Check Credit
Command
Credit ReservedEvent
Approve OrderCommand
Create OrderCommand
External request
Emits events Event ⇒ Command
Page 51
@crichardson
Request handling in an event-sourced application
HTTP Handler
Event Store
pastEvents = findEvents(entityId)
Order
new()
applyEvents(pastEvents)
newEvents = processCmd(SomeCmd)
saveEvents(newEvents)
Microservice A
(optimistic locking)
Page 52
@crichardson
Event Store publishes events - consumed by other services
Event Store
Event Subscriber
subscribe(EventTypes)
publish(event)
publish(event)
Aggregate
CQRS View
update()
update()
Microservice B
send notifications
…
Page 53
Event store = database + message broker
Hybrid database and message broker
Implementations:
Home-grown/DIY
geteventstore.com by Greg Young
http://eventuate.io (mine)
Event Store
Save aggregate
events
Get aggregate
events
Subscribe to events
Page 54
@crichardson
Benefits of event sourcingSolves data consistency issues in a Microservice/NoSQL-based architecture
Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…
Eliminates O/R mapping problem (mostly)
Reifies state changes:
Built-in, reliable audit log,
temporal queries
Preserved history ⇒ More easily implement future requirements
Page 55
@crichardson
Drawbacks of event sourcing
Weird and unfamiliar
Events = a historical record of your bad design decisions
Handling duplicate events can be tricky
Application must handle eventually consistent data
Event store only directly supports PK-based lookup => use Command Query Responsibility Segregation (CQRS) to handle queries
Page 56
@crichardson
Agenda
Events on the outside
Events on the inside
Events at the core with event sourcing
Designing event-centric domain model
Page 57
@crichardson
Use the familiar building blocks of DDD
Entity
Value object
Services
Repositories
Aggregates ⟸ essential
Page 58
Aggregate designGraph consisting of a root entity and one or more other entities and value objects
Each core business entity = Aggregate: e.g. customer, Account, Order, Product, ….
Reference other aggregate roots via primary key
Often contains partial copy of other aggregates’ data
Order
OrderLine Item
quantity productId productName productPrice
customerId
Address
street city …
Page 59
@crichardson
Partition the domain model into Aggregates
Order
OrderLine Item
quantity
…
Addressstreet city …
Customer
Productname price
Page 60
@crichardson
Transaction = processing one command by one aggregate
No opportunity to update multiple aggregates within a transaction
Aggregate granularity is important
If an update must be atomic (i.e. no compensating transaction) then it must be handled by a single aggregate
e.g. scanning boarding pass at security checkpoint or when entering jetway
Page 61
@crichardson
Aggregate granularity
Consistency Scalability/ User experience
Customer
Order
Product
Customer
Order
Product
Customer
Order
Product
Page 62
Designing domain eventsRecord state changes for an aggregate
Part of the public API of the domain model ProductAddedToCart
id : TimeUUID senderId: UUID productId productName productPrice …
Required by aggregate
Enrichment: Useful for
consumers
Event metadata
Page 63
@crichardson
Example event
Page 64
@crichardson
Designing commandsCreated by a service from incoming request
Processed by an aggregate
Immutable
Contains value objects for
Validating request
Creating event
Auditing user activity
Page 65
@crichardson
Example command
Page 66
@crichardson
Hybrid OO/FP domain objects
Page 67
@crichardson
OO = State + Behavior
creditLimit creditReservations : Map[OrderId, Money]
Customer
List<Event> processCommand ( Command aCommand) { … }
void applyEvent (Event anEvent) { … }
State
Behavior
Page 68
@crichardson
Aggregate traits
Map Command to Events
Apply event returning updated Aggregate
Used by Event Store
to reconstitute aggregate
Page 69
@crichardson
ReflectiveMutableCommandProcessingAggregate
Page 70
@crichardson
Customer - command processing
Page 71
@crichardson
Customer - applying events
Page 72
@crichardson
Event Store APIT is a subclass of Aggregate[T]
Rx Observable = Future++
Page 73
@crichardson
Creating an order
save() concisely specifies: 1.Creates Customer aggregate 2.Processes command 3.Applies events 4.Persists events
Page 74
@crichardson
Event handling in Customers
1.Load Customer aggregate 2.Processes command 3.Applies events 4.Persists events
Triggers BeanPostProcessor Durable subscription name
Page 75
@crichardson
Summary
Events are a central to modern applications
Events integrate applications
Events maintain data consistency in a microservices architecture
⇒
Build events into the core of your application using event sourcing
Page 76
@crichardson
@crichardson [email protected]
http://plainoldobjects.com http://microservices.io
Questions?