Transcript
Hello!
My#name#is#Jacopo&Nardiello
• Programmer,*currently*located*in*London
• I*care*about*code*quality
• If*it's*not*tested,*it*doesn't*exist.
If#you#want#to#follow#me#@jnardiello
About&Onebip
Mobile'payments!pla%orm.Start,up!born!in!2005,!acquired!by!Neomobile!group!in!2011.!
Onebip'today:.'70'countries.'200+'carriers.'5'billions'registered'users
#1
Avoid&Mul*ple&sources&of&truthIt's%hard%to%compare%different%data%from%different%in%a%distributed%
system%spli7ed%across%mul8ple%domains
Eventsourcing
The$fundamental$idea$of$Event$Sourcing$is$that$of$ensuring(every(change(to(the(state(of(an(applica3on(is(captured(in(an(event(object,$and$that$these$event$objects$are$themselves$stored(in(the(sequence(
they(were(applied
—(Mar3n(Fowler
Unrolling(a(stream(of(eventsStar%ng(from(the(beginning(of(%me,(you(are(
literally(unrolling(history(to(reach(the(state(in(a(given(%me
Idea%#1
Every&change&to&the&state&of&your&applica4on&is&captured&in&an&event&object
"User%Logged%In",#"Payment%Sent",#"User%Landed"
Commands
Anything(that(happens(in(one(of(your(domains(is(triggered(by(a(command(and(generate(one(or(more(events.
Order received --> payment sent --> Items queued --> Confirmation email sent
Query
Generate'read'models'from'events'depending'how'data'needs'to'be'actually'used'(by'users'and'other'applica9on'internals)
Idea%#4
One$way$to$store$data/events$but$poten2ally$infinite$ways$to$read$them
A"prac'cal"example
Tech%ops,%Business%control,%Monitoring,%Accoun4ng%they%are%all%interested%in%reading%data%from%different%views.
You$start$with$this
{ "_id": ObjectId("123"), "username": "Flash", "city": …, "phone": …, "email": …,}
Soon$you$might$end$up$with
{ "_id": ObjectId("123"), "username": "Flash", "city": …, "phone": …, "email": …, "created_at": …, "updated_at": …, "ever_tried_to_purchase_something": …, "canceled_at": …, "acquisition_channel": …, "terminated_at": …, "latest_purchase_date": …, …}
A"bomb"wai)ng"to"detonate
It's%impossible%to%keep%adding%state%changes%to%your%documents%and%then%expect%to%be%able%to%extract%them%with%a%single%query.
EventStore• Engineered)for)event)sourcing
• Supports)projec4ons
• By)the)father)of)CQRS
• Great)performances
The$badWin$Only.$Run$on$*nix$with$Mono,$s4ll$too$unstable.
LevelWHEN)
An#eventstore#built#with#Node.js#and#LevelDB8#Faster#than#light8#Completely#custom,#no#tool#to#handle#aggregates
The$write$architecture
Service | \ | \ [event payload] | \ | Service --- Queue System <------------> API -> MongoDB / | / [event payload] | / |Service |
MongoDB'RS
A"mongod"replica*set"with"two"logical"dbs."1."Eventstore"DB"where"we"would"store"2."Repor=ng"DB"where"we"would"store"aggregates"and"final"reports
Anatomy(of(an(event
{ '_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 'type': 'an-event-type', 'data': { 'meta' : { … }, 'payload' : { … } }}
Storing(events
Anatomy(of(an(event
'meta' : { 'creation_date': ISODate("2014-21-11T00:00:01Z"), 'saved_date': ISODate("2014-21-11T00:00:02Z"), 'source': 'some-bounded-context', 'correlation_id': 'a-correlation-id'},'payload' : { 'user_id': '1234', 'animal': 'unicorn', 'colour': 'pink', 'purchase_date': ISODate("2014-21-11T00:00:00Z"), 'price': '20\fantaueros'}
Storing(events
Don't&trust&the&network:&Idempotence{ '_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', …}
The$_id$field$is$actually$defined$client0side$and$ensures$idempotence$if$an$event$is$received$two$8mes.
Indexes• Events(collec,ons(are(HUGE((~100M*N(documents)
• Use(indexes(wisely(as(they(are(necessary(yet(expensive(
• With(suggested(events(structure:(type + data.meta.created_at
BenchmarkingHow$many$events/second$can$you$store?
Our$machines$were$able$to$store$roughly$150$events/sec.$This$number$can$be$greatly$increased$with$dedicated$IOPS,$more$aggressive$inser@ng$policies,$etc..
#1#The#registra-on#event{ '_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 'type': 'user-registered', 'data': { 'meta' : { 'save_date': ISODate("2014-21-11T00:00:09Z"), 'created_at': ISODate("2014-21-11T00:00:01Z"), 'source': 'core-domain', 'correlation_id': 'user-123456' }, 'payload' : { 'user_id': 123, 'username': 'flash', 'email': 'a-dummy-email@gmail.com', 'country': 'IT' } }}
#2#The#purchase#event{ '_id' : '3318c11e-fe60-4c80-a2b2-7add681492d9', 'type': 'user-purchased', 'data': { 'meta' : { 'save_date': ISODate("2014-21-11T00:10:09Z"), 'created_at': ISODate("2014-21-11T00:10:01Z"), 'source': 'payment-gateway', 'correlation_id': 'user-123456' }, 'payload' : { 'user_id': 123, 'email': 'a-dummy-email@gmail.com', 'amount': 20, 'value': EUR, 'payment': 'credit_card', 'item': 'fluffy cat' } }}
Sequen&al)projector)(1)
Divides'the'stream'of'events'into'batches,'filters'events'by'type'and'pass'those'of'interest'to'the'mapper'
[]->[x]->[]->[x]->[]->[]->[]->[] |--------------| |------------| | | | | ---> Projector
Sequen&al)projector)(2)• It's&a&good&idea&to&select&fixed&sizes&batches&to&avoid&memory&problems&when&you&load&your&Cursor&in&memory
• Could&be&a&long=running&process&selec>ng&events&as&they&arrive&in&real%me
Event&Mapper&(1)
Translates)event)fields)to)the)Read)Model)domain.
Takes&an&event&as&input,&applies&a&bunch&of&logic&and&will&return&a&list&of&Read&Model&fields.
Event&Mapper&(2)
Input&Event:&
user-registered
Output:
$output = [ 'user_id' => 123, // simply copied 'user_name' => 'flash', // simply copied 'email' => 'a-dummy-email@gmail.com', // simply copied 'registered_at' => "2014-21-11T00:00:01Z" // From the data.meta.created_at event field ];
Event&Mapper&(3)
Input&Event:&
user-purchased
Output:
$output = [ 'user_id' => 123, // simply copied 'email' => 'a-dummy-email@gmail.com', // simply copied 'purchased_at': "2014-21-11T00:10:01Z" // From the data.meta.created_at event field ];
The$Projec*on$,$a.er$event#1$(2)
db.users_conversion_rate_projection.findOne()
{ 'user_id': 123, 'user_name': 'flash', 'email': 'a-dummy-email@gmail.com', 'registered_at': "2014-21-11T00:00:01Z"}
The$Projec*on$,$a.er$event#2$(3){ 'user_id': 123, 'user_name': 'flash', 'email': 'a-dummy-email@gmail.com', 'registered_at': "2014-21-11", 'purchased_at': "2014-21-11" // Added this field and rewrote others}
The$Projec*on$,$collec*on{ 'user_id': 123, 'user_name': 'flash', 'email': 'a-dummy-email@gmail.com', 'registered_at': "2014-21-11", 'purchased_at': "2014-21-11" // Added this field and rewrote others}{ 'user_id': 456, 'user_name': 'batman', 'email': 'a-dummy-email@gmail.com', 'registered_at': "2014-21-11", 'purchased_at': "2014-21-11" // Added this field and rewrote others}{ 'user_id': 789, 'user_name': 'superman', 'email': 'a-dummy-email@gmail.com', 'registered_at': "2014-21-12", 'purchased_at': "2014-21-12" // Added this field and rewrote others}
The$Projec*on$,$A$few$thoughts$(4)
Note%that%we%didn't%copy%from%events%to%projec6on%all%the%available%fields.%Just%relevant%ones.
From%these%two%events%we%could%have%generated%infinite%read%models%such%as:
• List&all&purchased&products&and&related&amounts&for&the&company&buyers&
• Map&all&sales&and&revenues&for&our&accoun8ng&dept
• List&transac8ons&for&the&financial&department
The$aggrega(on$(1)$.$Total$registered$usersvar registered = db.users_conversion_rate_projection.aggregate([ { $match: { "registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") } } }, { $group: { _id: { }, count: { $sum:1 } } }]);
The$aggrega(on$(2)$.$Users$with$a$purchasevar purchased = db.users_conversion_rate_projection.aggregate([ { $match: { "registered_at": { $gte: ISODate("2015-11-21"), $lte: ISODate("2015-11-22") }, "purchased_at": { $exists: true } } }, { $group: { _id: { }, count: { $sum:1 } } }]);
The$aggrega(on$(3)$.$Automate$all$the$things• You%can%easily%create%the%aggrega2on%framework%statement%by%composi2on%abstrac2ng%the%concept%of%Column.
• This%way%you%can%dynamically%aggregate%your%projec2ons%on%(for%example)%an%API%requests.
• If%your%Projector%is%a%long%running%process,%your%projec2ons%will%be%updated%to%the%second%and%you%automagically%get%real%me%data.
Mongo%won't%help%you.Actually(it(will(make(your(life(incredibly(difficult(with(hidden(bugs(and(leaking(
documenta8on.
top related