Start Small Bigger Biggest The Event Driven Organisation Based on the Lambda Architecture
Start small bigger biggest
Jun 29, 2015
The Event Driven Company
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Start Small Bigger BiggestThe Event Driven Organisation
Based on the Lambda Architecture
Who Is That?
BigData.be
Belgian BigData Group
Founded in juli 2011
+200 members
http://www.bigdata.be
http://www.meetup.com/bigdatabe
@bigdatabe
Agenda
TransCo
The Event Driven OrganisationEventsViewsIntegration
TransCo
Meet TransCoInternational Transport
Track & TraceFreelance DriversCustom SoftwareEnterprise Service Bus
TransCo
DepartmentsITFinanceLegalLogisticsSalesCommunicationsMarketing...
TransCo
A customer requests a quote
A server goes down
A website visitor clicks on a link
A financial transaction is booked
A delivery truck pings its GPS coördinates
TransCo
All these have a similar thing:
They Generate Events
Events
ITFinanceLegalLogisticsSalesCommunications...
Events
Events
Events used to manipulate their master data
Events
Today, events ARE their master data
TransCo
Isn't this a big data presentation?
Hang on, We'll get to that
Events
An event isTime basedRecord linkedField orientedIncrementalImmutable
Events
A new customer is addedADD [ts=1,id=123,field="name",value="Daan Gerits"]
ADD [ts=1,id=123,field="address",value="MyStreet 40"]
A customer is updatedADD [ts=13,id=123,field="address",value="AnotherStreet 1"]
Eventually a customer is removedADD [ts=43,id=123,field="address",value=]
Events
Notice I only created new records
No UpdatesNo Deletes
CRUD becomes CR
Events
Oh no, I shouldn't have done that!DELETE [ts=43,id=123,field="address"]
TransCo
Discover market trends
Application and server statistics
Direct Marketing
Financial validation
Tracking Dashboard
Views
View = Fx All Data
Views
View =
Different View TypesReal-timeRelationalReportingApplicationAPI...
Views
Views areFull data modelsBased on all dataFocussed
Views
Integration
How do I integrate with my existing infrastructure?
Views can also be stored in- RDBMS- Indexing Engine- Files- DataWarehouse- ...
But these views are read-only!If you want read-write use the API
Integration
Big Data
So what has all this to do with Big Data?
Only appending eventsneed to store a lot of data
=> HDFS, HBase, ...
need batch processing=> MapReduce, Pig, Hive, ...
need real-time processing=> Storm, S4, ...
Big Data
Allows trendingdriver statistics for his whole career
Allows state regenerationthe state of all master data on februari 12, 2005
Is human-error-proofremove the last event on this record
Scales very well
Event Driven Organisation
By storing events
By generating focussed views
Event Driven Organisation
Backup SlidesGenerating Views
Views
Realtime Views
Views
Latency
Views
Generating takes time=> View is always x hours old
View Ready
Start Generating + + + ++
Views
How to solve this?Store events in 2 locations
Master Data SetBuffer DB
Buffer DB has - data of last x hours- data, not events
Views
Speed View
API
Batch View
Views
Batch ViewsUse MapReduce
Speed ViewsUse Storm or something similar
Related Documents
