Page 1
Architecting theInternet of Things@blimpyacht
Page 7
Event SourcingTHE BEST, UBIQUITOUS PATTERN
YOU’VE NEVER HEARD OF
Page 9
User: ABalance: 427
User: BBalance: 550
Page 10
User: ABalance: 427
User: BBalance: 550
Page 11
User: ABalance: 427
User: BBalance: 550
+5-5
Page 12
Event Sourcing
State
Page 13
+4-2+5+7-4+9-3-6+2+1+1-5 t(12) = 9
Event Sourcing
Page 14
+4-2+5+7-4+9-3-6+2+1+1-5
t(5) = 10
Event Sourcing
Page 15
+4-2+5+7-4+9-3-6+2+1+1-5
t(9) = 12
Event Sourcing
Page 16
+4-2+5+7-4+9-3-6+2+1+1-5
Event Sourcing
a = f ’(x)
Page 17
+4-2+5+7-4+9-3-6+2+1+1-5
Event Sourcing
+5-4+5
Page 19
Res Gestae Divi Augusti
Page 27
+4-2+5+7-4+9-3-6+2+1+1-5
Event Sourcing
Page 28
+4-2+5+7-4+9-3-6+2+1+1-5
t(4) = 14
Event Sourcing
t(8) = 10
t(12) = 9
Page 29
+4-2+5+7-4+9-3-6+2+1+1-5
t(4) = 14
Event Sourcing
t(8) = 10
t(12) = 9
t’(4) = -2
t’(8) = -17
t’(12) = -22
Page 31
{ _id: {
ts: 1213084687000, taxi: "abboip" }, loc: {
geometry: [ -122.39488, 37.75134
] }, fare: 0, ts: ISODate("2008-06-10T07:58:07Z")}
Page 33
{ _id: ObjectId("56a4215a46778b69e08ff8ff"), taxi: "abboip", start: ISODate("2008-05-17T14:51:10Z"), route: { type: "LineString", coordinates: [ [ -122.39724, 37.74977 ], [ -122.40619, 37.74896 ], [ -122.41335, 37.74831 ], [ -122.414, 37.75157 ], [ -122.41438, 37.75552 ]
] }, end: ISODate("2008-05-17T14:55:58Z")}
Page 35
Data Modeling Domain EventsDomain Objects
Page 38
Accounts Payable
Accounts Receivable
Page 39
Accounts Payable
Accounts Receivable
Journal
Page 40
Accounts Payable
Accounts Receivable
Journal
Ledger
Page 41
Command QueryResponsibility
Segregation
Page 43
CRUD vs. CQRS
CRUD
Page 44
CRUD vs. CQRS
WRITE
READ
Page 47
WRITE
READ
READ
CRUD vs. CQRS
Page 48
WRITE
READ
Variant Read Models: VIEWS
READREADREADREAD
Page 49
WRITE
READ
Primary
READ
CRUD vs. CQRS
Page 50
CRUD vs. CQRS
WRITE
READ
READ
Primary
2ndary
2ndary
Page 51
Where it Gets Tricky
Page 52
CRUD vs. CQRS
WRITE
READ
READ
Primary
2ndary
2ndary
Page 53
CRUD vs. CQRS
WRITE
READ
READ
Primary
2ndary
2ndary
Page 54
Consistent Replay+4-2+5+7-4+9-3-6+2+1+1-5
t(5) = 10 Exchange Rate Server
10 / 0.88 = 8.82€
Page 55
Consistent Replay+4-2+5+7-4+9-3-6+2+1+1-5
t(5) = 10 Exchange Rate Server
10 / 0.87 = 8.70€
Page 56
+4-2+5+7-4
+9-3-6+2+1+1-5
0.88 Exchange Rate Server
Consistent Replay
Page 57
Where it Gets Trickier Still
Page 63
External Updates+4-2+5+7-4+9-3-6+2+1+1-5
Notification
Subscriber
Page 66
External Updates+4-2+5+7-4+9-3-6+2+1+1-5
Notification
Subscriber
Page 78
TROUGH OF DISILLUSIONMENT
Page 79
Interactive ShellEasy (-er)Caching
Page 80
HDFS
Distributed Data
Page 81
HDFSYARN
Hive
Pig
Domain Specific Languages
MapReduce
Page 82
Spark Stand Alone
YARN
Mesos
HDFS
Distributed Resources
Page 83
YARN
SparkMesos
HDFS
Spark Stand Alone
Hadoop
Distributed Processing
Page 84
YARN
SparkMesos
Hive
Pig
HDFS
Hadoop
Spark Stand Alone
Domain Specific Languages
Page 85
YARN
SparkMesos
Hive
Pig
SparkSQL
Spark Shell
SparkStreaming
HDFS
Spark Stand Alone
Hadoop
Page 86
YARN
SparkMesos
Hive
Pig
SparkSQL
Spark Shell
SparkStreaming
HDFS
Spark Stand Alone
Hadoop
Page 87
YARN
SparkMesos
Hive
Pig
SparkSQL
Spark Shell
SparkStreaming
HDFS
Spark Stand Alone
Hadoop
Page 88
YARN
SparkMesos
Hive
Pig
SparkSQL
Spark Shell
SparkStreaming
Spark Stand Alone
Hadoop
Page 89
Stand AloneYARN
SparkMesos
Hive
Pig
SparkSQL
SparkShell
SparkStreaming
MapReduce
Page 90
Stand AloneYARN
SparkMesos
SparkSQL
SparkShell
SparkStreaming
Page 91
Stand AloneYARN
SparkMesos
SparkSQL
SparkShell
SparkStreaming
Page 92
executor
Worker Node
executor
Worker Node Master
Java Driver
Hadoop Connector
Driver Application
Page 93
Parallelization
Parellelize = x
Page 94
Transformations
Parellelize = x t(x) = x’ t(x’) = x’’
Page 95
Transformationsfilter( func )union( func )intersection( set )distinct( n )map( function )
Page 96
Action
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’
Page 97
Actionscollect()count()first()take( n )reduce( function )
Page 98
Lineage
f(x’’) = yParellelize = x t(x) = x’ t(x’) = x’’
Page 99
Transform Transform ActionParallelize
Lineage
Page 100
Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize
Lineage
Page 101
Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize
Lineage
Page 102
Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize Transform Transform ActionParallelize
Lineagehttp://www.blimpyacht.com/2016/02/03/a-visual-guide-to-the-spark-hadoop-ecosystem/
Page 103
https://github.com/mongodb/mongo-hadoop
Page 104
Spark ConfigurationConfiguration conf = new Configuration();conf.set(
"mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat”);conf.set(
"mongo.input.uri", "mongodb://localhost:27017/db.collection”);
Page 105
Spark ContextJavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf,
MongoInputFormat.class,Object.class,BSONObject.class
);
Page 106
Spark Submit
/usr/local/spark-1.5.1/bin/spark-submit \ --class com.mongodb.spark.examples.DataframeExample \ --master local Examples-1.0-SNAPSHOT.jar
Page 107
Stand AloneYAR
N
SparkMesos
SparkSQL
SparkShell
SparkStreaming
Page 108
JavaRDD<Message> messages = documents.map (
new Function<Tuple2<Object, BSONObject>, Message>() {
public Message call(Tuple2<Object, BSONObject> tuple) { BSONObject header = (BSONObject)tuple._2.get("headers");
Message m = new Message(); m.setTo( (String) header.get("To") ); m.setX_From( (String) header.get("From") ); m.setMessage_ID( (String) header.get( "Message-ID" ) ); m.setBody( (String) tuple._2.get( "body" ) );
return m; } });
Page 111
THE FUTUREAND
BEYOND THE INFINITE
Page 113
Aggregation Filters$match | $project | $group
Page 114
Data Locality mongos
Page 115
THANKS!@blimpyacht