Intelligent Distributed Systems Optimizations

MANCHESTER LONDON NEW YORK

Martin Zapletal @zapletal_martin#ReactiveSummit

Intelligent Distributed SystemOptimizations

@cakesolutions

https://twitter.com/zapletal_martin

val in: Array[Int]

var i = 0var result = 0while (i <= in.size) { result += in(i) i += 1}

Motivation

val in: Array[Int]

in.foldLeft(0)(_ + _)

Distributed System Optimizations

● Performance, cost, reliability, uptime● Logical, physical, intelligence, .... Optimized

single machine

Optimized single

machine

Optimized single

machine

Optimized distributed

system

Logical Optimizations

● Databases● Spark● Akka Streams● SystemML● TensorFlow● Free Monad

[6]

val users = spark.read.json("...").as[User]val posts = spark.read.json("...").as[Post]

users .where(not(not($"active"))) .join(posts, posts("author") === users("id")) .select($"userName", initcap($"title")) .filter($"userName" !== "Mike") .filter(_ => 1 == 1) .groupBy($"userName") .agg(count("*")) .select($"userName", $"count(1)")

Author, text, title

Active, id, age, name, userName

NOT(NOT(Active))

userName, initcap(title)

Username != Mike

userName, count

posts

users

filter

join

project

filter

aggregate

project

Graph Optimizations

Graph Optimizations

posts users

Author, text, title Active, id, age, name, userName

filterActive && Username != Mike

join

project userName

aggregate

project userName, count

posts

usersAuthor, text, title


filter NOT(NOT(Active))

join

project userName, initcap(title)

filter Username != Mike

aggregate


projectprojectAuthor, title Id, userName

Id == author

Id == author

Graph Optimizations

posts users

Author, text, title Active, id, age, name, userName

filterActive && Username != Mike

join

userName

aggregate


posts

usersAuthor, text, title


filter NOT(NOT(Active))

join

project userName, initcap(title)

filter Username != Mike

aggregate


projectprojectAuthor, title Id, userName

Id == author

Id == author

project

Physical Optimizations

posts users

filter

5 billion records, 500 GB

join

projectproject

100 million records 20 GB

t1.value=200 1 million records, 100 MB

Id == author

[3, 8]

Whole Stage Code Generationprivate void agg_doAggregateWithKeys() throws java.io.IOException { agg_hashMap = agg_plan.createHashMap();

while (inputadapter_input.hasNext() && !stopEarly()) { InternalRow inputadapter_row = (InternalRow) inputadapter_input.next(); boolean inputadapter_isNull = inputadapter_row.isNullAt(0); UTF8String inputadapter_value = inputadapter_isNull ? null : (inputadapter_row.getUTF8String(0)); long inputadapter_value1 = inputadapter_row.getLong(1);

UnsafeRow agg_unsafeRowAggBuffer = null; UnsafeRow agg_fastAggBuffer = null;

if (agg_fastAggBuffer == null) { agg_holder.reset();

agg_rowWriter.zeroOutNullBytes();

if (inputadapter_isNull) { agg_rowWriter.setNullAt(0); } else { agg_rowWriter.write(0, inputadapter_value); }

[0, 1, 2, 4, 5, 7]

Manual Optimizations● Coordination avoidance● Single writer principle● Load balancing● Speculative Execution● Colocation● Decoupling● Caching● Publish Subscribe● Batching● Rate Limiting● Low Level Optimizations (memory, object allocation, GC, CPU, IO, parallelization, …)● ...

Machine learning

● Stock market predictions● Product recommendations● Facial recognition● Object recognition● Speech understanding● Self-driving cars● Distributed system optimizations● ...

Observability

● Observability○ Logs○ Metrics○ Traces

● Analytics● Understanding● Actions● Reactive

[9]

Metrics and Monitoring

private[this] val latency = Histogram .build() .name(s"requests_latency_seconds") .help("Request latency in seconds.") .register(registry)

val timer = latency.startTimer()

…

timer.observeDuration()

[9]

Metrics and Monitoring

● Time series database● Aggregate queries

Actor Processing Time (max)

200

150

100

50

0

600

400

200

0

800

600

400

200

0

Running Actors

Processed Messages (1 Minute Rate)

17:30 17:45 18:00 18:15

17:30 17:45 18:00 18:15

17:30 17:45 18:00 18:15

6

4

2

0

Actor mailbox Time (max)

17:30 17:45 18:00 18:15

[10, 11]

Logs and Context

● Automated context

log.logInfo( s"""{ | "traceId"="$traceId", | "userId"="$userId", | "scope"="$scope", | "service"="$service", | "userData"=${userData.asset} | "message"="Don't log like this! Automate context!" |""".stripMargin)

Logs and Context{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "uid": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}

Logs and Context{

"datetim

e": "2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.920Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

2.dispatch

er.Disp

atchRequestH

andler",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce2",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"eve

ntPublished": {

"someAttri

bute1": "2",

"someOtherAttri

bute1": "None",

"dateTimePublish

ed": "2017-09-20T15:01:28.917Z",

"eventTyp

e": "updateAsse

t",

"company":

"cake

Solutions",

"assetDescr

iption": "

some asse

t descr

iption",

"updateId": "0",

"userId

": "54947df8-0e9e-4471-a2f9-9af509fb5889",

"userD

etails": "n

o way to notice

the use

rId here actu

ally is i

ncorre

ct",

"versi

on": "v1

",

"data": {

"updateAsse

tData":

"http://w

ww.longurlm

aker.c

om/go?id=7s80lengthened87EzU

RL11aDecentURLsd

136ShoterLinkdeepSnipURLcfa

r%2Breac

hingt108cNutsh

ellURLgbsp

read%2Bout13EasyURLendurin

gtall0xURl.ie

18418stretch

edbastretch

ingxm62d1faraway0

URl.ie

ny0exp

andedNe1longish419Smallr3

9far%2Bofflo

fty1EzU

RL151Shim1aURLPieaShredURLURL.co

.ukNanoRefendurin

gxlof

tynfar%

2BreachingtallM

ooURL57i09stretch

edk014longish

090rangyn9c5

h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo

ut50301URL2aspread%2Bout01blnkf5

1gangling2101k9far%

2Boff0a0910d0Shortli

nksfaraway7

8Redirxdrawn%2Bout3Doio

p"

}

}

}

"datetime": "

2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.

{

"timesta

mp": "2017-09-20T15:01:28.931Z",

"lo

gLevel":

"warn",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.asset.E

ventParse

r.reco

verFromMiss

ingData",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"m

issingDataFailure": {

"Miss

ingData": {

"failure": {

"property

": "referre

r",

"optio

nal": "tru

e"

}

}

}

}

"datetime": "

2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.

{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",

"traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),

AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),

AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,

10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/asse

ts/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-

8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,275420

8,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(Ass

etIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http:/

/example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",

"traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),


AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIden

tifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutio

ns.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,Ams

Id(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.14

0350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/

file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),De

scriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"

}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",

"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" }

} }}

{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"

}

{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",

"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" }

} }}



}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067 "datetime":

"2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",


}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.e6162-3b6f-4ae2-a171-2470b63dff00"}

{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser", "traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}

{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",

"scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "5760302508966145110",

"service": "microservice1",

"userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889",

"serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"

}{ "datetime": "2017-09-20T15:01:28.920Z",

"level": "info",

"scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",

"traceId": "5760302508966145110",




"eventPublished": {

"someAttribute1": "2",

"someOtherAttribute1": "None",

"dateTimePublished": "2017-09-20T15:01:28.917Z",

"eventType": "updateAsset",

"company": "cakeSolutions",

"assetDescription": "some asset description",

"updateId": "0",

"userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",

"userDetails": "no way to notice the userId here actually is incorrect",

"version": "v1",

"data": {

"updateAssetData":

"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac

hingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ie

ny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof

tynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo

ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doio

p" }

}} "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",





}{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",




"serviceUrl": "http://cake.

{ "tim

estamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn",

"scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",

"traceId": "5760302508966145110",




"missingDataFailure": {

"MissingData": {

"failure": {

"property": "referrer",

"optional": "tru

e"

}

}

}} "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",





}{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",




"serviceUrl": "http://cake.

{ "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"ser

vice

": "m

icro

serv

ice1

",

"use

rId":

"067

e616

2-3b

6f-4

ae2-

a171

-247

0b63

dff0

0",

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.92

0Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e2.d

ispa

tche

r.Dis

patc

hReq

uest

Han

dler

",

"tra

ceId

": "5

7603

0250

8966

1451

10",

"s

ervi

ce":

"mic

rose

rvic

e2",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"ass

etId

": "5

4947

df8-

0e9e

-447

1-a2

f9-9

af50

9fb5

889"

,

"eve

ntP

ublis

hed"

: {

"

som

eAttr

ibut

e1":

"2",

"so

meO

ther

Attr

ibut

e1":

"Non

e",

"da

teTi

meP

ublis

hed"

: "20

17-0

9-20

T15:

01:2

8.91

7Z",

"ev

entT

ype"

: "up

date

Ass

et",

"co

mpa

ny":

"cak

eSol

utio

ns",

"as

setD

escr

iptio

n": "

som

e as

set d

escr

iptio

n",

"up

date

Id":

"0",

"us

erId

": "5

4947

df8-

0e9e

-447

1-a2

f9-9

af50

9fb5

889"

,

"

user

Det

ails

": "n

o w

ay to

not

ice

the

user

Id h

ere

actu

ally

is in

corr

ect",

"ve

rsio

n": "

v1",

"da

ta":

{

"up

date

Ass

etD

ata"

: "h

ttp://

ww

w.lo

ngur

lmak

er.c

om/g

o?id

=7s8

0len

gthe

ned8

7EzU

RL1

1aD

ecen

tUR

Lsd1

36S

hote

rLin

kdee

pSni

pUR

Lcfa

r%2B

reac

hing

t108

cNut

shel

lUR

Lgbs

prea

d%2B

out1

3Eas

yUR

Lend

urin

gtal

l0xU

Rl.i

e184

18st

retc

hedb

astre

tchi

ngxm

62d1

fara

way

0UR

l.ie

ny0e

xpan

dedN

e1lo

ngis

h419

Sm

allr3

9far

%2B

offlo

fty1E

zUR

L151

Shi

m1a

UR

LPie

aShr

edU

RLU

RL.

co.u

kNan

oRef

endu

ringx

lof

tynf

ar%

2Bre

achi

ngta

llMoo

UR

L57i

09st

retc

hedk

014l

ongi

sh09

0ran

gyn9

c5h5

YA

TUC

rem

oteb

1Doi

op00

dgre

at2a

01sp

un%

2Bo

ut50

301U

RL2

aspr

ead%

2Bou

t01b

lnkf

51ga

nglin

g210

1k9f

ar%

2Bof

f0a0

910d

0Sho

rtlin

ksfa

raw

ay78

Red

irxdr

awn%

2Bou

t3D

oio

p"

}

}} {

"dat

aTim

e": "

2017

-09-

20T1

5:01

:28.

925Z

",

"lev

el":

"info

",

"sco

pe":

"net

.cak

esol

utio

ns.m

icro

serv

ice1

.ass

et.E

vent

Pro

cess

or",

"t

race

Id":

"576

0302

5089

6614

5110

",

"ser

vice

": "m

icro

serv

ice2

",

"use

rId":

"067

e616

2-3b

6f-4

ae2-

a171

-247

0b63

dff0

0",

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"a

sset

Det

ails

": "(

List

(Ass

etId

entif

ier(

Pro

vide

r,123

4156

7433

.112

2-23

44,1

1220

303,

0128

479)

, A

sset

Iden

tifie

r(P

rovi

der,1

9460

718.

1403

5097

4037

952,

2762

537,

1946

0718

), A

sset

Iden

tifie

r(P

rovi

der,0

9320

418u

7.10

9238

4,p9

1234

,018

2374

))),(

List

(Ass

etId

entif

ier(

Pro

vide

r,102

9389

0.10

000,

12-8

3,10

7237

9777

99))

,Ass

etU

rl(ht

tp://

cake

.qa.

solu

tions

.us-

east

-1.n

et/v

4/as

sets

/des

crib

e/ad

8ecd

76-7

36e-

4816

-8c5

9-48

4ba1

d229

a4/

desc

riptio

r,Am

sId(

ad8e

cd76

-736

e-48

16-8

c59-

484b

a1d2

29a4

),0,0

,0))

,(Lis

t(Ass

etId

entif

ier(

Pro

vide

r,194

8756

.140

3509

740

3842

4,27

5420

8,19

4875

6)),D

escr

ipto

rFile

(For

mat

,http

://ex

ampl

e.or

g/so

me/

Des

crip

tor/f

ile.m

p4,0

,0,0

)),(L

ist(A

sset

Iden

tifie

r(P

rovi

der,1

9487

20.1

4035

0974

0383

7,27

5426

4,19

4872

0)),D

escr

ipto

rFile

(For

mat

,http

://ex

ampl

e.or

g/so

me/

othe

r/Des

crip

tor/f

ile/

vide

o.m

p4,0

,0,0

))"

} { "d

ateT

ime"

: "20

17-0

9-20

T15:

01:2

8.92

5Z",

"l

ogLe

vel":

"inf

o",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.a

sset

.Eve

ntP

arse

r",

"t

race

Id":

"904

4230

9659

1682

8392

",

"ser

vice

": "m

icro

serv

ice1

",

"use

rId":

"067

e616

2-3b

6f-4

ae2-

a171

-247

0b63

dff0

0",

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"a

sset

Det

ails

": "(

List

(Ass

etId

entif

ier(

Pro

vide

r,194

6071

8.14

0350

9740

3795

2,27

6253

7,19

4607

18),

Ass

etId

entif

ier(

Pro

vide

r,194

6071

8.14

0350

9740

3795

2,27

6253

7,19

4607

18),

Ass

etId

entif

ier(

Pro

vide

r,194

6071

8.14

0350

9740

3795

2,27

6253

7,19

4607

18))

,nul

l),(L

ist(A

sset

Iden

tifie

r(P

rovi

der,1

9487

588.

1403

5097

4038

400,

2754

237,

1948

7588

)),A

sset

Url(

http

://ca

ke.q

a.so

lutio

ns.u

s-ea

st-1

.net

/v4/

asse

ts/d

escr

ibe/

ad8e

cd76

-736

e-4

816-

8c59

-484

ba1d

229a

4/de

scrip

tior,A

msI

d(ad

8ecd

76-7

36e-

4816

-8c5

9-48

4ba1

d229

a4),0

,0,0

)),(L

ist(A

sset

Iden

tifie

r(P

rovi

der,1

9487

56.1

4035

0974

0384

24,2

7542

08,1

9487

56))

,Des

crip

torF

ile(F

orm

at,h

ttp://

exam

ple.

org/

som

e/D

escr

ipto

r/file

.mp4

,0,0

,0)

),(Li

st(A

sset

Iden

tifie

r(P

rovi

der,1

9487

20.1

4035

0974

0383

7,27

5426

4,19

4872

0)),D

escr

ipto

rFile

(For

mat

,http

://ex

ampl

e.or

g/so

me/

othe

r/Des

crip

tor/f

ile/v

ideo

.mp4

,0,0

,0))

"} {

"tim

esta

mp"

: "20

17-0

9-20

T15:

01:2

8.93

1Z",

"l

ogLe

vel":

"war

n",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.a

sset

.Eve

ntP

arse

r.rec

over

From

Mis

sing

Dat

a",

"t

race

Id":

"576

0302

5089

6614

5110

",

"ser

vice

": "m

icro

serv

ice1

",

"use

rId":

"067

e616

2-3b

6f-4

ae2-

a171

-247

0b63

dff0

0",

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"m

issi

ngD

ataF

ailu

re":

{

"

Mis

sing

Dat

a": {

"f

ailu

re":

{

"p

rope

rty":

"ref

erre

r",

"opt

iona

l": "t

rue"

}

}

}}

{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "5760302508966145110",





}{

"datetime": "2017-09-20T15:01:28.920Z",

"level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",

"traceId": "5760302508966145110",




"eventPublished": { "someAttribute1": "2",






"updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",


"version": "v1", "data": { "updateAssetData":






p" } }

}{

"dataTime": "2017-09-20T15:01:28.925Z",

"level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",

"traceId": "5760302508966145110",




"assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479),


AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,10

7237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a

4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.1403509740

38424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(P

rovider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/

video.mp4,0,0,0))"}

{ "dateTime": "2017-09-20T15:01:28.925Z",

"logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",

"traceId": "9044230965916828392",




"assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),


AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.14

0350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4

816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provide

r,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)

),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/so

me/other/Descriptor/file/video.mp4,0,0,0))"

}{

"timestamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",

"traceId": "5760302508966145110",





"MissingData": { "failure": {


"optional": "true"

} } }}

{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"

}

{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}

{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067 "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.e6162-3b6f-4ae2-a171-2470b63dff00"

}

{ "datetime": "2017-09-20T15:01:28.834Z", "level": "info", "scope": "net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "serviceUrl": "http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z", "eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description", "updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "userDetails": "no way to notice the userId here actually is incorrect", "version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xURl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39far%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2BreachingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlinksfaraway78Redirxdrawn%2Bout3Doiop" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor", "traceId": "5760302508966145110", "service": "microservice2", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,12341567433.1122-2344,11220303,0128479), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser", "traceId": "9044230965916828392", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "assetDetails": "(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718), AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIdentifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z", "logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData", "traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00", "assetId": "54947df8-0e9e-4471-a2f9-9af509fb5889", "missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}


"traceId": "5760302508966145110", "service": "microservice1", "userId": "067e6162-3b6f-4ae2-a171-2470b63dff00",



}{ "datetime": "2017-09-20T15:01:28.920Z", "level": "info", "scope": "net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",



"eventPublished": { "someAttribute1": "2", "someOtherAttribute1": "None", "dateTimePublished": "2017-09-20T15:01:28.917Z",

"eventType": "updateAsset", "company": "cakeSolutions", "assetDescription": "some asset description",

"updateId": "0", "userId": "54947df8-0e9e-4471-a2f9-9af509fb5889",


"version": "v1", "data": { "updateAssetData": "http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac





p" } }}{ "dataTime": "2017-09-20T15:01:28.925Z", "level": "info", "scope": "net.cakesolutions.microservice1.asset.EventProcessor",










video.mp4,0,0,0))"}{ "dateTime": "2017-09-20T15:01:28.925Z", "logLevel": "info", "scope": "net.cakesolutions.microservice1.asset.EventParser",










me/other/Descriptor/file/video.mp4,0,0,0))"}{ "timestamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn", "scope": "net.cakesolutions.microservice1.asset.EventParser.recoverFromMissingData",



"missingDataFailure": { "MissingData": { "failure": { "property": "referrer", "optional": "true" } } }}

{

"datetime": "

2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.920Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

2.dispatch

er.Disp

atchRequestH

andler",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce2",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"eve

ntPublished": {

"someAttri

bute1": "2",

"someOtherAttri

bute1": "None",

"dateTimePublish

ed": "2017-09-20T15:01:28.917Z",

"eventTyp

e": "updateAsse

t",

"company":

"cake

Solutions",

"assetDescr

iption": "

some asse

t descr

iption",

"updateId": "0",

"userId

": "54947df8-0e9e-4471-a2f9-9af509fb5889",

"userD

etails": "n

o way to notice

the use

rId here actu

ally is i

ncorre

ct",

"versi

on": "v1

",

"data": {

"updateAsse

tData":

"http://w

ww.longurlm

aker.c

om/go?id=7s80lengthened87EzU

RL11aDecentURLsd

136ShoterLinkdeepSnipURLcfa

r%2Breac

hingt108cNutsh

ellURLgbsp

read%2Bout13EasyURLendurin

gtall0xURl.ie

18418stretch

edbastretch

ingxm62d1faraway0

URl.ie

ny0exp

andedNe1longish419Smallr3

9far%2Bofflo

fty1EzU

RL151Shim1aURLPieaShredURLURL.co

.ukNanoRefendurin

gxlof

tynfar%

2BreachingtallM

ooURL57i09stretch

edk014longish

090rangyn9c5

h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo

ut50301URL2aspread%2Bout01blnkf5

1gangling2101k9far%

2Boff0a0910d0Shortli

nksfaraway7

8Redirxdrawn%2Bout3Doio

p"

}

}

}

"datetime": "

2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.

{

"timesta

mp": "2017-09-20T15:01:28.931Z",

"lo

gLevel":

"warn",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.asset.E

ventParse

r.reco

verFromMiss

ingData",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"m

issingDataFailure": {

"Miss

ingData": {

"failure": {

"property

": "referre

r",

"optio

nal": "tru

e"

}

}

}

}

"datetime": "

2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.qa.us-e

ast-1.so

lutions.n

et/micr

oservi

ce2/067e6162-3b6f-4

ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"le

vel":

"info",

"sc

ope": "net.c

akeso

lutions.m

icrose

rvice

1.dispatch

er.Asyn

cRequestD

ispatch

er",

"tra

ceId": "

5760302508966145110",

"se

rvice

": "micr

oservi

ce1",

"use

rId": "

067e6162-3b6f-4ae2-a171-2470b63dff0

0",

"asse

tId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"se

rvice

Url": "h

ttp://c

ake.

{ "

datetime": "

2017-09-20T15:01:28.834Z",

"level": "

info",

"scope": "

net.cakesolutions.microservice1.dispatcher.AsyncRequestDispatcher",

"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"serviceUrl":

"http://cake.qa.us-east-1.solutions.net/m

icroservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"

}{

"datetim

e": "2017-09-20T15:01:28.920Z",

"level": "

info",

"scope": "

net.cakesolutions.microservice2.dispatcher.DispatchRequestHandler",

"traceId": "

5760302508966145110",

"service": "

microservice2",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"eventPublished": {

"s

omeAttribute1": "

2",

"s

omeOtherAttribute1": "

None",

"d

ateTimePublished": "2017-09-20T15:01:28.917Z",

"e

ventType": "updateAsset",

"c

ompany": "cakeSolutions",

"a

ssetDescription": "some asset description",

"u

pdateId": "0",

"u

serId": "54947df8-0e9e-4471-a2f9-9af509fb5889",

"u

serDetails": "no way to notice the userId here actually is incorre

ct",

"v

ersion": "v1",

"d

ata": {

"updateAssetData":

"http://www.longurlm

aker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLinkdeepSnipURLcfar%2Breac


ny0expandedNe1longish419Smallr39far%

2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxlof

tynfar%2BreachingtallM

ooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2a01spun%2Bo

ut50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Shortlin

ksfaraway78Redirxdrawn%2Bout3Doio

p"

}

}}

"datetim

e": "2017-09-20T15:01:28.834Z",

"level": "

info",

"scope": "


"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"serviceUrl":



}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"level": "

info",

"scope": "


"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"importantPresentationMessage": "

It is very simple to miss things in such amount of data. I a

m pretty sure noone will

ever read this message. R

elationships, context and the overall data can be used to im

prove the system as a whole.

Consider for example understanding failures, predicting failures, understanding system and its

behaviour in specific

context

and much more. Also, I h

ope you have a great day!"

{ "

timestamp": "

2017-09-20T15:01:28.931Z",

"logLevel": "

warn",

"scope": "

net.cakesolutions.microservice1.asset.EventParser.re

coverFromMissingData",

"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",


"M

issingData": {

"failure": {

"property": "

referrer",

"optional": "

true"

}

}

}}

"datetim

e": "2017-09-20T15:01:28.834Z",

"level": "

info",

"scope": "


"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"serviceUrl":



}{

"datetim

e": "2017-09-20T15:01:28.834Z",

"level": "

info",

"scope": "


"traceId": "

5760302508966145110",

"service": "

microservice1",

"userId": "

067e6162-3b6f-4ae2-a171-2470b63dff00",

"assetId": "

54947df8-0e9e-4471-a2f9-9af509fb5889",

"serviceUrl":

"http://cake.

{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",





}{

"datetime": "2017-09-20T15:01:28.920Z",

"level": "info",


"traceId": "5760302508966145110",




"eventPublished": {







"updateId": "0",



"version": "v1",

"data": {

"updateAssetData":






p" } }}

{ "dataTime": "2017-09-20T15:01:28.925Z",

"level": "info",

"scope": "net.cakesolutions.microservice1.asset.EventProcessor",

"traceId": "5760302508966145110",











video.mp4,0,0,0))"

}{

"dateTime": "2017-09-20T15:01:28.925Z",

"logLevel": "info",

"scope": "net.cakesolutions.microservice1.asset.EventParser",

"traceId": "9044230965916828392",












}{

"timestamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn",


"traceId": "5760302508966145110",





"MissingData": {

"failure": {


"optional": "true"

} } }}

{ "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl":

"http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "t

imes

tam

p": "

2017

-09-

20T1

5:01

:28.

931Z

",

"l

ogLe

vel":

"war

n",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.a

sset

.Eve

ntPa

rser

.reco

verF

rom

Mis

sing

Dat

a",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"m

issi

ngD

ataF

ailu

re":

{

"M

issi

ngD

ata"

: {

"f

ailu

re":

{

"pro

perty

": "r

efer

rer"

,

"opt

iona

l": "t

rue"

}

}

}} {

"dat

etim

e": "

2017

-09-

20T1

5:01

:28.

834Z

",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl":

"http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67

"dat

etim

e":

"201

7-09

-20T

15:0

1:28

.834

Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl":

"http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.e

6162

-3b6

f-4ae

2-a1

71-2

470b

63df

f00"

}

{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",




"serviceUrl":

"http://cake.qa.us-east-1.solutions.net/microservice2/067e6162-3b6f-4ae2-a171-2470b63dff00"

}{ "datetime": "2017-09-20T15:01:28.920Z",

"level": "info",


"traceId": "5760302508966145110",




"eventPublished": {







"updateId": "0",



"version": "v1",

"data": {

"updateAssetData":

"http://www.longurlmaker.com/go?id=7s80lengthened87EzURL11aDecentURLsd136ShoterLink

deepSnipURLcfar%2Breachingt108cNutshellURLgbspread%2Bout13EasyURLenduringtall0xU

Rl.ie18418stretchedbastretchingxm62d1faraway0URl.ieny0expandedNe1longish419Smallr39fa

r%2Bofflofty1EzURL151Shim1aURLPieaShredURLURL.co.ukNanoRefenduringxloftynfar%2Bre

achingtallMooURL57i09stretchedk014longish090rangyn9c5h5YATUCremoteb1Doiop00dgreat2

a01spun%2Bout50301URL2aspread%2Bout01blnkf51gangling2101k9far%2Boff0a0910d0Short

linksfaraway78Redirxdrawn%2Bout3Doiop"

}

}}{ "dataTime": "2017-09-20T15:01:28.925Z",

"level": "info",


"traceId": "5760302508966145110",






AssetIdentifier(Provider,09320418u7.1092384,p91234,0182374))),(List(AssetIdentifier(Provider,

10293890.10000,12-83,107237977799)),AssetUrl(http://cake.qa.solutions.us-east-1.net/v4/asse

ts/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,AmsId(ad8ecd76-736e-4816-

8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.140350974038424,275420

8,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/file.mp4,0,0,0)),(List(Ass

etIdentifier(Provider,1948720.14035097403837,2754264,1948720)),DescriptorFile(Format,http:/

/example.org/some/other/Descriptor/file/video.mp4,0,0,0))"

}{ "dateTime": "2017-09-20T15:01:28.925Z",

"logLevel": "info",


"traceId": "9044230965916828392",




"assetDetails":

"(List(AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718),


AssetIdentifier(Provider,19460718.140350974037952,2762537,19460718)),null),(List(AssetIden

tifier(Provider,19487588.140350974038400,2754237,19487588)),AssetUrl(http://cake.qa.solutio

ns.us-east-1.net/v4/assets/describe/ad8ecd76-736e-4816-8c59-484ba1d229a4/descriptior,Ams

Id(ad8ecd76-736e-4816-8c59-484ba1d229a4),0,0,0)),(List(AssetIdentifier(Provider,1948756.14

0350974038424,2754208,1948756)),DescriptorFile(Format,http://example.org/some/Descriptor/

file.mp4,0,0,0)),(List(AssetIdentifier(Provider,1948720.14035097403837,2754264,1948720)),De

scriptorFile(Format,http://example.org/some/other/Descriptor/file/video.mp4,0,0,0))"

}{ "timestamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn",


"traceId": "5760302508966145110",





"MissingData": {

"failure": {


"optional": "true"

}

}

}}

{ "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.92

0Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e2.d

ispa

tche

r.Dis

patc

hReq

uest

Han

dler

",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e2",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"e

vent

Publ

ishe

d": {

"so

meA

ttrib

ute1

": "2

",

"so

meO

ther

Attri

bute

1": "

Non

e",

"da

teTi

meP

ublis

hed"

: "20

17-0

9-20

T15:

01:2

8.91

7Z",

"ev

entT

ype"

: "up

date

Asse

t",

"co

mpa

ny":

"cak

eSol

utio

ns",

"as

setD

escr

iptio

n": "

som

e as

set d

escr

iptio

n",

"up

date

Id":

"0",

"us

erId

": "5

4947

df8-

0e9e

-447

1-a2

f9-9

af50

9fb5

889"

,

"us

erD

etai

ls":

"no

way

to n

otic

e th

e us

erId

her

e ac

tual

ly is

inco

rrect

",

"ve

rsio

n": "

v1",

"da

ta":

{

"

upda

teAs

setD

ata"

:

"http

://w

ww

.long

urlm

aker

.com

/go?

id=7

s80l

engt

hene

d87E

zUR

L11a

Dec

entU

RLs

d136

Shot

erLi

nkde

epSn

ipU

RLc

far%

2Bre

ac

hing

t108

cNut

shel

lUR

Lgbs

prea

d%2B

out1

3Eas

yUR

Lend

urin

gtal

l0xU

Rl.i

e184

18st

retc

hedb

astre

tchi

ngxm

62d1

fara

way

0UR

l.ie

ny0e

xpan

dedN

e1lo

ngis

h419

Smal

lr39f

ar%

2Bof

flofty

1EzU

RL1

51Sh

im1a

UR

LPie

aShr

edU

RLU

RL.

co.u

kNan

oRef

endu

ringx

lof

tynf

ar%

2Bre

achi

ngta

llMoo

UR

L57i

09st

retc

hedk

014l

ongi

sh09

0ran

gyn9

c5h5

YATU

Cre

mot

eb1D

oiop

00dg

reat

2a01

spun

%2B

o

ut50

301U

RL2

aspr

ead%

2Bou

t01b

lnkf

51ga

nglin

g210

1k9f

ar%

2Bof

f0a0

910d

0Sho

rtlin

ksfa

raw

ay78

Red

irxdr

awn%

2Bou

t3D

oio

p"

}

}}

"dat

etim

e": "

2017

-09-

20T1

5:01

:28.

834Z

",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.

{ "t

imes

tam

p": "

2017

-09-

20T1

5:01

:28.

931Z

",

"l

ogLe

vel":

"war

n",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.a

sset

.Eve

ntPa

rser

.reco

verF

rom

Mis

sing

Dat

a",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"m

issi

ngD

ataF

ailu

re":

{

"M

issi

ngD

ata"

: {

"f

ailu

re":

{

"pro

perty

": "r

efer

rer"

,

"opt

iona

l": "t

rue"

}

}

}}

"dat

etim

e": "

2017

-09-

20T1

5:01

:28.

834Z

",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.q

a.us

-eas

t-1.s

olut

ions

.net

/mic

rose

rvic

e2/0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

} { "d

atet

ime"

: "20

17-0

9-20

T15:

01:2

8.83

4Z",

"l

evel

": "in

fo",

"s

cope

": "n

et.c

akes

olut

ions

.mic

rose

rvic

e1.d

ispa

tche

r.Asy

ncR

eque

stD

ispa

tche

r",

"t

race

Id":

"576

0302

5089

6614

5110

",

"s

ervi

ce":

"mic

rose

rvic

e1",

"u

serId

": "0

67e6

162-

3b6f

-4ae

2-a1

71-2

470b

63df

f00"

,

"a

sset

Id":

"549

47df

8-0e

9e-4

471-

a2f9

-9af

509f

b588

9",

"s

ervi

ceU

rl": "

http

://ca

ke.

{ "datetime": "2017-09-20T15:01:28.834Z",

"level": "info",


"traceId": "5760302508966145110",





}{ "datetime": "2017-09-20T15:01:28.920Z",

"level": "info",


"traceId": "5760302508966145110",




"eventPublished": {







"updateId": "0",



"version": "v1",

"data": {

"updateAssetData":






p" }

}}{ "dataTime": "2017-09-20T15:01:28.925Z",

"level": "info",


"traceId": "5760302508966145110",











video.mp4,0,0,0))"

}{ "dateTime": "2017-09-20T15:01:28.925Z",

"logLevel": "info",


"traceId": "9044230965916828392",












}{ "tim

estamp": "2017-09-20T15:01:28.931Z",

"logLevel": "warn",


"traceId": "5760302508966145110",





"MissingData": {

"failure": {


"optional": "tru

e"

}

}

}}

Traces

[12]

clientservice1

service1service1service1

service1service1service1

service2

service2

service2

service2

service2

service2service2

service2

service2

service2

service3

181.126ms : client-calls-service1180.527ms : service1-internal-method1

605µs : service1-internal-method2

54.152ms : service1-internal-method3


46µs : service1-internal-method540.910ms : service1-internal-method6





11.492ms : service2-internal-method4105µs : service2-internal-method5

11.494ms : service2-internal-method610.511ms : service2-internal-method7


29.816ms : service2-internal-method912.153ms : service2-internal-method10


client x4 service1 x10 service3 x2 service2 x11unknown-service x2

Economies of scale

[13, 14]

Economies of scale

[13, 14]

Economies of scale

[13, 14]

Economies of scale

[13, 14]

Economies of scale

● Find the perfect balance● Align the risk taken by a service with the risk the business is willing to

bear● Explicit decisions

[15]

Node Assignment

Node Assignment

Matrix MultiplyNetwork Bound

QR FactorizationMem Bandwidth Bound

[16]

Node Assignment

128 192 256 320 384 448 512 576 640 704 768 832 896 960 1024 1088 1152

Function 1

Function 2

Function 3

Function 4

Function 5

Function 6

...

Serverless

Automated ScalingRate of game events

Number of workers

40,000/s

30,000/s

20,000/S

10,000/s

30

20

10

0

[17, 18]

Automated ScalingRate of game events

Number of workers

40,000/s

30,000/s

20,000/S

10,000/s

30

20

10

0

Configurationval data = spark.read.format("libsvm") .load("sample_multiclass_classification_data.txt")

val splits = data.randomSplit(Array(0.6, 0.4), seed = 1234L)val train = splits(0)val test = splits(1)

val layers = Array[Int](4, 5, 4, 3)

val trainer = new MultilayerPerceptronClassifier() .setLayers(layers) .setBlockSize(128) .setSeed(1234L) .setMaxIter(100)

val model = trainer.fit(train)

val result = model.transform(test)val predictionAndLabels = result.select("prediction", "label")val evaluator = new MulticlassClassificationEvaluator() .setMetricName("accuracy")

evaluator.evaluate(predictionAndLabels)





























Configurationval spark = SparkSession .builder() .config("spark.reducer.maxSizeInFlight", maxSizeInFlight) .config("spark.reducer.maxReqsInFlight", maxReqsInflight) .config("spark.shuffle.file.buffer", shuffleFileBuffer) .config("spark.memory.fraction", sparkMemoryFraction) .config("spark.shuffle.service.index.cache.entries", shuffleServiceIndexCacheEntries) .config("spark.memory.storageFraction", sparkMemoryStorageFraction) .config("spark.shuffle.memoryFraction", sparkShuffleMemoryFraction) .config("spark.storage.memoryFraction", sparkStorageMemoryFraction) .config("spark.storage.unrollFraction", sparkStorageUnrollFraction) .config("spark.broadcast.blockSize", sparkBroadcastBlockSize) .config("spark.executor.cores", sparkExecutorCores) .config("spark.default.parallelism", defaultParallelism) .config("spark.files.maxPartitionBytes", sparkFilesMaxPartitionBytes) .config("spark.files.openCostInBytes", sparkFilesOpenCostInByes) .config("spark.rpc.message.maxSize", sparkRpcMessageMaxSize) .config("spark.storage.memoryMapThreshold", sparkSorageMemoryMapThreshold) .config("spark.cores.max", sparkCoresMax) .config("spark.speculation", "true") .config("spark.speculation.interval", sparkSpeculationInterval) .config("spark.speculation.multiplier", sparkSpeculationMultiplier) .config("spark.speculation.quantile", sparkSpeculationQuantile) .config("spark.task.cpus", sparkTaskCpus)

Random trials

● Labelled training examples (feature vector, label)● Classification, regression, clustering, ...● Find an algorithm that for given feature vector finds the correct label● Optimization of an objective function with respect to model parameters

Features Label

100mb,10,100kb,0.75,0.1,0.001,0.75,0.75,0.1,1g,2,8,10000,1000000,100,1g,1ms,1.5,0.01,1g

61.9799

1kb,2147483647,10mb,0.75,0.001,0.1,0.001,0.5,0.001,10mb,8,2,10000,1000000,100,1kb,1ms,1.1,0.75,512mb

12.2511

100mb,10,1mb,0.75,0.5,0.001,0.1,0.001,0.5,100mb,4,1,100000000,1000,1000,10mb,100ms,1.5,0.01,8g

20.7461

... ...

Supervised Learning

Supervised Learning

Supervised Learning

Supervised Learning

Supervised Learning

Supervised Learningdef deep_nn(x, hidden1_units, hidden2_units): with tf.name_scope('hidden1'): weights1 = tf.Variable(tf.truncated_normal([20, hidden1_units], stddev=1.0 / math.sqrt(float(20))), name='weights') biases1 = tf.Variable(tf.zeros([hidden1_units]), name='biases') hidden1 = tf.nn.relu(tf.matmul(x, weights1) + biases1)

with tf.name_scope('hidden2'): weights2 = tf.Variable(tf.truncated_normal([hidden1_units, hidden2_units], stddev=1.0 / math.sqrt(float(hidden1_units))), name='weights') biases2 = tf.Variable(tf.zeros([hidden2_units]), name='biases') hidden2 = tf.nn.relu(tf.matmul(hidden1, weights2) + biases2)

with tf.name_scope('linear'): weights3 = tf.Variable(tf.truncated_normal([hidden2_units, 1], stddev=1.0 / math.sqrt(float(hidden2_units))), name='weights') biases3 = tf.Variable(tf.zeros([1]), name='biases') logits = tf.matmul(hidden2, weights3) + biases3 return logits










Supervised Learning example_batch, label_batch = read_example_batch(filename=FLAGS.data_file, batch_size=batch_size) x = tf.placeholder(tf.float32, [None, 20]) y = deepnn(x, hidden1_units, hidden2_units) y_ = tf.placeholder(tf.float32, [None, 1]) mse = tf.reduce_sum(tf.pow(y-y_, 2))/(2*batch_size) train_step = tf.train.GradientDescentOptimizer(learning_rate=0.25).minimize(mse)

sess = tf.Session() with sess.as_default(): sess.run(tf.global_variables_initializer()) coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord)

for i in range(1000): examples, labels = sess.run([example_batch, label_batch]) sess.run(train_step, feed_dict={x: examples, y_: labels})










Supervised Learning

Problem solved

PROBLEM SOLVED!

Data● What about training data?● Random runs● In-line optimization● Synthetic data● Generative models● Unsupervised learning

General case

Akka-httpKafka

Microservice CockroachDB

Dynamo

Redis Cache

Akka Cluster

Load Balancer

Lambda

General case● Complex● Long running● Large number of variables (many unknown)● Many possible actions● Temporal reward

Reinforcement Learning● Trial and error (positive reinforcement)● Set of states S● Set of actions A● Reward r

● P(s’ | a, s)● s0, a0, r1 -> s1, a1, r2 -> …● Q(s, a) = r + γ * maxa’(Q(s’, a’))

[19. 20]

Environment

Agent

Interpreter

Reward

State Action

Deep Reinforcement Learning● Complex hierarchical actions and planning● Temporal reward

[21]

Deep Reinforcement Learning - Gym

RewardNumber of errors, Cost

Concurrent user load 10,000 12,000 20,000 ...

Instances and sizes 1,1,2 ? ? ?

Action

Nothing

Inc inst size

Dec inst size

Add inst

Remove inst

State

Deep Reinforcement Learning - Gymdef _cost(self, dist_sys, load): cost = 0 can_handle_load = 0 for i in range(len(dist_sys)): if dist_sys[i] == INST_SIZE_0: cost += INST_SIZE_0_COST can_handle_load += INST_SIZE_0_LOAD elif dist_sys[i] == INST_SIZE_1: cost += INST_SIZE_1_COST can_handle_load += INST_SIZE_1_LOAD elif dist_sys[i] == INST_SIZE_2: cost += INST_SIZE_2_COST can_handle_load += INST_SIZE_2_LOAD errors = min((abs(min(can_handle_load - load, 0)) *

ERROR_COEFFICIENT + max(cost, 0)), MAX_COST)





Deep Reinforcement Learning - Gymdef _step(self, action): if (action == ADD_NODE): ... elif (action == INCREASE_INSTANCE_SIZE): ... elif (action == REMOVE_NODE): ... elif (action == DECREASE_INSTANCE_SIZE): ... else: self.dist_sys = self.dist_sys cost = self._cost(self.dist_sys, self.current_load[self.progress]) reward = -1 * ((2 * (cost / MAX_COST)) - 1) self.progress += 1 done = self.progress >= len(self.current_load) - 1 return self.current_load[self.progress], reward, done



Deep Reinforcement Learning - Agentclass agent(): def __init__(self, s_size,a_size): self.inputs = tf.placeholder(shape=[None, s_size],dtype=tf.float32) W = tf.Variable(tf.random_uniform([s_size,a_size],0,0.01)) self.Q_values = tf.matmul(self.inputs,W)

self.largest_Q_value_index = tf.argmax(self.Q_values,1) self.largest_Q_value = tf.reduce_max(self.Q_values)

self.next_Q_values = tf.placeholder(shape=[1,a_size],dtype=tf.float32) loss = tf.reduce_sum(tf.square(self.next_Q_values - self.Q_values)) trainer = tf.train.GradientDescentOptimizer(learning_rate=0.1) self.updateModel = trainer.minimize(loss)

[22, 23]




[22, 23]




[22, 23]




[22, 23]

Deep Reinforcement Learning - Agentwith tf.Session() as sess: sess.run(init) for i in range(num_episodes): state = env.reset() d = False j = 0 while j < env.episode_length() + 1: j+=1 fixed_in = fixed_length_history(episode_history[:,0]) chosen_action, all_Q_values = sess.run( [myAgent.largest_Q_value_index,myAgent.Q_values], feed_dict={myAgent.inputs:fixed_in}) if np.random.rand(1) < random_action_probability: chosen_action[0] = env.action_space.sample()

[22, 23]


[22, 23]


[22, 23]

Deep Reinforcement Learning - Agent next_state,r,d,_ = env.step(chosen_action[0])

episode_history.append([state, chosen_action, r, next_state]) next_largest_Q_value = sess.run( myAgent.largest_Q_value, feed_dict={myAgent.inputs:next_fixed_in}) all_Q_values[0,a[0]] = r + gamma * next_largest_Q_value _ = sess.run( myAgent.updateModel, feed_dict={myAgent.inputs:fixed_in,myAgent.next_Q_values:all_Q_values}) state = next_state

[22, 23]



[22, 23]



[22, 23]



[22, 23]

Learn

[24]

Learn

[25]

Conclusion

● Optimize your code and architecture● Measure, gather data● Use the data!● Continuously improve and evolve

Questions


0845 617 1200

@zapletal_martin @cakesolutions


[email protected]

We are hiringhttp://www.cakesolutions.net/careers

We are hiringhttp://www.cakesolutions.net/careers

References[0] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-whole-stage-codegen.html[1] https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6122906529858466/293651311471490/5382278320999420/latest.html[2] http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html[3] https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html[4] http://www.vldb.org/pvldb/vol4/p539-neumann.pdf[5] https://blog.acolyer.org/2016/05/23/efficiently-compiling-efficient-query-plans-for-modern-hardware/[6] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-rdd-lineage.html[7] https://databricks.com/blog/2017/02/16/processing-trillion-rows-per-second-single-machine-can-nested-loop-joins-fast.html[8] https://www.slideshare.net/SparkSummit/costbased-optimizer-framework-for-spark-sql-spark-summit-east-talk-by-ron-hu-and-zhenhua-wang[9] https://medium.com/@copyconstruct/monitoring-in-the-time-of-cloud-native-c87c7a5bfa3e[10] https://www.lightbend.com/blog/lightbend-monitoring-now-integrates-with-datadog-for-monitoring-akka-based-reactive-applications[11] https://www.lightbend.com/blog/how-to-get-monitoring-right-for-streaming-and-fast-data-systems-built-with-spark-mesos-akka-cassandra-and-kafka?utm_content=buffera2917&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer[12] http://zipkin.io/[13] https://www.theverge.com/2016/7/21/12246258/google-deepmind-ai-data-center-cooling[14] http://www.npr.org/sections/thetwo-way/2017/03/03/518322734/amazon-and-the-150-million-typo.[15] http://money.cnn.com/2016/09/07/technology/delta-computer-outage-cost/[16] https://www.slideshare.net/SparkSummit/ernest-efficient-performance-prediction-for-advanced-analytics-on-apache-spark-spark-summit-east-talk-by-shivaram-venkataraman[17] https://www.slideshare.net/SparkSummit/auto-scaling-systems-with-elastic-spark-streaming-spark-summit-east-talk-by-phuduc-nguyen[18] https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow-autoscaling-to-spark-and-hadoop[19] https://www.youtube.com/watch?v=URWXG5jRB-A[20] https://en.wikipedia.org/wiki/Reinforcement_learning[21] https://stats.stackexchange.com/questions/234891/difference-between-convolution-neural-network-and-deep-learning[22] https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0[23] https://github.com/awjuliani/DeepRL-Agents/blob/master/Q-Network.ipynb[24] https://www.youtube.com/watch?v=22g14GtVhXk[25] https://www.youtube.com/watch?v=C-BY3JhXTiE

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-sql-whole-stage-codegen.html

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/6122906529858466/293651311471490/5382278320999420/latest.html

http://hydronitrogen.com/in-the-code-spark-sql-query-planning-and-execution.html

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html

http://www.vldb.org/pvldb/vol4/p539-neumann.pdf

https://blog.acolyer.org/2016/05/23/efficiently-compiling-efficient-query-plans-for-modern-hardware/

https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-rdd-lineage.html

https://databricks.com/blog/2017/02/16/processing-trillion-rows-per-second-single-machine-can-nested-loop-joins-fast.html

https://www.slideshare.net/SparkSummit/costbased-optimizer-framework-for-spark-sql-spark-summit-east-talk-by-ron-hu-and-zhenhua-wang

https://www.lightbend.com/blog/lightbend-monitoring-now-integrates-with-datadog-for-monitoring-akka-based-reactive-applications

https://www.lightbend.com/blog/how-to-get-monitoring-right-for-streaming-and-fast-data-systems-built-with-spark-mesos-akka-cassandra-and-kafka?utm_content=buffera2917&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

https://www.lightbend.com/blog/how-to-get-monitoring-right-for-streaming-and-fast-data-systems-built-with-spark-mesos-akka-cassandra-and-kafka?utm_content=buffera2917&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer

http://zipkin.io/

https://www.theverge.com/2016/7/21/12246258/google-deepmind-ai-data-center-cooling

http://money.cnn.com/2016/09/07/technology/delta-computer-outage-cost/

https://www.slideshare.net/SparkSummit/ernest-efficient-performance-prediction-for-advanced-analytics-on-apache-spark-spark-summit-east-talk-by-shivaram-venkataraman

https://www.slideshare.net/SparkSummit/auto-scaling-systems-with-elastic-spark-streaming-spark-summit-east-talk-by-phuduc-nguyen

https://cloud.google.com/blog/big-data/2016/03/comparing-cloud-dataflow-autoscaling-to-spark-and-hadoop

https://www.youtube.com/watch?v=URWXG5jRB-A

https://en.wikipedia.org/wiki/Reinforcement_learning

https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0

https://www.youtube.com/watch?v=22g14GtVhXk

Thank you

Intelligent Distributed Systems Optimizations

Software