BENDING SPARK TOWARDS ENTERPRIZE NEEDS BORIS TROFIMOV @ SIGMA SOFTWARE
BENDINGSPARKTOWARDSENTERPRIZE NEEDS
BORISTROFIMOV@SIGMASOFTWARE
USERINTERFACE
SERVICELAYER
DATASTORE
CLASSIC APPROACH
PRESENTATIONLAYER
APPLICATIONSERVICES
DOMAINLAYER
DAL
INFRASTRUCTURE
CLASSIC LAYERS
CLASSIC APPROACH
USERINTERFACE
SERVICELAYER
DATASTORE
CQRS
USERINTERFACE
SERVICELAYER
DATASTORE
COMMAND/WRITEMODEL READMODEL
CQRS NAKEDUSERINTERFACE
SERVICELAYERWRITESTORE
COMMANDBUS
COMMANDHANDLER
REPOSITORY
DomainModel Domain
Model
EVENTBUS
EVENTHANDLER
Commands
READMODELSTORE
QUERYFACADE
Query DTO
WRITEMODEL READMODEL
USERINTERFACE
MULTIPLE READMODELSUSERINTERFACE
SERVICELAYERWRITESTORE
COMMANDBUS
COMMANDHANDLER
REPOSITORY
DomainModel Domain
Model
EVENTBUS
EVENTHANDLER
Commands
EVENTHANDLER
READMODELSTORE1
READMODELSTORE2
QUERYFACADE QUERYFACADE
WRITEMODEL READMODEL
BULLETIN BOARDAPP
APPLICATIONFEATURES
•ADDBulletinwithspecificauthornameandmessage•VIEWlistofpublishedbulletins
SERVICELAYER
SPARKISCOMINGUSERINTERFACE
MONGOREAD/WRITESTORE
KAFKA
COMMANDHANDLER
REPOSITORY
DomainModel Domain
Model
Commands
QUERYFACADE
Query DTO
WRITEMODEL READMODELSPARK
SERVICELAYER
SPARKISCOMINGUSERINTERFACE
MONGOREAD/WRITESTORE
KAFKA
COMMANDHANDLER
REPOSITORY
DomainModel Domain
Model
Commands
QUERYFACADE
Query DTO
WRITEMODEL READMODELSPARK
SHAREDSTORE
TECHNOLOGY STACK
FAÇADE• Java8• SpringBoot&MVC• Mongo
COMMANDPROCESSOR• Kafka0.9.0.1• Scala2.11• Spark2.0.0Streaming
DEPLOYMENT VIEW
DOCKER
KAFKAZOOKEEPER
MONGO
DEVMACHINE
LOCALSPARK
FACADE
DEMO
https://github.com/btrofimov/spark-enterprise-example/tree/master/nonblocking-bulletinboard
WAIT
WELCOMETOASYNCHONOUSHELL
ATLEASTFOURSOLUTIONS
• C'mon,it’sOK
• Pullingserviceuntilentityisadded
• Addpushchannelanddelivereventsupstairs
• MakeRESTmethodsblockeduntilcommandisfinished
MAKEITSYNCHRONOUS
DEMO2
https://github.com/btrofimov/spark-enterprise-example/tree/master/blocking-bulletinboard
RETROSPECTIVE
READMODELinstances
WRITEMODELinstances
EFFICIENT SCALEOUTSEPARATEDSCALEOUTDEPENDINGONMODEL
NEEDS
dataStream.foreachRDD { rdd =>
val values = rdd.values
values.map { cmdMessage =>…
}.foreach { cmdMessage =>
…}
}
DESERIALIZATION ERRORS
MIGHTFAILDURINGLAUNCHIFPIPELINEHASBEENCHANGED
RISK: AFFECTSCHECKPOINTS
DESERIALIZATION ERRORS
• Hardtokeepcodetoleranttotheseerrorsandtestableatthesametime.• SparkwithKafkaconnector0.10allowstosaveKafkaoffsetsanditispossibletorestorefromtheminsteadofcheckpoints.
THROUGHPUT VSLATENCY
Spark-basedserviceshavenicethroughputhoweverduetointernaloverheadstheyhavelowerlimitonlatency
IDENTIFIER ASSIGNRESPONSIBILITY
• Clientgeneratesuniqueidentifiersandpassesittoservice• Servicegeneratesuniqueidentifierandpassesitbacktoclient
EVENTUALVS STRONG CONSISTENCY
• CQRSappsusuallyareeventuallyconsistent• ChoosewiselybetweenCQRSandtraditionalapproach• Consideroptimisticorpessimisticlockingtoachievestrongconsistency• Insomecasesdefinedreasonabletimeoutsorin-progresslistsmighthelptosolveraceconditions(dependsoncommandroundtrips)
FAULTTOLERANCE
FAULTTOLERANCE
• Sparkautorerunstaskstomitigatenetworkorotheroutages• Easyandsafeapplicationrestartbasedoncheckpoints
THANKYOU
slides code