Micro- Services DevOps Distributed Systems

Micro-Services

DevOpsDistributedSystems

Microservices andScienceGateways

MartinFowler’sDefinition

• Developasingleapplicationasasuiteofsmallservices• Eachservicerunsinitsownprocess• Services communicatewithlightweightmechanisms

• “Often anHTTPresourceAPI”• Butthathassomeproblems• Messagingisbetter.

• Theseservicesarebuiltaroundbusinesscapabilities• Independentlydeployablebyfullyautomateddeploymentmachinery.• Minimumofcentralizedmanagementoftheseservices,

• Maybewrittenindifferentprogramminglanguages• Mayusedifferentdatastoragetechnologies.

http://martinfowler.com/articles/microservices.html

MonolithicApplications

•aclient-sideuserinterface•adatabase•aserver-sideapplication.

Enterprise applicationsthree tieredarchitecture

•handleHTTPrequests,•executedomainlogic,•retrieveandupdatedatafromthedatabase,•selectandpopulateHTMLviewstobesenttothebrowser.

Server-side Application

•Asinglelogicalexecutable.•Changesrequirebuildinganddeployinganewversionoftheserver-sideapplication.

Thisserver-sideapplicationisamonolith

OtherQuestionableMonolithicPatterns

• Theserver-sideapplicationinteractswithasingledatabase• NoteverythingneedstofitintooneDBsystem• NotjustCAPchoices:usetheDBthatfitseachsubsytem’s problemandyourdevelopmentpatterns

• Splittingteamsintotiers• TheDBteam,theUXteam,theServerAppteam• Thedeploymentandoperationsteam

CAP:Consistency-Availability-PartitionTolerance

Browser

Web InterfaceServer

Application Server

ServerSDK

Client SDK

Karst:MOAB/Torque

ResourcePlugins

Stampede:SLURM

Comet:SLURM Jureca:SLURM

HTTPS

HTTPorTCP/IP

RecalltheGatewayOctopusDiagram

“Super”SchedulingandResourceManagement

Inmicro-servicearch,thesealsoneedscheduling

APIServer

ApplicationManager

MetadataServerApplication

ManagerApplicationManagerApplicationManagerApplicationManagerApplicationManager

MetadataServerMetadataServerMetadataServerMetadataServerMetadataServer

APIServerAPIServerAPIServer

APIServer

ApplicationManager

MetadataServer

DecouplingtheApp

Server,Take1

WhatScienceGatewayMicroservices CanWeIdentify?

MySuggestedList(1/2)

• APIServer• Highavailability,load-balancedEntryPoint

• Scheduler• HighlyavailableleaderOrchestrator

• Resources,Applications• Highlyavailable?

StatelessMetadataServer

MySuggestedList(2/2)

• Experiment stateStatefulMetadataServer

• Submitsjobs• FireandforgetApplicationManager

• MonitorsjobsonremoteresourcesMonitorService

• Centralizeyourservice logsLogging

DrawandThink

ThoughtExercise

• AssumeAmazonCloudandabigbudget• Howdothesecomponentsinteract?• HowcanIdistributecopiesofcomponentsacrossmultipleavailabilityzones?

• Whatkindsoffailurescanoccur?• HowcanIhandlefailuresofdifferentcomponents?

LastThoughts:AreMonolithicApplicationsBad?• Amonolithicserverisanaturalwaytoapproachbuildingmanysystems.• Monolithicapplicationscanandshouldstillbemodular• Allyourlogicforhandlingarequestrunsinasingleprocess

• Youcanusethebasicfeaturesofyourlanguage todivideuptheapplicationintoclasses,functions,andnamespaces.

• MonolithscanstilluseDevOps• Youcanrunandtesttheapplicationonadeveloper'slaptop,• Useadeploymentpipeline toensurethatchangesareproperlytestedanddeployedintoproduction.

• Youcanhorizontallyscalethemonolithbyrunningmanyinstancesbehindaload-balancer.

• DBCAPproblemsdon’thitmostapplications

TheDividingLine

Whendoyoudecoupleyourcode?Whenisamonolithicapplicationbetter?

MonolithicApplications:TraditionalSoftwareReleases• Softwarereleasesoccurindiscreteincrements• Runonclients’systems• Releasesmaybefrequentbuttheyarestilldistinct

• Firefox• OSsystemupgrades

• Traditionalreleasecycles• Extensive testing• Alpha,beta,release candidates,andfullreleases

• Extensiverecompilingandtestingrequiredaftercodechanges• Codechangesrequiretheentirereleasecycletoberepeated

MicroServices:SoftwareasaService

• Doesyoursoftwarerunasaservice?• Doyourunthisserviceyourself?Ordoesanotherpartofyourorganizationruntheservicethatyourteamdevelops?

• Traditionalreleasecyclesdon’tworkwell• Test-release-deploy takestoolong• Maymakereleasesmanytimesperday

• Youcanbealittlemoretolerantofbugsdiscoveredafterreleaseifyoucanfixquicklyorrollbackquickly.

• Getnewfeaturesandimprovementsintoproductionquickly.

Microservices:StartwithModularity

• Thinkaboutyourcurrentapplication.• Howcanyoubreakitintomodularcomponentparts?• Whataretheinterfaces?Whatarethemessagesthatyouneedtosendbetweentheparts?

• Yourcodeshouldbemodularanyway.• OGSIandSpringdothisforsingleJVMJavaapplications• Youcanmaintainamodulewithouttouchingothercode.

SpringCoders,SpeakUp!

Whatcomponents,POJOs,etc doyouhaveinyourcodenow?

Shouldapersonorgroup“own”asubsetofmodulesormicroservices?

• Teammemberscomeandgo• Theyaren’talwaysavailableanyway• Work-lifebalancewilleventuallybecomeimportanttomost• Nooneshouldbeindispensable

Forsmallteams,NOinmyexperience.

• “Possessiveownership”culture• Opaquecode• “Notmyproblem”ifithappens inanothermodule

Moduleownershipleadstootherproblems

Ownership and Conway’s Law

• Anyorganizationthatdesignsasystem(definedbroadly)willproduceadesignwhosestructureisacopyoftheorganization'scommunication structure.–MelvinConway

InOtherWords

• If youseparateyourorganizationinto• Systemsspecialists,• DBadmins,• Softwaredevelopers,• Userexperiencespecialists, etc,

• Andaskthemtobuildsomething,• Then

• thesystemwilllookliketheorgchart.• ThisistheoppositeoftheDevOps approach• Createsthebureaucraticinfightingthatplagueorganizations.

Products,NotProjects

• Thealternative:organizearoundproducts• Aproductteamhasalltheexpertise itneeds• JimGrayinterviewsWernerVogels:https://queue.acm.org/detail.cfm?id=1142065

• “Youbuildit,youownit”

• Problemswiththisapproach:• Someonemayworkonmultipleproducts• Whoisthatperson’sboss?

MyAdvice:OwntheEntireProduct

Evenifyouaren’tthemanager.Owntheproduct.

Doyouseethisinthewayyouworknow?

Orinthewayyouworkedbeforeyoucamebacktoschool?

HowBigIsaModule?Howlittle ismicro?

RulesofThumbYouMightEncounter

• Amazon’stwo-pizzarule• Teamsshouldn’tbelargerthanwhattwopizzascanfeed

• “Agoodprogrammershouldbeabletorewriteamicroservice inacoupleofweeks”

• BoundedContextsfromDomainDrivenDesign

CanWeMakeThisMoreRigorous?

Messagingpatternsbetweencomponents

•REST•CommoninInternetapplicationsbutyouwouldneverdothiswithincodeinasingleJVM

Request-response

•CommoninsingleJVMapplication:Observer,listenerpatterns•Butnot socommonindecoupledapplications.•Moregenerally,asynchronousmessagingisalittlescary.

Push

•Thecomponentknowswhichcomponentneedstogetthenewinformation.One-to-one

•Componentsdon’tneedtoknowwhichothercomponentsneedstheinformation.Theyjustbroadcastit.Many-to-many

HowBigIsaComponent?

• Oneanswer:theoptimalmessagepatternshouldnotchangewhengoingfromsingleJVMtodistributedcomponents.

• Telescope inandtelescopeout

• Componentsthatruninthesameprocesscanbechattyandsendlotsofmessages.

• Messages canbepointerstoshareddataobjects• Messages arealwaysreceived• Thisistrueforbothrequestandpushmessaging.

Pan-GalacticComponents

• Componentsthatrunacrosstheuniverseneeddifferentmessages,eitherinfrequentorterse.

• Noshareddataobjects topointto.• Distributedcomponentsmustalsoassumeawidevarietyofmessagefailures

• Lostconnections• Recipients neverreceived themessage• Recipients receivedmessages inthewrongorder• Recipients crashedwhileprocessing• ...

• Microservices aredistributedsystems• ShouldknowanduseDSideas

Microservices AreNotDistributedObjects

• AcomponentisnotanobjectintheOOsense.• AcomponentshouldbeaserviceintheSOAsense• RecallRESTarchitecture:services shouldbeasidempotentaspossible.

• Ifyousendthesamemessagemorethanonce, theserverstateafterprocessingthemessageisthesame.

• Ex:Ifacomponentreceives10identicalmessagesayingsubmitajob,thecomponentshouldonlysubmitthejobonce.

• Howdoyouknowthemessageisreally identical?Timestamps,otheridentifiers,etc

• “Sharednothing”betweenservices• Messageprocessingshouldhavenosideeffectsthatneedtobeexposedtoothercomponents

• Anti-pattern:allthoseoutputfilesgeneratedbyscientificapplications

DecentralizedDataManagement

• TheRegistrycomponent,backedbyasingleDBsystem,holdsallthedata

Thisisanopenchallengefor

ApacheAiravata

• Onedatabasehasall theinformationfortheapplication• Limitsyourchoiceofdatastores:youhavetousethesameDBtechnologyforeverything.

• Microservices constantlyneedtomakeremoteDBcalls• Performanceandsecurity

MicroserviceAntipattern

SettingDataBoundaries

• Thisiseasywhenthedataisn’tconnectedorlightlyconnected.• ButnoteasytosplitupatraditionallydesignedDBschema.• Sciencegatewaydataboundaries

• Youmaywanttosplitstateless andstateful data

• Ifyousplitevenlightlyconnecteddataacrossservices,youwillneedtoconsiderconsistency

• Strong:requiresdistributedtransactions,whichcanbetricky• Weak:eventualconsistency.Canyoulivewiththis?

• Whatdataboundariesdosciencegatewayshave?

Howdoservicescommunicate?

• Ideally,alwaysdothesamething.• Yourdevelopmentenvironmentshouldbeaminiatureversionofyourdeploymentenvironment:containerization

• Encapsulateandabstract• ThistouchesDevOps CI/CD,sothoseareseparate lectures

• Keepthecommunicationpatternsthesame• Push,pull• 1-1,orN-N

• We’lllookatmessagingsystemsindetailnext• Pushmessaging,many-to-many,reliabledelivery,replay,etc

IfImakeeverythingdistributed,howcanItest?• Unittestswithincomponentsaresimple• Buttestingtheentiresystembecomesanoperationaltest

• Thisisnotabadthingifyoucanscriptyouroperationtests

• Andyoushould

• Monolith:testyourcodewhilebuildingwithMaven

• BringupsimpletestDBslikeSQLiteorDerby• Onebigbuilddoeseverything

Microservice Testing

• Microservice:testyourcodeinanoperation-likesetting• Don’tbuildtheentirecodeeverytime• Buildonlythemicroservice youareworkingon.• Soyourbuildshouldbemodular

• Somaybeyoudon’tneedonebigbuildsystemafterall• Andmaybeyoudon’tneedonegit repoforeverything...• Andmaybeyoudon’tneedoneversion#thatcoversallyourcode...

• Soreleasesbecomeincremental• Testing,monitoring,andloggingbecomemorecontinuous

• ThisisatouchpointbetweenMicroservices andDevOps

DesigningforFailure

• Microservices needtobebuilttotolerateerrors.• Whatkindsoferrorswillsciencegatewaymicroservices encounter?• Howdoyoudetectthesefailures?• Whattoolsdoyouneed?• Whatstrategiescanyouusetodosystemtesting?

MessaginginDistributedSystems

Zab:AtomicBroadcasting

ClientsSeeSpaces

Zookeeper’sclientAPIresemblesastructuredversionoftheTupleSpacesconceptofparallelprogramming.

AdvancedMessageQueuingProtocolOverview

MoreDistributedServiceConcepts

AMQP:NetworkProtocolandArchitecture,NotAnAPI• ManyImplementations

• RabbitMQ• ApacheActiveMQ• ApacheQpid• SwiftMQ• Andothers:• Thiscontradictsmyearlierdistinctionbetweenhorizontalandverticalstandardizationefforts

• AtleastforVersion0-9-1• EachimplementationcanhaveitsownAPI• I’llfocusonVersion0-9-1• ThisisnotanAMQPtutorial

BasicConcepts

AnAMQPServer(orBroker)

• Exchange• Message Queue• Exchanges caninteract withmultiple message queues

Twomainparts

• Accepts producermessages• SendstotheMessage QueueExchange

• Routesmessages todifferentconsumers depending onarbitrarycriteria• Buffersmessages when consumers arenotabletoaccept them fastenough.

MessageQueue

ProducersandConsumers

• ProducersonlyinteractwithExchanges• ConsumersinteractwithMessageQueues• Consumersaren’tpassive

• Cancreateanddestroymessagequeues

• Thesameapplicationcanactasbothapublisherandaconsumer• Youcanimplement Request-Response withAMQP• Exceptthepublisherdoesn’tblock

• Ex:yourapplicationmaywantanACKorNACKwhenitpublishes• Thisisareplyqueue

MessageQueue

• Typesofqueueproperties:• Privateorshared• Durableortemporary• Client-named orserver- named,etc.

• Store-and-forward queue:holdsmessagesanddistributesthesebetweenconsumersonaround-robinbasis.

• Durable andsharedbetweenmultipleconsumers.• Privatereplyqueue:holdsmessagesandforwardsthesetoasingleconsumer.

• Replyqueuesaretypicallytemporary,server-named, andprivatetooneconsumer.• Privatesubscriptionqueue:holdsmessagescollectedfromvarious"subscribed"sources,andforwardsthesetoasingleconsumer.

ConsumersandMessageQueues

• AMQPConsumerscancreatetheirownqueuesandbindthemtoExchanges

• Thishasaninterestingimplication• Queuescanhavemorethanoneattachedconsumer• AMQPqueuesareFIFO

• AMQPallowsonlyoneconsumerperqueue toreceivethemessage.• Useround-robindeliveryif>1attachedconsumer.• Thisgreatlysimplifiesthings

• Ifyouneed>1consumertoreceiveamessage,youcangiveeachconsumertheirownqueue.

• Usetopicmatchingtoroutemessages• ComparethistoZab andatomicbroadcastprotocols,generally.

• AMQPmakessomesimplifications

TheExchange

• Receivesmessages• Inspectsamessageheader,body,andproperties• Routesmessagestoappropriatemessagequeues• Routingusuallydonewithroutingkeys inthemessagepayload

• Forpoint-to-pointmessages,theroutingkeyisthenameofthemessagequeue

• Forpub-subrouting,theroutingkeyisthenameofthetopic• Topicscanbehierarchical

Publish-SubscribePatterns

• Usefulformany-to-manymessaging• Inmicroservice-basedsystems,severaldifferenttypesofcomponentsmaywanttoreceivethesamemessage

• Buttakedifferentactions• Ex:youcanalwaysaddaloggerservice

• Youcanalwaysdothiswithexplicitlynamedroutingkeys.• Youmayalsowanttousehierarchical(namespace)keynamesandpatternmatching.

• gateway.jobs.jobtype.gromacs• Gateway.jobs.jobtype.*

TheMessagePayload

• Readthespecificationformoredetails.• IngeneralAMQPfollowsthehead-bodyformat• Themessagebodypayloadisbinary• AMQPassumesthecontentishandledbyconsumers

• Itistheconsumer’sjobtodosomethingwiththecontent.• UnlikeJMS

• YoucouldserializeyourcontentwithJSONorThriftanddeserialize ittodirectlysendobjects.

MessageExchangePatterns

DirectExchange

• Apublishersendsamessagetoanexchangewithaspecificroutingkey.

• Theexchangeroutesthistothemessagequeueboundtotheroutingkey.

• Oneormoreconsumersreceivemessagesiflisteningtothequeue.

• Default:round-robinqueuingtodelivertomultiplesubscribersofsamequeue

Queue.Declare queue=app.svc01 Basic.Consume queue=app.svc01 Basic.Publish routing-key=app.svc01

Fanout Exchange

• MessageQueuebindstoanExchangewithnoargument

• PublishersendsamessagetotheExchange

• TheExchangesendsthemessagetotheMessageQueue

• AllconsumerslisteningtoallMessageQueuesassociatedwithanExchangegetthemessage

TopicExchange

• MessageQueuesbindusingroutingpatternsinsteadofroutingkeys.

• APublishersendsamessagewitharoutingkey.

• ExchangewillroutetoallMessageQueuesthatmatchtheroutingkey’spattern

MoreExamples

• TheRabbitMQ tutorialpagehasseveralniceexamplesofclassicmessageexchangepatterns.

• https://www.rabbitmq.com/getstarted.html

SomeUsefulCapabilitiesofMessagingSystemsfor

MicroservicesOverarchingRequirement:Itshouldsupportyoursystem’s

distributedstatemachine

Let’sbrainstormsome

UsefulCapabilities:MyList(1/2)

• Supportsbothpushandpullmessaging• Delivermessagesinorder• Successfullydeliveredmessagesaredeliveredexactlyonce

• MultiplerecipientsOK• Delivermessagestooneormorelistenersbasedonpre-definedtopics.

• Storemessagespersistently• Therearenoactiverecipients.• Allrecipientsarebusy

• Determineifcriticalmessagesweredeliveredcorrectly

UsefulCapabilities:MyList(2/2)

• Redelivermessagesthatweren’tcorrectlyreceived• Corruptedmessages,norecipients,etc• Recipientcanchange

• Redelivermessagesonrequest• Helpsclientsresynchtheirstates

• Allowothercomponentstoinspectmessagedeliverymetadata.• Supportselasticity, faulttolerance

• Prioritymessaging?• QualitiesofService

• Security,faulttolerance

WhichMessagingSoftwaretoChoose?

• AMQPdoesnotcoverallthecapabilitieslistedabove.• Itcanbeextendedtocoverthese inmanycases

• AMQPmessagingsystemimplementationsarenotnecessarilycloud-ready

• Theyhavetobeconfiguredashighlyavailableservices.• Primary+failover

• Nofancyleaderelections, etc asused inZookeeper+Zab• Havescalinglimitations,althoughthesemaynotmatteratourscales.

• Othermessagingsystems(Kafka,HedWig)arealternatives

SomeApplications

SimpleWorkQueue

• Queueupworktobedone.• Publisher:pushesarequestforworkintothequeue

• Queue shouldbeasimpleDirectExchange• MessageQueueshouldimplement“onlydelivermessageoncetoonceconsumer”.

• Round-robinscheduling.• RabbitMQ doesthisoutofthebox

• Consumer:SendsanACKaftercompletingthetask• IfaQueue-ClientclosesbeforeanACK,resendmessagetoanewconsumer.

• RabbitMQ detectsthesetypesoffailures.

GatewayAPIServer

AppMan Worker

AppMan Worker

AppMan Worker

Broker

MSG1, 2,3MSG2

MSG1

MSG3

ACK

ACK

ACK

• 3Job requests come totheAPIServer• APIServer publishes toBroker(Exchange+Message

Queue)• Brokersends toworkersinround-robin• Workers sendACKswhendone• IfBrokerdetectsaclosed connection beforeanACK, sends

MSGtoadifferentworker

SimpleWorkQueue

WhatCouldPossiblyGoWrong?

• Jobstakealongtimetofinish,soACKsmaynotcomeforhours.• DurableconnectionsneededbetweenConsumersandMessageQueues• Alternatively,theACKcouldcomefromadifferentprocess

• Jobsmayactuallygetlaunchedontheexternalsupercomputer,soyoudon’twanttolaunchtwicejustbecauseofamissingACK

• Clientshavetoimplementtheirownqueues• Couldgetanotherworkrequestwhiledoingwork.

WorkQueue,TakeTwo

• Orchestratorpushesworkintoaqueue.• Haveworkers requestworkwhentheyarenotbusy.

• RabbitMQ supportsthisas“prefetchCount”• Useround-robinbutdon’tsendworktobusyworkerswithoutstandingACKs.• Workers donotreceiveworkrequestswhentheyarebusy.

• WorkersendsACKaftersuccessfullysubmittingthejob.• Thisonlymeansthejobhasbeensubmitted• Worker cantakemorework

• Aseparateprocesshandlesthestatechangesonthesupercomputer• Publishes”queued”, “executing”, ”completed”or“failed”messages

• Whenjobisdone,Orchestratorcreatesa“cleanup”job• Anyworker availablecantakethis.

WorkQueue,Take2

Orchestrator Broker

AppMan Worker

EmailListener

1a.Ready

2.RunJob

3.ACK

AppMan Worker

5.Done

1b.Ready

8.CleanUp

9.ACK

0.RunJob

ExperimentMetadataManager

4.,6.,10.JobStatus

7.Cleanup

WhatCouldPossiblyGoWrong?

• AWorkermaynotbeabletosubmitthejob• Remotesupercomputer isunreachable, forexample• WeneedaNACK

• TheOrchestratorandExperimentMetadatacomponentsarealsoconsumers.

• ShouldsendACKstomakesuremessagesaredelivered.• OrchestratorandExperimentMetadataManagermayalsodieandgetreplaced.

• UnlikeAppMan workers,OrchestratorandEMMmayneed aleader-followerimplementation

• Brokercrashes• RabbitMQ providessomedurabilityforrestarting• Possibletolosecachedmessages thathaven’tgonetopersistentstorage

WhatElseCouldGoWrong?

• Lotsofthings.• Howdoyoudebugunexpectederrors?

• Logs

• AloggerlikeLogStash shouldbeoneofyourconsumers• Noone-to-onemessagesanymore.• Everythinghasatleast2subscribers

• Yourlogservice• Themaintarget

• OryoucoulduseFanout

Summary

• Microservices areapplicationsbuiltoutofdistributedcomponents• ImportantifyourunSoftwareasaService• ConnectiontoDevOps:

• Acceleratethedevelop-test-deploycycle• ContinuousIntegrationandContinuousDeliveryareessential

• Servicesneedtocommunicate• Asynchronousmessaging ispowerfulmechanism• Push,many-to-many• Caneasilyextendtoaddmorecomponents

Micro-Services

DevOpsDistributedSystems