Distributed Systems Communica3on and models Rik Sarkar Spring 2018 University of Edinburgh
DistributedSystems
Communica3onandmodels
RikSarkarSpring2018
UniversityofEdinburgh
Models
• Expecta3ons/assump3onsaboutthings• Everyideaorac3onanywhereisbasedonamodel
• Determineswhatcanorcannothappen
2
Communica3on&modeling• Modelingdistributedsystems:– Howwecanthinkaboutthem
• Communica3onbetweennearbynodes• Communica3onbetweendistantnodes• Communica3onwithmanynodesSometerminology:• Oneto“all”:Broadcast– All:insomesetofinterest
• Onetoone:pointtopoint3
Packets
• Networkscommunicatedatainmessagesoffixed(bounded)size–calledpackets
• Moredatarequiresmorepackets
• NumberofmessagesorpacketstransmiWedisameasureofcommunica3onused
4
Localareanetworks• Medium:Broadcast
– Messagegoesfromonecomputertoallothercomputers(restrictedtosomeset)• Forexample,allothercomputersintheLAN,orsomeothersysteminconsidera3on
– EthernetLANisabroadcastmedium• Allcomputersareconnectedtoawire.Theytransmitmessagesonthewireandallcanreceive
– WirelessLAN(WiFi)isabroadcastmedium• Electromagne3cwavesisthecommonmedium
5
Localareanetworks
Advantages:• Sendingacommonmessagetoeveryoneiseasy• Findingdes3na3oniseasy– Messagegoestoeveryone– Justhavea“des3na3on”field
Mainissue:Mediumaccess• Sincemediumisbroadcast,twopeopletransmiangatthesame3megarblesmessage
6
Mediumaccess• Onlyonetransmissionata3mecanbeallowed– Mutualexclusionproblem(sharedresourceofcommunica3onwire)
– Wecannotusemessagestosolveit– Protocols:
• TDMA:Everyonehasaperiodicslot• CSMA:Seeifanyoneelseistransmiang.Ifso,defer.• Usuallyacksarealsousedtoensuretransmission
– Retransmitifnecessary
– Bandwidthreduceswithnumberofnodestryingtotransmit.• OneLANshouldnothavetoomanynodes
7
Mediumaccess
• Wireless:morecomplicated• Hiddenterminalproblem• Morecomplexprotocolusingacks
8
OurmodelsofLAN
• Graph:Everynodehasanedgetoeveryother
• Weofenassumethattosendamessage(packet)toanodeonthesamenetworktakesoneunitof3me(or,atmostaconstant)
• ThismaynotbetrueiftherecanbemanynodesinthesameLAN– Butusuallythenumberisnotverylarge
9
Reallifenetworks
• LANsconnectedbyrouters
Ethernet/WifiEthernet/Wifi
PointtoPoint
Routers
10
1 4
2 4
3 4
4 4
1 3
2 3
3 3
5 5
1 1
2 2
4 4
5 4
2 2
3 3
4 3
5 3
Rou3ng• Findingapathinthenetwork• Everynodehasarou3ngtable• EquivalenttoaBFStreeforeverynode
1
2
4
3
5
1 1
3 3
4 3
5 3 11
1-4 4
1-3 3
5 5
1 1
2 2
4-5 42 2
3-5 3
Rou3ng:Distributedsearchforapath
• Smallerrou3ngtablesbycombiningaddresses• UsedinIP(Internet)rou3ng• Smallerrou3ngtablesarepreferable
1
2
4
3
5
1 1
3-5 3
12
Rou3ng
• Realrou3ngismorecomplicated• Withmorethanonepathtoades3na3on,backupsetc
13
Related:Loca3onbasedrou3ng
• Geographicalrou3ngusesanode’sloca3ontodiscoverpathtothatnode.
x
y
GreedyRou3ng:Forwardtotheneighborthatisnearesttothedes3na3on
14
Largenetworks• Communica3onistypicallypoint-to-pointusingrou3ng
• Broadcastisnotautoma3c– Ifweneedbroadcast,wewillhavetoarrangeaflood(orsomeothermethod)
Ethernet/WifiEthernet/Wifi
PointtoPoint
Routers
15
Transportmanagement• UDP:
– Sendapacket,hopethatthenetworkroutesanddeliversit,in3me
– NoSequencenumber• NotnecessarilyFIFO
– Usefulinstreamingaudio/video.Notforimportantdata.
• TCP:– Sendapacket(orfewpackets)– Packetshavesequencenumber
• FIFO– Ifnoacksarrive,resendpackets– Ifnoacksarefoundafermanytries,returnerror
16
TCP
• Doesdistributedconges3oncontrol– Whenpacketsdon’tgetdelivered,TCPslowsdownthestream• Assump3on:routersdroppacketswhentherearetoomany
• Difficulty– Acksmaynotarriveduetootherfactors• Someconnec3onfailedtemporarily• Usermovedfromonenetworktoanother
17
Networkstack
• Eachlayersolvesadifferentdistributedproblem
18
Formwikipedia
Communica3on:Overlaynetwork• Wemaysome3mesignorepartsofthenetwork
– Nodesthatcarrymessagesbutdonotdirectlypar3cipateeg.routers
– Oredgesthatexistbutwearenotusing– Orwedon’tknowabout
• Ofenusedinpeer-to-peernetworks– Noteverynodeknowsallothernodesinthenetwork– Butcommunicatestoknownnodesthroughrou3ng
19
Blockingandnon-blockingcommunica3on
• Blockingcommunica3on– Sendersendsmessage,waitsun3lreceiverreplies– Doesnotdoanythinginthemean3me
• Non–blockingcommunica3on– Sendersendsmessage,thencon3nuesitsownworkwithoutwai3ng
– Whenreceiverreplies,orsomeothermessagearrives,nodeinterruptscurrentworktohandlemessage
• Some3mesthesearecalledsynchronous/asynchronous,butwewilltrynottousethat
20
Computa3on• Synchronous:– Opera3oninrounds– Inaround,anodeperformssomecomputa3on,andthensendssomemessages
– Allmessagessentattheendofroundxareavailabletorecipientsatstartofroundx+1• Butnotearlier
21
Computa3on
Communica3on
Round1 Round2 Round3
Communica3on
• Synchronous– Canbeimplementedifmessagetransmission3meisboundedbysomeconstantsaym
– Computa3on3mesforallnodesareboundedbysomeconstantc
– Clocksaresynchronized(sufficiently)– Thenseteachroundtobem+cindura3on
22
AsynchronousCommunica3on
• Nosynchroniza3onorrounds– Nodescomputeatdifferentandarbitraryspeeds– Messagesproceedatdifferentspeeds:maybearbitrarilydelayed,maybereceivedatany3me
• Worstcasemodel– Noassump3onaboutspeedsofprocessesorchannel– (Butdoesnotincludecommunica3on/computa3onerrors)
23
AsynchronousCommunica3on
• Hardertomanage– Messagecanarriveatany3meaferbeingsent,mustbehandledsuitably
– Possibletomakesomesimplifyingassump3onsE.g.:• ChannelsareFIFO:orderofmessagesonachannelarepreserved• Somecodeblocksareatomic(notinterruptedbymessages)• Eithercommunica3onorcomputa3on3mesbounded
24
Synchronouscommunica3oninRealsystems
• Synchronouscommunica3oncanbeafairmodel• Moderncomputersandnetworksarefast– (thoughnotarbitrarilyfast)
• Easiertodesignalgorithmsandanalyze• Welldesignedalgorithmsarefasterandmoreefficient
• Ofencanbeadaptedtoasynchronoussystems– Ofenastar3ngpointfordesign
25
Failures
• Nodesmayfail– Hardwarefailure– Runoutofenergyorpowerfailure– Sofwarefailure(crash)– Permanent– Temporary(whathappenswhenitrestarts?Recoversthestate?Startsfromini3alstate?)
– Modeldependsonsystem.E.g.differenttypesoffailuresoccurwithcorrespondingprobabili3es
26
Nodefailures• Commonabstractmodels
– Stoppingfailure:nodejuststopsworking• Mayneedassump3onsaboutwhichcomputa3on/communica3onitfinishesbeforestopping
• Mayneedassump3onaboutneighborsknowingoffailure– Byzan3nefailure:nodebehavesasanadversary
• Imagineyourenemyhastakencontrolofthenode• Istryingtospoilyourcomputa3on
• Nodesmayfailindividually– E.g.eachnodefailswithprobabilityp
• Nodesmayhavecorrelatedfailure– E.g.allnodesfailinaregion(datacenter,sensorfield)
27
Link/communica3onfailure• Maybetemporary/permanent• Mayhappendueto
– Hardwarefailure– Noise:electronicdevices(microwavesetc)maytransmitradiowavesatsimilarfrequenciesanddisruptcommunica3on
– Interference:Othercommunica3ngnodesnearbymaydisruptcommunica3on
• Effects– Channelsilentandunusable(hardwarefailure)– Channelac3ve,butunusableduetonoiseandinterference– Channelac3ve,butmaycontainerroneousmessage(maybedetectedbyerrorcorrec3ngcodes)
28
Security• Issues:– Unauthorizedaccess,modifica3on.Makingsystemsunavailable(DOS)
– AWackononeormorenodes• Causingtoitfail• Readdata• Takingcontroltoreadfuturedata,disruptopera3on
– AWackoncommunica3onlinks/channel• Blockcommunica3on• Readdatainthechannel(easyinwirelesswithoutencyp3on)
• Corruptdatainthechannel
29
Security
• Solu3onsusuallyhavespecificassump3onsofwhattheadversarycando
• E.g.Ifadversaryhasaccesstochannel– Cryptographymaybeabletopreventreading/corrup3ngdata
30
Mobility
• Movementmakesithardertodesigndistributedsystems– Communica3onisdifficult
• Delays,lostmessages• Edgeweightscanchange
– Applica3onsthatdependonloca3onmustadapttomovement
• Howdopeoplemove?Whatisamodelofmovement?– Notyetwellunderstood
31
Modelingdistributedsystems
• Manypossibili3es• Chooseyourassump3onscarefullyforyourproblem
• PaycloseaWen3ontowhatisknownaboutcommunica3on/network
• Startwithsimplermodels– Usuallymoreassump3ons,fewerparameters– Seewhatcanbeachieved– Thentrytodrop/relaxassump3ons
32
33