Scalable Control Plane Substrate - platformlab.stanford.edu · MoEvaon • Separaon of control plane is a common trend: networks/systems – Challenging to design, hard requirements

ScalableControlPlaneSubstrate

SachinKa3,JohnOusterhout,GuruParulkar,MarcosAguilera,CurtKolovson

ONRC+ON.Lab+RAMCloud,VMWare

MoEvaEon

•  SeparaEonofcontrolplaneisacommontrend:networks/systems–  Challengingtodesign,hardrequirementsonthroughput,latency,consistency

•  Wehavebeenbuildingcontrolplanesforourspecificsystems–  ONOS,SoORAN,andRAMCloudCoordinator

•  Canwedesignageneralizedscalablecontrolplanewith–  AcommonfoundaEonthatcanbecustomizedfordifferentcontexts

ExamplesofControlPlanes

•  SDNControllers–  Datacenter,enterprise,serviceproviderwiredandcellularnetworks–  IoT

•  VMmanager(e.g.NovainOpenStack)–  DataCenters

•  Nameservers(“namenode”)–  distributedfilesystem

•  Scheduler–  Manydistributedsystems

•  SoOwaredefinedstorage•  …

New“Control”Planes–Drones,Cars

•  Controllingfleetsofdronesandcars–  Collisionavoidance,sharingtheairspace–  Airtrafficcontrolfordrones–  LogisEcsanddelivery

App1 App2

ControlPlane

OpenInterface(s)

SeparaEonofControlPlane

SeparaEonofControlPlane

GlobalViewApp3

OpenInterface

Server

Server

Storage

SwitcheseNBsServersStorage

Server

LogicallyCentralizedControlPlane

•  Providesglobalview

•  Makesiteasytoprogramcontrol,management,configapps

•  Enablesnewapps

KeyPerformanceRequirements

Control

Apps Apps

Global Network View / State Global View / State

HighVolumeofState:~500GB-2TBHighThroughput:~500K-20Mops/second~100Mstateops/secondLowLatencytoEvents:1-10sms

Difficultchallenge!

Highthroughput|Lowlatency|Consistency|Highavailability

Server

Server

Server

Storage

●  Scalability,HighAvailabilityandPerformance●  NorthboundandSouthboundAbstracEons●  Modularity

ControlPlaneChallenges

ExampleScalableControlPlanes

ONOSSoORAN

RAMCloudCoordinator

Scale-OutDesignw/MulEpleONOSInstances

ONOS

Apps Apps

Global Network View / State Global Network View / State

Each instance is identical

One can add and remove instances seamlessly

Each instance is a master for a subset of switches

It works like a single system for apps and network devices

ONOS Instance 1 ONOS Instance 2 ONOS Instance 3

ONOSArchitectureTiers

Northbound - Application Intent Framework (policy enforcement, conflict resolution)

OpenFlow NetConf . . .

Apps Apps

Distributed Core (scalability, availability, performance, persistence)

Southbound (discover, observe, program, configure)

Northbound Abstraction: - network graph - application intents Core: - distributed - protocol independent Southbound Abstraction: - generalized OpenFlow - pluggable & extensible

DistributedCore

Southbound

“Provision 10G path from Datacenter 1 to Datacenter2 optimized for cost”

Intents translated and Compiled into specific instructions for network devices.

ApplicaEonIntentFramework:AbstracEons,APIs,PolicyEnforcement,ConflictresoluEon

DistributedCore

SouthboundCoreAPI

OpenFlow NETCONF SouthboundInterface

ONOSApplicaEonIntentFrameworkIwanttodefine“what”Ineedwithoutworryingabout“how”…

DistributedCore:StateManagement

●  Topology●  Flows●  Intents●  Switchtocontrollermapping●  ResourceallocaEons●  NetworkConfiguraEon●  AndaplethoraofapplicaEon

generatedstate

DifferentstateshavedifferentsizeandR/WpakernsTheycanusedifferentconsistency,replicaEon,sharding

KeytoPerformance/Scalability:DistributedStateManagement

172.16.0.0 172.16.0.0

172.16.0.0 172.16.0.0

Application Intents - immutable - durable & replicated Global Network View - eventually consistent - fully replicated

Flow Table Entries - strongly consistent - partitioned

Optimistic Replication - gossip based - anti-entropy

Optimistic Replication - gossip based - anti-entropy - partial ordering

Master/Backup Replication

172.16.0.0 172.16.0.0

172.16.0.0 172.16.0.0

172.16.0.0 172.16.0.0

172.16.0.0 172.16.0.0

Switch è Master Switch è Master Switch è Master SwitchtoController-stronglyconsistent-replicatedfordurability

Consensusforstrongconsistency

Switch è Master Switch è Master Switch è Master ResourceAssignment-stronglyconsistent-parEEonedforscale

Consensusforstrongconsistency

ONOSPerformanceMetricsandResults●  Device&linksensinglatency

o  Lessthan100mso  ONOSprocessinglessthan10ms

●  FlowruleoperaEonsthroughputo  500Kto3Mops/sec

●  IntentoperaEonsthroughputo  200kops/sec

●  IntentoperaEonslatencyo  Lessthan50ms

SoORAN

GlobalNetworkView:eNBs+smallcelleNBs+signalstrengthsfromdevicesto

neighboringeNBs

So#RAN:So#wareDefinedRadioAccessNetwork

Macrocell

Picocell

Microcell Macrocell

PicocellMicrocell

So#RANOS

LTE-A LTE-IoT MVNO

Upto100Kmobiledevices;fewflowsperdevice

Asdevicesmove,needtohandover

50MacroeNBs,600MicroeNBs

Signalstrengthsconstantlychange–withinterference

Goal:Maximizeefficiencyofpreciousspectrum

Createpersonalizeduserexperience

Minimizecapex/opex

withrapidly

increasingdevicesandtheirbandwidth

requirements

CriEcaldecisions:Handover

DevicetoeNBassignmentPowerassignment

GlobalNetworkVieweNBs+smallcelleNBs+signalstrengthsfromdevicesto

neighboringeNBs

ControlPlaneRequirementsforSo#RAN

Macrocell

Picocell

Microcell Macrocell

PicocellMicrocell

So#RANOS

LTE-A LTE-IoT MVNO



AllocaEonof50resourceblockspereNBstomobiledeviceseveryms

SoORANKeyPerformanceRequirements

Control

Apps Apps

Global Network View / State Global View / State

HighVolumeofState:~500GB-2TBHighThroughput:~500K-5Mops/second~20Mstateops/secondLowLatencytoEvents:1-10sms

Difficultchallenge!Server

Server

Server

Storage

Orderofmagnitudehigherthroughputandlowerlatencyrequirement


OpportunityforParallelismStatemanagement/processingcanbedoneindependently–

notasEghtlycoupled

GlobalNetworkViewMulEpleTBs

Updatedeveryfewms

ControlPlaneChallengesforSo#RAN:SimplificaConandOpCmizaCon

So#RANOS

LTE-A LTE-IoT MVNO


OnlydevicesintransiConrequireimmediateprocessing

RAMCloudCoordinator

RAMCloudControlPlane

RAMCloudArchitecture:ControlPlane

Appl.Library

DatacenterNetwork

…1000–100,000ApplicaConServers

Appl.Library

Appl.Library

Appl.Library

MasterBackup

MasterBackup

MasterBackup

MasterBackup…

Coordinator

CoordinatorStandby

ExternalStorage(ZooKeeper)

1000–10,000StorageServers●  Meta-state:mappingofkeyspacetotablesonservers,listofacEveservers,

listofacEveclient-leases●  AppseverscachethestateandsocoordinatornotinthecriEcalpath

RAMCloudCoordinator:Requirements

Apps Control Apps

Global Network View / State Global System View / State

High Volume of State: ~500GB Medium Throughput: ~M updates per second Low Latency to Response: usec to msec

MasterBackup

MasterBackup

MasterBackup…

10,000Servers

…

100,000AppServers

Appl.Library

Appl.Library

Appl.Library

GeneralizedandCustomizableScalableControlPlane

Whatisdomainspecificandwhatisnt?

•  Domainindependent–  Statemanagementwithconfigurableconsistency,availability–  Streamingeventprocessing–  GraphabstracEons

•  Domainspecific–  Northboundandsouthboundinterfaces–  NoEficaEonAPIs


Northbound Absrtractions/APIs (C/C++, Declective Programming, REST) Strongly consistent, trasaction semantics?

OpenFlow/NetConf RAN X2 Interface RPC

Apps

Distributed Core Cluster of Servers, 10-100Gbps, low latency RPC

Distributed State Management Primitives

Southbound Plug-in for different contexts

NorthboundAbstracEon:-ProvidedifferentAPIs-CustomizeforthecontextCore:-distributed-contextindependentSouthboundAbstracEon:-Plug-insfordifferentcontexts

Server

Server

Storage

SwitcheseNBsServersStorage

Server

TimeVaryingStreamingGraphAbstracEon

•  CommonstateabstracEonformanycontrolplanesàGraph–  E.g:Innetworks,graphsofnodes(routers,basestaEons,phones)with

statechangingfrequently(linkload,cellconnecEvity)–  O(1million)nodesingraph,relaEvelysmallcomparedtowebgraphs

•  Updatestographsinstreamingfashion&highlydynamic–  E.g:Incellularnetworks,everyedgeupdatedevery3-5msà

O(2-5million)updateseverysecond

ComplexEventProcessing

•  ComputeabstracEon:Updatestographs(events)triggercomplexprocessingthat’sdomainspecific–  E.g:linkdowneventtriggeringarouteupdatetomaintainconnecEvity

•  TriggeredeventprocessingcanhaveEghtlatencyconstraints–  E.g:linkdowneventcancausedisconnecEon,needfastacEon

(~10-100ms)tokeepthenetworkconnected

•  Ideally,oneshouldbeabletowritetheeventlogicinahighleveldeclaraEvefashion

Summary•  SeparaEonofcontrolplaneisacommontrend:networks/systems

•  Wehavebeenbuildingcontrolplanesforourspecificsystems–  ONOS,SoORAN,andRAMCloudCoordinator

•  Canwedesignageneralizedscalablecontrolplanewith–  AcommonfoundaEonthatcanbecustomizedfordifferentcontexts

Backup


•  AcommonfoundaEonforallcontrolplanes–  Aclusterofcommodityservers–  Highperformancenetwork:10-100Gbps,speedoflightlatency–  LowlatencyRPC:~5usecendtoendlatency–  Customizeddistributedstatemanagement–  Differentconsistency,replicaEon,andshardingmodelsfordifferentstate

•  Customizableforspecificcontexts–  Allowthespecificsystemtopickdistributedstatemanagements–  AppprogrammingabstracEons/models/APIs

•  C/C++,DeclaraEveProgramming,RESTAPIs

–  Supportdifferentconsistencymodels•  Canwegetawaywitheverythingbeingstronglyconsistent?

Outline•  Examplecontrolplanes:requirementsanddesigns

–  ONOSforserviceprovidernetworks–  SoORANforcellularRANs–  CoordinatorforRAMCloud–  Otherexamples

•  Opportunityforaunifiedscalablecontrolplane–  Whatmakesusbelievethisispossible–  Whatarethechallenges

•  Highlevelarchitectureofaunifiedscalablecontrolplane–  Commonbuildingblocks–  CustomizaEon

•  Nextsteps

So#RAN:DataPlaneAbstracCon

•  Flow-graph:thepipeline/graphofpacketprocessingatomsthatneedtobeexecutedonalldata/packetsmatchingtheflow

•  Flowisflexiblydefined(e.g.IPflow,UEbearer,QoSclass)

•  AtomsaresimplepacketprocessingfuncEons(e.g.headercompression,segmentaEon,FFT)

•  Flow-graphscanhavelatencyandcomputethroughputrequirementsassociatedwiththem

A1

A2

A3

A4 A5

An

Match:IfPktàFlow1AcEon:ExecuteFlow-GraphF

Flow

-Graph

Fwith

throughp

utand

latency

requ

iremen

t

So#RAN:ControlPlaneAbstracCon•  Update/Reduceoncontrol-graphsthat

representnetwork/applicaEonstate–  Control-graphisdefinedflexibly(e.g

networkgraphwitheNBs,UEsandedgesannotatedwithlinkstrength,flowrecords,CDNgraphwithcontentnodes)

–  LocalupdatecomputaEonateachnodeinthegraph,messagepassingonedgesandglobalreducefuncEons

•  SimpleabstracEonissufficienttoexpressalargerangeofcontrolfuncEons

N1 N2 N3

N4 N6 N5

R

Update(NodeState) Update(NodeState) Update(NodeState)

Reduce(GlobalState)


DistributedMessagePassing

So#RAN:Expressingnetworkservicesusingflowgraphs&controlgraphs

A1

A2

A3

A4 A5

An

N1

N2

N3

N4

N6

N5

R


Reduce(GlobalState)


DistributedMessagePassing

SoORANOS(Compiler,scheduler&resourceorchestrator)

Macrocell

PicocellMicrocell Macrocell

PicocellMicrocell

So#RANOS:PlasormtoexecuteanycombinaEonofflowandcontrolgraphsexpressedinhighlevelprogramsonCOTSdistributedhardwarewhilemeeEnglatency&throughputrequirements

Data/ControlInterface

H1

H2

H3

H4

H5

H6

H7

H8

H9

H1H2H3H4H5H6 H7H8H9H1H2H3H4H5H6 H7H8H9

ControlPlaneforServiceProviderNetworks

ONOS•  Networkstate

–  1TB–  Changesmoreslowly

•  Controlops–  1-3Mops/sec

•  Latencyforprocessing–  10ms

•  Highavailability

So#RAN•  Networkstate

–  MulEpleTBs–  Changeseveryfewms

•  Controlops–  20Mops/sec

•  Latencyforprocessing–  1ms

•  Highavailability

SoORANoffersopportuniEesforsimplificaEonandopEmizaEons

RAMCloudControlPlane

RAMCloudArchitecture

Appl.Library

DatacenterNetwork

…1000–100,000ApplicaConServers

Appl.Library

Appl.Library

Appl.Library

MasterBackup

MasterBackup

MasterBackup

MasterBackup…

Coordinator

CoordinatorStandby

ExternalStorage(ZooKeeper)

1000–10,000StorageServers

High-speednetworking:•  5µsround-trip•  FullbisecEonbandwidth

CommodityServers

64-256GBperserver

ScalableControlPlane:BreakoutSessionSummary•  HeardposiEvecommentsabouttheoverallgoalsandproject

–  withareminderthatwearetakingonsomethingchallenging–  generalandcustomizableandhighperformancereobviouslyatodds

witheachother

•  Scalablecontrolplanesfornetworking—easiertoseefromIoTtoRANtoothernetworks–  Aretheyinadifferentcategorycomparedtoscalablecontrolplanefor

distributedsystems

•  Examplesofscalablecontrolplanesfordistributedsystems–  Schedulers:clustersandmulEpleclusters–  Schedulingofmicro-servicesorassociatedworkload

•  Whatisdomainorcontextspecificandwhatisdomain/contextindependent–  Distributedstatemanagementisdomainindependent–  ApplicaEonabstracEonsandAPIsandinterfacestodevicesbeingcontrolledare

domainspecific

•  Whatistherightmodelorparadigmforthedomainindependentpart–  Distributedstatemanagementbutthereareothermodels–  Emevaryingnetworkgraphasadatabaseandstandingqueries–  streamingdatabasewithabilitytocustomize:consistency,replicaEon,sharding–  Complexeventprocessingonstreamingdatabase

•  ApplicaEonabstracEonsandAPIs–  DeclaraEve/easytoprogram

Scalable Control Plane Substrate - platformlab.stanford.edu · MoEvaon • Separaon of control plane is a common trend: networks/systems – Challenging to design, hard requirements

Documents