Network Disaggregation Ronald van der Pol SURFnet Routz SharePlus, 6 February 2017, Diemen
NetworkDisaggregation
RonaldvanderPolSURFnet
RoutzSharePlus,6February2017,Diemen
Overview
• Whatisnetworkdisaggregation?• OpenComputeProject- Networking• demo• OpenFlow,PipelinesandSilicon• TableTypePattterns (TTPs)andP4• demo• Summary
RoutzSharePlus,6February2017,Diemen
Vertically integratedClosed, proprietary
Slow innovationSmall industry
SpecializedOperatingSystem
SpecializedHardware
AppAppAppAppAppAppAppAppAppAppApp
SpecializedApplications
HorizontalOpen interfacesRapid innovation
Huge industry
Microprocessor
Open Interface
Linux MacOS
Windows(OS) or or
Open Interface
(slidebyNickMcKeown,StanfordUniversity)
Vertically integratedClosed, proprietary
Slow innovation
AppAppAppAppAppAppAppAppAppAppApp
HorizontalOpen interfacesRapid innovation
ControlPlane
ControlPlane
ControlPlane or or
Open Interface
SpecializedControlPlane
SpecializedHardware
SpecializedFeatures
MerchantSwitching Chips
Open Interface
(slidebyNickMcKeown,StanfordUniversity)
NetworkDisaggregation
• Bestofbreedinhardwareandsoftware• OpenAPIs• OpenHardware• User/operatorincontrol
– Lessdependentonvendorroadmaps– Implementandexperimentwithnewprotocols
RoutzSharePlus,6February2017,Diemen
BenefitsofDisaggregation
• From– Closedvendorproprietaryall-in-oneblackboxes
• Vendordecideswhichfeaturestosupport• Vendordecideswhenthesefeaturebecomeavailable• Dependentoninnovationstrengthofvendor
• TO– Whitelabelswitcheswithdisaggregatedfirmware
• Choosebestvendorforhardware• Choosebestvendorforfirmware• Useopensourcefirmware(communitycontroloverfeaturesandinnovation)
RoutzSharePlus,6February2017,Diemen
Computingvs Networking
ClosedSystems
OpenHardwareOpenSoftware
1970 1980 1990 2000 2010
ClosedSystems
closed
open
ClosedhardwareOpenFlow APIOpenSourceApplicationsPortableapplications
OpenHardwareOpenFirmware
ClosedhardwareWorkstations+UNIXUNIXSystemCallAPIStartofOpenSourceSoftwarePortableapplications
SlidebyR.vanderPol
OpenComputeProject- Networking
• ONIE(switchbootloader)• SAI(SwitchAbstractionInterface)• OpenNetworkLinux(ONL)• SoftwareforOpenNetworkingintheCloud(SONiC)
RoutzSharePlus,6February2017,Diemen
ONIEBootloader
RoutzSharePlus,6February2017,Diemen
OpenNetworkLinux
RoutzSharePlus,6February2017,Diemen
ADDITIONAL / FUTURE APPLICATIONS HERE
ONL: THREE+ FORWARDING AGENTS; 20+ PLATFORMS
8 © 2015 BIG SWITCH NETWORKS, INC.
CPU (PowerPC, x86)
Miscellaneous Hardware (Fans, LED controllers, SFP sensors, Power…)
Packet Forwarding Chip (ASIC)
OCP Switch Hardware (Facebook Wedge, IM Niagara, Accton 6712, 7512, Dell S6000-ON, Quanta LY6, ...)
Broadcom SDK (others coming soon)
OF-DPA
Platform Specific ASIC Drivers
Open NSL
SAI Interface
ONL Linux Kernel (Includes extra drivers: I2C, MUX, management Ethernet, …)
Open Network Linux Platform Abstraction Layer (Platform specific drivers including Optics)
Installer (ONIE)
FORWARDING AGENT PROGRAMMING APIS
Indigo OpenFlow
Agent
Facebook FBOSS
HA
RD
WA
RE
PLA
TFO
RM
Platform Driver
Microsoft SONiC
SONiC
SWITCH(HOST)
RoutzSharePlus,6February2017,Diemen
OpennetworkingHardware
BaseOSandinstaller
H/WProgramminginterface
NetworkSoftwareComponentsNoOCPSolutionAvailable
SwitchAbstractionInterfaceSAI
OCPHardware
OpenNetworkLinuxONL
OpenNetworkInstallEnvironment
ONIE
SoftwareforOpenNetworkingintheCloudSONiC
CumulusNetworks
• CommercialL2/L3• HardwareCompatibilityList:
– Dell,EdgeCore,Facebook,HPE,Mellanox,Penguin,QCT,Supermicro,Agema
RoutzSharePlus,6February2017,Diemen
Pica8
• PicOS commercialL2/L3andOpenFlow• HardwareCompatibilityList:
– EdgeCore,HPE,Inventec,InterfaceMasters,Pica8,Dell,Penguin,QCT
RoutzSharePlus,6February2017,Diemen
OpenNetworkLinux
• OpenSourceL2/L3andOpenFlow• HardwareCompatibilityList:
– QuantaMesh,Accton/EdgeCore,DNI/Agema,Dell,InterfaceMasters,Mellanox
RoutzSharePlus,6February2017,Diemen
OpenSwitch
• OpenSourceL2/L3andOpenFlow• Switchplugins:
– OVS– OpenNSL (Broadcom)– P4– XPliant– SAI– Marvell
RoutzSharePlus,6February2017,Diemen
OpenSwitch Architecture
Virtual Interfaces I2C/other drivers SDK driver
ops-powerd
ops-tempd
ops-fand
SDK
SDK specific plugin
SDK independent layer
OpenFlow
sflow
OVSDB
CLI, REST, Ansiblesystem daemonsL2/L3 daemons
RFC 7047
KERNEL
RoutzSharePlus,6February2017,Diemen
Demo
RoutzSharePlus,6February2017,Diemen
OpenFlow
• OpenFlow givesuser/operatordirectaccesstoflowforwardingtables
• OpenFlow providesMatch/Actionsemantics• Supportedonmanyhardwareswitches
– PureOpenFlow switches– Hybridswitches(conventionalswitchadd-on)
• Many(opensource)controllerplatforms• OpenFlow startedthenetworkdisaggregationefforts
RoutzSharePlus,6February2017,Diemen
SoC ASICbasedOpenFlow Switches
• ManybasedonBroadcomASICs(e.g.Trident)• Onlyasmallfixedamountoflookuptables
– TCAM(wildcardentries,ACLs)– MACForwardingDatabase– L3longestprefixmatchtable– L3hostroutes
RoutzSharePlus,6February2017,Diemen
OpenFlow 1.3MultipleTables
• Preventflowentryexplosion• Multi-tablepipeline
table0
table1
tablen
ExecuteAction
Set
packet in packet outIngressPort
ActionSet = {}
ActionSet
Packet +IngressPort +
metadata Packet
ActionSet
RoutzSharePlus,6February2017,Diemen
MappingofFlowTables
ExecuteAction
Set
packet in packet outIngress
Port
ActionSet = {}
ActionSet
Packet +IngressPort +
metadata Packet
ActionSet
ingressport
packet in IPtable
MACtable
ExecuteAction
Set
egressport
packet out
OpenFlow 1.3 Pipeline
Broadcom Pipeline (simplified)
table0
ingressTCAM
based on Pica8 documentation
table1
tablen
RoutzSharePlus,6February2017,Diemen
BroadcomTridentII
• ThereisverylittlepublictechnicalinformationbecauseofBroadcom’sNDA
• SeveralTCAMs,L2,L3,LPMtables• UnifiedForwardingTable(UFT)memorybankscanbeallocatedto:– L2entries– ARPentries– L3LPMentries– ExactmatchACLentries
RoutzSharePlus,6February2017,Diemen
BroadcomTridentIIUFTBANK SIZE
0 4Kx420bits
1 4K x420bits
2 16Kx420 bits
3 16Kx420 bits
4 16Kx420bits
5 16Kx420bits
6 1Kx420bits
7 1Kx420bits
8 1K x420bits
9 1Kx420bits
RoutzSharePlus,6February2017,Diemen
} DedicatedL2MAC(32Kx105bits)
} SharedEntries(256Kx105bits)
} DedicatedL3hostentries(16Kx105bits)
TridentIIUFTCombinationsMode L2 L3hosts LPM
0 288K 16K 0
1 224K 56K 0
2 160K 88K 0
3 96K 120K 0
4 32K 16K 128K(77K– IPv6)
RoutzSharePlus,6February2017,Diemen
LimitationsofSoCASICs
• Fixedsemanticstables(L2,L3,LPM,TCAM)• Fixedsizetables(orlimitedresizing)• Norecirculationofpackets(onepassthroughpipeline)
RoutzSharePlus,6February2017,Diemen
ASIC/OpenFlow LimitationExamples
• LimitationofSoC ASICs– OpenDaylight ServiceFunctionChaining(SFC)projectconfiguresmultipletables
– Theseendupin1TCAManddoesnotwork– Result:genericapplicationslikeODLSFCcannotbeused;applicationneedstobeadaptedtoASIC
• LimitationofOpenFlow– StilldependenceonSDOsandvendorsfornewencapsulations/protocols
– WewanttoexperimentwithNetworkServicesHeader(NSH),butnosupportinOpenFlow
RoutzSharePlus,6February2017,Diemen
ProgrammableNetworkSilicon• FPGAs(FieldProgrammableGateArrays)+TCAM+DDR– Corsa DP2200*)
• NetworkProcessors(NPUs)+TCAM+DDR– NoviFlow NS2128*)
• FlowProcessor– Netronome NFP-4000*)
• ProgrammableSwitchSilicon– CaviumXpliant,BarefootTofino
*)presentinSURFnet testbed
RoutzSharePlus,6February2017,Diemen
Corsa (FPGA/TCAM/DDR3)
RoutzSharePlus,6February2017,Diemen
CONFIDENTIAL 10
Network Hardware Virtualization
Multiple SDN applications controlling Virtual Switches
Overlay
Hardware Resource Pool
Single piece of hardware
OpenFlow SDN App L3 Routing App
OpenFlow Switch
Any OpenFlow match ANY RateANY PortAny Protocol
L3 Routes + ACLsANY RateANY PortAny Protocol
Underlay
Optimized L3 Router
NoviFlow NS2128
SerD
es
MAC
s
TOPPacket Processors &
Search EnginesTraffic
Manager
OAM
Internal Memory
Classification &Queuing
Internal TCAM
SDRAM DDR3 Controllers
RoutzSharePlus,6February2017,Diemen
Mellanox EZchip NP-5
NoviFlow PipelineConfiguration
• Setconfig pipeline<id><size><width><type>– <type>isexact(DDR)orwildcard(TCAM)– Default
• 28wildcard+28exacttables• 4096rows• 40bytewide
RoutzSharePlus,6February2017,Diemen
PipelineAbstractions
• Flexibleprogrammablepipelinesneedanabstractiontodescribethem
• Twopopularapproaches:– TableTypePatterns(TTP)– OpenFlow pipelines– P4(ProgrammingProtocol-IndependentPacketProcessors)
• Bothcanbeusedto– Lettheswitchadvertiseitssupportedpipeline(s)– Telltheswitchwhatpipelinetoconstruct
• P4ismuchmorepowerfulthanTTPs
RoutzSharePlus,6February2017,Diemen
TableTypePatterns(TTPs)
• ATTPisanabstractmodelthatdescribes(inJSONsyntax)theforwardingbehaviour– Descriptionofflowtables– Descriptionofvalidflow_mods,group_mods andmeter_mods
• SwitchandcontrollermaysupportmultipleTTPs
• Atstartup thereisanegotiationbetweenswitchandcontrolleraboutwhichTTPtouse
RoutzSharePlus,6February2017,Diemen
P4Language• P4:ProgrammingProtocol-IndependentPacketProcessors
• DomainSpecificLanguageforprogrammabledataplanes
• P4programà P4compilerà targetcode
• TargetcodeisloadedonP4switch• Consistsofpacketparserandlookuptables
RoutzSharePlus,6February2017,Diemen
P4Switch
INPUT
MatchAction
MatchAction
OUTPUT
ParseGraph
ControlProgram
Match+ActionTable Config
Ingress Egress
Switch Configuration
P4 Switch
PARSER
Queuesand/orBuffers
Source: The P4 Language SpecificationVersion 1.0.2
RoutzSharePlus,6February2017,Diemen
ExampleP4HeaderDefinitions
header_type ethernet_t {fields{dstAddr :48;srcAddr :48;etherType :16;
}}
header_type ipv4_t{fields{version:4;ihl :4;diffserv :8;totalLen :16;identification:16;flags :3;fragOffset :13;ttl :8;protocol:8;hdrChecksum :16;srcAddr:32;dstAddr:32;
}}
RoutzSharePlus,6February2017,Diemen
ExampleP4Parserparserstart{returnparse_ethernet;
}
parserparse_ethernet {extract(ethernet);returnselect(latest.etherType){ETHERTYPE_IPV4:parse_ipv4;default:ingress;
}}
parserparse_ipv4{extract(ipv4);returningress;
}
RoutzSharePlus,6February2017,Diemen
P4SupportedTableTypes• Exact:value==tableentry
– E.g.IPv4hostroute• Ternary:valueANDmask==tableentry
– Wildcard• LPM:LongestPrefixMatch
– Specialcaseofternary(1111….11110000.....0000)• Range:lowentry<=value<=highentry• Valid:tableentry={true,false}
– True:headerfieldisvalid– False:headerfieldisnotvalid
RoutzSharePlus,6February2017,Diemen
P4SupportedChecksumAlgorithms
• XOR16
• CSUM16
• CRC16
• CRC32
• Programmable_CRC– ArbitraryCRCpolynomial
RoutzSharePlus,6February2017,Diemen
AdditionalP4Features• Counters
– Type:bytesorpackets– Min-width– Saturating:stopcounting;defaultiswrap
• Meters
• Registers
• Resubmit(originalpacket+metadata)
• Recirculate(packetafteregressmodifications)
RoutzSharePlus,6February2017,Diemen
P4ControlFlow• If/else
• +,*,-,<<,>>,&,|,^
• ~,-
• OR,AND
• >,>=,==,<=,<,!=
RoutzSharePlus,6February2017,Diemen
WorkFlow
• WriteP4program,typicallythesesourcefiles:– foo.p4– headers.p4– parser.p4
• ConvertP4programtoJSONconfiguration
• LoadJSONconfigurationonP4switch
RoutzSharePlus,6February2017,Diemen
Demo
RoutzSharePlus,6February2017,Diemen
Summary
• OpenFlow startedthenetworkingdisaggregation
• Manycompanieshavejoinedthenetworkingdisaggregationefforts
• Manyopenhardwarevendors• SeveralcommercialNOS’savailable• SeveralopensourceNOS’savailable
RoutzSharePlus,6February2017,Diemen