RouterVM A High-Level Programming Model and Virtual Machine Architecture for Next-Generation Programmable Routers Mel Tsai [email protected].

RouterVMA High-Level Programming Model and Virtual Machine Architecture for Next-Generation Programmable Routers

Mel [email protected]

2

OutlineThe Changing Landscape of RoutersTechnical goals of RouterVMThe RouterVM ArchitectureGeneralized Packet Filters

PropertiesFormalized ModelGPF Examples

Property Guarantees through RestrictionsExperimental Features

3

Changing Landscape of Routers

Today we see enhanced network functionality as an enterprise requirement

Increased security requirementsManaging and prioritizing trafficBalancing loadOffloading functionality that is too data intensive for servers

To serve these functions, recent hardware advancements allow application-level processing to be incorporated into router-like devices

Routers are no longer “dumb…” Hardware can support wire-speed packet classification, computation, and state management on thousands/millions of flows

4

New Requirements in the Enterprise

ISP

Edge Router

Firewall / VPN

Server LoadBalancer

IP StorageGateway

IntrusionDetection

Content Cache

LinkCompressor Switch

Switch

Switch

Switch

Switch

Server Blades

SAN

ClientWorkstations

200 Mbps

2.5 Gbps

1 Gbps

1 Gbps

1 Gbps

1 Gbps

1 Gbps

1 Gbps

1 Gbps

40 Mbps

Offsite 1-2.5 Gbps2.5 - 10 Gbps

5

New ChallengesHow does the vendor seamlessly integrate a large number of applications onto one device?

How to present an intuitive, unified management interface that properly hides the complexity of integrating multiple functions while simultaneously minimizing potential for errors?

How to achieve end-user flexibility and programmability?Vendors are reluctant to expose interfaces and allow open programmability of devicesCan customers implement new functions, collect new statistics, or reconfigure functions in ways for which the device was not intended?More generally, can programmability be achieved through a high-level interface, without requiring customers to write new code?

Can the vendor’s development framework and integrated application solution effectively target the underlying programmable hardware?

Network processors, ASICs, specialized function units, FPGAs

6

Technical Goals of RouterVM1. Support the “programming” of functionality through a high-level, expressive,

and functionally-complete building block called generalized packet filters (GPFs)

2. Show how an integrated management interface for GPFs on programmable hardware has the potential to reduce management complexity and minimize errors

3. To complement a set of standard out-of-the-box edge router functions, implement an example library of GPFs and show that it is representative of network routing and appliance applications

4. Formally and experimentally analyze the properties of the GPFs and the RouterVM execution model to understand its completeness, expressiveness, ease of specification, and other characteristics

5. Through formal analysis and by building prototype implementations, show that GPFs and the execution model can be effectively mapped onto existing and future programmable router hardware

7

Generalized Packet Filters (1)

GPFs are based on filters found in commercial routers

Packet filter 1

Packet filter 2

Packet filter n

Default filter

FILTER 19 SETUP

NAME - SIP -

SMASK - DIP -

DMASK -PROTO -

SRC PORT -DST PORT -

VLAN - ACTION -

exampleany255.255.255.25510.0.0.0255.255.255.0tcp,udpany80defaultdrop

ClassificationParameters

Action to bePerformed

…simple and easy to use, but not very powerful

Notice how a user configures relatively high-level parameters to specify the filter characteristics. New code and general programmability is not required, because in most cases, users of typical routers and switches don’t need much more flexibility here!

8


Key observation: the operation of packet filters can generalize to the following fundamental steps:

IP

TCP

HTTP

iSCSI

FCIP

MPLS

Ethernet

ATM

…?

Intrusion Detect

NAT

Store/Ret. State

TCP/IP lookup

Checksum

Count/Tag

…?

Error Detect

Drop

Route

Load Balance

Replace Fields

Resize Pkt

Encrypt

Forward

Compress

…?

ClassificationParameters

Infer based onpast observations Action(s)

What if all these options were made available in a packet filter?

9


While maintaining the parameterized and high-level specification interface of a normal packet filter, a GPF also has:

A widely expanded set of classification criteria

Normal header fieldsApplication-level headers such as URLsRegular ExpressionsMultiple combinations of the above

The ability to maintain information about packet flows

E.g., information sharing among GPFs

A widely expanded set of actionsAllow, drop, load balance, replace fields, encrypt, storage virtualize, NAT, compress, tag

The ability to implement arbitrary sequences of control flow among other GPFs

NAT, traffic shaping and monitoringL7 traffic detection (Kazaa, HTTP, AIM, POP3, etc.)QoS and packet schedulingIntrusion detectionSpam filteringProtocol conversion (e.g. IPv6)Content cachingLoad balancingRouter/server health monitoringStorage, Fibre Channel to IP, iSCSIXML preprocessingTCP offload (TOE)Encryption/compression, VPNsMulticast, Overlays, DHTs

10

GPF Example

Back

pla

ne

Back

pla

ne

ControlProcessor

ControlProcessor

QoS ModuleQoS Module

L2 SwitchingEngine w/ARP


IP Router Engine

IP Router Engine

GPF 5:

SLB

GPF 10:

P2P…

Servers

To Clients

A Server Load Balancer and L7 Traffic Detector

10.0.0.1

10.0.0.2

Ext. IP = 24.0.5.6

10.35.x.x

11

GPF Example

Back

pla

ne

Back

pla

ne

ControlProcessor

ControlProcessor




IP Router Engine

IP Router Engine

GPF 5:

SLB

GPF 10:

P2P…

Servers

To Clients


10.0.0.1

10.0.0.2

Ext. IP = 24.0.5.6

GPF 5 Setup

name -algorithm -

flowid -sip -

smask - dip -

dmask - proto -

action1 -action2 -action3 -

Server Load Balancerequal flowssip, sportanyany24.0.5.6255.255.255.255udp, tcpslb nat 10.0.0.1, 10.0.0.2log connections, file = log.txttag “skip Yahoo Messenger Filter”

10.35.x.x

12

GPF Example

Back

pla

ne

Back

pla

ne

ControlProcessor

ControlProcessor




IP Router Engine

IP Router Engine

GPF 5:

SLB

GPF 10:

P2P…

Servers

To Clients


10.0.0.1

10.0.0.2

Ext. IP = 24.0.5.6

GPF 10 Setup

name - type -

pattern -

timeout - flowid -

sip - smask -

dip - dmask -

proto - action1 -

action2 -

Yahoo Messenger Filteryahoomessenger^(ymsg|ypns|yhoo).?.?.?.?.?.?.?(w|t).*\xc0\x8010 minsip, dip, sport, dportanyany10.35.0.0255.255.0.0tcplimit 1 kbpsemail root 10.35.x.x

13

Formalized GPF Model

Forwardpacket flow

Redirectedpackets

Packetbuffer

Classify Infer

Packetmodification,

tagging,message

generation

External and/orshared state

InternalState

To downstreamcomponent

External computeengines

Config info Config info Config info

GPF statistics and status

Sche

dule

r

14

GPF Execution Sequences

L3Classifier




IP Router EngineIP Router Engine

RateLimiter

NetworkAddress

Translate

IntrusionDetection

LoadBalancer

?Many paths exist for packets…

15

Performance and Reliability Guarantees through Restrictions

Goal: be able to understand and analyze the formal properties of the RouterVM paradigm

Difficult with a fully general framework… Guarantees are more readily achieved by restricting the GPFs functionality and control flow flexibility

ExamplesExplicit declarations of shared tables and strict hardware enforcement of access and consistency

Logically or physically restricting packet buffers to one line card?

Allow only well-defined and small packet tags that makes only small adjustments to packet control flow

I.e. references to shared data structures, extracted fields, QoS priorities

16

Limits on Control FlowAllowing only forward jumps eliminates deadlock and livelock

Alternative might be implementing an in-router TTLBonus: achieves the property that multiple packets can be in-flight in the filter chain, but the processing is still analyzable and more-or-less deterministic

L3Classifier




IP Router EngineIP Router Engine

RateLimiter

NetworkAddress

Translate

IntrusionDetection

LoadBalancer

?

17

Limits on Regular Expressions

Complex regular expressions may be intractable on most hardware, and may not even be necessary for most apps

Limiting searches to simple/simpler expressions allows you to bound processing time

Complex examples:snmp:

^\\x02\\x01\\x04.+([\\xa0-\\xa3]\\x02[\\x01-\\x04].?.?.?.?\\x02\\x01.?\\x02\\x01.?\\x30)|(\\xa4\\x06.+\\x40\\x04.?.?.?.?\\x02\\x01.?\\x02\\x01.?\\x43)

Java DirectConnect:\\$mynick[\\x09-\\x0d -~]*\\|\\$lock[\\x09-\\x0d -~]*\\||\\$lock[ \\x09-\\x0d -~]*pk=[\\x09-\\x0d -~]*\\|\\$hubname[\\x09-\\x0d -~]*\\||\\$key[\\x09-\\x0d -~]*\\|\\$validatenick[\\x09- \\x0d -~]*\\|

18

Experimental FeaturesMore general and user-friendly interface to look-up tables and databases

Examples: Access to an email white list for a GPF-based spam filterList of “cachable” pages in a content cache

“Define your own” classification fieldA typedef for GPF classification fields?

Conditional actions, conditional classification fieldsConditional execution of one or more actions (potentially AND’ed / OR’ed together) based on classification results

“Switch” actions:A variety of actions can be performed based on a certain field, but you don’t need to write a filter for each caseAlso gives you a “default” action when nothing matches

19

Backup

20

Management ConcernsSeparate appliances create a management headache… If an organization uses 10 appliances, then

Network admins must learn 10 interfacesExpensive rack space is consumed by 10 devicesSoftware updates come from 10 sources under 10 service contractsPinpointing failures involves 10 devicesChanging the network topology involves reorganization of 10 devices

Separate appliances do not communicate or share informationBecause the applications are often similar, they duplicate work waste resources, and can interact with undesirable resultsE.g., firewall is accidentally configured to block “good” traffic that is destined to a load balancer

SNMP-based management approaches (e.g. HP OpenView) may not be enough

Still has the problem of duplicate resourcesCannot always trust vendor interoperabilityHardware does not have consistent capabilities, e.g. ability to report statistics and status

21

…No Existing Solution

Network Processor Support

Supports programmability without writing new code

Considers OtherProgrammable Hardware

Considers multiple apps running simultaneously

Management and usability is a first-class consideration

Dynamic reconfiguration support

Out-of-the-box edgerouter features

Ideas can be easily deployedinto existing routers today

Net

Bin

d

Clic

k

Scou

t

NEP

AL

Library or API based approach

Teja

IP F

abric

s

Act

ive

Net

wor

ks

Gen

esis

Ker

nel

Net

Kit

Inkr

a N

etw

orks

Rou

terV

M

XOR

P

Pron

to

partially or maybeyes

unknown or unable to comment?

?

?

?

?

?

??

?

???

? ??

Management, Configurability, and

Deployment Concerns for Future Programmable

Appliances

22

GPF Hardware Execution Model

QoS ModuleQoS ModuleL2 Switching

Engine w/ ARP

L2 SwitchingEngine w/ ARP

IP Router Engine

IP Router Engine

ControlProcessor

ControlProcessor

Packet filter 1

Packet filter 2

Packet filter n

Default filter

EthernetPort


BackplaneBackplane



IP Router Engine

IP Router Engine

Packet filter 1

Packet filter 2

Packet filter n

Default filter

EthernetPort

ComputeEngine

ComputeEngine

ComputeEngine

A virtual line card is instantiated for

every port required by the

application

A virtual backplane shuttles packets between line cards

A control CPU handles routing protocols and management tasks

When required, compute engines perform complex, high-latency processing on flows

Blue components are “standard” and are instantiated by

default. GPFs are configured and installed on a per-

application basis

23

A Mapping Example

SWITCHFABRIC

MANAGEMENTCPU

CPU1

PHY

MEM ASIC

CPU1

PHY

MEM ASIC

CPU1

PHY

MEM ASIC

CPU1

PHY

MEM ASIC

8-port RouterVM Configuration

Typical Switch Hardware Configuration

“Mapping distance”is minimized due tostructurally-parallel

organization of RouterVM

24

Limited Hardware Resources

Hardware is not infinitely fast with infinite resources; it is generally limited in the following ways:

Number and type of computational unitsAvailable parallelism (separate CPUs, hardware threads, SIMD, VLIW, etc.)Special-purpose engines (search coprocessors, table lookup, hash units)Communication bandwidth (between computation units, between units and memory, between memory and I/O)Memory (SRAM, SDRAM, local registers, cache)Memory controllers (SDRAM controllers, DMA)

Certain GPFs, and even configurations of GPFs, may not effectively map to certain hardware at the required performance

Future work is to determine where the overheads are, how to determine whether performance can meet hardware abilities, etc.Open question: when and how does RouterVM inform the user when performance cannot be met? By what mechanism does it determine this?

25

Mapping to NPU Linecards

OC-192 NetworkProcessor

SDRAM

Traffic Mangr.

Line card A

<>

GbEMAC

GbEPHY

ChProc

Backplane

SwitchFabric

<>

GbEMAC

GbEPHY

<>

GbEMAC

GbEPHY

<>

GbEMAC

GbEPHY

SearchCoproc

TCAM

SerDes

SerDes

Line card B

SerDes

ChProc

SerDes

10/1

00

Octa

l MA

C1

0/1

00

Octa

l MA

C

NetworkProcessor

X-Bar

SRAM

SDRAM

NetworkProcessor

SRAM

SDRAM

GPF1 GPF2 GPF3 GPF4 GPF5

26

Enriched Features through Packet Tagging

Tags are an interesting way to enhance the functionality of GPFs

Tags can contain extracted link layer, IPv4, TCP, iSCSI, or HTTP fieldsA different way of transferring state…Useful e.g. when hardware filters perform extraction, while software-only filters use it downstream

Tags can be mini-instructions that affect control flow:“This packet is special, ignore downstream filters of type {X, Y, and Z}”“This packet is low-priority, downstream filters can drop this if resources are low”“Ignore this packet’s L2 header and route only based on its IP header”

Tags can contain application-specific data such:Useful statisticsSCSI block numbers in IP storageFlags for an out-of-order streamSymbolic references to downstream router resources

27

28

Optimal Mapping to Hardware

ApplicationsApplicationsArchitecture Instance

Mapping

Applications

PerformanceAnalysis

PerformanceNumbers

Suggest architecturalimprovements

Rewrite theapplications

Use differentMapping strategies

The MESCAL Y-chartSource: unknown member

of the Mescal team

RouterVM A High-Level Programming Model and Virtual Machine Architecture for Next-Generation Programmable Routers Mel Tsai [email protected].

Documents

gbps slide

new functions

routervm execution model

devices routers

new requirements

dumb hardware

box edge router functions

highlevel programming