1 Quality of Service vs. Any Service at All IWQoS 2005 Passau Germany Randy H. Katz Computer Science Division Electrical Engineering and Computer Science.

1

Quality of Service vs.Any Service at All

IWQoS 2005 Passau Germany

Randy H. KatzComputer Science Division

Electrical Engineering and Computer Science DepartmentUniversity of California, Berkeley

Berkeley, CA 94720-1776

2

Presentation Outline

• The Problem• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action

3



4

Some Observations

• Internet reasonably robust to point problems like link and router failures (“fail stop”)

• Successfully operates under a wide range of loading conditions and over diverse technologies

• During 9/11/01, Internet worked reasonable well, under heavy traffic conditions and with some major facilities failures in Lower Manhattan

5

The Problem

• Networks awash in illegitimate traffic: port scans, propagating worms, p2p file swapping

– Legitimate traffic starved for bandwidth– Essential network services (e.g., DNS, NFS)

compromised

• Needed: better network management of services/applications to achieve good performance and resilience even in the face of network stress

– Self-aware network environment– Observing and responding to traffic changes– While sustaining the ability to control the network

6

From the Frontlines

• Berkeley Campus Network– Unanticipated traffic surges render the network

unmanageable (and may cause routers to fail)– Denial of service attacks, latest worm, or the newest file

sharing protocol largely indistinguishable– In-band control channel is starved, making it difficult to

manage and recover the network

• Berkeley EECS Department Network (12/04)– Suspected denial-of-service attack against DNS– Poorly implemented/configured spam appliance adds to

DNS overload– Traffic surges render it impossible to access Web or

mount file systems

• Network problems contribute to brittleness of distributed systems

7

Why and How Networks Fail

• Complex phenomenology of failure• Recent Berkeley experience suggests that traffic

surges render enterprise networks unusable• Indirect effects of DoS traffic on network

infrastructure: role of unexpected traffic patterns– Cisco Express Forwarding: random IP addresses flood route

cache forcing all traffic to go through router slow path—high CPU utilization yields inability to manage router table updates

– Route Summarization: powerful misconfigured peer overwhelms weaker peer with too many router table entries

– SNMP DoS attack: overwhelm SNMP ports on routers– DNS attack: response-response loops in DNS queries generate

traffic overload

8

Possible Approach• New technology: packet flow manipulations at

L4-L7 made possible by new PNEs and stateful routers

• Enables identification/segregation of traffic– Good: protect it– Bad: block it– Suspicious: slow it

• Check/Observe/Protect Services (COPS)– Inspection-and-Action Boxes (iBoxes)– Annotation layer between routing and transport

• Yielding new service building blocks– Beyond packet marking and annotation …– To flow extraction and path-oriented statistics collection …– Based on traffic analysis, model extraction, statistical

correlation & causality testing

9



10

Edge Network

WideArea

Network

ServerServer

ServerServer

Managing Edge Network Services and Applications

• Not shrink wrap software—but cascaded “appliances”• Data Center in-a-box blade servers, network storage• Brittle to traffic surges and shifts, yielding network

disruption

FirewallIDSTrafficShaper

EgressChecker

LoadBalancer

Blades

Edge Network Middleboxes

11

Appliances Proliferate:Management Nightmare!

F5 Networks BIG-IP LoadBalancerWeb server load balancer

Packeteer PacketShaperTraffic monitor and shaper

Ingrian i225SSL offload appliance

Network Appliance NetCacheLocalized content delivery platform

Nortel Alteon Switched FirewallCheckPoint firewall and L7 switch

Cisco IDS 4250-XLIntrusion detection system

Cisco SN 5420IP-SAN storage gateway

Extreme Networks SummitPx1L2-L7 application switch

NetScreen 500Firewall and VPN

12

Network Support for Tiered Applications

LAN LAN LAN WideArea

Network

ServerAppTier

EgressChecker

ServerServerWebTier

ServerServer

ServerDatabaseTier

Firewall

LoadBalancer

Datacenter Network(s)

Unified LAN

WideArea

Network

ServerServer

ServerServerson

Demand

LoadBalancer

+Firewall

+Egress

CheckerBlades

ServerServer

ServerServerson

DemandConfigure servers, storage, connectivitynet functionality as needed

13

“The Computer is the Network”

• Emergence of Programmable Network Elements– Network components where net services/applications execute– Virtualization (hosts, storage, nets) and flow filtering (blocking,

delaying)

• Computation-in-the-Network is NOT Unlimited– Packet handling complexity limited by latency/processing

overhead– NOT arbitrary per packet programming (aka active networking)– Allocate general computation like proxies to network blades

• Beyond Per Packet Processing: Network Flows– Managing/configuring network for performance and resilience– Emergence of stateful routers for flow-based management– Adaptation based on Observe (Monitor), Analyze (Detect,

Diagnose), Act (Redirect, Reallocate, Balance, Throttle)

14



15

Check

• Checkable Protocols: Maintain invariants and techniques for checking and enforcing protocols

– Listen & Whisper: well-formed BGP behavior– Traffic Rate Control: Self-Verifiable Core Stateless Fair

Queuing (SV-CSFQ)

• Existing work requires changes to protocol end points or routers on the path

– Difficult to retrofit checkability to existing protocols without embedded processing in PNEs

– Develop building blocks for new protocols » Observable protocol behavior» Cryptographic techniques» Statistical methods

16

Observe

• Observation and Action Points– Network points where control is exercised,

traffic classified, resources allocated– In the datapath statistical collection +

annotating, prioritizing, shaping, blocking, …– Inspection-and-Action Boxes (iBoxes)

» Prototyped on commercial PNEs» Placed at Internet and Server edges of enterprise

net» Cascaded with existing routers to extend their

functionality» Migration into (some current and) future router

architectures

17

Protect

• Protect Crucial Services– Minimize and mitigate effects of attacks and traffic surges– Classify traffic into good, bad, and ugly (suspicious)

» Good: standing patterns and operator-tunable policies» Bad: evolves faster, harder to characterize» Ugly: cannot immediately be determined as good or bad

– Filter the bad, slow the suspicious, maintain resources for the good (e.g., control traffic)

» Sufficient to reduce false positives» Some suspicious-looking good traffic may be slowed

down, but won’t be blocked

18



19

iBoxes: Observe, Analyze, Act

AccessTier

AccessTier

AccessTier

R R

R R

I nternet orWAN Edge

RDistribution

Tier

I

EE

EE

EE

EE

EE

EE

UserEnd Hosts

ServerEnd Hosts

Network ServicesEnd Hosts

I

I

I R I

R

Enterprise Network Architecture

Inspection-and-Action Boxes:Deep multiprotocol packet inspectionNo routing; observation & markingPolicing points: drop, fence, block

20

Generic Network Element Architecture

InterconnectionFabric

Inp

ut

Port

s

Outp

ut

Port

s

Buffers

Buffers

Buffers

“Tag”Mem

CPCPCPAP

ActionProcessor

CPCPCPCP

ClassificationProcessor

Rules &Programs

21

RouterVM

• High-level specification environment for describing packet processing

• Virtualized: abstracted view of underlying hardware resources of target PNEs

– Portable across diverse architectures– Simulate before deployment

• Services, policies, and standard routing functions managed thru composed packet filters

– Generalized packet filter: trigger + action bundle, cascaded, allows new services and policies to be implemented / configured thru GUI

– New services can be implemented without new code through library building blocks

Mel Tsai

22

QoS ModuleQoS ModuleL2 Switching

Engine w/ ARP

L2 SwitchingEngine w/ ARP

IP Router Engine

IP Router Engine

ControlProcessor

ControlProcessor

Packet filter 1

Packet filter 2

Packet filter n

Default filter

EthernetPort

QoS ModuleQoS Module

Backp

laneBackp

lane



IP Router Engine

IP Router Engine

Packet filter 1

Packet filter 2

Packet filter n

Default filter

EthernetPort

ComputeEngine

ComputeEngine

ComputeEngine

Extended Router Architecture• Virtualized components representative of a

“common” router implementation• Structure independent of particular hardware

Virtual line card instantiated for every

port required by application

Virtual backplane shuttles packets between line cards

CPU handles routing protocols & mgmt tasks

Compute engines perform complex, high-latency processing on flows

Blue “standard” components

Yellow components added & configured per-application

Filters are key to flexibility

Mel Tsai

23

GPF “Fill-in” Specification

“Packet filter” as high-level, programmable building-block

for network appliance apps

FILTER 19 SETUP

NAME - SIP -

SMASK - DIP -

DMASK -PROTO -

SRC PORT -DST PORT -

VLAN - ACTION -

exampleany255.255.255.25510.0.0.0255.255.255.0tcp,udpany80defaultdrop

ClassificationParameters

Action

Traditional Filter

RouterVM Generalized Packet Filter (type L7)

24

GPF Action Language

• Basic set of assignments, evaluations, expressions, control-flow ops, “physical” actions on packets/flows– Control-flow: If-then-else, if-not– Evaluation: ==, <=, >=, !=– Packet flow control: Allow,

unallow, drop, skip filter, jump filter

– Packet manipulation: Field rewriting (ip_src == blah, tcp_src = blah), truncation, header additions

– Actions: NAT, loadbalance, ratelimit, (perhaps others)

– Meta actions: packet generation, logging, statistics gathering

• Basic Filter– Simple L2-L4 header classifications– Any RouterVM actions

• L7 Filter– REs, TCP termination, ADU recon

• NAT Filter– Capabilities beyond simple NAT action

available to all GPFs

• Content Caching– Builds on L7 filter functionality

• WAN Link Compression– Simple to specify, but requires lots of

computation

• IP-to-FC Gateway– Requires own table format & processing

• XML Preprocessing– Not very well documented, and

difficulty is unknown…

25



26

Network-LevelObserve-Analyze-Act

• Observe – Packet, path, protocol, service invocation statistical

collection and sampling: frequencies, latencies, completion rates

– Construct the collection infrastructure

• Analyze– Determine correlations among observations– “Normal” model discovery + anomaly detection– Exploit SLT

• Act – Experiment to test correlations– Prioritize and throttle– Mark and annotate– Control theory? Distributed analyses and actions

27

Network Layer Mechanism: Annotations

• Enhance network visibility: disseminate observations, communicate actions, provide in-band network management actions, iBox-to-iBox communications

• iBoxes label packets at annotation layer but do not rewrite packet contents

• Annotations stack, must be removed from packets before delivery to A-layer unaware end nodes

• Expose annotations to application layer?

Phy

Link

Network

Annotation

Transport

Session

Presentation

Application

28

Annotation Layer:Simple Marking Example

• Marking vs. rewriting approach– E.g., mark packets as internally vs. externally sourced using

IP header options

• Prioritize internal vs. external access to services solves some but not all traffic surge problems

iBoxBoundaryRouter

NetworkServices

iBoxI nternalRouter

Enterprise Network

External Traffi c

I nternal Traffi c

Packet

PacketLabel PacketLabel

PacketLabel

Action: Markpackets

Detect load and trigger action: Slow traffi c with “external” labels

29

Annotation Layer:iBox Piggybacked Control

Plane• Problem: Control plane starvation• Use A-layer for iBox-to-iBox communication

– Passively piggyback on existing flows– “Busy” parts of network have lots of control plane b/w– Actively inject control packets to less active parts– Embedded control info authenticated and sent

redundantly– Priority given to packets w/control when net under stress

• Network monitoring and statistics collection dissemination subsystem

30



31

R

R

DistributionTier

E

E

E

S

S

II

R IA

E

InternetEdge

AccessEdge

ServerEdge

SpamAppliance

Primary &Secondary

DNSServers

ISS

MailServer

S

Scenario: Traffic Surge Inhibiting Network Services

• DNS Server swamped by excessive request traffic– Observe: DNS time outs, Web access traffic slowed, but also

higher than normal mail delivery latency implying busy server edge (correlation between Mail Server and DNS Server utilization?)

– Root Cause: High DNS request rates generated by Spam Appliance triggered by mail surge

32

Scenario Continued

• How Diagnosed?– I-S detects high link utilization but abnormally high DNS traffic– Stats from I-I: high mail traffic, low outgoing web traffic, in

traffic high but link utilization not high– Stats from I-A: lower web traffic, no unusual mail origination– Problem localized to Server edge, but visibility limited

R

R

DistributionTier

E

E

E

S

S

II

R IA

E

InternetEdge

AccessEdge

ServerEdge

SpamAppliance

Primary &Secondary

DNSServers

ISS

MailServer

S

33

Scenario Continued

• Possible Action Responses– Experiment: Redirect local DNS requests to Secondary DNS

server: if these complete, can infer the server is the problem, not the network

– Throttle: Due to MS-DNS correlation, block/slow email traffic at Server Edge: should expect reduced DNS server utilization

R

R

DistributionTier

E

E

E

S

S

II

R IA

E

InternetEdge

AccessEdge

ServerEdge

SpamAppliance

Primary &Secondary

DNSServers

ISS

MailServer

S

34


• Problem and Approach• System and Network Trends• Checking-Observing-Protecting Services• Inspection-and-Action Boxes• Annotation Layer• Scenario• Call to Action

35

EdgeNetwork

System Perspective Needed!

DistributedMiddleware

Client

SLT Services DistributedMiddleware

Server

InternetIP Network

Router Router

EdgeNetwork

iBox iBox

Prototype ApplicationsProgrammingAbstractions

For Roll-back andwide-area distributed

computations

Crash-only services+ Observation

Infrastructure forSystem SLT

Checkable ProtocolsFast Detection &Route Recovery

ObservationInfrastructure for

network SLT

CommodityInternet

OperatorUser

Application-Specific

Overlay Network

36

iBoxes implemented on commercial PNEs

– Don’t: route or implement (full) protocol stacks

– Do: protect routers and shield network services

» Classify packets» Extract flows» Redirect traffic» Log, count, collect stats» Filter/shape traffic

Hope for EmergingPlatforms

37

Summary and Conclusions

• Processing-in-the-Network is real– Networking plus processing in switched and routed

infrastructures– Configuration and management of packet processing

cast onto PNEs (network appliances, blades, stateful routers)

• Needed: Unifying Framework– Methods to specify functionality and processing

» RouterVM: Filtering, Redirecting, Transformation» Map from policy intentions to network actions?» Local observations/correct global behavior?

• Application-specific network processing based on session extraction

38

Summary and Conclusions

• PNEs: foundation of a pervasive infrastructure for observation and action at the network level

– iBoxes Observation and Action points– Annotation Layer for marking and control

• Check-Observe-Protect paradigm for protecting critical resources when network is under stress

• Functionality eventually migrates into future generations of routers

– E.g., Blades embedded in routers

39

Quality of Servicevs.

Any Service at All

Randy H. Katz

Thank You!

40

1 Quality of Service vs. Any Service at All IWQoS 2005 Passau Germany Randy H. Katz Computer Science Division Electrical Engineering and Computer Science.

Documents

network slide

traffic overload slide

network trends

traffic analysis

illegitimate traffic

traffic changes

network infrastructure

network unmanageable