Ben Teitelbaum January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery QBone Service Specification and TE Early DiffServ Deployment Experiences.

Ben Teitelbaum <[email protected]>January 25th, 2001TEQUILA Workshop on Internet Design for SLS Delivery

QBone Service Specification and TEEarly DiffServ Deployment Experiences and Reduced Expectations

Service Specification and TE for the QBone—Amsterdam (January 25th, 2001) 2

“The Holy Grail of computer networking is to design a network that has the flexibility and low cost of the Internet, yet offers the end-to-end quality of service guarantees of the telephone network.”

- S. Keshav


The Paradox of Internet QoS

Internet’s success largely due to its lack of QoS!

Best-Efforts: scalability on steroids• Routers: complexity of forwarding very low • L2: minimal requirements• Providers: “magic”-free provisioning and low user expectations• Developers: E2E transport and simplicity of Berkeley socket

interface

QoS complicates everything (costs), but opens many opportunities (benefits)

• Developers: new apps exploit the ubiquity of the Internet• Providers: may make more efficient use of the network

Big Question: How can we build upon DiffServ without compromising scalability?


architecture

deployment

How We Got Here (short version)

• Began chanting: “enable advanced applications,…”

• Assessed requirements “What do you want?”“What can you give us?”

• Recommended DiffServ• Selected “Premium” service to meet demands of loss/jitter sensitive apps

• Charted QBone initiative • Specified QBone architecture• Now proceeding to implement it and tweak the architecture


QBone Architecture

A Service: QBone Premium Service• Built on Expedited Forwarding (EF) (RFC 2598)• Assurance: near-zero loss & low, bounded jitter for marked traffic conforming to a specified peak rate

–a.k.a. “virtual leased line”, “virtual wire”

Reservation Setup Protocol• Now: long-lived, manual setup• Proposed: SIBBS protocol between QBone domains; RSVP end-to-end between hosts

QBone Measurement Architecture• Uniform collection of QoS metrics• Uniform dissemination interface


QBone Measurement Architecture1/2

Collection metrics, EF and BE...• Active metrics (paths)

– One-way delay-variation

– One-way loss

– Traceroutes

– e.g. IPPM Surveyors

• Passive metrics (interfaces)– Load

– EF reservation load

– Discards (suggested)

– Link bandwidths (suggested)

– e.g. OCxMon, RTFM, MIBs

Boundary Router

Intra-Domain Premium Path

Inter-Domain Premium Path

ActiveMeasurements

MIB-basedstatistics

PassiveMeasurements

PassiveMeasurements

PM node PM node

AM node

QBone Domain2

QBone Domain1

QBone Domain3


QBone Measurement Architecture2/2

Dissemination • Standard URL query syntax:

• whois server to learn canonical names for QBone domains, routers, sniffers, etc

label ::= <alphanum> { <alphanum> }router ::= label"-ROUTER”probe ::= label"-PROBE”sniffer ::= label "-SNIFFER”host ::= router | probe | snifferpath ::= host "/" hostPHB ::= "BE" | "EF”metric ::= "LOSS" | "ONEWAY" | "PING" | "IPDV" | "LOAD" | "TRACEROUTE" | "COMMITMENT" | "RESERVATION"year ::= digit digit digit digitmonth ::= "01" | "02" | ... | "12”day ::= "01" | "02" | ... | "31”YYYYMMDD ::= year month dayaggregation ::= <unsigned_integer>prefix ::= “http” | “ftp” | <other>query ::= prefix ”://" path "/" PHB "/" metric "/” YYYYMMDD "/" metric "/" aggregation


Simple Interdomain Bandwidth Broker Signaling (SIBBS)

Basic Idea• Simple domain-to-domain reservation request/response protocol to

signal changes in the aggregate reservations of globally well-known services

Design Goals• Simple & extensible• Bootstrap wide-area experimentation with QoS-needy apps, while

providing hooks for clouds to experiment with various QoS TE approaches

• Integrate with end-to-end signaling capabilities of hosts

Status• Protocol draft nearing completion

(see: http://qbone.internet2.edu/bb/)• New protocol for now; looking at mapping to COPS


SIBBS: Basic Reservation Setup

BB

BB

BB

RTR ES RTR RTR RTR ES

RAR RAR

RAA RAA

Web interface RSVPDIAMETERCOPSRYO

CLISNMPCOPS


Virtual Peerings & Core Tunnels

BB BB BB SIBBS SIBBS

Src

RTR

RTR

Dest

“Virtual” SIBBS peering BB BB

“Core Tunnel”


SIBBS Approach to SLSs

Basic approach:• Define services and reservations, but punt on SLSs

Services • Globally specified, standardized, and named• Include many parameters from draft-tequila-sls (scope,

traffic envelope, performance guarantees, etc)

Reservations• Identify a service ID• Specify service parameters

SLSs• Each SLS along a reservation path should be regarded as a black

box that answers admissions questions and extends reservations across the peerings


Simple Interdomain Bandwidth Broker Signalling (SIBBS)

BB BB BB

H H

DS Domain1 DS Domain3DS Domain2

S LS1-2 S LS2-3

R RRR

Control

Data


QBone E2E Picture

GigaPoPA

CampusA

CampusB

GigaPoPB

CampusC

CampusD

Backbone

X Kbps of QPS from hither to………..yon

Key


Abilene TopologyJanuary, 2001

•47 connectors •183 participants

•34 connections to 20 peer networks


Miami

NYC (60 Hudson)

LA

AbileneSingaren

SEATTLE

NYC (Telehouse

25 Broadway)

STARTAP

ArgentinaBrazilChileColumbiaBrazil

Developing International Peerings

Courtesy: Linda Winkler, STAR TAP

CERNSINET

AbileneESnetvBNS

JAnetSURFnetNORDUnet

AbileneESnetCA*net3AARnetCERNET

CUDIHARNETSINET

AbilenevBNSESnetDRENNRENNISN

CA*net3SURFnetMIRnetNORDUnetREUNA

IUCCAPANCERNSINETSingarenTanet

GEMNETRenater

DFNINFNDANTECA*net3


Abilene Load Snapshot


APS Participation

Goal:• Make APS a reference implementation of the QBone architecture

Current Participants• MAGPI (U. Penn)• iCAIR• PSC (Penn State)• OARNet (Ohio State)• NASA EOS

Others in the wings• TF-NGN (through DANTE)• MIRnet• Various other international

• ANL• UIUC• DOE Science GRID (peering transit network)


Initial Engineering Plan (obsolete)

Measurement

Edge Policing

Manual Setup

EF Core Forwarding

Automated Setup

EF Edge Forwarding

Shaping

Sweetwater Midland Odessa Pecos

(GTS)

(Surveyor + SNMP + HTTP)

(“Firehose” CAR)

(Whiteboard + CLI)

(MDRR)

(MDRR)

(BB)


APS Measurements

Status: • Collecting EF/BE loads and CAR conform/exceed stats• Not currently monitoring IPDV, but Abilene Surveyor nodes now OC-3 connected and operational

• Ohio-ITEC hosting APS measurements and QBone-wide whois server

Near Future: • IPDV along edge-to-edge QBone paths• Abilene Surveyor timing improvements

–Better NTP–New CDMA timing sources (can't get GPS in Qwest POPs)

• Collection of AS-level traffic matrices


Committed Access Rate (CAR)

Classifies traffic based on certain matching criteria (interface, DSCP, or ACL) and meters it to a leaky bucket traffic profile

Depending on metering result, different actions applied (drop, transmit, set DSCP,…)

Syntax:rate-limit {input | output} [access-group [rate-limit] acl-index] bps burst-normal burst-max conform-action action exceed-action action


CAR Experience

For the most part, CAR is exactly what the DiffServ doctor ordered

However, there are some limitations…• PPS performance cost• Quirky constraints on token bucket depths• Not easy to do "virtual trunk" style classification

Problem: we want...

"virtual trunk""firehose"

But what we get is…


Abilene Architecture Limitation: “Porous” Edge Problem

DoQoS problem with current architecture

MDRR (EF forwarding) on interior interfaces easily subverted by unpoliced connectors


How to “Crisp” the Edge?

Problem: EF requires that all connectors be policed

Solution:• Short term: Stochastically detect illegal EF traffic with NetFlow and/or OCxMon passive monitoring; gigaPoP would be asked to police

• Longer term: Wait for E3 edge cards, deploy them aggressively, and police everywhere

• Road not taken:Re-write all non-participant traffic with DSCP 000000 (using PIRC hack); need to pass some DSCP values (reason why coming up…)


Looking Forward: Resource Accounting and Admissions

Problem: how to account for link EF capacity and commitment in routed network?

Potential solution: DiffServ-Aware MPLS-TE• Basic idea:

–Edge-to-edge MPLS tunnels–LSP setup constrained by available EF bandwidths–OSPF augmented to carry QoS link state attributes–See draft-lefaucheur-diff-te-reqts

• “Solves” DiffServ admissions problem• Does not solve general DiffServ provisioning problem• Status: completed extensive lab test of Cisco’s implementation of this idea; considering field trial


Current Engineering Plan

Measurement

Edge Policing

Manual Setup

EF Core Forwarding

Automated Setup

EF Edge Forwarding

Shaping

Sweetwater Midland Odessa Pecos

(GTS)

(Surveyor + SNMP + HTTP + whois + traffic matrices)

(CAR + QPPB)

(Whiteboard + CLI)

(MDRR)

(MDRR)

(SIBBS? + DSTE?)


QPS Status (Good News)

The Good News:• With enough elbow grease, E2E services can be built• DiffServ functionality in most modern routers• Many hosts now support QoS signaling (RSVP)• Numerous testbed trials• Partial implementations in ESnet and Abilene will form nucleus of QBone


SC2000 Interdomain QoS DemoNovember 6-9, 2000

Premium service over two wide-area paths• LBNL-ESnet-Abilene-SCinet-Internet2 booth• Stanford-CalREN2-Abilene-SCinet-Internet2 booth

Congestion induced at multiple points

CD-quality interactive audio application shown with/without QoS

SC2000 Network Challenge Winner: Award for "Most Captivating and Best Tuned" Demo



QPS Status (Bad News)

The Bad News:• Router vendor hype/reality mismatch• Deploying QPS “requires upgrading the world”• Low demand for QPS (app/net chicken/egg dynamic)• Elbow grease in short supply in most campus IT shops


Bottom Line

Progress on QPS will continue to be slow

Internet2 will continue with a QPS work program in 2001…

• Deploy incrementally where it counts• Revise/complete architecture (joint work with TF-NGN)• Work to build QoS-sensitive application user communities who

understand their needs

.However, QPS will no longer be the only Internet2 QoS activity

• Need to reap lower hanging fruit from DS (I2SS)• Much stronger push on measurements/monitoring• Big Internet2 E2E performance initiative ramping up


E2E Performance and Measurements

Internet2 E2E Performance Initiative• Typical BE TCP throughputs often much less than one would expect• Common problems

– Broken TCP stacks

– Ethernet auto-negotiation failures

• Evolving attributes of initiative– Performance Emergency Response Teams (PERTs)

– Sharper tools for measurement, monitoring, and analysis

Measurement arsenal• WEB100 (Mathis et al.)• Open Internet2 measurement architecture (à la QBone)• Open source and protocols for one-way delay measurement• Application-level performance fault-isolation (reflector infrastructure)


Internet2 Scavenger Service

Basic idea• A lower priority class of best-effort• Voluntary marking hints to network that degraded service is OK (think of a "nice" for the network)

Intended uses• Non-time-critical traffic (e.g. server-to-server NNTP, anonymous FTP, network backups)

• Bulk data transfers using TCP• Non-mission applications (e.g. Napster, games, etc)• New kinds of distributed applications that attempt to use idle network capacity


I2SS Service Specification

Rigorously defining the E2E I2SS is difficult!• Hoping to define relative to best effort

I2SS traffic indicated by DSCP 001000• Modification of class selector PHB• Note that the I2SS codepoint has global significance

I2SS domain requirements• Traffic leaving must be marked I2SS, if it entered so

Router requirements (all SHOULDs)• Forward I2SS independently giving it a lower probability of timely

forwarding OR forward in same manner as BE• Offer I2SS a very small minimum departure rate (unlike BH PDB)• Offer I2SS all un-used bandwidth


For more information...

Internet2 Home:• http://www.internet2.edu/

Internet2 QoS Working Group Home:• http://www.internet2.edu/wg/qos/

QBone Home:• http://qbone.internet2.edu/

Abilene Premium Service Home:• http://www.internet2.edu/abilene/qos/


Ben Teitelbaum January 25th, 2001 TEQUILA Workshop on Internet Design for SLS Delivery QBone Service Specification and TE Early DiffServ Deployment Experiences.

Documents

qbone premium service

qbone architecture

service id

architecture slide

service parameters slss

peerings slide

keshav slide

sibbs protocol