Policy-based QoS Management on the Internet2 QoS Backbone (QBone) Dr Tham Chen Khong, Michael Yuan Lihua, Zhang Yi, Mayank Agarwal, Paul Ng Dept of Electrical & Computer Engineering National University of Singapore SingAREN Broadband 21 Seminar
Policy-based QoS Management on the Internet2 QoS Backbone (QBone)
Dr Tham Chen Khong, Michael Yuan Lihua, Zhang Yi, Mayank Agarwal, Paul Ng
Dept of Electrical & Computer EngineeringNational University of Singapore
SingAREN Broadband 21 Seminar
Motivationn Set up infrastructure for
inter-domain end-to-end QoS(QBone: Internet2-SingAREN-NUS),including measurement infrastructure
n Deploy high-bandwidth and time-sensitive applications like video streaming and VoIP
n Develop and deploy QoS management functionalityn Resource allocation in local domain coordinated by
Bandwidth Brokersn Policy-based QoS managementn Accounting
Overviewn Need for Quality of Service (QoS)n Differentiated Service (DiffServ)n Internet2 QoS Backbone (QBone)n QBone Measurement Architecturen Inter-Domain QoS Management
n Service Level Agreements (SLAs)n Bandwidth Broker
n NUS (CCN-CIR)-SingAREN-QBone project with iCAIR, NWU/IBM
n Policy-based QoS Managementn Reservations using Bandwidth Brokers and
signallingn Future Work
Need for Quality of Servicen Applications that need QoS
n multimedia, real-time control, distributed interactive simulation (DIS), distributed computing, real-time information transmission
n Common QoS Parametersn Throughputn Delayn Delay variationn Loss rate
Differentiated Servicesn Goal: Provide preferential service without
increasing overhead in core routersn Scalable service differentiation in the Internet:
aggregate flows in the core of the network
n Each core and edge router has a set of pre-defined Per Hop Behaviors (PHBs)n Expedited Forwarding (EF)/Virtual Leased Line
(VLL) n Assured Forwarding (AF)n [mechanisms: PQ, WFQ, CBQ, CB-WFQ, WRR]n Best Effort (BE)
DiffServ Architecture
AppBB
App
BB
Bandwidth Brokers(perform admission control, manage network resources,
configure leaf and edge devices)
Leaf Router(police, mark flows) Egress Router
(shape aggregates)
Ingress Router(classify, mark, police)
Source
Destination
DiffServ Architecturen Traffic Classification and Conditioning
n Classifiersn Traffic conditioners
n Meter, Marker, Shaper/Dropper
n Location of traffic conditionersn Leaf, ingress, egress nodes
Meter
packets Classifier Marker Shaper/Dropper
Differentiated Servicesn Data packets are classified and are marked at the leaf
router by having their DS Code Point (DSCP) set to indicate the required PHB
n Packet classification, marking, policing and shaping need only be implemented at the network boundary or hosts
n Can have multiple DS-domains with inter-domain negotiation conducted by Bandwidth Brokers
DiffServ terminologyn Per-Hop-Behavior (PHB)
n PHB+policy rules=range of servicen Codepoint (in header of IP packet)
n Codepoints-->PHB mapping (e.g. 101110 for EF)n Service Level Agreement (SLA)
n service contract between customer and service provider that specifies the forwarding service a customer should receive; may include traffic conditioning rules
n Traffic Conditioning Agreement (TCA)n agreement specifying classifier rules and any corresponding traffic
profiles and metering, marking, discarding and shaping rules which apply to traffic streams selected by the classifier
0 1 2 3 4 5 6 7D S C P C U
Bandwidth Brokersn Manages the QoS resources within a given domain
n Performs admission control and provisioning/manage domain resources
n Configure leaf and edge routersn Manages the inter-domain communication with adjacent
regions’ BBsn Enforcement of Service Level Agreements (SLAs)
n Communicate with policy manager and perform subset of policy management
BB1
endusers
RARs
AS1
BB2 BB3
SLDsSLDs
AS2 AS3
Inter-domain Communication
Intra-domain Communication
edgerouter
Internet2 – vBNS (Backbone)
Internet2:A university-led effort involvingacademia, government & industry
Note: GigaPOPs connectedby High Speed ATM Backbone
UCAID Abilene POS Network
The new preferred backbone networkOC48 (2.4 Gbps) trunks
Real Time Measurements on Abilene
Note: vBNS and SingAREN peer at STAR TAP. QBone peering ON with 1Mbps CBR initially.Abilene and SingAREN peer at Indianapolis. Will soon have Abilene-SingAREN QBone peering.
Internet2 QBone Initiativen Build inter-domain testbed infrastructure
n Experiment and improve understanding of DiffServn Incrementally improve testbed
n Support intradomain & interdomain deploymentn Lead and follow IETF standards work
n Some parts of DiffServ architecture mature; others far from it
n Experience gained will inform standards processn Openness of R&E community makes it easier
n Users will tolerate the flakiness of an experimental infrastructuren Engineers will share experience and measurement data
QBone Architecture (High Level)n IETF “Diff” (EF PHB) + QBone “Serv”n QBone Premium Service (QPS)
n Use Van Jacobson VLL “Premium” servicen Well-defined SLS:
n Peak rate R & “Service MTU” M implying a token bucket metern Near-zero lossn Low jitter
n Delay variation due to queuing effects should be no greater than the packet transmission time of a service MTU-sized packet (provided reserved interdomain route does not flap)
n QPS has been implemented in core Abilene and ESnetn Plus important value-adds:
n Integrated measurement/dissemination infrastructuren Experimentation with pre-standards inter-domain bandwidth
brokering and signaling
QBone B/W Broker architecturen Based on Simple Interdomain Bandwidth
Broker Signaling (SIBBS)n an end-to-end signaling architecture for QBonen product of Bandwidth Broker Advisory Council
(BBAC)n Define BB model to be used in the QBonen Recommend deployment phasing
(Phase 0, 1, 2)n Specify common inter-domain protocol
Reservationsn Reservation Setup Protocol
n Reservation Request {source, dest, route, startTime, endTime, peakRate, MTU, jitter}
n Now: long-lived manual setupn Proposed: Simple Interdomain Bandwidth Broker
Signaling (SIBBS) protocol between QBone domains, RSVP end-to-end between hosts
Inter-domain Communication (SIBBS)n Simple Inter-domain Bandwidth Broker
Signallingn TCP is transportn Fundamental messages:
n Resource Allocation Request (RAR)n Resource Allocation Answer (RAA)
n Simple request-response protocoln Requires some basic authenticationn Supports setup, modification, takedownn Intended to be flexible and extensible
Draft SIBBS RAR/RAA
SIBBS version RAR ID
Sender ID Sender Signature
Source Prefix Destination Prefix
Ingress Router ID Start Time
Stop Time Flags
GWS ID Service Paramter TLV
Other (Optional) TLVs
Sample Setup
BB
BB
BB
RTR ES RTR RTR RTR ES
RAR RAR
RAA RAA
From end system to end system, e.g. microflows
Policy-based Networking
Directory Serverwith Policy-relatedData
PolicyServer
Other (unspecified)policy-relevantInformation Servers(e.g. time, certificates)
...Host
Information S
tores
PolicyTransaction
Protocol
DirectoryAccess
Protocol
QoSSProtocol
LeafDevice
(Access Router)
Directory Server
LDAPServer
DirectoryAccess
Protocol
(LDAPv3)
Policy-related Data
... Other InformationServers
PolicyTransaction
Protocol
(e.g. COPS,DIAMETER)
Policy Server
LDAPClient
PolicyInterpreter
PTServer
Host
QoSSClient
Policy-ControlledApplication
Access Device
PTClient
PolicyEnforcer
QoSSServer
SignallingProtocol
(e.g. RSVP)
Other
Access Protocols
Policy Mgmt
PBNM researchn Traffic Engineering for Quality of Service in
the Internet, at Large Scale (TEQUILA) project (Europe) (www.ist-tequila.org)
n Objectives: n study, specify, implement and validate a set of
service definition and traffic engineering tools to obtain quantitative end-to-end QoS guarantees through careful planning, dimensioning and dynamic control of scalable and simple qualitative traffic management techniques within the Internet (i.e. DiffServ).
Commercial offeringsn Orchestream, Allot, Cisco etc.
Edge Router
Edge Router
Edge Router
BandwidthBroker
Pol icy Server
Policy Database
Policy Manager
MySQL• PIB• High Level Policy
GUI Java Based
SQL
SQL
RARCOPS
Policy Based Network Management
DiffServ QoS PIBØ A Traffic Control Block (TCB) in the DiffServ interface
consists of zero or more classifiers, meters, actions, algorithmic droppers, queues and schedulers
Ø The PIB models the individual elements that make up the TCBs
Ø Designed for functional abstractions rather than device implementations (very flexible but hard to directly map to a proprietary device)
draft-ietf-diffserv-pib-06.txt, March 2002
QoS DiffServ PIBn Data Path Tablen Classifier Tablen Meter Tablen Action Tablen Queue Tablen Scheduler Tablen Algorithmic Dropper
Table
n Classification Capabilities Table
n Metering Capabilities Table
n Scheduling Capabilities Table
n Element Depth Capabilities Table
n Element Link Capabilities Table
Policy Managern Defines the policy condition and action:
high level and PIBn Control the deployment of a policy on an
interface or a group of interfacesn Add, view and delete the users, servers
applications, services, devices and their groups
Policy Clientn Communicate with Policy Server using COPS
protocol, e.g. issue policy request (REQ)n Receives policy from Policy Server and
configures the Network Element, e.g. edge router
n Usually deployed within the Network Element
Policy Servern Communicate with the Policy Client to service
policy requestsn Execute SQLQuery to retrieve the policy from
policy repository
Policy Client for BBn Accept the Resource Allocation Request (RAR)
from the BB in the same domainn Contact Policy Server which will check the
request against the policy in databasen If the request conforms to the policy
n reply to the BB using an accept message, otherwise send a reject message
COPS Message Types
Ø Request (REQ) PEP →PDP
Ø Decision (DEC) PDP →PEP
Ø Report State (RPT) PEP → PDP
Ø Delete Request State (DRQ) PEP → PDP
Ø Synchronize State Request (SSQ) PDP →PEP
Ø Client Open(OPN) PEP→PDP
Ø Client Accept (CAT) PDP → PEP
Ø Client Close (CC) PDP ?PEP
Ø Keep Alive (KA) PDP ?PEP
Ø Synchronize Complete (SSC) PEP → PDP
Common Open Policy Service (COPS)n COPS policy client models
n Outsourcing (RSVP)n Provisioning (DiffServ)
The COPS message: 0 1 2 3 +--------------+--------------+--------------+--------------+ |Version| Flags| Op Code | Client-type | +--------------+--------------+--------------+--------------+ | Message Length | +--------------+--------------+--------------+--------------+ Op Code=<Request> <Request> ::= <Common Header> <Client Handle> <Context = config request> *(<Named ClientSI>) [<Integrity>] Op Code=<decision> <Decision Message> ::= <Common Header> <Client Handle> *(<Decision>) | <Error> [<Integrity>] <Decision> ::= <Context> <Decision: Flags> [<Named Decision Data: Provisioning >]
also, SIBBS
Program Interaction (Start Up)
Policy Manager
0
7
1
2
34 5
6PolicyCfg
prgPolicyClient
prgPolicyServer
SQLQuery
Policy RepositoryConfiguration Files
COPS REQ
COPS DEC
Policy Client
NetworkElement
Policy Server
8
A High Level Policy:
IF the traffic from CCN video server to CIR video client requesting for resources from the Ingress Router in CCN marked EF from 28/01/2000 to 28/02/2002 not exceeding the peak rate 1MThen acceptedIF it exceeds the peak rate Then rejected.
A Sample Policy for Bandwidth Broker
Policy Data in the Policy Databasen Bandwidth Broker Address: 202.8.94.186n Source Address: 202.8.94.169/29n Destination Address: 202.8.94.149/29n Ingress Router Address: 202.8.94.161/30n Peak Rate: 1Mn Class: EFn Start Time: 28/01/2002n Stop Time: 28/02/2002
Policy-based Management System
Policy [policy_11][general][in][clfr_11]: [clfr_11][clfr_11]: [clfrem111][clfr_11][1][meter_11][filter_11]: [filter_11][Internet][196.168.1.2][0.0.0.255][Internet][202.8.94.139][0.0.0.255][0][tcp][3288][3288][0][32767][Enabled]: [meter_11][action_11s][0.0][tbmeter_11]: [tbmeter_11][tokenBucket][200][4000][50000]: [action_11s][algdrop_11][marker_11][2]: [marker_11][ef]: [algdrop_11][wred][queue_11][11][0][rddrop_11]: [queue_11][schdlr_11][schdpm_11]: [schdpm_11][4][200][0][500][0]: [schdlr_11][][4][0.0]: [rddrop_11][500][0][1000][0][10][10][10]:
Input Interface configuration set timeout 20 spawn telnet 192.168.1.4 expect "Username:" send "zhangyi\r" expect "Password:" send "*****\r" expect "ccn-4700#" send "configure terminal\r" expect "ccn-4700(config)#" send "access-list 111 permit tcp 202.8.94.139 0.0.0.255 196.168.1.2 0.0.0.255 eq 3288\r" expect "ccn-4700(config)#" send "interface ethernet2\r" expect "ccn-4700(config-if)#" send "ip access-group 111 in\r" expect "ccn-4700(config)#" send "class-map clfr_11\r" expect "ccn-4700(config-cmap)#" send "match access-group 111\r" expect "ccn-4700(config-cmap)#" send "exit\r" expect "ccn-4700(config)#" send "policy-map meter_11\r" expect "ccn-4700(config-pmap)#" send "class clfr_11\r" expect "ccn-4700(config-pmap-c)#" send "police 200 4000 4000 conform-action set-dscp-transmit ef exceed-action drop\r" expect "ccn-4700(config-pmap-c)#" send "exit\r" expect "ccn-4700(config-pmap)#" send "exit\r" expect "ccn-4700(config)#" send "interface ethernet2\r" expect "ccn-4700(config-if)#" send "service-policy input meter_11\r" expect "ccn-4700(config-if)#" send "exit\r" expect "ccn-4700(config)#"
Output Interface configuration set timeout 20 spawn telnet 192.168.1.4 expect "Username:" send "zhangyi\r" expect "Password:" send "*****\r" expect "ccn-4700#" send "configure terminal\r" expect "ccn-4700(config)#" send "class-map action_11s\r" expect "ccn-4700(config-cmap)#" send "match ip dscp ef\r" send "exit\r" expect "ccn-4700(config)#" send "policy-map schdlr_11\r" expect "ccn-4700(config-pmap)#" send "class action_11s\r" expect "ccn-4700(config-pmap-c)#" send "random-detect 10\r" expect "ccn-4700(config-pmap-c)#" send "exit\r" expect "ccn-4700(config-pmap)#" send "exit\r" expect "ccn-4700(config)#" send "interface ethernet1\r" expect "ccn-4700(config-if)#" send "service-policy output schdlr_11\r" expect "ccn-4700(config-if)#" send "exit\r" expect "ccn-4700(config)#"
Bandwidth Broker
A Resource Reservation System for DiffServ
Bandwidth Broker – Designn One BB per DiffServ domainn BB manages resources in its own domainn Negotiate with up- and down-stream BB to
establish end-to-end path
Idle
Self-Initializationget list of all
peers
Start TCP sessionbb_start_peer
bb_stop_peer
ActiveConn-ect
bb_connect_fail
Estab-lished
bb_connect_success
bb_accept_peer
Start
reject
bb_stop_peer
bb_stop_peer
Keep-Alive
Maintain a long-lived TCP connection with up- and down-stream brokers (peers)
Connection State Machine
Reservation State Machine
One RSM instance per RAR
Supports both Immediate Reservation and Advance Reservation
Provisioning of QPS Bandwidthat Edge Routers
U32Classifier
EFMarker
EFMeterDropp
er
U32C lassif ier
E FMarke r
E FM e terD ropp
er
U32C lassif ier
E FMarke r
E FM e terD ropp
er
O ther EF
D ropper
NUS (CCN/CIR)-SingAREN-QBone projectn Partners: iCAIR USA, Japan,Korea (APAN)n Applications: video streaming; later: VoIPn Objectives
n Connect to QBone(initially, 1.5 Mbps CBR between STAR TAP and SingAREN QBone routers)
n Set up measurement architecture (Surveyor, RTFM)
n Deploy real-time applications: video and VoIPn Develop and deploy QoS management &
accounting system: BB, policy-based, AAA
NTRCATM
CCN EE4700
STARTAPATM
APANTOKYO
ATM
NUSCCPRODATM
CCN EEATM
1-NETATM
CIRATM
NUSCCRESATM
Sing-ARENSTCATM
SingAREN
HQATM
TRANSPACATM
CIR7507
7505
3COM
7505
HQ12000
STC7507
NTRC7507
NTU CCATM
ROU-TER
ATM & Fast EthernetTestbeds
ATM & GigabitEthernet Testbeds
NUS CampusNetwork
NTU CampusNetwork
KRDLATM & other
Testbeds
Singapore ONENetwork
vBNS &Abilene
Ca*Net II
Japanresearchnetwork
ATM & Fast EthernetTestbeds
IP-levelQuality of Service (QoS)-enabled NetworkNUS-SingAREN-QBone
SingTelATM
SingTel12000
SingTel InternalTestbeds
34 Mbps
14 Mbps
2 Mbps
155 Mbps
155 Mbps
622 Mbps
155 Mbps
155 Mbps
155 Mbps
155 Mbps
155 Mbps
155 Mbps
155 Mbps
155 Mbps
USA
Singapore
CanadaJapan
L.A.TransitATM
SingARENLocal
SingARENInternational
SPRINGi connections
Digital Video StreamingNUS/SingAREN + iCAIR(NWU) & IBM
n Based on IBM VideoChargern Scalable and easy to usen Integrated into applicationsn Streaming and interactiven Real-time and asynchronous (stored)n Unicast and native multicastn Single source to multi-sourcen Resolutions up to HDTV
n Joint work looks at network QoS + server adaptationaccording to QoS
QBone Connectivity -- Physical
KRDLATM
SwitchDS domain in KRDL
Internet 2(QBONE)
1MbpsPVC toStarTap
(Chicago)
NUSCCFORESwitch
CIR7505
ForeAsx200Bx
LS1010 LinuxRouter
(Humper)
LinuxRouter
(Dumper)
Clark
CEO
STC7507
SingARENAOS
Switch
STC7507
CCNFORESwitch
CCN4700
CCNEdge
Video Server
ATMOC-3
Ethernet
QBone Connectivity – IP-level testbed
Internet2/QBone
CIR7505
LinuxRouter
(Humper)
LinuxRouter
(Dumper)
CoreRouter
EdgeRouter
EdgeRouter
DS DomainCiR
CCN4700
LinuxRouter
(ccnedge)
LeafRouter
Video StreamServer
DS DomainCCN
202.8.94.166/29
202.8.94.165/29
202.8.94.162/30
202.8.94.161/30
202.8.94.154/30
202.8.94.185
Clark
CEO202.8.94.186
202.8.94.187
202.8.94.153/30
202.8.94.146/30202.8.94.145/30
202.8.94.129/30 202.8.94.130/30
202.8.94.250/30
STC7507
202.8.94.249/30
192.5.172.70/30
192.5.172.69/30
NUS
SingAREN/NUS
SingAREN
CCN/CIR Video testbed
14Mbps or1Mbps QBone
Note: not traversing Abilene or vBNSVideoCharger server
9,000 miles
(CB-WFQ)
DiffServ over MPLSn Multi-Protocol Label Switching (MPLS) domain
consisting of Cisco and Linux routersn Differentiated Services Over MPLS via
EXP-Inferred-PSC Label Switched Paths (E-LSPs)
n SNMP probes in Linux routers to provide MPLS-specific information which are then graphed using MRTG
DiffServ over MPLS
Measurements
n Cisco NetFlow export from Cisco routers
n Capture and analyse using Splintered flow-tools (used also by Abilene NOC)
Future Workn Design and deployment of traffic-aware
policies, including automatic discovery of good policies
n Advance reservations using Probing Requestmechanism
n Traffic Engineering over MPLSn Provisioningn LSP selection
n Wireless access network, multicast QoSn Server QoSn QoS in Grid and cluster computing
15
Proposed Clearing House ArchitectureProposed Clearing House Architecture
• Introduce logical hierarchy
• Distributed database• reservation status, % link utilization, optimum path
source
ISP1
ISP n
destination
Edge Router
LCH LCH LCH
CH1CH1
CH2
ISP2
Note: make advance reservations based on traffic prediction to reduce latency
(in AIX 4.3.3)
Conclusionn QBone as a testbed to:
n implement QoS services and the applications which use these services
n experiment with advanced inter-domain signalling, measurement and QoS management
n We welcome collaboration with other groups working on QoS, measurements, policy-based managementn QoS in the Internet has to be deployed and not
remain a research curiosity
Acknowledgementsn Funding
n SingAREN/A*STAR Broadband 21 Grant
n NUS QBone Project Team Membersn A/Prof AL Ananda (CIR/SoC)n Michael Yuan (CIR/ECE)n Zhang Yi, Mayank Agarwal, Paul Ng, He You Ming,
Kong Kean Fong, Tan Eng Theng (ECE)
n NUS Computer Centre staffn Roland Yeo
n SingAREN staffn Jonathan Ongn Pua Chin Kok
Bandwidth Broker and Policy-Based Management
Demo by Michael Yuan Lihua, ECE/CIR
Bandwidth Brokern What is it?
n An agent-based resource reservation systemn Receiver-based – RSVPn Sender-based – ST-II n Advantage:
n Signaling only between BBs at the control planen Reservation can be initiated by anybody (Sender,
Receiver, even third party)n Per-domain reservation – scalability
Bandwidth Broker: The nodal model
Bandwidth Broker: The Implementationn Cisco IOS-like TTY user interface
n Most network admins are familiar with itn Apply changes on-the-flyn A single configuration file for all the modulesn A GUI under development
Bandwidth Broker: The Implementationn Immediate Reservation (IR)
n Reserve and use n Release the reservation when finish
n Book-Ahead (BA)n Reserve in advancen Necessary if resource is limitedn Resource usage/reservation information need to be
kept over a long time periodn Implemented using AVL-tree
The Demo: Testbed setup
The Demo: BB-based deploymentn Best-Effort is not enoughn A DiffServ network with admission-control
n No Reservation ? n Go awayn Or? Ask your BB to talk to me (RAR)
n Made Reservation ? n You are most welcome
The Demo: Policy-based Deploymentn Different policy for different user/user group
n Nobody? Poor Guy?n “I will try my best to help, but ……”
n VIP?n Premium Service