Tiziana Ferrari Quality of Service Support in Packet Networks1 Quality of Service Support in Packet Networks Tiziana Ferrari [email protected] Italian.

Tiziana Ferrari Quality of Service Support in Packet Networks 1

Quality of Service Supportin Packet Networks

Tiziana Ferrari

[email protected]

Italian National Institute for Nuclear Physics

INFN - CNAF


Overview

• Problem statement

• technical solutions:

– ATM

– RSVP and RSVP to ATM SVC mapping

– differentiated services (diffserv)

• Diffeserv in detail

• Diffserv: a case study

• Diffserv test activities (TF-TANT)

• comments


Quality of Service

• Heterogeneous networks– capacity– transmission technology– bottlenecks

• congestion and lack of transmission guarantees

• heterogeneous application requirements– interactive: telnet, remote X sessions, web browsing– non-interactive and packet loss tolerant: ftp, mailing– delay sensitive: real time applications (e.g. remote control)– delay variation sensitive and packet loss: voice over IP, videoconferencing– ...

Quality of Service


Quality of Service: basic concepts

• Service: a pre-defined type of packet treatment during transmission across the network – qualitative– quantitative

> delay> instantaneous delay variation> packet loss probability> throughput > MTU (Maximum Transfer Unit)> priority (e.g. for congestion treatment)

• Class: set of packets to which a given service applies. Classification is based on traffic filters. A filter defines a set of

packetmatching rules. Matching is based on the content of packet fields.E.g. filter := (pack(src) = SRC) && (pack(dest) = DEST) &&

(pack(pro) = TCP)


Quality of Service: benefits

• Benefits:

– network. congestion management, congestion independent QoS

parameters, traffic engineering> differentiated allocation of expensive network resources e.g.

over intercontinental connections> multiple services

– application:

> within a single application: differentiated treatment of streams

according to their requirements and priority

> multiple applications: applications hierarchy according to

priorities

• today: single service, best-effort


Solutions

• Cell based networks:

– ATM (permanent and/or switched connections)

• IP based networks:

– RSVP (resource ReSerVation Protocol) and integrated services

• heterogeneous networks:

– RSVP to ATM SVC mapping (IP and ATM)

– differentiated services


ATM (Asynchronous Transfer Mode)

• Cell based• pre-defined classes of services:

– Constant Bit Rate (peak cell rate - PCR-)– Variable Bit Rate (sustainable cell rate, PCR, maximum burst size

-MBS-)> real time> non real time

– Available Bit Rate (minimum cell rate -MCR-, PCR, rate increase/decrease factors)

– Unspecified bit rate (PCR)• today: deployed as backbone technology (GARR, European national

research networks, TEN-155, ESnet), not common as LAN technology. Only permanent connections are commonly deployed to support traffic engineering.


ATM (cont)

• Permanent and/or dynamic connections (PVC, SVC)• 1-to-1 or 1-to-many• end-to-end signalling protocol for ATM connection set-up• traffic engineering• Disadvantages:

– not widely deployed as LAN technology (no end-to-end connectivity)

– no native ATM applications– IP over ATM: overhead– few applications supporting traffic profile definition– addressing scheme not compatible with IP– signalling only in few backbones -> lack of interoperability


RSVP• RSVP: signalling protocol for IP based applications

– traffic profile definition - source -

– reservation profile specification - destination -

– 3 classes of service: best-effort, controlled load, guaranteed

– reservation: (soft state)

– QoS support in heterogeneous network

– multicast is supported

– QoS support to the application

• Advantage: IP compatible

• Disadvantages: – requires RSVP support on each router on the path from tx to rx

– lack of scalability

– admission control and policy management


RSVP to ATM SVC mapping

• ATM signalling in the core

• RSVP signalling at the edge (access networks)

• reservation parameter mapping at the boundary

• Advantages:

– deploys ATM features in the backbones– QoS support in heterogeneous environments– QoS to the application– interim solution -> intserv - diffserv (scalability)

• Disadvantages:– application: RSVP capable, traffic profile specification– RSVP support at both edges– connection set-up latency: not convenient for delay sensitive

applications transmitting small chunks of data– still relays on ATM signalling in the core


RSVP to ATM mapping: features

• Translation of intserv classes of services and parameters into ATM classes of services and parameters

• deployment of best-effort connections (UBR in the ATM core) for initial transmission of RSVP messages (PATH, RESV)

• combination of RSVP and ATM admission control

• ATM: tx initiated signalling vs RSVP: rx initiated reservation request


Scenarios

• Mapping in the end-system

• mapping in the router

ATM

NRN - TEN-155 - NRNRSVP -> ATM

ATM

RSVPRSVP

3. RSVP -> ATM

1. PATH

2. RESV4. SVC

ATMATM

LIS 1 LIS 3LIS 2


Differentiated Services:Architecture


Differentiated services

• Why diffserv?– new technology

– independent of layer 2 technologies

– interoperability between independent national research networks (different requirements, infrastructures, policies and management)

– traffic aggregation

– scalability: no reservation state maintained in the routers

– no signalling

– QoS for networks not ATM based• RFC 2474: Definition of the Differentiated Services Field (DS Field) in the

IPv4 and IPv6 headers• RFC 2475: An Architecture for Differentiated Services• RFC 2598: An Expedited Forwarding PHB• RFC 2597: Assured Forwarding PHB Group

cont.


Diffserv architecture: building blocks

• Label: DS field (1 byte), DS Code Point (6 bits)

• packet classification

• packet scheduling

• traffic conditioning: – metering– marking– policing– shaping

DSCP CU0 6 7


DS building blocks: logical view

classifier marker Shaper/policer

meter

scheduler


Diffserv: traffic aggregation and (re)marking

aggregation

aggregation

aggregationand re-marking

re-markingmarking


Terminology

• Per Hop Behaviour (PHB): the externally observable forwarding behaviour applied to a DS-compliant node to packets with same label (DS codepoint)

• PHB Group: set of PHBs which can only meaningfully specified and implemented simultaneously (e.g. with common constraints on queue servicing and queue management). E.g. 4 PHB each associated to a different drop priority. A single PHB is a special case of PHB Group.

• Service: quantitative or statistical definition of significant characteristics of packet transmission in one direction across the network in terms of throughput, delay, jitter, loss, priority in access to network resources. Services are implemented through PHBs. The service describes the overall treatment end-to-end.


Terminology (cont)• DS codepoint: specific value of the DSCP field

• DS behaviour aggregate: packets with same code point

• DS domain: contiguous set of nodes with same service provisioning policies and same code point numbering scheme

• DS region: set of contiguous DS domains

• DS ingress/egress node: DS node handling packets entering/leaving the DS domain it belongs to

• classifier: entity selecting packets according to the content of packet headers according to a defined rule

• BA classifier: a classifier which only takes into account the DS field content

DS RegionDS Domain

Interior/Ingress/Egress Node


Terminology (cont)• Conditioning: metering, marking• Policing: packet discard according to the state of a corresponding

meter enforcing a traffic profile• Metering: the process of measuring the temporal properties of a

traffic stream selected by a classifier• Marking: the process of setting the DS codepoint in a packet based on

defined rules• Service Level Agreement: traffic contract between a customer and

service provider specifying the forwarding service the customer’s traffic should receive

• Service Provisioning Policy: specification of– microflow mapping into a DS Behaviour Aggregate– conditioning configuration


Diffserv architecture: network model for TEN-155

DS domain

DS domain

DS domainDS domain

Non DS capabledomain

MPLS

NRN

NRN

TEN-155

marking

Markingpolicing scheduling

shaping

DS domain


Diffserv architecture: PHBs

• Standardised PHBs:

– Expedited Forwarding (low delay, low delay variation,

guaranteed bandwidth)

– Assured Forwarding (Behaviour Aggregate, 4 classes,

3 drop priorities per class)

• Experimental PHBs

• PHB class selectors

0 3 7

precedence


PHB: Expedited Forwarding

• For the implementation of services requiring a reservation profile like: low loss, low latency, low jitter, assured bandwidth

• loss, latency and jitter queue management (small queues)

in order to prevent a queue from building up, in the EF queue of

each transit node, the aggregate maximum arrival rate < departure rate• EF implementation based on

– scheduling (for traffic isolation and support of bandwidth guarantees)

– policing – shaping

• EF traffic can preempt other classes, for this reason the maximum EF rate has to be limited through policing

• EF codepoint: 101|110


PHB: Assured Forwarding

• 4 independent AF forwarding classes

• 3 drop priorities in each AF class

• given two packets in node with drop precedence p and q respectively, with p < q, pack(p) is always transmitted BEFORE pack(q)

• at the boundary between two AF domains, traffic conditioning can apply: shaping, per class discarding, drop precedence remarking and AF class reassignment. Traffic conditioning has to avoid REORDERING -> performance gain on the rx side


PHB: Assured Forwarding (cont)

• In case of long term congestion AF packets are dropped

• drop algorithm:

WRED (Weighted Random Early Discard) for the implementation of a gradual discard mechanism based on congestion levels and proportional to the drop precedence of a given microflow

• Codepoints:

class 1 class2 class3 class4

low drop prec 001|010 010|010 011|010 100|010

medium drop prec 001|100 010|100 011|100 100|010

high drop prec 001|110 010|110 110|110 100|110


PHB: Class Selectors and Experimental PHBs• Class selectors:

– XXX000 where x = 1 or 0

– for backward compatibility with precedence field of the old so called TOS (Type Of Service) byte. TOS was replaced by the DS field. Precedence = [0, 7]

– if pred(pack1) < pred(pack2) then

p_drop(pack1) > p_drop(pack2)

• Experimental PHB: – not standardised codepoint and packet forwarding behaviour

– definition up to the ISP

– requires PHB mapping at the boundary

0 3 7

precedence TOS byte


Diffserv architecture:scheduling

policing and classificationtraffic metering


Scheduling• Scheduling: queue service policy for differentiated treatment of

packets among queues

• Examples of packet scheduling algorithms:

– simple priority queue (high priority queues have the highest priority, arrival rate < departure rate, low priority traffic starvation is possible)

– weighted round robin queuing (queues serviced in round robin fashion, service time proportional to the weight)

– weighted fair queuing (minimum rate guaranteed per class, service time of each packet in each queue is a function of the packet size and of the queue weight. Current service time is updated every time a packet is sent)

– class based queuing (maximum rate per class is configured)


Congestion management and service enforcement: policing

CustomerPremise

PolicyPolicySpecificationSpecification

Network Edge Packet Classifier and Policer

Policing: traffic which exceeds a given rate threshold is treateddifferently from conforming traffic. E.g. Exceeding packets can be dropped, re-marked, transmitted as best-effort etc.Policing is deployed for service level agreement enforcement:

- to limit the input rate at the edge- at the boundary between domains to guarantee a fair deployment of the service among different domains


Policing: token bucket

• Policing is based on traffic metering. A typical algorithm is called token bucket.Input pack stream

output pack stream

Exceed burst0 < drop probability < 1

Normal burst (number of tokens available),drop probability = 0

R: departure rate at which tokens are replenished

drop probability = 1


Token bucket: algorithm• tokens replenished at regular intervals

• Normal burst: max number of tokens which can be in the bucket (in bytes)

• Exceed burst: to avoid tail drop in favour of gradual drop

• Actual debt AD = ADi where is the number of borrowed tokens,

– ADi decreases of R tokens per time unit

• Compounded debt

CD = ADi where ADi

CD = 0 after a packet drop

packi is dropped if CB > exceed burst


Token bucket: algorithm (cont)Example (by BoB Olsen)

token rate = 1 data_unit/time_unitnormal burst size = 2 data_units (DUs)extended burst = 4 DUs. rate = 2 DUs arrive per time unit.

After 2 time units, the stream has used up its normal burst and must begin borrowing one DU per time unit, beginning at time unit 3.

Time tocken available DU arrivals Actual Debt Compounded Debt0 21 2-2+1=1 2 0 02 1-2+1=0 2 0 03 0-2+1=-1 2 1 14 -1-2+1=-2 2 2 35 -2-2+1=-3 2 3 (temp) 6 (temp) > 4


Token bucket: algorithm (cont)

• At this time a packet is dropped because the new compounded debt (6) would exceed the extended burst limit (4). This causes CD to effectively become 0, and lowers AD back down to 2. The values 3 and 6 were only temporary and do not remain valid in the case where a packet is dropped. The final values for time unit 5 are given below.

• Time tocken available DU arrivals Actual Debt Compounded Debt

5 -2-2+1=-3 2 3 (temp) 6 (temp) > 4

drop

5 -2 2 2 (*) 0

6 -2-2+1=-3 2 3 3

7 -3-2+1=-4 2 4 (temp) 7 (temp)

drop

7 -3 2 3 0


Congestion Avoidance:Weighted Random Early Discard (WRED)

• Goal: to avoid congestion before it occurs by dropping isolated packets among different streams. Packets are dropped according to the current amount of data into a buffer. If

buffer_level < thr1 p(drop) = 0 case 1

thr1 < buffer_level < thr2 0 < p(drop) < 1 case 2

otherwise p(drop) = 1 case 3• (W)RED is a congestion avoidance algorithm for TCP traffic based on the

TCP flow control features (TCP reduces the output rate when a single packet is dropped before real congestion occurs)

• WRED: like RED but p(drop) in case 2 depends on the DS codepoint of the packet. Packets with low priority experience packet drop before packets with higher priority

• with both RED and WRED packet drop is randomly distributed among several flows


Weighted Random Early Discard (WRED) - cont -

• Gold Class 6 will never get drop unless extreme congestion : 90%of queue depth

• Silver Class 4 will not get dropped unless severe congestion :70%of queue depth

• Bronze Class 2 will start drop at 60%queue occupancy

• Standard The remaining traffic will start drop at 40% of queue occupancy

Pa

ck

et

Dro

p

Pro

ba

bil

ity

Queue Length

Standard

Max

Gold

40%

Bronze

Silver

70%60% 90%

Pa

ck

et

Dro

p

Pro

ba

bil

ity

Class 6 gets 50% minimumClass 4 gets 30% minimumClass 2 gets 20% minimumThe remaining traffic gets 10%


Differentiated Services:

Case Study


Characterisation of the application

Characterisation needed to identify the requirements of the application: service definition

• 1. Hardware of the trigger: remote control - ROBIN -– few transactions (low bandwidth), TCP traffic on a limited well-known set

of TCP ports, IP address of the server known– client - server, one connection to a server at a time

low paket loss, delay sensitive application, reliability, burst tolerance

• 2. Monitoring of quality and correctness of the results of the analysis - ROOT -– exchange of analysis results (root object)– low bandwidth consumption– client - server, IP address of the server is known

bandwidth guarantee, more tolerance to packet loss


Models of network deployment1. Trigger hardware control

Server m

VME

Server 1

Power PC

VMEServer 2

VME

...

Client 1

Client 2

Client 3

Client n

...

1. Monitoring of analysis

server

...

Browser 1

Browser 2

Browser 3

Browser n

bottlenecks


Service for trigger control packets

• SERVICE 1:– low drop probability – delay bound (minimum queue size)– precedence: highest precedence, higher than monitoring

packets precedence– Capacity: a minimum network capacity guaranteed, in

case of spare capacity, more bandwidth can be allocated.

– No upper limit in rate for maximum burst tolerance


Service 1: implementation

If pack(src/dest) = (s1 || s2 || … || s m) && pack(TCP_port) in [x, y] then pack(label) = max precedencepolicing: always transmitdelay: buffer size of 2*MTUminimum service rate = m * r * Nclient or rate = m * R * Nclient

m: overbooking factorr: estimated rate consumed by 1 client, R: estimated rate consumed by 1 serverNclient : number of clients downstream

Scheduling client -> server

VME

Server 1

VMEServer 2

VME

...

Client 1

Client 2

Client 3

Client 4

r

r

r

r

2*m*r

2*m*r

4*m*r

R

R

R

Server 8

Marking client -> server Marking server -> client

Scheduling server -> client


Service for monitoring traffic

• SERVICE 2:

– precedence: higher than best effort, lower than service 1 packets

– drop probability: packets can be dropped in case of congestion.

– Label: lower precedence

– capacity: minimum guaranteed bandwidth, more bandwidth can be allocated if available

– maximum upper rate threshold: for fair bandwidth allocation between several clients


Service 2: implementation

If pack(src/dest) = s1 thenif rate < max, pack(label) = medium prcedence (marking)else drop (policing)

shaping: buffer size > server or client burst sizeminimum service rate = r * Nclient (client), or rate = R * Nclient (server)

r: estimated rate needed for 1 client, R: estimated rate needed by the server to support m clientsNclient : number of clients downstream

precedence: < precedence(service 1)

Scheduling client -> server

Server ...

Client 1

Client 2

Client 3

Client 4

r

r

r

r

2*r

2*r

4*rR

Marking client -> server

Policing client -> server Policing server -> client

Marking server -> client

Scheduling server -> client


Diffserv testing and QoS measurement


Test network


LAN layout (example)


Equipment

• Test workstations

• traffic generators (3 SmartBits, Netcom System loan)

• ATM switches in the test sites

• cabletron ethernet switch

• DS capable platforms:– CISCO: C7200 or C7500 (partial CISCO loan)– IBM: IBM 2212 and IBM 2216 (2 routers in 5 sites,

IBM donation)– Linux– Cabletron (LAN switch)– (Nortel, Torrent)


CISCO: traffic policing (CAR)

• CAR: Committed Access Rate

• Multi-field classification: OK

• packet marking (precedence setting): OK

• exceed action testing: OK

• policing (at a configurable rate): two parameters token bucket for TCP performance optimisation

– normal burst

– exceed burst

--> parameter tuning


CAR: test of exceed actions

Throughput:SWITCH: 1.20 MbpsDANTE: 0.38 Mbps


CAR: TCP burst tolerance

• Normal and exceed burst tuning with TCP traffic

• single and multiple TCP streams

optimum values are functions of the rate R at which

traffic is policed, in particular:

normal burst = 0.5 sec * R

exceed burst = 2 * normal burst


CAR: TCP burst tolerance (cont)

Throughput of 1 TCP connection (Mbps)Exceed (bytes)Normal

(bytes) 32000 48000 64000 96000 12800032000 0.98 1.23 1.23 1.25 1.2548000 1.09 1.21 1.25 12564000 1.18 1.24 1.2596000 1.24 1.25128000 1.25

Table 4: throughput of 1 TCP connection for increasing values of the normal and exceed burst size

Aggregate throughput of 5 concurrent TCP connection (Mbps)Exceed (bytes)Normal

(bytes) 32000 48000 64000 96000 12800032000 1.26 1.26 1.25 1.26 1.2548000 1.25 1.26 1.25 12664000 1.25 1.27 1.2596000 1.26 1.26128000 1.25

Table 5: throughput of 5 TCP connections for increasing values of the normal and exceed burst size


CISCO: scheduling (CB-WFQ)

Scheduling mechanism to provide minimum bandwidth

guarantees to classes

• class definition: precedence or MF classification --> OK

• bandwidth allocation: no starvation, no bandwidth consumption --> OK

• traffic isolation: scenarios– UDP high priority + UDP best-effort

– TCP high priority + UDP best-effort

– TCP high priority + TCP best-effort

UDP --> OK (always)

TCP: inconsistent results with 1 TCP stream due to cell shaping problems

in the ATM part of the network, good results with several TCP strams


CB-WFQ: set-up


IBM: scheduling (Self Clocked Fair Queuing)

• Policy = (traffic profile, validity period, diffserv action)

• diffserv action = (type of marking, queue type, bandwidth)

• optimum traffic isolation (tests only with UDP)

Premium TCP traffic throughput, target rate: 163 KbpsTest

numberStreams BE

throughput(Kbps)

AssuredThroughput

(Kbps)

Premiumthroughput

(Kbps)

Total throughput(Kbps)

1 BE 1967.7 / / 1967.72 A / 1968.0 / 1968.03 P / / 159.8 159.84 BE + A 649.8 1367.0 / 2016.85 BE + P 1852.5 / 159.8 2012.36 A + P / 1852.0 159.8 2011.87 BE + A + P 617.8 1236.9 159.8 2014.6

Premium: 163.8 Kbps guaranteed (8% PPP link bw)Assured: 819.2 Kbps (40% PPP bw)


IBM: EF policing

Small TCP burst tolerance in a policer can completely starve a TCP stream. Token bucket depth is key parameter --> tuning need according to the rate at which traffic is policed


IBM: EF policing (cont)

Bucket size(bytes)

Test length(sec)

TCP Throughput(Kbps)

2200 Connection stalled ~ 04400 60 0.97

60 35.2120 74.7180 89.8240 88.5300 95.6360 98.2420 99.3

6600

480 100.68800 300 118.911000 300 124.413200 300 124.815400 300 126.017600 300 125.364000 300 125.0

EF with TCP traffic, target rate = 163 Kbps


QoS measurement

• Resource allocation monitoring

– for resource allocation and network dimensioning

• performance measurement

– passive

– active (invasive traffic)

– for service validation

– to understand the effect on end-to-end performance of buffering in one router or in a chain


Performance measurement

• parameters of interest:

– one-way delay, requirements: clock synchronisation

> GPS based synchronisation

• SmartBits (Netcom Systems)

> NTP (Network Time Protocol)

• NTP client / server hierarchy

– one-way delay variation

– packet loss

– throughput

– RTT


Services: examples

• Virtual leased line:

– point to point

– one to many

• capacity allocation on congested links (e.g. US links)

• better-than-best-effort (qualitative definition)

• delay bound and delay variation sensitive traffic classes

• rate limiting of invasive traffic

• ...


More information at

• diffserv:

– http://www.cnaf.infn.it/~ferrari/tfng/ds

• QoS measurement:

– http://www.cnaf.infn.it/~ferrari/tfng/qosmon

• QBONE: US initiative for testing, validation and deployment of services based on the expedited forwarding PHB of the pr


Comments

• Diffserv building blocks: already supported by several vendors

• diffserv: no changes in applications required

• diffserv goal: simplified approach to QoS for its support in backbones from today

• good interim test results

• diffserv in the future:

– Packet Over Sonet (POS) vs ATM

– VLL vs ATM permanent connections


Comments (cont)

• diffserv still requires QoS support end-to-end (but diffserv can be implemented in some domains, provided that the end-to-end service is homogeneous)

diffserv diffserv

ATM/POS

Dedicated connection

Production network

diffserv diffservdiffserv

Production network


Comments (cont)

• Issues

– effects of high degree aggregation? More testing needed

– interoperability between different platforms: effect on end-to-end services?

– performance of marking, classification and scheduling at high speed?

– Tools for service monitoring….

– Diffserv in production? 1 year?


Discussion

• Deployment of diffserv in HEP

• Issues in diffserv deployment in HEP

• recommendations

Tiziana Ferrari Quality of Service Support in Packet Networks1 Quality of Service Support in Packet Networks Tiziana Ferrari [email protected] Italian.

Documents

atm rsvp

quality of service slide

atm mapping

packet networks9 rsvp

packet networks10 rsvp

packet networks11 rsvp

atm core

single service