Shivkumar KalyanaramanRensselaer Polytechnic Institute
1
Edge-based Traffic Management Building Blocks
David Harrison, Yong Xia, Shiv Kalyanaraman,
Rensselaer Polytechnic Institute
http://www.ecse.rpi.edu/Homepages/shivkuma
I E
I
EI
E
Logical FIFO
B
Shivkumar KalyanaramanRensselaer Polytechnic Institute
2
Private Networks vs Public Networks QoS vs Congestion Control: the middle ground ?
Overlay Bandwidth Services: Key: deployment advantages A closed-loop QoS building block
Services: Better best-effort services, Assured services, Quasi-leased lines, App-level QoS…
Overview
Shivkumar KalyanaramanRensselaer Polytechnic Institute
3
Motivation: Site-to-Site VPN Over a Multi-Provider Internetwork
Shivkumar KalyanaramanRensselaer Polytechnic Institute
4
Virtual ISP: Network-level Overlay Avoid crossing ISP boundaries
Each ISP will provide good service; V-ISP can easily verify it Allocate/buy service across each ISP and compose them Network (IP)-level overlay
ISP 1
ISP 2
ISP 3
Proxy(edge)
GPoP(core)
GPoP(core)
Proxy(edge)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
5
Our Model: Edge-based building blocks
New: Closed-loop control !Policy/Bandwidth Broker
I E
I
EI
E
Logical FIFO
B
Model: Inspired by diff-serv; Aim: further interior simplification
Shivkumar KalyanaramanRensselaer Polytechnic Institute
6
Closed-loop BB: Bandwidth Sharing
FIFO
B
Loops: differentiate service on an RTT-by-RTT basis using edge-based policy configuration.
B
Priority/WFQ
Scheduler: differentiates service on a packet-by-packet basis
Shivkumar KalyanaramanRensselaer Polytechnic Institute
7
Queuing Behavior: Without Closed-loop Control
End system
Bottleneckqueue
Shivkumar KalyanaramanRensselaer Polytechnic Institute
8
Queuing: With Closed Loops Bottleneck management issues consolidated at edges
Key: Transparent and lossless loop schemes
Potential: Edge-based QoS services, Edge plays in application-level QoS, active networking..
Shivkumar KalyanaramanRensselaer Polytechnic Institute
9
Closed-loop Building Block Reqts
#1. Edge-to-edge overlay operation, #2. Robust stability #3. Bounded-buffer/zero-loss,
#4. Minimal configuration/upgrades + incremental deployment
#5. Rate-based operation: for bandwidth services
Not available in any congestion control scheme… Related work: NETBLT, TCP Vegas, Mo/Walrand, ATM
Rate/Credit approaches
Shivkumar KalyanaramanRensselaer Polytechnic Institute
10
Queuing at One Router: Arrival / Service Curves
),(),(
)],(),([
)]()([)]()([
)()(),(
)()()(
)()()(
ttOttI
ttttt
tSttStAttA
tqttqttq
ttSttAttq
tStAtq
ijij
ijij
ijijijij
ijijij
ijijij
ijijij
flow i at router j arrival curve Aij(t)
& service curve Sij(t)
cumulative continuous non-decreasing
if no loss, then
time
Aij(t)
Sij(t)
queue
delaybit
t2t1
b1
b2
Shivkumar KalyanaramanRensselaer Polytechnic Institute
11
Accumulation: Series of Routers
J
j
J
jkkiji dtqta
1
1
)()(
11,)()( 1, Jjitdt jijij we have
define accumulation
which is a time-shifted, distributed sum of buffered bits of flow i at all routers 1 through J
1 j j+1 J
μij Λi,j+1
dj
fi
Λiμi
ingress egress
Shivkumar KalyanaramanRensselaer Polytechnic Institute
12
Accumulation (Contd)
12
1
1
1
11
1
1
1
1
1
1
),(),(
)],(),([
)],(),([
),(
)()(
)()(),(
J
jj
fi
if
ii
if
ii
J
j
J
jkkij
J
jkkij
J
j
J
jkkij
J
j
J
jkkij
J
j
J
jkkij
iii
ddwhere
ttOtdtI
ttttdt
ttdttdt
tdtq
dtqdttq
tattatta
then
1 j j+1 J
μij Λi,j+1
djfi
Λiμi
ingress egress
Shivkumar KalyanaramanRensselaer Polytechnic Institute
13
Accumulation vs Queuing
queue qij(t) -- num of bits of flow i queued in a fifo router j
accumulation ai(t) -- num of bits of flow i queued in a set of fifo routers 1~J
),(),(),(
)(
ttOttIttq
tq
ijijij
ij
fi
if
iii
J
j
J
jkkiji
d
ttOtdtItta
dtqta
),(),(),(
)()(1
1
the collective queuing behavior of a series of fifo routers looks similar to that of one single fifo router
is the forward direction propagation delay.
Shivkumar KalyanaramanRensselaer Polytechnic Institute
14
Accumulation: Physical Meaning
1 j j+1 J
μij Λi,j+1
dj
fi
Λiμi
… …
14time
)(1f
ii dtq )(
1
J
jkkij dtq
)(tqiJ
1 j j+1 J
jd 1Jd
),( tdtI fii
)(tai
)( ttai
),( ttOi
fid
t
Shivkumar KalyanaramanRensselaer Polytechnic Institute
15
Edge-based Control (EC) policy
1 j j+1 J
μij Λi,j+1
dj
fi
Λiμi
0)( ii ta control objective : keep if , no way to probe increase of available bw;0)( tai
ttttdtttarec
thentaif
thentaif
if
iii
iii
iii
)],(),([),(:
)(
)(
control algorithm :
Shivkumar KalyanaramanRensselaer Polytechnic Institute
16
16
EC schemes
monaco accumulation estimation: out-of-band / in-band congestion response: additive inc/additive dec (aiad), etc
vegas accumulation estimation: in-band congestion response: additive inc / additive dec (aiad)
riviera accumulation estimation: in-band congestion response: additive inc / multiplicative dec
using egress rate (aimd-er)
Shivkumar KalyanaramanRensselaer Polytechnic Institute
17
Recall: accumulation theory
J
j
J
jkkiji dtqta
1
1
)()(
… …
time
)(1f
ii dtq )(
1
J
jkkij dtq
)(tqiJ
1 j j+1 J
jd 1Jd
)(tai
)( ttai
fid
Shivkumar KalyanaramanRensselaer Polytechnic Institute
18
Accumulation vs. Monaco Estimator
1 j j+1 J
μij Λi,j+1
dj
fi
Λiμi
… …
time
)(1f
ii dtq )(
1
J
jkkij dtq
)(tqiJ
1 j j+1 J
jd 1Jd
)(taq im out-of-band
in-band ctrl pkt
),,(1
J
jkkq dtjit
Shivkumar KalyanaramanRensselaer Polytechnic Institute
19
19
Accumulation vs. Monaco estimator
1 jf Jf
μij Λi,j+1
djf
fi
Λiμi
Jb jb+1 jb 1djb ctrl
data
jf+1
out-of-bd ctrl
in-band ctrl,data pkt
classifier
ctrl
fifo
Shivkumar KalyanaramanRensselaer Polytechnic Institute
20
ec: monaco
20
congestion estimation:out-of-band and in-band control packets
congestion response: (AIAD)if qm < α, cwnd(k+1) = cwnd(k) + 1;
if qm > β, cwnd(k+1) = cwnd(k) – 1;[ 1 = α < β = 3 ]
Shivkumar KalyanaramanRensselaer Polytechnic Institute
21
ec: vegas
congestion estimation:define qv = ( cwnd / rttp – cwnd / rtt ) * rttp;
where rttp is round trip propagation delay (basertt)
congestion response:if qv < α, cwnd(k+1) = cwnd(k) + 1;
if qv > β, cwnd(k+1) = cwnd(k) – 1; [ 1 = α < β = 3 ]
Time
cwnd
slow start
congestionavoidance
Shivkumar KalyanaramanRensselaer Polytechnic Institute
22
Vegas Accumulation Estimator
22
the physical meaning of qv
rtt = rttp + rttq [ rttq is queuing time ]
qv = ( cwnd / rttp – cwnd / rtt ) * rttp
= ( cwnd / rtt ) * ( rtt – rttp )
= ( cwnd / rtt ) * rttq [ if rtt is typical ]
= sending rate * rttq [ little’s law ]
= packets backlogged [ little’s law again ]
so vegas maintains α ~ β number of packets queued inside the network
it adjusts sending rate additively to achieve this
Shivkumar KalyanaramanRensselaer Polytechnic Institute
23
23
Accumulation vs. Vegas estimator
)()(
)(
)(
)()()(
1,
1,
tadta
dtq
ddtq
rttrttttq
bi
bi
fi
J
j
J
jn
bnji
J
j
J
jm
fm
biji
bq
fqiiv
b
b
b
b
b
f
f
f
f
f
Backlogv
1 jf Jf
μij Λi,j+1
djf
fi
Λiμi
Jb jb+1 jb 1djb ack
data
jf+1
Shivkumar KalyanaramanRensselaer Polytechnic Institute
24
Vegas vs. Monaco estimators
Vegas accumulation estimatoringress-basedround trip (forward data path and backward ack path)sensitive to ack path queuing delaysensitive to round trip propagation delay measurement
error
Monaco accumulation estimatoregress-basedone way (only forward data path)insensitive to ack path queuing delayno need to explicitly know one way propagation delay
Shivkumar KalyanaramanRensselaer Polytechnic Institute
25
Riviera
25
congestion estimation:in-band techniques, similar as vegas
congestion response:
ttttdtttarec
where
kaifkk
kaifkk
if
iii
ii
iiiii
iiiii
)],(),([),(:
10,0
)()()1(
)()()1(
Shivkumar KalyanaramanRensselaer Polytechnic Institute
26
Riviera: stability and fairness
lyapunov function
26
iiiii
iiiii
kaifkk
kaifkk
)()()1(
)()()1(
Ll
lliiIi
iiiii
l
dxxcpsswU
0
),(])1(log[)(
i
dxx
xswB ii
iiiii
0
log)(
each flow i maximizes ( utility – penalty )
proportionally fair
Shivkumar KalyanaramanRensselaer Polytechnic Institute
27
Linear Network Topology
27
I0
I1
I2
E0
E1
E2
B0 B1 Bn
100Mbps
4ms
I00
E00
I10
En0U
U
U
U U
88
88
8
U U
U
U
U
send rate (Mbps)
All links are 4ms, 100 Mbps.I=ingress, E=egress, U=UDP, B=Bottleneck
Shivkumar KalyanaramanRensselaer Polytechnic Institute
28
Stability and Fairness
28
Shivkumar KalyanaramanRensselaer Polytechnic Institute
29
Utilization
29
Shivkumar KalyanaramanRensselaer Polytechnic Institute
30
Utilization w/ Reverse Path Congestion
30
Shivkumar KalyanaramanRensselaer Polytechnic Institute
31
Queue, Utilization w/ Basertt Errors
31
Shivkumar KalyanaramanRensselaer Polytechnic Institute
32
Service Differentiation: Loss-based or Accumulation-based ?
32
Shivkumar KalyanaramanRensselaer Polytechnic Institute
33
Overlay Edge-to-edge Bandwidth Services
Idea: Use the EC scheme as a closed-loop building block for a range of QoS services
Basic Services: no admission control “Better” best-effort services Denial-of-service attack isolation support Weighted proportional/priority services
Advanced services: edge-based admission control Assured service emulation “Quasi-leased-line” service
Key: no upgrades; only configuration reqts…
Shivkumar KalyanaramanRensselaer Polytechnic Institute
34
Without Overlay Scheme With Overlay Scheme
Queue distribution to the edges => can manage more efficiently
CoV vs. No of Flows
FRED at the core vs. FRED at the edges with overlay control between edges
Scalable Best-effort TCP Service
Shivkumar KalyanaramanRensselaer Polytechnic Institute
35
Scalable Best-effort TCP Service
Shivkumar KalyanaramanRensselaer Polytechnic Institute
36
Edge-based Isolation of Denial of Service/Flooding
TCP starting at 0.0s UDP flood starting at 5.0s
Shivkumar KalyanaramanRensselaer Polytechnic Institute
37
Backoff Differentiation Policy:
Backoff little (as) when below assurance (a), Backoff (as) same as best effort when above assurance (a) Backoff differentiation quicker than increase differentiation
Service could be potentially oversubscribed (like frame-relay) Unsatisfied assurances just use heavier weight.
Edge-based Assured Service Emulation
1 > AS >BE >> 0
r =r + min(r, AS aa
if no congestion
if congestion
Shivkumar KalyanaramanRensselaer Polytechnic Institute
38
Bandwidth Assurances
Flow 1 with 4 Mbps assured + 3 Mbps best effort
Flow 2 with 3 Mbps best effort
Shivkumar KalyanaramanRensselaer Polytechnic Institute
39
Assume admission control and route-pinning (MPLS LSPs). Provide bandwidth guarantee. Key: No delay or jitter guarantees!
Adaptation in O(RTT) timescales Average delay can be managed by limiting total and per-
VL allocations (managed delay) Policy:
Quasi-Leased Line (QLL)
1 > BE >> 0
r =r + if no congestion
if congestionmax(aaa
Shivkumar KalyanaramanRensselaer Polytechnic Institute
40
Quasi-Leased Line Example
Background QLL starts with rate 50Mbps
Best-effort VL quickly adapts to new rate.
Best-effort rate limit versus time
Best-effort VL starts at t=0 and fully utilizes 100 Mbps bottleneck.
Shivkumar KalyanaramanRensselaer Polytechnic Institute
41
Quasi-Leased Line Example (cont)
Bottleneck queue versus time
Starting QLL incurs backlog.
Unlike TCP, VL traffic trunks backoff without requiring loss and without bottleneck assistance.
Requires more buffers: larger max queue
Shivkumar KalyanaramanRensselaer Polytechnic Institute
42
Quasi-Leased Line (cont.)
Worst-case queue vs Fraction of capacity for QLLs
Single bottleneck analysis:
q < b
1-bB/w-delay products
For b=.5, q=1 bw-rtt
Simulated QLL w/edge-to-edge control.
Shivkumar KalyanaramanRensselaer Polytechnic Institute
43
Current Work With bottlenecks consolidated at the edge:
What diff-serv PHBs or remote scheduler functionalities can be emulated from the edge ?
What is the impact of congestion control properties and rate of convergence on attainable set of services ?
Areas: Control plane architecture for large-scale overlays Application-level QoS: edge-to-end problem Dynamic (short-term) services Congestion-sensitive pricing: congestion info at the edge
Edge-based contracting/bidding frameworks
Shivkumar KalyanaramanRensselaer Polytechnic Institute
44
Summary
Private Networks vs Public Networks QoS vs Congestion Control vs Throwing bandwidth
Edge-based Building Blocks & Overlay services: A closed-loop QoS building block: EC framework Accumulation concept Monaco, Vegas, Riviera Schemes: estimation issues Basic services, advanced services