Buffer requirements for TCP: queueing theory & synchronization analysis

Buffer requirements for TCP:queueing theory & synchronization analysis

Gaurav Raina Damon WischikCambridge UCL

TCPif (seqno > _last_acked) {

if (!_in_fast_recovery) {

_last_acked = seqno;

_dupacks = 0;

inflate_window();

send_packets(now);

_last_sent_time = now;

return;

}

if (seqno < _recover) {

uint32_t new_data = seqno - _last_acked;


if (new_data < _cwnd) _cwnd -= new_data; else _cwnd=0;

_cwnd += _mss;

retransmit_packet(now);

send_packets(now);

return;

}

uint32_t flightsize = _highest_sent - seqno;

_cwnd = min(_ssthresh, flightsize + _mss);


_dupacks = 0;

_in_fast_recovery = false;

send_packets(now);

return;

}

if (_in_fast_recovery) {

_cwnd += _mss;

send_packets(now);

return;

}

_dupacks++;

if (_dupacks!=3) {

send_packets(now);

return;

}

_ssthresh = max(_cwnd/2, (uint32_t)(2 * _mss));

retransmit_packet(now);

_cwnd = _ssthresh + 3 * _mss;

_in_fast_recovery = true;

_recover = _highest_sent;

}

time [0-8 sec]

traf

fic r

ate

[0-1

00 k

B/s

ec]

Motivation: buffer size• Internet routers have buffers,

to accomodate bursts in traffic.

• How big do the buffers need to be?– 3 GByte? Rule of thumb is C*RTT—what Cisco does today

– 300 MByte? C*RTT/√N [Appenzeller, Keslassy, McKeown, 2004 ]

– 30 kByte? constant buffer size, independent of line rate

• Large buffers are unsustainable:– Data volumes double every 10 months– CPU speeds double every 18 months– Memory access speeds double every 10 years

Buffers & synchronization

Desynchronized TCP flows:– aggregate traffic

is smooth

– small buffers are OK

SynchronizedTCP flows:– aggregate traffic

is bursty

– large buffers are needed

++=

++=

aggr

egat

etr

affi

c ra

te

time

indi

vidu

alfl

ow r

ates

Network traffic model

• When there are many TCP flows, the average traffic rate xt varies smoothly, according to a differential equation[Misra, Gong, Towsley, 2000]

– involving average round trip time RTT

– and packet loss probability pt

time

aggr

egat

etr

affi

c ra

te

desynchronized synchronized

Network traffic model

• When there are many TCP flows, the average traffic rate xt varies smoothly, according to a differential equation[Misra, Gong, Towsley, 2000]

– involving average round trip time RTT

– and packet loss probability pt

• The packet loss probability pt satisfies a differential equation, which depends on buffer size:[Raina, W., 2005]

– either for small buffers (max queueing delay « RTT)

– or for large buffers (max queueing delay RTT)time

val1

40 41 42 43 44 45

05101520

20

05101520

200

05101520

2000

time

val1

40 42 44 46 48 50

050

100

150

queueapprox

time

queu

esi

ze

Analysis

• Write down differential equations– for average TCP traffic rate xt

– for queue dynamics and loss prob pt

taking account of buffer size

• Calculate– average link utilization– average queue occupancy/delay– extent of synchronization

and consequent loss of utilization, and jitter

Small buffers[20-50 pkts]

Large buffers[>50ms queue delay]

low-bandwidth flows[window <5pkts]

• slightly synchronized;

• high utilization;

• low delay/jitter.

• desynchronized;


• high delay, low jitter.

high-bandwidth flows[window >10pkts]

• desynchronized;

• moderate utilization;

• low delay/jitter.

• severely synchronized;


• high delay/jitter.

Results

There is a virtuous circle:desynchronization permits small buffers; small buffers induce desynchronization.

Illustration: 20 flows

Standard TCP, single bottleneck link, no AQMservice C=1.2 kpkt/sec, RTT=200 ms, #flows N=20

B=20 pkts(small buffer)

B=54 pkts(intermediate buffer)

B=240 pkts(large buffer)


Standard TCP, single bottleneck link, no AQMservice C=12 kpkt/sec, RTT=200 ms, #flows N=200



B=2,400 pkts(large buffer)


Standard TCP, single bottleneck link, no AQMservice C=120 kpkt/sec, RTT=200 ms, #flows N=2000



B=24,000 pkts(large buffer)

Limitations/concerns• Surely bottlenecks are at the access network,

not the core network?– Unwise to rely on this!

– The small-buffer theory works fine for as few as 20 flows

• Perhaps TCP is constrained by receiver window, not by congestion window?– Unwise to rely on this!

Probably there are congestion-constrained flows which cause congestion

– Would small buffers hurt flows that are constrained by receiver window?Not if they respond appropriately.

Further work

• These results have been partially validated(Internet2 & Level3)

• Full validation needs– goodly amount of traffic

– full measurement kit

– ability to control buffer size

• Alternatives to TCP – should pace packet injections

– can achieve desynchronization at all window sizes

time

val1

40 41 42 43 44 45

05

1015

20

20

05

1015

20

200

05

1015

20

2000

Small buffers

As line rate increases– queue size fluctuates more and more rapidly

• so pt depends only on the instantaneous utilization at time t• can model packet arrivals by a Poisson process of rate xt

[Cao+Ramanan 2002]

– queue size distribution depends only on link utilization,not on line rate

time

val1

40 41 42 43 44 45

05

1015

20

20

05

1015

20

200

05

1015

20

2000

queu

e si

ze[0

-15

pkt]

time [0-5 sec]

queueing delay19 ms

queueing delay1.9 ms

queueing delay0.19 ms

time

val1

40 41 42 43 44 45

05

1015

20

20

05

1015

20

200

05

1015

20

2000

Large buffers (queueing delay 200 ms)

• When Nxt<C the queue size is small (C=line rate)

• No packet drops, so TCP increases xt

time

val1

40 42 44 46 48 50

050

100

150

queueapprox

queu

e si

ze[0

-160

pkt

]

time [0-10 sec]




• When Nxt>C the queue fills up and packets begin to get dropped

time

val1

40 42 44 46 48 50

050

100

150

queueapprox

queu

e si

ze[0

-160

pkt

]

time [0-10 sec]




• When Nxt>C the queue fills up and packets begin to get dropped

• TCP may ‘overshoot’, leading to synchronization

time

val1

40 42 44 46 48 50

050

100

150

queueapprox

queu

e si

ze[0

-160

pkt

]

time [0-10 sec]


• Drop probability depends onboth average traffic rate xt and queue size qt

time

val1

40 42 44 46 48 50

050

100

150

queueapprox

queu

e si

ze[0

-160

pkt

]

time [0-10 sec]

System summary• xt = average traffic rate at time t

pt = packet loss probability at time tC = line rate B = buffer size RTT = round trip time N=# flows

• TCP traffic model

• Small buffer queueing model(though this is sensitive to traffic statistics)

• Large buffer queueing model

Stability/instability analysis

• For some values of RTT*C/N, the dynamical system is stable

• For others it is unstable and there are oscillations(i.e. the flows are partially synchronized)

• When it is unstable, we can calculate the amplitude of the oscillations

20 40 60 80 1000.60.8

1.21.4

20 40 60 80 1000.60.8

1.21.4

time

trafficrate

Nxt/C

0.5 1 1.5 2

-4

-3

-2

-1

Instability plot

traffic intensity Nx/C

log10 ofpkt loss

probability p

TCP throughput equation

queue equation

extent ofoscillationsin Nx/C

0.5 1 1.5 2

-4

-3

-2

-1

Instability plot

RTT*C/N=4 pkts

RTT*C/N=20 pkts

RTT*C/N=100 pkts

traffic intensity Nx/C

log10 ofpkt loss

probability p

Alternative buffer-sizing rules

b25 b100 b400

0.5 1 1.5

-4

-3

-2

-1

b10 b20 b50

0.5 1 1.5

-4

-3

-2

-1

0.5 1 1.5 2

-4

-3

-2

-1

b50 b1000

0.5 1 1.5-6

-5

-4

-3

-2

-1

r

p

Intermediate buffers buffer = bandwidth*delay / sqrt(#flows)

or Large buffers, no AQM buffer = bandwidth*delay

Large buffers with REDbuffer=bandwidth*delay*{¼,1,4}

Small buffers, no AQMbuffer={10,20,50} pkts

Small buffers, no AQM, ScalableTCPbuffer={50,1000} pkts

Buffer requirements for TCP: queueing theory & synchronization analysis

Documents