Reliable Transport Protocols Should Forbid Reneging

Reliable Transport Protocols Should Forbid Reneging

Nasif EkizPaul D. Amer, Professor

Preethi NatarajanErtugrul Yilmaz

Jon LeightonAbu Rahman

sponsored byU.S. Army Research Lab

2

Outline

• transport layer selective acknowledgments (SACKs)– what are SACKs?– how are SACKs renegable?

• what is the gain in making selective acks non-renegable?• what is the penalty in making selective acks non-renegable?

Conclusion: selective acks for reliable transport protocols (e.g., TCP, SCTP) should be non-renegable

Transport layer receive buffer

3 4 5 7 9 11 12 13

ordered data (ACKed)out-of-order data (SACKed)available space

receiving application

Internet

data sender

receive buffer

4

Types of acknowledgments

• For ordered data - cumulative ACK n– bytes [ ... to n-1 ] (TCP) [RFC 793]

– segments [ ... to n ] (SCTP) [RFC 2960]

• For out-of-order data - selective ACK (SACK) m-n– bytes [ m to n-1 ] (TCP) [RFC 2018]

– segments [ m to n ] (SCTP) [RFC 2960]

prevent unnecessary retransmissions during loss recovery improve throughput when multiple losses in same window

5

Types of ACKs

data senderreceive buffer

data receiver

3

11

ACK 1 2

ACK 2 2

4ACK 2, SACK 4-4 45ACK 2, SACK 4-5 4 56

4 5 6ACK 2, SACK 4-6

ACK 6 3

63 4 5

6

What is reneging?

[RFC 2018]: “The SACK option is advisory, in that, while it notifies the data sender that the data receiver has received the indicated segments, the data receiver is permitted to later discard data which have been reported in a SACK option.”

discarding SACKed data before delivery to the receiver application (or socket) is “reneging”

TCP and SCTP allow reneging - data sender retains copies of all SACKed data until ACKed

7

data senderreceiver buffer

1

1

2

3

4

5

6 4 5 6

data receiver

ACK 1

ACK 2, SACK 4-4

ACK 2, SACK 4-5

ACK 2 2

4

4 5

ACK 2, SACK 4-6

Reneging example

3ACK 3 3

ACK 3, SACK 7-7 77

OS needs memory and reneges

8

Summary of reneging

• Reneging : data receiver SACKs data, and later discards it (i.e., SACK information is “advisory”, not a delivery guarantee)

• Reneging is discouraged but permitted• Data sender keeps data in a send buffer until

cumulatively ACKed (i.e., cum ack is a guarantee)

Special Case for SCTP – out-of-order data already delivered to the application is non-renegable by definition

Special case for SCTP: unordered data


2

data receiver

11

ACK 1

3 3ACK 1, SACK 3-3

3 55

ACK 1, SACK 3-5

9

3 44

ACK 1, SACK 3-4 unordered

6

? ?unordered

Special case for SCTP: unordered data


1

3

3

3

4

5

1

2

3

4

5

data receiver

ACK 1

ACK 1, SACK 3-3

232 5ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

77ACK 7

10

unordered

3 5

6 3 5 6ACK 1, SACK 3-6

unordered

11

We argue that tolerating reneging is wrong

• Suppose SACKs were a guarantee of delivery, not advisory– All SACKs are non-renegable (NR-SACKs)– Data receiver takes responsibility for all selectively acked data– Data sender can remove NR-SACKed data from send buffer

Part I: What is the gain in forbidding selective acks?

always improved send buffer utilization (TCP and SCTP)“Non-renegable selective acks for SCTP” Int'l Conf on Network Protocols (ICNP), Orlando, 10/08

sometimes improved throughput (SCTP)“Throughput analysis of Non-Renegable Selective Acks for SCTP” Computer Communications, 33(16), 10/10

Send buffer utilization (SACK)

13

data sender receive bufferdata receiver

send buffer

11

ACK 1 1100%

221100%

3 3ACK 1, SACK 3-3

2 31100%

44

ACK 1, SACK 3-4 42 31100%

55

ACK 1, SACK 3-5 42 3 5100%

send buffer blocking

2ACK 5

42 3 525%2

42 3 575%

42 3 525%42 3 550%

66100%

Send buffer utilization (NR-SACK)

data sender receiver bufferdata receiver

send buffer

11

ACK 11100%

ACK 7 2262 5 7100%

72 6 8100%ACK 8 8

8

982 7 9 9100%

ACK 9

221100%

4ACK 1, NR-SACK 3-4

42 31100% 4

5ACK 1, NR-SACK 3-5

42 3 5100% 5

52 4 6100% 6ACK 1, NR-SACK 3-6 6

3ACK 1, NR-SACK 3-3

2 31100% 3

62 5 7100% 7ACK 1, NR-SACK 3-7 7

1092 8 10100% 10

ACK 10100% 11108 9 11

no send buffer

blocking

15

NR-SACK ns-2 simulation

16

NR-SACK FreeBSD implementation

17

Send buffer utilization (ns-2)

NR-SACK

As traffic load increases, NR-SACKs better utilize

send buffer

Send

Buff

er U

tiliza

tion

SACKSACK 64KSACK 32K

∞

18

Send buffer utilization (FreeBSD)Se

nd B

uffer

Util

izatio

n

∞

19

Throughput gains (ns-2) (only for SCTP not TCP)

-1% 0% 1% 2% 3% 4% 5% 6%0%

2%

4%

6%

8%

10%

12%

14%

16%SCTP 32K 1000Mbps 45ms (BDP>32K)SCTP 32K 100Mbps 45ms (BDP>32K)SCTP 32K 10Mbps 45ms (BDP>32K)SCTP 32K 3Mbps 45ms (BDP=32.95K)SCTP 32K 2Mbps 45ms (BDP<32K)SCTP 32K 1Mbps 45ms (BDP<32K)

Loss Rate

Thro

ughp

ut G

ain

NR-SACKs never do worse than

SACKs

20

Changing TCP or SCTP to non-reneging protocol is easy:• SACK semantics changed from advisory to permanent• if data receiver needs to renege, data receiver MUST

RESET the connection ( this is the penalty)

Let’s assume transport protocols are designed to forbid data reneging

Part II: What is penalty in forbidding selective acks?

We argue that tolerating reneging is wrong

• Suppose reneging occurs 1 in 100,000 TCP (or SCTP) flows• Case A (current practice): reneging allowed

– 99,999 non-reneging connections underutilize send buffer (and for SCTP may achieve lower throughput)

– 1 reneging connection continues (maybe...)• Case B (proposed change): reneging forbidden

– 99,999 connections have equal or better send buffer utilization (and for SCTP potential greater throughput)

– 1 reneging connection is RESET

Hypothesis: “data reneging rarely if ever occurs in practice”

• Data reneging has never been studied– Does data reneging happen or not?– If reneging happens, how often?

How big is the penalty?answer: depends on how often reneging happens

receive buffer

1

3

1

2

3

3 4 55

22

Data Sender Data Receiver

Router

OS reneges

State of receivebuffer

Detecting TCP reneging at a router

4

ACK 1, SACK 3-4 3 4ACK 1, SACK 3-4

6 3 4 5 6ACK 1, SACK 3-6

ACK 1, SACK 3-6

77ACK 2, SACK 7-7

ACK 2, SACK 3-6 ? reneging detected

Model to detect reneging• Current state (C) and new SACK (N) are compared• 4 possibilities:

SACK 12-17 SACK 12-15 NewCurrent

SACK 12-13 SACK 12-17

SACK 22-25 SACK 12-17

SACK 12-17 SACK 15-20

Model to detect renegingCurrent state (C)New SACK (N)Reneging (R)

Model to detect reneging

CAIDA* trace TCP flow filter

RenegDetect

tsharkeditcapmergecap

~4600 lines of C codeACK reordering check

TCP flows with SACKs reneging?

yesorno

.pcap

*Cooperative Association for Internet Data Analysis

Model verification• RenegDetect was tested with synthetic TCP flows

– created reneging flows with text2pcap– all reneging flows were identified correctly

• RenegDetect was tested with real TCP flows from CAIDA Internet traces– at first, reneging seemed to occur frequently– on closer inspection, we found that many SACK

implementations are incorrect !

“Misbehaviors in TCP SACK Generation” (Ekiz, Rahman, Amer) (ACM SIGCOMM Computer Communication Review, April 2011)

Incorrect SACK implementations

Operating System MisbehaviorA B C D E F G

FreeBSD 5.3, 5.4 Y YLinux 2.2.20 (Debian 3) YLinux 2.4.18 (Red Hat 8) YLinux 2.4.22 (Fedora 1) YLinux 2.6.12 (Ubuntu 5.10) YLinux 2.6.15 (Ubuntu 6.06) YLinux 2.6.18 (Debian 4) YOpenBSD 4.2, 4.5, 4.6, 4.7 Y YOpenSolaris 2008.05 Y YOpenSolaris 2009.06 Y YSolaris 10 YWindows 2000 Y Y Y Y YWindows XP Y Y Y Y YWindows Server 2003 Y Y Y Y YWindows Vista Y YWindows Server 2008 Y YWindows 7 Y Y

• Event A: TCP flow reneges• Hypothesis:

• We want to design an experiment which rejects H0 with 95% confidence to conclude

• Our experiment will observe n TCP flows hoping to NOT find even a single instance of reneging

• Using MAPLE, n ≥ 299,572

Experiment design – how to “prove” reneging does not happen?

31

Questions ?

(thank you)

TCP Send buffer utilization (SACK)

32

data sender receiver buffer

1

2

3

4

5

6

2

data receiver

ACK 1

ACK 1, SACK 3-3

ACK 5

ACK 1, SACK 3-4

ACK 1, SACK 3-5

send buffer

1

21

2 31

42 31

42 3 5

42 3 5

42 3 5

42 3 5

6

100%100%

100%100%

100%

75%

50%

25%

100%

send buffer blocking

1

3

32 4 5

3 4

3 4 5

3 4 5

3 4 5

3 4 5

ACK 6 6

TCP Send buffer utilization (NR-SACK)

33

data sender receiver buffer

1

2

3

4

5

8

data receiver

ACK 1

ACK 1, NR-SACK 3-3

ACK 7

ACK 1, NR-SACK 3-4

ACK 1, NR-SACK 3-5

send buffer

1

21

2 31

42 31

42 3 5

52 4 6

62 5 7

72 6 8

100%

100%

100%100%

100%

100%100%

100%

82 7 9100%

1

3

3 4

5 3 4

5 3 4 6

5 3 4 6 7

5 2 3 4 6 7

8

6

7

2

ACK 1, NR-SACK 3-6

ACK 1, NR-SACK 3-7

9

100% 10 9

10

ACK 9

ACK 10

62 5 7100%

ACK 8

92 8 10

100% 11108 9 11

NO send buffer blocking

Data reneging in OSes

• Reneging in Linux (version 2.6.28.7)– tcp_prune_ofo_queue() deletes out-of-order data

• Reneging in FreeBSD, Mac OS– net.inet.tcp.do_tcpdrain sysctl turns reneging on/off– tcp_drain() deletes out-of-order data

Data reneging in Linux

3. Inferring the state of receive buffer

TCP Segments with n SACK options

Enough space for another SACK

option

Not enough space for another SACK

option

n=1 ~88% 0%

n=2 ~11% 0%

n=3 0.7% 0.20%

n=4 n/a 0.15%

Total number of TCP segments 780,798 (100%)

3. Inferring the state of receive buffer

TCP Segments with n SACK options

Enough space for another SACK

option

Not enough space for another SACK

option

n=1 ~88% 0%

n=2 ~11% 0%

n=3 0.7% 0.20%

n=4 n/a 0.15%

Total number of TCP segments 780,798 (100%)

38

NR-SACK Negotiation

INIT – NR-SACKs Supported

INIT-ACK – NR-SACKs Supported

COOKIE-ACK

COOKIE-ECHO

SCTP Data Transfer

SCTP Association Startup

DATA

NR-SACKs(cum-ack,gap-ack,nr-gap-ack,dup-

TSN)

39

NR-SACK Chunk

Type = 0x10 Chunk Flags Chunk Length

Cumulative TSN Ack

Advertised Receiver Window Credit

Number of NR-Gap-Ack Blocks = MNumber of Gap Ack Blocks = N

Gap Ack Block #1 Start Gap Ack Block #1 End

Duplicate TSN 1

Duplicate TSN X

Gap Ack Block #N Start Gap Ack Block #N End

Number of Duplicate TSNs = X Reserved

NR-Gap Ack Block #1 Start NR-Gap Ack Block #1 End

NR-Gap Ack Block #M Start NR-Gap Ack Block #M End

ACK

SACK

NR-SACK

D-SACK

40

When is data non-renegable? case 2: multistreaming


data receiver

SID : Stream Identifier SSN : Stream Sequence Number

41



11

data receiver

ACK 1 SID: 1 SSN: 1


42



1

1

2

data receiver

ACK 1 SID: 1 SSN: 1

SID: 2 SSN: 1


43



1

3

1

2

3

data receiver

ACK 1

ACK 1, SACK 3-3

SID: 1 SSN: 1

SID: 2 SSN: 1

SID: 1 SSN: 2


44



1

3

4

1

2

3

4

data receiver

ACK 1

ACK 1, SACK 3-3 ACK 1, SACK 3-4

SID: 1 SSN: 1

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2


45



1

3

4

1

2

3

4

5

data receiver

ACK 1


SID: 1 SSN: 1

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2

SID: 1 SSN: 3


? ?

46



1

3

4

4 5

1

2

3

4

5

data receiver

ACK 1


ACK 1, SACK 3-5

SID: 1 SSN: 1

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2

SID: 1 SSN: 3


47



1

3

4

4

4

5

6

1

2

3

4

5

6

data receiver

ACK 1


ACK 1, SACK 3-5

ACK 1, SACK 3-6

SID: 1 SSN: 1

4 6

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2

SID: 1 SSN: 3

SID: 2 SSN: 3


48



1

3

4

4

4

5

6

1

2

3

4

5

6

242 6

data receiver

ACK 1

ACK 1, SACK 3-3

ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

ACK 1, SACK 3-6

SID: 1 SSN: 1

4 6

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2

SID: 1 SSN: 3

SID: 2 SSN: 3

SID: 2 SSN: 1


49



1

3

4

4

4

5

6

1

2

3

4

5

6

7

242 6

7

data receiver

ACK 1

ACK 1, SACK 3-3

ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

ACK 1, SACK 3-6

SID: 1 SSN: 1

4 6

ACK 7

SID: 2 SSN: 1

SID: 1 SSN: 2

SID: 2 SSN: 2

SID: 1 SSN: 3

SID: 2 SSN: 3

SID: 2 SSN: 1

SID: 1 SSN: 4


50

When is data non-renegable? Case 3: OS guarantee

Data SenderReceiver Buffer

Data Receiver

51



11

Data Receiver

ACK 1

52



1

1

2

Data Receiver

ACK 1

53



1

3

1

2

3

Data Receiver

ACK 1

ACK 1, SACK 3-3

54



1

3

3 4

1

2

3

4

Data Receiver

ACK 1


55



1

3

3

3

4

4 5

1

2

3

4

5

Data Receiver

ACK 1


ACK 1, SACK 3-5

56



1

3

3

3

3

4

4

4

5

5

1

2

3

4

5

66

Data Receiver

ACK 1


ACK 1, SACK 3-5

ACK 1, SACK 3-6

3 4 5 6

57



1

3

3

3

3

4

4

4

5

5

1

2

3

4

5

6

22

6

Data Receiver

ACK 1

ACK 1, SACK 3-3

ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

ACK 1, SACK 3-6

3 4 5 6

3 4 5 6

58



1

3

3

3

3

4

4

4

5

5

1

2

3

4

5

6

7

2

6

7

Data Receiver

ACK 1

ACK 1, SACK 3-3

ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

ACK 1, SACK 3-6

3 4 5 6

ACK 7

2 3 4 5 6

59



1

3

3

3

3

4

4

4

5

5

1

2

3

4

5

6

7

2

6

7

Data Receiver

ACK 1

ACK 1, SACK 3-3

ACK 6

ACK 1, SACK 3-4

ACK 1, SACK 3-5

ACK 1, SACK 3-6

3 4 5 6

ACK 7

*

* OS guarantees not to renege

2 3 4 5 6

Reliable Transport Protocols Should Forbid Reneging

Documents

data receiver ack

data receiver sacks

reneging data sender

sacked data

data receivers

data cumulative ack

ack values

later discard data