Experimental analysis of the impact of RTT differences on ... · Experimental analysis of the impact of RTT differences on the throughput of two ... Simulations in the Ns2 network

Individual Research Assignment, June 2008

Committee

Dr. ir. Pieter-Tjerk de Boer

Dr. ir. Geert Heijenk

Experimental analysis of the

impact of RTT differences on

the throughput of two

competing TCP flows

The network phase effect explored

Maarten Burghout

2

Abstract Two TCP flows sharing a common bottleneck link, obtain a fraction of the available throughput,

based on the (inverse) ratio of their roundtrip times (RTT): the higher the RTT, the less throughput.

This report investigates how a higher RTT can deliver a higher throughput, in contrast with normal

TCP calculation rules.

Underlying cause for this deviation of regular throughput predictions is the fact that modern TCP

traffic shows a strong periodicity at bottleneck links, as a result of most packets carrying maximum

payload. Traffic sources that tune their RTT to be in phase with the transmissions at that bottleneck

are known to obtain a higher throughput at that link. This phenomenon is called the Network Phase

Effect.

In this report, results of a series of experiments that address this phenomenon are discussed. A

physical implementation of a simple network was constructed to further investigate preliminary

simulation results. A theoretical analysis is also given, with conditions for the phase effect to occur

derived and tested against physical experiments. These experiments confirm the possibility of a

higher throughput on a bottleneck link, as a result of RTT tuning according to phase effect

conditions.

3

Table of contents

1 INTRODUCTION ....................................................................................................................................... 5

1.1 SCOPE ...................................................................................................................................................... 5

2 PROBLEM DEFINITION ............................................................................................................................. 6

2.1 SIMULATION SETUP .................................................................................................................................... 6

2.2 NS2 SIMULATION RESULTS ........................................................................................................................... 7

3 ANALYSIS ................................................................................................................................................. 8

3.1 NEUTRAL SITUATION ................................................................................................................................. 10

3.2 CONGESTION WINDOW INCREASE ................................................................................................................ 10

3.3 DELAY .................................................................................................................................................... 10

3.4 THE NETWORK PHASE EFFECT ...................................................................................................................... 11

3.5 NS2 SETUP RESULTS EXPLAINED ................................................................................................................... 11

4 EXPERIMENTS ........................................................................................................................................ 13

4.1 THE ASSIGNMENT: TESTS TO BE CONDUCTED ................................................................................................. 13

4.2 TEST ENVIRONMENT ................................................................................................................................. 13

4.3 MEASUREMENT METHOD........................................................................................................................... 13

4.4 COMPARISON TO NS2 SETUP AND PREDICTIONS ............................................................................................. 14

4.5 DIFFERENT TCP FLAVORS ........................................................................................................................... 14

5 RESULTS ................................................................................................................................................ 16

5.1 DATA PROCESSING .................................................................................................................................... 16

5.2 PHASE EFFECT VISIBLE ............................................................................................................................... 16

5.3 MEASUREMENT ABNORMALITIES ................................................................................................................. 17

5.3.1 Small payload packets .................................................................................................................... 17

5.3.2 Terminated measurements ............................................................................................................. 17

5.4 PHASE EFFECT FOR DIFFERENT TCP FLAVORS .................................................................................................. 18

5.4.1 Equal tripping points ....................................................................................................................... 19

5.4.2 Bandwidth sharing ratio ................................................................................................................. 19

5.4.3 Low delay and Scalable ................................................................................................................... 20

5.5 COMPARISON OF DIFFERENT FLAVORS .......................................................................................................... 20

5.6 EQUAL DELAY SETTINGS ............................................................................................................................. 21

6 CONCLUSIONS ....................................................................................................................................... 24

6.1 PHASE EFFECT POSSIBLE IN LAB ENVIRONMENT ............................................................................................... 24

6.2 DIFFERENT TCP FLAVORS AGAINST EACH OTHER ............................................................................................. 24

6.3 PREDICTIONS ON (SEMI-)PUBLIC NETWORKS .................................................................................................. 24

APPENDIX A - DEFINING GOODPUT................................................................................................................ 25

APPENDIX B – NS2 GIVES FAULTY/UNEXPECTED BEHAVIOR IN A SPECIFIC CASE ............................................ 26

WHY................................................................................................................................................................... 26

NETWORK SETUP ................................................................................................................................................... 26

NETWORK PROPERTIES ........................................................................................................................................... 26

DESCRIPTION OF THE PROBLEM ................................................................................................................................ 26

TEST RESULTS ....................................................................................................................................................... 27

Both flows start at 0 ..................................................................................................................................... 27

4

Starting both flows separate ........................................................................................................................ 28

Conclusion .................................................................................................................................................... 29

LITERATURE ................................................................................................................................................... 30

5

1 Introduction In today’s network topology, a not-uncommon scenario has two or more TCP flows sharing a

common bottleneck link, e.g. multiple users on a LAN, sharing a DSL link. When competing for

bandwidth on that bottleneck, the round-trip time (RTT) is a determining factor for the throughput

obtained by those flows.

In general, a higher RTT means a lower throughput. Under general conditions, flows share the

available bandwidth by the inverse of their RTT ratio.

In this project, we will investigate how tuning the RTT for a flow can help in obtaining a higher

throughput. Simulations in the Ns2 network simulator suggest that by opting for a higher RTT, a flow

can actually get more throughput than its competitors (which do not tune their RTT). Main focus for

this project is a physical implementation of such a setup and parameter strategy. Measurements will

be done to see whether a higher RTT will give a flow a greater share of the available bandwidth, or

even make it the sole user on the bottleneck, as some simulations suggest.

1.1 Contents

The cause for this assignment was a set of simulation results on the Ns2 network simulator, which

was believed to be erroneous. These simulation results are discussed in chapter 2, with a theoretical

analysis for this (correct!) behavior given in chapter 3. Based on this analysis, a series of experiments

were drawn up, as is discussed in chapter 4. Results of these experiments are shown in chapter 5.

Finally, chapter 6 gives conclusions based on the analytic investigation and practical results.

1.2 Scope

This project was conducted as an Individual Research Assignment (Individuele Onderzoeks Opdracht,

IOO), part of the undergraduate program for Electrical Engineering at the University of Twente. Main

activity was performed in 2005.

6

2 Problem definition While working on his BSc-assignment entitled “TCP synchronisation effect in TCP New Reno and TCP

Hybla”, Bert Baesjou stumbled upon an apparent flaw in simulation results of the Ns2 network

simulator. Since this flaw was not of direct interest for his assignment, he summarized his findings in

a memo, which can be found in appendix B. His findings were worth investigating further, which

therefore led to the formulation of my assignment.

2.1 Simulation setup

Figure is a graphical representation of the network as simulated in Ns2, when the problem occurred.

Figure - Lay-out of network in the Ns2 simulation

The network consists of:

• 2 source nodes (1 and 2) connected to the first router via 100 Mbit/s links

• 2 end nodes (5 and 6) connected to the second router via 100 Mbit/s links

• 2 routers, interconnected via a 100 kbit/s link

Only two flows exist: node 1 sends to node 5, and node 2 to node 6, accompanied by their respective

acknowledgment return flows. All links are lossless, but packets may be dropped at the routers when

their queues fill up to their maximum of 10 packets.

All traffic is TCP (New Reno), packet size is set to 1,000 bytes; network protocol is IPv4.

The 100 Mbit/s ethernet links for the 1 � 5 flow are set to a fixed delay of 5 ms, while delays for the

links for 2 � 6 are variable (and set between 5 and 25 ms). The first flow experiences a smaller RTT

and is therefore expected to be the ‘faster’ connection.

7

Given that the clear bottleneck for both flows is the 100 kbit/s link between the routers, one may

expect a quick fill of the queue at the first router, causing packets to be dropped and the packet

windows of both flows being adjusted in classic TCP form (the ‘saw tooth’).

2.2 Ns2 simulation results

For delays up to 19 ms per link on the variable delay links the goodput is shared equally between the

two links, giving both flows around 50% of the capacity on the 100 kbit/s link. This is shown in Figure

. The TCP windows of both flows increase to 6 (sending 12 packets to the first router) which causes

packets to be dropped since no more than 10 can be accommodated in the queue, the window

decreased to half the size and thereafter increasing to 6 again.

However, when the variable delay is set to 20 ms, the flow between 1 � 5 comes to a complete halt.

The window of node 2 still shows the normal saw tooth form, but now increases to 12 and decreases

to 6. Goodput on the bottleneck is entirely given to the 2 � 6 flow.

This behavior continues to show in the simulation results up to a delay of 25 ms. When increased

further (i.e. 26, 27… ms) the goodput is divided in a 63-37 ratio between 1 �5 and 2 � 6

respectively. Window size reaches 7 (node 1) and 5 (node 2), still adding up to 12 at which point

both nodes notice packet loss due to the queue of 10 packets at the first router.

While it seems awkward that the bottlenecks goodput isn’t shared equally between the competing

flows, there is actually a sound explanation for this behavior, as we will see in the analysis.

0

20

40

60

80

100

0 20 40 60 80 100 120

Go

od

pu

t o

n b

ott

len

eck lin

k in

B/s

Total delay over 4 links for flow from 2 to 6 in ms

Goodput from 1 to 5

Goodput from 2 to 6

Figure - Goodput over the bottleneck link for both flows, for different delay settings on the links from node 2

to node 6

8

3 Analysis The behavior of the setup during simulation in Ns2 seems awkward: a link deliberately made slower

by means of a larger RTT still manages to capture the entire bandwidth available on a bottleneck

link. In theory however, there is a good explanation for this phenomenon, which is known as the

Network Phase Effect.

The analysis is based on the network setup of Figure and uses the following assumptions and

parameters:

• The traffic consists of 1,000 byte packets (including headers)

• Traffic is already flowing and the 10 packet queue at the first router is filled entirely

• Transmission times over the 100 Mbit links are neglected, since these are a magnitude

smaller than other relevant transmission times (i.e. over the bottleneck link)

• The individual delays on all the 100 Mbit links are concatenated1 and set at the source nodes

• We denote the transmission time of a packet over the bottleneck (TP), of the accompanying

ACK packet (TA), the delay set on the bottleneck link (dc) and the total delays for both flows

(d1 and d2) set at source 1 and 2 respectively.

1 This is allowed, since packet loss can only occur at the back of the queue; therefore, only the amount of delay

is of importance, not the location where it is added.

9

Figure - Transmission of two packets over the bottleneck link, without delay (1) and with delay (2)

t = 0

t = dc

t = TA + 2dc

t = TP

t = TP + TA + 2dc

t = TP + TA + 2dc + d2

1

2

d1

t = -TP

d2

Time

Source nodes

Router 1 Router 2

Destination nodes

dc

Packet from source 2

ACK towards source 2

t = TA + dc

2 places available in queue

Packet from source 2

1 place available in

queue

ACK towards source 1

10

To illustrate a chronological sequence of events, Figure shows two packets transmitted over the

bottleneck link.

3.1 Neutral situation

We start our analysis at the first router, where transmission of the first packet (originating in source

1) starts at t = -TP. The packet takes TP to be transmitted over the bottleneck link, so at t = 0 a place

in the queue becomes available.

The first packet continues to its destination node, where it generates an ACK reply. This ACK travels

back to source 1, taking TA back over the bottleneck. Since an additional delay of dc was set on the

bottleneck (in both ways!), source 1 receives the ACK packet at t = TA + 2dc

If receiving this ACK allows source 1 to send a new packet to the first router, this packet will fill the

free spot in the queue (indicated in Figure at �). Source 1 thereby has the same amount of packets

in the queue as it had at t = -TP. In this way, each packet sent over the bottleneck will allow for its

source to queue a new packet, thereby maintaining the amount of bandwidth each flow gets.

Sources cannot place a packet in an empty spot, created by transmission of an opponent’s packet,

because TCP only allows for packets to be sent when receiving an ACK. As TA is only a fraction of TP, a

source can reclaim its own spot long before the other source can.

3.2 Congestion window increase

When receiving an ACK, TCP usually allows for sending a new packet. Since this is a straightforward

TCP connection, where only packet loss can occur (packet reordering is not possible), some ACKs will

allow an increase of TCP’s congestion window, thus sending (at least) a second packet to the queue.

These two packets arrive at the back end of the queue where only one spot is available, causing the

second packet to be dropped. This packet loss will result in a decrease of the congestion window,

forcing Source 1 to wait for one or more ACKs from its packets still in the queue before sending new

packets to the queue. By then, the other source may have already claimed the empty spots and

temporarily taking a larger share of the bandwidth on the bottleneck.

However, as soon as the queue has been completely filled again, source 2 will also have one of its

packets dropped, allowing for source 1 to reclaim some queue spots.

Over a greater period of time and with other criteria (such as TCP congestion avoidance mechanism)

equal, this sequence of increase and decrease of bandwidth share should level out, with both flows

obtaining an equal goodput.

3.3 Delay

In the basic setup (i.e. without delays) a spot in the queue becomes available at t = 0 and source 1

can have a packet delivered to fill it at t = TA + 2dc. When the ACK that allowed the new packet of

source 1, also allowed for an increase of the send window, source 1 may send two packets.

However, only one spot is available in the queue, so for both packets to be accommodated, source 1

should wait until that second spot becomes available at t = TP. If source 1 had not waited but instead

increased its window and sent two packets to the queue, the second packet would have been

dropped.

11

Clearly, source 1 can have its two packets accepted in the queue if it waits until after t = TP. It should

however not wait too long, as source 2 had the next packet in line and will receive an ACK at t = TP +

TA + 2dc. Therefore, source 1 has to set its additional delay d1 in the window

�� � �� � 2�� � � � ��

Source 2 had also been given an extra delay d2, which extends the above window to

�� � �� � 2�� � � � �� ��

This scenario is also illustrated in Figure , indicated at �.

Of course, this short example will not hold indefinitely. Eventually, source 1 will enlarge its

congestion window even further, which will allow for sending more packets to the queue than there

are empty spots. Source 1 will experience packet loss, reduce its congestion window and has to wait

for a while before it can send its next packet. By then, source 2 may have stepped in and obtained

some extra spots in the queue, thereby taking a larger share of the bandwidth on the bottleneck.

However, as soon as the packet order as described in the example above takes place, source 1 again

will have a change to regain goodput. Source 2 on the other hand always has a disadvantage when

recovering from packet loss. Thus, over a greater period of time, Source 1 can obtain more goodput.

3.4 The network phase effect

The phenomenon described above can be seen as a phase effect in a traffic flow through a network.

Here, phase denotes the time between the end of transmission of a packet out of a stream, and the

arrival of another packet belonging to that stream at the transmission queue.

Because all traffic in this setup is of the same size (i.e. 1000 bytes), all packet transmissions over the

bottleneck link take the same time, making the RTT a multiple of the transmission time.

Adding delay in the amount of this transmission time to another part of the total connection can

make a flow dominant when competing for goodput over a bottleneck link. Extra delay that sets the

RTT of a flow to a non-multiple of the transmission time makes that flow loose out over a greater

period of time. End hosts can use this method to their advantage, especially if they ‘know’ that the

major part of the RTT is caused at one particular part of the link, and provided almost all traffic is of

same size.

Earlier investigations on this effect were conducted by Sally Floyd and Van Jacobsen (Floyd &

Jacobsen, 1992). They established a strong relation between the roundtrip time and arrival of a

packet in a queue. Different drop strategies for TCP were found to be of influence, as was the

addition of random (i.e. out-of-phase) traffic.

3.5 Ns2 setup results explained

Adding a delay for source 1 in the amount of one transmission time does not fully explain why this

allows source 1 to take more spots in the queue and thereby more goodput over the bottleneck link.

If both packets that left a vacant spot in the queue belonged to source 1, then no gain is obtained:

source 1 still has as many packets in the queue as it had when transmission of its packet began. But

if packet 2 originated from source 2, source 1 has gained an additional packet over source 2, and

thereby also gained some goodput.

12

As we saw, source 1 only gains a larger share if it steals a spot from the other source, which can only

occur if that source does not reclaim its vacant queue position in time.

Calculating for the Ns2 simulation setup, the total delay for the flow from node 2 to node 6 should

be between 74.8 and 100 ms. Since the delay was set distributed over the various links (and in

multiples of 1 ms), individual delay settings of 20 ms to 25 ms should result in a strong phase effect

condition, which is exactly shown in Figure .

As the term ‘phase’ implies, adding an extra transmission time �� to the total delay setting should

yield similar results. The theory described above supports this, but the Ns2 experiments did not

investigate higher delay settings. For the physical world experiments in this assignment, we will

explore those delay settings.

13

4 Experiments

4.1 The assignment: Tests to be conducted

First aim within this assignment was to create a close physical copy of the setup used in the Ns2

simulations. Using this setup, initial tests were to be conducted using the parameters of the

simulation and determine how close the measured behavior aligns with Ns2 results.

Next, and most important, the setup would be used to (try to) show the ‘network phase effect’.

Once established, it was to be determined within what range of delays the effect would show.

Finally, possibilities for application in (semi-)public network would be explored and possibly tested.

4.2 Test environment

Physical implementation of the Ns2 setup, as shown in Figure , was created using six identical PC-

compatibles, running Debian Linux. An IPv4 subnet was created, IP range 10.0.0.0/16.

Network infrastructure consisted of 905TX series 3Com network interfaces (either onboard the

mainboards or as PCI cards). Standard CAT5 UTP cables were used.

The bottleneck link between the two routers was established by means of an SLIP (RFC 1055) link,

operating at 115,200 bps, over a 3-wire null modem cable. The SLIP protocol adds a start- and stop

bit to each byte, putting 10 bits on the wire for every byte of user data. Furthermore, each datagram

is preceded and concluded by an additional symbol.

SLIP also uses two special characters (ESC and END), which must be ‘escaped’ when the same byte is

present in the user data. Payload for all traffic is however strictly chosen to avoid these characters.

Occasionally, a byte in the TCP- or IP-header has to be ‘escaped’ but it is highly unlikely that multiple

need for escaping causes a significant increase in transmission time, relative to other packets, since

each escaped character adds about 87 µs or 1 per mille.

Each datagram transmitted over SLIP is preceded and concluded by an END character, thus for each

IP datagram an additional 2 bytes are sent.

Traffic Control Queueing Discipline (“tc qdisc”) is used to limit (IP-)packet size to 1000 bytes and to

set the delays for the links.

Netcat is used to create the desired traffic (in most tests an endless stream of zeroes, to avoid

escape characters).

4.3 Measurement method

The main interest of these tests is the effect of the additional delay added to links on the phase

effect. Therefore sequential runs, each with an increase in delay, were conducted in an automated

manner. Cron (a daemon to execute scheduled commands) was used at all nodes to start and stop

traffic, and for logging. Since cron has a one-minute resolution for jobs, the measurements had to be

adopted to function accordingly.

For the main runs, the flow from node 1 � 5 (experiencing a constant delay of 5 ms per link, totaling

to 20 ms for the whole flow) was started first, with the other flow, set to the appropriate delay for

that run, following a minute later. This method was selected, because a simultaneous start (i.e.

14

within the same ‘cron-minute’) might result in a winner, should one node start slightly early. Starting

the adjustable flow a minute late ensured an established flow of traffic was already present, and

measurements could be done on the amount of goodput the second link was able to conquer from

the first.

All flows were logged at all nodes using the tcpdump tool. Main source for analysis were the logs of

the first router, at the egress point to the SLIP link.

Since the main focus of this project was on goodput in phase-effect affected links, the total goodput

of both links was calculated during the ‘stable’ period of each measurement, i.e. while both source

nodes were sending traffic. Therefore, goodput was calculated over the period between the 60th

and 400th second of each run, thereby ignoring the first minute, when only the first link was active,

and the latter part of transmissions, where the flow was terminated by the cron scripts thus no

longer containing homogenous traffic.

4.4 Comparison to Ns2 setup and predictions

The physical layout was chosen to match the Ns2 setup as close as possible, but slight differences

could not be avoided. Most obvious is the use of a SLIP link to act as bottleneck link, which not only

has a somewhat higher transmission speed but also increases overhead per datagram.

The main parameter influenced by this setup is the transmission time over the bottleneck link. IP-

packets from source nodes (1,000 bytes each) take 10,020 bits over the SLIP link, which results in a

transmission time of 86.98 ms. This number increased slightly when header bytes need to be

escaped. ACK-packets on their way back amount to between 52 and 80 bytes (depending on SACK

usage), taking 4.69 to 7.12 ms over the bottleneck.

Based on the theory of the network phase effect, the 'slow' flow from source 2 to destination 6

should be dominant in obtaining goodput over the bottleneck from 80,3 ms total delay, up until

104,3 ms. Those delay settings should ensure that, after receiving an ACK, source 2 sends its next

packets no sooner than a second spot in the queue is available, but before source 1 has a chance to

claim one of those spots.

4.5 Different TCP flavors

During analysis of some early results, it became apparent that one of the sending nodes used a

different TCP congestion control mechanism, due to erroneous setup. This was of course an

undesirable situation, so all machines were reconfigured with the same TCP implementation (BIC,

the Linux kernel default at the time).

Analysis of the results of the erroneous setup however showed a small influence on the outcome of

the tests. This prompted for a comparison of 3 widely used TCP congestion control/avoidance

mechanisms:

• TCP BIC

• TCP Reno

• TCP Scalable

Two scenarios were investigated:

15

• The above setup, with both flows using the same TCP flavor

• Different TCP flavors competing against each other, with equal delay settings for both flows

and a common starting time (instead of the one minute head start for flow from node 1 to 5)

16

5 Results

5.1 Data processing

During the experiments, detailed dataflow information was logged at both routers, on all ingress and

egress points. To obtain goodput information per flow, the tcpdump log files of the egress point at

the first router were used. These logs were processed by scripts, mostly in AWK, to determine how

much useful data a source was able transmit over the bottleneck link in the interval between t=65s

(the start of the second flow) and t=400s. The goodput figure per source was obtained by summing

all payloads in this interval; for a justification and definition of this goodput measurement method,

see appendix A. Averaging the goodput over more than 5 minutes filters out any startup effects and

will indicate whether or not a flow was dominant over a longer period of time.

5.2 Phase Effect visible

The data processing described above yields a goodput figure for all delay settings of the second flow.

Figure represents a graphical representation of the network phase effect results.

Figure - Goodput for 2 flows, experiencing the phase effect2

For delay settings from 0 ms to 56 ms, the flow from node 1 to 5 gets around 63% of the total

goodput that both flows obtain on the bottleneck. This link even obtains this larger share for delay

settings of the other flow below 20 ms (the amount of delay it had itself), indicating that this might

be a result of the one minute head start it got. Normal TCP goodput calculation suggest that flows

with an (almost) equal RTT and equal conditions should get an equal amount of goodput, but

apparently being the first flow to fill the queue at the first router gives an advantage over the second

2 The measured values for 76 ms are omitted as a result of a prematurely terminated measurement; an

estimated figure based on partial results is in line with neighbouring results.

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for flow from 2 to 6 in ms

Goodput from 1 to 5

Goodput from 2 to 6

17

flow. Defending spots in the queue is easier than stealing them, because the low delay settings

almost always allow a source to reclaim the vacant spot in the queue that was created by the

transmission of one of its own packets.

Flow 2 (from node 2 to 6) is clearly dominant from 60 to and including 104 ms delay, taking 80% of

the total goodput; a clear example of the network phase effect. This window has the same end as

predicted during our analysis of the phase effect, but starts at considerably smaller delay settings.

This is probably due to buffering at the front of the queue: a packet being sent will free its spot in

the queue before it is transmitted entirely. Based on the difference between the theoretical delay at

which the phase effect start (80 ms), and our measurement (60 ms), this buffer is probably 256

bytes.

Some further conclusions can be drawn from these results:

• The phase effect does not only show up at the predicted interval around a single

transmission time, but also at around twice the transmission time (148 ms up to 192 ms)

• When either of the flows takes a larger part of the bandwidth, it has a clear advantage over

the other flow, but the ratio decreases at higher delay settings (dropping from 80/20 to

65/35 in these tests)

• The ratio between the goodputs remains constant during the delay range at which the phase

effect is present (goodput is constant for 0 to 56 ms, 60 to 104, 108 to 144 and 148 to 192

ms)

5.3 Measurement abnormalities

During the (highly automated) measurements, a few small glitches occurred.

5.3.1 Small payload packets

Close analysis of the captured data showed unintended and unpredicted behavior of the sending

nodes: some packets did not carry a full payload. Netcat should have filled all packets up to the MTU

with input taken from /dev/zero, but apparently something interfered. Due to late discovery and

breakdown of the network setup, no further investigation to this matter could be done.

These smaller packets did however allow for a look at the stability of the network phase effect. Since

these packets have a shorter transmission time over the bottleneck, they may disrupt the phase

effect, as it heavily depends on a constant packet transmission time. Detailed observation however

did not show an interruption of the dominance (if present) of one flow over the other.

On rare occasions, these smaller packets occurred in a strict periodic pattern, being transmitted

every 6.5 or 16.5 seconds. No logical explanation for this strange behavior has been found.

5.3.2 Terminated measurements

A few measurements were cut short due to an inexplicable error. Unfortunately, these flaws were

not discovered until after the network setup had been dismantled, so there was no opportunity to

re-do these measurement. Missing data is left out of the associate graphs, and these occurrences

are indicated in footnotes at those graphs.

18

5.4 Phase effect for different TCP flavors

The results in the previous paragraph were obtained with two TCP BIC flows. To investigate the

effect of different flow control mechanisms employed by various TCP flavors, the above tests were

repeated with both flows set to TCP Reno (a predecessor to BIC) and TCP Scalable (intended for high

bandwidth networks). Figure and Figure show their respective results.

Figure - Phase effect in TCP Reno

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for flow from 2 to 6 in ms

Goodput from 1 to 5

Goodput from 2 to 6

19

Figure - Phase effect in TCP Scalable3

These TCP mechanisms too show the phase effect occurring, but some differences can be observed.

5.4.1 Equal tripping points

All mechanisms have the same tripping points (i.e after 56, 104, 144 and 192 ms). As these points

are entirely determined by the transmission time of a (fully filled) packet, different TCP flavors

should not influence them, which is affirmed by the measurements.

5.4.2 Bandwidth sharing ratio

Reno and BIC share the bandwidth (both during and outside of phase effect periods) in the same

ratio, which becomes more equal for higher delay settings. Scalable on the other hand always

divides the bandwidth 85 to 15 percent. This is caused by Scalable’s low penalty for loss (the

congestion window is reduced to 7/8

th instead of ½ !) and quicker recovery from loss. Those

conditions make packet loss a minor issue for the dominant flow, whereas the dominated flow has

almost no opportunity to steal bandwidth from its competitor.

Figure shows goodput figures for the flow from source 1 to destination 5, for all three investigated

flavors. This plot clearly illustrates both of the above observations.

3 Measurement for 28 ms failed; results omitted

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for flow from 2 to 6 in ms

Goodput from 1 to 5

Goodput from 2 to 6

20

Figure - Goodput comparison of BIC, Reno and Scalable (only one flow plotted)

5.4.3 Low delay and Scalable

Before the first phase effect period (i.e. for small delay settings, up to 54 ms) Scalable has no

constant dominant flow. This seems to be caused by luck: a source that manages to obtain incidental

burst of packets to be transmitted, can maintain that burst easily.

5.5 Comparison of different flavors

Since a slightly different behavior of the three TCP flavors was observed, it would be interesting to

see how these flavors would compete against each other. Since the main focus of these tests would

now be on throughput and not the ability to ‘break in’ on the other link, both flows were started at

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for flow from 2 to 6 in ms

BIC

Reno

Scalable

21

the same time (t=0). See Figure .

As with the previous tests on asymmetrical startup times, Scalable proves to be the most aggressive.

All other protocols loose out to it. BIC and Reno do not show a clear winner.

5.6 Equal delay settings

For comparison, a series of ‘equal opportunity’ tests was done. In these tests, both flows were given

the same delay settings on each link, the same TCP flavor was used at all nodes and both flows

started at the same time.

These tests would show what influence the delay would have on the distribution of goodput without

the phase effect, giving an insight in what part of the phase effect is caused by synchronizing

transmission time and RTT.

Figure , Figure and Figure show the results of these tests with two flows of BIC, Reno and Scalable

respectively.

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for both flows in ms

Reno v. Scalable, Scalable

BIC v. Scalable, Scalable

BIC v. Reno, BIC

BIC v. Reno, Reno

BIC v. Scalable, BIC

Reno v. Scalable, Reno

Figure - Different TCP flavors competing against each other, when started at the same time

22

Figure - Goodput for two TCP BIC streams, started simultaneous and with equal delay settings

Figure - Goodput for two TCP Reno streams, started simultaneous and with equal delay settings

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for both flows in ms

Goodput from 1 to 5

Goodput from 2 to 6

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for both flows in ms

Goodput from 1 to 5

Goodput from 2 to 6

23

Figure - Goodput for two TCP Scalable streams, started simultaneous and with equal delay settings

These results clearly illustrate that the phase effect is not invoked by the kind of TCP flavor used,

only by the unequal delay settings, as all test using equal delay settings yield a random winner, if

any, and independent of the amount of delay set. Reno and BIC share the available bandwidth

almost entirely fairly, especially after the delay setting is greater than one RTT, minus the

(suspected) 256 bytes buffer of the UART interface.

Scalable shows the same behavior as during the phase effect experiments: one flow almost always

outperforms the other, probably due to the ‘forgiving’ nature of its congestion avoidance window

scaling.

0

2000

4000

6000

8000

10000

0 50 100 150 200

Go

od

pu

t o

n S

LIP

lin

k in

B/s

Total roundtrip delay for both flows in ms

Goodput from 1 to 5

Goodput from 2 to 6

24

6 Conclusions

6.1 Phase Effect possible in lab environment

The results of the tests on the phase effect confirm the predictions based on the Ns2 simulations,

which motivated this project.

In addition to the Ns2 predictions, the phase effect was shown to hold for (total) delay times up to

several times the bottleneck link transmission time. TCP Scalable maintained a clear ‘winner/loser’

bandwidth sharing ratio, whereas Reno and BIC see the results of the phase effect shrink for delay of

more than one transmission time.

The experiments of both flows given the same extra delay show no phase effect occuring. From

these experiments, we can conclude that the phase effect only works when some flows do, and

some do not, tune their transmissions to be in phase with transmission periodicity at a bottleneck

link.

TCP Scalable shows similar behavior (i.e. one flow obtaining much more goodput over the bottleneck

link) even without fulfilling phase effect conditions, but this is a result of its congestion window

scaling mechanism and yields a random dominant flow, rather than a predetermined winner as a

result of phase effect timing of the RTT.

6.2 Different TCP flavors against each other

When different TCP implementations were put up to each other, competing for the bottleneck link

capacity under equal circumstances (delay settings and starting time), Scalable proved to be the

most aggressive, gaining the most goodput over Reno and BIC. This indicated that Scalable is not

suitable in a mixed-traffic environment, because its window scaling mechanism violated TCP friendly

rules.

6.3 Predictions on (semi-)public networks

No experiments were conducted on semi-public or public networks. However, the unintentional

errors during the experiments, which caused some IP packets not to be filled up to the MTU, gave a

valuable insight as to how the phase effect would hold in uncontrolled traffic environments. As the

phase effect was shown to hold even when slightly disturbed by smaller packets, we expect the

phase effect to also hold on public networks (where non-MTU traffic is far likelier to occur), provided

that MTU-sized traffic makes up for the bulk of the competing traffic. Thus, the results of the lab

experiments suggest that in an environment where traffic is mostly full-load TCP over IP, the phase

effect can occur and could be invoked by means of delay addition at a source node.

Future work on this subject might focus on the possibility of a phase effect in less controlled

environments, with more than one competing flow.

25

Appendix A - Defining goodput In computer networks analysis, goodput is the figure that represents the amount of useful bits (or

bytes) that have been carried from source to destination, per unit of time. This figure excludes

overhead like headers of transport protocols and retransmissions of lost or corrupt packets, which

are needed for communication to be possible (and are included in throughput figures) but carry no

useful data as far as the application or user is concerned.

In this project, goodput figures are calculated based on the packet flow on the bottleneck link, as

present in the test network. Due to the setup of this network, packets that make it to the bottleneck

will always reach their destination, and so will their acknowledgments. Therefore, it is fair to

determine the goodput at that point.

To calculate goodput in a simple, automated way, there are a couple of options, which give slightly

different results:

• Difference between first and last sequence number.

Packet loss can not occur beyond the measurement point, but can occur before. Therefore,

the first and last packet may not necessarily carry the first and last bytes of data transmitted

within that period of time. Since we only observe a part of a whole communication between

end hosts, out-of-sequence packets at the start or end of the measurement period are

omitted in the calculation of goodput, but may be useful beyond that period.

• Packet count.

This method assumes all packets carry useful data, which is true if when assuming out-of-

sequence packets are still useful beyond our observed period. The network topology ensures

nu duplicate data will ever be transmitted, as all observed packets make it to their

destination (note that packets are observed at the egress of the 10 packet queue, i.e. when

they are put on the bottleneck link). Since the sources had an indefinite amount of data to

transmit, all packets should be filled to carry maximum payload. Goodput is therefore

calculated by multiplying the packet count with the MTU size, minus header sizes.

• Payload count.

Close observation of logged traffic showed some packets not filled to carry the maximum

amount of payload, allowed by the MTU. No logical explanation could be found for this

behavior, but it does affect the goodput figure when calculated according to the packet

count method. Calculating goodput as the sum of all payloads is more correct method,

although differences with the previous method are small, as the payload of a single packet is

much less than the total amount of data transmitted, and occurrence of this behavior is

almost always seldom.

The last method was used for all goodput figures in this report, since it includes all data, even out-of-

sequence, but does not count non-existing payload.

26

Appendix B – NS2 gives faulty/unexpected behavior in a specific case

Bert Baesjou, University of Twente, 24-12-2004

Why

While running tests with the tcp-hybla module some unexpected results were generated. After some

testing it did not seem to be the tcp-hybla module but a part of the tcp suite (at least TCP-Hybla,

TCP-Reno and TCP-New Reno suffer from it). The tests were performed on two separate installations

of ns-2.27, one with changes with respect to the hybla module which was inserted and an other

without any alterations (except for some code printing the congestion window size along with a

timestamp to a file and some standard bugfixes ns2 needs when it comes fresh out of the box).

Network setup

Network properties

• There goes a flow from node 0 to 4 and from node 1 to 5

• No changes are made in the link capacities

• A variable delay is set for the flow between 1 and 5 (this is made the 'slow' link)

• No loss in the network, only at routers when the queue fills up

• Queue size at the router is 10

• Both flows are New Reno

• TTL = 64

Description of the problem

When the delay between node 1 and the first router is 20 or larger until 25, and between the second

router and node 5 is set to the same delay (both 20, both 21, etc.) the link between 1 and 5

completely starves. For values under 20 or above 25 both links show more common and expected

TCP-New-Reno behavior.

27

Test results

Both flows start at 0

Up until 19 ms between node 1 and the fist router, and between the second router and node 5 we

see the following type of expected behavior (Red is the 0->4 link Green is the 1->5 link4)

Between 20 and 25 we see the following unexpected behavior:

4 I believe this is the wrong way round, since this contradicts the figures in the table below

28

After 25 we see the following behavior:

The following table shows the goodput of the both connections for the different links (kb/s), all

figures are rounded.

5 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

0->4 50 50 49 49 49 49 0 1 1 1 1 1 63 63 63 63 63

1->5 50 50 50 50 50 50 99 99 99 99 99 99 37 37 37 37 37

Starting both flows separate

Starting both flows with a gap of 20 seconds gives us the following result (0->4 starts firsts, 1->5 20

seconds later)

29

As one can see the flow 1->5 almost directly starves out.

Conclusion

NS2 has unexpected and unwanted behavior. Strange is that the 'fast' link is starving out (instead of

the one with the higher delays). What the exact cause of this behavior is, is not exactly clear. The

only workaround found is increasing the queue size at the router, but this is not always wanted. A

real solution would only be possible with more insight in the exact code running the simulator,

which means that tracing every move of the simulator has to be done. This unwanted behavior does

not strengthen the thrust in accuracy of the simulations.

30

Literature

Floyd, S., & Jacobsen, V. (1992). Traffic Phase Effects in Packet-Switched Gateways.

Scalable TCP, http://www.deneholme.net/tom/scalable/ (retrieved june 2008)