SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Prototype C. Jin, D. Wei Theory D. Choe (Postech/Caltech), J. Doyle,

SCinet Caltech-SLAC experiments

netlab.caltech.edu/FAST

SC2002 Baltimore, Nov 2002

PrototypeC. Jin, D. Wei

TheoryD. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)

Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.

Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.

Navratil, J. Williams StarLight: T. deFanti, L. Winkler

Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF

Acknowledgments

FAST Protocols for Ultrascale Networks

Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information

Rf (s)

Rb’(s)

l ctyc

)()(1)( tan)(

)()(1-2

tqtttT

wx iid

TCP AQM

Theory

Calren2/Abilene

Chicago

Amsterdam

Geneva

SURFNet

StarLight

WAN in LabCaltech

research & production networks

Multi-Gbps50-200ms delay

Experiment

155Mb/s

slowstart

equilibrium

FASTrecovery

FASTretransmit

timeout

10Gb/s

Implementation

Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS)

Industry Doraiswami (Cisco) Yip (Cisco)

Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA)

Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR)

Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco

People

netlab.caltech.edu

Outline

Motivation Theory

TCP/AQM TCP/IP

Experimental results

netlab.caltech.edu

HEP high speed network

… that must change

netlab.caltech.edu

HEP Network (DataTAG)

NLNLSURFnet

GENEVA

UKUKSuperJANET4

ABILENE

ESNETESNET

CALREN

ItItGARR-B

NewYork

FrFrRenater

STAR-TAP

STARLIGHT

Triangle

2.5 Gbps Wavelength Triangle 2002 10 Gbps Triangle in 2003

Newman (Caltech)

netlab.caltech.edu

Network upgrade 2001-06

’01155

’02622

’032.5

’04 5

’05 10

netlab.caltech.edu

Projected performance

Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps100 sources, 100 ms round trip propagation delay

’01155

’02622

’032.5

’04 5

’05 10

J. Wang (Caltech)

netlab.caltech.edu

Projected performance

Ns-2: capacity = 10Gbps100 sources, 100 ms round trip propagation delay

FAST TCP/RED

J. Wang (Caltech)

netlab.caltech.edu

Outline

Motivation Theory

TCP/AQM TCP/IP

Experimental results

netlab.caltech.edu

Congestion control

Example congestion measure pl(t) Loss (Reno) Queueing delay (Vegas)

netlab.caltech.edu

TCP/AQM

Congestion control is a distributed asynchronous algorithm to share bandwidth

It has two components TCP: adapts sending rate (window) to congestion AQM: adjusts & feeds back congestion information

They form a distributed feedback control system Equilibrium & stability depends on both TCP and AQM And on delay, capacity, routing, #connections

xi(t)TCP: Reno Vegas

AQM: DropTail RED REM/PI AVQ

netlab.caltech.edu

Network model

Rb’(s)

TCP Network AQM

lieR lis

lif link uses source if

lieR lislib link uses source if H

netlab.caltech.edu

Vegas model

Rb’(s)

TCP Network AQM

netlab.caltech.edu

Methodology

Protocol (Reno, Vegas, RED, REM/PI…)

Equilibrium Performance

Throughput, loss, delay

Fairness Utility

Dynamics Local stability Cost of stabilization

))( ),(( )1(

txtpGtp

txtpFtx

netlab.caltech.edu

Summary: duality model

subject to

)( max0

Flow control problem

TCP/AQM Maximize utility with different utility functions

Primal-dual algorithm

))( ),(( )1(

txtpGtp

txtpFtx

VegasDropTail, RED, REM

Theorem (Low 00): (x*,p*) primal-dual optimal iff 0 ifequality with ** lll pcy

netlab.caltech.edu

Equilibrium of VegasNetwork

Link queueing delays: pl

Queue length: clpl

Sources

Throughput: xi

E2E queueing delay : qi

Packets buffered:

Utility funtion: Ui(x) = i di log x Proportional fairness

iiii dqx

netlab.caltech.edu

Persistent congestion

Vegas exploits buffer process to compute prices (queueing delays)

Persistent congestion due to Coupling of buffer & price Error in propagation delay estimation

Consequences Excessive backlog Unfairness to older sources

Theorem (Low, Peterson, Wang ’02)

A relative error of i in propagation delay estimation distorts the utility function to

iiiiiiiii xdxdxU log)1()(ˆ

netlab.caltech.edu

Validation (L. Wang, Princeton)

Source rates (pkts/ms)# src1 src2 src3 src4 src51 5.98 (6) 2 2.05 (2) 3.92 (4)3 0.96 (0.94) 1.46 (1.49) 3.54 (3.57)4 0.51 (0.50) 0.72 (0.73) 1.34 (1.35) 3.38 (3.39)5 0.29 (0.29) 0.40 (0.40) 0.68 (0.67) 1.30 (1.30) 3.28

(3.34)

# queue (pkts) baseRTT (ms)1 19.8 (20) 10.18 (10.18)2 59.0 (60) 13.36 (13.51)3 127.3 (127) 20.17 (20.28)4 237.5 (238) 31.50 (31.50)5 416.3 (416) 49.86 (49.80)

netlab.caltech.edu

Methodology

Protocol (Reno, Vegas, RED, REM/PI…)

Equilibrium Performance

Throughput, loss, delay

Fairness Utility

Dynamics Local stability Cost of stabilization

))( ),(( )1(

txtpGtp

txtpFtx

netlab.caltech.edu

TCP/RED stability

Small effect on queue AIMD Mice traffic Heterogeneity

Big effect on queue Stability!

netlab.caltech.edu

Stable: 20ms delay

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (ms)

individual window

Window

Ns-2 simulations, 50 identical FTP sources, single link 9 pkts/ms, RED marking

netlab.caltech.edu

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

800Instantaneous queue

time (ms)

Stable: 20ms delay

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (ms)

individual window

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (ms)

individual window

average window

Window

netlab.caltech.edu

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (10ms)

individual window

Unstable: 200ms delay

Window

netlab.caltech.edu

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (10ms)

individual window

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

70Window

time (10ms)

individual window

average window

Unstable: 200ms delay

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

time (10ms)

QueueWindow

netlab.caltech.edu

Other effects on queue

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

time (ms)

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

time (10ms)

0 10 20 30 40 50 60 70 80 90 1000

800Instantaneous queue (50% noise)

time (sec)

30% noise

0 10 20 30 40 50 60 70 80 90 1000

800Instantaneous queue (50% noise)

time (sec)

30% noise

0 10 20 30 40 50 60 70 80 90 1000

time (sec)

Instantaneous queue (pkts)

avg delay 16ms

0 10 20 30 40 50 60 70 80 90 1000

time (sec)

Instantaneous queue (pkts)

avg delay 208ms

netlab.caltech.edu

Theorem (Low et al, Infocom’02) Reno/RED is stable if

Stability: Reno/RED

Rb’(s)

TCP Network AQM

TCP: Small Small c Large N

RED: Small Large delay

netlab.caltech.edu

Stability: scalable control

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Paganini, Doyle, Low, CDC’01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology

netlab.caltech.edu

Stability: Vegas

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Choe & Low, Infocom’03) Provided R is full rank, feedback loop is locally stable if

), ;( max 20 kMTx ii

netlab.caltech.edu

Stability: Stabilized Vegas

)()(1)( tan)(

1 )()(1-

tTx iid

tqtxi ii

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Theorem (Choe & Low, Infocom’03) Provided R is full rank, feedback loop is locally stable if

),( max aTx ii

netlab.caltech.edu

Stability: Stabilized Vegas

)()(1)( tan)(

1 )()(1-

tTx iid

tqtxi ii

Rb’(s)

TCP Network AQM

l ctyc

tp )(1

Application Stabilized TCP with current routers Queueing delay as congestion measure has right scaling Incremental deployment with ECN

netlab.caltech.edu

Fast AQM Scalable TCP

Equilibrium properties Uses end-to-end delay and loss Achieves any desired fairness, expressed by utility function Very high utilization (99% in theory)

Stability properties Stability for arbitrary delay, capacity, routing & load Robust to heterogeneity, evolution, … Good performance

Negligible queueing delay & loss (with ECN) Fast response

netlab.caltech.edu

Implementation

Sender-side kernel modification Build on

Reno, NewReno, SACK, Vegas New insights

Difficulties due to Effects ignored in theory Large window size

First demonstration in SuperComputing Conf, Nov 2002 Developers: Cheng Jin & David Wei FAST Team & Partners

netlab.caltech.edu

Outline

Motivation Theory

TCP/AQM TCP/IP

Experimental results WAN in Lab

netlab.caltech.edu

Network

(Sylvain Ravot, caltech/CERN)

netlab.caltech.edu

FAST BMPS

Internet2Land Speed

Record

Baltim

unnyvale

#flows

FAST Standard MTU Throughput averaged over > 1hr

netlab.caltech.edu

FAST BMPS

flows BmpsPeta

ThruputMbps

Distancekm

Delayms

Durations

TransferGB

Alaska-Amsterdam

9.4.2002

1 4.92 401 12,272 - - 13 0.625 Fairbanks, AL – Amsterdam,

MS-ISI29.3.2000

2 5.38 957 5,626 - 4,470 82 8.4 MS, WA – ISI, Va

Caltech-SLAC19.11.2002

1 9.28 925 10,037 180 1,500 3,600 387 CERN -Sunnyvale

2 18.03 1,797 10,037 180 1,500 3,600 753 CERN -Sunnyvale

7 24.17 6,123 3,948 85 1,500 21,600 15,396 Baltimore -Sunnyvale

Mbps = 106 b/s; GB = 230 bytes

netlab.caltech.edu

Aggregate throughput

1 flow 2 flows 7 flows 9 flows 10 flows

Average utilization

88%FAST Standard MTU Utilization averaged over > 1hr

1hr 1hr 6hr 1.1hr 6hr

Experiment

Sunnyvale Baltimore

Chicago

Geneva

3000km 1000km

C. Jin, D. Wei, S. LowFAST Team and Partners

Internet: distributed feedbacksystem Rf (s)

Rb’(s)

TCP AQM

Theory

FAST TCP Standard MTU Peak window = 14,255 pkts Throughput averaged over > 1hr 925 Mbps single flow/GE card

9.28 petabit-meter/sec 1.89 times LSR

8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR

21TB in 6 hours with 10 flows

Implementation Sender-side modification Delay based

Highlights

a-Sunnyv

Baltim

#flows

netlab.caltech.edu

FAST vs Linux TCP

flows BmpsPeta

ThruputMbps

Distancekm

Delayms

Durations

TransferGB

Linux TCPtxqueulen=100

1 1.86 185 10,037 180 1,500 3600 78 CERN - Sunnyvale

1 2.67 266 10,037 180 1,500 3600 111 CERN - Sunnyvale

FAST19.11.2002

1 9.28 925 10,037 180 1,500 3600 387 CERN -Sunnyvale

2 3.18 317 10,037 180 1,500 3600 133 CERN - Sunnyvale

2 9.35 931 10,037 180 1,500 3600 390 CERN - Sunnyvale

FAST19.11.2002

2 18.03 1,797 10,037 180 1,500 3600 753 CERN -Sunnyvale

Mbps = 106 b/s; GB = 230 bytes; Delay = propagation delayLinux TCP expts: Jan 28-29, 2003

netlab.caltech.edu

Aggregate throughput

Linux TCP Linux TCP FAST

Average utilization

92%FAST Standard MTU Utilization averaged over 1hr

txq=100 txq=10000

Linux TCP Linux TCP FAST

netlab.caltech.edu

Effect of MTU

(Sylvain Ravot, Caltech/CERN)

Linux TCP

Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S.

Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J.

Navratil, J. Williams StarLight: T. deFanti, L. Winkler

Major sponsorsARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF

Acknowledgments

netlab.caltech.edu

FAST URL’s

FAST websitehttp://netlab.caltech.edu/FAST/

Cottrell’s SLAC websitehttp://www-iepm.slac.stanford.edu/monitoring/bulk/fast

netlab.caltech.edu

Outline

Motivation Theory

TCP/AQM TCP/IP Non-adaptive sources Content distribution

Implementation WAN in Lab

fiber spool

Max path length = 10,000 kmMax one-way delay = 50ms

: server

: router

electroniccrossconnect(Cisco 15454)

EDFA EDFA

500 km

netlab.caltech.edu

Unique capabilities

WAN in Lab Capacity: 2.5 – 10 Gbps Delay: 0 – 100 ms round trip

Configurable & evolvable Topology, rate, delays, routing Always at cutting edge

Risky research MPLS, AQM, routing, …

Integral part of R&A networks Transition from theory, implementation,

demonstration, deployment Transition from lab to marketplace

Global resource

(a) Physical network

netlab.caltech.edu

Unique capabilities

Global resource

(b) Logical network

netlab.caltech.edu

Global resource

Unique capabilities

Calren2/Abilene

Chicago

Amsterdam

Geneva

SURFNet

StarLight

WAN in LabCaltech

Experiment

netlab.caltech.edu

Coming together …

Clear & presentNeed

Resources

netlab.caltech.edu

Clear & presentNeed

Coming together …

Resources

netlab.caltech.edu

Clear & presentNeed

Coming together …

Resources FASTProtocols

FAST Protocols for Ultrascale Networks

Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information

Rf (s)

Rb’(s)

l ctyc

)()(1)( tan)(

)()(1-2

tqtttT

wx iid

TCP AQM

Theory

Calren2/Abilene

Chicago

Amsterdam

Geneva

SURFNet

StarLight

WAN in LabCaltech

Experiment

155Mb/s

slowstart

equilibrium

FASTrecovery

FASTretransmit

timeout

10Gb/s

Implementation

Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS)

Industry Doraiswami (Cisco) Yip (Cisco)

Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA)

Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR)

Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco

People

netlab.caltech.edu

Backup slides

netlab.caltech.edu

TCP Congestion States

Established

Slow Start

High Throughput

ack for syn/ack cwnd > ssthreshpacing? gamma?

netlab.caltech.edu

From Slow Start to High Throughput

Linux TCP handshake differs from the TCP specification

Is 64 KB too small for ssthresh? 1 Gbps x 100 ms = 12.5 MB !

What about pacing? Gamma parameter in Vegas

netlab.caltech.edu

Established

Slow Start

High Throughput

FAST’sRetransmitTime-out *

3 dup acks

retransmision timer fired

netlab.caltech.edu

High Throughput

Update cwnd as follows: +1 pkts in queue < + kq’ - 1 otherwise

Packet reordering may be frequent Disabling delayed ack can generate

many dup acks Is THREE the right number for Gbps?

netlab.caltech.edu

Established

Slow Start

High Throughput

FAST’sRecovery

FAST’sRetransmit

3 dup acks

retransmit packetrecord snd_nxt

reduce cwnd/ssthresh

snd_una > recorded snd_nxt

send packet if in_flight < cwnd

netlab.caltech.edu

When Loss Happens

Reduce cwnd/ssthresh only when loss is due to congestion

Maintain in_flight and send data when in_flight < cwnd

Do FAST’s Recovery until snd_una >= recorded snd_nxt

netlab.caltech.edu

Established

Slow Start

High Throughput

FAST’sRecovery

FAST’sRetransmitTime-out *

3 dup acks

netlab.caltech.edu

When Time-out Happens

Very bad for throughput Mark all unacknowledged pkts as lost and

do slow start Dup acks cause false retransmits since

receiver’s state is unknown Floyd has a “fix” (RFC 2582).

netlab.caltech.edu

Established

Slow Start

High Throughput

FAST’sRecovery

FAST’s RetransmitTime-out *

ack for syn/ackcwnd > ssthresh

3 dup acks

snd_una > recorded snd_nxt

netlab.caltech.edu

Individual Packet States

Birth Sending In Flight Received

Queued Dropped Buffered

Freed Delivered

queueing

out of order queueand no memoryack’d

SCinet Bandwidth Challenge

Experiment

Sunnyvale Baltimore

Chicago

Geneva

3000km 1000km

C. Jin, D. Wei, S. LowFAST Team and Partners

Internet: distributed feedbacksystem Rf (s)

Rb’(s)

TCP AQM

Theory

22.8.02IPv6

9.4.021 flow

29.3.00multiple

-Geneva

Baltim

eSC20021 flow

SC20022 flows

SC200210 flows

I2 LSR

Sunnyvale

-Geneva

FAST TCP Standard MTU Peak window = 14,100 pkts 940 Mbps single flow/GE card

9.4 petabit-meter/sec 1.9 times LSR

9.4 Gbps with 10 flows 37.0 petabit-meter/sec 6.9 times LSR

16TB in 6 hours with 7 flows

Implementation Sender-side modification Delay based Stabilized Vegas

Highlights

netlab.caltech.edu

Baltim

29.3.2000multiple

22.8.2002IPv6

9.4.20021 flow

SC2002 1 flow

SC200210 flows

FAST BMPS

Bmps Thruput Duration

37.0 9.40 Gbps min

9.42 940 Mbps 19 min

5.38 1.02 Gbps 82 sec

4.93 402 Mbps 13 sec

0.03 8 Mbps 60 min

netlab.caltech.edu

FAST: 7 flows

Statistics Data: 2.857 TB Distance: 3,936 km Delay: 85 msAverage Duration: 60 mins Thruput: 6.35 Gbps Bmps: 24.99 petab-m/sPeak Duration: 3.0 mins Thruput: 6.58 Gbps Bmps: 25.90 petab-m/s

Network SC2002 (Baltimore) SLAC (Sunnyvale), GE , Standard MTU

18 Nov 2002 Mon

cwnd = 6,658 pkts per flow

17 Nov 2002 Sun

netlab.caltech.edu

FAST: single flow

Statistics Data: 273 GB Distance: 10,025 km Delay: 180 msAverage Duration: 43 mins Thruput: 847 Mbps Bmps: 8.49 petab-m/sPeak Duration: 19.2 mins Thruput: 940 Mbps Bmps: 9.42 petab-m/s

Network CERN (Geneva) SLAC (Sunnyvale), GE, Standard MTU

17 Nov 2002 Sun

cwnd = 14,100 pkts

SCinet Bandwidth Challenge

Experiment/facilities Caltech: J. Bunn, S. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J.

Pool, S. Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, L. Cottrell, C. Logg, W. Matthews, R. Mount, J. Navratil StarLight: T. deFanti, L. Winkler

Major sponsors/partnersARO, CACR, Cisco, DataTAG, DoE, Lee Center, Level3, NSF

Acknowledgments

SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Prototype C. Jin, D. Wei Theory D. Choe (Postech/Caltech), J. Doyle,

gbps triangle

wang cds z

gbps wavelength triangle

deploymentpaganini ee

tb data

mbsecmany pb of data

networkingnewman physics

congestion informationthey

Documents

Caltech Architectural Heritage

SC2002, Baltimore ( From the Earth Simulator to PC Clusters...

Caltech Connect

Operazionali Caltech

CS 145A Implementation Issues Netlab.caltech.edu/course.

Energy procurement in the presence of intermittent sources.....

Catalog Caltech

Colin Borys (Caltech) Andrew Blain (Caltech) Darren Dowell.....

CALTECH NEWS - CaltechCampusPubs

Caltech Tutorial

CS/EE 145A Reliable Transmission over Unreliable Channel...

FAST TCP Cheng Jin David Wei Steven Low netlab.CALTECH.edu.

Weak lensing mass-selected cluster catalogues Richard Massey...

CS/EE 145A Congestion Control Netlab.caltech.edu/course.

ORGANIZATIONAL COMMUNICATION - Caltech

Scaling to New Heights Retrospective IEEE/ACM SC2002...