vlbi_udp Throughput Performance and Stability.

ESLEA VLBI Bits&Bytes Workshop , 31 Aug 2006, R. Hughes-Jones Manchester1

vlbi_udp

Throughput Performance and Stability.

Richard Hughes-Jones The University of Manchester

www.hep.man.ac.uk/~rich/ then “Talks”


vlbi_udp Flow Diagram

Ring buffer

Input thread

Ring buffer

Send thread

Disk Memory

Receive thread

Output thread

Disk Memory

From Simon Casey

UDP Data

TCP Control

Control thread Control thread

Vlbi_control

vlbi_send vlbi_recv


iGrid2002 monolithic code Convert to use pthreads

control Data input Data output

Code branch for Simon’s file transfer tests Work on vlbi_recv:

Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on

semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used

Network performance – in progress

History or the story so far:


Instability: 2 streams into JIVE, March 2006From Simon Casey


L1 ; 0; 0; 0; 100; 0; 0; 0;;L2 ; 0; 0; 0; 99; 0; 0; 0;;L3 ; 0; 0; 0; 99; 0; 0; 0;;L4 ; 2; 0; 43; 53; 0; 1; 18

Early B2B UDP Tests Kernel 2.6.6 (May06) vlbi_recv semaphore Wait 15 us; signal 200 events Unstable throughput

~815 Mbit/s Packet loss ~ 0.05%

Lost at top of IP stack

Inter-packet time

Processing time mean 0.1166 sigma 0.3493

CPU load:

vlbi_recv_k266_105d_w12_11May06

770

775

780

785

790

795

800

805

810

815

820

0 10 20 30 40 50 60 70 80

Time between frames us

Wir

e R

ate

Mb

it/s

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

Packet

loss


110

1001000

10000100000

100000010000000

0 20 40 60 80 100 120 140 160


N(t

)


110

1001000

10000100000

100000010000000

0 20 40 60 80 100 120

Packet process time usN

(t)


Confirm with B2B UDP Tests Kernel 2.6.6 (Aug06) vlbi_recv semaphore Wait 15 us; signal 100 events Throughput a bit unstable

~815 Mbit/s Packet loss ~ 0.05%

Lost at top of IP stack

CPU number & % packet loss Have loss when using same CPU

as that servicing interrupts

Confirmed by many udpmon tests !

vlbi_udp-sig k266 w15 N100 17Aug06

0

2

4

6

8

0 10000 20000 30000 40000 50000 60000 70000 80000

ThousandsPacket Number

CP

U n

o

0

5

10

15

20

Pac

ket

loss

CPU num

num lost


700

720

740

760

780

800

820

840

0 200 400 600 800 1000 1200 1400

Time s

Wir

e R

ate

Mb

it/s

0

0.02

0.04

0.06

0.08

0.1

% P

acke

t lo

ss



1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120 140 160


N(t

)Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.3% hi, 0.0% siCpu1 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% siCpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.7% id, 0.0% wa, 0.0% hi, 0.0% siCpu3 : 6.0% us, 17.0% sy, 0.0% ni, 40.3% id, 0.0% wa, 1.0% hi, 35.7% si

B2B UDP Tests – different kernel Kernel 2.6.9 vlbi_recv semaphore Wait 12 us; signal 200 events Stable throughput

999 Mbit/s variation less than 1 Mbit/s No packet loss

Inter-packet time


CPU load:


980

985

990

995

1000

0 500 1000 1500 2000 2500 3000 3500 4000 4500


Wir

e R

ate

Mb

it/s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Packet

loss


1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120

Packet process time usN

(t)


Cpu0 : 3.7% us, 8.0% sy, 0.0% ni, 88.4% id, 0.0% wa, 0.0% hi, 0.0% si Cpu1 : 1.0% us, 4.0% sy, 0.0% ni, 95.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.0% sy, 0.0% ni, 100.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu3 : 11.6% us, 22.9% sy, 0.0% ni, 41.9% id, 0.0% wa, 1.0% hi, 22.6% si

B2B UDP Testsvlbi_recv_k269_110d_w20_29Aug06

600

602

604

606

608

610

612

614

616

618

620

0 1000 2000 3000 4000 5000


Wir

e R

ate

Mb

it/s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Packet

loss

vlbi_recv_k269_110d_w20_29Aug06

1

10

100

100010000

100000

100000010000000

1000000001000000000

0 20 40 60 80 100 120 140 160


N(t

)

vlbi_recv_k269_110d_w20_29Aug06

110

1001000

10000100000

100000010000000

1000000001000000000

0 20 40 60 80 100 120

Packet process time us

N(t

)

Kernel 2.6.9 vlbi_recv sched_yield() Wait 20 us Stable throughput

612.5 Mbit/s variation less than 1 Mbit/s No packet loss

Inter-packet time


CPU load:


vlbi_udp_recv_k269_110d_w12_29Aug06

980

985

990

995

1000

0 500 1000 1500 2000 2500 3000


Wir

e R

ate

Mb

it/s

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Packet

loss

Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si Cpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si Cpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si

B2B UDP Tests Kernel 2.6.9 vlbi_recv sched_yield() Wait 12 us Stable throughput

999 Mbit/s variation less than 1 Mbit/s No packet loss

Inter-packet time


CPU load:


1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120 140 160


N(t

)


1

100

10000

1000000

100000000

10000000000

0 20 40 60 80 100 120

Packet process time us

N(t

)


Next Steps

Fix the timing precision – again! Test with latest 2.6.16 and 2.6.17 kernels Why does shed_yield() take more CPU time? Is it

a problem or not? Test on long-distance Networks

Production (packet switched) UKLight

Test at 4-8 Gbit


Any Questions?


More Information Some URLs 1 UKLight web site: http://www.uklight.ac.uk MB-NG project web site: http://www.mb-ng.net/ DataTAG project web site: http://www.datatag.org/ UDPmon / TCPmon kit + writeup:

http://www.hep.man.ac.uk/~rich/net Motherboard and NIC Tests:

http://www.hep.man.ac.uk/~rich/net/nic/GigEth_tests_Boston.ppt& http://datatag.web.cern.ch/datatag/pfldnet2003/ “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue 2004 http:// www.hep.man.ac.uk/~rich/

TCP tuning information may be found at:http://www.ncne.nlanr.net/documentation/faq/performance.html & http://www.psc.edu/networking/perf_tune.html

TCP stack comparisons:“Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004

PFLDnet http://www.ens-lyon.fr/LIP/RESO/pfldnet2005/ Dante PERT http://www.geant2.net/server/show/nav.00d00h002


Lectures, tutorials etc. on TCP/IP: www.nv.cc.va.us/home/joney/tcp_ip.htm www.cs.pdx.edu/~jrb/tcpip.lectures.html www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm www.cis.ohio-state.edu/htbin/rfc/rfc1180.html www.jbmelectronics.com/tcp.htm

Encylopaedia http://www.freesoft.org/CIE/index.htm

TCP/IP Resources www.private.org.il/tcpip_rl.html

Understanding IP addresses http://www.3com.com/solutions/en_US/ncs/501302.html

Configuring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt

Assigned protocols, ports etc (RFC 1010) http://www.es.net/pub/rfcs/rfc1010.txt & /etc/protocols

More Information Some URLs 2

http://www.nv.cc.va.us/home/joney/tcp_ip.htm

http://www.cs.pdx.edu/~jrb/tcpip.lectures.html

http://www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200/CCONTENTS

http://www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ap1.htm


Backup Slides


VLBI Network Topology


Geant2 Launch event, June 2005

vlbi_udp Throughput Performance and Stability.

Documents

cpu load

semaphore packet loss

packet losshave loss

cpu time

recv semaphore

b2b udp testscpu0

b2b udp testskernel

cpu usednetwork performance