High Rate Internet Data Transfer for eVLBI Ralph Spencer, Richard Hughes-Jones and Simon Casey The University of Manchester All Hands Sept 2005
Jan 02, 2016
High Rate Internet Data Transfer for eVLBI
Ralph Spencer, Richard Hughes-Jones and Simon Casey
The University of ManchesterAll Hands Sept 2005
Radio Astronomy• The study of celestial objects at <1 mm to >1m
wavelength.• Sensitivity for continuum sources• B=bandwidth, integration time.• High resolution achieved by interferometers.
Some radio emitting X-ray binary stars in our own galaxy:
B/1
GRS 1915+105 MERLIN
SS433MERLIN and European VLBI
Cygnus X-1VLBA
GRS 1915+105: 15 solar mass BH in an X-ray binary: MERLIN observations
receding
600 mas = 6000 A.U. at 10 kpc
Very Long Baseline Interferometry
How interferometry works
• Interferometer: signals from each telescope brought together coherently for correlation (multiplication)
• Resolution /D << that possible from a single telescope, e.g the pimples on a golf ball at 3000 miles at the highest resolution
• Output of the correlator is one component of the Fourier transform of the sky brightness
Dcos
X
D
Earth-Rotation Synthesis
.
The baseline between each pair of telescopes forms an ellipse as the Earth rotates
The Aperture Plane (u,v)
• Need ~ 12 hours for full synthesis, not necessarily collecting data for all that time.
• Telescope data correlated in pairs: N(N-1)/2 baselines
• NBTrade-off between B and t for sensitivity ~sqrt(Bt)
• Data are calibrated, Fourier inverted and then deconvolved to give the images.
EVN
MERLIN
The European VLBI NetworkEVN
• Detailed radio imaging uses antenna networks over 100s-1000s km
• At faintest levels, sky teems with galaxies being formed
• Currently use disk recording at 512Mb/s (MkV)
• real-time connection allows greater – response– reliability– sensitivity
– Need Internet eVLBI
WesterborkNetherlands
Dedicated
Gbit link
EVN-NREN
OnsalaSweden
Gbit link
Jodrell BankUK
DwingelooDWDM link
CambridgeUK
MERLIN
MedicinaItaly
Chalmers University
of Technolo
gy, Gothenbu
rg
TorunPoland
Gbit link
eVLBI Milestones
• January 2004: Disk buffered eVLBI session:– Three telescopes at 128Mb/s for first eVLBI
image– On – Wb fringes at 256Mb/s
• April 2004: Three-telescope, real-time eVLBI session.– Fringes at 64Mb/s– First real-time EVN image - 32Mb/s.
• September 2004: Four telescope real-time eVLBI– Fringes to Torun and Arecibo– First EVN, eVLBI Science session
• January 2005: First “dedicated light-path” eVLBI– Australia to JIVE via Canarie: data from Huygens
descent transferred at ~450Mb/s
• 20 December 20 2004• connection of JBO to Manchester by 2 x 1 GE• eVLBI tests between Poland Sweden UK and Netherlands at 256 Mb/s
• February 2005• TCP and UDP memory – memory tests at rates up to
450 Mb/s (TCP) and 650 Mb/s (UDP)• Tests showed inconsistencies between Red Hat
kernels, rates of 128 Mb/s only obtained on 10 Feb• Haystack (US) – Onsala (Sweden) runs at 256 Mb/s
• Regular tests with VLBI data every ~6 weeks • 128 Mpbs OK, 256 Mpbs often,• But not 512 Mbps – WHY NOT?
• Correlator can cope with large error rates• up to 2 %• but need high throughput for sensitivity• implications for protocols
Gnt5-DwMk5 11Nov03-1472 bytes
0
2
4
6
8
10
12
0 5 10 15 20 25 30 35 40Spacing between frames us
% P
acket
loss
Gnt5-DwMk5
DwMk5-Gnt5
Throughput vs packet spacing Manchester: 2.0G Hz Xeon Dwingeloo: 1.2 GHz PIII Near wire rate, 950 Mbps UDPmon
Packet loss
CPU Kernel Load sender
CPU Kernel Load receiver 4th Year project
Adam Mathews Steve O’Toole
Gnt5-DwMk5 11Nov03/DwMk5-Gnt5 13Nov03-1472bytes
0
200
400
600
800
1000
1200
0 5 10 15 20 25 30 35 40Spacing between frames us
Recv W
ire r
ate
Mbits/s
Gnt5-DwMk5
DwMk5-Gnt5
Gnt5-DwMk5 11Nov03 1472 bytes
020406080
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% K
erne
l S
ende
r
Gnt5-DwMk5 11Nov03 1472 bytes
020406080
100
0 5 10 15 20 25 30 35 40Spacing between frames us
% K
erne
l R
ecei
ver
UDP Throughput Oct-Nov 2003 Manchester-Dwingeloo Production
Throughput (JbGig1 - Mark608 @ JIVE via production)
500
600
700
800
900
1000
25/0
1/20
0500
:00
25/0
1/20
0512
:00
26/0
1/20
0500
:00
26/0
1/20
0512
:00
27/0
1/20
0500
:00
Time of day
Wir
e r
ate
(M
b/s
)
12us
18us
20us
% Packets lost (JbGig1 - Mark608 @ JIVE via production)
0
0.5
1
1.5
2
25/0
1/20
0500
:00
25/0
1/20
0512
:00
26/0
1/20
0500
:00
26/0
1/20
0512
:00
27/0
1/20
0500
:00
Time of day
% L
os
t
12us
18us
20us
January 2005 repeated UDP over 2 days
ESLEA
• Packet loss will cause low throughput in TCP/IP
• Congestion will result in routers drooping packets: use Switched Light Paths!
• Tests with MB-NG network Jan-• JBO connected to JIVE via UKLight
in June (thanks to John Graham, UKERNA)
• Comparison tests between UKLight connections JBO-JIVE and production (SJ4-Geant)
UKLight Switched light path
Tests on the UKLight switched light-path Manchester : Dwingeloo
• Throughput as a function of inter-packet spacing
• Packet loss • Maximum size packets can
reach full line rates with no loss, and there was no re-ordering (plot not shown).
gig03-jiveg1_UKL_25Jun05
0100200300400500600700800900
1000
0 10 20 30 40Spacing between frames us
Rec
v W
ire r
ate
Mbi
t/s
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes
gig03-jiveg1_UKL_25Jun05
0.0001
0.001
0.01
0.1
1
10
100
0 10 20 30 40Spacing between frames us
% P
acke
t lo
ss
50 bytes
100 bytes 200 bytes
400 bytes 600 bytes
800 bytes 1000 bytes
1200 bytes 1400 bytes
1472 bytes
Tests on the production network Manchester : Dwingeloo.
• Throughput
• Small (0.2%) packet loss was seen
• Re-ordering of packets was significant
gig6-jivegig1_31May05
0.0001
0.001
0.01
0.1
1
10
100
0 10 20 30 40Spacing between frames us
% P
acket
loss
50 bytes
100 bytes 200 bytes
400 bytes 600 bytes
800 bytes 1000 bytes
1200 bytes 1400 bytes
1472 bytes
UKLight using MkV terminals
The GÉANT2 Launch June 2005
Jodrell BankUK
DwingelooDWDM link
MedicinaItaly Torun
Poland
e-VLBI at the GÉANT2 Launch Jun 2005
UDP Performance: 3 Flows on GÉANT• Throughput: 5 Hour run 1500 byte MTU
• Jodrell: JIVE2.0 GHz dual Xeon – 2.4 GHz dual Xeon670-840 Mbit/s
• Medicina (Bologna): JIVE 800 MHz PIII – Mk5 (623) 1.2 GHz PIII 330 Mbit/s limited by sending PC
• Torun: JIVE 2.4 GHz dual Xeon – Mk5 (575) 1.2 GHz PIII
245-325 Mbit/s limited by security policing
(>600Mbit/s 20 Mbit/s) ?
• Throughput: 50 min period• Period is ~17 min
BW 14Jun05
0
200
400
600
800
1000
0 500 1000 1500 2000Time 10s steps
Rec
v w
ire ra
te M
bit/s
JodrellMedicinaTorun
BW 14Jun05
0
200
400
600
800
1000
200 250 300 350 400 450 500Time 10s steps
Rec
v w
ire ra
te M
bit/s
JodrellMedicinaTorun
18 Hour Flows on UKLightJodrell – JIVE, 26 June 2005
• Throughput:• Jodrell: JIVE
2.4 GHz dual Xeon – 2.4 GHz dual Xeon
960-980 Mbit/s
• Traffic through SURFnet
• Packet Loss– Only 3 groups with 10-150 lost
packets each– No packets lost the rest of the
time
• Packet re-ordering– None
man03-jivegig1_26Jun05
0
200
400
600
800
1000
0 1000 2000 3000 4000 5000 6000 7000
Time 10s steps
Rec
v w
ire r
ate
Mbi
t/s
w10
man03-jivegig1_26Jun05
900910920930940950
960970980990
1000
5000 5050 5100 5150 5200
Time 10s
Recv w
ire r
ate
Mbit/s w10
man03-jivegig1_26Jun05
1
10
100
1000
0 1000 2000 3000 4000 5000 6000 7000
Time 10s steps
Packet
Loss
w10
Conclusion and Future directions• eVLBI needs to get to rates of 512 Mpbs – not there yet.
• Rates of 256 Mpbs on production networks possible now using TCP/IP in good conditions, but not higher rates.
• G2 launch showed continuous long term flows as in VLBI have different characteristics to those in short term tests (Iperf, UDPmon) : more investigation needed – collaborating with DANTE and SURFnet
• Packet loss is a killer for TCP – new protocols needed (ESLEA) - VSI-E standard
• The End Host Problem– The performance of Motherboards, NICs, RAID controllers and
Disks matter: Mk V units (Conduant)– MkV-B 2 Gbps data recording in 2 yrs?
• Next generation correlator 10 Gbps in 5-10 yrs
Thanks to
• Paul Burgess JBO• Steve Parsley, Arpad Szomoru, Cormac Reynolds JIVE• Nicola Pezzi, John Graham, Colin Greenwood, Peter
Clarke ESLEA• EVN observatories staff