Connect. Communicate. Collaborate 4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit Development of Real Time eVLBI at Jodrell Bank Observatory 7 th International eVLBI Workshop, Shanghai, 16-17 th June 2008 Richard Hughes-Jones DANTE Jonathan Hargreaves JBO
40
Embed
Richard Hughes-Jones DANTE Jonathan Hargreaves JBO
4 Gigabit Onsala - Jodrell Lightpath for e-VLBI The iNetTest Unit Development of Real Time eVLBI at Jodrell Bank Observatory 7 th International eVLBI Workshop, Shanghai, 16-17 th June 2008. Richard Hughes-Jones DANTE Jonathan Hargreaves JBO. The 4 Gig Path for EXPReS Onsala – Jodrell. Multi: - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Connect. Communicate. Collaborate
4 Gigabit Onsala - Jodrell Lightpath for e-VLBI
The iNetTest Unit
Development of Real Time eVLBI at Jodrell Bank Observatory
7th International eVLBI Workshop, Shanghai, 16-17th June 2008
Richard Hughes-Jones DANTE Jonathan Hargreaves JBO
• Classic Bottleneck• 10 Gbit/s input 4 Gbit/s output• Use udpmon to send a stream of spaced UDP packets• Measure packet number of first lost frame as function of w packet spacing
)R*w(PNQ out1lostlen )/Q(R*wP/QN/1 lenoutlen1lost
Slope gives buffer size ~57 kBytes
Connect. Communicate. Collaborate
Use UDP to emulate TCP slowstart
• udpmon sends bursts of spaced packets:– 32 packets– Jumbo 8000 bytes– back2back– 4 ms between bursts
• PathLon-Ams_FF-Prague-Paris-Lon
• Rtt 55.5 ms
• See 13 packets then loose 1 in 3
• Confirm the TCP problem!
0
5
10
15
20
25
30
0 10 20 30 40 50 60
packet number
1-w
ay t
ime
Connect. Communicate. Collaborate
iNetTest: iBoB FPGA with 10 GE
Connect. Communicate. Collaborate
iNetTest: Control & Operation
10/100 Ethernet
10Gb Port 0
iNetTest iBOB 1
Send and receive packets of selected length and spacing
Count packets transmitted and received, calculate histograms of arrival times
Control PC
IP control of multiple iBOBs
10Gb Port 1
10Gb Port 0
10Gb Port 1
Ten Gb Network
iNetTest iBOB n
Expansion to more than two iBOBs
iNetTest iBOB 2
Send and receive packets of selected length and spacing
Count packets transmitted and received, calculate histograms of arrival times
Connect. Communicate. Collaborate
iNetTest: Simulink design
Jonathan HargreavesJBO
Network test parameter CSRs
10GE MAC core
Inter-packet timehistogram
Transmit & receive event time log
Connect. Communicate. CollaborateiNetTest: Details• iNetTest FPGA runs with a 200 MHz (5 ns) clocks • Two iNetTest units can be controlled over ethernet IP from a PC• iNetTest Ethernet IP address and MAC preset in code
– last digit selected by jumper• Ten GE IP address, Port and Gateway can be user configured• Automatic or manual ARPing. ARP tables for each port can be examined• iNetTest responds to and generates PING• UDP packets can be sent by firmware between two iNetTests
or between iNetTest and PCs• User selects the number of packets to send, their length,
and the time between them• iBOBs count received and transmitted packets, and store up to 2048 time
stamped events per port• iBOBs generate histograms of the arrival time distribution for received
eMERLIN Import – Onsala to Jodrell eMERLIN Export – Jodrell to JIVE
Connect. Communicate. Collaborate
Overview of eVLBI at JBO
eMERLIN CORRELATOR
JBO JIVE
4Gbps light path
OnsalaJBOCX4 4GbpsOr fibre if > 15m
CX4 4Gbps iBOB 5SwitchiBOB 4 Switch ADCStation Board VSI
VSI
4Gbps light
path
CX4 1Gbps
CX4 1Gbps
CX4 1Gbps
CX4 1Gbps
iBOB 0 SwitchStation Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
Switch
iBOB 1Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
iBOB 2Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
iBOB 3Station Board VSI
VSI
VSI to ZDOK
VSI to ZDOK
VLBI Mk V b receiv
ers
Connect. Communicate. Collaborate
Onsala to JBO: Transmit eMERLIN Import
ZDOK0
iADC
2 channels
1024MSPS per channel
8 bit sampling
10GB PORT
. 4GB/s out
ASSEMBLE PACKETS
Divide data into 8192 byte packets plus 16 byte MkVC compatible header
Alternatively generate test packets as per iNetTest
Count packets transmitted
DATA Left Pol
ADCCLK
ONEPPS
Convert data to 2 bit
Measure signal power on both channels
Generate real time clock based on ADC clock
Pass data & sync from ADC clock to iBOB 200MHz clock domain
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
Measured RMS Signal Power
Read and Set Real Time Clock
Set up test mode parameters
Control Data Flow ON/OFF
Set Network Source and Destination
DATA Right Pol
iBOB
Connect. Communicate. Collaborate
Onsala to JBO: Receive
ZDOK0
10GB PORT
. 4GB/s in
RE-ORDER PACKETS
Store data in SRAM location according to its time stamp and sequence number
Regenerate Sync and Data Valid signals
FIFO
16k deep
Smooth out bunching due to network delay variations
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
iBOB
STREAM OUT
Clock data out of SRAM on 128MHz correlator clock
Stream the left and right polarisations to the Station Board VSI chips
ZDOK1
VSI Chip 0
Route left polarisation data to input stage of Station Board
Send correlator clock to iBOB
VSI Chip 1
Route right polarisation data to input stage of Station Board
Station Board
SRAM0
512 packets L pol. data
SRAM1
512 packets R pol. data
DATA 32 bit
DATA 32 bit
128MHz clock
Connect. Communicate. CollaborateOnsala to JBO - Status
• iADC and transmit iBOB tested with ADC clocked at 1GS/s in diagnostic mode – ADC is set to output a fixed bit pattern
• iBOB receiver and VSI chip coded in Simulink and VHDL
Next Steps
• Test iADC and transmit iBOB with sinusoidal test signal• Generated deliberately out-of-order packets and bursts of
greater than 4Gbps to test receiver buffering
Connect. Communicate. Collaborate
JBO to JIVE: Transmit – OvervieweMERLIN Export
ZDOK0
10GB PORT
. 1 Gb/s out
REMOVE DELAY MODEL
(COURSE)
Use 72 bit wide bt 512k deep SRAM to remove eMERLIN delay model with 16 microsecond resolution
FORMAT
Generate 10000 byte MkVb frames
Power PC
iBOB control and monitoring via registers on the OBP bus
10/100 ethernet
iBOB
ZDOK1
VSI Chip 0
Process left polarisation data
Detail on next slide
VSI Chip 1
Process right polarisation data
Detail on next slide
Station Board
DATA 32 bit
DATA 32 bit
CLOCK
VALID
PDATA
CLOCK
VALID
PACKETISE
Divide each MkVb frame into two 5008 byte packets (incl header)
Add Mk5C header to each packet
Connect. Communicate. Collaborate
JBO to JIVE: Transmit -VSI Chip Detail
VSI Chip 0 - Process left polarisation data
Monitor & Control Bus (MCB)
Station Board
DATA 32 bit
CLOCK
VALID
PDATA
Select a 128MHz
band from the
filter bank
Subtract Merlin Delay (Fine)
Variable coefficient FIR filter changes delay in steps of 1/16th of original sample period
Reclock for delays up to 16ns
Mixer 0
FIRLow pass filter
Decimate by 8/16128 taps
Convert 4/8 bit data to 2 bit VLBI
Mixer 1
Mixer 8/16
Stream data out to iBOB over VSI
cable
MCB InterfaceLoad band selection, delay model and filter
parameters
0
1
15
Mixer Bank
Select 8 or 16 VLBI bands from
128MHz input
Remove eMERLIN offset
(a multiple of 10kHz)
Connect. Communicate. CollaborateJBO to JIVE: Status
• VSI chip DSP functions currently under development and simulation
• Transmit iBOB re-uses packetising code from iNetTest
• Next Steps• VSI chip MCB interface – based on Verilog code provided
by Dave Fort at Penticton• Transmit iBOB needs coarse delay and MkVb formatting
Connect. Communicate. Collaborate
Summary
• UDP on a 28 VC-4, 4.2 Gigabit lightpath provides the performance required for EXPReS
• TCP performance poor due to small Ethernet buffer size on the equipment connecting 10GE to the 4 Gig Lightpath
• Ethernet flow-control with short tails helps TCP
• The network behaves correctly.• Care needed on choice of PC for 10 Gigabit.• FPGA iNetTest solutions work very well.• Onsala – Jodrell 4 Gigabit path now in place.• Packet loss and packet spacing need more understanding. • VLBI FPGA application progressing well.
Connect. Communicate. Collaborate
Any Questions ?
Connect. Communicate. CollaborateFor More Information
• www.geant2.net• www.dante.net• For latest news and factsheets http://www.geant2.net/media
• For research activities http://www.geant2.net/research• http://expres-eu.org/ [note: only one “s”]• http://www.jive.nl/ • Contact information: