High Bandwidth Data Acquisition and Network Streaming in VLBI
Jan Wagner Guifreacute Molera et alTKK Metsaumlhovi Radio Observatory
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 2
bull EXPReS funds to connect stations at 1G10Gbull 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport
protocols
bull ToDo new system for gt1 Gbps rates
Basic e-VLBI Data AcquisitionIntroduction
bull Basic 8ndash16 stations rates 128ndash512 Mbits 24h 1ndash4 TBstation all to correlatorbull VLBI observed galaxyrsquos noise recorded to disks shipped to correlatorbull e-VLBI transfer station data via fast research networks two alternatives
bull post-observation record locally transfer to correlator laterbull real-time stream to the correlator for temp storage or processing
1Apr08 JIVE vlbi_udp incoming aggregate rate record
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3
Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum
bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station
bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator
bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming
Introduction
C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4
High Bandwidth DAQ DevelopmentDAQ steps
Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)
Development Environment
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5
DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform
bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards
Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC
=gt 32 Gbps ADC data 20 Gbps net
bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development
bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 2
bull EXPReS funds to connect stations at 1G10Gbull 1G fiber up to capacity with Tsunami UDP (2006) or JIVE vlbi_udp (2007) real-time transport
protocols
bull ToDo new system for gt1 Gbps rates
Basic e-VLBI Data AcquisitionIntroduction
bull Basic 8ndash16 stations rates 128ndash512 Mbits 24h 1ndash4 TBstation all to correlatorbull VLBI observed galaxyrsquos noise recorded to disks shipped to correlatorbull e-VLBI transfer station data via fast research networks two alternatives
bull post-observation record locally transfer to correlator laterbull real-time stream to the correlator for temp storage or processing
1Apr08 JIVE vlbi_udp incoming aggregate rate record
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3
Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum
bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station
bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator
bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming
Introduction
C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4
High Bandwidth DAQ DevelopmentDAQ steps
Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)
Development Environment
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5
DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform
bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards
Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC
=gt 32 Gbps ADC data 20 Gbps net
bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development
bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 3
Goals for High(er) Bandwidth DAQbull Observe a broader radio spectrum
bull broader spectrum =gt higher data ratebull correlation result and final image improvesbull high EXPReS goal 1248 Gbps per station
bull Higher rates =gt need at leastbull new analog-to-digital converter system (current 10+ years old)bull faster higher capacity localremote data storage systemsbull suitable transport protocols - stations to correlator
bull Ultimately e-VLBI data processing in real-timebull get rid of high capacity station data storagebull JIVE hardware correlator successful in many le512 Mbps real-time runsbull distributed software correlators more natural with data streaming
Introduction
C Phillips e-VLBI Scientific Benefits ATNF CSIRO memo 2004
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4
High Bandwidth DAQ DevelopmentDAQ steps
Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)
Development Environment
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5
DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform
bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards
Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC
=gt 32 Gbps ADC data 20 Gbps net
bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development
bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 4
High Bandwidth DAQ DevelopmentDAQ steps
Digitize wide-band signal ndash Stream over 10G ndash Capturestore with PC(s)
Development Environment
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5
DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform
bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards
Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC
=gt 32 Gbps ADC data 20 Gbps net
bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development
bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 5
DAQ ndash Digitizingbull UC Berkeleyrsquos iBOB FPGA hardware platform
bull Virtex2 Pro FPGA with PowerPC405bull two CX4 10G portsbull 12 Berkeley iADC expansion boards
Atmel AT84AD001 two channels 8-bit 1 Gsamples=gt max 8+8 Gbps from one iADC
=gt 32 Gbps ADC data 20 Gbps net
bull Semi-ready component libraries from Berkeley for ldquofasterrdquo start of development
bull Analog signal digitizing setupbull iBOB FPGA reads raw 2-channel iADC databull FPGA processes data implements network streaming =gt hellip
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 6
DAQ ndash Streamingbull TCP is not a good streaming choice
bull No unreliable best-effort delivery no delivery time constraints
bull Real-time VLBI stream requirementsbull data loss fle10 is tolerable (impact is ) occasional re-order okbull data integrity non-critical BER210-9 frame CRC32 checksumless UDPbull data must have time stamp a 64-bit PSN sufficient
35kYears1Tbps8Kbull congestion control none data thinning change sampling mode
bull iBOB UDP streaming first implementationbull iBOB has only 14MB RAM (asymp30ms4 Gbps)
=gt no retransmissions =gt plain UDP packetizer 1 stream per channelbull streaming configurable remotely via Telnetssh
overall iBOB test designhellip =gt
~ 1 f
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 7
DAQ ndash iBOB RF-to-10G Test DesignBasic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 8
DAQ ndash Storage RAID Performancebull Should use RAID-0 disk striping for best throughputbull Hardware RAID eg PCIe cards XRaid VTrak too expensive and slowbull Software RAID baseline performance test on
~1keuro workstation PC Linux RAID-0 12 x SpinPoint F1 9TBTest method- source buffer ~90 RAM size- sequential fwrite() to a single file- not using 10G UDP yet
=gt Inexpensive 1ndash4 Gbps data storage system
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 9
DAQ ndash Stream Recordingbull Wrote simple UDPraw capture programs udp2raid raw2raid
bull Essential for performance in 10G stream recordingbull 10G NIC throughput
bull 479K jumbo frames UDP checksums offbull prefer larger UDP packets larger OS buffers
=gt Myri Chelsio NIC PC iperf 99 Gbps UDP 0 lossbull Application and disk throughput
bull use multi-core system try to utilize (nearly) all cores separate network and disk IO into own threads
=gt udp2raid raw2raid currently min 4 Gbps from 10G to RAID-0 =gt we have 1ndash4 Gbps iBOB stream to Abidal PC disk recording
Basic Data Acquisition System
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 10
Fast ldquooff-linerdquo e-VLBI File Transferbull Transfer station files to where you process the data one option Tsunami UDP v11 fast lsquoFTPrsquo and real-time (httptsunami-udpsfnet)bull Weekly used in geodesic VLBI for le1 Gbps file transfer occasional real-timebull Current v11 version first result on 10G LAN
bull high CPU load 3 ndash 35 Gbps to RAID protocol rate limit 4 Gbpsbull Started v12 of protocol to use udp2raid improvements
bull 4 ndash 44 Gbps to Abidal PC RAID bull 7 Gbps to RAM (vs UDT4-protocol 52 Gbps to RAM)
Basic Data Acquisition System
=gt Improved rates for ldquooff-linerdquo e-VLBI
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 11
Summary
Summarybull Many European VLBI stations start to reach ~1 Gbps
streaming with the old hardware
bull Basic iBOB design can further improve the rate digitize and stream 1 ndash 20 Gbps UDP
bull A consumer ~1 keuro PC + 12 disks is capable of 4 Gbps storagecan capture the iBOB stream(s) to PC RAID
bull Tsunami v12 has improved file transfer ratesbull Can transfer from station to similar ge4 Gbps PC elsewherebull ToDo port real-time Tsunami server onto iBOB board
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642
10 April 2008 EXPReS- High Bandwidth Data Acquisition and Network Streaming in VLBI Slide 12
QuestionsAnswersbull Contact information
Jan WagnerMScResearcherEXPReS (Metsaumlhovi)jwagnerkurphutfi
bull Additional Informationhttpexpres-euorg [note only one ldquosrdquo]httpwwwmetsahovifienvlbi
bull EXPReS is made possible through the support of the European Commission (DG-INFSO) Sixth Framework Programme Contract 026642