An FX software correlator for VLBI Adam Deller Swinburne University Australia Telescope National Facility (ATNF)
Jan 05, 2016
An FX software correlator for VLBI
Adam DellerSwinburne University
Australia Telescope National Facility (ATNF)
Outline
• Function of a correlator
• Reasons for software correlation
• Code and platforms
• Progress and current performance
• Immediate and near future plans
• Facilitation of real-time gigabit eVLBI
Correlator functionality
• Averaging the product of signals at two telescopes gives visibility
Correlator functionality
• Averaging the product of signals at two telescopes gives visibility
• To get frequency information, average additional multiplications by lagged signal, then FFT (or reverse for FX)
• What does it all boil down to? Gigabit interferometry, 1024 channels with
6 telescopes, FFT cost ~ 200 Gflops (FX)
Why software correlate?
• Flexibility Handle disparate file formats and
bands High spectral resolution + high b/w Real time fringe checking
• Rapid development• For Australian LBA
8 x bandwidth, using disk-based recorders instead of S2 tape system
Correlator code structure
• Developed in C++
• Disk read/write threaded to hide latency
• Currently using IPP for vector mathematics
fxcorr
Telescope Baseline
Mode InputStream
Platform options
• Initial development to run on Swinburne cluster: 200+ 3GHz Pentium 4 machines
• Cray XD-1 chassis: 12 AMD Opteron processors, 6 Xilinx FPGAs
Hybrid Architecture
• PC flexibility with FPGA power
• FPGA used for unpacking/fringe rotating and FFT
• Possibly incorporate cross-multiply• FPGA code compiled into library,
available as C subroutine call• Cray contributing via development
of FPGA subroutine library
Milestones to date
• ‘Basic’ and ‘complex’ FX algorithms implemented and verified
• Fortran driver for CALC ported to C for delay modeling
• Used in search for first trans-Tasman fringes to 6m antenna in NZ
• Range of support/analysis packages created
Correlator output
Current performance
• LBA: Complex correlation, 6 telescopes, 2x 16MHz bands (128 Mb/s), 4 products/band, one node of cluster: 512 channels: 21x real time 2048 channels: 25x real time 32786 channels: 32x real time
• 15 telescopes, 128 Mb/s, 4 products 512 channels: 88x real time
Immediate applications
• Improved LBA sensitivity: parallax of double pulsar J0737-3039
• High time and frequency resolution + high bandwidth: Wide field imaging
• Masers, SNRs, AGNs…NGC253 image credit:Emil Lenc (Swinburne)
Real-time gigabit eVLBI?
• All ATNF telescopes of LBA capable of 1 Gb/s, others 512 Mb/s
• Technical/cabling issues to overcome: disk-based recorders
• Software correlator ready now• Simulated Gb/s data, 512 channels
<160x real time Current system has disk transfer
limitation
In the near future…
• Fibre links overcome disk limitations?• Investigate alternate architecture:
‘Cell’ processor array at VPAC• Code improvements:
Sophisticated pulsar gating Polyphase filterbank alternative to FFT Automated configuration from schedule,
GUI frontend, output FITS compatible…
Integration into the LBA
• Available for use from now on a limited basis
• Final release planned before March 2006 observing session
• Will incorporate automatic configuration, GUI frontend etc
• Feedback on desired features encouraged
Applications outside the LBA
• Potentially useful for niche applications anywhere
• Applicable to connected-element as well as VLBI• “Piggy-backing” on existing interferometers - only need facility to get data to disk, and some processors
Existing correlator limited + disk space
available + processors = SOFTWARE CORRELATE
Conclusions
• Fast, flexible new FX software correlator for generic clusters developed
• Hybrid architecture will be explored• Will provide significant increase in
LBA sensitivity until next upgrade• Real-time gigabit eVLBI using the
software correlator is feasible on Swinburne cluster
• Final release planned in early 2006
Questions??
XD-1 Specifications
• 12x 2.2GHz Opteron processors - 106 Gflops total
• 1-8 GB RAM per processor, 12.8GB/sec• 4-8 GB/s interconnect (between nodes)• 6 Xilinx Virtex 4 FPGAs• 4 PCI-X slots for ethernet/fibre channel• Max 1.5 TB local storage