xNTD/SKAMP/LFD Correlator 4th RadioNet Engineering Forum Workshop Next Generation Correlators for Radio Astronomy and Geodesy 27-29 June 2006, Groningen, The Netherlands, John Bunton CSIRO ICT Centre, Sydney
Jan 17, 2016
xNTD/SKAMP/LFD Correlator
4th RadioNet Engineering Forum Workshop
Next Generation Correlators for Radio Astronomy and Geodesy
27-29 June 2006, Groningen, The Netherlands,
John Bunton
CSIRO ICT Centre, Sydney
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Common Design
Developing a set of technologies that will be used in xNTD, SKAMP3 and LFD
Common correlator, VHDL code and hardware elements
This talk mainly the correlator and xNTD beamformer
Team (SKAMP3, xNTD and LFD) Uni Sydney/ ATNF - Ludi de Souza,
Uni Sydney Duncan Campbell-Wilson, John Russel, Chris Weimann, Adrian Blake
ATNF/ ICT Centre John Bunton, Jaysri Joseph
MIT Roger Cappallo, Brian Fanous
ANU Frank Briggs
and growing
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
xNTD specification
Connected element telescope with 20-30 antennas within ~10km area
Each antenna generates 32 dual pol. beams giving a field of view of ~30 square degrees – frequency independent
Dynamic range ~50dB (8-bit A/D) aided by low RFI environment at Mileura WA, >60dB filterbanks
Frequency range below 1.7 GHz with 300 MHz BW
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
xNTD Correlator
FX correlator, FPGA based using polyphase filterbank
Large design with 20-40 antennas x ~32 beams Input to beamformers 1.5 THz, to correlator 500 GHz (16bits/Hz)
6,000-15,000 baselines, each full Stokes
~1000 Tops/sec (mainly beamformer/filterbanks)
Output data rate 1-2 Gcorrelation/visability set
Full 300 MHz processed to full frequency resolution ~50,000 channel – 6 kHz resolution (no options)
BUT Low resolution and continuum by averaging channels
Daisy chain data for special processing
Later can trade bandwidth for higher resolution or time sampling
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Design
Beamformers and filterbank ~200 inputs generating 32 dual pol. beams
Signal transport from feeds optical analogue (TBD)
Beamformer location TBD
Design similar to SKAMP3 filterbank (Ludi’s talk)
Correlator based on correlation cell Highly configurable as to number inputs
If the beamformer is at the antenna crossconnect in fibre network and O/E interface
Needs data re-ordering (Ludi’s talk)
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
xNTD Beamformer
Beamformer input 96 Dual polarisation inputs
300 MHz bandwidth
About 1 Terabit/sec input data rate per antenna
Each of 32 output beams is the sum of 20 to 96 inputs depending on frequency Would like to have data from all inputs to each unit that generates
a beam
Instead “Divide and Conquer”
Use the same topology as SKAMP3 filterbanks (Ludi’s talk)
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Filterbank to Beamformer
24 channelOS Polyphase
12x10 MHzcomplex
24 channelOS Polyphase 12x10 MHz
complex
~200 MS/s Real
~200 MS/s Real
192 inputs
192 feeds4MHz 800 channels
Dual 6.1 Gbits/s
12 sets
192 inputs10MHz band2k ChannelPolyphase
192 inputs10MHz band2k ChannelPolyphase
CrossConnectOutputs10 MHzfor all
signals toeach finefilterbank
32 channelOS Polyphase
Output 16x23MHzcomplex
32 channelOS Polyphase
Output 16x23MHzcomplex
~660 MS/s Real
~660 MS/s Real
Inputs from 96dual polarisation
feeds(192 signals)
Beamformer33x2 beam
23 MHz
Beamformer33x2 beam
23 MHz
32x2 beams309MHz309 MS/s
(4R,4I) complex163 Gbits/s
to centralcorrelator
20.6MHz4k ChannelPolyphase
20.6MHz4k ChannelPolyphase
CrossConnectOutputs23MHzfor all
signals toeach BF
SKAMP Filterbanks (one card cage)
xNTD Beamformer
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Correlator
The challenge was to develop a correlator concept the could be used in xNTD, SKAMP3 and LFD - 20-30, 192, and 512 antennas
Systolic array too inflexible xNTD 760 correlations/beam LFD 500,000 correlations
Approach developed – Correlation Cell Combination of multiply-accumulate and storage Each cell handles 256 correlations at a time 49,000 correlations per FPGA simultaneously 512 time sample short integration on chip
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Correlation Cell
Input 16+ pairs of data 4bit complex multiply in 18-bit
multiplier Accumulation to block RAM Calculate 256 correlation,
512 successive time samples Data reordering in filterbank
xNTD 4-7 cells for all correlations 30-70 MHz BW per FPGA
All baselines LFD 12, SKAMP3 1.5 FPGAs 1.2-1.5 MHz of bandwidth
Dual PortMemory
Complex multiplierResources one 18-bit multipliers
Dual 4+4 bit inputs - 9+9 bit output
Single Dual Port Memory - 2 x 256 36-bit words
+
Data loading
Two4+4 bit outputs
18+18 bit outputs
Dual PortMemory
MUX MUX
Latch Latch
8-bit shift register
Serial input data
36-bit shift re
gister
Serial output data
8-bit shift register
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
22 antenna xNTD configuration
Store 22 or 22 pairs per correlation cells
Within group 231 correlation
Between groups 484 correlations Need two cells
CorrelationCell
Correlationswithin a
group of 22
CorrelationCell
Half thecorrelationsbetween twogroups of 22
CorrelationCell
Half thecorrelationsbetween twogroups of 22
CorrelationCell
Correlationswithin a
group of 22
First group of 22 inputs
Second group of 22 inputs
48 sets of 22-antenna correlators per FPGA (Virtex4 SX35)
Clock cycles 242 for 44 inputs = 5.5 clocks per input
Input data rate 48/5.5 = 8.7 bytes/clock
As systolic array 192 multipliers gives 13x13 array
Input data rate 3 times higher at 26 bytes/clock
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Board Manufacture Simplification (1)
Manufacture of correlator board a major task Examples SKAMP1, EVLA …
Correlation cell reduces input data rate into correlation chip
For xNTD correlation cell data rate 3 times less compared to systolic array
For individual correlation cell 2 sets of 16 inputs requires 256 clock cycles to process. Data rate reduction up to a factor of 16 This value approached for SKAMP3 and LFD
For xNTD need ~5 byte wide busses into each correlator FPGA. (input data rate higher than correlator clock)
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Board Manufacture Simplification (2)
Correlation cell also reduces data duplication
SKAMP1 4x4 systolic array, EVLA 8x8 systolic array Data duplication 8 in EVLA, higher in SKAMP1 due to array reuse
Each Correlation Cell process 256 correlations at once Can reduce size of systolic array by sqrt(256)=16
No data duplication on board for up to 150 antennas
Data duplication none for xNTD, 1.5 for SKAMP3, and 3.5 for LFD, LFD 12 FPGAs data input 1/sqrt(12) of total
Correlation cell leads to a large simplification of correlator board
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Putting it Together – The SKAMP protoype
Routing
AutocorrelationsLong Term Accumulations
CorrelationsCorrelator interface
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Input, Daisy Chain, Route, & Autocorrelate
Two FX20
Interconnection for high antenna number designs
Input 16 rocket I/O on unidirectional Infinband
Output 16 Rocket I/O unidirectional Infiniband
Can daisy chain modules for reuse of data in further processing modules or conform to standards, Infiniband …
Two FX20
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Compute Engine
Six SX35 FPGAs Input 24, Output 18 LVDS
per FPGA 1152 Correlation cells total Up to 294k correlations on
board at a time (256 per cell) Data re-ordering in filterbank
to achieve Process of 512 time values for
each frequency channel Then dump to LTA
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Long Term Accumulator
Number of Correlation require DRAM for storage
Data rate requires two DIMM modules for three SX35
Half of Correlation and LTA hardware
Three SX35 Two LX15 + DDR2 DIMM
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Estimated Performance
Correlator board clock rate 330MHz, 192 cells/FPGA, 6 FPGAs Board processing rate 400GCMACs/s = 2.8Tops/s
Power consumption ~100W Power efficiency 0.25W/GCMAC (4bit FX)
Filterbank board 3.2Gsample/s, two polyphase filterbanks ~32 operations per sample Board processing rate 100Gops/s (18 bit)
Power consumption ~60W Power efficiency 0.6W/Gop
www.ict.csiro.au
New Generation CorrelatorsGroningen 26-29 June 2006
Conclusion
Common hardware, hardware modules and VHDL for xNTD, SKAMP3 and LFD
SKAMP3 in the lead with filterbank and correlator hardware well on the way
Initial Manufacture this year Correlator common to all using correlation cell to
gain required flexibility xNTD beamformer hardware starts July 06
LFD receiver will piggy back on this
Developing international project - distributed design team