Page 1
Galfa Spectrometer
Jeff Mock, Dan WerthimerJeff Mock, Dan WerthimerStudents: Henry Chen, Pavel Monat, Aaron Parsons, Wonsop SimStudents: Henry Chen, Pavel Monat, Aaron Parsons, Wonsop Sim
Science Advisors: Carl Heiles, Snezana StanimirovicScience Advisors: Carl Heiles, Snezana Stanimirovic
http://seti.berkeley.edu/galfa
Page 7
Image RejectionImage Rejection
• 45 dB at +- 10 MHz45 dB at +- 10 MHz
• 40 dB at +-20 MHz 40 dB at +-20 MHz
• 35 db at +-30 MHz35 db at +-30 MHz
Page 8
Gain ControlGain Control
• 0 to 40 dB gain – each of 16 channels0 to 40 dB gain – each of 16 channels– (2 spare IF to baseband converters)(2 spare IF to baseband converters)
• Controlled by RS232 serial portControlled by RS232 serial port
• Overflow/Saturation detectionOverflow/Saturation detection
• Remote Operation (Displays as well)Remote Operation (Displays as well)
Page 9
GALFA SpectrometerGALFA Spectrometer
Page 11
Multi-Purpose FPGA-Multi-Purpose FPGA-Based Spectrometer Based Spectrometer (NSF, A. Parsons)(NSF, A. Parsons)
XilinxVirtex-II 6000
FPGA
XilinxVirtex-II
1000FPGA
256 MB DRAM
AreciboFeed
Array
200 MhzADC
Compact PCIBackplane
Software
200 MhzADC
200 MhzADC
200 MhzADC
I
I
Q
Q
Pol. 1
Pol. 2
{
{
200 Aux. I/O
Page 12
compactPCIcompactPCI•PowerPower•CoolingCooling•Robust/ReliableRobust/Reliable•InexpensiveInexpensive
FPGA-Based FPGA-Based SpectrometerSpectrometer
Page 13
SETI ApplicationsSETI Applications• ALFA Sky Survey (300 MHz x 7 beams)ALFA Sky Survey (300 MHz x 7 beams)
• Parkes Southern SERENDIPParkes Southern SERENDIP
• JPL/UCB/SI Survey (20 GHz Bandwidth)JPL/UCB/SI Survey (20 GHz Bandwidth)
• SETI Italia (Bologna)SETI Italia (Bologna)
• SETI@homeSETI@home
Astronomy ApplicationsAstronomy Applications• GALFA Spectrometer – Arecibo Multibeam Hydrogen SurveyGALFA Spectrometer – Arecibo Multibeam Hydrogen Survey
• Astronomy Signal Processor – ASP – Backer/Stairs (pulsars)Astronomy Signal Processor – ASP – Backer/Stairs (pulsars)
– GreenBank and Arecibo, soon Nancay GreenBank and Arecibo, soon Nancay
• ATA4 Correlator F Engine ATA4 Correlator F Engine
• Two Reionization Experiments Two Reionization Experiments
(Backer/Demorest/Peeks (UCB), Chippendale/Green/Ekers (ATNF)) (Backer/Demorest/Peeks (UCB), Chippendale/Green/Ekers (ATNF))
Page 14
GALFA SpectrometerGALFA Spectrometer
sin
cos
LPF
LPF
100 MHz
-50 to +50 MHz
sin
cos
LPF
LPF
100 MHz
-50 to +50 MHz
QuadratureDownconverter
Board
IF Pol. 1
IF Pol. 2
Biplex256 pnt.
PFB
e^-it
e^-it
FIRLPF
FIRLPF
Digital LO
Decimateby 14
Decimateby 14
Biplex8192 pnt.
PFB
Detect
Detect
cPCIBackplan
eto
CPU
Multipurpose Spectrometer Board
Page 15
PFB vs. FFTPFB vs. FFT
Page 16
PFB vs. FFTPFB vs. FFT
Page 17
PFB vs. FFTPFB vs. FFT
Page 18
GALFA Lowpass FilterGALFA Lowpass Filter
Page 19
GALFA Lowpass FilterGALFA Lowpass Filter
Page 20
GALFA 2v6000 FPGA Resource Budget
Blockrams MultipliersBiplex 8k PFB
4x overlap FIR 16Coefficient tables 16delays 56
8k pt decimating FFT 3Twiddle tables 2delays 26
Biplex 256 PFB4x overlap FIR 16
Coefficient tables 4delays
256 pt FFT 18Twiddle tables 2delays
Dual Complex Mixer 8Dual lowpass filter 49Dual 256 point (x41) accumulator 4Dual 8k point (x32) accumulator 32Dual wideband power 4Dual narrowband power 4
Total 142 118
Chip resource 144 144
Page 21
GALFA DiagnosticsGALFA Diagnostics
Page 22
GALFA DiagnosticsGALFA Diagnostics
Page 23
GALFA DiagnosticsGALFA Diagnostics
Page 24
Integration TimeIntegration Time
• 988 mS fixed (99% duty cycle)988 mS fixed (99% duty cycle)
• Spectra read out every secondSpectra read out every second
• Integration starts 6 mS after 1 PPSIntegration starts 6 mS after 1 PPS
• Integration stops 6 mS before 1 PPSIntegration stops 6 mS before 1 PPS
• Timing accuracy +-10 nS wrt observatoryTiming accuracy +-10 nS wrt observatory
Page 25
Data OutputData Output
• Galfa.yyyymmdd.projname.sequence_number.FITSGalfa.yyyymmdd.projname.sequence_number.FITS
• file written every 15 seconds (programmable)file written every 15 seconds (programmable)
• 0.5 Mbyte/sec (8 MByte files)0.5 Mbyte/sec (8 MByte files)
• Files written over NFS to file serverFiles written over NFS to file server
• 2*8K + 2*256 32 bit powers (42 bit 2*8K + 2*256 32 bit powers (42 bit accumulators)accumulators)
• Time stamped (NTP), plus telescope dataTime stamped (NTP), plus telescope data
Page 26
Worries, WeaknessesWorries, Weaknesses
• ADC on same board with Digital Electronics ADC on same board with Digital Electronics
(spurs at +- 25.000000 MHz…)(spurs at +- 25.000000 MHz…)
• Dynamic Range Problems Dynamic Range Problems (Overflow/Saturation) (Overflow/Saturation)
(prog. gain/shifting, overflow detection) (prog. gain/shifting, overflow detection)
• Unknown LO phase shift (0 or 180 degrees) Unknown LO phase shift (0 or 180 degrees) between channels after power cyclebetween channels after power cycle– (LO is divided by two in downconverter)(LO is divided by two in downconverter)
Page 27
Spectrometer Spectrometer DocumentationDocumentation• Schematics, Cabling DiagramsSchematics, Cabling Diagrams
• Software (Source code)Software (Source code)
• Users ManualUsers Manual
• DiagnosticsDiagnostics
• About 100 Pages of stuffAbout 100 Pages of stuff
• http://seti.berkeley.edu/galfahttp://seti.berkeley.edu/galfa
Page 28
Short Term PlansShort Term Plans
• Tonight: Jeff Mock ArrivesTonight: Jeff Mock Arrives
• Monday – Thursday: Integration/TestMonday – Thursday: Integration/Test
• Friday – Sunday: ObserveFriday – Sunday: Observe
• This week: come and visit for demoThis week: come and visit for demo
feel, touch, and get to know your spectrometerfeel, touch, and get to know your spectrometer
Page 29
Long Term PlansLong Term Plans
• Find ETFind ET
Page 31
Future SETI Future SETI SpectrometersSpectrometers2015 4 THz 400 beams
10 GHz each2020 128 THz 12,800 beams
2025 4000 THz 40,000 beams
2030 128,000 THz 1M beams
Page 32
SERENDIP V
PolyphaseFilter Bank
Serverw/ EDT card
GbESwitch
PC
Serverw/ EDT card
Serverw/ EDT card
Serverw/ EDT card
PCPCPC
PCPC
GbESwitch
PCPC
PCPC
PCPC
GbESwitch
PCPC
PCPC
PCPC
GbESwitch
PCPC
PCPC
PCPC
100 MHz
100 MHz
Pol. 1
Pol. 2
Astronomy Signal Processor – Don Backer
Page 33
SERENDIP V SpectrometerSERENDIP V Spectrometer
Page 34
Biplex Pipelined FFTBiplex Pipelined FFTExample: 8 pnt. FFTExample: 8 pnt. FFT
-4Z
-2Z
Switch
Delay by NCoefficientMultiply
add
sub
Unity
-2Z
-1Z
add
sub
2nd Roots of Unity
-1Z
-0Z
add
sub
4th Roots of Unity
-4Z
Pol. 1
Pol. 2
Time Domain
Frequency Domain
Page 35
JFFT FFT controlsJFFT FFT controls• Any length transformAny length transform
• Input width and output widthInput width and output width
• twiddle coefficient widthtwiddle coefficient width
• 3 or 4 multiplier complex multiply3 or 4 multiplier complex multiply
• Specify downshift or programmable downshift Specify downshift or programmable downshift (optional rounding)(optional rounding)
• Decimate FFT optionDecimate FFT option
• Blockram / CLB memory threshold optionBlockram / CLB memory threshold option
• Overflow detectionOverflow detection
Page 36
JFFT additional PFB controlsJFFT additional PFB controls• Filter overlapFilter overlap
• Width of filter coefficientsWidth of filter coefficients
• Window function for filter (hamming, hann, etc.) Window function for filter (hamming, hann, etc.)
• Import filter coefficients for custom filter performanceImport filter coefficients for custom filter performance
Page 37
Moore’s Law in FPGA Moore’s Law in FPGA worldworldComputational Density Comparison
1000
10000
100000
1000000
10000000
10/28/1995
3/11/1997
7/24/1998
12/6/1999
4/19/2001
9/1/2002 1/14/2004
Release Date
(MO
PS
/MH
z)*l
am
da^
2 Processor Peak
FPGA 32-bit int MAC
FPGA maximum sustained performance
1
10
100
1000
10000
100000
12/1/1996
6/19/1997
1/5/1998
7/24/1998
2/9/1999
8/28/1999
3/15/2000
10/1/2000
4/19/2001
11/5/2001
5/24/2002
Release date
MO
PS
(3
2 b
it M
AC
)
100X More efficient100X More efficientthan micro-processors!than micro-processors!
3X improvement3X improvementper year!per year!
Page 38
Next Generation BoardNext Generation BoardBEE2 (2004/5) – Chen BEE2 (2004/5) – Chen ChangChang• 5 Xilinx XC2VP70 5 Xilinx XC2VP70
• 40 GB RAM (8 GB each chip, 13Gbit/sec/chip)40 GB RAM (8 GB each chip, 13Gbit/sec/chip)
• 18 10Gbit/sec infiniband ports18 10Gbit/sec infiniband ports
• 50 boards per rack, Tbit/sec infiniband switch50 boards per rack, Tbit/sec infiniband switch
• Applications:Applications:– 1 GHz, 1 Gchannel spectrometer (single board)1 GHz, 1 Gchannel spectrometer (single board)
– Next Generation ATA backends (ata32 = 2 boards)Next Generation ATA backends (ata32 = 2 boards)
– SKA imagingSKA imaging
Page 39
B2 Module: board layoutB2 Module: board layout• 5 compute
elements on a board
• Up to 400 billion CMAC/s performance
• communication bandwidth: – 240 Gbps on-
board 360 Gbps off-board
Page 40
Global InterconnectsGlobal Interconnects
• Commercial Infiniband switch from Mellanox, Voltaire, etc.– Packet switched, non-
blocking– 24 ~ 144 ports (4X) per
chassis– 480Gbps ~ 2.88Tbps
full duplex constant cross section bandwidth
– <$400 per port
ComputeNode
#N
ComputeNode
#1
Infiniband Crossbar Switch
Ethernet Switch
Page 41
19” 48RU Rack 19” 48RU Rack CabinCapacityCabinCapacity
• 40 compute nodes in 5 chassis (8U) per rack
• Up to 16 trillion CMac/s performance per rack
• 250 Watt AC/DC power supply to each blade
• 12.5 Kwatt total power
Page 42
Unified Digital Processing Unified Digital Processing Architecture Architecture
• Distributed per antenna spectral channel processing• Multiple reconfigurable backend application processing• Commercial packet switched interconnect• Backend data pulling through remote DMA access
PolyphaseFilter
Banks
ChannelReorderBuffer
Infin
iban
d S
wtic
h
XMAC imaging
Spectrometer
Pulsar SearchingB
eam
form
ing
PolyphaseFilter
Banks
ChannelReorderBuffer
An #1
An #N
Infin
iban
d
Sw
tich
Page 43
Happy Birthday CarlHappy Birthday Carl