APA integration test for the ProtoDUNE-SP LAr TPC at CERN Junbin Zhang On behalf of the cold electronics team Brookhaven National Laboratory 21 st IEEE Real Time Conference June 12,2018
APA integration test for the ProtoDUNE-SP LAr TPC at CERN
Junbin ZhangOn behalf of the cold electronics team
Brookhaven National Laboratory
21st IEEE Real Time Conference June 12,2018
Deep Underground Neutrino Experiment (DUNE)
6/12/2018
over 1000 collaborators from 175 institutions in 30 countries
4x10 kton LAr TPCsFar Detector at 1.5km underground:
4x10 kton Liquid Argon Time Projection Chambers (TPCs)
Two TPC concepts:• Single Phase (LAr)• Dual Phase (Ar gas + LAr)
J. Zhang - 21st IEEE RT
NP04 (ProtoDUNE-SP) ~ 1% of DUNE
6/12/2018
CPA modules
Field cage modules
Field cage modules with ground plane
APAsEnd wall field cage modules
– ProtoDUNE-SP TPC: ▪ 6 full-size (6x2.4 m2) Anode Plane
Assemblies (APAs) ▪ A total of 15,360 TPC channels
▪ 3 Cathode Plane Assemblies (CPA)
• 2 x 3.6 𝑚2 drift regions
– Goals:▪ Prototyping production and installation
procedures for DUNE▪ Validate design from perspective of basic
detector performance▪ Accumulate test-beam data to
understand/calibrate response of the detector
▪ Demonstrate long term operational stability of the detector.
J. Zhang - 21st IEEE RT
Readout electronics (cold side)
6/12/2018
Cold side
7m cold cables
20 FEMBs / APA
8 FE ASICs + 8ADC ASICs / FEMBJ. Zhang - 21st IEEE RT
Readout electronics (warm side)
6/12/2018
Warm side
1 WIEC / APA1 flange board / APA1 PTC / APA1 PTB / APA5 WIBs / APA
J. Zhang - 21st IEEE RT
FEMB production
6/12/2018
FE + ADC ASIC Screening FEMB Assembly FEMB Screening (Preliminary)
FEMB Final Screening
(FEMBs assembled in CE boxes)
Packaging
CERN
RT(300K)LN2(77K)
RT(300K)LN2(77K)
RT(300K)LN2(77K)
J. Zhang - 21st IEEE RT
APA Integration Test
CE box checkout (in barrack) Install CE boxes on the top of APA CE box checkout ( on APA)
Move APA into the cold boxExperimental area--- extension EHN1 hallMove APA into Cryostat
Warm test
Cooling down
Cold test(150 K)
Warming up
Warm test
6/12/2018 J. Zhang - 21st IEEE RT
ENC Performance vs Temperature
6/12/2018
APA2 (2018-01) Lowest temperature reached ~ 159K
1. Uniform gain (77 e-/bin) is applied for calculating noise of all channels2. HV Bias voltages were off3. Data are read out chip by chip over local diagnostic GbE port.J. Zhang - 21st IEEE RT
Cooling down
ENC at 159K:U-plane: 481 e-
V-plane: 481 e-
X-plane: 398 e-
Cryogenic temperature Warming up
Data flow (FEMB in cold side)
6/12/2018
FE_ASIC1-4
ADC_ASIC1-4
con
nec
tor
FE_ASIC5-8
ADC_ASIC5-8
con
nec
tor
FPGAEP4CGX50DF27C7N
TX0
TX1
TX2
TX3
Det
ecto
r (1
28
ch
ns)
16 × 4 = 64 𝑐ℎ𝑛𝑠
16 × 4 = 64 𝑐ℎ𝑛𝑠
29 × 16𝑏𝑖𝑡 × 2𝑀𝐻𝑧 × 1.25(8𝐵/10𝐵) = 1.16𝐺𝑏𝑝𝑠
32 × 16𝑏𝑖𝑡 × 2𝑀𝐻𝑧 × 1.25(8𝐵/10𝐵) = 1.28𝐺𝑏𝑝𝑠
Data rate of physical link
Data rate of payload
For each Tx link:
@2𝑀𝑠𝑝𝑠/ADC
@2𝑀𝑠𝑝𝑠/ADC
J. Zhang - 21st IEEE RT
Data flow (WIB in warm side)
6/12/2018
FEMB1
TX0
TX1
TX2TX3
FEMB2
TX0
TX1
TX2TX3
FEMB3
TX0
TX1
TX2
FEMB4
TX0
TX1
TX2
TX3
TX3
Equ
aliz
erEq
ual
izer
Equ
aliz
erEq
ual
izer
CO
LDA
TA_R
X(T
ran
scei
ver)
gro
up
1
CO
LDA
TA_R
X(T
ran
scei
ver)
gro
up
2
CO
LDA
TA_R
X(T
ran
scei
ver)
gro
up
3
CO
LDA
TA_R
X(T
ran
scei
ver)
gro
up
4
LIN
K F
IFO
sLI
NK
FIF
Os
LIN
K F
IFO
sLI
NK
FIF
Os
FEM
B d
ata
pro
cess
or
RCEPCS
FELIXPCS
EventBuilder
QSFP+
RCE
SFPGbEUDP_IO
Slow control
Timingprocessor
FELIX
link speed: 9.6 𝐺𝑏𝑝𝑠
Link speed: 1.0 𝐺𝑏𝑝𝑠
PC
link speed: 5 𝐺𝑏𝑝𝑠
Link speed: 1.28 𝐺𝑏𝑝𝑠
PTC
Warm Interface Board (WIB)
PTB
Timing (fiber)
J. Zhang - 21st IEEE RT
ProtoDUNE-SP WIB-FELIX readout
6/12/2018
• Xilinx Kintex UltraScale• 16-lane PCIe Gen3.0, throughput tested ~101Gb/s• 48 optical transceivers (MiniPOD)• On-board timing interface
BNL-712 (FLX-712) PCIe card
APA
WIB1
WIB2
WIB3
WIB4
WIB5
FELIX
FULL mode: @9.6 Gbps per link
One FLX-711/712 card has enough bandwidth to read out one APA with 10 FULL mode links.
J. Zhang - 21st IEEE RT
WIB-FELIX test stand @ BNL
6/12/2018 J. Zhang - 21st IEEE RT
FEMB1 FEMB2 FEMB2 FEMB3
WIB
QSFP+ GbE
FLX-711
Gb
EFELIX server
⚫ FPGA DAC (1,2,3,4,5,8)⚫ Gain = 25mV/fC⚫ Peaking time = 1us
Pedestal from one WIB
6/12/2018
Collection wires: X
Induction wires: V
Induction wires: U
J. Zhang - 21st IEEE RT
Calibration Waveform
6/12/2018
dac = 1,2,3,4,5,8
Collection wire
DAC = 1,2,3,4,5,8
J. Zhang - 21st IEEE RT
Calibration: ADC (Peak & Area) vs. Charge
6/12/2018
Peak vs. Charge
Area vs. Charge
DAC = 1,2,3,4,5,8 INL = 0.212%
DAC = 1,2,3,4,5,8INL = 0.184%
J. Zhang - 21st IEEE RT
ProtoDUNE-SP –Present and Future
6/12/2018
All 6 APAs and 3 CPAs have been installed into the ProtoDUNE-SP cryostat
• Cold Electronics (CE) and photodetectors (PD) have been tested before and after the installation on APAs
• All CE and PD cables are routed though the chimney out of the
cryostat and connected to the corresponding crate.
• Integration and readout tests are ongoing at CERN and BNL
• Beam run is scheduled from August 29th to November 11th
J. Zhang - 21st IEEE RT
Summary
6/12/2018
• BNL developed cold electronics towards low temperature (77K-89K) is an enabling technology for liquid detectors. All CE boxes were assembled at BNL and shipped to CERN after a comprehensive set of QA/QC tests.
• All APAs are fully characterized before moving into the cryostat.
• Good ENC performance reached at 159K (398 e- for X, 481e- for U,V)
• BNL will advance the design of DUNE cold electronics and continue to contribute efforts at FELIX readout for ProtoDUNE and DUNE in future.
J. Zhang - 21st IEEE RT
6/12/2018 J. Zhang - 21st IEEE RT
Thank You!
https://jobs.bnl.gov/job/upton/postdoctoral-research-associate/3437/8059129
•This job post is for multiple openings
•The successful candidate is expected to play an essential role in FELIX and
Global Trigger development in the ATLAS upgrades and other high-energy
physics, nuclear physics and astronomy experiments. • Design, develop, prototype, and produce hardware and firmware for the ATLAS experiment
• Evaluate hardware and firmware and characterize system performance
• Participate in system integration of multiple combined systems
Job opening at BNL for Trigger & DAQ
Backup slides
6/12/2018 J. Zhang - 21st IEEE RT
• FE ASIC
– Built-in 6-bit DAC for calibration pulse generation
– Built-in analog monitoring output for debug
– Address pole-zero cancellation and drive capability in buffer-off mode
– Add higher bias current (1nA and 5nA) options and smart reset
– Revise BGR start-up circuit and increase ESD protection on I/O
– Will be used to instrument SBND and ProtoDUNE-SP
• ADC ASIC
– Implement COLDATA (DUNE baseline design by FNAL, prototype expected in FY19) compatible interface and FE ASIC compatible configuration
– Address the early saturation and roll-back
– Implement power-on default configuration and extend soft-control functions
– Revise BGR start-up circuit and increase ESD protection on I/O
– Improve ADC INL/DNL → not completely resolved
– Will be used to instrument ProtoDUNE-SP
– SBND is exploring COTS ADC option
– Cold ADC ASIC development is very challenging given the amplified mismatch error and inaccurate simulation model in cryogenic temperature
CMOS Cold ASICs Upgrades Implemented
4.5 mm
6.1
mm
6/12/2018 J. Zhang - 21st IEEE RT
Front End Electronics production at BNL
• Based on DUNE design, 20CE boxes have been installed on each APA (120 CE boxes, 15360 electronic channela in total).
• CE boxes were assembled at Brookhaven National Laboratory (BNL) and shipped to Cernafter a comprehensive set of QA/QC tests.
• Integration tests in LN2 were performed at BNL by a 40% of DUNE APA (2.8 m x 1.0 m) in a smaller cold box.
J. Zhang - 21st IEEE RT
Inside Cold BoxFilled with LN2
WIEC
Ch
imn
ey
40% APA
6/12/2018
Final: CERN Grounding Scheme (with SHV cable connected)
20 CE boxes(FEMBs)
ProtoDUNE APA (6m x 2.5 m)
Cold Box
5WIB
Faraday cage
CR + PSL
WIENER PS
+48 V
0V(GND)230V
Detector ground
WIEC
Flange boardFT
Optical SW
Ethernet cable
Optical fibers
7 meter samtec power cables & data cables
48V PS cable (20m)
LV PS (-)detector ground
LV PS (+)
Fans PS cable (20m)
+18 V230V0V(GND)
DCFans
PS RACK
open
~35ft (10m)25 mm2
Building ground
PTB
PTC
Panel
Note: 1. Heater cable is connected but not power applied. The shielding of heater cable is connected to Heater PS but open on WIEC crate (same as Fans PS cable)2. Two SHV cables (-650V for G plane and 800V for X plane) connect to HV power supply3. Currently, U plane is shorted to Detector ground somewhere (investigate after cold test). The SHV cable for U plane is disconnected4. The SHV cable for ED outer is disconnected
HV
230V 0V(GND)
10 kohm
7 meterHV cables
HV
box
SHV cables (20m)
Note:Power supply for DC fans are separated from supply for CE, the shielding of fans PS cable disconnects to flange. (DC fans are isolated from WIEC)
6/12/2018 J. Zhang - 21st IEEE RT
Equivalent Noise Charge (ENC) – Integrals
Noise Coefficients A1, A2, A3
2 2 2 2 2 2 211 11 2 0 3 2 0 32 2n in in f n in in f
AENC e C I C A I qI I e C C A A qI A
= + + = + +
( ) ( )
( ) ( )
( ) ( )
2 2 2 11
221 2
2 2
22
3 3
1
2
1
2
1
2
AI w t dt H j d
I w dt H j d A
I w t dt H j d A
− −
− −
− −
= = =
= = =
= = =
( ) ( ) ( )2 22 2 21
2
0
1 1
2 2neq neqENC i H d i H d
−
= =
Series
white
1/f
Parallel
white
Time domain
(weighting function)Frequency domain
(transfer function)6/12/2018 J. Zhang - 21st IEEE RT
Noise (ENC) vs TPC Sense Wire and Signal Cable Length for CMOS at 300K and 89K
MIP Signal for 3x3 and 5x5 mm Sense Wire Spacing
J. Zhang - 21st IEEE RT
CMOS at 77K: ENC< 103 e rms
MicroBooNEENC~400 e rms
DUNE with warm electronics (300K)ENC~6x10^3 e rms
6/12/2018
At 77-89K, charge carrier mobility in silicon increases, thermal fluctuations decreasewith kT/e, resulting in a higher gain, higher gm /I, higher speed and lower noise.
Cold vs. Warm CMOS: static characteristics vs. T
0.0 0.3 0.6 0.9 1.2 1.5 1.810
-5
10-4
10-3
10-2
10-1
100
101
~18m
V/d
ec
~72m
V/d
ec
(ln(
10)n
VT)
CMOS018
MEASURED
ID
gm
LN
RT
I D [m
A],
gm
[mS
]
VGS
[V]
NMOS, L=0.18µm, W=10µm
~ 30 300
~ 116 77
m
D B
at T Kg q
at T KI nk T
=→ =
=
10-6
10-5
10-4
10-3
10-2
10-1
100
101
102
0
20
40
60
80
100
120 MEASURED
NMOS PMOS T=300K
L=360nm
L=270nm
L=180nm
NMOS PMOS T=77K
L=360nm
L=270nm
L=180nm
gm
/ID [
V-1
]
Drain Current Density [mA/mm]
CMOS018
Transconductance / drain
current