T2K Time Projection Chambers Front-end Electronics – Experience Return P. Baron, H. Bervas, D. Besin, D. Calvet, T. Chaleil, C. Coquelet, X. de la Broïse,

T2K Time Projection Chambers Front-end Electronics –

Experience Return

P. Baron, H. Bervas, D. Besin, D. Calvet, T. Chaleil, C. Coquelet, X. de la Broïse, E. Delagnes, F. Druillole, A. Le Coguie, E. Monmarthe, J-M. Reymond,

E. Zonca

DSM/IRFU, CEA Saclay, France

O. Ballester,

IFAE, Barcelona, Spain

[email protected] 2 Saclay 2-3 December 2009

Plan

Presentation of the T2K experiment

Architecture of TPC readout electronics

Principal components

Current status

Experience return


Tokai to Kamioka (T2K) experiment

Main Physics Goal: neutrino oscillation

• μ disappearance for improved accuracy on 23

• e appearance to improve sensitivity to 13

50 kT water


T2K Time-Projection Chambers

TPC features

• Resolution goal: 10% for p < 1 GeV/c

• Double wall design; inner wall is the field cage

• Total ~9 m2 instrumented with 72 Micromegas amplification modules segmented in ~7 mm x ~10 mm pads

• Custom readout electronics

1 of 3 TPCs shown

1728 pad (36 x 48)Micromegas module

34 cm x 36 cm

CalibrationGas system Electronics; on-line software

1 m

2 m


TPC Readout Challenges

Environment

• Embedded front-end: limited access, need low power

• No radiation, low magnetic field (0.2 T), Japanese underground facility

Highly segmented detector

• 124.000 channels over 9 m2 of instrumented area

→ Massive replication of identical modular building blocks

Extreme burstiness of data

• 124.000 channels; sampling: 12-bit 33 MHz → 50 Tbps

→ beyond capability of fully digital solution for power budget target

Large raw event size but modest average dataflow• 50 Tbps during ~15 µs drift time → 90 MBytes

• Allowable event size for storage: ~250 KBytes

→ data reduction of ~400 to achieve in real time• Spill repetition period: ~3.5 s; Cosmic calibration: 20 Hz max.

• DAQ average rate: ~250 KBytes × 20 Hz = 5 MByte/s

→ commercial computers and networking techniques


Plan




Current status

Experience return


Logical Read-Out Flow

Architecture principles

• Custom front-end ASIC; analog memory (Switch Capacitor Array)

• ADC + digital buffer mounted close to the detector

• Multiple optical fibers send data to off-detector concentrator cards

• Interface to common DAQ via standard gigabit Ethernet network

Pre-amp and shapers

Samplers and analog memory buffers

Analog to digital conversion

Digital buffer – zero-suppression

Data concentrator Clock/Trigger fanout

~124.000 channels

1728 ASICs

On

-det

ecto

r

72 Optical fibers

~4 Tbaud*/s peak*1 baud = 12 bit

~2 ms retention max.

34 Gbaud/s peak

400 Gbit/s peak

~1-10 Gbit/s averaged

Shared DAQ~0.1-1 Gbit/s

432 FrontEnd Cards

18 DataConcentrator Cards

72 Front-EndMezzanines cards

Off

-det

ecto

r

Global Clock/TriggerGigabit Ethernet


Architecture Implementation3 TPCs

1 m

2,5

m

2,5 m

Network

TCP/IP

PC Linux

DCCs

PrivateEthernet

GigabitEthernet

DAQ control

DetectorB

DetectorA

Global trigger

1 of 6 TPC end-plates(12-modules)

Outside magnetInside magnet

2 of 12 (or 18) DataConcentrator

Cardsx 6

12 duplexOptical fibres

1 of 72 modules

Front End MezzanineCard (FEM)

288 channelFront End Card (FEC)

1728 pad Micromegas plane

Slow controlnetwork

Optical fiberto/from DCC

Low voltagePower supply

1 of 1728 Front-End ASIC “AFTER”

72 channel x 511 time bucketsSwitched capacitor array


Plan




Current status

Experience return


ASIC “AFTER” Functional Diagram

AFTER

511 cells

SCAFILTER

100ns<tpeak<2us

CSA

1 channel

x72(76)

76 to 1BUFFER

SCA MANAGERSLOW CONTROL

Serial Interface

W / R

Mode

CK

CK

ADC

TEST

In Test

120fC<Cf<600fC

Power Supply Reference Voltage Reference Current

Asic Spy Mode

CSA;CR;SCAin (N°1)

Power

On Rese

t

Design features

• 72 channels x 511 analog memory cells; Fwrite: 1-50 MHz; Fread: 20 MHz

• 4 Charge Ranges (120 fC; 240 fC; 360 fC & 600 fC)

• Supports positive or negative input signals

• 16 Peaking Time Values (100 ns to 2 µs)

Asic

For

Tpc

Electronic

Read-out


The AFTER chipDesign facts

• 0.35 µm CMOS - 500.000 transistors

• 7.8 x 7.4 mm die – 160-pin 0.65 mm pitch LQFP

• single 3.3V supply - 8 mW/channel

Performance

• Integral non-linearity < 1.2% full range; ENC: 350 electrons (no load)

• Stored charge degradation after 2 ms retention: 0.18 LSB

→ All specifications met on first silicon!

AFTER chip used to readout TPC Micromegas and FGD Silicon PM

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 11000 12000 13000 14000 15000

DAC bin

Inte

gral

Non

_Lin

earit

y =

(dat

a - f

it)/fi

tmax

(%)

100ns

2µs


AFTER chip test and production

Production of the AFTER chip

• 5300 chip produced; 4750 OK; Yield: 89%

• 1800 chips delivered to T2K (TPC + FGD + monitoring chamber)

LABVIEW Test Software

Protection 1 Protection 2 No Protection

AFTER Test Card

Test bench

• AFTER test card served as pre-prototype of Front-End Card

• Xilinx Virtex 2 Pro eval. kit as pre-prototype of FEM board for readout

• DAQ and Analysis with Ethernet PC and LabView


Front-End Card (FEC)

Features

• All design concepts validated on AFTER test board

• 1 FEC reads out 288 channels – 6 layer PCB

• Consumption: 1 A – 4 W

Quad-channel12-bit ADC AD9229

Passive protection Circuits

4 AFTER chips

3.3V regulatorsSilicon ID chipV - I –T monitor

+4V input

Four dual row 80-pin 1.27 mm pitch connectors to detector

PulserClock fanout

PhotoMos relay

80-pin connector to FEM


Main functions

• Drive 6 FECs and aggregate data produced (1728 channels, 5.7 Gbps)

• Buffer one event (raw data), i.e. ~10 Mbit

• Deliver data to DCC upon request: raw data or zero-suppressed (one programmable threshold per channel)

• Configuration and slow control, voltage, current, temperature monitoring

Front-end Mezzanine (FEM) card

ZBT SRAM2 Gbps optical transceiver

CANbus slow control (RJ45)

FPGA

µC

FEC#1

Voltage regulators

Voltage regulators JTAGPower In

ConnectorFEC#0Buffers FEC#2

FEC#3 FEC#4 FEC#5


Detector Module Read-out Electronics

72-channel ASIC

Quad-channel ADC

digital Front-endMezzanine card (FEM)

Optical Transceiver

FPGA

80-pin connector

288-channel analogFront-End Card (FEC)

1728-pad detector plane

Slow-controlNetwork - CANbus

Fiber toDCC

Low voltagepower

Materializing the concept

• 3 year development of ~6 equivalent persons from ASIC specifications to working readout of one detector module


Plan




Current status

Experience return


Integration of TPC#0 at Triumf

Feb. ‘09


Integration of TPC#0 and #1 at TokaiPhotos: Claudio Giganti


Latest news from T2K nd280 m

TPC #2

• Tested at Triumf and now in the clean room at Tokai

• Installation in pit on Dec’ 17th

→ Fully commissioned TPCs by mid-Jan’10 for closing magnet

TPC #0 and TPC #1

• Installed in the magnet; DAQ commissioning in progress

• Awaiting final gas mixture to power central cathode and detector HV

→ Detector commissioning starts Dec. 7th

The nd280 collaboration announced the observation of the first beam neutrino candidate (in the INGRID detector) two weeks ago!


Plan




Current status

Experience return


Contribution of Irfu to T2K nd280m

Electronics

• ASIC design, production test, readout electronic cards (design and production supervision)

Physics

• Simulation and experimental data analysis of MicroMegas detectors

Detector engineering and integration

• Design of Micromegas detectors, production coordination, QA

• Cooling mechanics and structure, cabling, services (HV, LV, fibers…)

Firmware and software

• Embedded firmware and software in the front-end, slow control, DAQ for detector test bench, firmware and software for DCCs, LV control, etc.

Project management

• M. Zito co-leader of TPC project with D. Karlen (Canada), A. Delbart, D. Calvet coordinator of “Micromegas modules” and “Readout Electronics”

→ Project relies on the diversity of competence at Irfu, teams above critical size and adequate funding


Project Difficulties Production

• First batch of wafers of the After chip did not pass QA

→ 2 months delay to get After chips

• First batch of FECs failed: humidity stored in After chip package during test requires extended baking procedure

→ more than 2 months to understand the problem and find workaround

• Defects on 10% FEM for first batches (broken via below the BGA)

no consensus reached between designer, PCB manufacturer, and assembly company on exact cause (coating on vias?).

→ three month delay: defect analysis, modifications to PCB, new batch

• Could not solder some connectors on FEMs using lead free process

Front-end production took 1 year while 6 months had been planned


Project Difficulties (con’t)

Back-end electronics

• DCC prototype board did not become functional

Unanticipated contribution of Irfu to back-end HW/FW/SW and DAQ.

→ Back-up solution based on customized commercial evaluation boards deployed by common effort of Triumf/Irfu/Lpnhe.

CommercialXilinx Virtex 4 ML405

Evaluation board

CustomClock board

CustomOptical

Transceiverboard

FGD DCC Crate


What really helped

Commercial FPGA evaluation boards

• Inexpensive, readily available, customizable hardware

• Can develop and debug firmware/software on proven hardware– Firmware/software development before a dedicated board is available– Simplifies dedicated board debugging because firmware/software already

tested and only migration issues need to be solved

• Common platform for collaborative development among various group

→ ~45 Xilinx ML405 boards purchased by the TPC-FGD groups! Used at Saclay, Paris, Rennes, Geneva, Barcelona, Hamburg, Tokai, Vancouver

$995.00Part Number:

HW-V4-ML405-UNI-G


Summary

• What?

A 124.000 channel readout system for the 3 TPCs of the T2K experiment

• How?

A new 72-channel ASICs based on a 511 time-bucket SCA

Custom made readout cards at the back of each 72 Micromegas detectors

Extensive use of commercial FPGA evaluation boards

• Principal merit?

High density, low power: ~16 mW/channel (2 kW for 124.000 ch.); ~3 €/ch.

• Summary and Status

5 year development effort from concept to installation on site

→ TPC commissioning at J-PARC starting in the next few days

T2K Time Projection Chambers Front-end Electronics – Experience Return P. Baron, H. Bervas, D. Besin, D. Calvet, T. Chaleil, C. Coquelet, X. de la Broïse,

Documents

segmented detector

cea saclay

spain slide

data concentrator cards

kt water slide

kamioka t2k

gbits peak

s drift time