om Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 1 CMS Calorimeter Trigger CMS Calorimeter Trigger SLHC Regional Calorimeter Trigger System Design and Prototypes Tom Gorski University of Wisconsin July 20, 2009
Jan 13, 2016
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 1
CMS Calorimeter TriggerCMS Calorimeter TriggerCMS Calorimeter TriggerCMS Calorimeter Trigger
SLHC Regional Calorimeter Trigger
System Design and Prototypes
Tom Gorski
University of Wisconsin
July 20, 2009
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 2
GCT Muon Aux Card UpdateGCT Muon Aux Card Update
Pushbutton Reset
(for Microblaze)S-Link
Connectors
TTS
TTCrx
RS-232GTP Links
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 3
GCT Muon Aux Card UpdateGCT Muon Aux Card Update
• Currently developing test firmware for the Aux Card
• Firmware to be completed in Fall, 2009
• Test firmware is a hybrid microblaze/HDL implementation
• User interface: C program via RS-232 to PC terminal
• Dedicated HDL blocks to test GTP, TTC, TTS & S-Link Interfaces
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 4
SLHC Cal. Trig. DemonstratorSLHC Cal. Trig. DemonstratorSLHC Cal. Trig. DemonstratorSLHC Cal. Trig. DemonstratorSLHC Cal. Trig. Demonstrator Prototype Design
• First Step: determine requirements to provide proof of principle of major elements of SLHC Cal. Trig. System
Evaluate SLHC Cal. Trig. Conceptual Design• Needs performance to incorporate algorithms
• Shown by M. Bachtis, S. Dasu, K. Flood (UW) in this workshop• Compatible with firmware developed for these algos.
• K. Compton, M. Schulte, et al., (UW) Proceedings of IEEE Symposium on Field-Programmable Custom Computing Machines, April 2009.
• Must work with ECAL1, HCAL2 upgrade designs & eventually in combination with upgraded tracking trigger primitives• Ph. Busson1 (LLR), J. Mans2 (UMinn)
• Integrated with SLHC Optical SLB• J.C. da Silva, LIP.
• Clear evolutionary path from present LHC RCT• Backwards-compatible functionality maintained.
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 5
RCT
Calorimeter Trigger EvolutionCalorimeter Trigger EvolutionCalorimeter Trigger EvolutionCalorimeter Trigger Evolution
RCT
GCT:Sources
GCT:Main GT/GMT
GCT/uTCA
ETCC:TPGs
HTR:TPGs
Cu
FO
RCT
GCT:Sources
GT/GMT
GCT/uTCA
uTCA-HTR:TPG
oSLBRCT/
uTCA
GCT/uTCA
GT/GMT
ETCC:TPGs
uTCA-HTR:TPGs
oSLB
Step 1 (2009) Step 2: ↓ OR ↓
RCT/uTCA
ETCC:TPGs
uTCA-HTR:TPGs
oSLB
Step 3 Step 4
oSLBoSLB
GCT:Sources
GT/GMT
GCT/uTCA
RMCRMCRMC
SLB SLB
ETCC:TPGs
SLB
Matrix& AuxCards
oSLBoSLB oSLB
HTR:TPGs
oSLB
oSLB
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 6
RCT Receiver Card
Review: Current HCAL/ECAL to RCT LinkReview: Current HCAL/ECAL to RCT Link
HCALHTR orECALTCC
Vitesse
V7216 Tx
RCTRCVRMezz
Vitesse
V7216 Rx
RCT
Phase
ASIC
4X 1.2 Gbps Copper Links(19 bits data + 5 bits Hamming per link per crossing)
• Intersection of 4,032 links at common destination (RCT)
• Link Xmt & Rcv Clks are 3X the LHC Clock (~120 MHz)—no long term drift issues
• Elastic buffers in V7216 Rx chips to manage short term clk jitter
• RCT Phase ASIC provides channel bonding function, hamming code check, and buffering for 4X RCT processing pipeline (~160 MHz) in about 3 crossings
• 120 MHz Link Clock skew tolerances: ±6ns @ Tx, ±1ns @ Rx
• Scheme is stable, reliable, and has low latency
4.8 Gbps aggregate
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 7
Clock Frequencies and Data RatesClock Frequencies and Data Rates
• Calorimeter data produced by a 40.08 MHz synchronous process
• Carried by high-speed serial links between processing stages
• FPGAs support different clock domains for the high speed serial links and the programmable fabric, but at the cost of increased latency at the interface
• Governing parameter is the Link Parallel I/O Clock Frequency
• Key Decision: Run FPGA processing fabric and/or serial links synchronous to the LHC clock or on a decoupled timebase?
• Decoupling frees designers to chose frequencies that maximize link data rates
• Price of decoupling:
• Increased latency on link/fabric interface
• Increased complexity in pipeline control and link protocol (empty cycles)
• Advantages of coupling:
• Perfect match in bandwidth between calorimeter, links, and processing pipeline
• Lowest latency on interfaces
• Simplified pipeline control and link protocol
• Existing Latency Constraints + Xilinx Rocket I/O Tile design exerts a strong influence towards the coupled approach
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 8
TPG Decompression FunctionsTPG Decompression Functions
• RCT Currently uses full LUTs for each H/E tower pair (217 × 18 bits)
• Implements arbitrary conversion function plus H/E cut for electrons
• LUTs require significant board space, not practical for upgrade—need to go to functions instead
• Functions can place significant demands on special FPGA resources (e.g., DSP slices, block RAM)
• Demand depends on function type:
• Mantissa/Exponent least costly—simple shift
• Piecewise linear uses Add/Multiply per HCAL or ECAL tower
• Logarithmic/Exponential about 4-8× more costly than piecewise linear
• Will need to settle on general scheme early on in the process in collaboration with ECAL and HCAL groups
• Pursued design approach will have parameters stored in FPGA RAM, and allow changes to them without requiring FPGA resynthesis
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 9
oSLB RCT side (LIP Development)oSLB RCT side (LIP Development)
FPGAOpticalRcvr
OpticalXmtr
Incoming TPG(4.8 Gbps fromoSLB TPG side)
Repeated TPG to SLHC RCT
120.24 MHz Clock
120 MHz Synchronous Parallel Data to Phase ASIC
Latency for Current 7216-based Link:• 3.4 bunch crossings for cable (85ns)
• 0.8 bunch crossings for 7216 Tx (19ns)
• 2.5 bunch crossings for 7216 Rx (62ns)
• TOTAL: 6.6 bunch crossings (166ns)
• Based on what we know so far about Rocket I/O, the budget may need to be increased by 1 or 2 crossings
(from J. C. De Silva)
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 10
SLHC RCT Block Diagram(56η × 12φ slice)
SLHC RCT Block Diagram(56η × 12φ slice)
TT
C/D
AQ
Ca
rd
Pro
ce
ss
ing
Ca
rd
(Backplane)
Pro
ce
ss
ing
Ca
rd
Pro
ce
ss
ing
Ca
rd
Inter-crate Φ-sharing LinksInter-crate
Corner-sharingLinks
HCAL/ECAL TPGs from oSLB Cards
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
TriggerPartial-
Products
η-sharingLinks
Output Links to GCT
Clock and Control
TTC/DAQ Connections
TriggerData to
DAQ
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 11
SLHC RCT Crate (1 of 6)SLHC RCT Crate (1 of 6)
Inp
ut C
ard
TT
C/D
AQ
Ca
rd
Pro
ce
ss
ing
Ca
rd
Inp
ut C
ard
Inp
ut C
ard
Pro
ce
ss
ing
Ca
rd
Inp
ut C
ard
Inp
ut C
ard
Inp
ut C
ard
Pro
ce
ss
ing
Ca
rd
Inp
ut C
ard
Output Links to GCT
HCAL/ECAL TPGs from oSLB Cards
Φ-sharing Linksto Input Cards In
other Crates
Corner-sharingLinks to Input
Cards inother Crates
Crate Output to DAQ
Clock/Control from TTC
(uTCA form factor)
Hu
b C
on
trolle
r
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 12
InputCard
2
Input Card Coverage/SharingInput Card Coverage/Sharing
InputCard
3
56 towers in η
72 t
ow
ers
in φ
InputCard
2
InputCard
1
InputCard
0
InputCard
4
InputCard
5
InputCard
6
12
to
we
rs/c
rate
InputCard
3
InputCard
2
InputCard
1
InputCard
0
InputCard
4
InputCard
5
InputCard
6
Cra
te N
InputCard
3
InputCard
2
InputCard
1
InputCard
0
InputCard
4
InputCard
5
InputCard
6
Cra
te N
+1
Cra
te N
-1
8 towers/card
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 13
SLHC RCT Input Card (12η × 8φ)SLHC RCT Input Card (12η × 8φ)
LinkSwitch
Microcontroller(Mezzanine)
FPGA
HF TPG (1)
Ethernet
ClockCircuitryφ-sharing (4)
Frontpanel(Serial Links)
Backplane
Fast Buffer SRAM
ECAL TPG (12)
Corner-sharing (4)
η-sharing (6)
To Proc. Card (~12)
To DAQ Card (1-2)
Clock and Ctrl
HCAL TPG (12)
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 14
SLHC RCT Processing Card(~20(~20ηη × 12 × 12φφ))
SLHC RCT Processing Card(~20(~20ηη × 12 × 12φφ))
LinkSwitch
Microcontroller(Mezzanine)
FPGA
Ethernet
ClockCircuitry
Fast Buffer SRAM
To DAQ Card (1-2)
Output to GCT
Frontpanel(Serial Links)
Backplane
Clock and Ctrl
From Input Cards
(~30)
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 15
SLHC RCT “TTC”/DAQ Card(Evolution of UW GCT Aux Card)
SLHC RCT “TTC”/DAQ Card(Evolution of UW GCT Aux Card)
DAQXmt
Interface
Microcontroller(Mezzanine)
FPGA
Ethernet
TTC Interface
Clock and ControlTo Input andProc. Cards
To DAQ
Frontpanel Backplane
DAQ Data from
Input and Proc.
Cards (10-20)
Clock/ControlFrom TTC
To TTS
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 16
SLHC RCT Key Points:SLHC RCT Key Points:• Double-width AMC-style (uTCA) modules (148.8mm height, 181.5mm deep)• 3 Card Types: Input Card, Processing Card, TTC/DAQ Card (evolution of UW
GCT aux card, possibly common card)• Input Card receives HCAL/ECAL TPGs, performs inter-region data sharing needed
by algorithms (7 cards/crate)• Processing card receives partial products from Input Cards, completes regional
processing, delivers output to GCT (~3 cards/crate)• TTC/DAQ interfaces crate to the TTC and DAQ systems (1/crate)
• Single Crate Encompasses Full 56-tower η Width• Card microcontroller implemented as a Mezzanine
• Can perform uTCA arbitration if needed• High-performance I/O interface to card for Trigger Supervision functions• Single hardware/firmware implementation
• Backplane contains combination of passive and switched interconnections• Passive good choice for η-sharing• Switches for some routing between Input and Processing cards
• TPG Inputs are fundamentally compatible with the envisioned upgrade path• Affects link and pipeline clock frequency choices
• Goal is to have a single firmware image for each card type, and to configure individual cards via RAM
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 17
SLHC Cal. Trig Prototype(R&D Platform for Conceptual Design)
SLHC Cal. Trig Prototype(R&D Platform for Conceptual Design)
LinkSwitch
Microcontroller(Mezzanine)
FPGA(e.g., Xilinx
XC5VTX240T)
Ethernet
Link ClockConditioning
CircuitryFast Buffer SRAM
GTX Links (6)
GTX Links (12)
Clock and Ctrl
SNAP 12 Transceiver
(6X/6R)
SNAP 12 Transceiver
(6X/6R)
SNAP 12 Transceiver
(6X/6R)
GTX Links (6)
GTX Links (6)
Tom Gorski, U. Wisconsin, July 20, 2009 SHLC RCT - 18
SLHC Cal. Trig PrototypeSLHC Cal. Trig Prototype
• R&D Hardware Platform• Prove layout/PCB fabrication concepts and principal
technologies for Input and Processing cards
• Circuit Prototypes:
• Microcontroller Mezzanine Concept
• Link/Pipeline Clock Conditioning
• External SRAM/FPGA Interface
• Double-size AMC module (149mm x 182mm)
• Suitable for trigger algorithm development
• Capable of supporting hardware/firmware/system R&D for multiple years