CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 1 LHCb Readout System and Real-Time Event Management - Emphasis.

CERN

R. Jacobsson

CERN

16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 1

LHCb Readout System and Real-Time Event

Management

- Emphasis on the architectural and functional aspects not the particular technologies- Commissioning and status

“An Integrated Control System for the LHCb Experiment”, CMS1-1, Thursday 14.00

“Handling Online Information in the LHCb Experiment”, CMSP-8, Tuesday 16.40

Federico Alessio, CERN, in place of Richard Jacobsson, CERN,

on behalf of the LHCb Online Team

CERN

R. Jacobsson

CERN

16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009

Place in the World

2

Underground cavern at ~100m depth

LHC

LHC

in 50 institutes in 16 countries

700 people

~10m

~20m

CERN

R. Jacobsson

CERN


Challenge of an hadronic precision experiment Particle multiplicity Very large background Small ratio of interesting B-meson decays O(10−3 –10−9)

High statistics Design luminosity L~2x1032 cm-2s-1 (1/50 Atlas&CMS) 10 MHz visible interactions 100 kHz bb-event rate ~ 1012 bb-pair / year at LHCb 2 kHz event storage rate

LHCb – Single-arm forward spectrometer

3

p-beamp-beam

Efficient trigger for many B decay topologiesMuon system, ECAL+Preshower , HCAL, Vertex Locator

Efficient particle identificationRICH

Good decay time resolutionVertex Locator 5 mm from beam

Good mass resolutionTracker and Magnet

B

1 cm

CERN

R. Jacobsson

CERN


Trigger Architecture

4

Hig

h-Le

vel T

rigge

r

2 kHz

Level -0

L0 e, g

40 MHz

1 MHz

L0 had

L0 m

ECALAlley

Had.Alley

Global reconstruction30 kHz

HLT

1H

LT2

MuonAlley

Inclusive selectionsm, m+track, mm

Exclusive selections

Storage: Event size ~35kB

Level-0 Hardware Trigger 40 MHz 1 MHz Search for high-pT m, e, g, hadron candidates Latency 4 ms, i.e. pipelining 160 events

High Level Trigger Farm with O(1000) quadcores HLT1: Confirm L0 candidate with more complete info,

and add impact parameter and lifetime cuts HLT2: global event reconstruction + selections Processing time available O(milliseconds) Output rate 2 kHz

HLT needs to know how L0 is configured How to distribute to 1000 nodes simultaneously in

seconds when optimizing parameters during LHC fill?

CERN

R. Jacobsson

CERN


Evolution: Eliminating trigger levels

Readout Architecture RT2003

5

CERN

R. Jacobsson

CERN

16th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 6

LHCb Readout System RT2009

SWITCH

High-Level Trigger farm

Detector

Timing & Fast

Control

SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH

READOUT NETWORK

LHC clock

Event Requests

Event building

Front-End

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Readout Board

VELO ST OT RICH ECal HCal Muon

SWITCH

Mon. farm

CPU

CPU

CPU

CPU

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

320 ROBs• 24 [email protected] Gb/s• 4 outputs@1 Gb/s

50 TB with 70 MB/s

3000 GbE ports35 GB/s

50 subfarms of ~40 nodes

Sh

ieldin

g w

all

5000 optical/analog linksO (4 Tb/s)

Offline See “Controlling a Large Trigger Farm Using Industrial Tools”, OPF2, Wednesday 11.00and “Management of the LHCb Readout Network”, OPF-3, Wednesday 11.00

CERN

R. Jacobsson

CERN


Design principle: A limited number of protocols and technologies

Readout network: GbE IPv4

Farm-to-storage: GbE TCP/IP

Overhead reduction <Event fragment> ~ 120 Bytes Overhead on IP/Eth is 58 Bytes

Pack fragments of several consecutive events = MultiEvent Packet (MEP) Protocol

Data Transfer Protocols

7

SWITCH


Detector

Timing & Fast

Control


READOUT NETWORK

LHC clock

Event Requests

Event building

Front-End

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Readout Board


SWITCH

Mon. farm

CPU

CPU

CPU

CPU

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

Offline

CERN

R. Jacobsson

CERN


SWITCH


Detector

Timing & Fast

Control


READOUT NETWORK

LHC clock

Event Requests

Event building

Front-End

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Readout Board


SWITCH

Mon. farm

CPU

CPU

CPU

CPU

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

Offline

Calib

Readout control has two aspects: Control of data transfer

• MEP Packing • Destination assignment for event building and HLT• Load balancing• Partitioning for parallel activities

Management of event types and associated destinations/processing• Physics triggers• Calibration triggers• Luminosity triggers• Non-zero suppressed data• Luminosity scans (Vernier scan)

Driven and managed by the LHCb Timing and Fast Control System Responsible for distributing timing, trigger and synchronous and asynchronous information to entire

readout system FPGA based master: Readout Supervisor

• Also performs rate control and generates all types of auto-triggers and calibration sequences

Centralized Readout Control

8

CERN

R. Jacobsson

CERN


TFC Master Info Flow

9

Readout Supervisor

LHC accelerator Beam Phase and Intensity Monitor

Subdetectors

HLT farm

L0 trigger

RS event bank

Multi Event Requests

Bunch currentsClock/orbit,

UTC, LHC Info

HW and run parameters

Run statistics

Detector status L0 Decision

RO ElectronicsTrigger Throttle

TFC

FE ElectronicsTFC

TFC

CERN

R. Jacobsson

CERN


Data Transfer ControlReadout network: Push protocol with passive pull mechanism

MEP Packing Control by Readout Supervisor:1. Trigger Type transmitted synchronously to all Readout Boards

Determines processing in ROB and synch check

2. IP Destination transmitted synchronously to all Readout Boards for each MEP Reception triggers closure and sending of last MEP Interleaving Trigger Type and Destination determines MEP packing Dynamic packing factor depending on event types Trigger type determines destination type (HLT, calibration, etc)

Farm Destination for next MEP chosen based on Credit scheme1. Farm nodes transmits Event Requests with a Credit(“declare as ready to receive”)

2. Readout Supervisor round-robin in Destination Table

3. Select destinations with positive Credit and decrement

Effectively, load balancing of readout network and HLT Farm Static load balancing of network by organization of Destination Tables in Readout Supervisor Dynamic load balancing of HLT farm nodes

Nodes request events when ready In all cases, event loss is minimized in case of failing/blocked/slow links or nodes

Ultimately, credit scheme regulate readout rate when low on credits

SWITCH


Detector

Timing & Fast

Control


READOUT NETWORK

LHC clock

Event Requests

Event building

Front-End

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Readout Board


SWITCH

Mon. farm

CPU

CPU

CPU

CPU

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

Offline

Calib

10

CERN

R. Jacobsson

CERN


Event Management Readout Supervisor Data Bank appended to each event

Book-keeping Run Number regroups events taken with same configuration UTC Time to correlate event with Conditions DB Coarse quality bits from each sub-detector which may be used by HLT Trigger Type and Calibration Type determine type of processing Window of consecutive 25ns clock cycles which should be processed together (see later)

HLT needs to know how L0 is configured How to distribute to 1000 nodes simultaneously? Trigger Configuration Key distributed in RS data bank allows optimizing the trigger parameters in

real-time during LHC fill

Event Type

Run Number#D0

Step Number#D1

Trigger Configuration Key#D9

Bunch IDTAE

WindowTrigger

TypeCalType

FBX

TypeBunch Current

#D8

Detector StatusError Bits#D7

UTC Time (31 .. 0)#D6

UTC Time (63 .. 32)#D5

L0 Event ID (31 .. 0)#D4

L0 Event ID (63 .. 32)#D3

Orbit ID#D2

Event Type

Run Number#D0

Step Number#D1

Trigger Configuration Key#D9

Bunch IDTAE

WindowTrigger

TypeCalType

FBX

TypeBunch Current

#D8

Detector StatusError Bits#D7

UTC Time (31 .. 0)#D6

UTC Time (63 .. 32)#D5

L0 Event ID (31 .. 0)#D4

L0 Event ID (63 .. 32)#D3

Orbit ID#D2

11

CERN

R. Jacobsson

CERN


Detector Performance, Readout Performance and Data Quality Histogram collected from all systems Monitoring Farm spying on event streams at best effort

• Also produces histograms from an online reconstruction at best effort Histogram analysis Automatic checks and alarms Histogram inspected by Data Manager Shifter

Online Monitoring

12

SWITCH


Detector

Timing & Fast

Control


READOUT NETWORK

LHC clock

Event Requests

Event building

Front-End

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

Readout Board


SWITCH

Mon. farm

CPU

CPU

CPU

CPU

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

Offline

Calib

Histogram Handling

(ECS)

AutomaticHistogram

Analysis

Interactive Presenter

CERN

R. Jacobsson

CERN


LHCb Data and Process Flow

13

Online 35 kB/evt@2kHz = 70 MB/s or 2 GB file(~60 kevts) / 30sOffline

Tie

r-0

/ 1

Tie

r-2

Storage(CASTOR)

Reconstruction

Simulation

Stripping

Analysis

HLT Bulk Stream Express Stream5 Hz

Storage(CASTOR)

Reconstruction

CalibrationAlignmentQ

C

Offline Control Room

20h/file, 20 kB/evt

Bookkeping

Data and Production ManagementData Quality Checking

Test Jobs

QC

Run Info

Dat

a Q

ual

ity

CERN

R. Jacobsson

CERN


Real-Time scheme to validate High Level Trigger, data flow and offline processing

= Be ready to receive, process and analyze 7 million events in the first hour of collisions

No-beam System Tests

14

SWITCH

HLT farm

TFC System


READOUT NETWORK

LHC clock

Event building

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

CPU

SWITCH

MON farm

CPU

CPU

CPU

CPU

Offline

LHCbMEP Requests

InjectorSimulated events

Detector

Front-End

Readout Board


Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

Readout Board

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

FEElectronics

L0 trigger

L0 Trigger

MEP Request

Replacing detector with injection of 108 “accepted” simulated events real-time in Online system at HLT rate (2 kHz)

See poster “High-speed Data Injection for Data Flow Verification in LHCb, CMSP-28, Tuesday 16.40

CERN

R. Jacobsson

CERN


Commissioning LHCb

15

• Two years of intense work 2006 – 2008 with the aim to:

Operate the detector AND people as a unit with common tools Bring all components (sub-detectors and service systems) to operational state. Define, implement and validate the tools and procedures needed to run the detector as a whole Organise the activities to reach the ready state in time

Understand and calibrate the detector Test pulses, radioactive sources Cosmics LHC injection tests First days with beam

Operate with two shifters Operating the whole detector from one console Understandable high-level tools for diagnostics, alarms and data monitoring Homogeneity in the system Shifter training On-call Experts for all sub-systems and sub-detectors

Reach operational efficiency Starting (<10min) and restarting (<1 min) rapidly and smoothly

Crucial tool: Readout and processing of sets of consecutive 25ns clock cycles around “detector activity” trigger

• Time and space alignment• Leakage in preceding and subsequent clock cycles• Optimize signal over spill-over

CERN

R. Jacobsson

CERN


Commissioning with Cosmics

16

Challenge: LHCb geometry is NOT well suited for cosmics… “Horizontal” cosmics well below a Hz Still 1.6x106 good events (July – September 2008 ) recorded for the large sub-detectors

CERN

R. Jacobsson

CERN


First Glimpse of LHC Protons

17

LHCb@LHC Sector Tests Aug-Sep 2008 Beam 2 dumped on injection line beam stopper (TED)

TED

TI8

LHC

Vertex Locator

Scintillator Pad Detector

Muon

CERN

R. Jacobsson

CERN


Experience with 150 Joules

18

RICH2 “photon blast”

• LHC Turn-On - September 10, 2008 An all too short honeymoon with LHC… Contrary to what we wish for the future, the splashes were Highly Desired Events!

CERN

R. Jacobsson

CERN


Readout Architecture RT2014

19

See talk “A 40 MHz Trigger-free Readout Architecture for LHCb”, RTSA2, Tuesday 18.20

Upgrade plans already well underway: Another level of Natural Selection

L0 trigger

L0 Trigger

CERN

R. Jacobsson

CERN


LHCb has become an operational experiment “waiting” for beam

Readout system mature with advanced readout control and event management Reached a good compromise between use of COTS and custom electronics

Commissioning Still (too…) many experts in the control room…necessary ones or not…

Injection tests end of August 2008 gave the first ever LHC-induced tracks

Beam-collimator shots were obviously the high-light of 2008, “unfortunately”… LHCb very ready for the long run with LHC COLLISIONS 2009-10

Upgrade path is towards full 40 MHz trigger-free readout Becoming popular concept, cmp future acelerators Good topic for next years Real Time Conference

Conclusions

20

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 1 LHCb Readout System and Real-Time Event Management - Emphasis.

Documents

lhcb experiment

lhcb readout system

event size

lhcb online teamcernr

b decays

beamefficient trigger

trigger architecture4

hardware trigger