ST + Tracking in 2008: What did we learn ? M. Needham EPFL ‘Had we succeeded well, We had been reckoned ‘mongst the wise: our minds Are so disposed to.

ST + Tracking in 2008: What did we learn ?M. Needham

EPFL

‘Had we succeeded well,

We had been reckoned ‘mongst the wise: our minds

Are so disposed to judge from the event’ Euripides

Outline

IT in 2008

TED running

• Studies performed• Lessons

• Studies performed• To do• Lessons

Cosmics

General Lessons

• The detector• Lessons learned • The team

• Hopes for the future

ST group extremely active on softwareMany generic tools being developed that are being used by wider LHCb communityEvidenced by large number of commitsof fixes + improvements to cvs See also ST analysis meetings in December2008 + February 2009

TT in 2008

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

TT Status19 41 7 46 33 10 42 101 133 132 108 25 44 32 48 52 51 11 full module

15 14 1315 14 1315 14 1315 14 13

1 30 3 24 31 40 23 104 104 106 103 8 34 38 5 13 29 45 full module

10 9 810 9 810 9 810 9 8

36 14 12 20 21 22 107 103 134 105 35 37 39 28 27 26 full module

6 5 46 5 46 5 46 5 4

2 49 15 47 4 54 102 146 121 106 16 6 43 9 53 17 full module

3 2 13 2 13 2 13 2 1

# Sectors Percentage0 0.00 Not yet installed

0 0.00 Installed but not yet commissioned

1 0.36 Installed, but HV problems

7 2.50 Installed, broken bonds

0 0.00 Installed, optical path or Tell1 not ok

1 0.36 Installed, but problem with the SB.

271 96.79 Installed and commissioned

280 100

1 17 17 1733 33 3 233 33 33 33

18 18 18 1834 34 34 18

19 19 19 1935 35 35 19TTa x

35 35 35

34 34 34

20 20 20 Tell1 number

19

18

17 17 17

1 20 20 2036 36 3 236 36 36 36

4 21 21 2137 37 6 537 37 37 37

22 22 22 2238 38 38 22

23 23 23 2339 39 39 23TTa u

39 39 39

38 38 38

24 24 24 Tell1 number

23

22

21 21 21

4 24 24 24

25 25

40 40 40 40 40 40 6 5

25 25 25 2510 9 8 7

26 26 26

41 41 41 41 41 41 11

7 26 26 26

27 27 27

42 42 42 42 42 42 11

7 27 27 27

28 Tell1 number

TTb v43 43 43 43 43 43 11

28 28 28 289 8 7 28

29 29

44 44 44 44 44 44 11 10

29 29 29 2915 14 13 12

30 30 30

45 45 45 45 45 45 16

12 30 30 30

31 31 31

46 46 46 46 46 46 16

12 31 31 31

32 Tell1 number

TTb x47 47 47 47 47 47 16

32 32 32 3214 13 12 32

A-Side C-Side

48 48 48 48 48 48 16 15

Broken bonds

Broken bonds: glued

Broken bonds: modules removed

Dig board show problems recently.

HV problemFM36

0

50100

150

200

250300

350

400450

500

0 50 100 150 200

Voltage (V)

FM36

97 % of the detector workingLast week start of repair work on low power optical links71 VXCELs beings exchangedStart of run aim to have 98 - 99 % working detector

J. Van Tilburg

TT Timing from TEDFibre length (m)

Add. fibre delay (ns)

Add. cable delay (ns)

Time of flight IT-TT (ns)

Total delay wrt IT (ns)

IT 18 - - - -

TT AB 19 -4.9 -20 +19.9 -5.0

TT AT 21 -14.8 -20 +19.9 -14.9

TT CB 34 -79.0 -20 +19.9 -79.1

TT CT 36 -88.9 -20 +19.9 -89.0

Calculated timing for backward events using fibre length:

… agrees nicely with timing from Landau MPV fit

BTW, getting the signs right in this table is not so easy.

TT alignment with TEDExtrapolated Velo tracks to TT → VeloTT tracks

• Alignment in Tx for the full station.• Unbiased residuals ~ 500 micron.

→ extrapolated from Velo

Survey Database• Module position → from survey in pit.• Sensor position → from metrology in lab.

Main effect comes from survey of modules

Next: align also in Ry (long lever arm).

C. Salzmann

TT alignment with TEDA-C side asymmetry about ~62%

Christophe had a look at the possible causes:• Timing A-C side

• Clear effect when summing tracks in different spills.• Still to do: merge spills in TT to avoid double counting.

• Data quality• Clear effect when excluding runs with error banks• Still to do: recover partial banks.

• Geometrical acceptance• Clear effect when seen when taking beam pipe into account.• Use hit expectation tool to calculate efficiency (works)

Did not look at pattern recognition, but probably above effects are enough to explain asymmetry.

Rough estimate of effects:Timing: 62% → 41%Error banks: 41% → 26%Acceptance: 26% → 22%

C. Salzmann

IT Status IT Status now, few more problems developed since end of run

2.5 % dead 2 % noisy

Known this small lost leads to 1 % loss in tracking efficiency

Fixes for 2009 • Backplane problems + weakest optical links will be replaced• Precise # still to be worked out, likely to be ~ 83

• We start with the worst service boxes and see how it goes• Always a risk that in attempting to repair we cause more damage

• N.B. repair criteria is a matter of judgement of time/effort/parts versus time• ~ 100 more links in TT , ~ 100 more links in IT could be candidates for future repair

• During run careful monitoring to check if more link problems develop• In parallel new Tell1 firmware developed

• Can recover from loss of synchronization caused by ‘link’ loss• Data for that event on effected link is still lost• Error bank is sent and recoverary mode needed

IT Cosmic Running

21st August ? OT closed

5 th September

19 th - 21st September

July- Maybe 500 k events ?but IT/OT opened, alignment changed….around 18th August(according to emails, hints in logbook). Older firmware etc, Not considered

2000 k good IT events at least !Care: OT not present in all runsRuns with RICH TAE settings removedRuns with only Central spillTAE window changes run to runRuns with calibration events in data stream Not all data copied to castor

400 k good IT events OT in for all runsOnly cosmic runs good for TT1 Tell1 board in IT1A missingSix bad Beetles not masked [rec mode needed]

19/20th ~ 180k events21/21st ~ 200 k eventsrun 32406 50 k events 5th Septemberrun 32385 120 k events 5th Septemberrun 32197 2 k events 4th September (no events)32181 39k events 5th Septemberrun 31959 5k events 1st Septrun 31732 50 k events 30th August 2 bxrun 31731 30 k events 30 August 2 bxrun 31729 51 events 2 bx run 31720 37k events 2 bxrun 31715 8 k events 2 bxrun 31714 20 k events 30th Augustrun 31557 180 k events 29th Augustrun 31026 20 k events 26th Augustrun 30664 40k events 21st Augustrun 30662 20k 21st Augustrun 30654 90k 21st Augustrun 30653 8k 21st Augustrun 30648 23k 21st Augustrun 30636 11k 21st run 30639 13k 21st run 30640 1.3k 21strun 30641 15k 21strun 30645 6 k 21st run 30659 7k 22ndrun 30660 11k 22nd run 30661 17k 22ndrun 30666 41k 22nd run 30669 22k 22ndrun 31300 50k 27thrun 31537 5k events

IT Cosmic dataset run 32432 10 k events 5th Septrun 31542 200 events 29thrun 31723 76k event, 30thrun 31716 1k events, 30thrun 31719 19 k event 30thrun 31725 10k event 30thrun 31727 10 k events 30thrun 31626 60 events 30th run 31733 10k events 30th run 31734 37k events 30thrun 31736 4k events 30thrun 31737 62k events 30th run 31739 32k events 30thrun 31744 59k events 30thrun 31747 9k events 30thrun 31760 1.5 kevents 30thrun 31761 0.2k events 30thrun 31789 [no OT] 21 krun 31791 0.2 k 31th run 31792 83 k 31thrun 31793 11k 31thrun 31794 [calib] 85k 31thrun 31795 40 events 31st run 31797 120 k event 31strun 31817 10k events 31st run 31821 77k events 31st run 31838 31k 31st run 31841 85 k 1st Septrun 31842 1k 1st Sept

run 31843 14k 1st Septrun 31844 8k 1st Septrun 31845 1.8k 1st Septrun 31846 66 k 1st Septrun 31848 5k 1st Septrun 31849 24k 1st Septemberrun 31856 40 k 1st Septemberrun 31860 0.1k 1st September run 31874 150 k 1st Septemberrun 32381 15 k events th September

• 2.55 million events in 15 days at 10 Hz• Could have collected 13 million• Few days not running on cosmics• IT missing/giving bad data [10 %]• Room for improvement !• Filter to set of 27 k IT events• Filtering was a lot of time and effort• Really need to understand the logbook • Changing run conditions• Calibration data in the datastream• OT opening/closing around August 22nd

Cosmic Results

70 candidates, Calo confirmed

through two boxes

(300 MeV 'energy’)

Mainly on A or C side due to geometry

2 pass through TT

ty

Unbiasedresidual

Cosmic Alignment Align for Tx: Fix X1U in T1, T2 X2V in T3Require 10 hits in layers being aligned

ideal~390 m

survey~270 mmore entries

Cosmic Lessons • Many cosmic tools developed: filtering, spill merging, ITGenericTracking

• Important to have long runs with stable conditions

• Fighting the noise is important. Aim to reduce this year

• Good time alignment: reduce spills needed to 3

• Remove hotspots (working header correction ?)

• Important to have all detectors working to confirm tracks

• plots of track parameters important checks of tracking quality

• 2 - 3 weeks continuous cosmics running, stable detector collect 10 million events

• Working OT + Calo for confirmation

• Working book-keeping + Data Quality

• HLT alley or offline express stream to reduce to ~ 10,0000 or less

• Full Cosmic detector reconstruction CRAB

To collect ~ 500 IT cosmics + perform cosmic alignment of IT next year, Need:

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

In the ST mini-review management urged us to concentrate on TED [~August]. Still possibility of Detector Open TED in June. No effort on ST cosmic alignment from now on

TED: First Steps

First alignment in x using histograms Stereo is messy, ladders not surveyedVerified correctness of surveyPosition ladders + boxes to accuracy 20 micronsWork to appear soon in LHCb note

TED: First Steps Width tells us about ladder misalignments

[If we know the momentum]If dominated by primaries from TED p ~ 20 GeV

TED-like MC particle gun, 500 muons/eventVary p and apply same procedure as data

Either:

p ~ 5 GeV and no ladder misalignments

p > 20 GeV + 100 micron ladder Misalignments

Or something in between

Tracking Understanding • Looked to be a problem in T/B box survey

• In fact tracking problem• Two competing conventions for x in x search

• x = x(yMid) [LHCb 2007-037]• x = x(y = 0) [LHCb 2008-042]

y = 0 cm

y box

Survey indicates ladder rotations ~ 1mrad• Reverting to xMid convention better results• x=x(y=0) relies on incorrect assumption• Problem still exists in standard tracking code

Surveyed, T2, core 200 microns

Default, T2 core 130 microns

TED Tracking Assumptions:• x ladder position known to ~ 150 micron• y ladders worse [no survey], guess ~ 250 micron

Search windows:• x +/- 0.6 mm• y +/- 10 mm [4 0.25 sin 5o ]

Delayed decision:• Follow all valid candidates to the end [ie full Kalman fit]• Many incompatible candidates, choice needed• Winner takes all algorithm [TrackCompetition]• Rank tracks by # hits, then fit c2

Ensure uniqueness:• Many outliers in search window from other tracks• Many incompatible candidates, choice needed• Two hits from a sector in window, split candidate in two

Some tracks come from the TED:• tx < 0.02 mrad, ty < 0.05 mrad• - 3 < x(TED) < 7 m, |y(TED)| < 5 m

TED Tracking

C side A side

Unbiased residuals

Convincing for Top/BottomSide boxes: longer laddersWorse combinatorics ?More suspectible to misalignment ?

TED MC

20 GeV muonsTED occupancies: T/B boxes Ghost rate ~ 0.8 %L/R boxes Ghost rate ~ 6.2 %

TED occupancies: T/B boxes efficiency ~ 98 %L/R boxes efficiency ~ 81 %

Matching to TT

Top-IT TTb

Bottom-IT TTa

MC

Clear signals seen

Can be used for understanding

alignment

Imply p in the 10 - 20 GeV range

TED Alignment L. Nicolas

Survey +first alignment in x

y box alignment

Stereo layers worstAligned ?

TED Alignment L. Nicolas

TT validation TTaU, Top IT box

Align

TED Alignment L. Nicolas

In fact bugs in my hand-made TED alignment, where I had moved stereoLayer in pair wrongly with respect to xWhen this is corrected both methods agreelevel 20 microns

Still some subtle weak modes ?Things improve, e.g x ladder residuals~ 150 to ~ 100 microns

Side boxes no convincing results obtained

To Do • Speed improvements in generic tracking

• Documentation: alignment note, ITGenericTracking tracking note

• Alignment studies:

• ~ finished

• to be documented in Louis thesis

• A/C-side sensor survey: will be added by Geraldine

• Probably small effect, but every little helps

• Still few tools/studies to be done:

• Matching to Velo

• Use TT as confirmation device

• Not done yet due to manpower constraints

Lessons Care needed with assumptions in Pattern Recognition• Projecting to xMid assumes that ladder rotations small + they are not• Problem will never be fixed by better alignment • Finding this problem took a lot of time

Hotspots + memory management• High occupancies necessary to make full fit of all valid candidates • High densities: memory management the issue

• Only have ~100 events, so spending processing 10 minutes on hot event is ok, and I could fix this anyway• Some events ~ 100 - 200 k tracks made• No obvious hot spots [everything is hot…]• Less a problem in MC….

• Solution: only use memory really need• Kill obvious clones early. Use TrackCompetition algorithm • Standard clone killer tried but nested double loop means slow• Track competition: two single loops, fast, generic

Lessons Monte Carlo useful• Code developed + debugged without Monte Carlo• Proved very useful for understanding to have Monte Carlo samples• A lot of work to generate [tune particle gun, run Gauss, tune Boole,…]

• Easy to do this for ST • Other detectors: need to understand simulation/timing etc

Go generic…• Code looks for straight lines + is totally generic• Developed for TED, tested on cosmic data, tested on beam gas MC• Will work for beam gas, magnet off, collisions @ ~ 10 m • Easy to tune for different conditions with few options

Can go along way with simple residual plots• But time consuming, a lot of work by hand• Takes ~ day or two to make layer/box alignment • A lot of room for error: editing XML by hand +need XML not SQL

• …XML is not so nice if trying to combine with Velo• Try to automize ?

General Lessons • Real data is a lot tougher than Monte Carlo

• Its easy to get bogged down trying to understand problems

• That makes it hard to work on several things in parallel

• Be well prepared [a lot of work preparing ST software for 2009]

• Fixes have to be made quickly

• Issue of projecting to y=0 reported in December, but not yet fixed

• Find it very suprising will take 1 month for other subdetectors to deal with delta-rays

• For track confirmation/quality studies need inter-detector interaction

• Make sure technical problems solved quickly

• Make sure improvements fed in quickly

• Tools still missing to measure the detector/tracking efficiency + resolutions

ST Development • Have a lot of generic tools developed this year

• STEventMerge, ITGenericTracking, STCosmicFilter

• Shutdown work: Make code development easier

• New base class, helper functions/short-cut, print functions, monitoring

• Cluster selectors, clusters to lite clusters , ….

• Optimize for ease of use, avoid hacking ‘speed’

• Make it easy to tune MC

• Cluster killers in Brunel, changes to turn in delta-rays in Gauss

• Documentation: Updated and improved

• Aim: make code development faster and simple, ready for everything

• Always aim to develop generic code that can be used elsewhere

Higher level tools: efficiency calculation, resolution model, event merging • Discussed in tracking meetings since 2007, no progress on common tools• ST group will develop own tools building on the flexible framework we have developed over last months

Summary • The tracking must be prepared to react quickly to an imperfect detector

• Now is the time to ensure we have the tools to understand the detector

• Flexibility is crucial: react to the detector as it is, not as you would like it to be. Don’t make assume anything. Being fast but locked into one set of assumptions is a recipe for disaster.

• 2008 the disaster that never was, since we didn’t run:

• Understanding link loss problems/developing new firmware took ~ 1 month

• Understanding desychronization problems related to TTC fibers took time

• Always would be new problems [new bad links, broken bonds, failing Tell1s …]

• Debugging would require access/removing IT from data taking

• Hard to imagine how detector eff would be more than 90 %

• Fold in imperfect alignment, imperfect Velo, IT, …..

• Tracking must be able to respond quickly to possible problems during long run

• We hope no more links develop optical power problems, but it could happen

• We hope TT broken bond problem will not develop further

• Once we close the detector, no more hardware fixes possible

Backup

ITCosmic dataset

Second round of stripping:

• Better understanding of when OT closed on the 21st August

• Better understanding of the logbook, recover many runs around 31700-31800

• Recover many of the runs with calibration events mixed in the data

• Take all runs, even those with less than 10k eventsFound another 1.45 million events, now have 2.55 in total

• 70 - 80 tracks depending on the cuts

• Took 3 days of studying logbook, finding data, understanding…

• The largest reprocessing of real data performed so far on LHCb ?

• N.B, OT [A or C] not available in many of these runs

1st round of stripping, present in T-Rec October 15th 1.1 million events:

IT Cosmic dataset 2 tracks going through 3 IT boxes

~70 tracks going through 2 IT box

~1000 tracks going through 1 IT box

Too few, to do anything with apart from making nice plots

Any two boxes, mainly A or C sideTried to use for layer alignmentStatistics too poor, TED better

Not looked in detail OT geometry problems, lack of manpowerLook at residuals try to align IT to OT

IT Cosmic Alley Count clusters with > 12 ADC Reject

n = 0, n > 1000

> 50 clusters > 7 clusters 3 in one box + spill

> 3 clusters (15 ADC)

In 1 spill +/- 3 mm window

Accept Reduce 2.6 million eventsto ~27, 000

X search: Box Alignment Using survey [box residuals]

~ 0.13 mm ~ 0.23 mm

~ 0.27 mm ~ 0.43 mm

MC Expectation Running ITGeneric tracking on DC06 beam gas MC…

MC find 0.08 mm [1.4 * binary resolution]If survey accurate to 50 micron expect 0.09 mm100 micron expect 0.13 mmHmm…we are far from understanding IT alignment

X search (return) I only move the boxes in T2, not the layers, lets try that…

C-side, move 0.02, 0.03 mm ~ 0.13 --> 0.125 mm

A-side, move 0.14, 0.03 mm ~ 0.23 --> 0.16 mm

Bottom, move 0.14, 0.9 mm ~ 0.27 --> 0.25 mm

Top, move 0.1, 0.3 mm ~ 0.43 --> 0.33 mm

ITT2TopX1, ITT2BottomX2 are the ‘bad’ guys

Data available Going through the logbook is like an episode of CSI

Some things not clear in logbook or not in logbook

• Pages of ‘conditions same as the last run…’• Which detectors are in the run ?• What are the conditions ? TAE ? How many spills ? RICH TAE ?• What is the run number ? When did it start/stop ?

• I know (from email) that OT opened/closed ~18th - 21st August• Hard to know this from the logbook

• Calibration data in the physics data at start of September• I know this, reported on mailing lists at time, …

• Where is the data ? Copied to castor or not ?• Whats the directory ?• How does the non-expert know all this ?

IT Cosmic dataset

10 k events, in POOL_ROOT format

Merging spills Timing changes run to run

• Forward + backward cosmics have different timings

Solution: merge spills using the algorithm: STEventMerge

• In case two clusters have same key: take one with highest charge

• In case of overlapping clusters: take one with highest charge

• STCluster class extended to have information on spill

Typically ~ 3-4 noise clusters/spill

• Merging means we have ~ 15- 20 clusters in five spills

• 5 spills are needed: lose ~ 10 % tracks if take only 3 spills

The Golden Track

Very clean event, track with 10 hits

Have one hit just fail to include on track…..

Run 3172030th August

A-side All clusters > 10 ADC shown

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.

TED MC

Unbiased residual Biased residual

IT Overlaps

A-side with Bottom

C-side with Top

Look for overlaps between boxes

Take Top/Bottom tracksResiduals to Side boxesEarly on no signal seenWith final survey geometry overlaps !Bad combinatoricsConvincing signal for A-side/Bottom