- LETEM - HOSTLAB Phase 1 - HOSTLAB Phase 2 · EA coffee meeeting April 2nd, 2020 3 LETEM = LHC Experiments Technical and Engineering Meeting LETEM Committee (… should perhaps be

- LETEM

EA coffee meeting 2/4/2020Fabio Formenti

- HOSTLAB Phase 2- HOSTLAB Phase 1

EA coffee meeeting April 2nd, 2020 2

(†) ed.: LETEM and HOSTLAB look at first view similar but do not have precisely the same scope

Content LETEM (†)

• Overview and basic definitions• Budget line• “HOSTLAB support domain”, supported activities • Budget management• 2019 investments for EA

HOSTLAB (†)

• The scope• HOSTLAB Phase 1

Funded activitiesBudget structure, spending profile, follow-up2019 EA manpower by HOSTLAB

• HOSTLAB Phase 2Main programmesProject development historyBudget structure, spending profile


LETEM = LHC Experiments Technical and Engineering Meeting

LETEM Committee (… should perhaps be LETEC? … below the best definition I can write …)1.1 Steering committee aimed at assuring the long-term operation and upgrades of the LHC experiments,

1.2 by providing resource coordination support (planning, funding, manpower) among CERN Tech. Serv. and LHC Exp.,

1.3 for all needed service activities identified as part of the domain “Hostlab support”.

Committee compositionChair: EN DH, co-Chair: LHCb TC (as representative of EP Department management)Standing Members: EN/EP DDHs, TCs four LHC experiments (or respective deputies), Hostlab PLInvited according to agenda: EN/EP Technical Support Units, Safety, EN/EP DPOs, FAP

Meeting frequency: monthly/bi-monthly depending on criticality of on-going programmes

Budget of the LETEM Project (root allocation)Project type (negative and positive carryover)Resources are funded following approval of Directorate and as result of MTP exercise

LETEM overview, definition and composition


Material budget part

M available for:

Material budget shared among 4 LHC Exp and EN-EA (aim is balanced 20% share each) Expenses should comprise “regular” needs for goods/services in the “HOSTLAB support domain” Excluded activities already funded by SLAs (Service Level Agreements)

Fix yearly released budget

As of 2017, until …… deadline related to end of LHC Exp. physics (LS3 and LS4 at least OK)

Covers material/services/manpower provided to operate, maintain and assure safety of running LHC Exp. facilities

Includes EA services and related infrastructure upgrade in scope for LHC Exp. support

Manpower budget part

M->P consist of Appl-Phy/Eng, Tech

4 FTEs in EP + 4 FTEs in EN (2017 to 2025)

- EP : 1 FTE each LHC Exp.- EN : 2 FTEs for EA + 1 FTE for CV + 1 FTE under discussion

LETEM budget line


Definition of “HOSTLAB support domain” is derived from two main CERN official documents: “General conditions of Experiments at CERN” “Memorandum of Understanding” of Experiments

Boundaries are summarized in the HOSTLAB document “Analysis of CERN support requirements for …” and visualized by the picture below

“HOSTLAB support domain”, supported activities

† being fully moved to SLA

†

‡

‡ including safety coordination


Requesting year engagement plans at beginning of year (+ apply prioritization criteria) Monitoring orders and CET transactions (monthly/bi-monthly) Providing budget statistics (beg. yr. budget release - mid period status - end yr. closure)

Sharing 2019 closureStatistics “Order Type” 2019

50%40%

OK

OK

+10%

-7%

-5%

Budget management: maintain best resource balance among Users

Personal tool: Excel worksheet with CET extraction and filled with complementary order information


For Cabling Team- Hi Pot Tester fabrication (paid across 2018 and 2019)- Vector Network Analyzer and calibration accessories- Digital microscope, with lenses and base accessories- Pull-out tester for crimped connectors- Oven for ageing tests- Diamond wire saw- Electrical screwdriver adjustable torque- Torque wrench set

For Integration Team- Handled 3D scanner- Workstations and screens

For Mechanical Team- TIG Welding Station

Main items for EA invested through LETEM in 2019

2021 2024 2025 2026 2027LHCb & ALICE:major upgrades

ATLAS & CMS upgrades:Finish Ph1 Start Ph2

ATLAS & CMS upgrades:Finish Ph2

LHCb upgrades:Detector consolidation & enhancement

LHC

restart

EYETS 23/24


HOSTLAB project, the scopeOrigin: from initial request of LHC Exp. at Chamonix 2017 session

Context: in perspective of the important LHC Exp. detector upgrades already planned for: LS2 -> ALICE & LHCb major upgrades, ATLAS & CMS minor upgrades for end Phase1 + start Phase2 LS3 -> ATLAS & CMS major upgrades for end Phase2 LS4, LS5, … -> not yet planned (? e.g. ALICE proposal for LS4 new full detector under study ?)

Aim: assure resources & support from the Technical Sector to LHC Exp. upgrades according to “HOSTLAB domain”(cf. “… that were considerably reduced after successful installation of detector systems, RUN1 and first part of RUN2”)

(Experiments need to get prepared for EYETS and LSs)

Chamonix 2017


Period 2018 … 2022 (-> for LS2)Total approved budget 18.5 MCHF

(25 M originally requested)

PROJECTS ALICE Data Centre (C.E., containers, primaries, access, safety) LHCb Data Centre (Modules, C.E., containers, primaries, access, safety) ATLAS SR1 building extension and laboratory B180 refurbishment CMS extension SXA5 building (~ 20% participation to C.E.)

SYSTEM UPGRADES EL primary connections (all Exp.) CV primary connections (all Exp.) Data fibres (ALICE) Mod. underground infrastructure for detector services (CMS) Temporary storage (LHCb)

SERVICES and LOGISTICS (manpower) Integration, models and data base (all Exp.) Survey (all Exp.) Safety Coordination (all Exp.) Transport and crane driver (ALICE & LHCb) Radiation Protection (ALICE & LHCb)

HOSTLAB Phase 1 funded activities (target LS2)


For Run 3 and Run 4, Alice and LHCb aim at getting more statistics, more data collection, more interesting event for new physics

Need sustain 40 MHz data taking up to PC farm and increased data volumes provided by the upgrade of the central detectors

Most cost effective solution given by new Data centres, comprising: containers, adiabatic cooling on roof (90 % efficiency, closed circuit, separated temperature zones inside container), up to ~20 racks/container with horizontal cooling

6 containers for LHCb (2 for DAQ processing farm with home made readout cards and 4 for high-level trigger farm)

4 containers for Alice (new DAQ processing farm with LHCb home made readout cards)

~2 MW power at full load of all containers

10ths of thousands fibres carrying some 40 Tb/s data rate to record more than 20 PB(†) on data storage

Two LHCb containers are today lent to IT for their data centre, waiting for the new IT data centre in Prevessin to be completed

ALICE and LHCb new data centres

New data centre installation at P8 site

(†) 1 PB = 1’000 TB = 1’000’000 GB


ATLAS and CMS civil engineering

ATLASExtension of SR1 bldg. (~290 m2) to install larger clean room (paid by ATLAS) for assembling and testing ITK pixel detector

Ceiling reinforcement of existing clean room

CMS Extension SXA5 building for detector assembly, workshop utilities and temporary storage

Trk det.(new layers, DC-DC)

Fwd det.(timing resolution, coverage)

Beam pipe

H.Cal.(electronics)

Mu Det.(new technology, n-bkgnd shield)Pixels det.

Clean room extension (new)

SR1 bldg.

prototest

mech

stor

stor stor

electr

ALICE UPGRADE SERVICES AND SUPPORT & LOGISTICS (95421) ALICE DATA CENTRE (95422)

Jan

-18

Fe

b-1

8M

ar-

18

Ap

r-1

8

Ma

y-1

8

Jun

-18

Jul-

18

Au

g-1

8

Se

p-1

8

Oct

-18

No

v-1

8

De

c-1

8

Jan

-19

Fe

b-1

9M

ar-

19

Ap

r-1

9

Ma

y-1

9

Jun

-19

Jul-

19

Au

g-1

9

Se

p-1

9

Oct

-19

No

v-1

9

De

c-1

9

Jan

-20

Fe

b-2

0M

ar-

20

Ap

r-2

0

Ma

y-2

0

Jun

-20

Jul-

20

Au

g-2

0

Se

p-2

0

Oct

-20

No

v-2

0

De

c-2

0

Jan

-21

Fe

b-2

1M

ar-

21

Ap

r-2

1

Ma

y-2

1

Jun

-21

Jul-

21

Au

g-2

1

Se

p-2

1

Oct

-21

No

v-2

1

De

c-2

1

Jan

-22

Fe

b-2

2M

ar-

22

Ap

r-2

2

Ma

y-2

2

Jun

-22

Jul-

22

Au

g-2

2

Se

p-2

2

Oct

-22

No

v-2

2

De

c-2

2

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

Jan

-18

Fe

b-1

8M

ar-

18

Ap

r-1

8

Ma

y-1

8

Jun

-18

Jul-

18

Au

g-1

8

Se

p-1

8

Oct

-18

No

v-1

8

De

c-1

8

Jan

-19

Fe

b-1

9M

ar-

19

Ap

r-1

9

Ma

y-1

9

Jun

-19

Jul-

19

Au

g-1

9

Se

p-1

9

Oct

-19

No

v-1

9

De

c-1

9

Jan

-20

Fe

b-2

0M

ar-

20

Ap

r-2

0

Ma

y-2

0

Jun

-20

Jul-

20

Au

g-2

0

Se

p-2

0

Oct

-20

No

v-2

0

De

c-2

0

Jan

-21

Fe

b-2

1M

ar-

21

Ap

r-2

1

Ma

y-2

1

Jun

-21

Jul-

21

Au

g-2

1

Se

p-2

1

Oct

-21

No

v-2

1

De

c-2

1

Jan

-22

Fe

b-2

2M

ar-

22

Ap

r-2

2

Ma

y-2

2

Jun

-22

Jul-

22

Au

g-2

2

Se

p-2

2

Oct

-22

No

v-2

2

De

c-2

2

CHL budget profile Total engagements %

Jan

-18

Fe

b-1

8M

ar-

18

Ap

r-1

8

Ma

y-1

8

Jun

-18

Jul-

18

Au

g-1

8

Se

p-1

8

Oct

-18

No

v-1

8

De

c-1

8

Jan

-19

Fe

b-1

9M

ar-

19

Ap

r-1

9

Ma

y-1

9

Jun

-19

Jul-

19

Au

g-1

9

Se

p-1

9

Oct

-19

No

v-1

9

De

c-1

9

Jan

-20

Fe

b-2

0M

ar-

20

Ap

r-2

0

Ma

y-2

0

Jun

-20

Jul-

20

Au

g-2

0

Se

p-2

0

Oct

-20

No

v-2

0

De

c-2

0

Jan

-21

Fe

b-2

1M

ar-

21

Ap

r-2

1

Ma

y-2

1

Jun

-21

Jul-

21

Au

g-2

1

Se

p-2

1

Oct

-21

No

v-2

1

De

c-2

1

Jan

-22

Fe

b-2

2M

ar-

22

Ap

r-2

2

Ma

y-2

2

Jun

-22

Jul-

22

Au

g-2

2

Se

p-2

2

Oct

-22

No

v-2

2

De

c-2

2

0%

20%

40%

60%

80%

100%

0%

20%

40%

60%

80%

100%

Jan

-18

Fe

b-1

8M

ar-

18

Ap

r-1

8

Ma

y-1

8

Jun

-18

Jul-

18

Au

g-1

8

Se

p-1

8

Oct

-18

No

v-1

8

De

c-1

8

Jan

-19

Fe

b-1

9M

ar-

19

Ap

r-1

9

Ma

y-1

9

Jun

-19

Jul-

19

Au

g-1

9

Se

p-1

9

Oct

-19

No

v-1

9

De

c-1

9

Jan

-20

Fe

b-2

0M

ar-

20

Ap

r-2

0

Ma

y-2

0

Jun

-20

Jul-

20

Au

g-2

0

Se

p-2

0

Oct

-20

No

v-2

0

De

c-2

0

Jan

-21

Fe

b-2

1M

ar-

21

Ap

r-2

1

Ma

y-2

1

Jun

-21

Jul-

21

Au

g-2

1

Se

p-2

1

Oct

-21

No

v-2

1

De

c-2

1

Jan

-22

Fe

b-2

2M

ar-

22

Ap

r-2

2

Ma

y-2

2

Jun

-22

Jul-

22

Au

g-2

2

Se

p-2

2

Oct

-22

No

v-2

2

De

c-2

2

CHL budget profile Total engagements %

Spending profiles (cumulated %)

(Defined with Tech. Coordinators)


Budget structure, spending profile and follow-up(some key issues for a healthy project management: clear BS, reliable SP, rigorous FU)

9 HOSTLAB Ph1 budget codes.- 1 Common Project fund (shared by all Exp.)- 4 for Upgrades, Services and Logistics (1 per Exp.)- 4 for Projects (1 per Exp.)

Budget follow-up (ALICE Example)


EA manpower by HOSTLAB in 2019

AS Section

1.5 FTEs Wiring Technicians

0.25 FTE Fellow

DC Section

1.3 FTEs FSU

2 FTEs Technicians

PE Section

0.5 FTE Technician


1) ProjectsLarge LS3 activities involving complex tasks on multiple technical fields

2) Upgrades & ConsolidationsSpecific technical activities typically mono-domain

3) Technical Services and LogisticsSupport from service groups essential for two programmes above

4) Safety and Radiation ProtectionSafety of personnel, protection of equipment and environment

HOSTLAB Phase 2 main programmes (target LS3)

ATLAS & CMS- CO2 cooling system and CO2 facility building- Dry gas and Greenhouse gas recuperation system

ATLAS- New cooling system at P1- New electrical distribution at P1

CMS- P5 civil engineering (Control Room and Engineering Centre)- TDAQ upgrade at P5

All Experiments- Local primary services (EL, CV)- HVAC systems- Cranes and lifting equipment- Civil engineering structures- Beam pipes

As in Phase1, extended (all Experiments)

New in Phase 2 (all Experiments)

IntegrationSurveyTransportCablingScaffolding


ATLAS and CMS CO2 cooling systems and dry air

Needed for inner tracking detector cooling at low temperatureLow material budgetDistance from radiation environmentEnvironmental friendly gas

Working principle

2PACLs

Detector cooling power:ATLAS = 300 kWCMS = 600 kW

Existing CMS membrane dry gas system

No frost at CO2 cooled detectors

7-14 chiller units

2PACLs


ATLAS and CMS CO2 buildings

PX15

~60 m

P1

~75 m

PM54

P5

CO2 building

CO2 building

New IPP requirements:- 2 storey- 535 m2 total- CO2- TDAQ & HVAC- Large UPS & Batteries

New IPP requirements:- 2 storey- 1340 m2 total- CO2- Large UPS & Batteries- Diesel generator- 2 dry gas systems- CV technical room

30 MW-2500 m3/h


P1 cooling upgrade

Adding ~70 tall 63U electronics racks for DAQ and HLT

New technology based on ATCA (~20 kW full load/rack)

Required additional ~2 MW on mixed water

Required additional ~900 kW for HVAC in underground

Cooling power margins OK but flow rate is limited

Baseline: accept higher DeltaT at user equipment (6-8 DegC)

Baseline solution:- Cut EG and EM- Replace SDX1 EM with EP- Install new chillers SU1 for EGHp: HVAC limits in SDX1 and Underground no changes

Version 1 -> eff. ~84% Version 2 -> eff. ~94%

Under study: part of TDAQ out of SDX1


P5 civil engineering

~400 m2

Old 80’s barracks

New Control Room

Engineering Centre

New Centre ~900 m2

Better housing existing teams already working on-site

Regroup engineering integration teams

HOSTLAB provides slab and primaries only

Existing Control Room

Lack of space for additional Workstations and screens

Existing Control Room adapted from originally intended DAQ room

Has incompatible HVAC shared with DAQ room

Unsafe as to close to DAQ room UPS & batteries

Shall work during shut down Visitor viewing pathway ~40 seats Internal fitting equipment not

funded by HOSTLABNB: Existing Control Room reverting to DAQ room -> covered by the TDAQ Upgrade at P5 project

New Control Room New Engineering Centre


Period 2021 … 2027 (-> for LS3)Budget estimate under discussion at the 2020 MTP

HOSTLAB Phase 2, project development history

2018 -> collection of requirements Definition of preliminary cost estimate

2019 -> trimming estimates vs. activities Revision of activities -> narrowed budget error margins Agreement baseline solutions Scrutiny of activities:

- For HOSTLAB -> reduced scope, prioritization, delayed, further tech review, out of context, other budget

- For Experiments -> re-tuned (enhanced) sharing Definition of budget estimate

2020 -> civil engineering IPPs (Infrastructure Project Proposal) Update of baseline solutions

- Change of activity implementations, more precise estimates- Cost and schedule reviews

Release of most critical part of budget

% of HOSTLAB Grand Total ATLAS CMS ALICE LHCB

Projects ~30% ~30% X X

Upgrades & Consolidations ~5% ~5% ~3% ~3%

Technical Services & Logistics ~4% ~4% ~4% ~4%

Safety & Radiation Protection ~2% ~2% ~2% ~2%

GRAND TOTAL HOSTLAB Ph2 CERN

GRAND TOTAL COLLABORATION FUNDING

~50%

~50%


1) Critical early expenses -> Urgent funding by MTP 2020- C.E. designs- CO2 primary cooling tendering- Studies new cooling system at P1

2) Expenses prior-LS3 and for EYETS 23/24 - Completion C.E. works- Completion CO2 cooling implementation- Advanced implementations of cooling system at P1- Advanced implementations of TDAQ at P5

LS3

Budget structure, spending profile

~50-50 CERN-EXPERIMENTStotal contribution ratio

(Pre-approved 2019 project)

3) Completion priority activities for Run 4- Requiring operation shut down- Commissioning new equipment/facilities

Thank you for your attention

- LETEM - HOSTLAB Phase 1 - HOSTLAB Phase 2 · EA coffee meeeting April 2nd, 2020 3 LETEM = LHC Experiments Technical and Engineering Meeting LETEM Committee (… should perhaps be

Documents