CO Support proposal for 2014 Marine Pace Technical Committee -12 Dec 2013
Jan 05, 2016
CO Support proposal for 2014
Marine Pace Technical Committee -12 Dec 2013
Context
• Detailed proposal for the organisation and the process of the CO support in 2014
• Main focus on PS Complex as subject to major controls changes & to a new support model
2
3
Recap of phases from Eugenia’s presentation
DRY RUNS COMMISSIONING & EARLY BEAM OPERATION
STABLE OPERATION
CO Support phaseswith respect to re-commissioning schedule 2014
(Latest Official schedule still being discussed EN/GS/OP)
July3 4 5 6 7
Jan Feb Mar Apr May Jun198 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 28 29
# # # 3 17 # 3 # 17
PSB
# # # 7 #5 #
L2
# # 2 9# # 7 # # ##
LEIR
PS
AD
Services
Ventilatin commissioning
Electrical distribution(EL, EPC)
Controls PSB/TT2
LEIR - L2-L3 PSB Chilled W.
Compressed air
Bake-out
Hardware tests
Hardw. tests
PS main magnets
POPS MPS
Hardw.
AD target SPS last circuits
Hardw.
ADEast area
Glob. Acc. Test
Access N.C.
Glob. Acc. Test
& DSO
Commissioning BEAMSource
Cold Check-out Beam set-up BEAM
Cold Check-out Beam set-up BEAM
HW TEST period : CO is involved in many DRY RUNS with EQP GP + OPAim : TEST as much as possible the controls infrastructure
PHASE 1 : CO DRY RUNS -> all experts required for the dry run will be present
CO Support phaseswith respect to re-commissioning schedule 2014
(Latest Official schedule still being discussed EN/GS/OP)
July3 4 5 6 7
Jan Feb Mar Apr May Jun198 9 10 11 12 13 14 15 16 17 18 20 21 22 23 24 25 26 27 28 29
# # # 3 17 # 3 # 17
PSB
# # # 7 #5 #
L2
# # 2 9# # 7 # # ##
LEIR
PS
AD
Services
Ventilatin commissioning
Electrical distribution(EL, EPC)
Controls PSB/TT2
LEIR - L2-L3 PSB Chilled W.
Compressed air
Bake-out
Hardware tests
Hardw. tests
PS main magnets
POPS MPS
Hardw.
AD target SPS last circuits
Hardw.
ADEast area
Glob. Acc. Test
Access N.C.
Glob. Acc. Test
& DSO
Commissioning BEAMSource
Cold Check-out Beam set-up BEAM
Cold Check-out Beam set-up BEAM
PHASE 2 -> Commissioning & Early Operation phase -> all experts on best effort and selected experts on special shift work
Phase 1: HW Tests
• CO Dry Runs• Involved: CO + EQP Groups + OP • Working hours• BE/CO support organization -> next slide• MCCs are responsible for the dry-runs
6
7
Dry-runs Organization
• 1 Dry run expands over one week • Several dry-runs per week • Groups involved: CO + EQP Groups + OP • Each dry-run deals with a system (ex: Matrix 10
MHz)
Claude’s courtesy
Dry-runs Insertion in OP Schedule - example
List of DRY RUNSMany in parallel
Dry-runs Support Team
MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY
AM PM AM PM AM PM AM PM AM PM
Preparation 1
BE-CO Test 1
Preparation 2
BE-CO Test 2Debriefing/Debugging/Consolidation Debriefing
5 working days Period
A specific team of 5 experts is assigned to each dry-run by the MCC
CO experts for all systems involved are available on-call
Claude’s courtesy
Next Steps
1. The dry Run planning (duration, date, content) will be consolidated by Claude (as planning officer) with input from MCCs
2. Then will be discussed with EQP groups + OP + Katy Foraz
3. The final agreed planning will be incorporated in the 2014 OP Schedule
Milestone: these steps must be completed by end January 2014
Next Steps for the MCC
• Prepare for each dry-run : • List of pre-requisites defined as Jira issues• List of checks defined as Jira issues• Wiki page centralizing all information• List of experts to be present during the whole dry-run and
those to be available on-call.
12
Phase 2: Cold Checkout
Can be during night time• 24-hour planning of checks defined by OP • CO INTENSE validation phase• Experts on best effort• Experts PRESENT in the CCC (full availability expected) during day
time• Some experts placed under special shift work conditions • Organization under MCC responsibility
13
Phase 2: Beam Set up & Early Operations phase
BEAM SETUP & OPERATIONS• CO is considered operational by OP but is NOT • Experts on Best Effort but high availability is expected• Requested experts placed under special shift work
conditions • Organization under EXM responsibility
14
Special Shift work for phase 2• Motivation for shift work: speed up CO issues resolution • Inspired from BI Commissioners organization at LHC startup• Role of expert on shift
– Offer support to OP on CO problems to diagnose and solve the problem– Report feedback on DIAG tools as will be heavily used – If time left during shift, work on DIAG tools
• Will be a natural trigger to improve diagnostics tools to avoid constant calls to CO experts
• Organization– 3 shifts per day – Max 2 shifts / week per person and typically only one out of normal
working hours– Compensation: experts on shift are required to take the morning off
after an evening shift and the complete day off after a night shift
Special Shift work • Current estimate of total shift period:– Max 3 months to cover all accelerators
• To be adjusted according to real time requirements• AD case (starting later, will benefit from debugging done on previous
accelerators -> no shift work may be required)
15
Start CCO Lin2/PSB
CO system STABLE
Special Shift work : status
• The detailed organization will be presented to the group early 2014
16
17
Phase 3: Beam Operations
•
Beam OPERATION • Controls stabilized• Upgrades only for justified requests and under
OP authorization• BEST EFFORT support ONLY
Support model in stable phase• BE/CO support is based on Best Effort specialists teams
– For all CO Hardware and Software – We reach a unique model for all accelerators PSX SPS LHC– Based on 2014 interventions statistics, we’ll evaluate and tune our
support strategy• Previous ‘Piquet’ suppression motivated by :
– 1st line FE support transfer to EQP Groups– New HW -> less failures (teething problems solved in previous phase) – Many problems reported to the PIQUET were outside its scope of
responsibility (wrong timing config….)• Pre-requisites for Best Effort mode to perform well:
– The responsibility transfer of first line support for FEs from CO to EQP is effective
– Strong diagnostics infrastructure -> Vito’s talk
19
First line support for Front ends 1/2• Responsibility is transferred to EQP Groups – Except for pure BE/CO systems (LTIM, OASIS, …)– Policy agreed between CO an EQP GP – Special case of legacy systems (ex. CPS/PSB BLM)
• Responsibility share between CO and EQP GP• Case by case study to be done
– To minimize intervention time, the EQP group specialist will have access to centralized stock of BE/CO HW modules to perform the HW change if (s)he is already on site • BE-BI is on board• Training of EQP experts on procedure to extract HW modules from CO
stock
• CCC will call directly the FRONT END specialist in the EQP GP
First line support for Front ends 2/2
• CO requirements– Every FE is described in the CCDB with its responsible
group and its operational support team– Information is propagated from CCDB to DIAMON and
the OP WEB Support page ‘webPiquet’:• DIAMON should display the operational support team when
clicking on the front end• WEBPIQUET should display the phone number(s) related to
the operational support team
• Actions on MCC– Ensure interfaces and data propagation are available– Check that operational support team is filled
INCA /LSA
.
OASIS TIMING
CMW/JAPC DRIVER
HW SPECIALISTS
CONSOLE + SERVER
... .
FRONT END SYSTEMS
SIS Logging
FEC OS…
EQP GP FRONT-END(Ex: TE/EPC FEC) BECO BEST EFFORT SERVICES
DIAMON/LASER
…...
OTHER SERVICES.
..
EQP GP FRONT-END(Ex: TE/EPC FEC)
EQP GP FRONT-END(Ex: TE/EPC FEC)
FRONT END INSTALLATION
Organization of expert teams under discussion in CO- Team membership - Generic phone number - Anonymous phone number …
BEST EFFORT Expert Teams– not exhaustive nor correct list
OPTeams likely often contacted > hence the importance of good diagnostics
Scenario for a problem
• A problem manifests in a high-level application or on DIAMON screen1. OP will have improved error reporting from application and core
diagnostic tools to identify the source of problem2. Possible recovery solutions will be available via DIAMON (repair,
restart process, reboot FE, etc. ) to be done by OP• Requisite: DIAMON will display the right status of all FEs at all times
3. If no recovery was possible by available actions, OP will call the best effort expert from the OP Web support list & record the intervention
4. In case of a FE problem: EQP GP specialist is called and runs his expert diagnostics
5. if it turns to be a CO FE problem, he will ask the CCC to call a CO specialist from the Best Effort team identified as concerned.
Intervention Recording• The recording of each problem and intervention should be done by OP using
eLogbook– This is critical to ensure the compensation of the experts on best effort when they are
called by the CCC• The best effort specialist will ensure the information is updated at the end of his own
intervention
– A technical report of the intervention will be written by the specialist• This will be used by the second expert and ease the hand over
• This means extensions on e-logbook & Jira– e-logbook: Creation of FAULT sub-categories under ‘ CONTROLS FAULTS’ category,
basically one per BEST EFFORT system– e-logbook: Write access mode from home– …JIRA issues automatic creation: to be generalized to all issues & promoted to LHC &
SPS• OP agreed on all of these, design to be finalized for elogbook + JIRA
• CO recorded interventions will be monitored by EXM to ensure any persisting problems are properly followed up and resolved inside the group
Perspectives 2014 …… for LHC !
24
Early 2015 schedule “Nominal" 25 ns Physics from July’15
b* = 19 m
vdM
LHCf
Work in progress!
RECOMMISSIONING WITH BEAM
Preliminary Planning – Commissioning starts in 2014
• 2014
26
Month Activity2014
Jan
Feb
Mar
Apr
May
June
July
Aug
Sept
Oct
Nov
Dec
2015
Jan
Feb
Mar
Dry Run
Dry Run
Dry Run
Dry Run
Dry Run
Dry Run
Dry Run
Sector Test
Check-out
Beamcomm.
Dry Run,Settings,
Cycle
LBDS Prep
LBDS reliability run: CCC: 6 weeks June/July
From April:
Every 2nd week: dry run
October:
Every week: dry run + sector test preparation
Beginning of Nov:
Sector Test
January:
Machine Check-Out
Feb + Mar:
Beam Commissioning
LBDS
+LBDS Prep
+LBDS Prep
Additional slides
27
Additional slides
28
OP SUPPORT Web page webPiquet
Tentative Re-commissioning Schedule(still being discussed EN/GS/OP)