1 15 April 2010: Post Mortem Analysis by M.Zerlauth Automatic POWERING EVENT Analysis Adam, Arek, Gert-Jan, Ivan, Knud, Hubert, Markus, Nikolai, Nikolay, Odd, Reiner, Roman, Zinur and many others…
Dec 16, 2015
1
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
Automatic POWERING EVENT Analysis
Adam, Arek, Gert-Jan, Ivan, Knud, Hubert, Markus, Nikolai, Nikolay, Odd, Reiner, Roman, Zinur and many others…
2
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
Preamble
1st version of POWERING SERVERS ready for release early next week
– Support work of MP3 through continuous online analysis of powering events (with and
without beam)
– Data driven mechanism, ie arriving PM RAW DATA FILES will be grouped into powering
EVENTS, and are analyzed in SESSIONS
– Analysis session finishes typically some 7-8 minutes AFTER circuit trip and might (depending
on the analysis outcome) SUPERLOCK circuits in case of equipment malfunctioning
– Analysis module included in initial release• Single Circuit Event (SCE) module for dissection of event into individual circuit events• 600A Energy Extraction• 13kA Energy Extraction• Quench Heater Discharge Analysis (QHDA) • 600A Quench discard module• FGC_ext analysis (for possible feedback issues)• Automatic Powering Event Recognition (Summer module for final decision on event type,
identification of fault source, FGC faults, MP features, circuit lock,…)
3
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
Reminder: PM Server Architecture
PM Data files
PMA Scheduler
Event Builder
DataCollection
PM Data Manager
Libraries(references,DB AccessLogging)
GUI
ModulesModules
Database
LHCLogging
PMA server
PM buffers
notifies
triggers
runs
uses
reads writes
browses
triggers
Courtesy of Vito
4
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
ONLINE and PLAYBACK servers
ONLINE server – Launched from CCM or http://abwww/ap/dist/pm/powering/pm-powering-gui/PRO/pmpow-online-gui.jnlp
– For real-time analysis in CCC
– Completely automatic, including automatic confirmation & circuit locking
– Keeps history of past 15 analysis sessions in memory
PLAYBACK server– Launched from CCM or http://abwww/ap/dist/pm/powering/pm-powering-gui/PRO/pmpow-playback-gui.jnlp
– For offline analysis by equipment experts,…
– Allowing replaying ANY event in the past
– NO EFFECT ON LHC OPERATION, ie no circuit locking, etc….
– Manual confirmation/discard for potential saving of results
– Keeps history of past 15 analysis sessions in memory
5
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
1. Identify the super-locked circuit(s). Most likely the CCC will let you know, otherwise check the list of locked/super-locked circuits in the PIC SCADA
– Circuits locked by POWERING SERVER will be marked in Comment
6
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
2. Identify the EVENT during which the circuit was super-locked. If it is recent it will most likely still be present in ONLINE server, easiest to check PM Database (from CCM or https://cs-ccr-oas1.cern.ch/pls/htmldb_dbabco/f?p=117:1 )
7
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
3. Open the according analysis SESSION , either in the ONLINE server (if still present) or in PLAYBACK
In ONLINE: Use File -> Reconnect menu:
In PLAYBACK: Use File -> Load Event menu:
8
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
4. Identify event source + cause of locking in ‘Overview’ tab, eventually browse to details in ‘Modules Graph’ tab
9
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
4. Identify event source + cause of locking in ‘Overview’ tab, eventually browse to details in ‘Modules Graph’ tab
10
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
What to do in case a circuit gets super-locked
5. If applicable, document reason for failure in MP3 logbook, unlock circuit after authenticating in PIC SCADA
11
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
Let’s try some examples….
14-OCT-10 12.05.41.315000 PM
– RQX.R1 @ -1.493402958
– CIRCUIT_LOCK: Locked because of heater discharge
17-OCT-10 08.23.13.112000 PM
– Sector 23 trip from 3.5TeV following quench test, Heater firing on MQ.14R2
23-OCT-10 06.52.25.085000 AM
– Sector trip of A23 from injection current
– Trip of RQ9.R2 (+ heater discharge)
– RQTF.A23B2 @ 5.831202507 EE file not OK
12
15 A
pril
2010
: Pos
t Mor
tem
Ana
lysi
s by
M.Z
erla
uth
Conclusions
Server ready to be released mid next week (announcement will be sent to EIS’s, operators and MP3
members)
Limited number of modules included in initial version (a lot of tuning already done but tbc for e.g.
600A EE, 13kA EE, QPS_600….)
Expecting <1 circuit locks per day (depending on activities), please document in logbook cases where
you think the circuit lock was unjustified
I’m available to help, don’t hesitate to call me if needed (I’ll receive as well automatic e-mails)
Next weeks essential for learning with new system, will add new modules starting early next year