Stefan Schlenker, CERN ATLAS Post-LS1 Operations Workshop, 24th June 2013, CERN DCS Tools during ATLAS Run 1 Questionnaire Results Stefan Schlenker, CERN.

Stefan Schlenker, CERNStefan Schlenker, CERNATLAS Post-LS1 Operations WorkshopATLAS Post-LS1 Operations Workshop,, 24th June 2013, CERN24th June 2013, CERN

DCS Tools during DCS Tools during ATLAS Run 1 ATLAS Run 1 Questionnaire ResultsQuestionnaire ResultsStefan SchlenkerStefan Schlenker, CERN, CERN


Detector Control System – Detector Control System – Operation ToolsOperation ToolsArchitectureArchitecture► Designed to be controlled by Designed to be controlled by

single operator single operator at GCS layerat GCS layer

Shift operationShift operation► Alarm system Alarm system – low level– low level► Finite State machine Finite State machine – –

high level and navigation high level and navigation

Expert operationExpert operation► Remote access via Remote access via

WTS/gatewayWTS/gateway – full access – full access► Web online monitoringWeb online monitoring – at a – at a

glanceglance► Web history Web history – glance back in – glance back in

timetime► DCS Data Viewer (DDV)DCS Data Viewer (DDV) – –

interface to DCS archive on interface to DCS archive on Oracle serversOracle servers

► Notification framework Notification framework – manage – manage SMS/emailSMS/email


System Assignment / ToolsSystem Assignment / ToolsDCS TreeDCS TreeSystemSystem assignment @end of run 1 assignment @end of run 1► ID desk: ID desk: PIXPIX,, SCT SCT,, TRT TRT, IDE (incl. , IDE (incl. BCMBCM))► CAL desk :CAL desk : LARLAR, , TILTIL, FWD (=, FWD (=LUCIDLUCID+ZDC++ZDC+ALFAALFA))► Muon desk :Muon desk : MDT, CSC, RPC, TGC, MUON (= MDT, CSC, RPC, TGC, MUON (=MUONMUON))► Data quality desk :Data quality desk : LHC (= LHC (=DQDQ,, LUMI LUMI))► Trigger/DAQ desk :Trigger/DAQ desk : TDQ ( TDQ (TRIGGERTRIGGER, , DAQDAQ,, L1CALO L1CALO))► Slimos & ShiftLeader:Slimos & ShiftLeader: CIC, EXT, SAFETY, DCS BE (OPM, CIC, EXT, SAFETY, DCS BE (OPM, DCSDCS))

*System covered in questionnaire results (total 16, no ADC)*System covered in questionnaire results (total 16, no ADC)

Separation of toolsSeparation of tools► Desk Alarms: Desk Alarms: Assign respective DCS control stations to filterAssign respective DCS control stations to filter► Desk FSM: Desk FSM: Attach respective tree nodes to Attach respective tree nodes to artificial desk top nodeartificial desk top node, flexible in reassignment , flexible in reassignment

IDG, CAL, MUO, TI IDG, CAL, MUO, TI

Q: System shifter has DCS tasks?Q: System shifter has DCS tasks?► 14 Yes14 Yes, all but DCS, DAQ, all but DCS, DAQ

Q: Tools used?Q: Tools used?► Desk Alarms: Desk Alarms: 1313 (all but TRIGGER) (all but TRIGGER)► Desk FSM: Desk FSM: 1414

► Constant monitoring of specific panel? Constant monitoring of specific panel? 8 No, 2 Occasionally (for specific problems), 8 No, 2 Occasionally (for specific problems), 1 Permanently (RPC rates+tower status)1 Permanently (RPC rates+tower status)

► Non standard tools used for DCS?Non standard tools used for DCS? No No


Finite State Machine Finite State Machine (FSM)(FSM)

SCADA / PVSS Layer SCADA / PVSS Layer

GCSGCS

SCS 2SCS 2 SCS NSCS N

TTCN.1

LCSN.1. 1

DevN-1

LCSN. 1. 2

DevN

...

TTC1 . N

SCS 1SCS 1

TTC1. 1

TTC1. 2

LCS1 .1. 1

LCS1. 1. 2

Sub-sys1 .1. 2.1

Sub-sys1 .1. 2. 2

Dev1

Dev2

Dev4

Dev3

Dev5

...

ShiftOperator

DetectorExpert

AnotherDetector

Expert

STATE &COMMANDS STATUS

State machine object

OKOK

WARNINGWARNING

ERRORERROR

FATALFATAL

READYREADY

UNKNOWNUNKNOWN

STANDBY***STANDBY***

NOT_READYNOT_READY

GOTO_STANDBY

GOTO_SHUTDOWN

SHUTDOWNSHUTDOWN

TRANSITIONTRANSITION

State machine hierarchyState machine hierarchy► Detector hardware and Detector hardware and hierarchical structure hierarchical structure represented represented

by FSM entitiesby FSM entities

State, Status, and Command propagationState, Status, and Command propagation► State model State model for devices (for devices (ON,OFF,…ON,OFF,…) and logical objects ) and logical objects

((READY, NOT_READY,…READY, NOT_READY,…), ), error handling error handling with STATUSwith STATUS► Propagation upwards Propagation upwards (using programmed logic for parent (using programmed logic for parent

depending on child states), Shifter monitors everything = depending on child states), Shifter monitors everything = READY/OKREADY/OK

► CommandsCommands propagated downwards propagated downwards

Current procedure in SL training:Current procedure in SL training:► If problem cannot be fixed with instructions, If problem cannot be fixed with instructions, contact expertcontact expert► Report to Report to ELOGELOG► If problem cannot be solved on short timescale (<~15min), If problem cannot be solved on short timescale (<~15min),

Disable or Exclude Disable or Exclude corresponding FSM object(s) corresponding FSM object(s) in in agreement with expertagreement with expert

► In particular In particular Disable/ExcludeDisable/Exclude when in state other than when in state other than READYREADY

► Interventions Interventions should be done by handing over ownership should be done by handing over ownership ((ExcludeExclude) to experts on DCS desk, ) to experts on DCS desk, IncludeInclude sub-tree when sub-tree when intervention is overintervention is over


Finite State Machine:Finite State Machine: Questions Questions

SCADA / PVSS Layer SCADA / PVSS Layer

GCSGCS

SCS 2SCS 2 SCS NSCS N

TTCN . 1

LCSN . 1 . 1

DevN - 1

LCSN . 1 . 2

DevN

...

TTC1 . N

SCS 1SCS 1

TTC1 . 1

TTC1 . 2

LCS1 . 1 . 1

LCS1 . 1 . 2

Sub-sys1 . 1 . 2 . 1

Sub-sys1 . 1 . 2 . 2

Dev1

Dev2

Dev4

Dev3

Dev5

...

ShiftOperator

DetectorExpert

AnotherDetector

Expert

STATE &COMMANDS STAT

US

State machine object

OKOK

WARNINGWARNING

ERRORERROR

FATALFATAL

READYREADY

UNKNOWNUNKNOWN

STANDBY***STANDBY***

NOT_READYNOT_READY

GOTO_STANDBY

GOTO_SHUTDOWN

SHUTDOWNSHUTDOWN

TRANSITIONTRANSITION

Q: System shifter instructed to use FSM actions?Q: System shifter instructed to use FSM actions?► Yes:Yes: MUON MUON► Only by expert instruction:Only by expert instruction: L1CALO, LAR (almost never), LUCID, SCT, L1CALO, LAR (almost never), LUCID, SCT,

TIL, TRTTIL, TRT► Never: Never: ALFA, BCM, DQ, LUMI, PIX, TRIGGERALFA, BCM, DQ, LUMI, PIX, TRIGGER

Q: Actions on VME crates by whom?Q: Actions on VME crates by whom?► Experts:Experts: all except LUCID, MDT/CSC (also all except LUCID, MDT/CSC (also shiftersshifters))

Q: Lost data because only expert could take action?Q: Lost data because only expert could take action?► 13 No13 No► 1 Yes (VME crate power cycle: expert who was not in reach of computer)1 Yes (VME crate power cycle: expert who was not in reach of computer)

Q: Shifters were instructed to disable/exclude parts of tree?Q: Shifters were instructed to disable/exclude parts of tree?► No:No: ALFA, BCM, DAQ, L1CALO, LAR, LUCID, LUMI, PIX, TIL, TRT, ALFA, BCM, DAQ, L1CALO, LAR, LUCID, LUMI, PIX, TIL, TRT,

TRIGGERTRIGGER► Yes, when instructed by expert: Yes, when instructed by expert: SCTSCT► Yes, for specific cases from instructions/training: Yes, for specific cases from instructions/training: MUONMUON

Q: Shifters were instructed to leave FSM in a state other than READY for >1hQ: Shifters were instructed to leave FSM in a state other than READY for >1h► 9 No 9 No (however, 2 of them inconsistent answer)(however, 2 of them inconsistent answer)► 3 Yes3 Yes (TIL, TRT, LUMI) (TIL, TRT, LUMI)


Alarm ScreenAlarm ScreenConcept and tool optionsConcept and tool options►Alarm Alarm = parameter out of good range, ranges defined by experts= parameter out of good range, ranges defined by experts►Acknowledgement:Acknowledgement:

► Only for dedicated alarmsOnly for dedicated alarms► Makes sure Makes sure alarm does not disappear before operator actionalarm does not disappear before operator action: :

acknowledgeacknowledge►Options for each alarm (right-click menu):Options for each alarm (right-click menu):

► Mask Mask alarm on UI level until the alarm condition goesalarm on UI level until the alarm condition goes► Insert alarm to Insert alarm to ELOGELOG with comment with comment► Display Display trendtrend plot of value plot of value► Alarm help Alarm help on Twikion Twiki► Expert actionsExpert actions: change description, alert ranges: change description, alert ranges

►Summary alert:Summary alert:► Hides Hides several individual alerts in a single alarm entryseveral individual alerts in a single alarm entry► Access to individual alerts via “Details”Access to individual alerts via “Details”

►Filters:Filters:► Different sets of filters exist, e.g. sub-detectorsDifferent sets of filters exist, e.g. sub-detectors► ““Default” filter can be Default” filter can be pre-defined per deskpre-defined per desk

Current procedure in SL Current procedure in SL training:training:

►Understand alarmUnderstand alarm, check docu for , check docu for proceduresprocedures

►If problem cannot be fixed, If problem cannot be fixed, contact expertcontact expert

►If alarm condition cannot be If alarm condition cannot be removed on short timescale removed on short timescale (<~15min), (<~15min), maskmask alarm alarm in in agreement with expertagreement with expert

►If alarm condition cannot be If alarm condition cannot be removed on long timescale removed on long timescale (>1week), expert is responsible to (>1week), expert is responsible to deactivatedeactivate alarm or change alarm alarm or change alarm limits until problem is solvedlimits until problem is solved


Alarm Screen:Alarm Screen: Questions QuestionsQ: Alarms, shifters were told to ignore?Q: Alarms, shifters were told to ignore?► 6 No6 No► 2 Yes, but only during test runs2 Yes, but only during test runs► 3 Yes, but only for a very short time (few 3 Yes, but only for a very short time (few

minutes)minutes)► 2 Yes 2 Yes

Q: Acknowledgement: shifter or expert action?Q: Acknowledgement: shifter or expert action?► 8 experts only8 experts only► 1 Only on expert instruction1 Only on expert instruction► 2 Only “non mission-critical” alarms2 Only “non mission-critical” alarms► 1 Only for white-listed WENT alarms1 Only for white-listed WENT alarms

Q: Mask: shifter or expert action?Q: Mask: shifter or expert action?► 6 experts only6 experts only► 3 Only on expert instruction3 Only on expert instruction► 3 Only for white-listed alarms 3 Only for white-listed alarms

Q: How often did shifters ignore a critical (FATAL or Q: How often did shifters ignore a critical (FATAL or ERROR) alarm?ERROR) alarm?►6 Never6 Never►1 Single occurrence (hesitated to call during night)1 Single occurrence (hesitated to call during night)►2 Few times2 Few times►1 25% (ALFA)1 25% (ALFA)

Q: Alarm authorization scheme should change (who Q: Alarm authorization scheme should change (who is allowed acknowledge, mask)is allowed acknowledge, mask)►8 No8 No►2 No, but enforce expert notification (elog) on 2 No, but enforce expert notification (elog) on mask/acknowledgemask/acknowledge►1 Shift Leaders should not be able to 1 Shift Leaders should not be able to acknowledgeacknowledge►1 Only experts should be able to mask1 Only experts should be able to mask►1 Maybe1 Maybe►2 No opinion2 No opinion

??


Web Alarm HelpWeb Alarm Help

Q: Usage?Q: Usage?► 6 No6 No► 6 Yes6 Yes► 1 YES!1 YES!

Q: Filled?Q: Filled?► 2 100%2 100%► 4 > 50%4 > 50%► 1 most important 1 most important

onesones

ConceptConcept► For each For each alarm type alarm type a a TwikiTwiki page is assigned by naming page is assigned by naming

conventionconvention► DCS help web DCS help web portalportal: : daily scan of DCS alarms daily scan of DCS alarms in online in online

system system and Twiki serverand Twiki server, fills portal page with, fills portal page with► Existing help pages grouped by descriptionExisting help pages grouped by description► Not documented alarms suggesting Twiki page for creationNot documented alarms suggesting Twiki page for creation► Alarms with wrong descriptionsAlarms with wrong descriptions

► Help portal for individual alarm reachable Help portal for individual alarm reachable via direct alarm via direct alarm screen right-clickscreen right-click

TwikiTwiki can be can be createdcreated from alarm screen from alarm screen or by or by systematically browsing the portalsystematically browsing the portal

Q: Experience?Q: Experience?► Quality varies, some shifters Quality varies, some shifters

don’t knowdon’t know

Q: Not used because?Q: Not used because?► 2 Not necessary2 Not necessary► 1 Not aware1 Not aware► 1 Manpower/had no strong 1 Manpower/had no strong

needneed► 1 Small detector1 Small detector


Automatic ActionsAutomatic ActionsQ: Did you rely on automatic DCS actions?Q: Did you rely on automatic DCS actions?► 8 Yes, used regularly8 Yes, used regularly► 2 Yes, but rarely or never triggered2 Yes, but rarely or never triggered► 2 No 2 No

Q: Were automatic actions notified to shifter?Q: Were automatic actions notified to shifter?► 7 Yes, in FSM or alarm screen7 Yes, in FSM or alarm screen► 1 MRS only1 MRS only► 1 No, experts only: log files, SMS, 1 No, experts only: log files, SMS,

email…email…

Q: Shifter confusion due to automatism?Q: Shifter confusion due to automatism?► 2 Sometimes2 Sometimes► 2 Rarely2 Rarely► 4 Never4 Never

Proposal for introducing new DCS tool:Proposal for introducing new DCS tool:

► Central Central log viewer forlog viewer for DCS messagesDCS messages

► Currently, logs are only written to file but central message passing tool exists including Currently, logs are only written to file but central message passing tool exists including archiving to archiving to OracleOracle

► ATLAS DCS log message typesATLAS DCS log message types: : Debug, Information, Action, Warning, Error, Fatal

► Could use for Could use for shifter feedback messagesshifter feedback messages and and define additional message classes, e.g. define additional message classes, e.g. “UI Action”, “UI Action”, “Procedure Start”, “Procedure End”, … “Procedure Start”, “Procedure End”, …

► Different Different LogViewer default configuration LogViewer default configuration for ACR for ACR or Expert mode, Sub-det filters etc.or Expert mode, Sub-det filters etc.

► DDV plugin exists DDV plugin exists Web-accessible DCS logs Web-accessible DCS logs

► Discussed already at DCS workshop last yearDiscussed already at DCS workshop last year


Beam Actions, Beam Actions, LHC HandshakeLHC HandshakeQ: STANDBY/READY transition Q: STANDBY/READY transition

automatic?automatic?► automatic automatic ALFA, MUON, SCTALFA, MUON, SCT► manual, expert action manual, expert action PIXPIX

Q: STABLE BEAMS transition time?Q: STABLE BEAMS transition time?► MUON: MUON: 3~5 minutes3~5 minutes► PIX: PIX: About 3 minAbout 3 min► SCT: SCT: 60 seconds60 seconds

Q: Automate LHC Handshake or keep manual SL action?Q: Automate LHC Handshake or keep manual SL action?► 1 Keep SL approval1 Keep SL approval► Rest: No strong opinion, but keep possibility of disabling automatic Rest: No strong opinion, but keep possibility of disabling automatic

mode or have fallback in case of problemsmode or have fallback in case of problems

Train InjectionTrain Injection

Energy RampEnergy Ramp

CollisionsCollisions

Stable BeamsStable Beams

Pixel HVPixel HV

Drift Chamber HVDrift Chamber HV

Beam EnergyBeam Energy

LuminosityLuminosity

Beam IntensityBeam Intensity

Stable FlagStable Flag


DCS Notifications:DCS Notifications: Questions QuestionsQ: Did you rely on DCS SMS notifications?Q: Did you rely on DCS SMS notifications?► 10 Yes (FSM states, infrastructure alarms etc. all also available to shifters)10 Yes (FSM states, infrastructure alarms etc. all also available to shifters)► 2 No 2 No

Q: Did you rely on DCS email/Elog notifications?Q: Did you rely on DCS email/Elog notifications?► 8 Yes (equal or less as SMS sending)8 Yes (equal or less as SMS sending)► 4 No4 No

Q: Why automatic notifications?Q: Why automatic notifications?► 3 Redundancy to shift operation (e.g. compensate missing shifter reaction)3 Redundancy to shift operation (e.g. compensate missing shifter reaction)► 2 Timely reaction by experts on critical issues2 Timely reaction by experts on critical issues► 1 Make sure that correct expert is informed1 Make sure that correct expert is informed► 1 Wake up experts before shifter call1 Wake up experts before shifter call

Q: Data loss due to notification failure?Q: Data loss due to notification failure?► 9 No9 No► 1 Not sure1 Not sure

Q: Data loss due to no expert reaction to notification?Q: Data loss due to no expert reaction to notification?► 9 No9 No► 1 Not sure1 Not sure


DCS Notification FrameworkDCS Notification FrameworkManage NotificationsManage Notifications► Generic tool to Generic tool to

create/modify/manage SMS/email create/modify/manage SMS/email notifications notifications via FSM screen (DCS via FSM screen (DCS experts)experts)

► DefineDefine► AddressingAddressing (who gets it) (who gets it)

using phonebook searchusing phonebook search► Flood protection Flood protection (e.g. only 1 (e.g. only 1

SMS per hour)SMS per hour)► Trigger conditions Trigger conditions (e.g. (e.g.

alarms)alarms)► Boundary conditions Boundary conditions (stable (stable

beams = true)beams = true)► Generic handling of Generic handling of

activation/deactivationactivation/deactivation► Read-only Read-only browsingbrowsing by non- by non-

experts (who sends me SMS???)experts (who sends me SMS???)► Special mode: Special mode: reportsreports (e.g. send (e.g. send

me all FSM nodes of type X which me all FSM nodes of type X which are disabled but OK on every are disabled but OK on every weekday)weekday)

► Should Should facilitate debugging facilitate debugging of of notification problems notification problems by more than by more than one expertone expert

► Ready to be activated Ready to be activated for your for your sub-detsub-det


Remote Tools:Remote Tools: Questions QuestionsQ: Need for additional remote DCS monitoring tools?Q: Need for additional remote DCS monitoring tools?► 9 No9 No► 1 Applications logs1 Applications logs► 1 Smart phone apps?!1 Smart phone apps?!► 1 Alarm acknowledgement log1 Alarm acknowledgement log

mySCT

On my way!

Redirect on-call

How to avoid panic

Call PL


DDV: Alarm HistoryDDV: Alarm HistoryWeb Alarm HistoryWeb Alarm History► Plugin of Plugin of

DcsDataViewerDcsDataViewer► QueryQuery any time any time

interval for DCS interval for DCS alarmsalarms

► BrowsingBrowsing by sub- by sub-det/systemdet/system

► Easily handles Easily handles tenten thousands of alarm thousands of alarm eventsevents

► Powerful Powerful filtering filtering functions functions for resultfor result

► Store Store search criteria search criteria (e.g. last 24h LAR (e.g. last 24h LAR HV trips) and HV trips) and call call anytime using single anytime using single URLURL

https://atlas-ddv.cern.ch


DiscussionDiscussion

Are we ready for a common DCS shifter?Are we ready for a common DCS shifter?

Stefan Schlenker, CERN ATLAS Post-LS1 Operations Workshop, 24th June 2013, CERN DCS Tools during ATLAS Run 1 Questionnaire Results Stefan Schlenker, CERN.

Documents