Console OperationsConsole Operations(Service Desk)(Service Desk)
Console Operators are Console Operators are tasked with a wide variety of tasked with a wide variety of functions and functions and responsibilitiesresponsibilities• We are the first point of contact for We are the first point of contact for
– Qualified Scheduling EntitiesQualified Scheduling Entities– Transmission Distribution Service Transmission Distribution Service
ProvidersProviders– Market ParticipantsMarket Participants– General PublicGeneral Public– ERCOT Employees.ERCOT Employees.
• We monitor the collective health of We monitor the collective health of approximately 350 - 400 Production approximately 350 - 400 Production servers and all test and development servers and all test and development servers.servers.
• We monitor the following thresholds:We monitor the following thresholds: – Hardware thresholds - Disk, CPU, Memory Hardware thresholds - Disk, CPU, Memory
utilization and capacityutilization and capacity– Application thresholds – Deployment failure, Tibco, Application thresholds – Deployment failure, Tibco,
Paperfree transaction failure (determined by the Paperfree transaction failure (determined by the application administrators)application administrators)
– Database – internal errorsDatabase – internal errors– Network infrastructure – ports, routers, switches, Network infrastructure – ports, routers, switches,
firewall, any type of problem with network devicesfirewall, any type of problem with network devices– Telecom – Point to points and Frame Relays to Telecom – Point to points and Frame Relays to
QSE’s QSE’s
• The Node Map is one of the tools The Node Map is one of the tools used for monitoring the servers.used for monitoring the servers.
• The Alarm browser is a tool to The Alarm browser is a tool to monitor hardware, software, and monitor hardware, software, and threshold alarms.threshold alarms.
• We monitor batch processes and We monitor batch processes and automated jobs automated jobs – Commercial ApplicationsCommercial Applications
• Registration, Data Acquisition/AggregationRegistration, Data Acquisition/Aggregation• Load ProfilingLoad Profiling• Settlements and Billing SystemsSettlements and Billing Systems• Collectively about 400 - 500 jobs and a few Collectively about 400 - 500 jobs and a few
more jobs that run once a week or once a more jobs that run once a week or once a month, etc.month, etc.
– EISEIS• Data WarehouseData Warehouse• Enterprise Data WarehouseEnterprise Data Warehouse• Data Archive and other reporting and Data Archive and other reporting and
miscellaneous support functions - 180 jobs / miscellaneous support functions - 180 jobs / 18 schedules18 schedules
• Security Monitoring Security Monitoring – Login attemptsLogin attempts
• failed logins and account lock outsfailed logins and account lock outs
– Virus trapsVirus traps• port shutdown subsequent to virus detectionport shutdown subsequent to virus detection
– Port violationsPort violations• a user or computer tries to use a port not a user or computer tries to use a port not
designated for that machinedesignated for that machine
– Unauthorized wireless access attemptsUnauthorized wireless access attempts
• Dedicated EMMSDedicated EMMS monitoring monitoring – The health and heartbeat of production The health and heartbeat of production
EMS servers responsible for Energy EMS servers responsible for Energy Management and Market SystemManagement and Market System
– All the Grid Operator terminalsAll the Grid Operator terminals
• Provide level 1 support for the Provide level 1 support for the above functionsabove functions– Per specific alarm, the Operator begins Per specific alarm, the Operator begins
remedial action to resolve the problemremedial action to resolve the problem– Notify and escalate the issue to level 2 Notify and escalate the issue to level 2
support personnel when applicablesupport personnel when applicable– Establish conference bridge with the Establish conference bridge with the
Subject Matter Expert and managers to Subject Matter Expert and managers to facilitate the resolution of the issuefacilitate the resolution of the issue
• Reporting and Miscellaneous Reporting and Miscellaneous – Daily verifications of Settlement Statements Daily verifications of Settlement Statements
and Invoices – send verification report to and Invoices – send verification report to business business
– Provide daily ticket report, outstanding Provide daily ticket report, outstanding ticket report and monthly report to ticket report and monthly report to managersmanagers
– Development FOB access reportDevelopment FOB access report– Send internal IT system notificationsSend internal IT system notifications– Direct incoming calls - AP/AR, business Direct incoming calls - AP/AR, business
solicitation calls, informational, power solicitation calls, informational, power outages, switching questions, concerns and outages, switching questions, concerns and complaints, media and market participant complaints, media and market participant callscalls
All Service Desk functions - Console All Service Desk functions - Console Operator's collect information Operator's collect information needed for problem resolution and needed for problem resolution and input into the tracking system while input into the tracking system while managing the issue until completion managing the issue until completion or escalation. or escalation.
A monthly average of appx. 2100 A monthly average of appx. 2100 calls and 2300 tickets are processed calls and 2300 tickets are processed by the Console Operations by the Console Operations department.department.
Lifecycle of an Issue from Console Operations Perspective
The Operator is notified of an issue by phone, e-mail, or monitoring system
A Problem Ticket is now generated by the Operator
Begin Begin remedial remedial action to action to
resolve the resolve the problemproblem
Level 1 Support is initiated
Begin Begin remedial remedial action to action to
resolve the resolve the problemproblem
Notify and Notify and escalate escalate
the issue to the issue to level 2 level 2 support support
personnel personnel when when
applicableapplicable
Level 2 Support is contacted
Begin Begin remediremedi
al al action action
to to resolve resolve
the the probleproble
mm
Notify Notify and and
escalate escalate the the
issue to issue to level 2 level 2 support support personnpersonnel when el when applicabapplicab
lele
Establish conference bridge
with support
personnel as
requested
Level 1 and Level 2 support utilized
Begin Begin remediremedi
al al action action
to to resolve resolve
the the probleproble
mm
Notify Notify and and
escalate escalate the issue the issue to level to level
2 2 support support personnpersonnel when el when applicabapplicab
lele
Establish Establish conferenconference bridge ce bridge
with with support support
personnepersonnel when l when neededneeded
Resolution is Resolution is defined as defined as
service being service being restoredrestored
Resolution
Begin Begin remediremedi
al al action action
to to resolve resolve
the the probleproble
mm
Notify Notify and and
escalate escalate the issue the issue to level to level
2 2 support support personnpersonnel when el when applicabapplicab
lele
Establish Establish conferenconference bridge ce bridge
with with support support
personnepersonnel when l when neededneeded
Resolution is Resolution is defined as defined as
service being service being restoredrestored
Ticket Ticket is is
closed closed and and
user is user is notifienotifie
dd
Closure