Incident Management & Communication Incident Management & Communications Procedures Guide Version 1.91 Last Updated: December 2, 2009
Incident Management & Communication
Incident Management & Communications
Procedures Guide
Version 1.91
Last Updated: December 2, 2009
Incident Management & Communication
2
This page left intentionally blank
Incident Management & Communication
3
Table of Contents Introduction............................................................................................................................................................. 4 Severity Level Definitions ...................................................................................................................................... 5 Communication Checklist....................................................................................................................................... 8
Manager On Call (MOC)................................................................................................................................ 8 IT Center ......................................................................................................................................................... 9
Incident Manager On Call – IMOC List ............................................................................................................... 10 Incident Manager On-Call (IMOC) .............................................................................................................. 11 University IT Technical Staff / Technicians on Call .................................................................................... 12 University IT Director - of affected unit(s)................................................................................................... 12 Information Security ..................................................................................................................................... 13 CIO’s Office.................................................................................................................................................. 14 Communication Manager and/or Other Designated University IT Employees............................................ 15 Scribe ............................................................................................................................................................ 16 University IT Office Admins ........................................................................................................................ 16 University IT Staff Members ........................................................................................................................ 16 Provost .......................................................................................................................................................... 16 President........................................................................................................................................................ 16 Other University Executives ......................................................................................................................... 17 Students......................................................................................................................................................... 17 Faculty / Departments or Divisions .............................................................................................................. 17 University Staff............................................................................................................................................. 17 University Security ....................................................................................................................................... 17 University Facilities ...................................................................................................................................... 17 Rochester Management................................................................................................................................. 17 University Legal............................................................................................................................................ 17 University HR ............................................................................................................................................... 17
Communication Call Log...................................................................................................................................... 18 Security Level Definitions .................................................................................................................................... 20 Internal Communications Template...................................................................................................................... 21 External Communications Template..................................................................................................................... 23 University IT Technician Form ............................................................................................................................ 25 External Communication Matrix .......................................................................................................................... 26 Incident Command Center Wall Charts................................................................................................................ 31 IT Alert (G2Alert) – Steps to Send a Severity 3 IT Alert:.................................................................................... 40 ISD Manager On-Call - University IT (Data Center Services) Alert Notification ............................................... 41 Appendix............................................................................................................................................................... 43
Roles & Responsibilities................................................................................................................................... 44 Incident Manager On-Call (IMOC) .............................................................................................................. 44 Manager On-Call (MOC).............................................................................................................................. 45 Communications Manager ............................................................................................................................ 46
Web Content Hack – Immediate Actions ......................................................................................................... 51 Debrief Procedures............................................................................................................................................ 52 Debrief Agenda Template................................................................................................................................. 53 Updating Procedures......................................................................................................................................... 54 Change Control ................................................................................................................................................. 55
Incident Management & Communication
4
Introduction Leaders in the University Information Technology organization acknowledged the need to develop a wider view of incident management and communications. In the past, each University IT department utilized its own incident escalation path. Consistency in delivering incident management and expected communication levels were not meeting internal and external customer expectations, especially during high profile incidents. This Incident Management & Communication Procedures manual contains Severity 3 incident response tools. Severity 3 incidents are the highest level and most critical of events that occur within our organization. Immediate action is required by multiple people to assist in recovering services affected by the incident. By identifying scope and ownership of an incident early in the process, we can now triage to the appropriate teams, who in turn establish their communication protocols and management roles within the context of the broader incident management procedures. Incident management and communication processes that had been used independently across the organization are now merged into a single document and available across University IT. On-call escalation now has the ability to mobilize an Incident Manager On-Call (IMOC) who coordinates the Incident Command Center and communication methods to executives and customers. Each department’s Manager On-Call (MOC) can now concentrate on recovering services, without the need to communicate with multiple people. Technicians will also benefit from these procedures by eliminating multiple communication paths and allowing them to concentrate on technical issues. Each Severity 3 incident will have a Communication Manager, assigned to assist with the creation of communication materials. A scribe will detail incident events. After the recovery from an incident, a mandatory debrief meeting will be scheduled to complete the Sev 3. Documentation for the debrief methodology has been finalized and is included in this manual. A coordinated University IT response is essential to our business and services. Our customers demand it, our internal resources need it, and the Information Technology Services Incident Management & Communication Procedures Guide delivers it.
Incident Management & Communication
5
Severity Level Definitions
Severity Level 3.0 2.5 2.0 1.5 1.0 0.0
Service Impact Enterprise‐wide Enterprise‐wide Limited Single or None Single or None Single or None
Immediate need for service
No immediate need for service
Single department affected by service
interruption
Single user service impact
Single user service impact
Single user service impact
Scope may not be defined Scope is defined Aged General User
Ticket
No Service Impact with complex
elevated resolution
No Service Impact with elevated resolution
No Service Impact
Complete service outage VIP User Ticket Elevated User
Ticket (Director, Manager)
General User Ticket General User
Ticket General User
Ticket
Triggers formal communication plan
Triggers formal communication plan
Triggers informal communication
plan
Resolution by Tech Lead/System
Lead
Resolution by Subject Matter Expert (SME)
Resolution by IT Center staff
Multiple departments, groups, and individuals
Enterprise‐wide impact
Severity Level Defined
University‐wide security violation/compromise
Decision Maker IMOC/Director Director Manager Level III:
Tech Lead, System Lead
Level II: Subject Matter
Expert Call Agent
University IT Director University IT Director University IT Manager
Level III Support Level II Support Call Agent Only
University IT Manager University IT Manager Level III Support Level II Support Call Agent
IMOC IMOC Call Agent
CIO Office Appropriate University
IT Personnel
Senior Management IT Center
All University IT
IT Center
Involvement
ISD (if Data Center or Network Related)
Who is Notified ‐ By Whom
Immediate Notification
IMOC ‐ by Director Director ‐ by Manager Manager ‐ by Level
III Level III ‐ by Level II
Level II ‐ by Call Agent
Call Agent ‐ by User
CIO Office ‐ by IMOC IMOC ‐ by Director User ‐ by Ticket
Assignee User ‐ by Ticket
Assignee User ‐ by Ticket
Assignee User ‐ by Ticket
Assignee
Senior Management ‐ by CIO Office
IT Center ‐ by Director
All University IT ‐ by Hyper‐Reach or Email
All University IT ‐ by Hyper‐Reach or
IT Center ‐ by Director
Additional Notifications
ISD ‐ by Hyper‐Reach Communication
Plan Type Formal* Formal* Informal Informal Informal Informal
Direct Contact (phone, in‐person)
Direct Contact (phone, in‐person)
Direct Contact (phone, in‐person)
Service Ticket Service Ticket Walk‐in, phone call, email, web
form
Communication Methods
Hyper‐Reach Hyper‐Reach Real‐time
Communications
Incident Management & Communication
6
ITENS ITENS
Incident Management & Communication
7
Incident Management & Communication
8
Communication Checklist This document provides a high-level overview of the communication flow that needs to take place during a declared Severity 3 (Sev 3) incident. University IT divisions will assess incidents as normal until a Sev 3 has been declared – once elevated to a Sev 3, initiate this checklist. Normal Business Hours (8:00am – 5:00pm): Applies to weekdays and non-holidays.
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
1 Manager On Call (MOC)
Determines if University IT Security, University Security, University Facilities, and/or Rochester Management need to be engaged. If yes, engages each required units (6, 18, 19, 20).
Notify Unit Director Declares Severity 3 Incident. Notifies Customer Contact Centers: IT Center (5-2000)
and NC Ops Center (4-4357) and DC Ops (5-1205). If no one is available to answer the call, the answering service process will kick in.
Direct IT Center to maintain CHRON until scribe is identified.
Notifies IMOC (3). Provides them with a brief of the situation.
Assembles and leads technical teams/technicians that must be on-site unless otherwise directed by IMOC. Determines meeting location and initiates MOC Phone Bridge if needed (1-866-603-2932 Access #6608484 Pin #9058 (Host only))
Identifies relevant vendors that may be needed. MOC will determine if techs need to forward their phones
(internal calls only) allowing uninterrupted problem solving.
Determines if University IT Security, University Security, University Facilities, and/or Rochester Management need to be engaged. If yes, engages each required unit (6, 18, 19, 20).
Notify Unit Director Declares Severity 3 Incident. Notifies Customer Contact Centers: IT Center (275-2000)
and NC Ops Center (274-4357) and DC Ops (275-1205). If no one is available to answer the call, the answering service process will kick in.
Begins and maintains CHRON until scribe is identified.
Notifies IMOC (3). Provides IMOC with a situation brief and determines on-site support needs. Determines resources that need to be on-site. Potential use of IT Alert (www.g2alert.com)
Assembles and leads technical teams/technicians that must be on-site unless otherwise directed by IMOC. Determines meeting location and initiates MOC Phone Bridge if needed (1-866-603-2932 Access #6608484 Pin #9058 (Host only))
If incident is over 12 hours, coordinates staffing schedule Identifies relevant vendors that may be needed. If the IT Center is not open, MOC for affected
department(s) is responsible for coordinating customer communication.
Incident Management & Communication
9
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
2 IT Center
If the IT Center is open, provide customers with IMOC-supplied information
Ensures MOC(s)/MOC Designee of affected
department(s) was notified and is aware of the situation.
Notifies both service center staff members. If Towne House evacuation, contact NCS Manager
on Call to forward Operations phone numbers 5-9194 & 5-9195 to 5-2000
Notifies University IT-ORG. Communication should provide a brief of the situation, what the solution is, and if the event is still ongoing. Use IT Alert (www.g2alert.com).
Provides guidelines for customer communication as determined by the IMOC, MOC, and Communications Manager or other key players as needed based on incident type.
Triages calls and provides updates as requested by MOC.
The Networking Operations Center serves as a hub to coordinate the communication with customers and University IT contacts. Both centers are effective at handling this communication. Keeps the customer list up-to-date and monitors the service impact by customer base through direct customer contact.
Periodically checks in with customers to assess the situation (Are fixes working? Are users still experiencing problems?) – be sure to include University faculty, staff, and students in relevant locations.
If IT Center is open, provide customers with IMOC-supplied information. If neither center is open, MOC for affected department(s) is responsible for this communication.
Ensures MOC(s) of affected department(s) was
notified and aware of the situation.
If Towne House evacuation, contact NCS Manager on Call to forward Operations phone numbers 5-9194 & 5-9195 to 5-2000
Provides guidelines for customer communication as determined by the IMOC, MOC, and Communications Manager or other key players as needed based on incident type.
Triages calls and provides updates as requested by MOC.
The IT Center serves as a hub to coordinate the communication with customers and University IT contacts. Both centers are effective at handling this communication. Keeps the customer list up-to-date and monitors the service impact by customer base through direct customer contact.
Periodically checks in with customers to assess the
situation (Are fixes working? Are users still experiencing problems?) – be sure to include University faculty, staff, and students in relevant locations.
Incident Management & Communication
10
Incident Manager On Call – IMOC List Contact Operations (275-9194) or (275-1205) for most current IMOC list
Group covers the following area(s): The role of the Incident Manager On Call is to lead Severity 3 and Severity 2.5 incidents. The Incident Manager On-Call is available 24x7.
Schedule ROTATION START DATE PRIMARY SECONDARY TERTIARY OTHER 2009 May Crowley Wirley Barden June Wirley Barden Myers July Barden Myers Fredericksen August Myers Fredericksen Crowley September Fredericksen Crowley Wirley October Crowley Wirley Barden November Wirley Barden Myers December Barden Myers Fredericksen 2010 January Myers Fredericksen Crowley February Fredericksen Crowley Wirley March Crowley Wirley Barden April Wirley Barden Myers
Personnel NAME CALL FIRST CALL SECOND OTHER AVAIL. OTHER AVAIL. Barden 275.5458 cell - 317.3398 home - 627.1602 cottage - 315.536.6634 Crowley 275.8235 cell - 733.1365 pager - 220.3330 home - 924.3273 Fredericksen 273.1714 cell - 313.4003 home - 586.5986 Myers 273.1804 cell - 208.0939 home - 349.7211 Wirley 275.5615 cell - 638.2591 home - 671.9046
Incident Management & Communication
11
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
3
Incident Manager On--Call (IMOC)
Evaluates the situation and gathers all the facts from MOC.
Notifies CIO and Directors (5,7). Initiates ISD Phone Bridge, if ISD systems are involved
• 1-866-945-2255 Access Code: 608965# Initiates IMOC Phone Bridge, if necessary
• 585-273-3311 Access Code 144357 or • 1-866-871-2663 Access Code 144357
Call Information Security MOC (DCS Operations [275-1205] can provide contact number) to review situation and determine if there has been a breach [SKIP this step if it is clear that the event is NOT security related; see next page for detail]. Information Security Office will make one of three decisions (see item 6 for details):
1. Security Controlled 2. Security Related 3. No Security Impact
Engages Communications Manager and Scribe (8,9). Contact DCS Production Control MOC to review impact of
incident with scheduled production jobs. Internal communication should reflect potential impacts.
Contact SMS group to setup Service Monitoring (Uptime) if necessary
Notifies University IT Computer Store/Sales if Blackboard, Flex, or the Secure 1 server (front-end of CS online store) is down (10). Otherwise, CSS can be notified as part of University IT Org.
Provides regular updates to the CIO office. Scheduled IMOC (not acting IMOC) schedules and leads
post-mortem/debrief session within one week of incident.
Evaluates the situation and gathers all the facts from MOC.
Notifies CIO and Directors for after hour incidents. Initiates ISD Phone Bridge, if ISD systems are
involved
• 1-866-945-2255 Access Code: 608965# Initiates IMOC Phone Bridge, if necessary
• 585-273-3311 Access Code 144357 or • 1-866-871-2663 Access Code 144357
Calls in Information Security MOC to review situation and determine if there has been a breach [SKIP this step if it is clear that the event is NOT security related; see next page for detail]. Information Security Office will make one of three decisions (see item 6 for details): Security Controlled, Security Related, or No Security Impact
Coordinates CHRON and scribe duties. Calls in staff for communications and scribe duties if needed.
Contact DCS Production Control MOC (DCS Operations [275-1205] can provide contact number) to review impact of incident with scheduled production jobs. Internal communication should reflect potential impacts.
Contact SMS group to setup Service Monitoring (Uptime) if necessary
Communicates with key people & customers during event.
Prepares a communication for release to University IT-ORG and external groups in early AM next business day. Communication should provide a brief of the situation, what the solution is, and if the event is still ongoing. Use IT Alert (www.g2alert.com).
Meets next morning with communications manager to discuss future communications and follow-up (if required).
Scheduled IMOC (not acting IMOC) schedules and leads post-mortem/debrief session within one week of incident.
Incident Management & Communication
12
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
4 University IT Technical Staff / Technicians on Call
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC. If MOC determines, technicians can forward internal calls for short periods of time.
Troubleshoots problem and begins working on solutions.
Retrieve Technical Recovery Guides (TRG’s) for services affected.
Provides regular updates to MOC. Participates in vendor calls as needed. Periodically checks in with other University IT staff
members to assess the situation – be sure to include members in other locations.
Avoid incoming customer calls. These are distractions to solving the issue at hand. If they are calling your phone, route them to the Call Centers (2).
Do not speak with internal (Currents/Campus Times) or external (D&C/TV stations) media. Direct them to University Communications.
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC.
Troubleshoots problem and begins working on solutions.
Retrieve Technical Recovery Guides (TRG’s) for services affected.
Provides regular updates to MOC. If off-site, calls into MOC Phone Bridge if needed (1-866-603-2932 Access #6608484)
Participates in vendor calls as needed. Periodically checks in with other University
IT staff members to assess the situation – be sure to include members in other locations.
Avoid incoming customers calls. These are distractions to solving the issue at hand. If they are calling your phone, route them to the Call Centers (2).
Do not speak with internal (Currents/Campus Times) or external (D&C/TV stations) media. Direct them to University Communications.
5 University IT Director - of affected unit(s)
Participates in discussions lead by MOC and IMOC. Provides support to technical teams. Provides any other support that may be needed to
help resolve the incident.
May be onsite or working from home as determined by MOC.
Participates in discussions lead by MOC. Provides support to technical teams. Provides any other support that may be
needed to help resolve the incident.
Incident Management & Communication
13
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
1. Security Controlled Situation is critical and may involve highly sensitive
data. Security Office takes control of incident
management and IMOC coordinates communications.
Engages University Legal and/or University HR (21,22).
Develops and distributes communications on a limited basis. Some events will require Security Office to keep all details confidential. Determines (if critical security situation) what information can be shared beyond the Security office.
If services are impacted, public communications will be determined by Security Office. If servers are down, notifies Operations Centers.
2. Security Related Reviews situation and gathers facts from
technicians. Participate in troubleshooting and helps to
implement solution. Begins a parallel communication stream as may be
required by specific incidents. 3. No Security Impact
Takes no action unless specifically asked to. Incident is NOT security related in any way.
May be onsite or working from home as determined by the type of security incident. 1. Security Controlled
Situation is critical and may involve highly sensitive data.
Security Office takes control of incident management and IMOC coordinates communications.
Engages University Legal and/or University HR (21,22).
Develops and distributes communications on a limited basis. Some events will require Security Office to keep all details confidential. Determines (if critical security situation) what information can be shared beyond the Security office.
If services are impacted, public communications will be determined by Security Office. If servers are down, notifies Operations Centers.
2. Security Related Reviews situation and gathers facts from
technicians. Participate in troubleshooting and helps to
implement solution. Begins a parallel communication stream as
may be required by specific incidents. 3. No Security Impact
Takes no action unless specifically asked to. Incident is NOT security related in any way.
6 Information Security
Security Controlled Examples:
• Missing person
• Crimes (domestic and international)
• Major security breach
Security Related Examples:
• Worm outbreak
• Virus problems
After incident debrief, IT Security will notify University Audit of major University IT incident. Notification will include cc: to Julie Buehler for Audit communication retention.
Incident Management & Communication
14
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
7 CIO’s Office
Receives details about incident from IMOC. Provides incident brief to Provost and President
(12,13). Provides business perspective (big picture) for the
incident.
Receives details about incident from IMOC.
Decides if the Provost and President should be notified before the start of the next business day.
Gathers with IMOC next business day morning to review event and provides business perspective (big picture) for the incident.
Incident Management & Communication
15
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
8 Communication Manager and/or Other Designated University IT Employees (Set up where main communication is taking place)
Gathers details about incident. Crafts messages for internal and external use. Identifies appropriate communication channels. Deploys communications according to incident
timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver the same message (2,10).
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retain copy of all communications for debrief session and for audit purposes.
Picks up the next business day to continue on-going communications (internal and external) or to assist in closing out the incident. If incident is closed:
Sends final communications when incident is closed.
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retain copy of all communications for debrief session and for audit purposes.
If incident is still open: Gathers details about incident and reviews
CHRON. Crafts messages for internal and external
use. Identifies appropriate communication
channels. Deploys communications according to
incident timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver the same message.
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retain copy of all communications for debrief session and for audit purposes.
Incident Management & Communication
16
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
9 Scribe (Set up where main communication is taking place)
Takes detailed notes during event to help complete the CHRON and serve as a record of the event.
Types up info in CHRON template and distributes to team at regular intervals during incident.
Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
Picks up in the AM of next business day. If incident is closed:
Types up info in CHRON template and distributes to team at regular intervals during incident.
Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
If incident is still open: Reviews CHRON already completed. Continues CHRON and takes detailed
notes during the event. Types up info in CHRON template and
distributes to team at regular intervals during incident.
Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
10 University IT Office Admins
Uses guidelines for communications to customers when responding to calls that may come in from various areas.
In the AM of next business day: Uses guidelines for communications to
customers when responding to calls that may come in from various areas.
11 University IT Staff Members
Uses guidelines for communications to customers when responding to calls that may come in from various areas.
In the AM of next business day: Uses guidelines for communications to
customers when responding to calls that may come in from various areas.
12 Provost
Receives regular updates from CIO. Disseminates info as needed to key staff members.
13 President
Receives regular updates from CIO. Disseminates info as needed to key staff members.
Incident Management & Communication
17
Management Steps Communication Flow
Normal Business Hours (8:00am – 5:00pm) After Hours
14 Other University Executives
15 Students
16 Faculty / Departments or Divisions
17 University Staff
18 University Security
Participates as required by incident. Participates as required by incident.
19 University Facilities
Participates as required by incident. Participates as required by incident.
20 Rochester Management
Participates as required by incident, specifically when related to the Towne House building. 461-9440 or 467-2442 after hours
Participates as required by incident, specifically when related to the Towne House building. 461-9440 or 467-2442 after hours
21 University Legal
Participates as required by incident, specifically when security related.
Participates as required by incident, specifically when security related.
22 University HR
Participates as required by incident, specifically when security related.
Participates as required by incident, specifically when security related.
Incident Management & Communication
18
Communication Call Log Last revised On: 7/15/07
Who to contact Notify? Yes/No
Contacted By 1st
Contact At:
2nd Contact
At:
3rd Contact
At:
4th Contact
At:
IT Center: Provide key facts so centers can handle incoming calls consistently and triage accordingly.
IT Center @ 5-2000
Ops @ 5-9194
(TH Computer Room)
University IT Incident Management: Provide key facts and begin IM team mobilization and communications.
University IT Directors
(Sev 3 VIP list)
CIOs office @ 5-5240
Norm Acunis
(for Email Sevs)
Becky Kingcaid
(for Email Sevs or any Sev affecting Executives in Wallis)
Information Security Office
(as needed)
Michelle Rogers
Bill Waterhouse
Main University IT Communication Channels: Provide high-level status of the event with updates as needed.
3-3999 Recording & Sev Page Sent
University IT Notices Updated (University IT website)
IT Center Plasma Screen
University IT Org
Phone Tree and/or G2 Alert
University IT Office Admins: Provide key facts so this team can handle incoming calls consistently and provide departmental support as needed.
CIO’s Office
Finance/Admin/Comm Office
AA Office
NC Office
DC Office
Security Office
External to University IT: Provide high-level status of the event with updates as needed.
Phonedown
Netdown
President’s Office @ 5-8356
Nicholas Bigelow @ 5-8549 (President of Faculty Senate)
Provost’s Office @ 5-5931
Incident Management & Communication
Who to contact Notify? Yes/No
Contacted By 1st
Contact At:
2nd Contact
At:
3rd Contact
At:
4th Contact
At:
All Campus Admins.
(for email Sevs)
ISD @ 5-3200
Highland Hospital Comm Ctr
@ 473-2200
Michele Cairns @ 1-8463
Med Ctr Director’s office
(Julie Choate, Roberta Parker)
Comm Ctr @ 5-2222
(Voice Services including VM)
College Dean’s Office
@ 3-5000
University Security Office
Highland Hospital Security
University Facilities Office
University Human Resources
University Legal
Students
Faculty
University Staff Members
University IT Notices Post
(ITENS)
Campus Times
Currents Digest (Email Daily)
Currents (Print)
Incident Management & Communication
Security Level Definitions Department: Information Security – Guiding Criteria Security Controlled (Sec. 3) Definition Information has the potential of being disclosed or altered that would:
1. Violate Laws, Regulations or Contractual Obligations 2. Significantly impact the reputation of the University
OR
A significant and growing number of SERVICES are rendered unavailable without any operational remedy.
Examples □ Server has been compromised that has
Student Social Security Numbers. □ Major worm outbreak is taking down
email, HRMS, etc. □ Main University Web Page significantly
defaced.
Security Related (Sec. 2) Definition Information has the potential of being disclosed or altered that would:
1. Cause Significant Harm to the University 2. Alter or disclose information regarding an individual
or group in an unauthorized manner 3. Alter the results of Research or Business Processes in
an unauthorized manner.
OR A significant and growing number of SYSTEMS are rendered unavailable without any operational remedy.
Examples □ Student Changes Grades. □ Researcher changes research data □ Worm outbreak is spreading rapidly
across ResNet.
Security Notified (Sec. 1) Definition Information that has been deemed non-critical has the potential of being altered or disclosed, without adverse impact to the University
OR A number of information systems are rendered unavailable without any operational remedy
Examples □ Known information is taken from a
system without any impact. □ Individual systems are hit with a
virus/worm. No trend across the University is detected.
Incident Management & Communication
Internal Communications Template (( II nn tt ee rr nn aa ll –– UU nn ii vv ee rr ss ii tt yy II TT SS tt aa ff ff OO nn ll yy ))
Communications Contact:
Release Date:
Incident:
Communication
Frequency: 1 Time Only Initial Comm + Multiple Updates
University IT – Internal
Audience: (check all that apply)
Who needs the information?
University IT – ALL Employees
- - - - - - - - - - - - - - - - - - CIO Directors University IT Managers University IT Office
Admins University IT Operations
Centers (IT Center/NCS Ops/DCS Ops)
Executive Support Team University IT Student
Workers (IT Center) N&C EC A&A S&P Computer Sales/Store University IT Finance &
Admin
Other
Channels: (check all that apply)
What’s the best
way to reach them?
Email Web Phone/Conf. Bridge ITENS/University IT
Home Page G2 Alert ext. 3-3999 In Person/Meeting
Other
University IT Hotline – for follow-up/summary
What information do they need?
Key Facts:
• Item 1
• Item 2
• Item 3
• Item 4
• Item 5
• Item 6
Incident Management & Communication
Page 22 Last Revised On: 4/17/06
Initial Communication Copy
Version 1:
Version 2:
Version 3:
Special Instructions/Notes:
Communication Channel
University IT Audience Assigned To Copy Version
Updates
Time Date Message Channels
Incident Management & Communication
External Communications Template (( EE xx tt ee rr nn aa ll –– UU nn ii vv ee rr ss ii tt yy CC oo mm mm uu nn ii tt yy aa nn dd PP rr ee ss ss ))
Communications Contact:
Release Date:
Incident:
Communication
Frequency: 1 Time Only Initial Communication + Additional Updates as Needed
External Audience:
(check all that apply)
Who needs the information?
Entire University Community
- - - - - - - - - - - - - - - - - - All Faculty (All Schools) Staff All Students (All Schools) Student Workers
(University IT) Residential Assistants (RAs) University Administration Department Administrators
Deans (All Schools)
Provost
President
VP of Communications
Medical Center/ISD
Medical Center/Staff Medical Center/
Communications Center
Highland Hospital Communications Center
Memorial Art Gallery
Telephone Directory Contacts
Key University IT Contacts
University Legal University Security Office
University Facilities
University Human Resources Dept.
Campus Times/Currents
Local Press/TV and Print
Other (Use this area for communications to specific Colleges)
Channels: (check all that apply)
What’s the best way to reach them?
Email Web Phone Currents Digest ITENS - University IT
Home Page G2 Alert Fax In Person – Visit various
locations IT Center Plasma Screen Flyers – post in relevant
areas
Other
Following Incident: Currents Print Campus Times Flyer/Postcard Follow-up Phone Call
Incident Management & Communication
What information do they need?
Key Facts:
• Item 1
• Item 2
• Item 3
• Item 4
• Item 5
Communication Copy
Version 1:
Version 2:
Version 3:
Special Instructions/Notes:
Communication Channel
Audience
(External to University IT)
Assigned To Copy Version
Updates
Time Date Message Channel(s)
Incident Management & Communication
University IT Technician Form: Incident Management & Communications
Time Alerted Alerted by Notified OPS Date Time 275-9194
275-9195 220-3283 pager
Time OPS Notified
Message of initial alert:
Vendor Case / Contact:
Systems affected Services affected
MOC Conference Call Bridge & Pin 1-866-603-2932 pin 6608484#
IMOC Conference Call Bridge & Pin 1-866-871-2663 or 273-3311 pin 144357#
MOC IMOC SysAdmin (s)
Time Event MOC Notification& Updates
Incident Management & Communication
26
External Communication Matrix
External To University IT
Who To Contact Who Can Contact (from University IT) Email Web Phone
CCoolllleeggee ooff AArrttss,, SScciieennccee,, aanndd EEnnggiinneeeerriinngg Deans
Vice Provost and Dean of the College Faculty Peter Lennie [email protected] 3-5000
Dean of The College Richard Feldman [email protected]
3-5001
Dean of the School of Engineering and Applied Sciences
Robert Clark [email protected] 5-4151
Vice Provost and Dean of Research and Graduate Studies
Wendi Heinzelman [email protected] 5-4153
Dean of Sophomores Vicki Roth [email protected]
5-9049
Dean of Freshmen Marcy Kraus [email protected]
5-2354
School of Engineering Computing and Networking Group (CNG)
John Simonson John Strong Jim Prescott Bob Lindholm
[email protected] 5-3106 5-4873 5-8265 5-0870
Department Heads
All Faculty
All Students
EEaassttmmaann SScchhooooll ooff MMuussiicc Dean/Director Doug Lowry [email protected] 263-2807
Computing Services Jeremy Beyette [email protected]
4-1160
SScchhooooll ooff MMeeddiicciinnee && DDeennttiissttrryy Dean David Guzick [email protected]
5-0017
Incident Management & Communication
27
External Communication Matrix
External To University IT
Who To Contact Who Can Contact (from University IT) Email Web Phone
SScchhooooll ooff NNuurrssiinngg Dean Kathy Parker [email protected] 5-8902
WWiilllliiaamm EE.. SSiimmoonn GGrraadduuaattee SScchhooooll ooff BBuussiinneessss AAddmmiinniissttrraattiioonn Dean Mark Zupan [email protected]
5-3316
Department of IT Joe Scacchetti [email protected] 3-5215
MMaarrggaarreett WWaarrnneerr GGrraadduuaattee SScchhooooll ooff EEdduuccaattiioonn aanndd HHuummaann DDeevveellooppmmeenntt Dean Raffaella Borasi [email protected] 5-8300
Warner School Information Technology Service Dave Garcia
RRiivveerr CCaammppuuss LLiibbrraarriieess Dean Susan Gibbons [email protected] 5-4461
Information Technologies Mike Bell [email protected]
5-6875
MMeeddiiccaall CCeenntteerr//SSttrroonngg HHeeaalltthh//HHiigghhllaanndd Information Systems Division (ISD) Jerry Powell [email protected] 784-6118
Communications Center (Strong)
Communications Center (Highland)
Security (Strong)
Security (Highland)
Facilities (Highland)
Incident Management & Communication
28
External Communication Matrix
External To University IT
Who To Contact Who Can Contact (from University IT) Email Web Phone
UUnniivveerrssiittyy AAddmmiinniissttrraattiioonn President Joel Seligman
Deputy to the President Lamar Murphy [email protected]
6-3262
Provost Ralph Kuncl
Provost Exec Assistant Melinda Smith [email protected]
5-5931
Assistant Provost Kathleen Moore [email protected]
5-2497
VP & General Secretary, Senior Advisor to the President, and University Dean
Paul J. Burgett [email protected]
3-2284
VP of Communications William Murphy 5-4124
Communications Administrator Maureen Baisch [email protected]
5-4127
Sr. VP of Finance & Administration/CFO Ronald J. Paprocki [email protected]
5-2800
Admin. Asst. Helen W. Kostizak [email protected]
5-2792
Sr. VP for Institutional Resources Douglas W. Phillips [email protected]
5-3311
Secretary Dianne Wittman [email protected]
5-8051
Sr. VP & Chief Advancement Officer James D. Thompson [email protected]
3-2158
Sr. VP & Vice Provost for Health Affairs and Medical Center CEO
Brad Berk [email protected]
5-3407
VP and General Counsel Sue S. Stewart [email protected]
3-5824
Incident Management & Communication
29
External Communication Matrix
External To University IT
Who To Contact Who Can Contact (from University IT) Email Web Phone
MMeemmoorriiaall AArrtt GGaalllleerryy The Mary W. and Donald R. Clark Director Grant Holcomb [email protected]
6-8902
LLaabboorraattoorryy ffoorr LLaasseerr EEnneerrggeettiiccss Director Robert McCrory [email protected]
5-4973
LLE Computer Support Alex Rysken [email protected]
5-5333
OOtthheerr UUnniivveerrssiittyy DDeeppaarrttmmeennttss Security Office
Facilities 3-4567
Human Resources [email protected]
5-8747
Office of Communications
Public Information Coordinator Sharon Dickman [email protected] 5-4128
Publicist Helene Snihur [email protected] 5-7800
Editor, Currents Jenny Leonard [email protected] 5-6076
Web Editor Lori Packer [email protected] 5-5277
OOtthheerr Telephone Directory Contacts
Key University IT Contacts
Residential Assistants
University IT Student Workers
Incident Management & Communication
30
External Communication Matrix
External To University IT
Who To Contact Who Can Contact (from University IT) Email Web Phone
University Health Services (Director)
International Services Office (Director) Cary Jensen [email protected]
5-8928
Office of Technology Transfer
Susan B. Anthony Center for Women's Leadership
Nora Bredes [email protected]
5-9283
University Intercessors Gerald Gladstein Frederick Jefferson Ruth Lawrence Kathy Sweetland
MMeeddiiaa ((IInntteerrnnaall ttoo UU ooff RR aanndd EExxtteerrnnaall)) Campus Times Various [email protected]
5-5342
Currents Digest Jenny Leonard [email protected]
5-6076
Currents (Print) Jenny Leonard [email protected]
5-6076
Local TV Stations Sharon Dickman [email protected]
5-4128
Local Newspapers Sharon Dickman [email protected]
5-4128
Local Radio Stations Sharon Dickman [email protected]
5-4128
Incident Management & Communication
31
Incident Command Center Wall Charts
Respond Time Action
Are Employees Safe? x13 Injured:
Contact Security (if necessary) x13
Security Contact:
Personnel On-Site: Contact Information:
Contact Facilities (if necessary) x3-4567
Contact Rochester Management (if necessary)
University IT Security Controlled Event? { Contact Information}
SECURITY CONTROLLED EVENT if either of the following exist: 3. Information has the potential of being disclosed or altered that would:
a. Violate Laws, Regulations or Contractual Obligations b. Significantly Impact the University’s Reputation
OR 4. A significant and growing number of SERVICES are rendered unavailable without any
operational remedy. Contact University IT Security immediately University IT SECURITY WILL COORDINATE RECOVERY ACTIVITIES/COMMUNICATIONS
Incident Management & Communication
32
SECURITY RELATED EVENT if either of the following exist: 1. Information has the potential of being disclosed or altered that would:
4. Cause Significant Harm to the University 5. Alter or disclose information regarding an individual or group in an unauthorized manner 6. Alter the results of Research or Business Processes in an unauthorized manner.
OR 2. A significant and growing number of SYSTEMS are rendered unavailable without any
operational remedy. Contact University IT Security IMOC will engage University IT Security to assist in recovery
University IT Security Controlled Event? { Contact Information}
SECURITY NOTIFICATION EVENT if either of the following exist: 1. Information that has been deemed non-critical has the potential of being altered or disclosed,
without adverse impact to the University OR
2. A number of information systems are rendered unavailable without any operational remedy CONTACT University IT SECURITY – NOTIFICATION ONLY
Severity 3 Declared
Declared By:
Incident Command Center Contact Information
Phone Numbers: Fax Numbers:
Help Desk Notifications IT Center x5-2000
Who
CIO Notification
Contact Dave Lewis – Must Make Verbal Contact; Cell 1st, Home Phone 2nd
Time Action
Incident Management & Communication
33
Control Time Action
Technicians On-Site?
ISD Comm Bridge Setup (if necessary) 1-866-945-2255 Access Code: 608965#
IMOC Comm Bridge Setup x33311 or 1-866-871-2663 144357#
Notate Time Sent in “Command Center Information”
MOC Comm Bridge Setup 1-866-609-2932 6608484
Notate Time Sent in “Command Center Information”
IT Alert Sent www.g2alert.com
Notate Time Sent in “Communication Updates”
University IT-ORG Email Sent Notate Time Sent in “Communication Updates”
University IT Notices Updated Notate Time Sent in “Communication Updates”
x3-3999 NCS Notification Notate Time Sent in “Communication Updates”
Customer Communications
Incident Management & Communication
34
Incident #1 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
Incident Management & Communication
35
Incident #2 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
Incident Management & Communication
36
Command Center Information
Location
Address: Fax #:
IMOC Conference Bridge
IMOC Communication Only
Phone #
Access #/Pin Code
MOC Conference Bridge
MOC/Technician Communication Only
Phone #
Access #/Pin Code
ISD Conference Bridge If necessary
Phone #
1-866-945-8855
Access #/Pin Code
608965
Incident Management & Communication
37
Personnel Name Contact Info Location Relief Person & Next Shift
IMOC
IMOC Communication
Assistant
Scribe
Communication Manager
MOC – AA
MOC – DC
MOC – ISO
MOC – NC
Incident Management & Communication
38
Communication Updates Vehicle Contact Info Performed By Last Update
IT Center 275-2000
Data Center Operators 275-9194 275-1205
IT Alert https://g2alert.com
University IT Organization Updated
1. University IT-ORG email list (if avail)
2. IT Alert 3. Phone Tree
University IT Notices Updated
Phone Update 273-3999
Incident Management & Communication
39
Vendor Contact Information University
IT Contact
Service/ Server
Company Contact Name Phone # Case #
Incident Management & Communication
Version 1.9 Confidential 40
IT Alert (G2Alert) – Steps to Send a Severity 3 IT Alert: 1. Gather information concerning the incident: Incident details, Service(s) Application(s) and Server(s) affected 2. https://www.g2alert.net & login 3. Choose Messages, then choose Send A Message 4. Choose “Create or Edit a Message” or select an existing Message from the pulldown list
If Creating a New Message Message Setup Time of Day
Business Hours After Hours Select Message Choose “Create A Message”
Create A Message Choose “Start with a Template”, and choose appropriate timeframe template Business Hours - Template After Hours - Template
Remember: You must fill in EACH method below for the message to reach recipients via that method (Text, SMS and Voice)
Message Name ***** Change the Message Name *****
Voice Messages
Email Messages Type as you would say it; you may
need spaces between letters Text Messages
Maximum 108 characters
Change {service/application/server/event} to reflect actual incident.
Change {service/application/server/event} to reflect actual incident.
Send Press Send – go to Sending Message below Press Send – go to Sending Message below
If Editing a Existing Message Message Setup Business Hours Time of Day After Hours Select Message Choose Edit or Copy
Remember: You must fill in EACH method below for the message to reach recipients via that method (Text, SMS and Voice)
Voice/Email/Text Messages Change message to reflect actual incident. Change message to reflect actual incident.
Send Press Send – go to Sending Message below Press Send – go to Sending Message below
Sending Message Message Setup Time of Day
Business Hours After Hours
List (CL) ALERT: UnivIT Only DIRs/MGRs (CL) ALERT: ISD Only MOCS & Bat Line
Request Confirmation Of a Receipt Always Choose “Yes”
Prompt Voice Message Recipients to Join A
Conference Call No Yes
Device Preferences Choose “Send to ALL” Choose “Send to Preferred Only”
Send Alert Always choose “Now” 5. Press “Continue” in the lower right hand corner. 6. After verifying the Send Message, choose “Send”. This will invoke the service to distribute the message.
Incident Management & Communication
Version 1.9 Confidential 41
ISD Manager On-Call - University IT (Data Center Services) Alert Notification University Data Center Services uses IT Alert, automatic notification software, that contacts specified individuals automatically, via cell phone, pager, home phone, e-mail, fax, or other, in the event of an emergency. The ISD Manager on Call will be contacted by IT Alert for any Severity 3 incident. At all hours, IT Alert will contact the ISD Manager on Call listed below. NOTE: The ISD Manager on Call will follow the ISD Incident Management Procedures to activate and contact ISD Management as applicable. Single system outages will be escalated through normal University IT escalation procedures. IT Alert will not be activated. The ISD Manager on Call will be contacted by each of their communication devices.
• Contact will be made in the order shown below, pager, cell phone, work phone, home phone, and e-mail.
• The pecking order will continue until all of your devices have been reached. • The IT Alert Notification contacts all devices; it does not stop if it reaches you by one of your
contact devices, even if you have confirmed receipt. ISD Manager On Call Schedule 2009 (see next page)
Incident Management & Communication
Version 1.9 Confidential 42
ISD Manager On Call Schedule 2009
Section 2. Contact and Communication Information
Start Time End Time Mgr Person
OnCall
Primary Secondary OnCall
Business Phone
Pager Or Cell Phone
Home Phone
6/16/09 7/6/09 Rick Haverty Primary 784‐6126 313‐0485 586‐6384
6/16/09 7/6/09 Dave Lindsey Secondary 784‐2949 314‐5665 315‐589‐8776
7/7/09 7/20/09 Dave Lindsey Primary 784‐2949 314‐5665 315‐589‐8776
7/7/09 7/20/09 Diane Koretz Secondary 341‐0403 734‐8976 315‐524‐7430
7/21/09 8/3/09 Chip Nimick Primary 784‐6115 415‐9053 671‐7570
7/21/09 8/3/09 Gary Scialdone Secondary 784‐2480 /275‐1120 350‐9588 787‐1639
8/4/09 8/17/09 Gary Scialdone Primary
784‐2480/275‐
1120 350‐9588 787‐1639
8/4/09 8/17/09 Nancy Bales Secondary 784‐8322 507‐6791 393‐1229
8/18/09 8/31/09 Nancy Bales Primary 784‐8322 507‐6791 393‐1229
8/18/09 8/31/09 Sue Graves Secondary 784‐2435 730‐2299/755‐
5395cell 335‐3276
9/1/09 9/14/09 Sue Graves Primary 784‐2435 730‐2299/755‐
5395cell 335‐3276
9/1/09 9/14/09 Ted Vaczy Secondary 784‐6002 576‐3651 624‐2792
9/15/09 10/5/09 Ted Vaczy Primary 784‐6002 576‐3651 624‐2792
9/15/09 10/5/09 Chip Nimick Secondary 784‐6115 415‐9053 671‐7570
10/6/09 10/19/09 Diane Koretz Primary 341‐0403 734‐8976 315‐524‐7430
10/6/09 10/19/09 Kathrin Kenny Secondary 784‐6121 474‐3569 315‐524‐4821
10/20/09 11/2/09 Kathrin Kenny Primary 784‐6121 474‐3569 315‐524‐4821
10/20/09 11/2/09 Tina DePalo Secondary 784‐8338 507‐9270 507‐9270
11/3/09 11/16/09 Tina DePalo Primary 784‐8338 507‐9270 507‐9270
11/3/09 11/16/09 Halle McNaney Secondary 784‐4275 245‐1884/880‐
1022 245‐1884/880‐1022
11/17/09 11/30/09 Halle McNaney Primary 784‐8275 245‐1884/880‐
1022 245‐1884/880‐1022
11/17/09 11/30/09 Tina DePalo Secondary 784‐8338 507‐9270 507‐9270
12/1/09 12/14/09 Tina DePalo Primary 784‐8338 507‐9270 507‐9270
12/1/09 12/14/09 Marty Bush Secondary 784‐8331 472‐4184 458‐3519
12/15/09 1/3/10 Marty Bush Primary 784‐8331 472‐4184 458‐3519
12/15/09 1/3/10 Dawn Robinson Secondary 784‐6159 820‐9274 383‐1213
Incident Management & Communication
Version 1.9 Confidential 43
Appendix
Incident Management & Communication
Version 1.9 Confidential 44
Roles & Responsibilities
Incident Manager On-Call (IMOC) The Incident Manager On-Call is a Director-level role and is responsible for managing University IT-wide incidents. The IMOC serves as a liaison to University executive offices and the University IT Managers On-Call during SEVERITY 3 incidents (defined below). They are on-call for one month, and are supported by a secondary and tertiary backup. The IMOC is available 24x7 during their monthly assignment. Definition of Severity 3:
The problem has a critical impact on key functions within the University or its reputation. Resolution takes highest precedence.
IMOC responsibilities:
Evaluate the situation and gathers all the facts from all Managers On-Call. Determine if the MOCs should be onsite during an incident that occurs outside normal business hours (8am-5pm weekdays), also known as “AFTER HOURS”. Oversee the Severity 3: Communication Checklist & Call Log process
Contact the CIO Work directly with MOCs & technical teams as necessary
Notifies University IT Information Security Office to review incident and determine if a security breach has occurred.
Serve as incident Communications Manager and oversee the gathering of information (CHRON) and customer communications. Determines the need/location of an Incident Command Center to manage the incident (also referred to as the “University IT War Room”)
Designate an incident scribe. In direct contact with the incident scribe and oversees all notifications to University IT ORG and if necessary, key University division contacts; President’s Office, Provost’s Office, Office of Communications, College Dean’s Office, URMC (School of Nursing), Simon School, Warner School and Eastman School of Music.
IMOC Schedule Changes If an IMOC is unavailable (sick, vacation, etc.), the IMOC is responsible for the following:
1. Notifying the secondary or tertiary IMOC to serve in their place 2. Notify University IT Production Control of the change in schedule
a. Use the “ITS Production Control” distribution list in the GAL b. Include start and stop dates and times for schedule modification
University IT Production Control will provide the IMOC update to the following:
1. SharePoint On-Call List https://sharepoint.its.rochester.edu/sites/DataCenter_OnCall/default.aspx 2. University IT Directors DL “IT Leadership” in the GAL 3. University IT Managers “ITS Managers” in the GAL 4. IT Centers [email protected] and/or 5-2000 5. Ida Gatto [email protected] and/or 5-9510
Incident Management & Communication
Version 1.9 Confidential 45
Manager On-Call (MOC) The Manager On-Call is a Manager-level role and is responsible for managing business unit level incidents. The MOC serves as a liaison for after hours notifications of the situations that are subject to off-hours resolution; receive calls from the after hours dispatch service, provide severity level review, triage/filter and dispatch staff as required. They are generally on-call for one week, and are supported by backup MOCs. The MOC is available 24x7 during their assignment. For severe service outages referred to as SEVERITY 3, the MOC is required to contact the Incident Manager On-Call (IMOC). Definition of Severity 3:
The problem has a critical impact on key functions within the University or its reputation. Resolution takes highest precedence.
MOC is responsible to:
Ensure that each call is reported [Chronology, HEAT or some other logging tool?] Only summary information needs to be recorded for all of the single user problems. Severity 2 and Severity 3 problems require communication as specified to ensure proper notification of service outages and also require logging basic chronology of events to report significant progress in solving problems. General Rule - State what you can do for the customer and not what you can’t do by positive negotiations. Offer your office phone number to the IT Center and the Operations Center number for inquiries by the customer on the next business day.
Update the University IT MOC list, and individual unit on-call schedules should be used to determine the appropriate triage and notification(s)
Certain service disruptions require contact with general dispatch points:
ISD Help Desk at x53200 can be your reference point for any ISD staff on call for desktop or Med Center department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is
your link to all trades-people in Facilities. Please note that if there are any issues concerning what the dispatchers at x34567 ask you, you may ask them to “patch” you through to their Supervisor.
Communications Center at x52222 is your link to all Med Center On-Call people (with the exception of
ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife Staff may be utilized to refer issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the
Administrator On-Call via the Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult with Security, x13 or 275-3333, immediately.
Incident Management & Communication
Version 1.9 Confidential 46
Communications Manager The Communications Manager is responsible for managing University IT-wide and University-wide communications for University IT-wide incidents. [This is a role served during an incident and not a job title.] He/She serves as a communications liaison to the IMOC during SEVERITY 3 incidents (defined below). The IMOC may choose to serve in this role if another suitable employee is not identified. The Communications Manger must review all communications with the IMOC before releasing them, unless otherwise stated by the IMOC. In some cases, the CIO (or Assistant CIO, Other Directors) may require that all communications get reviewed by the CIO’s Office prior to deployment. Communications Manager needs to compose and deploy updates during the course of the incident. The Communications Manager should also send out a final message indicating the incident is closed and offering a status report to affected users. [See sample text at the end of this document.] In some cases, the Communications Manager will need to provide details and in other cases, it will be necessary to remain vague. IMOC and CIO will provide guidance on this. Be sure to communicate with Becky Kingcaid/Alivin Ruiz if it is an issue that affects Wallis Hall. Becky will often re-tool general messages based on the needs of users in Wallis hall. It is a good idea to send her copy before releasing to the general public so she has a heads up. Refer to the templates and checklists provided in the Incident Management Handbook for details on communication channels, etc. Definition of Severity 3: The problem has a critical impact on key functions within the University or its reputation. Resolution takes highest precedence. Communications Manager Responsibilities:
Gathers details about incident. Crafts messages for internal (University IT Only) and external (University-Wide) use. Works with Office of Communications if communication outside of the University is required. Depending on the
situation, the Communications Manager may or may not be asked to speak to the press. But should never do so unless given instructions to.
Identifies appropriate communication channels. Deploys communications according to incident timeframe through identified channels/Working with MOC and
IMOC. [All Channels] Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can
handle calls appropriately and deliver a consistent message. Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retains copy of all communications for debrief session and for audit purposes.
If an incident occurs after normal business hours: The Communications Manager picks up the next business day to continue ongoing communications (internal and external) or to assist in closing out the incident.
If incident is closed:
Sends final communications i Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retains copy of all communications for debrief session and for audit purposes.
Incident Management & Communication
Version 1.9 Confidential 47
If incident is still open:
Gathers details about incident and reviews CHRON. Crafts messages for internal and external use. Identifies appropriate communication channels. Deploys communications according to incident timeframe through identified channels/Working with MOC and
IMOC. [All Channels] Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can
handle calls appropriately and deliver the same message. Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retains copy of all communications for debrief session and for audit purposes.
Certain service disruptions require contact with general dispatch points (this is usually done by the IMOC - but you may be asked to continue to provide them with updates during the course of the incident):
ISD Help Desk at x53200 can be your reference point for any ISD staff on call for desktop or Med Center department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is
your link to all trades-people in Facilities. Please note that if there are any issues concerning what the dispatchers at x34567 ask you, you may ask them to “patch” you through to their Supervisor.
Communications Center at x52222 is your link to all Med Center On-Call people (with the exception of
ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife Staff may be utilized to refer issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the
Administrator On-Call via the Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult with Security, x13 or 275-3333, immediately.
Incident Management & Communication
Version 1.9 Confidential 48
Sample Communication Copy General Pointers:
Always include a heading/subject line – even if email isn’t used. Helps people get their bearings. Be careful not to over promise on a solution or quick outcome. Provide estimates when possible. Indicate where people can go for additional information. Use “Contact University IT” in most cases – with whatever
number is appropriate for the incident. Don’t provide too much technical information. Speak in terms the average end user will understand. Tell users what to expect. Keep track of all communications in a Word document and add the time the communication was sent out. Provide updates after major attempts to solve the problem, such as server reboots, hardware swaps, etc.
Sample INITIAL Messages: Exchange Email Disruption Between 8:30am and 10:00am today, some University faculty and staff experienced disruptions with email service. These disruptions were confined to a subset of Exchange email users. University IT support teams have isolated and resolved the issue. We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout the day. If you have additional questions or begin to experience problems with your email, please contact University IT at 5-2000. UNIX Email Disruption University IT Support teams are currently investigating issues that may be resulting in delayed email delivery. We apologize for the intermittent slowdowns you may be experiencing with email services. University IT teams are working diligently to address the issue as quickly as possible. As a precaution, our hardware vendors have been called in to assist with the investigation and we will be working with the vendors to identify actions to minimize this type of disruption in the future. We expect to provide additional information by 5:00 this afternoon (1/19). If you have additional questions, please contact University IT at 5-2000. Sample SUBSEQUENT Messages: UNIX Email Disruption – 6:15PM Update We are still experiencing intermittent email issues on the mail.rochester.edu mail server and we will be rebooting the server at 6:20 p.m. this evening. Mail services will be unavailable for approximately 20 minutes. We appreciate your patience as we continue to work on resolving this issue. Please continue to check back for regular updates. You can also call the University IT at 5-2000 or 3-3999 (recorded message).
Incident Management & Communication
Version 1.9 Confidential 49
UNIX Email Disruption – 7:15PM Update We are still experiencing email issues on the mail.rochester.edu server following the reboot performed at 6:20 p.m. As we work with our vendors to diagnose the problem, you may continue to experience intermittent availability of email. Please continue to check back for regular updates. You can also call the University IT 5-2000 or 3-3999(recorded message). UNIX Email Disruption – 9:00PM Update Faculty and students may still be experiencing intermittent disruptions with email service. Users experiencing these problems are primarily within the College. We will continue to work with vendors to isolate the source of slow email service. At this point, we are progressing through a detailed plan. We apologize for the inconvenience; we recognize the impact that this has on you and are working to remedy the remaining issues. UNIX Email Disruption – 8:00AM Update University faculty, students and staff who were experiencing disruptions with email on Thursday, January 19 can now log into their email. You may experience a delay with your initial log in if you have a large quantity of unchecked messages in your mailbox. University IT staff resolved some service disruptions and is maintaining a continuous effort to address the issue. University email services will be monitored throughout the day. Please contact Information Technology Services at 5-2000 if you need assistance. Generic NetID template to be used when LDAP is disrupted. We are experiencing a service disruption with the University’s LDAP service. This means that applications requiring a NetID for authentication are currently unavailable. IT support teams have identified what must be done to resolve the issue OR IT support teams are working to identify the cause of this disruption. [if the reason is known and can be shared in terms the users will understand, add a brief statement here] We apologize for the inconvenience and we expect to have the problem resolved by [enter info here]. We will provide additional updates as they are available [or enter a specific time(s)]. Please contact the IT Center at 275-2000 if you have additional questions. Sometimes, we think we have fixed a problem and it comes back (or was never really fixed to begin with). Here’s an example of how to handle that. First Message – We have received new information that some University faculty, students, and staff members are still experiencing intermittent email issues. We apologize for the slowdowns you have been experiencing the past few days. We recognize the importance of email service and that this disruption has happened at an inopportune time. We are working diligently to restore full email services. Please contact University IT at 5-2000 if you need assistance. Status information is also posted on the IT Notices found at www.rochester.edu/its/. Second Message – Improvements to the email environment continue. We recognize the importance of email services and Information Technology Services continues to work diligently to restore full email services. Please contact Information Technology Services at 5-2000 if you need assistance. Status information will continue to be posted on the IT Notices found at www.rochester.edu/its/.
Incident Management & Communication
Version 1.9 Confidential 50
Sample CLOSED Incident Message: [It is important to send out a final communication to let users know that all has been restored and to offer an explanation of what to expect.] Exchange Email Disruption On Friday, May 5 at 8:25 a.m., University IT became aware of an issue with one of the Exchange 2003 email servers that resulted in a brief email outage for a subset of Exchange email users. University IT support teams isolated and resolved the issue and had email restored by 10:00 a.m. During this time, emails were held in the queue and delivered when email services were brought back online. Please be assured that no emails were lost during this event. We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout the day. If you have additional questions please contact University IT at 5-2000. In some cases, it may be necessary to provide information about an incident to people NOT directly affected. An example communication is provided below. Message for IT Key Contacts (Includes IT people outside of University IT – was sent to help other IT support users who were affected by the outage) Dear IT Colleagues, On Thursday, January 19, University faculty, students and staff started to experience intermittent disruptions with UNIX email service on the mail.rochester.edu server. Users experiencing the problems were primarily within the College. University IT worked with our vendors to isolate the source of slow email service. Users who were experiencing disruptions with email on Thursday can now log into their email. They may experience a delay with initial log in if their mailbox contains a large quantity of unchecked messages. Processing capacity was added to help move mail through the various checkpoints (anti-spam/anti-virus). University IT staff resolved some service disruptions and is maintaining a continuous effort to address the issue and University email services will be monitored throughout the day. Regular updates will be posted online at: www.rochester.edu/its/ - IT NOTICES. Please use this information to keep your area up to date with this issue. University IT uses this area to communicate with the University community on a regular basis and will be a source of information for you on this issue, regular updates on maintenance outages, and other University IT services. If you have additional questions, please contact University IT at 5-2000.
Incident Management & Communication
Version 1.9 Confidential 51
Web Content Hack – Immediate Actions On receipt of WebWatcher or other notification of a hack
1. Go to the page reported and see what has happened 2. If there does not appear to be anything different check with the owner of the file.
3. If confirmed hack begin notification of …? How should we start the escalation?
4. Do not delete or move any of the hacked files until the evidence is reviewed
5. Is this a OS hack or a content hack? If content hack continue
(We should have a procedure for assessing an OS hack)
6. Get the modified date and time of the hacked file
7. Using that time minus one hour find all files that have been modified You are looking for hack tools and any additional hacked pages. If nothing turns up use minus two hours etc.
8. Review these files for hack tools
9. Review log files for the hacked file access record and note the IP number
10. If more than one file is hacked find those in the log and capture the IP number
11. Preserve copies of the hacked files
12. Redeploy or restore the hacked file
13. Identify the ISP of the hacker and their entire IP range
14. Block that range at the router.
15. File an abuse report with the ISP of the hacker
16. File an incident report with Campus Safety
17. Evaluate the methods used & determine what actions can be taken to prevent a repeat.
Incident Management & Communication
Version 1.9 Confidential 52
Debrief Procedures
1. Debrief will be scheduled by the scheduled IMOC, not the acting IMOC. Meaning, if a scheduled IMOC is unavailable to be IMOC and an acting IMOC is leading the incident, the scheduled IMOC will be responsible for scheduling and leading the incident debrief. If circumstances prevent the scheduled IMOC from assigned duties, the acting IMOC will be responsible.
2. Debrief should occur no later than one week after the incident, with one day after the incident preferred while
information and events are fresh in participants’ minds.
3. Debrief documentation is to be stored in the Incident Management and Communication SharePoint site, located: https://sharepoint.its.rochester.edu/sites/ITS-IMC/Shared%20Documents/Forms/AllItems.aspx under the Incident Reports and Debriefs.
4. After documentation is complete, send an email to Bill Waterhouse. He will in turn produce a UR Audit
update to be sent to the University’s Audit department and Julie Buehler.
Incident Management & Communication
Version 1.9 Confidential 53
Debrief Agenda Template Event Date: Event Time: Event Description: Attendees: Debrief Facilitator: Debrief Date: Item 1. Notification
Was everyone notified in a timely manner? What would have made it better?
2. Turnout Was everyone there who needed to be there? What other personnel would have helped?
3. Communications Did we communicate to each other well? Did we communicate to customers well? How can we improve the process?
4. Personnel Did we have the correct personnel on-site throughout the incident? Was the personnel rotation correct?
5. Equipment Were the room(s) equipped with the correct items to support the incident? What other equipment would have helped?
6. Intra-Departmental Cooperation
Did the University IT business units work together in the best manner?
7. Inter-Departmental Cooperation
Did University IT work together with other University departments in the best manner?
8. Initial Strategy Did we use the best strategy to minimize incident timeframe? What strategies would have improved turnaround?
9. Execution Did we execute the strategy in the best manner? What could we have done better to improve turnaround?
10. Clean Up Was incident closed so everyone knew to step down from a Severity 3? Was chronology published in a timely manner?
11. Customer Impact What feedback did we receive from customers?
12. Follow Up Items What open items still need attention?
13. Lessons Learned / Recommendations
What did we learn? What would make incident response and communication better?
14. Audit Notification IT Security will provide incident notification to University Audit
Incident Management & Communication
Version 1.9 Confidential 54
Updating Procedures The following procedures manual was initially developed by the Incident Management & Communications team between February and June 2006. If you have any questions, concerns, or modifications to the following procedures, please contact the IT Center (275-2000 [email protected]) The following people had a major role in the creation of this document:
Project Sponsors Kate Crowley
Network & Communications
Project Manager Bill Waterhouse Security & Policy
Project Participants Norm Acunis Mike Fitch Karen McVige Joe Pasquarelli Jay Riley Mercedes Fredericksen Jason Wagner
Network & Communications Network & Communications Data Center – Production Control Academic Technology Applications & Architecture Office of the CIO – Communications Academic Technology – Emergency Preparedness
Others assisted with its creation, and Information Technology Services is thankful for the participation and guidance to better serve our customers.
Incident Management & Communication
Version 1.9 Confidential 55
Change Control
Name Person Section change Description change Date Version Number
Bill Waterhouse IT Alert Modified procedures to follow G2Alert alert custom list modifications 12/08/2006 1.0
Bill Waterhouse
Communication Checklist – Section 6 Debrief Document
Add University IT Security to notify Audit of major University IT incident 12/13/2006 1.1
Bill Waterhouse ISD On-Call Update Updated ISD On-Call schedule for 2007 1/3/2007 1.2
Bill Waterhouse IM&C Quarterly Update – Q1 2007
1. ISD Conference Call # in IMOC checklist 2. Service Monitoring query in IMOC
checklist 3. NCS MOC to forward Operations phone
numbers if TH evacuation 4. Debrief is required, and scheduled IMOC
will schedule (not acting IMOC) 5. Add Services Monitoring (Uptime) to IMOC
checklist
1/9/2007 1.3
B.J. Block IM&C Quarterly Update – Q3 2007
1. Changed name from ITS to University IT 2. Updated Information Security and Policy
Director to Bill Waterhouse 3. Updated contact information for Bill
Waterhouse 4. Updated IMOC schedule through
beginning of 2008 5. Changed debrief documentation to state
that the debrief should be sent to Bill Waterhouse and he will send to audit
6. Updated email distribution lists to new naming convention
7. General editing updates
7/15/07 1.4
Bill Waterhouse
Appendix IMOC Schedule ‘08
Updated appendix to include Web Content Hack Procedures Updated 2008 IMOC schedule
12/10/07 1.6
Bill Waterhouse IM&C Quarterly Update – Q4 2007
1. Updated 2008 IMOC schedule 2. Updated 2008 ISD schedule 3. Recovered roles deleted from version 1.6 4. Updated MOC role to include University IT
MOC decision point
02/01/08 1.7
Bill Waterhouse IMOC Schedule IT Alert (G2Alert) ISD Manager On Call
1. Updated University IT IMOC Schedule 2. ISD notified during any Severity 3 alert 3. Updated ISD IMOC information
5/29/09 1.8
Bill Waterhouse Contact Information Bridge Phone #
1. Updated all internal & external contact information
2. Added 3-3311 bridge # throughout doc 6/17/09 1.9