Disaster Recovery Planning

DISASTER RECOVERY PL ANNING FOR HEALTHCARE

Ron Pelletier, CISSP, CBCP, CISA, QSAVP and Executive Manager

FOR HEALTHCARE ORGANIZATIONS

DISASTER RECOVERY PLANNING

INMGA – August 11, 2015

DISASTER RECOVERY PL ANNING FOR HEALTHCAREPONDURANCE 2

AGENDA

• Disaster Recovery Planning Overview• General DRP Considerations• A BCM Methodology (encompasses DRP)• Trends and Standards• Questions


DISASTER RECOVERY PLANNING OVERVIEW


SIMPLIFIED DRP TERMS

Disaster Recovery – Planning to sustain supporting technology & data.

Crisis Management – Preserving life safety and business image.

Business Impact Analysis – Establish the organization’s critical path.

Recovery Time Objective – When do the systems/processes need to be restored?

Recovery Point Objective – How much data can you stand to lose?

Maximum Tolerable Downtime – What is the point of unacceptable risk?

Risk Tolerance – Collective picture of risk management and BCM.

High Availability – When downtime of systems/data is not an option.

Minimum Operating Requirements – What do you need, and when, to get by.


WHERE DOES DRP FIT IN THE BCM LIFECYCLE?

BCM

Business Continuity Planning

Disaster Recovery Planning

High Availability

Risk Management

Incident Response

Crisis Management

(general, not all inclusive)


TRADITIONAL THINKING ON DISASTER RECOVERY

Disaster Recovery vs. Business Continuity

PEOPLE BUSINESS PROCESSES PROCESS

CONTINUITY

BUSINESS PROCESSES

DRPDRPDRPDisaster Recovery

Business Continuity

TECH/DATARESTORE

BUSINESS

CONTINUITY

BUSIENSS

CONTINUITY


THE INTEGRATED PERSPECTIVE

Defined Tolerance for Risk

Program Exercising, Change Management, Maintenance

(BCP)Business Continuity Planning

(DRP)Disaster Recovery Planning

DRP Strategies

BCPStrategies

DRP Documentation

BCP Documentation

The Risk Analysis Phase

Current State

Assessment

Threat and Risk

Assessment

Business Impact

Analysis

CRISIS MANAGEMENT• Owns Initial and Ongoing Response• Allocates Emergency Resources• MAKES DECISIONS AS REQUIRED• Functions as Steering Committee


GENERAL DRP CONSIDERATIONS


EVALUATE YOUR OPERATING RISK

• PREPARE TO AVOID BUT PLAN TO RESPOND!!

• Human, technical, operational and strategic threats MUST be considered to formulate a viable avoidance and/or response posture

• Look for single points of failure that might not have been considered (control systems, joined power junctions, shared data closets, shared passwords, single communication gateway)

• Consider your level of reliance on other entities (parent organizations, shared services, external service providers, etc.)

• Integrate your risk assessment process with Cyber Security efforts. According to KPMG’s BCM Survey Only 41% of Companies integrate BCM with Cyber Security

• Do you have specific technologies at your site that are not typically supported by a shared services organization?

• Do you have a defined owner or custodian for your Disaster Recovery Planning efforts?


DEFINING RISK TOLERANCE FOR DRP

$ and Operational Impacts

Manual ProcessingApplication ‘X’ in 72 Hours Application ‘X’ in24 Hours

Management Negotiation Based on Risk Tolerance

Recovery Time Objectives (RTO’s)Recovery Point Objectives (RPOs)

Current Recovery Capabilities (CRC’s)

Information Technology Group

Current State Assessment

Maximum Tolerable Downtimes(MTD’s)

Business Unit Personnel

Business Impact Analysis


RISK TOLERANCE AND HEALTHCARE

• Two schools of thought can muddy the water when considering technology downtime in healthcare:o “We have been treating patients for centuries without

technology…we can live without it indefinitely”o “We have grown so dependent on technology that we

cannot be inconvenienced by its loss for even a single hour”

• How do you appropriately consider the risk to your organization, without trying to “over-engineer” a solution?

• What happens if technology platforms are down for extended periods of time?


RISK TOLERANCE AND HEALTHCARE• Consider looking at 2 key criteria to arrive at true business impact:

1. Degradation of Care (Life Safety): The degradation of care considers specific risks of patient-safety, if care professionals do not have access to all patient records that may provide insight into patient profiles (e.g., pharmaceuticals, allergies, past procedures, etc).

2. Patient Throughput (Financial and Operational): “Throughput” represents the number of patients that can be reasonably and safely treated over a given period of time. Without a level of automation and record accessibility, it’s logical to assume that hospitals will not be able to attend to, admit, or discharge the “normal” volume of patients with the same level of efficiency. This can lead to direct financial impacts, as it would likely lead to a reduced and untimely level of billing for patient care.

• Use qualitative measures where they make sense, but attempt to arrive at a Recovery Time Objective for each key system (HIPAA denotes this as “addressable”)


DEVELOPING DRP ROLES/RESPONSIBILITIES• Consider a 360 degree approach to ensure appropriate organizational

coverage

• Look outside the organization to determine if there are groups/entities with whom you need to coordinate your strategies and plans

• If you are part of a hospital system, have you integrated with their Hospital Command?

• If you have personal that you contract to facilities, do you know what their plans are if their facilities are impacted?

• Break down the roles and corresponding plans to facilitate action and accountability

• How do you define an incident commander?

• What about facilities? Specific technologies?


DEVELOPING DRP ROLES/RESPONSIBILITIES(SAMPLE ORG CHART FOR DRP)

IT Incident Commander

TBD at Time of Incident

IT Customer Support Center Lead

IT Hospital Command Liaison

IT Safety/Security/Privacy

Officer

IT Command Group IT Operations Support Group

Infrastructure Team Leader

Applications Team Leader

Facilities Director

Logistics & Vendor Support

Finance/Administration

Hospital Command

IT Facility and Technical Teams

IT Facility Coordinators

Applications Teams

Infrastructure Teams

Data Recovery Teams IT Security

IT Executive(s)


DRP SCENARIO PLANNING

• Ensure your response procedures have adequate flexibility to respond to both common and unique situations

• Do not get too specific or box your plans to a certain scenario, use situations that may prompt a certain response

• Align your response planning with the applicable Hospital Command (if that is applicable)

• Remember that disasters related to technology could take on physical form, logical form, or a combination of the 2

• While area-wide disasters are less likely to occur, they need to be at least considered (think Hurricanes Sandy, Katrina; Northeast power outage; ice storms, etc.)


DRP SCENARIO PLANNING - EXAMPLE


REVIEW YOUR DATA BACKUP AND RECOVERY

• Ensure the data backup scheme complements the Recovery Time and Recovery Point Objectives (RTOs & RPOs)

• Tapes are fine, but often they are either not removed from the site or are taken offsite 1x per week

• If the backups (tape or disk) are not tested periodically to verify full restoration, the capability to restore is questionable

• If the backup tapes are not encrypted when removed offsite, you are introducing a whole new set of risk

• Don’t blindly jump to a high availability strategy if it is not justified. It is entirely possible that even a replication strategy is not necessary, and a high availability strategy may completely over-engineer the program

• BUT…only proper analysis can provide that answer


DRP DOCUMENTATION CONSIDERATIONS

• Consider segmented action plan documents that are managed by accountable person(s), but provide seamless integration and consistency in format

• Have a Central Plan to drive communications and Emergency Response

• Have “extracts” or job action sheets that represent specific technical procedures for rebuild, restore, recover, etc.

• Assign accountability as appropriate, and add depth to preserve continuity

• Ensure the procedures are fairly thorough, but do not drive inflexibility or box the responders into a single set of actions

• Store the plans where they are accessible, particularly if your internal systems fail

• Ensure the plan appendices have adequate reference information (key vendors and contacts, location of stored equipment, etc.)


DRP DOCUMENTATION CONSIDERATIONS


A FULL BCM METHODOLOGY(WITH DRP CONSIDERATIONS)


6 DOMAINS TO CONSIDER FOR BUILDING BCMAssess the entity controls that integrate, manage, and sustain a viable BCM throughout the enterprise

1. Program Management

• Program Definition – Establish the program is formally developed and integrated

• Support and Accountability – Establish the program is supported at the highest level of the org

• Budget Planning and Program Evaluation – The org is committed to sustaining program viability

The organization has defined its recovery, restoration, and high availability requirements related to business processes, applications, infrastructure & data

2. Requirements Definition

• Risk Analysis and Treatment – Establish the org has analyzed its risk posture, reasonably reduce risk• The BIA Methodology – Establish the org maintains formal method to define impacts, prioritize criticality• Data Flows and Dependencies – Establish that dependencies (internal/external) are documented• Analysis and Reporting – Establish that BIA results are aggregated, prioritized, and approved

Assess the organization’s method for developing continuity and availability strategies, within its maximum tolerable downtime.3. Strategy Selection

• Staff and Support Requirements – Establish that strategies are developed based on defined requirements• Course of Action Analysis – Establish that the cost to maintain strategies in line with risk tolerance• Monitor, Evaluate for Change – Establish that strategies are periodically evaluated for change


6 DOMAINS TO CONSIDER FOR AUDITAssess the sufficiency, completeness, applicability, and implementation of the organization’s documented BCP/DRP plans.4. Plan Development

• Plan Components & Framework – Establish plans are documented, align with requirements• Supporting, Storing Plans – Establish plans are accessible, assigned to process owners• Plan Updates – Establish plans change as processes, technologies, people change

Assess the organization’s method for vendor selection and oversight relevant to the BCM program.

5. Vendor Management

• Vendor Contracting – Establish vendors are screened, will meet contractual requirements• Critical Vendor Dependencies – Establish critical dependencies are known, accounted for• Vendor Integration, Testing – Establish vendors occasionally participate in tests/exercises

Assess the organization’s capability to test and maintain the viability of its BCM program.

6. Implementation, Maintenance

• Testing and Validation – Establish plans are valid through scheduled, ongoing testing• Change Management – Establish changes required to BCM are formalized• Workforce Awareness – Establish workforce members are aware of the BCM program


CONSIDER A MATURITY MODEL APPROACHAs of: SEPTEMBER 2012

Client:

Affiliate:

Maturity Rating Not Addressed Minimally

Addressed Emerging Managed

1 41% 0 0 5 71.1 25% 0 0 1 2

1.2 45% 0 0 2 3

1.3 54% 0 0 2 2

2 46% 0 2 10 42.1 25% 0 1 3 0

2.2 59% 0 0 0 4

2.3 25% 0 1 3 0

2.4 75% 0 0 4 0

3 61% 0 1 6 43.1 56% 0 0 3 3

3.2 47% 0 1 2 0

3.3 80% 0 0 1 1

4 38% 0 0 6 54.1 50% 0 0 4 2

4.2 40% 0 0 0 2

4.3 25% 0 0 2 1

5 30% 0 4 2 35.1 25% 0 0 1 2

5.2 40% 0 3 0 1

5.3 25% 0 1 1 0

6 67% 0 0 4 76.1 75% 0 0 1 3

6.2 50% 0 0 3 0

6.3 75% 0 0 0 4

47% 0 7 33 30

CLIENT NAME

SUB ORGANIZATION

QUANTIFIED BCM FINDINGS (# of findings per maturity level)

Vendor Contracting

Data Flows and Dependencies

Plan UpdatesSupporting and Storing the Plans

Program Definition

REQUIREMENTS DEFINITION

The BIA Methodology

Support and Accountability

Budget Planning and Program Evaluation

Risk Analysis and Treatment

Analysis and Reporting

STRATEGY SELECTION

Change ManagementWorkforce Awareness

Enterprise BCM Principles

Critical Vendor DependenciesVendor Integration and Testing

PLAN IMPLEMENTATION & MAINTENANCETesting and Validation

Scoring

PROGRAM MANAGEMENT

Staff and Support Requirements

VENDOR MANAGEMENT

Course of Action AnalysisMonitor and Evaluate for Change

PLAN DEVELOPMENTPlan Components and Framework

• Facilitates Scalable Program

• Isolates Highest Risk Areas

• Accounts for areas to sustain

• Incorporates All Findings from the Audit


DRP TRENDS & STANDARDS


EMERGING TRENDS IN DRP

• Virtualization helps reduce number of overall IT assets, improves system uptime…but beware of single points of failure!

• Cloud computing provides a viable outsourcing option for production technologies…but be sure your cloud vendor is capable of meeting your RTOs, RPOs!

• Mobile devices provide a means of portability for documented plans, communications, and rapid response…but be sure phones are secure, encrypt if possible!

• Social networking provides an effective way to broadcast incidents, particularly for crisis management…but be sure that the messages are controlled!


CURRENT AND EMERGING STANDARDS

• Business Continuity Institute - Good Practice Guideline (2010)

• BS 25999 Business Continuity – BSI’s practices guideline

• Disaster Recovery Institute (DRI) – Professional Practices for BCM

• ISO/IEC 22301:2012 – One of the newest, aligned with the ISO27k set of standards

DISASTER RECOVERY PL ANNING FOR HEALTHCARE

Ron Pelletier, CISSP, CBCP, CISA, QSAVP and Executive Manager

QUESTIONS

[email protected]

Pondurance3105 East 98th StreetSuite 120Indianapolis, IN 46280