Course Materials DISASTER RECOVERY AND BUSINESS CONTINUITY Tom Williams Centurion Business Continuity Strategy Manager Jack Henry & Associates Northville, Michigan [email protected] 800-299-4411 August 11 & 12, 2016
Course Materials
DISASTER RECOVERY AND BUSINESS CONTINUITY
Tom Williams Centurion Business Continuity Strategy Manager
Jack Henry & Associates Northville, Michigan
[email protected] 800-299-4411
August 11 & 12, 2016
1
Tom Williams - Centurion Business Continuity Strategy Manager
I/T Disaster Recovery/Incident ResponsePresented by
The Graduate School of Banking andCenturion Disaster Recovery Services –A Division of Jack Henry & Associates
Agenda
• Centurion Disaster Recovery Overview
• Financial Industry Risk Environment
• The FFIEC Business Continuity Guidelines
• Business Continuity / Disaster Recovery Plan Components
• Business Continuity Plan Assessment
• Recovery Timeline
• Recovery Strategies
• Testing the Plan using the Mock Disaster Drill Model
• Q & A
2
CENTURION DISASTER RECOVERY SERVICES
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
3
Who is Centurion
• Our History– Originally provided DR services to customers in 1992
under Bank Business Recovery Services (BBRS).– Re-Branded as Centurion Disaster Recovery in 1998.– Added Business Recovery Consulting to portfolio in
2000.– JHA Has Offered Some Form of Disaster Recovery
since the Mid 80’s.
3
Centurion Recovery Centers
San Diego, CA
St. Paul, MN
Lombard, ILAngola, IN
Windsor, CT
Charlotte, NCLenexa, KS
Monett, MD
Birmingham, ALAllen, TX
Who is Centurion
• Our Customers
– We support over 1,200 Financial Institutions for DR support.
• Smallest - $8 Million in Assets - Largest - $15 Billion in Assets
• 700+ Disaster Recovery tests annually
• Dozens of disaster situations supported each and every year.
• We have assisted over 350 Financial Institutions in building and testing their Business Continuity Plans.
4
Centurion Business Continuity Planning
• Enterprise Wide Business Continuity Planning
• Business Continuity Web-Based Planning Software Tool
– COPE (Centurion’s On-line Planning Expert)
• Business Continuity Plan Maintenance
• Business Continuity / Disaster Recovery Plan Reviews
• Custom Engagements – DR Testing Assistance i.e. Replication Testing – Replication Set Up
• Mock Disaster Drills
Centurion Suite of Services
5
A l a s ka
Hawa i i
Canada
CA
NV
AZ OK
UTCO
MSGA
FL
AL
SC
IA
MO
IL
WIM I
TNNC
IN OH
KY
PA
WV VA
NY
ME
MA
R ICTNJDE
MD
NHVT
OR
MT
WY
ID
NE
SD
ND
WA
MN
KS
TX
NM AR
LA
Data Replication
Jack Henry Disaster Avoidance Infrastructure
DP DRDP DR
DP 1DP 1
DP 2DP 2
DP DADP DA
DP 1DP 1
Branson
Core Director
DP 3DP 3
DP 2DP 2 CIF 20/20 DP 3DP 3 SilverLake
9
• Secure underground facility nestled in the Ozark Mountains in Branson, MO
• 175 feet below ground; enclosed under dome and two layers of granite-like shale
• Impervious to most natural disasters –hurricane/flood/tornado-proof – rated to withstand up to 1000 mph winds
• Two separate electrical transmission lines from different states
• Multiple levels of telecommunications resiliency
10
Branson Business Recovery Facility
6
Branson Recovery Center
FINANCIAL INDUSTRY RISK ENVIRONMENT
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
12
7
Banking is a Risky Business
RISK
INTERNAL
EXTERNAL
Some Risks Facing All Banks
INTERNAL» Insider Fraud» Policies and Procedures» Systems, Personnel, and
Budgetary Limitations» Infrastructure
EXTERNAL» Fraud» Regulatory Pressure
» Vendors» Infrastructure
» Disasters
8
What is Your FI’s Risk Profile?
LOWRISK
HIGHRISK
What is Your FI’s Holistic Risk Profile?
BSA/AML
BCP/DR
Internal Fraud
LOWRISK
HIGHRISK
Cyber Security
9
LOWRISK
HIGHRISK
What is the Bank’s Disaster Risk Mitigation Profile?
BSA/AML
No Business Continuity Program
Internal Fraud
Business Continuity Program
MODERATE RISK
Each organization should continually strive to move toward the Low Risk area
18
Why we need a Business Continuity / DR Plan
10
The Bank after the Disaster
20
Will your bank be able to put all the pieces together after the disaster?
11
What some Executives think about customer expectations after a disaster?
• Our customers are loyal so they will be understanding and patient until we recover, no matter how long it takes.
• Our I/T team has a plan to get the systems and applications up and that is all the bank needs to recover operations.
• We have a veteran staff and we can handle whatever comes up on the fly.
• All of our critical personnel will be available to assist in the recovery efforts.
• Our core processing is outsourced so we will not be impacted.
Customer Expectations – As Told by Actual Customers
• “I expect the same level of service immediately following a disaster as I had before the disaster.”
• “I want immediate access to my accounts via mobile, internet and telephone banking immediately following a disaster.”
• “I expect expedited, or a higher level of service if the disaster impacted me and my family and I needed emergency monies.”
• “I want the ability to do cash withdrawals immediately following a disaster with no restrictions on the amount I can withdraw.”
• “I want to be able to increase my line of credit, or apply for a loan to help me rebuild if the disaster impacts my family.”
12
After a disaster – How long will it take us to restore operations? Executive responses.
• Good Question, I am not sure. My IT team handles that.”
• “We will be fully operational by the next day.”
• “We really don’t have a real plan and we have not tested what we will do in a disaster situation in years, therefore I am not sure how long it will take to be operational.”
• “It depends on the disaster, but I will think we will be operational between 18 – 24 hours.”
• “I was told by my IT team that they will have Tier 1 applications up within 2 hours and Tier 2 applications will be available in 6 hours. The remaining applications will be available between 12– 18 hours.”
No Service
Same as Normal Service
Comparison ‐ Customer Expectations vs Executive Perception
Customer Expectations
Delayed Service
RECOVERY TIME LINE
1 Hour
24 Hours
48+Hours
12Hours
36Hours
Severely DelayedService
SlightlyDelayed Service
Actual Recovery
Level
Recovery Gap
Analysis
Executive’s Perceived Recovery
Level
Service Level after Disaster
13
The Gap: Customer Expectations-Actual Recovery Time?
• Lack of an Enterprise Wide Business Continuity Plan that has been tested at multiple levels.
• The recovery strategy for the Core (In-house processing) is tape based.
• The recovery strategy for the Win-Server environment needs improvement.
• Extensive data re-entry required due to lost of data.
• Did not account for physical recovery of personnel.
• Availability of skilled personnel.
25
THE FFIEC – FEDERAL FINANCIAL INSTITUTION EXAMINATION COUNSEL GUIDELINES ON BUSINESS CONTINUITY
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
26
14
FFIEC BCP Guidelines
Business Impact
Analysis (BIA)
Risk Assessment
Risk Management
Risk Monitoring
• Critical Business Functions
• Disaster Impacts• Prioritization• Recovery Windows• Recovery Strategies• Resources• Cyber Security
• Threats– Natural– Human– Technical– Cyber Attacks
• Enterprise‐wide BCP• Emergency Plans• Crisis Management
Plans• IT & Business Unit
Plans• Family Disaster Plan
• Plan Maintenance• Plan Testing
• Business Units• Systems / Apps
Regulatory Guidelines – The BIA
Source: FFIEC IT Examination Handbook, Business Continuity Planning, March 2008, Appendix F, p. F-3
Level 1
Level 2
Level 3
Level 4
Level 5
Prioritizing Critical Business Functions
15
RTO’s of Critical Business Functions – BIA Based
• Cash Checks• Customer Inquiries via phones• Handle deposits & withdrawals• Accept loan payments• Account transfers• Balance cash drawers• Handle security issues• Handle stop payments• Issue cashier’s checks
15 minutes – 4 Hours
4 – 8 Hours
8 – 24 Hours
24 – 48 Hours
RTO’s of Critical Business Functions – BIA Based
• Order ATM cards/debit card• Calculate Payments using projection screens• Loan status calls• Do cash advance• Fund home equity loans• Fund second trustee loans• Issue onsite ATM cards• Issue temporary checks
15 minutes – 4 Hours
4 – 8 Hours
8 – 24 Hours
24 – 48 Hours
16
Business Function Technology Requirements
Department or Business Unit
Business Function/Activity
Corporate Impact
System Required
Application Required
Manual Process
Recovery Time
Objective(RTO)
Recovery Point
Objective(RPO)
Branch Operations Cash checks High iSeries Silverlake Yes 4 Hours 15 MinTelephone Express Center Process loan payments High iSeries Silverlake Yes
8-24 Hours 15 Min
Telephone Express Center Process wire transfers High iSeries Silverlake No 8 Hours 15 Min
Information Technology Administer backups High
ClientServer ProcessPro Yes 4-8 Hours N/A
Depost ServicesSet up close day, close
month process HighClientServer ProcessPro Yes 3+ Days 24 Hours
Electronic Banking Prepare VRU report HighClientServer ProcessPro Yes 3+ Days 6 Hours
Electronic Banking Hot card entry LowClientServer InTouch Yes 8 Hours 12 Hours
Electronic BankingSet up new Internet
accounts MediumClientServer PinPoint No 8 Hour 12 Hours
Item ProcessingSet up new Internet
accounts MediumWork
station NetTeller No 4 Hours 2
BUSINESS CONTINUITY / DISASTER RECOVERY PLAN COMPONENTS
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
32
17
The Major Components of the BCP
PeoplePeople
Employees
Customers
BCP / DR Teams
Vendors
Fire / Police
Utilities
Regulators
Plans / Procedures
Documentation
FacilitiesFacilities
Alternate work areas
Repaired facilities
Recovery centers
Hospitals
Shelter areas
Mobile Recovery Units
Off‐site storage facilities
TechnologyTechnology
Systems
Servers
Applications
Data
Telecommunications
The Most Critical Component – People
34
Team members require consistent training and testing
18
Facilities - Physical Recovery Considerations
• Branch Offices
• Work from Home
• Vendor Recovery Site
• Internal Recovery Site
• Mobile Recovery Unit
• Office/Remote Workspace
• Temporary Lease Facility
Technology-Equipment Recovery Considerations
• Store equipment in advance• Purchase equipment when needed• Drop Ship Equipment Service
– Mainframe– Servers– Workstations– Printers / Fax Machines– Phones– Routers / Switches
• Vendor provided at Recovery Site
• What is your equipment recovery strategy?
19
Step 4 ‐ Draft Plans Generated
37
Emergency Management Plan (Per Facility)
Crisis Management Plans
Information Systems Recovery Plan Business Unit Recovery Plans Branch Office Recovery Plans
Executive Summary Plan Testing & Exercise Guide
Business Continuity Plan Documentation
Loan
Ops
Human
Reco
urse
Bran
ch Ops
Business Continuity Team Structure
Business Unit Recovery Teams
FinanceTeam Leader Alt. Team Leader
AdministrationTeam LeaderAlt. Team Leader
Information SystemsTeam Leader Alt. Team Leader
Loan OperationsTeam LeaderAlt. Team Leader
Deposit OperationsTeam LeaderAlt. Team Leader
Bookkeeping Finance Accounting eBanking
AuditComplianceHRTraining
Marketing InvestmentsMaintenance
Information Systems
Loan AnalystLoan ProcessingCommercial Lending RE Mortgage
Deposit OperationsRetail Banking/Consumer Lending
ManagementTeam Leader Alt. Team Leader
Crisis Management Team
20
Business Unit Recovery Team Plan - TOC
1. Team Information1. BU Recovery Team - Recovery Organization Charts2. Workspace & Equip. Summary - Facilities & Locations
2. Notifications1. Personnel Notification Script - Business Unit Call List2. Call List Team Leaders – External/Internal Notifications
3. Recovery Tasks1. Recovery Phases – Recovery Tasks for Rec. Team
4. Business Impact Analysis (BIA) Reports5. Business Function Recovery Procedures6. Recovery Forms7. Appendix
40
Business Continuity Planning Software Tool
• Based on Best Practices of the Financial and Business Continuity industry.
• Designed solely for financial institutions and based on the FFIEC Guidelines on Business Continuity Planning.
• Web-based and hosted at on an independent network and server.
• Built on a Relational Database platform, i.e. SQL. • Fosters plan ownership at the business unit level.• Access plans for planning purposes, testing,
maintenance and plan execution from any web browser.
• Supported by experts in the business with a solid financial backing and a strong record of consistency and support.
21
Communications Tool to communicate with Employees – Customers -Vendors
41
BUSINESS CONTINUITY PLAN ASSESSMENT
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
42
22
How would you answer the following questions?
Is it an “Enterprise Wide” plan or just an I/T Plan?
Will the plan meet the examination criteria?
Is the plan tested and maintained on a regular basis?
How effective is our BCP?
How would you answer the following questions?
The most important question;
Will our plan get us through the “Disaster Event”
23
Business Continuity Plan Assessment
Plan Elements In Plan Not In Plan
Comments
Emergency / Evacuation Plan
Succession Plan – Escalation Plan
Alternate Work Locations Identified
Business Impact Analysis Conducted
Recovery strategies in place to restore critical functions
Recovery Timeline Identified
Critical Documentation Identified
Resource Requirements Identified per business function
Risk Assessment‐Risks Identified and Prioritized
Business Continuity Plan Assessment
Plan Elements In Plan Not In Plan Comments
Media Spokesperson Identified
Command Center Identified
Plans per department with specific information for that team?
Designated employee call list documented
Manual Procedures Documented
Notification List – External Contacts
Notification List – Internal Contacts
Recovery Teams Identified
Recovery Tasks Identified for Personnel
Critical forms identified
24
Business Continuity Plan Assessment
Plan Elements In Plan Not In Plan Comments
Core Processing System (Software) Recovery Strategy / Procedures in place
Win Servers Recovery Strategies in place
Telecommunication Recovery Strategies
Electronic interface recovery strategies i.e., Internet, voice response,
Vital Records Recovery Strategy
Plan Testing Strategy
Evacuation Drills
Hot Site Test
Recovery Team Exercises
Plan Maintenance Program in place
RECOVERY TIMELINE
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
48
25
Plan Execution Phases
Pre – Disaster Activity
Crisis Management Phase
Relocate Phase
Restore Operations
Phase
Rebuild PhaseRebuild Phase
Crisis Management Phase – Use the Crisis Management Plan Busin
ess as U
sual
Emergency / Crisis Management Plan
Relocation Phase – Use Business Continuity Plan consisting of IT & BU Plans
I/T Disaster Recovery Plan
Recover Business Functions PhaseBusiness Unit
Recovery Plans Plans
Restoration Plan ‐ Rebuild & Return Phase
50
Plan Execution Phases
Disaster Timeline
26
Evacuation & SafetyDamage AssessmentCommunicationsDisaster Declaration
NotificationsMobilizationRelocationRestore
Plan Execution / Recovery Timeline
Crisis Management PhaseRelocate &
Restore Phase
Recover Business
Functions Phase
Disaster
Emergency Management Plan for each location
• Notification List for Main - Branch Offices – Local Authorities
• General Emergency Policies
• Evacuation Policy
• Emergency Tasks
• Accounting for Personnel After a Disaster
• Reacting to a Disaster
• Evacuation Meeting Place per Facility / Department
• Disaster Scenarios
– In Case of Fire:
– In Case of Severe Tornado:
– In Case of Flood:
– In Case of Hurricane Watches/Warnings:
27
Evacuation & SafetyDamage AssessmentCommunicationsDisaster Declaration
NotificationsMobilizationRelocationRestore
Plan Execution / Recovery Timeline
Crisis Management PhaseRelocate &
Restore Phase
Recover Business
Functions Phase
DisasterCrisis Management Phase
Relocate & Restore Phase
Recover Business
Functions Phase
Restore Business Unit Functions & Processes using Business Unit Recovery Plans
Plan Execution / Recovery Timeline
Evacuation & SafetyDamage AssessmentCommunicationsDisaster Declaration
NotificationsMobilizationRelocationRestore
Crisis Management Phase
Relocate & Restore Phase
Recover Business
Functions Phase
Disaster
28
Rebuild & Return Phase
• Warnings / Public Information• Debris clearance, removal & disposal• Utilities & communications restoration• Temporary housing• Detailed building inspections
Plan Execution / Recovery Timeline
• Redevelopment planning• Environmental assessments• Demolition• Reconstruction• Hazard mitigation
Disaster
Crisis Management Phase
Relocate & Restore Phase
Recover Business Functions Phase
Plan Execution / Recovery Timeline
DisasterCrisis Management
Phase
29
57
Crisis Management Phase Tasks
Provide safety and shelter for employees / customers. Provide medical assistance as required. Activate Business Continuity PlanProvide safety and shelter for employees / customers. Provide medical assistance as required. Activate Business Continuity Plan
Declare official disaster with employees, customers, vendors, etc. Declare official disaster with employees, customers, vendors, etc.
Shut off gas, water and other utilities. Secure vaults, cash drawers,etc. Power down and protect equipment. Perform damage assessmentShut off gas, water and other utilities. Secure vaults, cash drawers,etc. Power down and protect equipment. Perform damage assessment
Order fuel in advance for generator if applicableOrder fuel in advance for generator if applicable
Perform a system backup if applicablePerform a system backup if applicable
Take employee / customer head count prior / after the eventTake employee / customer head count prior / after the event
Notify Human Resources and authorities of people not accounted forNotify Human Resources and authorities of people not accounted for
C
R
I
S
I
S
58
Crisis Management Phase Tasks
Secure the facility. Establish a liaison to Interact with authorities as required. Establish Emergency Command Center for the bank.Secure the facility. Establish a liaison to Interact with authorities as required. Establish Emergency Command Center for the bank.
Determine who is available to work and conduct initial status meeting. Assign initial tasks based on available resourcesDetermine who is available to work and conduct initial status meeting. Assign initial tasks based on available resources
Communicate with vendors, i.e. alarm company, recovery center, temp.workspace provider, telecommunications, regulatory agency, etc.Communicate with vendors, i.e. alarm company, recovery center, temp.workspace provider, telecommunications, regulatory agency, etc.
Secure critical documents, files, etc. Secure critical documents, files, etc.
Salvage equipment as requiredSalvage equipment as required
Establish communication protocol.Establish communication protocol.
Update web site and status hotline as requiredUpdate web site and status hotline as required
C
R
I
S
I
S
30
59
Team members must protect their family first
Four Steps to Family Preparedness
Family Disaster Plan
Family Disaster Plan
Get Informed
Get Informed
Make A Plan
Make A Plan
Assemble Disaster KitAssemble Disaster Kit
Test Maintain Plan
Test Maintain Plan
31
Get Informed
• Go to www.ready.gov to build a plan• Identify community hazards – Risk Assessment
• Tornadoes – earthquakes – hurricanes – local treats – chemical plants – airports – military base
• Learn about your community's Emergency Response Plan
• Know your Community Warning Systems• NOAA / NWS • Warning levels• Warning response requirements
Make A Plan
• Conduct a family disaster planning meeting
• Select “Out of Town” Contacts
• Determine emergency meeting locations
• Develop a family communications plan
• Contact information for:
• Home – family members - work – school – emergency personnel - authorities – doctor – pharmacist – hotels – airlines
• Escape Routes and Safe Places
• Provide for those with disabilities/special needs
32
Make A Plan (cont.)
• Plan for your pets
• Boarding facilities – Pet friendly hotels
• Plan for multiple disaster scenarios
• Create an “Action Checklist” of things to do prior to a disaster
• First Aid training Utilities operations
• Fire extinguisher usage Smoke alarms
• Insurance coverage Home inventory
• Vital records / documents
• Reduce home hazards in advance
Create a Disaster Supplies Kit
• Three day supply of food Identification
• Three day supply of water Credit Cards
• Battery powered radio/TV Matches
• Flashlight & batteries Pictures
• First aid and medicine Cash & coins
• Sanitation supplies Matches
• Whistle Basic Tools
• Extra clothing /blankets Extra keys
• Credit cards Maps
• Masks / gloves Cellular phone
33
Maintain / Test Your Plan
• Review semi-annually
• Conduct drills, i.e. evacuation
• Determine and travel alternate evacuation routes
• Restock supplies, i.e. (food, water, etc.)
• Replace batteries in smoke detectors
• Update contact information
• Recharge fire extinguishers – train on use
• Update evacuation routes
MobilizationRelocationRestore
Plan Execution / Recovery Timeline
Relocate & Restore Phase
Disaster
34
67
Relocate and Restore Phase Tasks
Retrieve critical resources, documents from off‐site locationRetrieve critical resources, documents from off‐site location
Set up alternate work locations; mobile trailer, internal recovery center, alternate workspaceSet up alternate work locations; mobile trailer, internal recovery center, alternate workspace
Travel to alternate work locations. Set up security at alternate facilityTravel to alternate work locations. Set up security at alternate facility
Put signage on damaged facilityPut signage on damaged facility
Re‐establish communications between alternate data processing site and branchesRe‐establish communications between alternate data processing site and branches
Provide authorities and security staff with list of employees allowed to enter damaged facilityProvide authorities and security staff with list of employees allowed to enter damaged facility
Notify vendors of alternate work address for deliveries. Redirect mail and courier deliveriesNotify vendors of alternate work address for deliveries. Redirect mail and courier deliveries
R
E
S
T
O
R
E
Restore Business Unit Functions & Processes using Business Unit Recovery Plans
Plan Execution / Recovery Timeline
Recover Business
Functions Phase
Disaster
35
69
Recover Business Function Phase Tasks
Execute Business Unit Recovery PlansExecute Business Unit Recovery Plans
Determine time needed to process backlog workDetermine time needed to process backlog work
Reconstruct / Re-enter “Work in Process”Reconstruct / Re-enter “Work in Process”
Implement business unit contingency strategiesImplement business unit contingency strategies
Install salvaged equipmentInstall salvaged equipment
Develop personnel work scheduleDevelop personnel work schedule
Update web-site and media sourcesUpdate web-site and media sources
Rebuild & Return Phase
• Warnings / Public Information• Debris clearance, removal & disposal• Utilities & communications restoration• Temporary housing• Detailed building inspections
Plan Execution / Recovery Timeline
• Redevelopment planning• Environmental assessments• Demolition• Reconstruction• Hazard mitigation
Disaster
Crisis Management Phase
Relocate & Restore Phase
Recover Business Functions Phase
36
DR RECOVERY STRATEGIES
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
71
Levels of Recovery
Data
Resume Critical Business
Procure Hardwareto Restore Data(Dissimilar)
Procure LocationHVAC/Utilities
Provision Security
Assemble Technical & Recovery Teams
Enterprise Recovery
Geography Workspace
Procure CommunicationsApplications
File & Folder Recovery
Data
System Recovery
Data
Procure Hardware to Restore Data (Dissimilar)
Assemble Technical Team
Workgroup Recovery
Data
Procure Hardware to Restore Data (Dissimilar)
Assemble Technical & Recovery Teams
Applications
Procure Communications
37
The plan must address different Recovery Levels
System Failure
Site Failure
X XProduction Processing
Branch 1
Branch 2
Branch 3
Communications
Processor
Network Failure
X
System / Application Recovery Strategies
• Traditional Media Device Backup– Tape – USB –Hard Drive – CD
• Virtualization / Replication
• Electronic Vaulting
• High Availability
38
Cost Vs. Level of Recovery CommitmentTechnology Infrastructure
ContinuousAvailability
RPO=near zero, RTO <1min, AutomaticServer/Workload/Network/Data SYSPLEX
RPO=Near zero, RTO <1Hr. to 4 hours, AutomaticServer/Workload/Network/Data Automatic Site Switch
RPO=Near Zero, RTO <1Hr. to 4 hours, ManualDisk or Tape Data Mirroring
RPO > 15 min. RTO= 4+ hours, ManualPiT or SW Data Replication.
RPO=4+ hours, RTO=8 to 24 hours, ManualData Base Log Replication & Host Log Apply at Remote
RPO<24 hours RTO = 8‐24 hoursHot Site & Tape RTO=Days, RPO>24 hours
Tape, HW ATOD
Recovery Point Objectives (RPO) & Recovery Time Objective (RTO)
Point‐in‐TimeBackup toTape / Disk
ActiveSecondary Site
Multi‐SiteFailover /Fallback
RPO<24 hours RTO = 8‐24 hoursElectronic Tape Vaulting
Cost
Lower
Higher
HoursMinutes Days
Traditional
Tape Recovery
Inherent Problems with Traditional Tape Recovery
• Personnel may not be available to transport tape.
• Tapes may not be accessible due to the disaster.
• Data loss based on when the last backup was stored off-site.
• Tapes may be damaged as a result of the disaster.
• Tape errors when trying to restore at the recovery center.
• Roads may be damaged or crowded due to evacuations.
• Airports may be closed.
• Recovery Time Objectives (RTO) may not be met.
39
Take advantage of technology to:
• Reduce the human dependency on restoring your technology infrastructure.
• Eliminate having to transport tapes and paper documents to the recovery center.
• Electronically replicate systems & servers,• Communicate with employees through an Emergency
Notification System that they can also use to provide updates of their status.
Typical Operating Environments
SCENARIO 1
• CORE: In-House
• SERVERS: In-House
SCENARIO 3
• CORE: In-House
• SERVERS: Outsourced
SCENARIO 2
• CORE: Outsourced
• SERVERS: In-House
SCENARIO 4
• CORE: Outsourced
• SERVERS: Outsourced
40
SCENARIO 1 - CORE: In-House
SERVERS: In-House
In‐House Processing Considerations
• Responsible for the restoration of the following:
– Recovery of Core System
– Recovery of Server / Network Recovery
• Exchange Servers - Domain Controllers
• Application Servers
– Telecommunications - Voice Recovery
– Equipment setup & Reconfiguration
– Facilities
TIME
Last EOD Backup of usable data
Friday 8:00 pmDisasterStrikes
Monday 3:47 pm
Disaster
67.47 Hours of Data Loss Tape
Recovery Strategy
7Hours
Data Re‐entry
Catch up
Last EODTapesFriday8:00 PM
6Hours
Time to Recover = 31 Hours
13Hours
Travel toRecovery Center
5Hours
Declare Disaster
SCENARIO 1 – Bank A - CORE: In-House
SERVERS: In-House – Tape Recovery Strategy
41
TIME
Last Data Snapshot3:32 pmMonday Disaster
StrikesMonday 3:47 pm
Disaster
Data Loss15 Minutes
Time to Recover
30 Minutes
SCENARIO 1 – Bank B - CORE: In-House
SERVERS: In-House – Bank B – Replication
Disaster Avoidance Concept – Bank B
TIME
Recovery of Business still Required
Disaster Avoidance Decision
Disaster Avoidance Period
Recovery of TechnologyAvoided(RTO)
PotentialDisasterEvent
Disaster
Switch to Secondary System
42
Out‐Sourced Processing Considerations• Responsible for the restoration of the following:
– Connectivity back to the Core Processing Site
• (jConnect Backup Router)
– Server / Network Recovery
• Exchange Servers - Domain Controllers
• JHA & 3rd Party Applications
– Telecommunications - Voice Recovery
– Equipment setup & Reconfiguration
– Facilities
• A plan to deal with a disaster that strikes the facility
SCENARIO 2
• CORE: Outsourced
• SERVERS: In-House
BCP TESTING
July 21, 2016©2015 Jack Henry & Associates. All Rights Reserved.
84
43
The FFIEC Testing Principals
• Define roles / responsibilities for testing and evaluation.
• Use the BIA/Risk Assessment as the test foundation.
• Enterprise-wide testing should be conducted annually.
• Testing should be viewed as a continuous cycle.
• The testing program should be reviewed by an independent party.
• Test results should be compared against the BCP to identify any gaps between the testing program and business continuity plan. guidelines.
Test Plan
Set Test Objectives
Select Training Methods
Identify Resource
RequirementsIdentify Participants
Identify Schedule Options
Determine Test Budget
Conduct Test
Identify Scope
Develop a Test Plan