Planning for Disaster Recovery Using SUNY Facilities to Advantage Initial Planning Concept Presented by Eric Wilson and Bob Hemmings with Ron Brown and.

Planning for Planning for Disaster RecoveryDisaster Recovery

Using SUNY Facilities to AdvantageUsing SUNY Facilities to Advantage

Initial Planning ConceptInitial Planning Concept

Presented by Eric Wilson and Bob Hemmings Presented by Eric Wilson and Bob Hemmings

with Ron Brown and Mike Notariuswith Ron Brown and Mike Notarius

PurposePurpose• Provide reasonable and affordable Disaster Recovery

options for most ITEC campuses• Provide a formal disaster recovery plan for each campus

- and for ITEC• Develop an initial Disaster Recovery functional test and a

periodic retest for participating campuses• Gather information from participating campuses into a

Disaster Recovery database• Making campuses ready for emergencies and in

compliance with SUNY and State auditors’ recommendations

• Scope: Not a Disaster Response – that would be U Police, Residence Life, HR, etc. roles

IntroductionIntroduction

• Today is not “Business as Usual”, due to War on Terror– will be a part of our lives for decades

• American universities may become soft targets, because of:– Traditional “openness” of universities– Opportunity to cause large loss of lives at universities

• creating loss of confidence that universities are safe for youth• creating aversion overseas to sending their youth here

– Possible loss of US influence abroad by cutting University ties• Need for rapid, but careful and effective planning• Disaster planning used to be thought of as for an unlikely event

– Due to terrorism the possibility of disaster has become more likely– Earthquakes are not enemies, just natural events– Terrorists are enemies actively trying to “get us”

• Planning is therefore much more important than in the past• Need to develop plans expeditiously, but carefully

– Subject to frequent revision given new developments

BackgroundBackground• Traditional Information Technology Threats:

– Fire– Flooding– Earthquake– Hurricane/Tornado– Vandalism– Accidents– PCBs

New Information Technology Threats:– Terrorism

• Bombs• Chemicals• Biological• Combinations

– Viruses– Invasions: from hackers or even terrorists

Relatively Minor RisksRelatively Minor Risks

• Relatively Minor:– Equipment destruction– Minor Facilities destruction– Communications infrastructure destruction– Short-term Interruption

Major RisksMajor Risks• Mission Critical Risks:

– Destruction of College Records • Used to be on microfilm or paper, but now largely automated• Most Significant:

– Registration credits and grades– Graduation Records– Student Payment Records– Budgetary and Accounting Records– Alumni / Foundation Names and Giving– Loss of irreplaceable campus specific computer code

– Loss of Term• Having to close doors for semester or longer• Inability to grant expected credits/degrees

– Major Facilities Destruction– Loss of key Knowledge-holders

• Systems, applications, communications programmers• Key users and user managers

Disaster Recovery PlanningDisaster Recovery Planning

• Recovery of Critical Information and Systems

• Identifying Major Responders and Partners

• Carefully Pre-planned Steps– with flexibility to adjust

• Temporary Alternate Facilities

• Technology Specialists

Photos of the NYC Blackout

This can happen to you too!

Times Square during Blackout

Empire

State

Building

during

Blackout

Brooklyn Bridge during Blackout

Grand Central Station during Blackout

Upper Manhattan during Blackout

Lower Manhattan during Blackout

The Statue of Liberty

Recovery ofRecovery ofCritical Information and SystemsCritical Information and Systems

• Initial Scope: – BANNER and related systems– Email systems– Basic Web Services for campus– System Administration interface

• Scope will be enlarged– after some experience with initial scope

Identifying Major Identifying Major Responders and PartnersResponders and Partners

(Disaster Recovery is more than an IT function)(Disaster Recovery is more than an IT function)

• Responders: must be involved in planning process– Senior Management (President, VPs, Directors), University

Police– Information Technology Staff– Facilities Staff– End users– Others

• Partners– ITEC– SICAS Center– SUNYNet– Training Center– Systems Administration– Information Technology Industry

Carefully Pre-Planned stepsCarefully Pre-Planned stepswith flexibility to adjustwith flexibility to adjust

• Plan must be thought-through

• Reasonable expectation of success

• All aspects of plan must be available– concept of Disaster Recovery database

• Flexibility to adjust– no outside contractual barriers

• as can occur with vendors

– ability to revise general template of plans

Planning for temporary Planning for temporary alternate facilitiesalternate facilities

End-User space convenient to campus– on or off campus, depending on emergency– with appropriate equipment at facility

• on-loan temporarily• to be quickly ordered

– with appropriate point of presence for communication

• could be temporary wireless

Planning for immediately available Planning for immediately available technology specialiststechnology specialists

• Systems programmers

• Network communications specialist

• Database managers

• Application managers

• Others

Possible SUNY ITEC Role:Possible SUNY ITEC Role:Prior to DisasterPrior to Disaster

• Manage electronic backup vaults for campuses– BANNER databases, ORACLE/BANNER

code• related unique campus code, e.g. SQR, C, Cobol

– Email data and logs– Duplicate offsite electronic backup vaults

• at Systems Administration or elsewhere

Possible SUNY ITEC Role:Possible SUNY ITEC Role:Prior to Disaster (continued)Prior to Disaster (continued)

• Pre-Disaster Recovery– Stay up with industry Disaster Recovery concepts– Discovery of SUNY specific concepts– Preparation of Disaster Recovery plan template– Coordination of Disaster Recovery planning– Scheduling recovery functional testing with campuses

• initial functional testing• periodic retests, less comprehensive than initial test

– Identify equipment for Disaster Recovery deployment

Possible SUNY ITEC RolePossible SUNY ITEC RolePost DisasterPost Disaster

Provision of Disaster Recovery equipment and software– Systems and software for remote BANNER support

• Solaris UNIX, AIX UNIX, OVMS

– Systems and software to remotely support EMAIL• Lotus Domino, Exchange, Send Mail, Others

– Campus Web Services• Basic functionality for emergency purposes

– Systems Administration connections• Web interfaces• Print queues

Possible SUNY ITEC RolePossible SUNY ITEC RolePost Disaster (continued)Post Disaster (continued)

• Provision of Disaster Recovery technical support– Possible loss of campus Tech Support professionals– On-campus/remote ITEC support, as required– Host campus tech support at ITEC, if required

• Concerns– Best effort to avoid these problems

• Possible multiplicity of concurrent disasters• Unexpected complications of scenarios• Untested scenarios, e.g. lack of version control• Unexpected changes to supported environments• User expectations exceeding scope

Possible SUNYNet RolePossible SUNYNet Role

• Assist in acquiring new Internet/SUNYNet access, if/when needed

• Provide wireless communications facilities in case campus hub is out

• Communications hardware

• Security issues

Possible Possible SUNY Training Center RoleSUNY Training Center Role

• Emergency user office space and facilities, if needed

• Emergency personnel training, if needed

• Emergency equipment borrowing, if needed

Possible Possible SUNY SICAS Center RoleSUNY SICAS Center Role

• Participation in Disaster Recovery detailed planning

• Planning BANNER version control strategy

• Participation in Disaster Recovery tests

• Provision of BANNER systems at ITEC for functional Disaster Recovery testing

• Assisting in Post Disaster Recovery

Possible Campus RolePossible Campus Role• Scope: limited to ITEC members

• as identified in database

• Identify components of plan:• Contacts: managerial, technical, vendors, key users• Critical equipment and software• Most critical user stations

– BANNER Users, Email Users, System Admin based Business Office Users, Library

• Alternate locations for users, e.g. TC, or nearby campus• Alternate internet access points, e.g. Time Warner• Communications issues and strategies, e.g. firewalls,

packet shapers• Critical printer server needs• Current network diagram document• Others

Possible Campus Role (continued)Possible Campus Role (continued)

Scheduling the functional recovery tests– Identification of staff involved in test– Pre-test planning meeting

• Identifying success factors

– Initial functional test– Post-test evaluation meeting

• Recommendations• Additional test, if success factors not met• Transmittal of results to management

– Re-test issues

Possible Campus Role (continued) Possible Campus Role (continued) Get upper management involvement in planningGet upper management involvement in planning

• Management buy in is a critical success factor• Management planning beyond information systems

– Template to be developed to guide campus thought– Possible temporary closing of campus– Possible replacement of key management and staff

• Assuring management of– security and privacy of electronic vaults

• Bringing campus staff into the planning– Another major success factor

• Bringing in other areas of concern– scope has to be limited; cannot be all things for all people– need to include key vendors

• Consider impact of being without computing– 3 days, 7 days, longer

• Create plans based upon this expected impact

SummarySummary

• Provide reasonable and affordable Disaster Recovery options for most ITEC campuses

• Provide a formal disaster recovery plan for each campus- and for ITEC

• Develop an initial Disaster Recovery test and a periodic retest for participating campuses

• Gather information from participating campuses into a Disaster Recovery database

• Making campuses ready for emergencies and in compliance with SUNY and State auditors’ recommendations

Where we go from here:Where we go from here:• Continual campus input is needed• Bringing others into the process

– SUBOA/CCBOA, Systems Administration– others

• Please refer to the Gartner Group report– “Management Update: Best Practices in Business

Continuity and Disaster Recovery”

• First test campuses: Oneonta and Geneseo• Begin definition of the Disaster Recovery database• Possible training in Disaster Recovery by TC

CommunicationCommunication

• This PPT presentation will be available through ITEC’s home page

• Comments and Critiques welcome– Bob Hemmings

• [email protected]• 315-829-2213 or 315-391-2916 (cell)

– Eric Wilson• [email protected]• 607-432-5980

– Ron Brown