Municipal League Joint Venture Disaster Recovery Project

Municipal League Joint Venture Disaster Recovery Project

Lee M. Mandell, Ph.D.Director of Information Technology and Research/CIO North Carolina League of Municipalities

What is Disaster Recovery?

Disaster Recovery is the ability to recover from a loss of the IT facilities (or access to those facilities) and infrastructure used to conduct daily business.

Why Would You Need It?

The loss of the use of IT facilities can occur for many possible reasons:

hurricanes, floods, fires, tornados, chemical spills, earthquakes, power grid failures, HVAC failures, ice and snow storms, pandemics, terrorism, computer viruses, structural failures, vandalism, and the list can probably go on

DR Is Not BC

A disaster recovery solution should be

part of a comprehensive business continuity planHaving a hardware/software/ communications alternative to a data center is not enoughThis presentation is only about one piece of the DR/BC puzzle and needs to be put into the context of the broader problem and solution

The DR Center Project

The NC League of Municipalities, in partnership with the Municipal Association of SC, and its technical partner VC3, has developed a jointly-owned Disaster Recovery facility for Wintel and compatible systems

Our thanks to HP for technical and financial assistance on this project

Project Overview

Although initially serving only two municipal organizations, the project determined that a methodology and solution existed that would allow multiple Leagues to share the initial cost and overhead of a scalable, centralized solution and allow recovery of business functions in the event of a disaster.

Project Overview

The DR solution developed for this joint project leverages: technology changes that have

occurred and matured in the past few years (server virtualization and imaging software)

the public Internet, and sweat equity

to dramatically reduce the cost and increase the operational effectiveness of a DR facility

Physical Server

Virtual Machines

ESX Server

Server Virtualization Basics

Deploy multiple virtual machines on a single physical server

Unmodified Application

Unmodified OS

Virtual Hardware

Benefits Increase hardware

utilization by sharing hardware resources across a large number of virtual machines.

Each Virtual Machine is a

complete system encapsulated in a

set of software files

Options for Disaster Recovery

Physical to Physical Today’s scenario without

Virtual Infrastructure Requires identical DR site

Physical to Virtual Reduces cost, improves time

to recovery, and increases flexibility

Virtual to Virtual Greatest flexibility, lowest

cost, best time to recovery

Activities to Date

Two sub-committees formed: Technology Committee Business Plan Committee

Technology Committee: Documented initial technical

requirements Designed a technical framework Completed the Proof of Concept

Phase Completed the Pilot Phase Entered the Production Phase

Technology Committee AssumptionsSupport for Windows NT 4.0, 2000 & 2003 ServersDR Site is a “very warm” standby; NOT a live HOT site Data synchronized on a daily basis (in some cases more frequently)Images transferred and access to restored servers over public InternetDR Site initially sized to support one organization at a timeFull recovery achieved within 24 hours remotely; targeted goal and expectation is less than 12 hoursDR site must accommodate firewall needsDR site must accommodate backups after production services have been transferredTwo tests per year per organization at the DR site

Phase Objectives

Proof of Concept Phase Validate the technology Verify data transfer and conversion timings Refine budget and processes

Pilot Phase Utilize live production systems Build a functional environment with a sub-

set of production servers from each League Validate sustained functionality and

reliabilityProduction Phase

Implement final solution with production services from all Leagues

The History of the Solution

August 2005: MASC and NCLM decide to form a joint venture to solve their own DR needsSeptember 2005: High level solution architectedNovember 2005: Proof of concept phase successfully completedJanuary 2006: Initial pilot phase successfully completedApril 2006: Hardware and software refined and orders placedMay 2006: Data Center space identified in Charlotte, NCJune 2006: Data Center space securedJuly 2006: MASC went live with basic DR Solution; recovery test completed successfully

Key Technology ComponentsHardware

Three HP dual processor servers (shared)

• 2.6 Ghz• 16 GB • 1.8 TB

VPN concentrator (shared) Firewall appliance (shared) Gigabit switch (shared) Console KVM Switch (shared) Image Server (low cost/high capacity

data storage) Software

VMWare – Server Virtualization Suite Symantec LiveState – Imaging Software

DR Server Replication

DR Process

Nightly transfers/replication of incremental data and system configuration images to off-site facility: Secure FTP with 3DES encryption

Image production and transfer process automated

Images validated after creation and transfer

Monthly Disk Transports of full images (base size 300 GB): 128 bit AES encryption with encryption at the data and file system level

Remote connection to restored servers at DR Center through Citrix

DR Center Metrics

Creating LiveState images of 11 MASC core virtualized servers: Initial Image: 2 to 2.5 hours 300 GB Daily Incremental Image: 2 to 2.5

hours 3 to 4 GB

Transferring incremental images nightly to DR Site: Incremental Image : 2 Mb/sec (10 Mbs

burst) 3.5 to 4 hours

Remote recovery of all 11 systems from incremental images: 6 to 8 hours (parallel recovery)

DR Cost ComparisonsTraditional Vendor-Provided DR Services Costs:$3,000 to $4,000 per monthCosts per League for five-year planning horizon:$180,000 to $240,000

DR Center Costs:initial shared setup costs at $40,800; hosting charges of $1,150 per month; pre-funded replacement charges of $680 per month; local setup costs at $20,400 to $51,900Costs per League (2) for five-year planning horizon:$95,700 to $127,200

Key Benefits of Basic DR Center Solution

Captures all data and all system state informationAllows remote restoration of all serversUses public Internet for both image transfer and access to restored systemsScales more cost effectively as data grows as compared to other solutionsShares costs and resources for little-used but crucial capabilityEasy to plan and budget the continuing costsDemonstrates that a partnership among like organizations to support joint DR solution can work (formal working/ownership agreement developed and signed)

NCLM Enhanced DR Solution

Reduce the cost and complexity and increase the flexibility and robustness of business continuity by storing entire database and system files on a storage area network (SAN) that can be replicated and restored at DR CenterProvide near real-time, byte-level replication and point-in-time rollback recovery

Provide additional level of data recovery beyond nightly backups at the file/email message levelDirect broadband connection to DR CenterLocate server room appliances at DR CenterWould add around $92,000 to five-year costs, net

Disaster Recovery Using SAN/iQ Remote Copy

Perform disaster recovery using Remote Copy No need to upgrade secondary site server

hardware in lock-step with the primary site Easy to automate and no need for bare metal

recovery tools Virtual image and data all protected under one

DR planPrimary

SiteRecovery Site

The Rest of the Future

Expand basic or enhanced version to other southern municipal leaguesThe basic DR solution also has the promise of being able to provide shared DR services to cities and towns in NC and SC that have historically been cost prohibitive for them to purchase, allowing the recovery of their critical business functions in the event of a disaster

Thank You!