STAFF SYMPOSIUM SERIES · VMWare Essentials Bundle Plus provides for 6 sockets = 3 servers @ 2 sockets each. Get 3!!! If you are virtualizing more than 5 servers, consider 3 physical

STAFF SYMPOSIUM - IT TRACK

STAFF SYMPOSIUM SERIESINFORMATION TECHNOLOGY TRACK

FACILITATORS

4/14-15/2015 SESSION 1 - SYSTEM MANAGEMENT 1

Carl Brooks System Manager - Detroit, MIChapter 13 Standing Trustee – Tammy L. Terry

William Drake System Manager – Ruskin, FLChapter 13 Standing Trustee – Kelly Remick

Scot Turner System Manager – Las Vegas, NVChapter 13 Standing Trustee – Rick Yarnall

Tom O’Hern Program Manager, ICF International, Baltimore, MDSTACS - Standing Trustee Alliance for Computer Security


Information Systems ManagersSession 5

Virtualization and Backup for Disaster Recovery

Server Virtualization coupled with cloud backup services provides a sound foundation to maintain continuity of trustee operations through minor and

major disasters.

This session covers costs, benefits, and details on server virtualization, cloud backup, cloud operations, alternate site recovery

and disaster operations.



(40) Server Virtualization

(30) Backups and the Cloud

(20) Disaster Recovery and

Prevention Technology Solutions

Session Agenda


STAFF SYMPOSIUM IT TRACK

Server Virtualization HyperV, VMWare, Virtualbox, XenApp

What is it?

Why consider it?

Why avoid it?

Virtual Machine Architectures

Planning and Recommendations for Trustee office virtual servers

Virtual server backup strategies

4/14/2015 SESSION 5 - DISASTER RECOVERY 4


HyperV, VMWare, Citrix XenApp, VirtualBox

Hyper-V – Microsoft’s hypervisor. Classical Windows interface. Install Hyper-V as a “Role”

VMWare – Longest player in the game. Proprietary drivers. Costly Add-ons.

XenServer – was Xen, then bought by Citrix. Uses standard, open-source Drivers. “5 minutes to Xen”. Does require a Host and a separate XenCenter system (PC class OS). Free open-source but costs for Maintenance & Support

VirtualBox – Oracle’s open-source hypervisor. Desktop Virtualization. Not an operating system so needs to be loaded on a supported OS Host (Windows, Linux, Mac, Solaris). Targeted towards personal and small office environments.



What is it?

Software technology that makes it possible to run multiple operating systems simultaneously on a single physical server - Run Windows 8, Server 2012, RedHat, Ubuntu, etc.



Why Consider It?


• Leverages your hardware more effectively – utilizes more CPU, RAM, storage capabilities

• Provides scalability – add Hosts and VM’s at will• Reduces capital expenditures – 2 physical servers – 7 virtual servers exposed to

the network.• Provides portability and failover – much less downtime• Supports Business continuity and DR – replicated VM’s offsite.


Why Avoid It?

Some licenses just wont run in a virtualized environment

Graphic intensive applications (CAD)

VDI may be more expensive than good ‘ol desktops

Possible high I/O applications may perform better directly with storage



Virtual Machine Architecture



Planning and Recommendations for Trustee office virtual servers

Licensing – Windows server single license provides for 2 VM’s.

Need more memory Captain!

VMWare Essentials Bundle Plus provides for 6 sockets = 3 servers @ 2 sockets each. Get 3!!!

If you are virtualizing more than 5 servers, consider 3 physical hosts.



Virtual Server Backup Strategies

Snapshots are not your backup strategy!

Software bundle to replicate the VM’s

Replicate to a host offsite for DR

Replicate onsite for quick DR!



Backups and the Cloud

Backup for Dummies by Acronis

Handbook and general requirements

Local and Cloud Models

Cloud-based Virtualization: Virtual Servers as a Service (DRaaS)

Backup Strategy



Backup for Dummies by Acronis

Answers to common questions about backup and recovery.

Ten tips for easier backup and recovery.

How to address the modern data protection challenges caused by virtualization, the cloud and data growth.

http://www.acronis.com/en-sg/blog/posts/introducing-backup-dummies-download-your-free-copy-today


http://www.acronis.com/en-sg/blog/posts/introducing-backup-dummies-download-your-free-copy-today


Handbook and General Requirements

Chapter 4 > E -- Computer Systems

The standing trustee must back up the computer server weekly and the data files daily. A copy of the backup should be stored in a secured location offsite weekly.

Standing trustee may consider a software provider’s offsite backup service, which automates the backup process.

Standing trustee should test the backup system to verify that the images in a paperless or image based system are also being backed up and retained.

The backup diskette, tape, or other media should be tested or rotated periodically to ensure its continued reliability

The standing trustee must develop and maintain a written office disaster recovery plan for the financial and administrative records, as well as for the computer system and data. Off-site storage of backup files for all critical records should be maintained in the event of natural disaster or physical damage to the premises.

https://www.justice.gov/sites/default/files/ust/legacy/2015/05/05/Handbook_Ch13_Standing_Trustees_2012.pdf


https://www.justice.gov/sites/default/files/ust/legacy/2015/05/05/Handbook_Ch13_Standing_Trustees_2012.pdf


Local and Cloud Models

Local◦ Backup to tape◦ Backup to disk◦ Backup to NAS

Cloud◦ Azure – Microsoft’s Backup and storage in their cloud◦ Acronis - $850/yr/vm or 500gb/$400/yr◦ Barracuda Backup – 4TB ~$9500/yr◦ Storagecraft Cloud Services using ShadowProtect◦ eVault◦ Carbonite – Personal desktop solution

Hybrid (Personal cloud)◦ Replicate offsite to another office◦ DRAB (http://www.13drab.com)◦ Replicate to bank or another secure location


http://www.13drab.com)/


Cloud-based Virtualization: Virtual Servers as a Service (DRaaS)

Disaster Recovery as a Service – replication of hosted or virtual server to an offsite 3rd party vendor to provide quick failover in the event of a disaster

VMware vCloud Air Disaster Recovery/IBM Cloud Virtualized Server Recovery/Storagecraft Cloud Services/Veeam Cloud Connect/Barracuda Cloud LiveBoot

DRAB – Private DRaaS



Backup Strategy

Restore Points ◦ 7 days worth of daily incrementals (3-6 restore points

during the day)◦ 15 days of daily consolidated (wraps up incrementals into

a single EOD)◦ 5 weeks of weekly consolidated◦ Months – indefinite?

Test, test test. Test a file restore, test a server restore (if possible in virtual environment), test offsite replicated backups.





What to backup? The Obvious

Bankruptcy data◦ Case software database

◦ External case data files (PDFs)

◦ Case Software and configuration files

Expense Account data◦ Software and license key

◦ Database encrypted export



What to backup? The Not So Obvious

Electronic Disaster Recovery Kit◦ Licenses Microsoft Server and Seat licenses

Others purchased software license Keys◦ Ex: Acrobat, antivirus,

Other Device Activation Keys◦ Ex: Check printer, firewall

Backup solution license, serial numbers or keys

◦ Hard to get copies of software

◦ Encrypted Master Password list



What to backup? System & Software Configuration

◦ Full System state (for bare metal restore)

◦ Server Software configurations Web servers, ftp, email, exchange,

Exchange, IIS, SharePoint

Virtual Machines

◦ Device Configurations Firewall configuration backup and firmware release

Wireless/Router configuration backup

Managed switch configuration – Especially with VLANs

NAS/SAN – especially RAID and logical volume setup

Hypervisor configuration



What to backup?User Data

Email◦ Hosted Mailbox – outsourced to 3rd party

◦ Hosted Temporarily – outsourced POPs/IMAPs

◦ Exchange – centralized mailbox folders

◦ Outlook – decentralized on user computer systems

Desktop Files◦ User Profiles

◦ Documents and Settings



What to backup?Organizational of data

File Shares◦ Evaluate shares from all servers

◦ Backup agents need access to restricted folders

Accounting

Human Resources

System Admin (DR Kit)

Tax Returns

◦ AccessEnum From Microsoft’s Sysinternals tool suite

Scans directories for file permissions



Q&A "Trustees, ask your system managers to answer these questions and show proof

of the answers. Ex: backup status report, list of files and folders backed up. Review the list with you system manager to assure all critical data is backed up and recoverable if you lost everything last night. Ask what is not backed up, and review to make sure you can live without it. Otherwise get it on the backup list."

Any suggestions on how to prepare a backup list? How to prepare a NOT backed up list? I've been looking for some type of software that will prepare/save directory trees so that I can do this, but as yet have not found anything acceptable. Any suggestions here? Sure I can come up with some general details, - servers, drives, folders, etc, but it sounds like you want more than that - as do I. How about some specific suggestions on how to prepare these backup lists. Ideally, I would like a detail listing of everything and be able to flag what is or is not backed up, but I've been looking for the last few years and still haven't found a practical way to do this.



Virtual Server DemoMicrosoft HyperVVMware VSphereOracle VirtualBox



Disaster Mitigation and Recovery



Disaster Mitigation Items

Network enabled UPS units shut down servers gracefully. ◦ For virtualized world, APC has their own appliance you can

download and install to manage your servers and shutdown as needed.

Environmental monitors alert to air flow, temperature, door contacts, humidity, etc.

◦ IT Watchdog 1250

Remote monitoring such as PRTG.

◦ Setup on remote PC to monitor your email exchange



Disaster and Recovery Scenarios

Arbitrary deletion of file/folder discovered months later

SAN or NAS Failure / Server Failure◦ Multi disk failure (RAID with Dual Parity)

◦ Controller failure leading to …

Corruption of Virtual machine(s) over 3-5 days◦ (24-48 hours)

System Infection ◦ Requires a reformat of hard drives

◦ Requires a system rebuild or full system restore

◦ Requires full data restore – Data files and Databases



Types of Disaster Recovery Plans

No disaster plan at all

No disaster plan, but good backup procedures

A disaster plan, with no resources in place

A ‘cold site’ disaster recovery solution

A ‘split site’ or ‘warm site’ disaster recovery solution

A ‘hot site’ disaster recovery solution



No disaster plan at all

No disaster plan; good backup procedures

The absolute minimum companies must do – even the smallest business – to prevent a disaster from wiping out business information is to back up the data on your computers daily and store the backups offsite at a secure archival company. Never store it at employee’s homes.

That way even if your hardware and software is ruined, you can still replace it and load it up with all your irreplaceable data.



A disaster plan, with no resources in place

Once you have a good backup and archival procedure and your critical systems are fault tolerant, the next step is to put together procedures for remote disaster recovery.

This simply means you ask and answer the question, “What do we do if the computer center is utterly destroyed?”

Put your plan is written form and store a copy offsite



A ‘cold site’ disaster recovery solution

A simple yet effective business backup solution, a cold site is simply a reserved area on a data center where your business can set up new equipment in the event of a disaster. This is a popular disaster recovery method because it tends to be less expensive than other options, yet still gives a company the ability to survive a true disaster.



A ‘split site’ disaster recovery solution

If your organization is large enough, it may be feasible to house the IT department across more than one location. In the event of a disaster to one site, operations can then reasonably simply shift to the other site and any new equipment needed could be purchased as necessary as long as the backups were properly maintained. The advantage to this method is it eliminates the need for the major up-front costs of building a dedicated disaster center.

Also know as a ‘warm’ site

As your organization will need to purchase or lease the equipment in the warm site, this option does involve more set-up costs than a cold site, but has the advantage of being able to get your business systems up and running much faster. Even sites with multiple applications can generally be back to full operation within 24 hours.



A ‘hot site’ disaster recovery solution

A hot site is a premium level of disaster recovery where the business IT systems and up-to-date data are duplicated and maintained at a separate data center.

In this scenario, a duplicate computer center is set up in a remote location with communication lines set up and actively copying data at all times. The site has a duplicate of every critical server, with data that is up-to-date to within hours, minutes or even seconds.



Backup Plan

Backup Schedule

Documented Recovery Process

Retention Periods

Recovery Time

Testing and Evidence◦ Real Evidence vs. Misleading Evidence



STACS Memo 8/24/2014

If today at 5 PM I were to delete ALL the files on the network file shares, could you

◦ restore them from your backup?

◦ How much data would be lost?

◦ Is this data loss acceptable?

When was the last time you checked to make sure your backups were actually working?

Trustees, ask your system managers to answer these questions and show proof of the answers.

◦ Ex: backup status report, list of files and folders backed up.

◦ Review the list with you system manager to assure all critical data is backed up and recoverable if you lost everything last night.

◦ Ask what is not backed up, and review to make sure you can live without it. Otherwise get it on the backup list.



Materials

NACTT Disaster Recovery Template


../Supplementals/Emergency Response Disaster Recovery Plan_NACTT_2012.docx