© 2007 Expedia, Inc. All rights reserved Expedia Business ResiliencyPrepared by Expedia for PEP Conference. All Rights ReservedPage 1 Headquarters Closed:

Expedia Business Resiliency Prepared by Expedia for PEP Conference. All R

ights ReservedPage 1

© 2007 Expedia, Inc.

All rights reserved

Headquarters Closed: Now What? Live-Fire ExerciseLessons Learned

Howard Mannella, CBCPPrincipal Resiliency Strategist

Expedia, Inc.

Prepared for:Partners in Emergency Preparedness

April 14, 2009

Expedia Business Resiliency Prepared by Expedia for PEP Conference. All Rights Reserved Page 2

What we’ll cover

• What we did/how we did it• Success factors• What we learned• What we didn’t learn


Who is Expedia?

• World’s leading OLTC• A global retailer of travel• Broadest reach in the category

– Some of the most popular sites on the Web

• Operations in Americas, EMEA, APAC


Exercise regimes

Walk-through

TableTop

FunctionalSimulation

DisruptiveLive-Fire

Competency Building

Degree of Disruption

Pause/Rewind/Replay

Degree of Validation

Low

Low

High

Low

High

High

Low

HighIndependent variables: scope, element of surprise


Expedia’s Exercises

“Virtual IT”• Exercise regime: Semi-

announced disruptive ‘live-fire’

• Scope: All IT Staff in the Bellevue HQ(with exceptions)

• Scenario: “Building unavailable and inaccessible, travel to Bellevue not an option”

• Timeframe: SOD Thurs - EOD Wed

“Europe Interrupted”• Exercise regime: Semi-

announced disruptive ‘live-fire’

• Scope: All-Hands in the Paris and Munich offices(with very limited exceptions)

• Scenario: “Building closed due to limited fire”

• Timeframe: 4pm Day 1 – EOD Day 2


Exercise objectives

• Assess ability of staff to work and be managed remotely• Assess effectiveness of communication protocols and channels• Assess effect on interactions with business areas, partners,

other locations• Opportunity for managers to think through and act upon an

emergency scenario and manage unexpectedly-dispersed staff• Opportunity for teams to think about and work under

emergency conditions in a safe low-risk environment• (For Paris) Technical validation of new Emergency Work Center• Identify gaps and weaknesses through physical failure• Forcing function for updating playbooks and other emergency

documentation and validating content

Objectives unique to Disruptive Exercises noted with emphasis


Considerations

• Exercises are forcing functions to help/make teams ready– They generate work – before, during, after– Pre-announcements and prep are not ‘cheating’

• Exercises are not supposed to go smoothly– Issues are expected

(and welcomed)– It’s not “Business as Usual”– They are not “Pass/Fail”


Quotes from the Post-Exercise Survey

• Were there business impacts during the exercise?“Yes, but in a positive way.”

• Did you come onto the campus?“Yes, but I didn’t do any IT-related tasks!”

• “Which of CIO’s direct reports do you report to?”“I’m not sure.”

• “I thought the exercise was very valuable!” • “The most ridiculous exercise I ever saw!”


Success factors

Tools and Enablers

Sponsorship and Support

Planning

Building the Buzz

Metrics



• CIO and Europe Leadership solidly behind it• Provided communications to divisions, briefings to

executive leadership



Questions to ask

• What are we trying to accomplish?• Who are the audiences? How do we engage them?• What will we measure? How?

“Do a disruptive exercise” is not a question, it’s an answer



Objective-based selling

Exercise objectives Walk

Thru

Table

Top

Func

Sim

Disrupt

L.F.

Recovery strategy technically sound? X X X X

Do notification/escalation procedures work? X X X X

Is exercise realistic? X X X

Where are the gaps and weaknesses in the plan? X X X

Are business areas adequately prepared? x X X

Business feasible in emergency conditions? x X

Do all staff know their roles ATOD? X

Do all tools and enablers actually work under load? X

“Acid Test”? X



BCP isNotDR

BCP isNot

a checklist

BCP isNot

“WFH-WFS”

Educating managementSponsorship and Support


Planning

• Over six weeks elapsed time per exercise• Working sessions with IT and remote location area

heads• Briefing session at HQ Admin all-hands• Awareness of cultural issues


Tools and enablers

• SharePoint site• Discussion board• Pre-and post-exercise survey• Emergency Notification

for English divisions(phone/email blast)


Building the buzz

• Signage• Staff meetings• Emails• Test phone blasts

(with a challenge)

Enlist the communications experts


Metrics

• Help Desk tickets• Email volumes• IM volumes• Badge access• Problem email distros• Official email thread• Command Center log (multiple)• “Closing the circuit” on escalations and communications


Communication Effectiveness

0%

20%

40%

60%

80%

100%

0:00 0:15 0:30 0:45 1:00 1:15 1:30 1:45 2:00

Phone/Email Blast:• Two-thirds of staff confirmed within 6 minutes• 80% confirmed within 30 minutesNormal Expedia Email:• Confirmation estimated by email replies

(unsolicited)• Actual confirmation not easily tracked for

Exchange aliases

Emergency Notification Service (phone/email blasts)

Normal Expedia channels (email aliases)

Time (minutes)

IT S

taff


Positives• Creativity• Technology and other enablers worked

for the most part• Participation: teams were engaged

and “played” the exercises realistically and professionally

• Teams put significant work intoupdating playbooks, briefing staffand pre-exercise planning

• The Paris Emergency Work Center (EWS)passed Proof of Concept and contract acceptance

• Provided an opportunity for experience, practice and competency, assuring crisper future responses

• Surfaced many gap items and weaknesses for resolution before they are needed


Issues

• Vendor management• Inter-group choreography• Rumor management• Human factors• Business judgment• Cultural differences• WFH/WFS not a complete solution• Human factor gaps need (constant) remediation• There will always be technology “gotcha’s”• Education through failure more effective than

education through jawboningWe learned many things almost impossible to learn from tabletops or simulations


The Bottom Line

Expedia demonstrated

the ability to keep

divisions operating

during short-term

disruptions


Best practices and Lessons Learned

Best Practices:• Make it clear that it’s not about

individual’s performance• Work the communication plan• Begin with the end in mind• Publish success

Lessons Learned:• Big difference between “a

person can work from home” vs. “a division can operate under disrupted conditions”

• Issues increase exponentially with scope

• Live-fire exercises will uncover things that tabletops and simulations cannot

Lessons Not [Yet] Learned:• Socialization of results could have been better• Follow-up could have been better• Make it part of a program


Questions?


Thank you!

Howard [email protected]

© 2007 Expedia, Inc. All rights reserved Expedia Business ResiliencyPrepared by Expedia for PEP Conference. All Rights ReservedPage 1 Headquarters Closed:

Documents

pep conference

rights reservedwho

rights reservedsponsorship

rights reservedquotes

rights reservedquestions

ridiculous exercise

disruptive exercises

emergency documentation