Tuesday330-overthemountainside.ppt

Lessons Learned from our First Rapid Deployment

Jayne Ashworth

University of Virginia

The University of VirginiaPublic20,834 graduate and

undergraduate students

2,140 research and instructional faculty

10,030 staff200 staff centralized

ITAlmost 400

departmental IT professionals

A Bit About CommunityNew CIO June 2006

Training in Community way of working 1st and 2nd quarter 2007

New way to work: interested people coming together without hierarchy

One component: making strong requests of hierarchy

A Bit About CommunityOne component: making strong requests of hierarchy

Responses to requests:AcceptDeclineCounter-offer

Exchange RolloutSpring 2007What’s so

Health System, two schools have Exchange servers

Desire for inter-operability among schools, departments

Desire for smart-phone useDesire for integrated systemDepartments acquiring servers

Exchange Rollout Spring 2007

Saw opportunity to consolidate servicesto save University moneyto eliminate redundant services

to help others use technology as they needed/wanted

Exchange Rollout Spring 2007

Central IT Resources LimitedCharge minimal fee $5/user/month to subsidize engineers

Support would be from departments alone

Exchange Rollout Spring 2007

IMPLICATIONS

Two non-interoperable calendaring systems would be in use

No/None/Zero/Zip/Nada centralized help from IT Help Desk, IT Tier II support, IT Training, IT Documentation, IT Web support

We Begin to GallopEarly August 2007

Budget cuts announced Some departments backed awayMajor upgrade for January 15, 2008

Concerns about dual calendar systems

We Begin to GallopEarly August 2007

OptionsStopMove forward as plannedModify plan

We Begin to GallopEarly August 2007

DecisionMove forwardMove all of central IT staff to Exchange calendar

We Begin to GallopRequest made to do IT rollout in

SeptemberSeptember = Back-To-School

Publications group swampedUser Support swampedTraining swamped

Counter-offer: date be moved to early October and that date was agreed

We Begin to GallopExchange for central IT

Training worked to develop materialsPublications worked to document

move processRecruited, trained early-adoptersEarly-adopters have accounts in early

October

We Begin to GallopExchange for Central IT

Late October 2007IT staff receive accounts

Macintosh users unhappyUnix users have only Outlook Web Access

Training in Outlook 2007, users have Outlook 2003

We Begin to GallopExchange for Central IT

Late October 2007Outlook email and Outlook calendar difficult for some

Sole trainer became ill: training came to a virtual halt

By mid-November, most central IT staff were working on new system

Run-Away:Exchange for our Users

Early October 2007Current calendaring system would be discontinued January 15, 2008

Force-migration of users starting mid-December

Request for earlier access receivedDate modified to early December to accommodate concerns

Run Away: Exchange to our UsersNovember 2007

Engineers preprare to implement by sharing server hardware with existing email system

Migration website quickly developed: minimal instructions

Online training found

Run Away: Exchange to our UsersSchedule: What’s So for December Accounts available

Finals early December

Semester ends mid-December

Winter break

“J-Term” Existing Calendaring System Shutdown

Spring semester begins

Upgrade to component of enterprise system

Run Away: Exchange to our UsersSchedule: What’s So for December

We did not know:

the telephone system was also changing

Run Away: Exchange to our UsersWhat’s So for December

One contact address: [email protected]

Used for referral and for direct contact

Help Desk would receive calls Existing staff duties assessed and

divided

Cliff Approaches: the RolloutWhat’s So: December 2007

Multiple notifications to people who were being moved: they would have TWO email accounts (both had to be read!)

Accounts delayed 1 day by mail store crash

Questions flooded in

Cliff Approaches: the RolloutWhat’s So: December 2007

Support partners less than happy

Support partners vocal about unhappiness

People missed notification messages

Cliff Approaches: the RolloutWhat we did not know hurt

Instability on serverCaused extensive downtimes (3 – 5 days in some cases)

Problem source difficult to determine

Cliff Approaches: the RolloutWhat we did not know hurt

Open source migration tool had problemsWith some charactersWith nested folders

Cliff

Approaches: the

Rollout

What

we

did

not

know

hurt

Amount

of

mail

people

had stored

was

huge

Mail stored filled

Full mail stores crashed

Failovers failed

Migrations delayed

No migration for local folders

Frustrations rose

Plunging off the CliffEngineers worked more hours than

User SupportIn a group of five, four became ill

with fluOne was driven home because they

became too ill to drive themselvesPatience from our users came, but

they were frustrated, too

Plunging off the Cliff

SURPRISE!!

New Telephone System

Plunging off the cliff New Telephone System

Learned about it early-December

Should have been transparent Happened simultaneously with Exchange migration

Plunging off the CliffNew Telephone System

Implemented mid-December

People could not email, nor their telephones

Voice mail did not work

Volume at Help Desk phenomenal (and they had phone outages …)

Plunging off the cliff Email Support Team working 80 hours/ weeks:

Responding to Exchange-Consult

Training to learn products

Developing documentation

Doing normal duties

Slowing down Mid-January 2008

Old calendaring server shut down

Questions slowedStill had people migrating their email, worked from archived calendar data

Slowing downMid-January 2008

Continued to hear complaintsUsers did not feel they were heard

Project team did a self-evaluation

Slowing DownMid-January 2008

Scheduled half-day session for listeningStructured for learningIncluded CIO, Deputy CIO and project team

Everyone who had account was welcome to attend

Slowing downFebruary 2008

60 people attended (6000 users)mixed group 1/3 users, 1/3 IT, 1/3 support partners

Facilitator had them create lists:What worked, What could have gone better

Concerns slowed after session

Slowing downFebruary 2008

We heard:Smartphone's passwordsMore communication to them needed

Unhappy with timing

and …

Slowing downFebruary 2008

We also heard:Reliable migration tool wantedMacintosh users unhappyMore training/testing needed

… and …

Slowing downFebruary 2008

Simultaneous deployment of phone/Exchange unwise

Lack of local file migration tool problematic

Slowing downWhat We Heard …Not much was newThey felt heard

Slowing downOur Responses

Outside trainer providedContact with Microsoft about

EntourageBetter migration tool found and

purchased

… and …

Slowing downOur Responses

Looking at scheduling issues (there are no good times for changes)

Communications redoubling effortsReimbursed President’s Office for

local files tool and made other tools available

Future directionsChange in response to emergent technology is needed and needs will happen rapidly

Future DirectionsOur users felt this change was wild – we

need to address this perceptions:Work with users to address concerns

dates, testing, training and documentation standards

Utilize outside resources (one-time cost)

Continue to have our people do what they are good at doing

Make powerful requests

Jayne [email protected]