Transcript
Lessons Learned from our First Rapid Deployment
Jayne Ashworth
University of Virginia
The University of VirginiaPublic20,834 graduate and
undergraduate students
2,140 research and instructional faculty
10,030 staff200 staff centralized
ITAlmost 400
departmental IT professionals
A Bit About CommunityNew CIO June 2006
Training in Community way of working 1st and 2nd quarter 2007
New way to work: interested people coming together without hierarchy
One component: making strong requests of hierarchy
A Bit About CommunityOne component: making strong requests of hierarchy
Responses to requests:AcceptDeclineCounter-offer
Exchange RolloutSpring 2007What’s so
Health System, two schools have Exchange servers
Desire for inter-operability among schools, departments
Desire for smart-phone useDesire for integrated systemDepartments acquiring servers
Exchange Rollout Spring 2007
Saw opportunity to consolidate servicesto save University moneyto eliminate redundant services
to help others use technology as they needed/wanted
Exchange Rollout Spring 2007
Central IT Resources LimitedCharge minimal fee $5/user/month to subsidize engineers
Support would be from departments alone
Exchange Rollout Spring 2007
IMPLICATIONS
Two non-interoperable calendaring systems would be in use
No/None/Zero/Zip/Nada centralized help from IT Help Desk, IT Tier II support, IT Training, IT Documentation, IT Web support
We Begin to GallopEarly August 2007
Budget cuts announced Some departments backed awayMajor upgrade for January 15, 2008
Concerns about dual calendar systems
We Begin to GallopEarly August 2007
OptionsStopMove forward as plannedModify plan
We Begin to GallopEarly August 2007
DecisionMove forwardMove all of central IT staff to Exchange calendar
We Begin to GallopRequest made to do IT rollout in
SeptemberSeptember = Back-To-School
Publications group swampedUser Support swampedTraining swamped
Counter-offer: date be moved to early October and that date was agreed
We Begin to GallopExchange for central IT
Training worked to develop materialsPublications worked to document
move processRecruited, trained early-adoptersEarly-adopters have accounts in early
October
We Begin to GallopExchange for Central IT
Late October 2007IT staff receive accounts
Macintosh users unhappyUnix users have only Outlook Web Access
Training in Outlook 2007, users have Outlook 2003
We Begin to GallopExchange for Central IT
Late October 2007Outlook email and Outlook calendar difficult for some
Sole trainer became ill: training came to a virtual halt
By mid-November, most central IT staff were working on new system
Run-Away:Exchange for our Users
Early October 2007Current calendaring system would be discontinued January 15, 2008
Force-migration of users starting mid-December
Request for earlier access receivedDate modified to early December to accommodate concerns
Run Away: Exchange to our UsersNovember 2007
Engineers preprare to implement by sharing server hardware with existing email system
Migration website quickly developed: minimal instructions
Online training found
Run Away: Exchange to our UsersSchedule: What’s So for December Accounts available
Finals early December
Semester ends mid-December
Winter break
“J-Term” Existing Calendaring System Shutdown
Spring semester begins
Upgrade to component of enterprise system
Run Away: Exchange to our UsersSchedule: What’s So for December
We did not know:
the telephone system was also changing
Run Away: Exchange to our UsersWhat’s So for December
One contact address: exchange-consult@Virginia.edu
Used for referral and for direct contact
Help Desk would receive calls Existing staff duties assessed and
divided
Cliff Approaches: the RolloutWhat’s So: December 2007
Multiple notifications to people who were being moved: they would have TWO email accounts (both had to be read!)
Accounts delayed 1 day by mail store crash
Questions flooded in
Cliff Approaches: the RolloutWhat’s So: December 2007
Support partners less than happy
Support partners vocal about unhappiness
People missed notification messages
Cliff Approaches: the RolloutWhat we did not know hurt
Instability on serverCaused extensive downtimes (3 – 5 days in some cases)
Problem source difficult to determine
Cliff Approaches: the RolloutWhat we did not know hurt
Open source migration tool had problemsWith some charactersWith nested folders
Cliff
Approaches: the
Rollout
What
we
did
not
know
hurt
Amount
of
people
had stored
was
huge
Mail stored filled
Full mail stores crashed
Failovers failed
Migrations delayed
No migration for local folders
Frustrations rose
Plunging off the CliffEngineers worked more hours than
User SupportIn a group of five, four became ill
with fluOne was driven home because they
became too ill to drive themselvesPatience from our users came, but
they were frustrated, too
Plunging off the Cliff
SURPRISE!!
New Telephone System
Plunging off the cliff New Telephone System
Learned about it early-December
Should have been transparent Happened simultaneously with Exchange migration
Plunging off the CliffNew Telephone System
Implemented mid-December
People could not email, nor their telephones
Voice mail did not work
Volume at Help Desk phenomenal (and they had phone outages …)
Plunging off the cliff Email Support Team working 80 hours/ weeks:
Responding to Exchange-Consult
Training to learn products
Developing documentation
Doing normal duties
Slowing down Mid-January 2008
Old calendaring server shut down
Questions slowedStill had people migrating their email, worked from archived calendar data
Slowing downMid-January 2008
Continued to hear complaintsUsers did not feel they were heard
Project team did a self-evaluation
Slowing DownMid-January 2008
Scheduled half-day session for listeningStructured for learningIncluded CIO, Deputy CIO and project team
Everyone who had account was welcome to attend
Slowing downFebruary 2008
60 people attended (6000 users)mixed group 1/3 users, 1/3 IT, 1/3 support partners
Facilitator had them create lists:What worked, What could have gone better
Concerns slowed after session
Slowing downFebruary 2008
We heard:Smartphone's passwordsMore communication to them needed
Unhappy with timing
and …
Slowing downFebruary 2008
We also heard:Reliable migration tool wantedMacintosh users unhappyMore training/testing needed
… and …
Slowing downFebruary 2008
Simultaneous deployment of phone/Exchange unwise
Lack of local file migration tool problematic
Slowing downWhat We Heard …Not much was newThey felt heard
Slowing downOur Responses
Outside trainer providedContact with Microsoft about
EntourageBetter migration tool found and
purchased
… and …
Slowing downOur Responses
Looking at scheduling issues (there are no good times for changes)
Communications redoubling effortsReimbursed President’s Office for
local files tool and made other tools available
Future directionsChange in response to emergent technology is needed and needs will happen rapidly
Future DirectionsOur users felt this change was wild – we
need to address this perceptions:Work with users to address concerns
dates, testing, training and documentation standards
Utilize outside resources (one-time cost)
Continue to have our people do what they are good at doing
Make powerful requests
Jayne Ashworthjpa@virginia.edu