Project Recovery

Reviving the PhoenixReviving the PhoenixDealing with Failed or Failing ProjectsDealing with Failed or Failing Projects

Tom RethardTom RethardLecturerLecturer

Department of Computer Science and Department of Computer Science and EngineeringEngineering

The University of Texas at ArlingtonThe University of Texas at Arlington

AgendaAgenda

IntroductionIntroduction 4 Case Studies4 Case Studies

Case Study #1 - Mainframe System SoftwareCase Study #1 - Mainframe System Software Case Study #2 - Mission Critical Asset Scheduling Case Study #2 - Mission Critical Asset Scheduling

SoftwareSoftware Case Study #3 - Tax SoftwareCase Study #3 - Tax Software Case Study #4 - Mission Critical Financial Analysis and Case Study #4 - Mission Critical Financial Analysis and

Planning Sales ToolPlanning Sales Tool Symptoms of a project in troubleSymptoms of a project in trouble Causes Causes CuresCures Basic Recovery Process Basic Recovery Process Resources Resources Questions?Questions?

The University of Texas at ArlinThe University of Texas at Arlingtongton Also known as “The Stealth University”Also known as “The Stealth University” Approximately 25,000 studentsApproximately 25,000 students College of EngineeringCollege of Engineering: 4,000: 4,000 Computer Science and EngineeringComputer Science and Engineering: :

10001000 600 undergrad, 400 grad600 undergrad, 400 grad BSCS, BCCSE, BSSE, MSCS, MSCSE, MSSE, BSCS, BCCSE, BSSE, MSCS, MSCSE, MSSE,

MCS, PhDMCS, PhD Accredited by ABET and CSABAccredited by ABET and CSAB

IntroductionIntroduction

60% of our (software) projects fail!60% of our (software) projects fail! Failure means:Failure means:

Never delivered or cancelledNever delivered or cancelled Delivered significantly lateDelivered significantly late Delivered significantly over budgetDelivered significantly over budget Delivered with significantly reduced Delivered with significantly reduced

functionalityfunctionality

CaveatCaveat

Please note that the parties involved in Please note that the parties involved in these case studies are not explicitly these case studies are not explicitly identified herein.identified herein. To protect the innocentTo protect the innocent To protect the guiltyTo protect the guilty To avoid general embarrassment of anyone To avoid general embarrassment of anyone

connected to themconnected to them Please also note that you may be able to Please also note that you may be able to

identify the parties involvedidentify the parties involved

Case Study #1Case Study #1 Mainframe System SoftwareMainframe System Software

ApplicationApplication COTS Application to Monitor and Control COTS Application to Monitor and Control

MVS Storage SubsystemsMVS Storage Subsystems CustomerCustomer

Storage Peripheral Manufacturer, for Storage Peripheral Manufacturer, for sale to MVS Licensees with Large sale to MVS Licensees with Large Storage SubsystemsStorage Subsystems

DeveloperDeveloper Developed In HouseDeveloped In House

Initial SituationInitial Situation

Prototype product developed in Prototype product developed in AustraliaAustralia ““Artificial Intelligence” (i.e., rule-based Artificial Intelligence” (i.e., rule-based

system) running on a PCsystem) running on a PC Performed rudimentary analysis of Performed rudimentary analysis of

mainframe SMF and Logrec datamainframe SMF and Logrec data Too slow to be practicalToo slow to be practical

Resulted from brainstorming session Resulted from brainstorming session between the Aussie branch and the CS between the Aussie branch and the CS department at a major Aussie Universitydepartment at a major Aussie University

Initial Situation (con’t)Initial Situation (con’t)

Marketing proposed an integrated product:Marketing proposed an integrated product: Higher performanceHigher performance Mainframe-basedMainframe-based Common code frameworkCommon code framework ““Add-on” functions (“Several”)Add-on” functions (“Several”)

A year after initial conception, there was A year after initial conception, there was still no coherent set of individual product still no coherent set of individual product ideas, much less requirementsideas, much less requirements

Development team had been assembled Development team had been assembled and was trainedand was trained

Realizing It’s in TroubleRealizing It’s in Trouble

No requirements after a yearNo requirements after a year No sign of agreements on No sign of agreements on

requirementsrequirements Lots of finger-pointing: Marketing vs. Lots of finger-pointing: Marketing vs.

DevelopmentDevelopment Increasing doubts about feasibility of Increasing doubts about feasibility of

the few ideas floating aboutthe few ideas floating about Increasing pressure to start coding (!)Increasing pressure to start coding (!)

Fixing the ProblemFixing the Problem

All but two members of development All but two members of development staff (myself and one other on a three staff (myself and one other on a three month contract) “temporarily” month contract) “temporarily” reassignedreassigned

Two remaining members built a Two remaining members built a simulation model of the MVS storage simulation model of the MVS storage subsystem (in PROLOG) subsystem (in PROLOG) To better understand behaviorTo better understand behavior To (hopefully) find potential requirementsTo (hopefully) find potential requirements

The ModelThe Model

Worked quite wellWorked quite well Was extremely flexibleWas extremely flexible Showed us enough about the hidden Showed us enough about the hidden

behaviors to convince me we behaviors to convince me we shouldn’t be thereshouldn’t be there

I began movement to cancel the I began movement to cancel the projectproject

The Battle to Kill ItThe Battle to Kill It

Some stakeholders had more ego Some stakeholders had more ego than money at stake, so no supportthan money at stake, so no support

VP of development didn’t want to be VP of development didn’t want to be embarrassed, so no supportembarrassed, so no support

Marketing wanted the products to Marketing wanted the products to help sell hardware, so no supporthelp sell hardware, so no support

Ever feel like the lone voice in the Ever feel like the lone voice in the wilderness?wilderness?

The Final OutcomeThe Final Outcome

A new senior exec with a storage A new senior exec with a storage products background at IBM was hiredproducts background at IBM was hired VP of Software Development reported to VP of Software Development reported to

himhim One meeting with the new VP was One meeting with the new VP was

enough to get the project cancelledenough to get the project cancelled I was reassigned to outer SloboviaI was reassigned to outer Slobovia VP of Software Development was VP of Software Development was

moved to “Special Projects”moved to “Special Projects”

Case Study #2Case Study #2 Mission Critical Asset Scheduling Mission Critical Asset Scheduling

SoftwareSoftware ApplicationApplication

““Speed up” Existing Scheduling ModelerSpeed up” Existing Scheduling Modeler CustomerCustomer

A Major Transportation CompanyA Major Transportation Company DeveloperDeveloper

In HouseIn House

Initial SituationInitial Situation

Basic scheduling problem is an NP-Basic scheduling problem is an NP-complete problemcomplete problem ““Schedulers” were a small staff of Schedulers” were a small staff of

mathematiciansmathematicians Project had been running for 2 ½ yearsProject had been running for 2 ½ years No documentationNo documentation 35 developers on staff35 developers on staff LOTS of code, not much functionalityLOTS of code, not much functionality

Initial SituationInitial Situation

Just hired 3Just hired 3rdrd Project Manager Project Manager Several key developers were not trusted by Several key developers were not trusted by

the new PM, and the new PM, and vice versavice versa New PM hired me as a consultant New PM hired me as a consultant

architectarchitect Original solution (from a statement by Original solution (from a statement by

the team member who proposed this)the team member who proposed this) We’ll use a massively parallel computer, an We’ll use a massively parallel computer, an

object database and object-oriented object database and object-oriented analysis, design and coding to solve all our analysis, design and coding to solve all our problems.problems.

Realizing It’s in TroubleRealizing It’s in Trouble

First clue was the PM turnover!First clue was the PM turnover! Second clue was the lack of tangible Second clue was the lack of tangible

progressprogress 33rdrd PM realized there were problems PM realized there were problems

But VP of Development didn’t – or didn’t But VP of Development didn’t – or didn’t want to admit itwant to admit it

About half of the developers knew About half of the developers knew something was wrong, but didn’t something was wrong, but didn’t know whatknow what

RecoveryRecovery

I came in primarily as an architectI came in primarily as an architect Placed myself as a buffer between Placed myself as a buffer between

the various leads and the PMthe various leads and the PM Met with PM on daily basisMet with PM on daily basis Intent to build trust, initially through me, Intent to build trust, initially through me,

within their organizationwithin their organization Spent several month interviewing Spent several month interviewing

key developers and userskey developers and users

RecoveryRecovery

After 6 months, we hadAfter 6 months, we had A real, live, agreed upon architectural A real, live, agreed upon architectural

definition that supported the definition that supported the requirementsrequirements

A parallel computer company that went A parallel computer company that went out of business before delivering, but out of business before delivering, but after being paidafter being paid

An object database company that An object database company that showed no signs of deliveringshowed no signs of delivering

RecoveryRecovery

But we also hadBut we also had An study done with the users on the An study done with the users on the

performance problem:performance problem: 90% of elapsed time was human wait time, 90% of elapsed time was human wait time,

elimination of which would effectively be a elimination of which would effectively be a 10x speedup (NB: cost of $0)10x speedup (NB: cost of $0)

A VP of Development who was still in A VP of Development who was still in denialdenial

One developer (the original proposer) One developer (the original proposer) who continued to sabotage efforts to who continued to sabotage efforts to move the project forwardmove the project forward

The Final OutcomeThe Final Outcome

The 3The 3rdrd Project Manager was fired two Project Manager was fired two weeks after I leftweeks after I left

The saboteur became the 4The saboteur became the 4thth Project Project ManagerManager Scrapped the architectureScrapped the architecture Changed to a vector machine instead of Changed to a vector machine instead of

MPPMPP Most team leads leftMost team leads left Project eventually scrappedProject eventually scrapped

Case Study #3Case Study #3 Tax SoftwareTax Software

ApplicationApplication Final Release of a Tax Software Product Final Release of a Tax Software Product

(COTS)(COTS) CustomerCustomer

A Software Company, for Shipment to A Software Company, for Shipment to Existing CustomersExisting Customers

DeveloperDeveloper In HouseIn House

Initial SituationInitial Situation

Product originally designed as a Product originally designed as a single-user systemsingle-user system

Users discovered networks, and began Users discovered networks, and began to share the database!to share the database! With the expected result of data With the expected result of data

corruptioncorruption Assigned to in-house team (23 people)Assigned to in-house team (23 people)

Intent was to add database sharing and Intent was to add database sharing and stabilize the product as the last releasestabilize the product as the last release

Had a 6 month scheduleHad a 6 month schedule

Realizing It’s in TroubleRealizing It’s in Trouble

Project Manager and DirectorProject Manager and Director No clue until a few weeks before the No clue until a few weeks before the

ship dateship date Never asked why the staff needed to be Never asked why the staff needed to be

so largeso large Team lead requested doubling of his Team lead requested doubling of his

staff (to 46) and said the product staff (to 46) and said the product would be latewould be late

RecoveryRecovery

Dump it in Tom’s lapDump it in Tom’s lap Met with project teamMet with project team

Requested all pertinent docsRequested all pertinent docs Received a one-page list of 22 Received a one-page list of 22

“enhancements”“enhancements” Sought the team’s justification of each Sought the team’s justification of each

“enhancement”“enhancement” Bottom line: lot’s of gold-plating by Bottom line: lot’s of gold-plating by

the team leadthe team lead All but one enhancement was unjustifiedAll but one enhancement was unjustified

RecoveryRecovery

Removed all but two developers from Removed all but two developers from the projectthe project

Remaining two (not including the team Remaining two (not including the team lead) spent two days analyzing the lead) spent two days analyzing the code in light of the real requirementcode in light of the real requirement

One day spent with the developers One day spent with the developers and myself designing the solutionand myself designing the solution

Solution was implemented in 5 daysSolution was implemented in 5 days

The Final OutcomeThe Final Outcome

Product was tested and turned over Product was tested and turned over for shipment two weeks ahead of for shipment two weeks ahead of scheduleschedule

I made a few enemies……I made a few enemies……

Case Study #4Case Study #4Mission Critical Financial Analysis and Mission Critical Financial Analysis and

Planning Sales ToolPlanning Sales Tool ApplicationApplication

Mission-Critical Sales Tool to Analyze Mission-Critical Sales Tool to Analyze Client Needs and Design a Financial PlanClient Needs and Design a Financial Plan

CustomerCustomer A Financial Services and Planning A Financial Services and Planning

CompanyCompany DeveloperDeveloper

A Major Software Consulting FirmA Major Software Consulting Firm

Initial SituationInitial Situation

Original project on fixed-price bid of Original project on fixed-price bid of $3.8M$3.8M

Let to major national consulting firmLet to major national consulting firm Had also completed the “Requirements”Had also completed the “Requirements”

Went through 3 Project ManagersWent through 3 Project Managers Last PMLast PM

Requested, and got, 6 month extension on Alpha Requested, and got, 6 month extension on Alpha Test dateTest date

Extension included bonus to all contractors who Extension included bonus to all contractors who stayed through the end datestayed through the end date

Initial SituationInitial Situation

Final PM produced a status report on Final PM produced a status report on Friday before Alpha Test start: “AOK”Friday before Alpha Test start: “AOK”

On Monday of Alpha Test request On Monday of Alpha Test request another 6 month extension, and another 6 month extension, and more money.more money.

Client terminated the relationshipClient terminated the relationship Project had generated 550 KLOC of Project had generated 550 KLOC of

C++ code at time of terminationC++ code at time of termination

Realizing It’s in TroubleRealizing It’s in Trouble

First observed clue: project was late First observed clue: project was late (the first extension)(the first extension)

Second observed clue: the lie on the Second observed clue: the lie on the status reportstatus report

Third observed clue: the request for Third observed clue: the request for more moneymore money

Immediate First AidImmediate First Aid

Removed the consulting firm from Removed the consulting firm from the premises, but retained some the premises, but retained some “key” contractors“key” contractors

Froze all work and began gathering Froze all work and began gathering the ashesthe ashes

Began search for an independent Began search for an independent project manager who could “finish” project manager who could “finish” the projectthe project

RecoveryRecovery

Client made a good start by Client made a good start by stabilizing the projectstabilizing the project

I had two weeks to study the project I had two weeks to study the project and make a recommendation to the and make a recommendation to the Board of DirectorsBoard of Directors

I was given complete control over the I was given complete control over the project, including hire/fire of project, including hire/fire of contractors and even permanent contractors and even permanent staff (from the project)staff (from the project)

Interesting ObservationsInteresting Observations

Consultant wrote 50 KLOC to produce a Consultant wrote 50 KLOC to produce a generalized report writing function.generalized report writing function.

All “design documents” were done via All “design documents” were done via Rational Rose reverse engineering of Rational Rose reverse engineering of code after it was writtencode after it was written

The “scope control” process noted only The “scope control” process noted only a couple dozen requests for changesa couple dozen requests for changes

Client reps to project meetings were Client reps to project meetings were from VP level or higherfrom VP level or higher

Interesting ObservationsInteresting Observations

There was no usable scheduleThere was no usable schedule There was no source controlThere was no source control

Several “master copies”Several “master copies” Most programmers could not see others’ Most programmers could not see others’

codecode There was no quality assurance group There was no quality assurance group

- or processes- or processes There was absolutely no measurement There was absolutely no measurement

of progress or productivityof progress or productivity

Interesting ObservationsInteresting Observations

Database Architect had not been allowed Database Architect had not been allowed to determine the database architectureto determine the database architecture Programmers told him what they wantedProgrammers told him what they wanted Code revealed that programmers had NO Code revealed that programmers had NO

understanding of relational databasesunderstanding of relational databases Client had been physically excluded from Client had been physically excluded from

the project area, even though it was on the project area, even though it was on their own propertytheir own property

There were not even any project files – just There were not even any project files – just stacks of unorganized paperstacks of unorganized paper

Recommendation to the Recommendation to the BoardBoard

Three possible approachesThree possible approaches Scrap everything and start overScrap everything and start over Keep everything and debug the existing Keep everything and debug the existing

codecode Take the triage approach and minimize Take the triage approach and minimize

our costs (and time), but also cripple our costs (and time), but also cripple long term potentialslong term potentials

Board chose the lastBoard chose the last

TriageTriage

Classical Triage, just like an Classical Triage, just like an emergency room in a hospitalemergency room in a hospital Some of the project can be used as is Some of the project can be used as is

(very little)(very little) A slightly larger portion of existing code A slightly larger portion of existing code

could be fixed, although would be more could be fixed, although would be more difficult to support long termdifficult to support long term

Most of the code was useless due to Most of the code was useless due to scaffolding of internals and would be scaffolding of internals and would be scrappedscrapped

RecoveryRecovery

Reorganized the staff into multiple Reorganized the staff into multiple teamsteams Teams owned one or more functional areasTeams owned one or more functional areas Included a QA teamIncluded a QA team

Empowered the DBAEmpowered the DBA And got him an assistantAnd got him an assistant

Directed teams to take detailed Directed teams to take detailed inventoryinventory

RecoveryRecovery

Established essential processesEstablished essential processes Change control (PVCS Tracker)Change control (PVCS Tracker)

NO changes without approval, including NO changes without approval, including programmersprogrammers

Provided the basis for much of the schedulingProvided the basis for much of the scheduling Source Control (MS SourceSafe)Source Control (MS SourceSafe)

Single master copy of sourceSingle master copy of source Schedule Control (MS Project)Schedule Control (MS Project) Reporting Reporting

RecoveryRecovery

Opened the area to all Client Opened the area to all Client employeesemployees But don’t bother my staff without going But don’t bother my staff without going

through methrough me Maintained a “war room” that contained Maintained a “war room” that contained

weekly progress charts emphasizing the weekly progress charts emphasizing the growth of growth of functionfunction

Directly invited the CEO and all senior Directly invited the CEO and all senior executives to examine anything and executives to examine anything and everythingeverything

RecoveryRecovery

Determined appropriate order of Determined appropriate order of development to allow for incremental development to allow for incremental delivery to QAdelivery to QA

Fixed and/or created functionality in Fixed and/or created functionality in the prescribed orderthe prescribed order

One “new” subsystem was built with One “new” subsystem was built with JAD, using a clerk and a very good JAD, using a clerk and a very good designer/programmer in record timedesigner/programmer in record time

RecoveryRecovery

Actual recovery overallActual recovery overall UneventfulUneventful No real surprisesNo real surprises Relatively low turnover of personnelRelatively low turnover of personnel Entire staff remained focusedEntire staff remained focused Little overtime necessaryLittle overtime necessary

Final OutcomeFinal Outcome

Went to Alpha Test on time (recovery Went to Alpha Test on time (recovery schedule) after about 13 monthsschedule) after about 13 months

Product deployed to 12 sites at Product deployed to 12 sites at acceptance, to 180 sites within 6 acceptance, to 180 sites within 6 monthsmonths

First year savings from use of the First year savings from use of the product indicated a lost savings of product indicated a lost savings of $28.8M from the delays$28.8M from the delays

Final OutcomeFinal Outcome

Client eventually sued the original Client eventually sued the original Consulting Firm for the original Consulting Firm for the original $3.8M and the $28.8M in lost $3.8M and the $28.8M in lost savings.savings.

After several years, the Consulting After several years, the Consulting Firm made a settlement offer that Firm made a settlement offer that amounted to $5M, which was amounted to $5M, which was acceptedaccepted

Symptoms of a Project in Symptoms of a Project in TroubleTrouble

Late Delivery (most common)Late Delivery (most common) Involuntary OvertimeInvoluntary Overtime Non-specific Tasks on ScheduleNon-specific Tasks on Schedule Large Tasks on Schedule (over 24 man-hours)Large Tasks on Schedule (over 24 man-hours) No ScheduleNo Schedule Complaints About “Scope Creep”Complaints About “Scope Creep” High Turnover Rate, Especially ManagersHigh Turnover Rate, Especially Managers SecrecySecrecy Adding ResourcesAdding Resources No Measurements Against Estimates (or No No Measurements Against Estimates (or No

Estimates)Estimates) No Requirements or Design DocumentationNo Requirements or Design Documentation No Reviews or Review Results DocumentedNo Reviews or Review Results Documented

Symptoms of a Project in Symptoms of a Project in TroubleTrouble

PAIN in generalPAIN in general Everybody knows when a project is Everybody knows when a project is

going sour – most won’t admit to it, going sour – most won’t admit to it, except by jumping shipexcept by jumping ship

CausesCauses Too many to enumerate!Too many to enumerate!

Often people issuesOften people issues Seldom technology issues!Seldom technology issues!

Failure to plan, or failure to follow the Failure to plan, or failure to follow the planplan

Lack of risk managementLack of risk management Poor communication, both within and Poor communication, both within and

without the teamwithout the team Hiding of problems (lies)Hiding of problems (lies) Failure to use change control processesFailure to use change control processes

CausesCauses

Failure to use source control processesFailure to use source control processes ““Build it all at one time” approachBuild it all at one time” approach ““Code like hell” approachCode like hell” approach Unk-UnksUnk-Unks Failure to maintain focusFailure to maintain focus Use of unqualified personnel without Use of unqualified personnel without

training themtraining them Wishful thinkingWishful thinking

CausesCauses

Nearly everything you can think of!Nearly everything you can think of! See McConnell’s bookSee McConnell’s book

CuresCures

Only one known:Only one known: Good project management from day oneGood project management from day one

Basic Process for RecoveryBasic Process for Recovery

Stop EVERYTHING!Stop EVERYTHING! No more codingNo more coding No more designingNo more designing No more writingNo more writing Etc.Etc.

Gather the artifacts Gather the artifacts Assess the situationAssess the situation Take a deep breathTake a deep breath

Basic Process for RecoveryBasic Process for Recovery

Plan as if it were a new projectPlan as if it were a new project Give special attention toGive special attention to

CommunicationsCommunications Change controlChange control Risk managementRisk management Personnel issuesPersonnel issues

Report the bad news FIRSTReport the bad news FIRST And immediately!And immediately!

Basic Process for RecoveryBasic Process for Recovery

Be flexibleBe flexible There will be changes in the scopeThere will be changes in the scope You will find many things that were You will find many things that were

hidden from you initiallyhidden from you initially Remember that nobody tries to do a Remember that nobody tries to do a

bad job, but some need help in bad job, but some need help in learning how to do a good job.learning how to do a good job. And that’s the PM’s jobAnd that’s the PM’s job

Lead by exampleLead by example

Bottom LineBottom Line

There is no silver bulletThere is no silver bullet The best recovery manager is a good The best recovery manager is a good

project managerproject manager With a good track recordWith a good track record Often with a high degree of technical Often with a high degree of technical

knowledgeknowledge Stay FOCUSEDStay FOCUSED

How to Get ThereHow to Get There

Experience is the best teacherExperience is the best teacher Education (other people’s experience) Education (other people’s experience)

is the second best teacheris the second best teacher And will help to solidify your own And will help to solidify your own

experienceexperience Professional organizations (Professional organizations (e.ge.g., PMI)., PMI) Formal TrainingFormal Training ReadingReading

ResourcesResources

Rapid DevelopmentRapid Development – Steve McConnell – Steve McConnell And anything else he wrote!And anything else he wrote!

Griffin-Tate Group: Griffin-Tate Group: Project RecoveryProject Recovery UC Davis: UC Davis: Project Recovery Management Project Recovery Management University of Texas at Arlington: University of Texas at Arlington:

Project Recovery TechniquesProject Recovery Techniques Part of our Part of our Advanced Project Management Certificate Advanced Project Management Certificate programprogram

University of Washington: University of Washington: Managing Project ComplexitiesManaging Project Complexities Part of their Part of their Certificate Program in [sic] Senior Project ManagerCertificate Program in [sic] Senior Project Manager

Oklahoma State University/OKC: Oklahoma State University/OKC: Project Recovery ManagementProject Recovery Management

CertificationCertification

ACTP: Association of Certified ACTP: Association of Certified Turnaround ProfessionalsTurnaround Professionals Offer the “ACTP” Certificate – BUT it’s Offer the “ACTP” Certificate – BUT it’s

aimed at Corporate turnarounds, not aimed at Corporate turnarounds, not software project turnaroundssoftware project turnarounds

Others: None foundOthers: None found But there are a number of “Advanced But there are a number of “Advanced

Project Management” Certificate Project Management” Certificate programs that are more appropriateprograms that are more appropriate

Questions?Questions?