Using periodic audits to prevent catastrophic project failure

1 of 30

Dr. Paul Dorsey

Dulcian, Inc.

May 22, 2008

Using Periodic Audits to Prevent Catastrophic Project Failure

2 of 30

The VasaEarly 17th century (1628)“The greatest ship of all time”

3 years to build 2 gun decks with 64 cannons

King Gustavus Adolphus of Sweden set the measurements. 1000 trees were required Triple laminated oak walls (18in/46cm thick) Mast = 190 ft/57m Length = 201 ft/61m

Cost = 5% of Sweden’s GNP

3 of 30

Maiden Voyage

Set sail on a beautiful summer day August 10, 1628 Passed the royal palace of Stockholm Fired proud salutes from cannons Sailed 1400 yards Gust of wind came Ship sank in less than 1 minute

4 of 30

Why is the Vasa relevant?

What sank the Vasa sinks FMIS projects.We are still building Vasas.We can’t stop bad decisions.

We CAN stop ignoring them.If you spend $1M and fail, it is bad.

If you spend $100M and fail, it impacts the whole organization

If you spend $1B and fail, it impacts the country.

If you are going to fail - fail cheap.

5 of 30

6 of 30

The Essence of Project Management

Leading kids on a hikeFrom time to time, climb a treeCheck for obstaclesAdjust directionCall everyone together"LET'S GO!"

7 of 30

Periodic Audits

Critical success factor for failure preventionMust be external

Developers cannot self-assess.Big, substantial effort

Weak audit is worse than useless Provides illusion of safety

8 of 30

Audit Costs

Delay projectExpensive

5%-10% of project costIntrusiveAnnoyingPolitically costly

9 of 30

Team response to audit

Project Manager "Why don't you trust me?" "Waste of time"

Developers "We would rather be coding."

Team may feel threatened… If the team feels threatened, they probably have

reason to be.If the team welcomes the audit….

Sign of professional maturity

10 of 30

Audit Benefits

Early detection of failure If a $300 million project fails after $20 million, that

is a huge savings.Validation of system architectureSecond set of eyesGive team time to take stockMid-project course correction

11 of 30

Auditor Independence

Auditor must be told that there will be no chance for follow-on work. Otherwise the audit is suspect

To enforce independence: No economic attachment to the

results No incentive to skew the results No relationship to the

development team

During the audit: Limit contact with development

team "Stockholm Syndrome" After the audit, auditors can

help with the project plan.

Auditor is the agent of the person who hired him/her (no one else) Only reports to the contract

owner are objective. No professional standard or

certification for IT auditors exists.

Contract creates objectivity.

12 of 30

Audits are never 100% objective

Auditors bring their own biases.There are IT "religions."

.Net vs. Java "Thick database" vs. middle tier logic Service Oriented Architecture (SOA) Repository-based development Business rules Agile

A professional with a different perspective can still detect a "Vasa."

13 of 30

Justifying Audit Cost

An extensive audit requires approximately 10% of the total project cost.

Industry project failure rate is ~ 60%. Example: Audit at the halfway point of a $1 million

project Cost: (50% x 1,000,000) x 10% = $50,000 Benefit: (50%) x 1,000,00) x 60% = $300,000

Given the high failure rate, audits are very cheap insurance.

With other benefits, it is obvious that audits are a good deal.

14 of 30

Finding the right auditor

Not internalNot from the same company as the developersNot dedicated auditors

Must be real developers, DBAs, architects Must have built systems of similar scope and subject

area

15 of 30

The Audit Team

Project Manager Experience in the subject area with projects of similar

scopeDatabase Administrator (DBA)

Experience with same platform and similar database size

Application Architect Skill in the same area (Java, .Net, Oracle, etc.)

Security Specialist Military, financial system or health care experience

16 of 30

Structure of the Audit

Knowledge transfer Deep understanding As if auditor were taking over the project Understand the system before evaluating

Infrastructure audit Evaluate the existing infrastructure to support system Every area needs to be assessed.

Ability to meet current and future user requirements Auditor must understand the requirements

Financial review

17 of 30

Detailed Structure of Audit A. Knowledge Transfer

Allows auditors to understand the entire system architecture as if they were taking over system development.

The following areas should be reviewed for the knowledge transfer portion of the audit:

System Overview Data Model Walkthrough Review/Identify Transaction Use Cases Review User Interface Code

Architecture Review Reporting

Requirements/Architecture Review System Architecture Review System Installation/Upgrade

Mechanism. Internal Control review User Access review Handling system bugs/enhancements Multi-Lingual capabilities Basic System Requirements Process Flows Custom Framework Performance Standards Training

B. Infrastructure Audit Examine from technical soundness perspective. Compare to current industry best practices;

document any discrepancies. System and User Documentation Data Model Audit Database Review User Interface (UI) Architecture Review Distributed System/ETL Audit Security Audit Hardware/Software/Networking Review Backup/Recovery Procedures Appropriateness of system upgrade mechanism

C. Ability to Meet Current/Future Requirements

Examine the current system requirements, identify use cases, and review for suitability, specifically:

Compare documented requirements to the existing use cases and how they are handled.

Assess user satisfaction with the existing system. Are existing backup/recovery procedures sufficient to

meet maximum downtime requirements? Assess system flexibility to meet new requirements.

D. Financial review Review how resources have been spent over the

lifetime of the project including budgeted vs. actual expenditures

18 of 30

Knowledge Transfer

"Seek first to understand." Stephen Covey Learn enough to take over the process:

Architecture Data Model Use Cases Report Audit Configuration Management Internal Controls Documentation Training

System may be bad enough that this is not possible. Do it anyway. Preventing the Vasa from sailing is hard work.

19 of 30

Infrastructure Audit

Compare what was learned in the knowledge transfer with best practices

Each area needs to be assessed: Documentation Data model Database design User interface architecture Security Backup & Recovery Configuration Management Internal Controls

Identify weaknesses in each area: Corrective actions Exposure Controls needed

One mistake can sink the Vasa. System won’t scale Security hole Inflexible design

20 of 30

Ability to Meet Requirements

Requires careful use case documentation Assess user satisfaction

Sit with users Survey Request queue is not a good measure. If system is poor, users

give up. Assess each use case

Functional requirements Performance Downtime

Future requirements Flexibility Deployment

21 of 30

Financial Review

Stewardship Audit If requirements

change, price can balloon.

Does this make sense?

Sunk costs are "somewhat" relevant

Measure decision quality

Financial History

Date Budget $ Spent Mile-stones/

Achieved

Notes

22 of 30

COTS Project Audits

Not very different from custom projectsCOTS projects fail just as often.Review COTS architectureBe careful about how well COTS satisfies

requirements: COTS customizations can be VERY expensive. Customized COTS cannot be upgraded.

23 of 30

Audit Report

2-5 page Executive Summary Report Are we OK?

10-15 page CIO Report Are we OK? Why?

100 page detailed report What we did What we found What needs to be fixed Next steps

24 of 30

Acting on the Audit Report

If report concludes "Vasa…." Get a second opinion Let the development team respond

Sunk costs are sunk costs. The amount of money budgeted for the project is

irrelevant."Upgrade" is a way to change direction without

admitting failure.

25 of 30

Case Study: Ethiopia FMIS

Harvard University ProjectSmall part of major financial reform

$38 M USD over 12 years $3 M USD over 3 years for IT (very frugal)

Harvard was ending the project and wanted to assess quality of system.

Custom development projectDulcian was brought in to do the audit.

26 of 30

Audit Scope

Standard audit As described in this presentation

Total knowledge transfer and team back-up Support for any “what if?” scenarios Total system back-up

27 of 30

Audit Recommendations

Maintain current systemUpgrade system internals

Business rules approach Oracle DBMS Ultra-light Web architecture

28 of 30

Audit Results

Government and Harvard accepted recommendations. Maintain/evolve current system $1.5M USD Redesign architecture $1.5M USD

Dual nature of the audit (audit + handoff) made existing team very uncomfortable. Top three IT people resigned.

29 of 30

Conclusions

Audits don't prevent failure; they just catch failures earlier in the process.

Audits don't replace good design. The audit may only help fail more small projects rather than

one big project. Audits are resource intensive, expensive, annoying,

politically charged. But they are cheaper than not doing them at all in the long

run. Auditors must be kept independent.

No follow-on work Both COTS and custom projects need audits.

30 of 30

Result of Audit

Quite a good design Excellent documentation Very good developer coding Supported current requirements

Some architecture portions needed modification. Database design issues

No Foreign Keys Odd design (inherited from previous team)

Poor performance for parts of system Scalability questions Would not work on Ethiopian internet due to slow/unreliable

connections

31 of 30

Contact Information

Dr. Paul Dorsey – [email protected] Dulcian website - www.dulcian.com

Developer AdvancedForms & ReportsDeveloper AdvancedForms & Reports Designer

HandbookDesignerHandbook

Latest book:Oracle PL/SQL for Dummies

Design Using UMLObject ModelingDesign Using UMLObject Modeling