Defense Systems Acquisition LtCol Mike Varmette Capital and Northeast Regional Campus Program Management and Leadership Dept. July 2004.

Defense Systems Acquisition

LtCol Mike VarmetteCapital and Northeast Regional Campus

Program Management and Leadership Dept.July 2004

Why Worry About Risk?

• Over the last decade downsizing, consolidation, shrinking budgets, increasing technological sophistication and shorter development cycles have lead us to examining adverse impacts on projects costs, schedule and performance.

• The built in risk analysis we had with prescriptive standards (i.e. Mil Specs) is now gone as organizations and engineers move to performance standards.

• Proactive Management vice Reactive Management

Management 1974 Peter F Drucker

• The FORD vis SLOAN (GM) Story…”Management is needed not only because the job is too big for any one man to do himself, but because managing an enterprise is something essentially different from managing one’s own property” pg 384

• “Above all, disagreement is needed to stimulate the imagination. One may not need imagination to find the one right solution to a problem. But then this is of value only in mathematics. In all matter of true uncertainty such as the executive deals with – whether his sphere be political, economic, social, or military – one needs creative solutions which create a new situation. And this means that one needs imagination – a new and different way to perceiving and understanding.” pg 473

• “…the effective decision-maker compares effort and risk of action to risk of inaction. There is no formula for the right decision here. But the guidelines are so clear…act if on balance the benefits greatly outweigh cost and risk; and act or do not act; but do not “hedge” or compromise.” pg 476

• “As a specific discipline, management has its own basic problems, its own specific approaches, its own distinct concerns. A manger who understand the discipline of management will still be an effective – and may even be a first-rate – manager with no more than minimum competence in managerial skills and tools. A man who knows only the skills and techniques, without understanding the fundamentals of management, is not a manger ; he is, at best a technician.

• Management is a practice rather than a science. In this, it is comparable to medicine, law and engineering. It is not knowledge but performance. Furthermore, it is not the application of common sense, or leadership, let alone financial manipulation. Its practice is based both on knowledge and on responsibility.”

RISK MANAGEMENTWhat is the issue…

• PM Solutions Center for Business Practices (CBP)• Recent Study based on CMM and PMBOK• Risk Management lowest maturity in Programs

• Risk Documentation the worst at 1.42 • Technical Req Definitions best at 2.58

(www.cbponline.com)• DAU Risk Study showed….

• Strong on “knowledge” • Weak on application

• Plenty of studies showing continued cost overruns

Drucker…”knowledge and on responsibility”

IOCBA

System Development& Demonstration

Production & Deployment

Pre-Systems Acquisition

Systems Acquisition

Operations & Support

C

Sustainment

User Needs & Technology Opportunities

Concept Refinement

TechnologyDevelopment

System Integration

System Demonstration

LRIP Full-Rate Prod & Deployment

DesignReadinessReview

FRPDecisionReview

Sustainment

Disposal

FOC

Relationship to Joint Capabilities Integration & Development System

Initial Capabilities Document (ICD)

Capability Development Document (CDD)

Validated & approved by Validation Authority

• Process entry at Milestones A, B, or C• Entrance criteria met before entering phases• Evolutionary Acquisition or Single Step to

Full Capability

Capability Production Document (CPD)

ConceptDecision

The Defense Acquisition Management Framework

IOCBA

System Development& Demonstration

Production & Deployment

Pre-Systems Acquisition

Systems Acquisition

Operations & Support

C

Sustainment

Concept Refinement

TechnologyDevelopment

System Integration

System Demonstration

LRIP Full-Rate Prod & Deployment

DesignReadinessReview

FRPDecisionReview

Sustainment

Disposal

FOC

ConceptDecision

CURRENT STATUS*BASELINE -COST -SCHEDULE -PERFORMANCEEXECUTION STATUS

PLANS *PROGRAM PLANS *EXIT CRITERIA

RISK MANAGEMENT

ASSESSMENT *COST *SCHEDULE *PERFORMANCE

CURRENT STATUS*BASELINE -COST -SCHEDULE -PERFORMANCEEXECUTION STATUS

PLANS *PROGRAM PLANS *EXIT CRITERIA

RISK MANAGEMENT

ASSESSMENT *COST *SCHEDULE *PERFORMANCE

DoD “old” PolicySelected References to RISK

• DoDD 5000.1, The Defense Acquisition System

– 4.5. Effective Management….tailor considering risk

• DoD 5000.2-R, Mandatory Procedures for MDAPs/MAIS

– Numerous references to RISK....management and mitigation

– 1.2.4.2 Risk reduction in source selection criteria

– 1.4.3.3.2 Cost Estimates include assessment of RISK

– 2.3, 2.5, 2.9 Acquisition Strategy …reduce System-Level risk to acceptable levels…industry bear risks

- 5.2.3.4.3…establish a risk management process

– 7.4 Exit Criteria

• DoDD 5000.4, OSD CAIG

– The CAIG Chair report … include quantitative assessments of risk…

• DoD 5000.4-M, Cost Analysis Guidance and Procedures

– Para 1.E.1.2, … Subsystem Description address risk issues

– Para 1.E.2.0, Risk..PM assess & plan to address/reduce

DoD “new” PolicyMore References to RISK

• 30 Oct 2002 Deputy Sec Def Memo• 3.2 Tailoring…flexible approaches…risk, and complexity• 3.5 Reduced Cycle Time…Evol Acq preferred• 3.14 Knowledge-Based Acquisition…

– Tech, Integration, and manufacturing risk reduced• 3.15 Systems Engineering approach• 3.20 Program Goals…implement management controls

– APB with costs, schedule and performance– No discussion of risk...

• 3.24 Cost Realism…cost risk and monitor (contract)• 3.25 Cost Sharing…undue risk is not imposed (contractor)

Risk Management Policy

• Continually assess program risk• Develop Risk Management approaches prior to decision to

proceed to next phase• Ensure Risk Management Encompasses

– Identification– Tracking– Mitigation– Control

• Ensure equitable and sensible allocation of risk between Government and Industry

• Practice event-oriented management to emphasize prudent risk management.

Risk Management Procedures

• Acquisition Strategy shall consider risk areas.• Establish Risk Management program to identify

and control performance, cost and schedule risks.

• Include Risk reduction measures in cost performance trade offs.

• Monitor Risk throughout each acquisition phase to determine how risks have changed.

Evolutionary Acquisition Risk Management

• Evolutionary acquisition is designed to get new military capabilities from the Lab to the warfighter as quickly as possible.

• In the “old” process, we spoke of cost, schedule, performance. New process we speak of cost, schedule, performance and risk.

• Acquisition programs are taking more risk and it is showing up in operational testing.

Risk Definition

• Risk is a measure of inability to achieve overall program objectives within defined cost, schedule and technical constraints and has two components:– Likelihood of failing to achieve a particular outcome– Consequences of failing to achieve that outcome. For

processes, risk is a measure of the difference between actual performance of a process and the best practice for performing that process

*Risk Mgt Guide for DoD Acquisition, 4th Edition June 2003

DAU PM Tool Kit Feb 2002

Risk Planning

• Develop an organized, comprehensive, and iterative approach

• Identify adequate resources - People and $$• Organize/Train Risk Management IPT

Members• Develop Management Information System• Draft Risk Management Plan

(Format - Risk Management Guide, Chap 5)

Risk Assessment• Identify: Risk Events

– WBS elements analyzed against risk sources/areas

• Analyze: Probability, Consequences– Determination of causes, impacts,

sensitivity, and relationships

• Rate/Prioritize: Risk events for handling

Continuous ProcessContinuous ProcessThroughout Program Life CycleThroughout Program Life Cycle

Risk AssessmentRisk Analysis

• Refine description of risk events• Isolate causes of risk• Measure probability and impact of risk• What criteria should be used to determine

High, Medium, and Low risk?– Probability/Impact based largely on judgement– Impact on quantitative terms, if possible– Tailored to the Product or Process

IDENTIFY

Risk Events:Identification

• People– Users– Relationships– Decision Makers/Authorities– Organizations– Availability– Talent/Skill Level/education– Experience– Motivation/Morale

Risk EventsIdentification

• Process

Requirements Threat

Time/Schedule Cost-Estimation/Control

Design Budget

Logistics Test and Evaluation

Management Legal/Regulatory

Project size/scope Procurement

Management

Test

facilities

Risk Events

Identification

• Technology– Change – New or Obsolete– Adoption/Use– Integration/Interfaces– Team

(Government/Contractor) technology expertise

– Security– Architecture– Scalability

Top CLAWS Program Risk Areas

• Family of Systems Performance Risk

• Software & Algorithm Development & Integration Performance Risk

• Development Testing Approach & Schedule Performance/Schedule Risk

• Weight Budget & Platform Stability Performance Risk

Safety Reqt’s Exceeds Spec

Family of Systems

Safety of Crew in Cab

PRS #2 Availability

Reloader Redesign

Safety impacts Design

11

22

45

42

15

Insufficent Spares

Turret Servo Maturity

DT Duration too short

High Failure Rates

Moderate

642

18 32

28

12

15452232

18

60.6

1

Potential Severity of Consequence (Cf)

0.5

Low

High

•

•

•

•

••

•

•

•

•

0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.7

0.8

0.9

0

11

28

12

Pro

bab

ility

of

Occ

urr

ence

(P

f)

CLAWS Top Risks

Risk Rating Table

MODERATE

LOW

LOW

HIGH

MODERATE

LOW

HIGH

HIGH

MODERATE

HIGH

PROBABILITYOF

OCCURRENCE

LOW

LOW

SEVERITY OF CONSEQUENCES

(IMPACT)

HIGH

Risk EvaluationClassification/Ratings

LOW HIGH

HPHIHPLI

LPLI LPHI

MED HIGH

MEDLOW

H IGH

L O W

3 14 2

LiKELIHOOD

IMPACT

Likelihood and Impact Criteria Definition

The IPT needs to define the criteria for likelihood and impact for evaluating your project.

• Likelihood– Very Likely (Example: “expect this risk greater that one in three chance

to occur”)

– Not Likely (Example: “expect this risk less than a one in three chance to occur”)

• Impact (Criteria for Cost and/or schedule and/or performance and/or other programs)

– Little Impact – Big Impact

RISK HANDLING STRATEGIES

Risk Handling (CAAT)• CONTROL

– Reduce probability of event occurrence(P3I, Reuse S/W, Parallel Design)

• AVOID– Use another path

(Redesign, Eliminate Reqr, Change IOC, COTS)

• ASSUME (Accept)– Make no changes

(Trade Space = Cost, Schedule, & Performance; Risk Reserve)

• TRANSFER– Reduce Impact

(Warranties, FP Contracts, Insurance)

Risk Handling Strategy #1CONTROL: Reduce Likelihood

• DEFINITION: Lowering the frequency of risk event occurrence.

• EXAMPLE: Providing a redundant

power supply for a computer system to increase overall system availability and reducing the number of downing events.

CONTROL: Reduce Likelihood: Tools and Techniques

PEOPLE PROCESS TECHNOLOGY

TRAINING REQUIREMENTS

USE PROVEN TECHNOLOGY

HIRING/PROMOTION DESIGN REVIEWS

MOCK-UPS & PROTOTYPING

COMMUNICATION RIGOR TECHNOLOY MATURATION

LEADERSHIP TEST PROGRAM

REDUNDANT DESIGN

KNOWLEDGE SHARING

TRADE STUDIES

ROBUST DESIGN

MODELING & SIMULATION

OPEN SYSTEMS

MULTI-PHASE DEVELOPMENT

Risk Response Strategy #2CONTROL: Reduce Impact

• DEFINITION: A method of controlling the risk event by softening the effect or impact should the risk event occur.

• EXAMPLE: An aircraft having an auxiliary hydraulic system in the event the primary system fails

Reduce Impact: Tools and Techniques

PEOPLE PROCESS TECHNOLOGY

TRAINING CONTINGENCY PLANNING

USE FAMILIAR TECHNOLOGY

HIRING/PROMOTION LOW RATE PRODUCTION

PROTOTYPING

COMMUNICATION RESERVE BUDGET

TECHNOLOY MATURATION

LEADERSHIP SCHEDULE SLACK

REDUNDANT DESIGN

KNOWLEDGE SHARING

INSURANCE WARRANTEES

IPT REPORTS MULTIPLE DEVELOPMENTS

Risk Response Strategy #3Avoid

• DEFINITION: Avoidance of the risk areas or sources that could possibly lead to the risk event.

• EXAMPLE: Eliminate a requirement to build a subsystem because the technological risk is too high.

Avoid: Tools and Techniques

PEOPLE PROCESS TECHNOLOGY

REORGANIZE REENGINEER (DELETE FROM PROCESS

DELETE REQUIREMENT

HIRING/PROMOTION/FIRE

CHANGE VENDOR REDESIGN

COMMUNICATION CANCEL PROGRAM DIFFERENT TECHNOLOGY

GIVE TO ANOTHER PERSON OR TEAM

BLOCK UPGRADE

USE DIFFERENT PROCESS

Risk Handling Strategy #4ASSUME

• DEFINITION: Acknowledging a future risk event and accepting the potential consequences without any efforts to control it.

• EXAMPLE:Only having “one deep” project team member and consciously not training a backup. ( Either too costly or not enough resources available.)

ASSUME:Tools and Techniques

PEOPLE PROCESS TECHNOLOGY

ACCEPT THOSE GIVEN TO YOU

NO CHANGES ACCEPT THE DESIGN AND PLANNED TECHNOLOGY

Risk Handling Strategy #5Transfer

• DEFINITION: Reduction of risk exposure by reallocation of risk from one part of the system to another or redistributing risk between the Government and the prime contractor or between Government agencies. (Part of the functional allocation process)

• EXAMPLE: The Marine Corps decides to have the Army as lead service to develop a new tracked recon vehicle.

Transfer: Tools and Techniques

PEOPLE PROCESS TECHNOLOGY

ASSIGN TO ANOTHER PERSON

MODULAR DESIGN TRANSFER FUNCTION BETWEEN HARDWARE AND SOFTWARE

GIVE TO ANOTHER TEAM

FUNCTIONAL PARTITIONING

SUBSTITUTE A DIFFERENT COMPONENT

OUTSOURCE TO CONTRACTOR

SUBSTITUTE A DIFFERENT SYSTEM

Quantification Measures - Strategies box

• Strategies to handle risks

• Measures (quantification) related to the risk

• How does Strategy effect Measures– Costs – Schedule– Performance– Other Programs

Risk Monitoring & Reporting

• Systematically track & evaluate performance of identified risk areas & events against established metrics

• Periodic IPT meetings & Integrated Baseline Reviews (IBR) to discuss status

• Defense Acquisition Executive Summary (DAES) & Selected Acquisition Reports (SAR)

• Milestone, Decision, Progress Reviews• Continuous Risk re-assessment to reveal

Unk/Unks, refine Known/Unknowns.

Conceptual Risk Management Reporting System

RISK MANAGEMENT CONCEPT

OTHER

CONTRACTOR

FUNCTIONAL

IPTs

RISKCOORDINATOR

DATABASEMANAGEMENT

SYSTEM

STANDARDREPORTS

AD HOCREPORTS

HISTORICALDATA

SUBMIT DATAFOR ENTRY

REQUEST REPORTS ORINFORMATION (CONTROLLED ACCESS)

REQUEST ORCREATE REPORT

Risk Control

• TRACK AND EVALUATE RISK HANDLING AGAINST METRICS

– TEST AND EVALUATION– EARNED VALUE– TECHNICAL PERFORMANCE

MEASURES

WHEN TO REVIEW RISKS

High risks- weekly agenda items for team meetings

Medium risks – monthly reviews

Low risks – each milestone, large program changes, or when theRisk Plan is redone

Risk Documentation

• Important to Document– Lessons learned spread to other programs– Risk tracking

• Many tools available such as Risk Radar

• Easy to make up your own on on Excel, Access or Word

Risk Management: Deficiencies

• Process weakly structured.• Process too subjective.• Risk likelihood overemphasized and impact

underemphasized.• Risk plans unlinked to plans and milestones• Resources not assigned for risk mitigation.• Inadequate documentation.

Risk Management: Workarounds

• When changes occur, review project risk plan.• Check risks along critical path.• Check high risks outside of critical path.• Check WBS.• Involve all appropriate IPT members.• Ensure program team has expertise and/or

resources to mitigate new risks.

CONCLUSION

• Risk Management techniques can be applied in almost all areas of project management.– Budgets, Schedules, Requirements, Contracts

– all interrelated to Risk Management Plan

• RM techniques must be continuously and iteratively applied.

• This is a team not an individual effort!

Parking Lot Some questions

• Does your program have/had Risk Management Plan ?• Contractor plan or PROGRAM plan ? • Are you following the plan? On all activities?• Do you have a Risk Board/Panel/Group ?

• How often does it meet ? Does it have influence ?• How many risks are you tracking ?• Do you quantify your risks?

• How integrated is your risk process with other tools• Do you quantify risk in your…

• Cost/Budget Estimates … Schedule/Network?• Requirements/Technical requirements matrix

• Do you force issues into your risk process?• What percentage of issues were foreseen?

Why Good Projects Fail Anyway

By Matta and Ashkenas; HBR Sept 03

• Focus on “execution risks” and neglect;• “white space risk” – unknowns• “integration risk” – disparate activities won’t come together

• Suggest a “rapid-results initiative”…spirals!!

• Closing paragraph: “Attempting to achieve complex goals in fast-moving and unpredictable environments is humbling. Few leaders and few organizations have figured out how to do it consistently. … Managers expect they will be able to identify, plan for, and influence all the variables and players in advance, but they can’t. Nobody is that smart or has that clear a crystal ball. They can, however, create an ongoing process of learning and discovery, challenging the people close to the action to produce results – and unleashing the organization's collective knowledge an creativity in pursuit of discovery and achievement.”

Why Hard-Nosed Executives Should Care About

Management TheoryBy Christensen & Raynor; HBR Sept 03

• Understand more than jacket blurbs

• Need to “articulate a theory of causation”

• “Good theories are circumstance contingent”• “Under what circumstances will the use of this theory lead to failure?”

• Recommendations:• Go beyond description, explain what works under what circumstances• … fallacy of “all companies in all situations”• “Correlations that masquerade as causation…”

What Really Works…4+2By Nohria, Joyce, Roberson; HBR July 03

• Findings surprising, most tools no direct causal relationship. What matters is strong grasp of basics.

• Excel in primary management practices…the 4• Strategy – Devise and maintain a clearly stated focus• Execution – Develop/maintain flawless operational execution• Culture – Develop/maintain a performance-oriented culture• Structure – Build/maintain a fast, flexible, flat organization

• Secondary management practice…the plus 2• Talent – Winners hold onto talented employees/develop more• Innovation – Turns out innovative products/service and anticipates disruptive events rather than reacting • Leadership – Choosing great chief executives can raise performance significantly• Mergers and Partnerships – Internally generate growth is essential, but master of mergers and acq can also be winners

Delusions of SuccessHow Optimism Undermines Executive’s Decisions

By Lovallo and Kahneman; HBR July 03

• Lists numerous examples of failures in Industry• Reject “rational risks in uncertain situations”• Propose over optimism from cognitive biases

• errors in way mind processes information• organizational pressures

•Problems• Anchoring – initial plan accentuate the positive • Competitor Neglect – underestimation of negative events• Organizational Pressure – internal competition big incentive to accentuate positives in forecasts

• Optimism in Its Place – a distinction between• functions and positions that

• involve decision making• that promote or guide action

RISK MANAGEMENT ToolsWEB Sites

• Risk Management Guide Feb 2001

http://www.dau.mil/pubs/gdbks/risk_management.asp

• Risk Management Learning Modules

http://center.dau.mil

• PMI Risk Management Special Interest Group

http://www.risksig.com

• Office of Naval Research’s Best Manufacturing Practices (BMP) Program http://www.bmpcoe.org

• Program Management Community of Practice (PMCoP) http://pmcop.dau.mil