Levers of Control along with other Management Models & Risk Management May 04 John Driessnack.

Levers of Control along with other

Management Models&

Risk Management

May 04

John Driessnack

Outline

• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Driessnack Framework View

•Risk Management … Overview of Theory and Models• Sources on the WEB … lots of help/ ARQ articles • DoD Policy … more emphasis, but can avoid!• Summary• Possible Discussions …

• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV – need for RPI and TPI• Tools (databases) …

Management 1974

Peter F Drucker• The FORD vis SLOAN (GM) Story…”Management is needed not only because the job is too big

for any one man to do himself, but because managing an enterprise is something essentially different from managing one’s own property” pg 384

• “Above all, disagreement is needed to stimulate the imagination. One may not need imagination to find the one right solution to a problem. But then this is of value only in mathematics. In all matter of true uncertainty such as the executive deals with – whether his sphere be political, economic, social, or military – one needs creative solutions which create a new situation. And this means that one needs imagination – a new and different way to perceiving and understanding.” pg 473

• “…the effective decision-maker compares effort and risk of action to risk of inaction. There is no formula for the right decision here. But the guidelines are so clear…act if on balance the benefits greatly outweigh cost and risk; and act or do not act; but do not “hedge” or compromise.” pg 476

• “As a specific discipline, management has its own basic problems, its own specific approaches, its own distinct concerns. A manger who understand the discipline of management will still be an effective – and may even be a first-rate – manager with no more than minimum competence in managerial skills and tools. A man who knows only the skills and techniques, without understanding the fundamentals of management, is not a manger ; he is, at best a technician.

• Management is a practice rather than a science. In this, it is comparable to medicine, law and engineering. It is not knowledge but performance. Furthermore, it is not the application of common sense, or leadership, let alone financial manipulation. Its practice is based both on knowledge and on responsibility.”

Delusions of SuccessHow Optimism Undermines Executive’s Decisions

By Lovallo and Kahneman; HBR July 03

• Lists numerous examples of failures in Industry• Reject “rational risks in uncertain situations”• Propose over optimism from cognitive biases

• errors in way mind processes information• organizational pressures

•Problems• Anchoring – initial plan accentuate the positive • Competitor Neglect – underestimation of negative events• Organizational Pressure – internal competition big incentive to accentuate positives in forecasts

• Optimism in Its Place – a distinction between• functions and positions that

• involve decision making• that promote or guide action

Space Systems Development Growth Analysis5

SBIRS High Quantitative Framework (A Point in Time)

1996 Estimate

$3.2B1997

President’sBudget$2.6B

MPC $2.4B Award

$2.1B1

2

3

4

5

6

Bill

ions

of

Th

en Y

ear

Dol

lars

SBIRS High EMD CostsEstimate, PB, Bid, Award and EAC TY$

Note: PB - President’s Budget; MPC - Most Probable Cost; EAC - Estimate at Completion

(1) $6.2B from 29 March 2001 SBIRS High Program Office Estimate (POE). Includes 3600, 3020 and SPO Cost funds (satellites 3,4, & 5)

Other Sources: ABIDES; Source Selection Document; BAH analysis

ContractorBid

$1.6B

Mar 2002POE

Scope$1.3B

Variance$3.3B

Total = $6.2B

Growth Attributed to

Variance

Added Scope

S&V

• REQ GEN/TRANS

• JET/SEIT

• RISK

• BASELINE RESTRUCTURE

• OTHER ECPS

• ESTIMATING ERRORS

•INCOMPLETE INFORMATION

• PRICE INCREASES

•CHANGES FROM JET/SEIT

• CONTRACTOR SCHEDULE CHANGES

• RISK ESTIMATION

Why Good Projects Fail Anyway

By Matta and Ashkenas; HBR Sept 03

• Focus on “execution risks” and neglect;• “white space risk” – unknowns• “integration risk” – disparate activities won’t come together

• Suggest a “rapid-results initiative”…spirals!!

• Closing paragraph: “Attempting to achieve complex goals in fast-moving and unpredictable environments is humbling. Few leaders and few organizations have figured out how to do it consistently. … Managers expect they will be able to identify, plan for, and influence all the variables and players in advance, but they can’t. Nobody is that smart or has that clear a crystal ball. They can, however, create an ongoing process of learning and discovery, challenging the people close to the action to produce results – and unleashing the organization's collective knowledge an creativity in pursuit of discovery and achievement.”

Why Hard-Nosed Executives Should Care About

Management TheoryBy Christensen & Raynor; HBR Sept 03

• Understand more than jacket blurbs

• Need to “articulate a theory of causation”

• “Good theories are circumstance contingent”• “Under what circumstances will the use of this theory lead to failure?”

• Recommendations:• Go beyond description, explain what works under what circumstances• … fallacy of “all companies in all situations”• “Correlations that masquerade as causation…”

What Really Works…4+2By Nohria, Joyce, Roberson; HBR July 03

• Findings surprising, most tools no direct causal relationship. What matters is strong grasp of basics.

• Excel in primary management practices…the 4• Strategy – Devise and maintain a clearly stated focus• Execution – Develop/maintain flawless operational execution• Culture – Develop/maintain a performance-oriented culture• Structure – Build/maintain a fast, flexible, flat organization

• Secondary management practice…the plus 2• Talent – Winners hold onto talented employees/develop more• Innovation – Turns out innovative products/service and anticipates disruptive events rather than reacting • Leadership – Choosing great chief executives can raise performance significantly• Mergers and Partnerships – Internally generate growth is essential, but master of mergers and acq can also be winners

Driessnack

Contract Cost Track

$M (TY)

* 3 PIPs & 96 RS* 38 SRS & 1 TIP* 7 SRS & 15 GRS

(LRIP)

50

100

200

150

0

ContractSCP

OriginalContract

CAIV TODAYPre-CAIV

* 3 PIPs & 96 RS* 38 SRS & 1 TIP* 7 SRS & 15 GRS

(LRIP)

* 3 PIPs & 150 RS** 41 SRS & 1 TIP

** 7 SRS & 15 GRS (LRIP)

* 3 PIPs & 150 RS

$172M * $116M

$205M

$85M

$136M

* 3 PIPs & 150 RS* 41 SRS& 1 TIP

* 7 SRS & 15 GRS (LRIP)

$13M Award Fee

$19M O&M Options

$19M (41SRS & 1 TIP)

$65M ECPs Cost Growth

Scope Changes #

$85M

$17M Award Fee

$19M O&M Options

$19M (38SRS & 1 TIP)

$85M

$19M O&M Options

$12M Award Fee

$30M O&M Options

$14M**

$65M ECPs

Cost Growth Scope Changes

$85M

$17M Award Fee

$19M O&M Options

$19M (38SRS & 1 TIP)

$8M VAC

$232M$245M

$19M ORD Requirements

# $65M ECP Changes; 50% overrun, 50% scope change consisting of S/W growth, PMO and NR engineering

$29M Award Fee

Driessnack

Schedule History (as briefed on 3 May to DAE)

285 days

405 days

J F M A M J J A S O N D1998



S O N D1997

J F M A M J J A S O ND2001

XPS Impact

TRANSMIT S/WIAT & C

SYSTEM TEST / TSATMOT & E IOC & MS III

(OBJECTIVE)

WAHIAWA NORFOLK

SHIP INSTALLTIME

2002

J FM A

2 MONTHMARGIN

AS OF 2/00

4/99

AS OF PDR

GOVT CONFIDENCE TESTADDED (GREEN)

210 days

210 days

CONTRACT AWARDAS OF11/97

SIGONELLA

Driessnack

-16

-14

-12

-10

-8

-6

-4

-2

0

2

Jan Apr Jul Oct Jan Apr Jul Oct Jan

Cost "A Leg"

Schedule

Cost "B Leg"

GBS EVMS EvolutionCumulative Dollar Variance

Cu

m V

aria

nce

($M

)

CAIV Mid-CourseCorrection

B Leg WBSA Leg WBS

As of Jun 00 Rebaseline

$-5M CV

$-5.5M SV

98 99 00

Delay PACOM

(3 Months)

ATR Delayed

(3 Months)

-$13.1M A Leg CV

C Leg WBSUpdateEVM

Simons, 94; Figure 1

Simons, 94; Table 1

Simons, 94; Table 4

Simons - Levers of ControlAn integrated approach is required

BusinessStrategy

Risks to Be Avoided

CoreValues

Strategic Uncertainties

CriticalPerformance

Variables

- Positive- mindfulness- process - innovation

BELIEF SYSTEM(Perspective - commitmenton grand purpose)

- Negative- mindless- outcomes- efficiency

BOUNDARY SYSTEM(Position - stakingout the territory)

DIAGNOSTIC CONTROL SYSTEMS(Plan - getting the Job done)

INTERACTIVE CONTROL SYSTEM(Patterns in Action - positioning for tomorrow)

Defines the Domain

Internal Controls

Driessnack

DoD Weapons AcquisitionPM Framework Measures

Costs, Schedule, Performance

Product Process

Defined - Stability Repeatable Accurate

Technology Challenge People Tools

Training WBS/EV Experience Integrated

Management and Leadership – over Periods

Pro

du

cer - Po

litics

Driessnack

Driessnack Framework View(visual – not really 5x3x4)

Product

Producer

Process

People

Politics

Cost Sched Perf

Belief Boundary DiagnosticInteractive

Visual Representation

What Period – look over time(time phasing different for each program)

Assess the • Capability Maturity• Risks/Opportunities

Capability Maturity ModelGeneric view of levels

• Level 1: Initial/Performed– Simply performing processes in a manner that is often ad hoc– Competence/heroics of individuals doing the work drive activity

• Level 2: Managed/Repeatable– Processes managed, they are planned, performed, monitored,

and controlled for individual projects and groups

• Level 3: Defined– Defines processes and tailors them based on organization set of

standards. Deviations beyond allowed tailoring is documented

• Level 4: Quantitatively Managed– Control processes using statistical and other quantitative

techniques. Process performance is understood

• Level 5: Optimizing– Continually improved based on understanding of variations

Outline

• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Driessnack Framework View


• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV – need for RPI and TPI• Tools (databases) …

Driessnack

Acquisition Review QuarterlySpecial Edition on Risk - Spring 2003

• Copy in Print or DAU web page (www.dau.mil)• Editors – LtCol John Driessnack & Noel Dickover

• My two cents- quotes from article

• …implementation of risk management is often shallow and not well integrated with other program management tools• Transparency relative to program status is going to increase as each service expands its information systems, …• The emphasis on risk, realism, and knowledge-based acquisition has the potential to truly transform the way we view acquisition programs.• … move beyond standard program baselines with point estimates…transparency into the risks in a program by exposing the range in the estimate, given the risks involved, …

•

http://www.dau.mil/

Driessnack

Acquisition Review QuarterlySpecial Edition on Risk - Spring 2003

• Article from PMCoP Risk Community Members

• 2nd – Research…“there was no correlation found between cost and schedule length. In addition, we did not find the anticipated connection between cost growth and schedule growth…”• 3rd – Overview that asks you questions about Managing Risk in a Program Office Environment • 4th – Seven specific issues relative to contracts and risk program • 5th – An approach that breaks out the business (project) and performance (product) side of decision supports• 6th – Industry view with PMI PM Body of Knowledge (PMBOK) and defense extension - comparisons• 7th –Normalize your measures with an Index to Measure a System’s Performance Risk • 8th – Army’s Health Hazard Assessment Program – one of many views of a program

Driessnack

Driessnack

Driessnack

Break Out One Step Added Implementation

Driessnack

Driessnack

DAU PM Tool Kit Feb 2002

Driessnack

Proposed Risk Management ProcessRisk CoP Community Effort

Prioritized List

AvoidTransferAssume

Quantify Some

Qualify All

Identify

Select how to Handle

Assess ListMonitor

status toplan

Mitigate

Plan for eachProgram byPhase/Evolution

Implementation through IPPD

PeriodicUpdate

Raw List

Integrate Into Program Plans& Control Toolsas necessary

Update asneeded

Risk/Opportunity Analysis

Negative

CriticalSeriousModerateMarginalMinorLimited Value

Some Value

Moderate Value

High Value

Very

High Value

Positive -- Consequence --

Remote

Unlikely

Likely

Highly Likely

Near Certainty

Pro

bab

ilit

y

Need Unknown and Effect

Driessnack

Some Analysis Usingthe Triangular Distribution

Average (expected) cost = (low + most likely + high) / 3(70 + 100 + 180) / 3 = 350 / 3 = 116.7

RelativeLikelihoodofOccurring

Possible Element Costs

Expected Cost = 116.7

70 100 180

Driessnack

Logic of Probabilistic Branch

Test Unit

Start: 8/10/02 ID: 5

Finish: 9/3/02 Dur: 25 d

Res:

FIXIT

Milestone Date: Tue 9/3/02

ID: 6

Retest


ID: 7

Finish


ID: 8

Succeed 70% Branch

Fail 30% Branch

New Activities

Driessnack

Driessnack

Driessnack

DoD “old” PolicySelected References to RISK

• DoDD 5000.1, The Defense Acquisition System– 4.5. Effective Management….tailor considering risk

• DoD 5000.2-R, Mandatory Procedures for MDAPs/MAIS – Numerous references to RISK....management and mitigation

– 1.2.4.2 Risk reduction in source selection criteria

– 1.4.3.3.2 Cost Estimates include assessment of RISK

– 2.3, 2.5, 2.9 Acquisition Strategy …reduce System-Level risk to acceptable levels…industry bear risks

- 5.2.3.4.3…establish a risk management process

– 7.4 Exit Criteria

• DoDD 5000.4, OSD CAIG – The CAIG Chair report … include quantitative assessments of risk…

• DoD 5000.4-M, Cost Analysis Guidance and Procedures– Para 1.E.1.2, … Subsystem Description address risk issues

– Para 1.E.2.0, Risk..PM assess & plan to address/reduce

Driessnack

DoD 5000.2-R Exit Criteria

“Phased-specific exit criteria normally track progress in important technical, schedule, or management risk areas. Unless waived or modified by the MDA, exit criteria must be substantially satisfied for the program to continue with additional activities within an acquisition phase or to proceed into the next acquisition phase, depending on the decision with which they are associated. Exit criteria shall not be part of the APB and are not intended to repeat or replace APB requirements or the entrance criteria specified in DoD Instruction 5000.2. They shall not cause program deviations. The DAES (see paragraph C7.15.3. and Appendix 1) shall report the status of exit criteria.”(5000.2-R, Para 7.4)

New 5000.2 - no mention of risk in criteria

Driessnack

Program Baselines

Threshold Objective

Performance

Cost

Schedule

Minimum acceptable level which will meet user needRDT&E, MilCon, Proc & AUPC, PAUC

Objective + 10%

6 Months beyond objective date (3 months for ACAT IA)

Cost effective increment in operational capability above threshold

Planned cost to meet program objectives

Planned event dates to meet program objectives

Does not represent the RISK on a ProgramNo baseline for, or tracking of Risk

New 5000.2 – no suggested 10% or 6 months

Driessnack

DoD “new” Policy References to RISK – 22 to 22!

• 5000.1– 4.3.1 Flexibility …no one best way– 4.3.2 Responsiveness …time phased capability – 4.3.4 Discipline …program goals for minimum number…– 4.5 Streamlined and Effective Management …decentralize to maximum extent

practicable– E1.5 Cost Realism…proposal that are realistic– E1.6 Cost Sharing…undue risk is not imposed (contractor)– E1.14 Knowledge-Based Acquisition…

• Tech, Integration, and manufacturing risk reduced– E1.21 Program Stability …realistic program schedules.– E1.27 Systems Engineering approach

• 5000.2 – 19 references to Risk

Risk and Realistic in enough paragraphs?

DoD Acquisition Guide ..Draft Risk Areas

5.3.12.2 ESOH Risk Management5.5.2 Management Activities5.5.2.1 Risk Management

7.5 Human Systems Integration (HIS) Strategy, Risk, and Risk Mitigation

9.4 Acquisition Protection Strategy for PMs9.4.4 Risk Management

11 Decisions, Assessments, and Periodic Reporting • The review associated with each decision point typically

addresses program progress and risk, affordability, program trade-offs, acquisition strategy updates, and the development of exit criteria for the next phase or effort.

12 Program Management Activity12.4 Risk Management

PROGRAM SUCCESS PROBABILITY SUMMARY

Program Success(2)

Program Requirements (3)

Program Execution

Contract Earned Value Metrics (3)

Program “Fit” in Capability Vision (2)

Program Parameter Status (3)

DoD Vision (2)

Transformation (2)

Interoperability (3)

Army Vision (4)

Current Force (4)

Stryker ForceTesting Status (2)

Program Risk Assessment (5)

Contractor Performance (2)

Program Resources

Budget

Contractor Health (2)

Manning

Program Advocacy

OSD (2)

Joint Staff (2)

War Fighter (4)

Army Secretariat

Congressional

Industry (3)

Fixed Price Performance (3)

Other

Program “Smart Charts”

Program Scope Evolution

Sustainability RiskAssessment (3)

Joint (3)

Technical Maturity (3)

Legends:Colors: G: On Track, No/Minor Issues Y: On Track, Significant Issues R: Off Track, Major Issues Gray: Not Rated/Not Applicable Trends: Up Arrow: Situation Improving (number): Situation Stable

(for # Reporting Periods) Down Arrow: Situation Deteriorating

PEOPEOXXXXXX COL, PM Date of Review: dd mmm yy

ProgramProgramAcronymAcronym

ACAT XXACAT XX

INTERNAL FACTORS/METRICS EXTERNAL FACTORS/METRICS

Program Life Cycle Phase: ___________

Future ForceInternational (3)

Force Management Risk

Definition: Challenge of sustaining personnel,

infrastructure and equipment

Risk Mitigation Examples

• Manage careers and rotations

• Modernize infrastructure and facilities

• Training, spares and overall readiness

Future Challenges Risk Definition: Challenge of dissuading, deterring,

defeating longer-term threats Risk Mitigation Examples

Experiment with new concepts, capabilities and organizational designs

Investing in transformational capabilities for portions of the force

Foster a spirit of innovation and risk taking

Operational Risk Definition: Challenge of deterring or defeating

near-term threats Risk Mitigation Examples

Plan and prosecute war on terror Elevate role of homeland defense Develop forward deterrence posture Enhance operational capabilities with Allies

Balanced Scorecard Approach(Proposed DoD Scorecard Areas)

Institutional Risk Definition: Challenge of improving efficiency

represented by unresponsive processes, long decision cycles, segmented information, etc.

Risk Mitigation Examples Modernize financial management systems and

approaches Acquisition excellence initiatives Improve planning and resource allocation

Driessnack

Intro Summary

• Lots of information….

• Several models to follow…don’t be constrained• think about how to incorporate “opportunity” • how to integrate across broad framework

• Get off point solutions…understand “range”• you pick the confidence level…not the technician

• Policy is not specific…may be changing

• “Realistic” is the new buzz word…use it !

Outline

• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Defense Acquisition View


• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV• Tools (databases) …

Driessnack

Parking Lot Some questions

• Does your program have/had Risk Management Plan ?• Contractor plan or PROGRAM plan ? • Are you following the plan? On all activities?• Do you have a Risk Board/Panel/Group ?

• How often does it meet ? Does it have influence ?• How many risks are you tracking ?• Do you quantify your risks?

• How integrated is your risk process with other tools• Do you quantify risk in your…

• Cost/Budget Estimates … Schedule/Network?• Requirements/Technical requirements matrix

• Do you force issues into your risk process?• What percentage of issues were foreseen?

RISK IDENTIFICATION:Output-risk events

Use the yellow cards. Write one risk per card. Use complete sentencesDo not discuss with your team…yet ( take about 20 Minutes)

WORK HERE

1. I dentify Clearly Describe Risk Event

Work Package #________

2. Analyze

Very Likely

Not Likely

Big I mpact Little I mpact

Risk Priority # ________

Risk Areas People_____ Process_____ Technology_____

1

2

3

4

Long Term ______ Short Term _____ Program Phase(s) and/ or Expected Date(s) of risk fi rst occurrence ___________________________________

I nternal Control___ _ External Control_______ Are the causes of the risk event within or outside the control of program team?

WORK HERE

Also check if The risk event Is PEOPLEPROCESS ORTECHNOLOGY

Driessnack

Driessnack

Driessnack

UNCLASSIFIED

UNCLASSIFIED

48

— ILS Software— Supportability Engineering— Training— Peculiar Support Equipment— Contractor Logistics Support

B

B

B

B

B

A

A

A

A

A

— Launcher Software— Launcher Transporter/

Missile Round Pallet

BB

AA

THAAD 02T-1036.02a

High- State-of-the-art research is required, and failure is likely and will cause seriousdisruption of program schedule and/or degradation of system performance even with special attention from contractor and close government monitoring

Moderate- Major design changes in hardware/software may be required with moderateincrease in complexity. If failure occurs, it will cause disruption in scheduleand/or degradation in performance. Special attention from contractor andclose government monitoring can overcome the risk.

Low- Existing hardware is available and/or proven technology application can overcome risk. Failure is unlikely to cause disruption in program schedule or degradation in system performance. Normal efforts from contractor/government can overcome risk.

20 Oct 2003

= Before Mitigation

= After Mitigation

B

A

THAAD Development Program Risk Assessment (U)

Risk reduced from moderate to low since contract award

THAAD System

Weapons Systems Engineering Integration Team (WSEIT)

System Test & Evaluation

C2/BM Missile

— System Engineering— Integration & Test— Software Management— Battle Management— Operations Management— Communications— System Support— Operator System Interface— Embedded Training— C2/BM Hardware— C2/BM Segment I&T

Environment (BSITE)

ILS

Launcher

Program Mgt

RadarB A

B

B

B

B

B

B

B

B

B

B

B

B

A

A

A

A

A

A

A

A

A

A

A

A

B A

AB

AB

AB

B A

B A AB

B A

— Product Test Equipment— Mission Software— Lethality— Missile Fore-Body— Missile Mid-Body— Range Safety Equipment— Ordnance Initiation System/

Flight Termination System— Seeker— Mission Computer— Inertial Measurement Unit— Two Axis Rate Sensor— Interstage & Booster Intr— Flare & Aft Skirt— Canister & MR Assembly— Booster Motor— Thrust Vector Actuator— Divert & Attitude Control System— Communications Systems— Power System & Interconnects— Instrumentation & Telemetry— Systems Engineering

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

B

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

A

— Radar Software— Antenna Equipment— T/R Module— Electronics & Shelters— Systems Engineering

B

B

B

B

B

A

A

A

A

A

THAAD 03T-1300.88

Driessnack

RISK ASSESSMENT Sub-process: Identification

• What Are the Risk Events in the Program?– Known / Knowns … Known / Unknowns– Unknown / Unknowns (uncertainty….)

• Where Are Risk Events Located in the Program? – Technology, Design, T&E, M&S, Cost, Funding, Schedule

• The IPPD process and empowerment…but measure!

– Program Office Structure – Functional or Product or both– Oversight and Politics…MDA/Space making major changes!– US or World Markets – IT (hot/cold cycles)

• Examine Sources/Areas of Risk – Tactical and STRATEGIC (inside/outside PM Office)– Product or Process (predicts the product problems)

Don’t narrow vision of your Risk Program

Driessnack

Understand External Risk In each decision system

• Requirements Generation – JCS/Service Chief Oversight• Requirements Generation System (CJCSI 3170.01B)

• Track beyond your KPP/thresholds• What is the changing environment – Crusader!

• Interoperability and Supportability of National Security Systems, and Information Technology Systems

• Worry about the “other guy”…two in “interoperability”

• PPBS Process – SECDEF Oversight and Politics• There really is no book! A Political “Free” Market!

• Defense Acquisition – MDA Oversight• DoD 5000….but this is only the formal rules• Who are the players…what are their agendas

Driessnack

Technology Readiness Levels (TRL) DoD 5000.2-R (Appendix F)

Technology Readiness Levels (TRL) DoD 5000.2-R (Appendix F)

• TRL #1 - Basic principles observed & reported• TRL #2 - Technology concept and/or application

formulated• TRL #3 - Analytical & experimental critical function

and/or characteristic proof of concept• TRL #4 - Component and/or breadboard validation in

laboratory environment.• TRL #5 - Component and/or breadboard validation in

relevant environment• TRL #6 - System/subsystem model or prototype

demonstration in a relevant environment• TRL #7 - System prototype demonstration in an

operational environment• TRL #8 - Actual system completed and qualified

through test & demonstration• TRL #9 - Actual system proven through successful

mission operations

(IOT&E during LRIP)

(System Demonstration)

(ATD/ACTD/ System Integration)

(basic research)

(applied research)

(advanced technologydevelopment)

(Component Advanced Development)

Driessnack

PlannedProfile

AchievedTo Date

Tolerance Band

Current Estimate

Threshold

Milestones

PlannedValue

VariationTECHPARAMVALUEe.g.,VehicleWt (lb)

TIMEVerPDRSFREqns

10

5

15

Technical Performance Measurement

Dr Falk Chart - modified

Normalize and track on index (like SPI & CPI)

Driessnack

Driessnack’s Thoughts…why EV evolved for 35 plus years

• Tracks programs on THEIR baseline, not rubrics• Rubrics – such as OSD O&E rates – drive behaviors

• Want program managed to particular baseline• Normalize the metric

• EV CPI/SPI track to the plan• TPMs that are normalized (see ARQ article)• O&E rates could be normalized to plan• ….almost anything can be put into an index

• Index (normalization) allows comparisons

• Each program has unique path of unique events• Need to characterize the uncertainty….

Driessnack

Driessnack’s Thoughts…why EV evolved for 35 plus years

• AT&L Staff working on next CARS (DAES/SAR) system• What data is needed…what process is needed…?

• EV has method of validating the overall process• Improve trust in the data reported…know detail exists • Reputation dependent on validation…real impacts• Need to incorporate RISKs into baselines…ranges!

• Industry builds tools to comply with criteria…• Need new criteria that drives industry to develop tools…incorporate Risk plus schedule into wInsight• Other tools manufactures do the same….

Driessnack

• Overall goal - integration of tools• WUG …

• Integrate Risk into EV develop RPI & TPI• CS Solutions - Risk+ schedule inside wInsight

• DoD baselines and tracks programs…risk neutral!

• Discuss of Risk • What is taught….some program examples• Various Management Tools not integrated

• General Proposal…beyond risk reserves• Discussion….• Discussion on RISK CoP…DAU WEB pages

Outline

Driessnack

Most trying to integrate with other tools…schedule, cost, TPMs, etc

Driessnack

- RIGHT PARTNERS (CPAR)- RISK MANAGEMENT PLANS/TEMPLATES- TPMs- EARNED VALUE ANALYSIS- COMMUNICATION

Insight not Oversight- TESTING AND MORE TESTING

…..ALL COMBINE TO YIELD AN INTEGRATED APPROACH TO DEVELOPING AN AFFORDABLE FAMILY OF …..

RISK REDUCTION TOOLS

Chart from Major DoD Program

Driessnack

- RIGHT PARTNERS (CPAR)- RISK MANAGEMENT PLANS/TEMPLATES- TPMs- EARNED VALUE ANALYSIS- COMMUNICATION

Insight not Oversight- TESTING AND MORE TESTING

…..ALL COMBINE TO YIELD AN INTEGRATED APPROACH TO DEVELOPING AN AFFORDABLE FAMILY OF …..

RISK REDUCTION TOOLS

Chart from Major DoD Program

Programs understand

they must integrate

Driessnack

Supporting Processes

• Quality management

• Contract acquisition

• Contract execution

Risk Identification• Change in threat• Technology

introduction• Design cost• Funding• Schedule• Field problems• Pollution

prevention initiatives

• Spare replacement• Environmental

compliance directives

Risklevel

Retirerisk?

Onplan

?

Monitor lowand retired risks

Risk MitigationPlanning/Implementation

Input Process Output

Statuschange

?

Perform riskassessment

analysis

No

Yes

Yes

• Review Analysis• Assess impact• Assign levels

System Wide

……. Risk Management Process from …. Risk Management Process from …

Med, High

Low

No

No

Yes

Track progress

Implement risk plan

Plan risk mitigations

• Change specification• Re-evaluate impact• Revise mitigations

Replan

Risk Management Board (RMB)

RiskAssessment

• Risk forms• Watchlist• Mitigation

Plans• Data Book

Driessnack

Supporting Processes

• Quality management

• Contract acquisition

• Contract execution

Risk Identification• Change in threat• Technology

introduction• Design cost• Funding• Schedule• Field problems• Pollution

prevention initiatives

• Spare replacement• Environmental

compliance directives

Risklevel

Retirerisk?

Onplan

?

Monitor lowand retired risks

Risk MitigationPlanning/Implementation

Input Process Output

Statuschange

?

Perform riskassessment

analysis

No

Yes

Yes

• Review Analysis• Assess impact• Assign levels

System Wide

……. Risk Management Process from …. Risk Management Process from …

Med, High

Low

No

No

Yes

Track progress

Implement risk plan

Plan risk mitigations

• Change specification• Re-evaluate impact• Revise mitigations

Replan

Risk Management Board (RMB)

RiskAssessment

• Risk forms• Watchlist• Mitigation

Plans• Data Book

Most Program have separate

Risk and EV processes

Driessnack

Risk Management Products Overview

Risk Management BoardRisk Management Board

Risk FormsRisk Item Description: Very brief description of the risk item. Be sure to use words like“Potential for…” instead of implying that risk is actually occurring (unless it really is).

Product/Task: Examples: Maintain WS Reliability, Maintain Availability of XX assets.

Responsible Individual/IPT: Firstname Lastname/Level 3 IPT Date of Evaluation: M/D/Y

Performance Measures: Examples: Voltage at GPR vs. Age, Subsystem Reliability vs. Age.

T: TechnicalConsequence if Risk I tem Occurs :

High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of technical risk

Probability of Risk Item Occurring:Low - What is the probability of it this consequence happening & why itis Low (or High or Medium).

S: ScheduleConsequence if Risk I tem Occurs :

High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of schedul risk


C: CostConsequence if Risk I tem Occurs :

High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of cost risk


Mitigation Plan:• Description 1• Description 2• Description 3

Conseque

nce

Probability of Risk Item Occurring

H M L

H

M

L6

4

9

7

52 1

8

3

TSC

Risk ID #MM & PKRisk Assessment Summary

Mitigation PlansID Task Name Duration Start

1 Task 1 14.5 days 12/13/99

2 Task 2 4 days 12/13/99

3 Milestone A 1 day 12/14/99

4 Task 3 7 days 12/15/99

5 Task 4 10.5 days 12/17/99

6 Task 5 9 days 12/17/99

7 Task 6 5 days 12/20/99

8 Milestone B 1 day 12/21/99

9 Task 7 3 days 12/27/99

10 Task 8 7 days 12/17/99

11 Task 9 4 days 12/27/99

12 Task 10 4 days 12/28/99

13 Milestone C 1 day 12/31/99

12/14

12/21

12/31

S S M T W T F S S M T W T F S S M T W T F SW3 W2 W1

Risk Quantification

23David Lindblad / TRW 05/16/2000 4:13 PM

Risk Quantification Sample

* Readiness is measured by Availability (Alert Readiness Rate: ARR and Operational Readiness Rate: ORR,Reliability, Accuracy, and Survivability (Hardness).

Readiness Projections

WeaponSystem

Readiness*Measure

Assessed at Component, Subsystem, and System levelsNotional Data

Fiscal Year

Acceptable Minimum

With Mitigation

Without Mitigation

WatchlistChange

RiskID #

WSRisk

Factor/Opened

MitigStatus

LMCFundingStatus

Risk Item Description

New S98 PK1

9/9/99Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).

M: W to GF: W to G

S99 MM2

9/9/99Green Green

Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).

M: R to Y Gr99 Both3

9/9/99Yellow

SCGreen


F: R to Y Pr99 PK4

9/9/99Green Yellow


2 to 4 Gu99 MM 4Red

TGreen


Closed R99 Both 6Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).

Watchlist SummariesWatchlist Count Summary

IPT 1 2 3 4 5 6 7 8 9 TotalSEIT # # # # # # # # # ##Ground # # # # # # # # # ##Propulsion # # # # # # # # # ##Guidance # # # # # # # # # ##RS/RV # # # # # # # # # ##Total ## ## ## ## ## ## ## ## ## ###

Risk Factor

M itig atio nS tatu s 1 2 3 4 T o ta lG reen # # # # ##Y ello w # # # # ##

R ed # # # # ##T o ta l ## ## ## ## ###

R isk F acto r

M itig a tio n S ta tu s S u m m ary

M itig a tio nS ta tu s 1 2 3 4 T o ta lG reen # # # # ##Y ello w # # # # ##

R ed # # # # ##T o ta l ## ## ## ## ###

R isk F acto r

F u n d in g S ta tu s S u m m ary

C h a n g eR is kID #

W SR is k

F a c to r/O p e n e d

M itigS ta tu s

M itigF u n d in g

S ta tu sR is k Ite m D e s c rip tio n

N e w S 9 8 P K1

9 /9 /9 9V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).

M : W to GF : W to G

S 9 9 M M2

9 /9 /9 9G re e n G re e n

V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).

M : R to Y G r9 9 B o th3

9 /9 /9 9Y e llo w

S CG re e n


F : R to Y P r9 9 P K4

9 /9 /9 9G re e n Y e llo w


2 to 4 G u 9 9 M M 4R e d

TG re e n


C lo s e d R 9 9 B o th 6V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).

W a tc h lis t C h a n g e S u m m a ry



Driessnack



Driessnack

• Overall goal - integration of tools• WUG …

• Integrate Risk into EV develop RPI• CS Solutions - Risk+ schedule inside wInsight

• DoD baselines and tracks programs…risk neutral!

• Discuss of Risk • What is taught….some program examples• Various Management Tools not integrated

• General Proposal…beyond risk reserves• Discussion….• Discussion on RISK CoP…DAU WEB pages

Outline

Driessnack

-16

-14

-12

-10

-8

-6

-4

-2

0

2


Cost "A Leg"

Schedule

Cost "B Leg"


Cu

m V

aria

nce

($M

)


B Leg WBSA Leg WBS


$-5M CV

$-5.5M SV

98 99 00

Delay PACOM

(3 Months)

ATR Delayed

(3 Months)

-$13.1M A Leg CV

C Leg WBSUpdateEVM

Driessnack

-16

-14

-12

-10

-8

-6

-4

-2

0

2


Cost "A Leg"

Schedule

Cost "B Leg"


Cu

m V

aria

nce

($M

)


B Leg WBSA Leg WBS


$-5M CV

$-5.5M SV

98 99 00

Delay PACOM

(3 Months)

ATR Delayed

(3 Months)

-$13.1M A Leg CV

C Leg WBSUpdateEVM

As PM I knew I would continue to rebaseline and develop C, D, E legsUsed Critical Chain – separate EV

baseline

Management Reserve

ACWPC

$

TimeNow

BCWSC

Completion Date

Schedule VarianceC

Cost VarianceC

PROJECTED

SLIPPAGE

OverBudget

EACContract at a Glance

BCWPC

ScheduleSlip

PMB

BAC

Driessnack

Driessnack

Driessnack

Driessnack

Driessnack

Driessnack

PRODUCTION

MILESTONES

DOCUMENTATION

EMD

12/93

MS II

4/00

LRIP DAB

9/01

LRIP 2

6/02

LRIP 3

7/03

MS III/FRP

F/A-18 TEMP

APB/ASR

JTEMP SINGLE MIDS ORD

11/95 5/96 3/029/0110/00 6/01

NAVYORD

SHIPTEMP

NOW

2/00OA

5/03IOC

2/01OA

10/02

Pilot Prod IOT&EAward DEL FRP IOC

SHIPS

ARMY

F-16

12/99INTEGRATION

LUT IOT&E1/01

7/011ST FLIGHT

6/02

F-15LVT (3)

9/989/96 10/99 3/00 6/00 2/01

9/02LVT(2)

FRP/IOC

INTEGRATION TECHEVAL OPEVAL

F/A-18

OA 11/00

OPEVALTECHEVAL

FY06PLATFORM INTEGRATION & TEST

3/94 2/98 1/99 6/00

MIDSCO 1st DELIVERY CDT&EMIDSCO ENDS SE&I

7/00

PROD READINESS

1/0011/011ST US DELIVERIES

8/97

PROD LONG LEAD & NRE

6/00 12/00

DLS/VIASAT EUROMIDS

3/031ST EU DELIVERIES

4/02

2004200320022001200019991998FY

DT6/02

DT 3/03

MIDS Program Overview

1/991ST FLIGHT

2/98

DT 3/01

6/02 10/02 1/04 6/04

Del Compl

11/03

FOE II IOT&E

Driessnack

COBRA JUDY (O-IPT CHARTS MS A)

Risk Assessment

L

Functional Area Risk

Technology

Design and Engineering

Human Systems

Support

Cost1

Schedule

Programmatics2

L

L

H

M

L

M

1High risk at current funding levels2Requires Acquisition Agent designation

Driessnack

MUOS Risk Assessment

Risk: CAD Contract award delayed

Mitigation Plans:

– Source selection process– Well defined selection criteria– Four versions of draft RFP– MS A scheduled for Aug 02

x

Schedule Risk #1

Risk: IOC of July 2008

Mitigation Plans:

– EVMS– Contract Incentives– Design off-ramps– Need funding stability

x

Schedule Risk #2

Risk: Cost estimates inaccurate

Mitigation Plans:

– Better system definition– Fund to CAIG estimate at MS B– Cross-check PO model– CAD Ktr bottoms-up estimate

x

Cost Risk #1

Risk: MUOS Cost Control

Mitigation Plans:

– CAIV– EVMS– Multi-year procurement– Incentives in SD&D

x

Cost Risk #2

Risk: Software development

Mitigation Plans:

– Software development plan– SEI Level III certification– OPTEVFOR EOA– Independent assessment

x

Technical Risk #1

Risk: System Integration

Mitigation Plans:

– JTRS MOA– Modeling/simulation– CAD demonstrations– Interface Control IPTs

x

Technical Risk #2

Driessnack

IPPD ENVIRONMENTEmpowerment

“ Decision making should be driven to the lowest possible level commensurate with risk. Resources should be allocated to levels consistent with risk assessment authority, responsibility & ability of people.

– The team should be given the authority, responsibility, & resources to manage its product & its risk commensurate with the team’s capabilities.

– The authority of team members needs to be defined & understood by the individual team members.

– The team should accept responsibility & be held accountable for the results of its efforts.”

IPPD Guide,” 5 Feb 96 - OUSD(DTSE&E)

You get what your measure

Driessnack

Critcal Risk ProcessesRisk Management Guide 2001

Driessnack

What is the Likelihood the Risk will happen?

Level Chance of Occurrence Your Approach and Processes

a

b

c

d

e

Not Likely:10% chance

Low Likelihood:25% chance

Moderate:50% chance

Highly Likely:75% chance

Near Certainty:90% chance

…Will effectively avoid this risk based on standard practices

…Have usually avoided this type of risk with minimal oversight in similar cases

…May avoid this risk, but workarounds will be required

…Cannot avoid this risk with standard practices, but a different approach may work

…Cannot avoid this risk with standard practices, probably not able to mitigate

Given the risk is realized, what would be the magnitude of the impact?

Level Technical Performance Schedule Impact

1Negligible

Small performance shortfall inspecific technical area; overall system performance unaffected

Minimal schedule slip but able to meet need dates w/o additional resources. Critical path unaffected

2Marginal

3Moderate

4Critical

5Catastrophic

Con

seq

uen

ce

Addition resources required to meetneed dates, Critical path unaffected

Minor performance shortfall inspecific technical area; overallsystem performance below goal but w/in acceptable limits

Mod performance shortfall inspecific technical area; overallsystem performance below goal& possible below acceptable limits

Minor schedule slip; will missneed date. Critical pathunaffected

Overall system performancebelow acceptable limits

Overall system performanceunacceptable to the degree thatthe ship is undeliverable

Major schedule slip. Program criticalpath affected (<1 month)

Major schedule slip. Program criticalpath affected (>1 month)

CostMillions

Costincrease <1

Costincrease 1-6

Costincrease 6-20

Cost increase

21-50

Costincrease >50

Li k

eli h

ood

Consequence

Lik

elih

o od

1 2 3 4 5

e

d

c

b

a

HIGH--Major program disruption,immediate priority management actionrequired.

MODERATE--Moderate disruption,possible management action required

LOW--Minimum impact

Questions about Risk Management?

Call Systems Engineering - Risk Management,Dept. E47

Steve Waddell, NNS 757-688-3760

CVN 77 Program Risk Assessment

Driessnack

NSSN Risk Assessment ProcessNSSN Risk Assessment ProcessQuestions about Risk

Management?

Call a Member of the Process Integration Team for Risk.

1 Minimal or No Impact Minimal or No Impact Minimal or No Impact None

2 Acceptable with Some Additional Resources Required ; < 5%Some Impact

Reduction in Margin Able to Meet Need Dates

3 Acceptable with Minor Slip in Key Milestone; 5 - 7% Moderate Impact

Significant Reduction Not Able to Meet Need Datesin Margin

4 Acceptable, No Major Slip in Key Milestone > 7 - 10% Major Impact

Remaining Margin or Critical Path Impacted

5 Unacceptable Can’t Achieve Key Team or > 10% Unacceptable

Major Program Milestone

CONSEQUENCE:Given The Risk is Realized, What is the Magnitude of the Impact?

NSSN Risk Process Card

February 1996

RISK ASSESSMENT

HIGH - Unacceptable. Major disruption likely. Different approach required. Priority management attention required.

MODERATE - Some disruption. Different approach may be required. Additional management attention may be needed.

LOW - Minimum impact. Minimum oversight needed to ensure risk remains low.

a Remote

b Unlikely

c Likely

d Highly Likely

e Near Certainty

Level What Is The Likelihood The Risk Will Happen?

LIKELIHOOD:

Level Technical Schedule Cost Impact on Other Teams

Performance

and/or and/or and/or

edcba

1 2 3 4 5

Lik

elih

ood

Consequence

ASSESSMENT GUIDE

Safety Reqt’s Exceeds Spec

Family of Systems

Safety of Crew in Cab

PRS #2 Availability

Reloader Redesign

Safety impacts Design

11

22

45

42

15

Insufficient Spares

Turret Servo Maturity

DT Duration too short

High Failure Rates

Moderate

642

18 32

28

12

15452232

18

60.6

1

Potential Severity of Consequence (Cf)

0.5

Low

High

•

•

•

•

••

•

•

•

•

0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.7

0.8

0.9

0

11

28

12

Pro

bab

ility

of

Occ

urr

ence

(P

f)

CLAWS Top Risks

Driessnack

Levers of Control along with other Management Models & Risk Management May 04 John Driessnack.

Documents

discipline of management

risk program risk

fundamentals of management

risk of action

john driessnack slide

guide action slide

tpi tools databases

right decision