November 8, 2007 Conference Brief RISC-IQ: Risk Integrated w / Schedule & Cost – Intelligent Quantification Applications for Systems Engineers This document.

November 8, 2007

Conference Brief

RISC-IQ: Risk Integrated w / Schedule & Cost – Intelligent Quantification Applications for Systems Engineers

This document is confidential and is intended solely for the use and information of the client to whom it is addressed.

International Council on System Engineering

2

International Council on System Engineering Booz Allen Proprietary

The need for Risk Management to be Integrated with Systems Engineering is well understood

In a recent Department of Defense briefing titled “Technical Excellence through Systems Engineering”, it was determined that there was a lack of understanding in programs on systems engineering. One of the reasons sighted was:

– Cost and schedule estimation and risk management processes inconsistently aligned with systems engineering processes

26% of surveyed systems engineers listed “Integration with other program management” as a way to drive technical rigor back into programs. This included:

– Linkage with acquisition strategy, IMP, IMS, logistics, testing, and risk management

As a result, a recommended strategy for driving technical rigor back into programs was to have risk management integrated, effective, and well resourced

3


So what?

The question isn’t “Should we do risk management?”

The question is, as a Systems Engineer:

– What do I do with the risk program I have?

– Is the risk program providing a good return for the expense?

– What, if anything, are the risk program outputs telling me?

– Can I use the risk program to communicate more effectively?

The rest of this discussion supposes a mature risk program exists:

– Roles and Responsibilities are defined (i.e. a Risk Mgt plan exists)

– Training has been provided

– Resources have been provided (tools, staff)

– Risk Management products are reviewed at regular intervals in formal reviews (Risk Review Boards)

4


RISC-IQ was developed to help extract actionable information from traditionally stove-piped data streams

RISC-IQ enables critical decision makingRISC-IQ enables critical decision making

Risk Exposure?

Impact Relationships?

Goals Too Risky?

Which Design?

More Reserves?

Major Drivers?

Adequately Mitigated?

5


RISC-IQ works by establishing a structured, repeatable process that integrates the potential impacts of risk on program baselines

Risk Integrated with Schedule and Cost – Intelligent QuantificationRisk Integrated with Schedule and Cost – Intelligent Quantification

Objective: Assess completeness of baseline documentation and understand existing processes.

Objective: Assess completeness of baseline documentation and understand existing processes.

Objective: Compare program risk profile at mitigation completion to determine outstanding risk exposure.

Objective: Compare program risk profile at mitigation completion to determine outstanding risk exposure.

Objective: Provide insight into where risks affect the program and uncover their true impacts.

Objective: Provide insight into where risks affect the program and uncover their true impacts.

Mitigation Strategy Revision

Mitigation Strategy Revision

RiskAnalysis

RiskAnalysis

ScheduleDevelopment

ScheduleDevelopment

CostEstimating

CostEstimating

Program Risk

Baseline

Program Risk

Baseline

Risk-WBS MappingRisk-WBS Mapping

Risk Impact AnalysisRisk Impact Analysis

Integrated Master

Schedule

Integrated Master

Schedule

Most Likely Cost

Estimate

Most Likely Cost

Estimate

Pre-Mitigation Cost EstimatePre-Mitigation Cost Estimate

Economic Effectiveness AnalysisBaseline Product Development

Pre-Mitigation Schedule Estimate

Pre-Mitigation Schedule Estimate

Mitigation Plan Development

Mitigation Plan Development

Mitigation Resource Analysis

Mitigation Resource Analysis

Economic Effectiveness Assessment

Economic Effectiveness Assessment

Post-Mitigation Cost Estimate

Post-Mitigation Cost Estimate

Initial Risk Accounting

Schedule Factor

Analysis

Schedule Factor

Analysis

Cost Factor Analysis

Cost Factor Analysis

Post-Mitigation Schedule Estimate

Post-Mitigation Schedule Estimate

6

International Conference on System Engineering Booz Allen Proprietary

Agenda

Project / Program Initiation

Design Cycle Selection

Engineering Trades

Defining Resources

Measuring Execution

Managing Groups of Projects

7


RISC-IQ uses explicit, quantified statements for risk probability and impact to determine risk scores

Probability

5 Very High

4 High

3 Medium

2 Low

1 Very Low

Impact

5 Very High

4 High

3 Medium

2 Low

1 Very Low

Probability

5 >70%

4 50-70%

3 30-50%

2 10-30%

1 <10%

Cost Impact

1 >70%

2 50-70% of Project Budget



5 <10% of Project Budget

The right-hand set of tables:

– Communicates the leadership’s sensitivity to risk

– Reduces personal bias

– Creates a foundation for quantitative analysis

VS

8


Agenda



Engineering Trades

Defining Resources

Measuring Execution


9


RISC-IQ can be used to help select and support multiple types of development cycle

Waterfall Development Model & Derivatives

Assists with estimates of overall cost and schedule estimates

Maps risk mitigation activities to development phases

Quantified risk score can be used to establish threshold for entering the next development phase

Spiral Type Develop Model

Enables capability vs. risk trade study for design features

Economic analysis of risk help apportion development resources to highest need areas

Tracking of risk profile useful in determining if the current spiral met its goals

10


Agenda



Engineering Trades

Defining Resources

Measuring Execution


11


RISC-IQ is useful for evaluating and communicating the impacts of design selection on risk exposure

As part of design trade activities, RISC-IQ provides input to the engineering processes by analyzing design options based on risk profiles

The risk team helps determine the risk variability across the options and the dependency of any one risk to key features/designs within an option

RISC-IQ applies a “Dependency” framework for this activity.

– Design Independent Risks-Design Independent Risks- Risks that remain relatively constant when key aspects of the system are altered

– Design Dependent Risks-Design Dependent Risks- Risks that vary significantly based on changes to the system design

Design Independent Risk Example

Option 1 Option 2 Option 3

32.50 32.00 33.50Risk Score

Design Dependent Risk Example

Option 1 Option 2 Option 3

11.23 5.48 55.75Risk Score

Little to no change in the risk score as the design or Little to no change in the risk score as the design or option changesoption changes

Significant change in the risk score as the design or Significant change in the risk score as the design or option changesoption changes

12


RISC-IQ provides a simple analytical framework for quickly highlight what risks drive any given design option

Dependency analysis is performed for every risk to determine whether the risk applies to a given option and the strength of dependency by design parameter

Shown below is a sanitized example from a recent client engagement

Risk # Prob Mit Sev Score Risk # Prob Mit Sev Score Risk # Prob Mit Sev ScoreProbability Variance

Mitigation Variance

Severity Variance

Score Variance Risk

Extent of Risk WBS

1 0.37 0.5 22 4.07 1 0.37 0.5 22 4.07 1 0.37 0.5 22 4.07 0 0 0 0 Weight issue None 1.3.55 0.37 0.25 22 2.04 5 0.37 0.25 22 2.04 5 0.37 0.25 22 2.04 0 0 0 0 Deployment None 1.3.36 0.87 0.5 43 18.17 6 0.87 0.5 43 18.17 6 0.37 0.5 43 7.96 0.5 0 0 10.75 Staff resource Moderate 1.3.4

7 0.37 0.5 43 7.96 7 0.37 0.5 43 7.96 0 0 0 0 Space availability Minimal 1.3.425 0.37 0.75 22 6.11 25 0.37 0.75 22 6.11 0 0 0 0 Deployment Minimal 1.2.2.3.3

48 0.98 0.75 43 31.61 48 0.98 0.75 43 31.61 48 0.63 0.5 22 6.93 0.35 0.25 21 24.675 Weight issue Significant 1.3.449 0.63 0.25 22 3.47 0 0 0 0 Deployment Minimal 1.2.3

54 0.13 0.75 36 3.51 54 0.63 0.75 36 17.01 0.5 0 0 13.5 Staff resource Moderate 1.2.2.3, 1.255 0.13 0.75 36 3.51 55 0.37 0.75 36 9.99 0.24 0 0 6.48 Space availability Moderate 1.2.2.4

62 0.63 0.75 36 17.01 0 0 0 0 Deployment Moderate 1.2.3.263 0.37 0.75 75 20.81 0 0 0 0 Weight issue Moderate 1.2.2.264 0.13 0.75 75 7.31 0 0 0 0 Deployment Minimal 1.2.3

70 0.13 0.75 95 9.26 70 0.63 0.75 75 35.44 70 0.63 0.75 75 35.44 0.5 0 20 26.175 Staff resource Significant 1.2.3.471 0.13 0.75 95 9.26 71 0.63 0.75 75 35.44 71 0.37 0.75 75 20.81 0.5 0 20 26.175 Space availability Significant 1.2.3.475 0.13 0.5 75 4.88 75 0.13 0.5 75 4.88 75 0.13 0.5 75 4.88 0 0 0 0 Weight issue None 1.2.2.3.178 0.87 0.25 75 16.31 78 0.63 0.25 75 11.81 78 0.63 0.25 75 11.81 0.24 0 0 4.5 Space availability Moderate 1.2.180 0.37 0.75 50 13.88 80 0.37 0.5 22 4.07 80 0.37 0.50 22 4.07 0 0.25 28 9.805 Weight issue Significant 1.2.2.2.5

91 0.37 0.5 75 13.88 91 0.37 0.5 75 13.88 0 0 0 0 Staff resource Minimal 1.2.392 0.13 0.25 75 2.44 0 0 0 0 Space availability Minimal 1.2.2.498 0.87 0.5 62 26.97 98 0.87 0.5 62 26.97 98 0.37 0.5 62 11.47 0.5 0 0 15.5 Deployment Moderate 1.2

100 0.37 0.75 36 9.99 0 0 0 0 Weight issue Minimal 1.2.3103 0.13 0.75 43 4.19 103 0.37 0.75 22 6.11 0.24 0 21 1.9125 Deployment Significant 1.2104 0.87 0.75 50 32.63 104 0.87 0.75 50 32.63 0 0 0 0 Space availability Moderate 1.2.2.2.4

106 0.13 0.25 29 0.94 106 0.13 0.25 29 0.94 106 0.63 0.75 95 44.89 0.5 0.5 66 43.945 Staff resource Significant 1.2.4.4157 0.87 0.75 95 61.99 157 0.87 0.75 95 61.99 0 0 0 0 Space availability Significant 1.2.4.4162 0.87 0.75 50 32.63 162 0.87 1 50 43.50 0 0.25 0 10.875 Deployment Moderate 1.2.5

175 0.87 0.5 95 41.33 175 0.87 0.5 95 41.33 175 0.87 0.5 95 41.33 0 0 0 0 Weight issue None 1.2.2.5

Design Option 1 Design Option 2 Design Option 3

Results

Sam

ple

13


The RISC-IQ design dependency approach establishes a method to evaluating requirements and overall risk tolerance

Using the quantitative risk scores we can compare a single risk area across platforms, or the variability of a single risk within a given platform

Ris

k L

ev

el

Weight RiskSatellit

e A optio

n

Satellite B

option

Satellite D

option

Satellite C

option

Medium

Low

High

Critical

Ris

k L

ev

el

Communications Risk

Option 1

Option 2

Option 4

Option 3

Medium

Low

High

Critical

(Placing the options in order of complexity, i.e. performance or technology, allows the program to understand the ‘risk curve’ as the options become more aggressive or challenging in their profile.)

14


Once the analytical framework and quantitative scoring is established, risk can then be treated as a design variable

Risk can be treated as an independent variable and applied in a similar fashion to cost in CAIV methodologies; “Risk As An Independent Variable”“Risk As An Independent Variable”

Ris

k L

ev

el

Technology Design Variable/Option

(1) Risk curves are generated for designs or aspects of each design using dependency model

Pe

rfo

rma

nc

e

Me

tric

Technology Design Variable/Option

(2) Performance Analysis working group determines relationship between the design variable and system performance

Variable

Risk

Metric

Variable

Effectiveness

Metricƒ(Variable) = X X

Eff

ec

tiv

en

es

s

Performance Metric

(3) Utility Analysis working group performs effectiveness analysis to show relationship to performance metric

Risk Sensitivity

Performance Sensitivity

Effectiveness Analysis

15


RISC-IQ then uses this analytical framework show when additional performance isn’t worth the additional risk

By combining these curves, it’s possible to identifying the point of diminishing risk tolerance for any design variable or option

– Point of Diminishing Risk Tolerance- “The point at which the increase in additional design capability, or the selection of a proposed option, reduces effectiveness by increasing risk to the program.”

Ris

k T

ole

ran

ce

Design Variable/Option

Effectiveness

Risk=

Point of Diminishing Risk Tolerance

16


RISC-IQ takes the results of the design trades and evaluates them by source category and the work to satisfy the requirements

WB

S #

WB

S E

lemen

t

Co

mm

s

Co

ncu

rrency

Co

st

Desig

n

Develo

per

Lo

gistics

M&

S

Mg

mt

Perfo

rman

ce

Pro

du

ction

Req

uirem

ents

Sch

edu

le

T&

E

Tech

no

log

y

Th

reat

To

tal

1.0 Prog Mgmt 1 1

2.0 Network 1 1 1 3

3.0 Mission 2 1 1 1 5

4.0 Ground 4 1 1 6

4.1 GSE 3 3

4.2 Program Management 1 1

4.3 Subcontractor 2 3 1 1 2 2 1 1 12

4.4 TT&C 2 1 2 1 1 7

4.5 Communications

5.0 Launch 1 2

6.0 Space PMO

7.0 Space SEIT 1 1

8.0 Bus 1 5 2 2 2 1 1 2 1 3 20

9.0 Payload 5 5 10

10.0 Spacecraft AI&T 1 1 1 1 4

Total 3 6 11 8 3 9 3 15 7 7

A review by WBS elements shows which areas of the program need additional risk

identification.

A review by WBS elements shows which areas of the program need additional risk

identification.

Critical Risk Categories with Few Risks Identified

A review by categories show holes in existing risk areas that should be

investigated further.

A review by categories show holes in existing risk areas that should be

investigated further.

The following is an example from a commercial satellite project analysis

17


Agenda



Engineering Trades

Defining Resources

Measuring Execution


18


RISC-IQ uses a convention of aligning risks to the Work Breakdown Structure (WBS) to help define resource requirements Aligning risks to the program WBS serves as the basis for allocating cost and schedule impacts

throughout the program

During the risk analysis phase, WBS elements are correlated to each other and risks mapped to WBS elements

Mapping provides an easily understood foundation that is repeatable, traceable, and defensibleMapping provides an easily understood foundation that is repeatable, traceable, and defensible

19


After mapping, any WBS element may have multiple risks that affect it

The combined affect of the risks on the cost element is related to the type of the risk, its likelihood and severity, and the cost of mitigation actions that could avert the risk

Using probabilistic models, the combined affect is determined and applied to the base cost estimate

RISC-IQ uses this WBS mapping to show the impacts of risk on the original estimate

WBS

1 3 2 11.1.2.3

Baseline Estimate

Risk-Adjusted Estimate

$1,179,000

50% 50%

20


This activity helps justify mitigation requests and can highlight when mitigation is no longer effective

This activity helps justify mitigation requests and can highlight when mitigation is no longer effective

Represents the initial baseline cost distribution without any included risk effects

Represents the cost distribution with potential risk effects BEFORE any mitigation

Represents the cost distribution AFTER successful mitigation, PLUS the required mitigation investment.

As a result of a $500,000 investment in risk mitigation, the program avoided a potential increase of $645,000.

$645,000

Mitigation Investment= $500,000

RISC-IQ then applies the WBS mapping and pre- and post- mitigation scores to guide mitigation activity

The final determination of cost considers the remaining risk exposure AFTER mitigation, plus the mitigation investment required

21


Relationships between risks and schedule elements for critical path determination must account for:

– Predecessor-Successor logic

– Parent-Child

– Parallel-Effort

– Conditional Dependencies

The same process can be performed on any unit of measure. Risk Adjusted Schedule is particularly complex

Cum

ulat

ive

Fre

quen

cy

Hits

0

24

48

72

96

120

Distribution (start of interval)

13/Jul/05 26/Jul/05 08/Aug/05 21/Aug/05 03/Sep/05 16/Sep/05

5% 29/Jul/05

10% 02/Aug/05

15% 04/Aug/05

20% 06/Aug/05

25% 08/Aug/05

30% 09/Aug/05

35% 11/Aug/05

40% 12/Aug/05

45% 14/Aug/05

50% 15/Aug/05

55% 17/Aug/05

60% 18/Aug/05

65% 20/Aug/05

70% 21/Aug/05

75% 23/Aug/05

80% 25/Aug/05

85% 26/Aug/05

90% 29/Aug/05

95% 02/Sep/05

100% 18/Sep/0500017 - Review production performance results 46%

00008 - Design and order new equipment 45%

00005 - Confirm product specification 33%

00018 - Identify long term production requirements 32%

00003 - Confirm product safety 25%

00021 - Determine if product is meeting customer cost/performance expect 21%

00004 - Re-evaluate manufacturing processes 17%

00020 - Affirm decision to continue commercialization program 15%

00019 - Recalculate product economics 14%

00022 - Finish 7%

00009 - Install and test new equipment or equipment modifications 3%

00002 - Confirm sales and marketing strategies 3%

00012 - Confirm manufacturing capabilities 3%

00016 - Perform internal product testing 3%

00014 - Start production of commercial product quantities 2%

00010 - Test failure 2%

00011 - Test pass 0%

00013 - Develop and review production trial plan (resolve logistics issu 0%

00006 - Confirm customer specification acceptance and trial participatio 0%

00015 - Monitor customer acceptance testing 0%

Risk Adjusted Delivery Likelihood Schedule Sensitivity By Task

22


Agenda



Engineering Trades

Defining Resources

Measuring Execution


23


RISC-IQ can help determine if the engineering activity is proceeding as planned

As a “snapshot” of project health, RISC-IQ can reveal whether or not the engineering activity is likely to deliver the expected results

In the example below, a RISC-IQ analysis demonstrated a previously unknown risk exposure of $148M and potential project cost growth of 25%

Incorporating RISC-IQ throughout a project’s lifecycle results in better initial planning and helps answer questions like:

– Are prototype or testing activities reducing risk as planned?

– Have risks been reduced sufficiently to pass entry / exit criteria for established technical reviews?

– Are new risks been uncovered faster than they are being closed?

– What are the impacts of “mid-stream” requirements changes?

24


Agenda



Engineering Trades

Defining Resources

Measuring Execution


25


RISC-IQ practices for a single project are extensible to groups of projects or systems of systems the must interact to deliver capability…

0

10

20

30

40

50

60

70

GroupAverage

Prog 1 Prog 2 Prog 3 Prog 4 Prog 5 ClassifiedProg Ave

CommercialProg Ave

Low

Med

High

Critical

Average Composite Risk Ranking by Program

Ave

rag

e R

isk

Sco

re

By standardizing risk analysis across all multiple programs, we provide insight into high risk programs and overall trends

26


400000

600000

800000

1000000

1200000

1400000

1600000

Prog 1: Pre-Mitigation

Prog 1:Post-

Mitigation


Prog 2:Post-

Mitigation


Prog 3:Post-

Mitigation

Prog N: Pre-Mitigation

Program N:Post-

Mitigation

… and program-by-program risk exposure analysis provides comparative insights what programs can be improved

Pro

gram

Cos

t (in

thou

sand

s)

$39,100

Initial Program Estimate

Revised Estimate

Budget at Risk

Mitigation Investment

Reduced Cost Exposure

Analysis will lead to a cumulative assessment of the total risk exposure, the investment allocated to reduce it, and the resultant investment benefit.

Analysis will lead to a cumulative assessment of the total risk exposure, the investment allocated to reduce it, and the resultant investment benefit.

27


Added DurationContract Length Total Risk Exposure

In a matrixed organization where Systems Engineering is a shared resource, using RISC-IQ to align risk-adjusted program schedules helps prioritize the timing of resources and risk reduction

Post-Mitigation Delivery

Pre-Mitigation Delivery

Contract Delivery

Program1

Program4

Program2

Program3

ProgramN

T0 T3 T18T6 T9 T12 T15T2 T5 T20T8 T11 T14 T17T1 T4 T19T7 T10 T13 T16

The overlap of additional program duration requires allocation of program

and/or enterprise resources

The overlap of additional program duration requires allocation of program

and/or enterprise resources

Understanding most probable program delivery allows S&IS to:

Risk exposure is the amount of schedule slip reduced through successful risk

mitigation

Risk exposure is the amount of schedule slip reduced through successful risk

mitigation

Inform clients of most probable schedule Plan for resources in advance Address contract issues Evaluate effectiveness of proposed mitigation Understand when enterprise reserves will be

needed

28


In summary, risk management provides the Systems Engineer with quite a bit of “so what”

Concept Development

Design Planning & Execution Enterprise

RISC-IQ can help established risk tolerance and incorporate risk as variable benefit trades

RISC-IQ can help established risk tolerance and incorporate risk as variable benefit trades

RISC-IQ provides unique insight into project cost and schedule realism, reserve requirements and health during execution

RISC-IQ provides unique insight into project cost and schedule realism, reserve requirements and health during execution

RISC-IQ highlights what specific requirements or technologies are project drivers

RISC-IQ highlights what specific requirements or technologies are project drivers

RISC-IQ at the enterprise level highlights overall reserves needs, resource shortfalls due to slips, and supports benchmarking

RISC-IQ at the enterprise level highlights overall reserves needs, resource shortfalls due to slips, and supports benchmarking

Cum

ula

tive F

requency

Hits

0

21

42

63

84

105

Distribution (start of interval)

15/May/12 18/Jun/12 22/Jul/12 25/Aug/12 28/Sep/12 01/Nov/12

5% 12/Jun/12

10% 21/Jun/12

15% 27/Jun/12

20% 03/Jul/12

25% 09/Jul/12

30% 12/Jul/12

35% 17/Jul/12

40% 20/Jul/12

45% 25/Jul/12

50% 31/Jul/12

55% 03/Aug/12

60% 08/Aug/12

65% 14/Aug/12

70% 20/Aug/12

75% 24/Aug/12

80% 31/Aug/12

85% 10/Sep/12

90% 19/Sep/12

95% 28/Sep/12

100% 05/Nov/12 Analysis

Simulation: Latin HypercubeIterations: 1000

Convergence

Plan Finish Date:Converged in 200 iterations(variation < 1% over 100 iterations)Total Plan Cost:Converged in 200 iterations(variation < 1% over 100 iterations)

Statistics

Minimum: 15/May/12Maximum: 05/Nov/12Mean: 01/Aug/12Max Hits: 111Std Deviation: 33.62

Selected Confidence75%: 24/Aug/12Deterministic Finish: 03/Apr/12Probability (less than 1%)

System Engineering & IntegrationEntire Plan : Finish Date

Booz Allen Hamilton

29


For more information on how risk management can be applied to your specific challenges, please contact:

Tweed RossColorado Springs, [email protected]

Robert MakarSan Diego, [email protected]

Michael LopezLos Angeles, [email protected]

November 8, 2007 Conference Brief RISC-IQ: Risk Integrated w / Schedule & Cost – Intelligent Quantification Applications for Systems Engineers This document.

Documents

program baselines risk

program risk profile

mature risk program

risk program outputs

risk management processes

risk mgt plan

potential impacts of

outstanding risk exposure