1 Oct 2016 Cost Risk/Uncertainty Analysis Overview Learning Objectives • Terminal Learning Objective: Explain the differences between cost risk analysis and uncertainty analysis as applied to major weapon systems • Enabling Learning Objectives: – Categorize the various methods used to perform cost risk & uncertainty analyses – Classify the types of risk faced by defense acquisition programs. – Illustrate the difference between risk and uncertainty – Explain the Monte Carlo Simulation process used in cost risk analyses 2
18
Embed
Cost Risk/Uncertainty Analysis Overview...Cost Risk & Uncertainty Analysis (CRUA) • CRUA provides insights into these questions • CRUA is a process of quantifying the cost impacts
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1Oct 2016
Cost Risk/Uncertainty Analysis Overview
Learning Objectives
• Terminal Learning Objective: Explain the differences between cost risk analysis and uncertainty analysis as applied to major weapon systems
• Enabling Learning Objectives:– Categorize the various methods used to perform cost risk & uncertainty analyses
– Classify the types of risk faced by defense acquisition programs.
– Illustrate the difference between risk and uncertainty
– Explain the Monte Carlo Simulation process used in cost risk analyses
2
General Cost Risk Modeling Process
3
PointEstimate
Statistical UncertaintyDistributions
Program RiskRegister
Uncertainties
Framing AssumptionsUncertainties
RunSimulation(s)
ApplyCorrelation
ApplyAny OtherInfluences
Interpret Results
Allocate andPhase Risk
Dollars
ReportResults
Consider other possible risks
Document as you go!
A Typical Program Cost Estimate
• The standard product of a cost estimating exercise is a single number – the point estimate – an anchor point
• We know that the estimate is wrong – we hope the actual cost will be close to the estimate
• We are not comfortable presenting the point estimate• many assumptions that may or may not come
true• many guesses were required to generate the
point estimate• how do we express the uncertainty in the
estimate?
4
Risk vs Uncertainty
Risk: the probability of a loss or injury
Opportunity: a favorable event or outcome
Uncertainty: the indefiniteness about the outcome of a situation
5
The goal of the simulation model is to combine all the sources of cost uncertainty in order to estimate the risk of exceeding a given budget.
Source: The Joint Agency Cost Schedule Risk & Uncertainty Handbook (CSRUH), p. 3
Typical Questions
• What distribution best fits our total cost data?
• What is the probability costs will exceed $1B?
• What is probability of a 30% or greater cost overrun?
• If X~Triang(15,35,85), what is P(X≥60)?
• If my funding is $150M, what is probability my program will exceed this funding level?
• What funding is required at the 60% confidence level?
6
Cost Risk & Uncertainty Analysis (CRUA)
• CRUA provides insights into these questions
• CRUA is a process of quantifying the cost impacts of uncertainties associated with a systems technical definition, cost estimating methodology, requirements, threat and schedule
7
Elements of Risk/Uncertainty in a Cost Estimate
• Technical Risk/Uncertainty– Risk/uncertainty due to inability to conquer technology problems
posed by the intended design in the current CARD or system specs
• Schedule Risk/Uncertainty*– Technical and other risks impact the schedule
• Concurrent development
• Test failures
• Delayed Milestone approval
• Optimistic task durations
• Requirements Risk/Uncertainty– Variations caused by unforeseen design shift from the current CARD
or system specs
• Programmatic Risk/Uncertainty– Beyond control of PM office; program decisions made at higher levels
of authority; includes budgeting decisions
8*Adapted from GAO Cost Estimating & Assessment Guide (2009)
Examples
• Threat Risk– Risk due to an unrevealed threat
– The problem changed (which changes the solution, design, & cost)
• Cost Estimating Risks– Variations despite a fixed configuration baseline
– Inaccurate, imprecise, or incomplete data leading to biased or imprecise estimates
– CER model misspecification
– Mathematical/statistical errors
• Business or Economic– Variations caused by changing assumptions
• Other?Cost Risk = ∑ All the Above Risks 9
Elements of Risk/Uncertainty in a Cost Estimate
• Parametric CERs including factors and cost improvement curve (CIC) equations
• CER inputs, complexity factors for analogies, engineering judgment
• Any other cost drivers (man-hours, head counts, rates, ratios, overhead, fee, etc.)
• The planned schedule (durations)
• Risk register events, both probability of occurrence and the consequence
• Framing Assumptions
10
Sources of Uncertaintyto be Captured in CRUA
See CSRUH p. 3 for more on sources of uncertainty
Types of CRUA
• Historically-based methods– Uses mean growth or distribution of growth from a set of
analogous programs
– Example: “On average, this type of program has experienced 17% cost growth. Therefore, we will increase the point estimates by 17%.”
– Growth factors should be adjusted based on extent of risk present in program being estimated vs average program from analogous data set
• Expert Opinion-based methods– Ask experts to assess the uncertainty around cost estimates, by
cost element
– Have them “score” a program (and other programs) in terms of the various risk elements; map scores to dollars
11
Types of CRUA
• Input-based methods– Assess uncertainty around inputs to the cost model, as well as the CER
equations themselves (where applicable)
– Most simulations use Input-Based methods (e.g. Monte Carlo)
– Symmetric approximation (a Method of Moments technique) is an input-based method
• Output-based methods– Apply uncertainty directly to estimate outputs (e.g., WBS elements) by
assigning distributions and simulating
• Scenario-based methods– From the point estimate, define a “protect scenario” that captures the
impacts of the major known risks to the program
– “Protect scenario” is combination of multiple sensitivities
– Determine cost of protect scenario; delta between this and point estimate is risk dollars
– May be used to capture potential impacts of framing assumptions
12See CSRUH p. 5 for more on types of CRUA, p. 80 for discussion on Method of Moments
Cost as a Probability Distribution (PD)
• Cost is an uncertain quantity • Cost is highly sensitive to conditions and assumptions
that change across a systems life• Sensitivity analysis
– Change in cost…varying conditions…isolates cost drivers– A deterministic process…defined by set of single variable
changes– Does not offer insights into simultaneous changes– Does not result in a distribution representing potential range of
costs
• A PD is a way to address many questions– A mathematical rule associating a probability to each outcome– Two types are PDF and CDF
13
Cost as a Probability Distribution (PD)
• Basis – if, individual cost elements are random variables and their distributions determined, then system costs can be expressed as PD
• Problem – how to determine PDs
• Problem – how to combine many individual cost elements and their uncertainty into a total estimate of cost representing all the inherent uncertainty
• Solution – Symmetric Approximation (Summation of Moments) and Monte Carlo Simulation
14
Cost Risk/Uncertainty Analysis
• Using Input-based Monte Carlo Simulation
– “….offers the user a powerful and precise method of assimilating the various uncertainties of a problem and producing a realistic appreciation of the problem’s uncertainty.” (David Vose, Quantitative Risk Analysis, p1)
– Accounts for values across a range and weights them by probability of occurrence
– Models each uncertain variable by a probability distribution function…assigns probabilities
– Objective – calculates the combined impact of the model’s various risks in order to determine a PD of possible model outcomes
15
• A PDF characterizes the probability associated with all possible outcomes of a random variable
• If we view cost as a random variable, then a PDF is an effective way to depict the possible outcomes
A Cost Probability Density Function (PDF)
0
1
2
3
4
5
6
7
8
1.11
1.13
1.15
1.17
1.19
1.21
1.23
1.25
1.27
1.29
1.31
1.33
1.35
1.37
1.39
1.41
1.43
1.45
1.47
1.49
1.51
1.53
1.55
1.57
1.59
Cost
Once PDF determined, can make probability statements with respect to costs.P
roba
bili
ty
16
The Total Cost PDF
17
AREA = 10%There is a 10% chance
that cost will exceed $1.40MOR
There is a 90% chance that the program will not exceed a $1.40M budget
AREA = 50%There is a 50% probability
that cost will be less than $1.32MP
roba
bili
ty
18
0
1
2
3
4
5
6
7
8
1.11
1.13
1.15
1.17
1.19
1.21
1.23
1.25
1.27
1.29
1.31
1.33
1.35
1.37
1.39
1.41
1.43
1.45
1.47
1.49
1.51
1.53
1.55
1.57
1.59
Cost
Pro
babi
lity
Probability Density Function
Cost
The Total Cost PDF
Using PDFs
• Identify a PDF for the uncertain inputs in the cost estimate
Cum Frequency Diagram to a Cum Density Function (CDF)
Risk (Contingency) Dollars
• The point estimate will usually be somewhere between the 15th and 50th percentile*° Depends on skewness of the PDFs
• Decision-makers may budget at a higher level
• Risk (Contingency) is the difference between the point estimate and the budget
° Where do you put that money?
31*See Smart, 2015, p.201 and CSRUH p. 60
General Cost Risk Modeling Process
32
PointEstimate
Statistical UncertaintyDistributions
Program RiskRegister
Uncertainties
Framing AssumptionsUncertainties
RunSimulation(s)
ApplyCorrelation
ApplyAny OtherInfluences
Interpret Results
Allocate andPhase Risk
Dollars
ReportResults
Consider other possible risks
Document all along!
Summary
• Several methods available for conducting a CRUA– This course will focus on the Input-based method
using Monte Carlo simulation
• Requires understanding of simulation concepts and probability distributions
• A good CRUA looks at all facets of uncertainty in the estimate……..– It should have a strategy
– It should be able communicate the results
33
References
34
Department of Defense. 2014. Joint Agency Cost Schedule Risk and Uncertainty Handbook(CSRUH). http://cade.osd.mil/cade/CSRUH.aspx
Smart, Christian. 2013 “Default Correlation for Cost Risk Analysis.” Missile Defense Agency, Paperpresented at the ICEAA Professional Development and Training Workshop, New Orleans, LA. June18-21.
Smart, Christian, 2015 “Covered with Oil: Incorporating Realism in Cost Risk Analysis”, Journal of Cost Analysis and Parametrics, 8:3, 186-205, DOI: 10.1080/1941658X.2015.1096220
Capen, E.C. 1976. “The Difficulty in Assessing Uncertainty.” Society of Petroleum Engineers.28(8):843-850. doi: 10.2118/5579-PA.
Government Accountability Office. 2009. Cost Estimating and Assessment Guide: Best Practices forDeveloping and Managing Capital program Costs. http://www.gao.gov/products/GAO-09-3SP .
Department of Defense. 2008. Risk Management Guide (RMG) For DoD Acquisition Sixth Edition (Ver1.0). http://www.everyspec.com/DoD/DoD-PUBLICATIONS/RISK_MGMT_GUIDE_FOR_DOD_ACQUISTION_2605/
Department of Defense. 2015. Risk, Issue, and Opportunity Management Guide for Defense Acquisition Programs. www.acq.osd.mil/se
Book, Stephen A. 2007. “Allocating “Risk Dollars” Back to Individual Cost Elements.” Paper presented at 40th Annual DoD Cost Analysis Symposium, Williamsburg, VA, February 15-18.
Book, Stephen A. 1999. “Why Correlation Matters in Cost Estimating.” Paper presented at 32ndAnnual DoD Cost Analysis Symposium, Leesburg, VA, February 2-5.
BACKUP
35
• STEP 1: Determine program cost drivers and associated risks
• STEP 2: Develop probability distributions to model uncertainty
• STEP 3: Account for correlation between cost elements
• STEP 4: Perform uncertainty analysis (e.g. with Monte Carlo simulation)
• STEP 5: Identify the probability associated with the point estimate
• STEP 6: From confidence level, determine risk dollars
• STEP 7: Allocate, phase, and convert the risk-adjusted estimate to then-year dollars and identify high risk elements
GAO Cost Risk Modeling Process
36Source: GAO Cost Estimating and Assessment Guide (2009), Chapter 14