Homayoon.dezfuli

Risk-informed Decision Making

Presented at the Seventh Annual NASA Project Management Challenge

Galveston, Texas

February 9-10, 2010

Homayoon Dezfuli, Ph.D.Office of Safety and Mission AssuranceNASA Headquarters

Gaspare MaggioTechnology Risk Management OperationsInformation Systems Laboratories, Inc.

Used with Permission

Acknowledgments

• This presentation is based on the material contained in the first draft of NASA Risk-informed Decision Making Handbook, released in October 2009. The authors acknowledge the contribution of the following individuals in the preparation of this handbook:

– Chris Everet, Information Systems Laboratories, Inc.– Rod Williams, Information Systems Laboratories, Inc.– Robert Youngblood, Idaho National Laboratory– Curtis Smith, Idaho National Laboratory– Peter Rutledge, Quality Assurance & Risk Management Services, Inc.

2

3

Background

4

NPR 8000.4A

• The latest version of NPR 8000.4A, Agency Risk Management Procedural Requirements, was issued on December 16, 2008

– Accessible from NASA Online Directives System (NODIS) Library– http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=8000&s=4A

• This directive evolves NASA’s Risk Management (RM) approach to entail two complementary processes:

– Risk-informed Decision Making (RIDM)Emphasizes the proper use of risk analysis in its broadest sense to make risk informed decisions that impact all mission execution domains (e.g., safety, technical, cost, and schedule) for program/projects and mission support organizations for supporting development of baseline performance requirements by selecting performance commitments

– Continuous Risk Management (CRM) Focuses on the management of risk associated with implementation of baseline performance requirements

RM ≡ RIDM + CRM

http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=8000&s=4A�

5

General Definition of “Risk” per NPR 8000.4A

“Potential for performance shortfalls, which may be realized in the future, with respect to achieving explicitly established and stated Performance Requirements”

• This definition of “risk” guided the development of some of the RIDM concepts

• The performance shortfalls may be related to any one or more of the following mission execution domains

– Safety– Technical– Cost– Schedule

6

The RIDM Process as defined in NPR 8000.4A

• What is RIDM? – A risk-informed decision-making process that uses a

diverse set of performance measures along with other considerations within a deliberative process to inform decision making. (Paragraph A-14)

A decision-making process relying primarily on a narrow set of model-based risk metrics would be considered “risk-based.” (Note to Paragraph A-14)

• What does it involve? – Identification of decision alternatives (decision

context) and considering a sufficient number and diversity of Performance Measures

– Risk analysis of decision alternatives (uncertainty analysis of performance associated with the alternative

– Selection of a decision alternative informed by (not solely based on) Risk Analysis Results

To Requirements Baselining

Risk-Informed Decision Making (RIDM)

Identification of AlternativesIdentify Decision Alternatives (Recognizing Opportunities) in the Context of Objectives

Risk Analysis of AlternativesRisk Analysis (Integrated Perspective) and

Development of the Technical Basis for Deliberation

Risk-Informed Alternative SelectionDeliberate and Select an Alternative and Associated Performance Commitments Informed by (not solely based on) Risk

Analysis

7

The RIDM Process Begins with NASA Strategic Goals• Within NASA’s organizational

hierarchy, high-level objectives (NASA Strategic Goals) flow down in the form of progressively more detailed performance requirements, whose satisfaction assures that objectives are met

• RIDM is designed to maintain focus on strategic goals as decisions are made throughout the hierarchy

RIDM P

roce

ss

8

RIDM and CRM Within the NASA Hierarchy

• RIDM and CRM operate at each level of the NASA hierarchy, with interfaces for the flowdown of requirements, the elevation of risk issues, and the communication of risk information

9

RIDM Handbook Development

10

RIDM Handbook

• OSMA has developed a Special Publication (in draft) to provide implementation guidance

– It decomposes RIDM into specific process steps, with specific guidance provided for each step

– It elaborates on the relationships between RIDM, requirements development, requirements baselining(or rebaselining), and CRM

– The present emphasis is on Programs and Projects; however, the process is generally applicable to all activities covered by NPD 7120.4

• The development team observed/reviewed a number of NASA decision-making activities for good practices & lessons learned

– Altair Buyback Analysis – Ares I Recovery Assessment– Ares Launch Order Analysis– Exploration Systems Architecture Study

NASARisk Informed Decision MakingHandbook

NASA/SP-2009-XXXXRev0

T H I S H A NDB OOK H A S NOT B E E N R E V I E W E D F OR E X POR T C ONT R OL R E ST R I C T I ONS; C ONSUL T Y OUR C E NT E R /F A C I L I T Y /H E A DQUA R T E R S E X POR T C ONT R OL

PR OC E DUR E S/A UT H OR I T Y PR I OR T O DI ST R I B UT I ON OF T H I S DOC UM E NT .

NASA/SP-2009-XXXXRev0

NASARisk Informed Decision MakingHandbook

Office of Safety and Mission Assurance NASA Headquarters

RIDM Process Themes

• The importance of close ties between the selected alternative and the requirements derived from it

– The RIDM process should promote the generation of achievable requirements (e.g., mean value results from high-level analyses should not become requirements)

– As alternatives are modified, derived requirements should be rebaselined to follow suit

• The importance of maintaining a focus on high-level objectives, for decisions made at all levels of the NASA hierarchy

• The importance of considering multiple objectives across all mission execution domains (safety, technical, cost, schedule)

• The importance of a documented decision

11

Information Flow in RIDM

12

Coordinated byRisk Manager

13

RIDM Process Steps

RIDM ProcessPart 1 – Identification of Alternatives

14

15

• The goal of Step 1 is the development of unambiguous objectives, reflecting stakeholder expectations.

• Typical inputs to Step 1 include:– Upper Level Requirements and Expectations: The needs, wants, desires,

capabilities, constraints, external interfaces, etc., that are being flowed down from a higher level (e.g., program, project, etc.)

– Identification of Stakeholders: Individuals or organizations that are materially affected by the outcome of a decision or deliverable but are outside the organization doing the work or making the decision

• Typical outputs for capturing stakeholder expectations include the following:

– Top-Level Requirements and Expectations: These are the top-level needs, wants, desires, capabilities, constraints, external interfaces, etc., for the product(s) to be developed

– Top-Level Conceptual Boundaries and Functional Milestones: How the selected alternative will be operated to meet expectations. It describes the alternative’s characteristics from an operational perspective

RIDM Process – Part 1Step 1 – Receive Objectives & Understand Stakeholder Expectations

16

• In general, it can be difficult to assess decision alternatives against multifaceted and/or qualitative top-level objectives

• To deal with this situation, objectives are decomposed, using an objectives hierarchy, into a set of lower-level performance objectives that any attractive alternative should have

• A performance measure is then developed for each performance objective, as the quantity that measures the extent to which a decision alternative meets the performance objective

RIDM Process – Part 1Step 2 – Derive Performance Measures from Objectives

Notional Objectives Hierarchy

17

RIDM Process – Part 1Step 2 – Derive Performance Measures from Objectives

• Imposed Constraints

– Performance objectives whose performance measures must remain within defined limits for every feasible alternative, give rise to imposed constraints that reflect those limits

– Imposed constraints propagate through the objectives hierarchy

– Imposed constraints include the success criteria for the undertaking, outside of which the top-level objectives are not achieved

• Example: If an objective is to put a satellite of a certain mass into a certain orbit, then the ability to loft that mass into that orbit is an imposed constraint, and any proposed solution that is incapable of doing so is infeasible

18

RIDM Process – Part 1Step 3 – Compile Feasible Alternatives

• Structuring the set of alternatives – Trade trees– Initially, the trade tree contains a number of high-level classes of decision

alternatives representing different strategies– The tree is developed in greater detail by determining option categories for each

strategy– Defined to the level required to quantify performance measures

• As the tree is developed, alternatives may be pruned

– Criteria• Infeasibility (e.g., does not

meet imposed constraints)• Inferiority to other alternatives

– Methods• Bounding analysis using point estimates• Expert opinion / deliberation

19

RIDM Process ReviewPart 1 – Identification of Alternatives

20

RIDM ProcessPart 2 – Risk Analysis of Alternatives

21

• Goal: to develop a risk analysis framework that integrates domain-specific performance assessments and quantifies the performance measures

– Risk Analysis - probabilistic modeling of performance

• The challenge is to establish a transparent framework that:– Operates on a common set of performance parameters for each

alternative – Consistently addresses uncertainties across mission execution domains

and across alternatives– Preserves correlations between performance measures

RIDM Process – Part 2Step 4 – Set Framework & Choose Analysis Methodologies (1)

Risk Analysisof an Alternative

Uncertain Conditions

Performance Measure 1

Performance Measure n

Probabilistically - Determined Outcomes

FundingEnvironment

Technology Development

Limited Data

OperatingEnvironment

Etc.

* Performance measures depicted for a single alternative

Design, Test & Production Processes

…• Safety Risk• Technical Risk• Cost Risk• Schedule Risk

22


• Setting the risk analysis framework (alternative specific)

23


• Choosing the analysis methodologies

– Detailed domain-specific analysis guidance is available in domain-specific guidance documents like the NASA Cost Estimating Handbook, the NASA Systems Engineering Handbook, and the NASA Probabilistic Risk Assessment Procedures Guide

– Depending on project scale, life cycle phase, etc., different levels of analysis are appropriate. The rigor of analysis should be enough to:

• Assess compliance with imposed constraints

• Distinguish between alternatives

– Iteration is to be expected as part of the analysis process, as analyses are refined and additional issues are raised during deliberations

24


• Choosing the analysis methodologies – “Consumer Guide” chart

• The rigor of theanalysis should besufficient tosupport robustdecision-making(i.e., the decisionmaker is confidentthat the selectedalternative is best,given the state ofknowledge)

25

RIDM Process – Part 2Step 5 – Quantify Performance Measures (1)

• Once the risk analysis framework is established and risk analysis methodologies determined, performance measures can be quantified

• Since performance measures are typically not independent, correlation between performance measures should be preserved

– For example, cost and schedule tend to be highly correlated. High costs tend to be associated with slipped schedules

• One way to preserve correlations is to conduct analysis within a common Monte Carlo “shell”

– For each iteration of the Monte Carlo shell, a common set of performance parameters is sampled and propagated through the entire suite of analyses, to produce the performance measures for that iteration

26

RIDM Process – Part 2Step 5 – Quantify Performance Measures (2)

• Quantification via probabilistic modeling of performance

27

RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate Performance Commitments (1)

• Performance measure pdfs constitute the fundamental risk analysis results

• However, there are practical difficulties comparing performance measures whose values are expressed as pdfs:

– Overlapping pdfs– Relationships between pdfs and imposed constraints– Relationships between pdfs and derived requirements

• To simplify the problem, one might use mean values to compare alternatives, but this approach can:

– Produce values that are disproportionately influenced by the tail ends of the pdfs– Introduce significant probabilities of falling short of imposed constraints, even

when the mean values meet imposed constraints– Lead to derived requirements that are not achievable

• What is needed is a technique for selecting alternatives, that is informed by an understanding of each alternative’s chances of not meeting performance expectations

28


• Performance Commitments– A performance commitment is a performance measure value set at a

specified percentile of the performance measure’s pdf

– Performance commitments help to anchor the decision-maker’s perspective to specific performance expectations for each alternative

– For a given performance measure, the performance commitment is set at the same percentile for all decision alternatives

– Performance commitments supporta risk-normalized comparison ofdecision alternatives, at a level ofrisk tolerance determined by thedecision maker


29

Imposed Constraint

Risk of not meeting specified

performance

Candidate Performance Commitments facilitate comparison of performance across alternatives, subject to the decision-maker’s risk tolerance for each Performance Measure

AlternativeA

AlternativeB

AlternativeC

PayloadCapability

30

RIDM Process – Part 2Step 6 – Develop Risk-Normalized Candidate Performance Commitments (4)• Developing Performance Commitments:

– The inputs to performance commitment development are:• The performance measure pdfs for each decision alternative• An ordering of the performance measures• A risk tolerance for each performance measure, expressed as a

percentile value

– For each alternative, performance commitments are developed by sequentially determining the value of each performance measure that matches the decision maker’s risk tolerance for that performance measure, conditional on meeting previously-defined performance commitments.

• This value becomes the performance commitment for the current performance measure

– The process is repeated until performance commitments have been developed for all performance measures

31


32

RIDM Process – Part 2Step 7 – Develop the Technical Basis for Deliberation

• The Technical Basis for Deliberation (TBfD) contains the information needed to risk-inform the selection of a decision alternative

• The TBfD contains:– A statement of the top-level objectives and imposed constraints– The objectives hierarchy and performance measures– A summary description of the compiled decision alternatives, indicating pruned

alternatives– A summary of the risk analysis framework and models– Scenario descriptions– Marginal performance measure pdfs and a summary of significant correlations– A tabulation of risk with respect to imposed constraints– Identification of significant risk drivers with respect to imposed constraints– Candidate performance measure risk tolerances

33

RIDM Process ReviewPart 2 – Risk Analysis of Alternatives

34

RIDM ProcessPart 3 – Risk-Informed Alternative Selection

35

RIDM Process – Part 3Step 8 – Deliberate (1)

• In Step 8, Deliberate, relevant stakeholders, risk analysts, and decision makers deliberate the merits and drawbacks of each alternative, given information in the TBfD

• This step is iterative, and may involve additional risk analysis and/or information gathering

• The decision maker, or his proxy, may also invoke deliberation as an intermediate step to cull the alternatives going forward (i.e., downselection)

Deliberation: Any process for communication and for raisingand collectively considering issues. In deliberation, peoplediscuss, ponder, exchange observations and views, reflect uponinformation and judgments concerning matters of mutualinterest, and attempt to persuade each other. Deliberationsabout risk often include discussions of the role, subjects,methods, and results of risk analysis.

36


• Step 8, Deliberate, is structured in terms of:

– Generate candidate performance commitments• Establish risk tolerances on the performance measures• Order the performance measures

– Assess the credibility of the estimation methods

– Identify contending alternatives• Infeasibility• Dominance• Inferior performance in key areas

– Additional uncertainty considerations• The potential for exceptionally high or poor performance

• Deliberation is iterative

37


• Generate Candidate Performance Commitments -- Candidate performance commitments are generated by the deliberators for the purpose of deliberation and down-selection prior to finalization by the decision maker. This is done by:

– Establishing risk tolerances on the performance measures:• Relationship to imposed constraints – Low risk tolerances on

performance measures that have imposed constraints assure a high likelihood of program/project success

• High-priority objectives – Low risk tolerances are appropriate for objectives that have high priority, but for which imposed constraints have not been setNote: The lack of an imposed constraint on a performance measure does not necessarily mean that the objective is of less importance; it may just mean that there is no well defined threshold that defines success

• Low-priority objectives and/or “stretch goals” – Higher risk tolerances may be appropriate for objectives that are not crucial to program/project success

38

RIDM Process – Part 3Step 9 – Select an Alternative (1)

• In Step 9, Select an Alternative, the deliberators present the contending alternatives to the decision maker, along with supporting information. The decision maker selects an alternative and documents his/her rationale

• In addition to information in the TBfD, information produced during deliberation should also be summarized and forwarded to the decision-maker. This includes:

– Risk tolerances and performance commitments – These are key pieces of information for the decision-maker. They strongly influence requirements development and the corresponding program/project risk that is to be accepted going forward.

– Pros and cons of each contending alternative – An itemized table of the pros and cons of each alternative is recommended for the contending alternatives. This format has a long history of use, and is capable of expressing qualitative and contentious issues

RIDM Process – Part 3Step 9 – Select an Alternative (2)

• Information forwarded to the decision-maker should also include:

– Risk lists – Each alternative will have different contributors to its performance commitment risks. Correspondingly, a risk list can be compiled for every contending alternative, which identifies the major uncertainties that contribute to risk

– Analysis credibility matrix – Communicates the credibility of the risk analysis methods and results

39

40

RIDM Process – Part 3Step 10 – Document the Decision Rationale• The Risk-Informed Selection Report (RISR) is a record of the risk-

informed decision, and documents the decision rationale. The RISR contains:

– The TBfD– From deliberation:

• Assessment of the credibility of the risk analysis• Identification of the contending decision alternatives• Pros and cons of each contending alternative• Any briefing material presented by the deliberators to the decision-maker

– From the decision-maker:• Identification of the selected alternative• The finalized risk tolerance for each performance measure, along with the

corresponding performance commitments for the selected alternative• Comparison of the selected alternative to the non-selected contending

alternatives, summarizing the relative pros and cons, and the reasons why the selected alternative is preferred

• Assessment of the robustness of the decision

41

RIDM Process ReviewPart 3 – Risk-Informed Alternative Selection

Summary - 1

• Risk-Informed Decision Making (RIDM) attempts to respond to some of the primary issues that have derailed programs in the past:

– the “mismatch” between stakeholder expectations and the “true” resources required to address the risks to achieve those expectations,

– the miscomprehension of the risk that a decision-maker is accepting when making commitments to stakeholders, and

– the miscommunication in considering the respective risks associated with competing alternatives

• A multi-step process has been developed to take advantage of existing systems engineering practices while also introducing risk analysis and systematic deliberative techniques into the decision-making process

42

Summary - 2

• OSMA has developed a Special Publication (in draft) and associated training material to provide implementation guidance

– Comments and suggestions from an agency-wide review cycle are being compiled and will be used in revising the draft for final publication

– To download a copy and participate in the process go to: https://secureworkgroups.grc.nasa.gov/armwg (PBMA registration and site access approval required)

• Future steps include:– Revision of how CRM should be conducted to be

consistent with 8000.4A and take advantage of the information provided by the RIDM process

– Better integration of the RIDM and CRM processes with the ultimate goal of a completely integrated and seamless Risk Management process

43

https://secureworkgroups.grc.nasa.gov/armwg�

Homayoon.dezfuli

Technology

management of risk

risk informed decisions

risk analysis results6

i t y h e

based risk metrics

ridm process themes

b e e n r e v

nasa hierarchy ridm