1 INCOSE HRA Advanced Risk Management Conference 2007 Courtney Lane INCOSE HRA Risk Management Conference November 9, 2007 Its More Than Just Numbers:

1

INCOSE HRA Advanced Risk Management Conference 2007

Courtney LaneINCOSE HRA Risk Management Conference

November 9, 2007

It’s More Than Just Numbers: How to Implement a Qualitative

Risk Analysis Approach

This document is confidential and is intended solely for the use and information of the client to whom it is addressed.

2


Purpose

Describe the difference between qualitative and quantitative risk analysis

Discuss common problems associated with qualitative analysis

Examine best practices for implementing a qualitative risk analysis approach

3


Standard Risk Management Process

Develop Strategy

This presentation will focus on the analysis sub-process. This sub-process should be defined in an organization’s risk management strategy

Analyze

PlanMonitor

and Control

Identify

4


Components of Risk Analysis

Probability of occurrence

Impact to a project’s cost, schedule, and technical objectives

Timeframe

Handling method

By identifying these components, project managers will be able to determine the magnitude of the risk and effectively prioritize handling of the risks

5


Difference Between Qualitative and Quantitative Analysis

Qualitative vs. Quantitative

Use ranges to select probability and impact rankings

Absolute numerical estimates are used for impact and probability

Focuses on overall level of individual risks (e.g. high, medium, low)

Allows project’s to aggregate individual risks to determine the overall level of project risk

Allows programs to quickly prioritize risks based on a number of criteria

Uses complicated analysis such as Monte Carlo simulation to prioritize risks

6


Why use Qualitative Analysis over Quantitative?

Lack of historical data

Lack of the necessary time and resources to conduct complicated analysis

Desire to use criteria for prioritization that is difficult to quantify– Timeframe

– Critical dependencies

– Political considerations

The type of analysis appropriate for an organization is dependent upon the risk management maturity level of the organization

7


Risk Management Maturity ModelLevel Characteristics

1: Chaotic Risks are ignored or managed in an ad hoc fashion

2: Reactive Processes definedManagement uncomfortable discussing riskMost efforts spent on resolving issues

3: Proactive Processes standardized across the organizationManagement comfortable with risk managementRisk management data used in decision making processes

4: Predictive Use of quantitative analysis to predict future resultsCost/benefit analysis implemented

5: Optimization Opportunities to exceed project objectives are identified and managed

Source: Modified from SEI’s Capability Maturity Model

Most organizations are not at an appropriate maturity level to implement quantitative risk analysis

8


Top Problems Related to Qualitative Analysis

#1Using inappropriate criteria to determine probability and consequence levels

#2 Trying to “quantify” qualitative data

#3Relying on risk scores generated by risk management tools to determine a list of top risks

9


Problem #1: Inappropriate Criteria

Criteria is often too detailed

Subject matter experts lack the necessary expertise to appropriately select the criteria

Easy to “game” the analysis since justification of criteria selection is extremely subjective

Poor criteria definitions make it difficult for risk analysts to accurately reflect impacts to cost, schedule, and performance

10


Solution: Understand Your Audience

Determine what questions decision makers want the risk management process to answer

Assess the level of expertise of those performing the risk assessments

Select criteria that balances the needs and abilities of both parties

11


Examples of Qualitative Analysis CriteriaNegligible

(A)

Marginal

(B)

Significant

(C)

Critical

(D)

Catastrophic

(E)

Cost Minimal or no impact

$25K-$50K $50K-100K $100K-250K >$250K

Schedule Schedule slip < 7 days

Schedule slip between 7 and 14 days



Schedule slip greater than 1 month

Performance Minimal or no impact

No impact to key performance parameter; degraded capability performance

Degraded enterprise capability performance

Degraded capability performance with impacts to one key performance parameter

Degraded capability performance with impacts to more than one key performance parameter

Criteria is too detailed and thus difficult to assess

12



(A)

Marginal

(B)

Significant

(C)

Critical

(D)

Catastrophic

(E)


<5% of overall budget

5-7%of overall budget

7-10% of overall budget

>10% of overall budget

Schedule Minimal or no impact

Additional resources required; able to meet need dates

Minor slip in key milestones; not able to meet need dates

Major slip in key milestone or critical path impacted

Can’t achieve key team or major project milestone


Acceptable with some reduction in margin

Acceptable with significant reduction in margin

Acceptable; no remaining margin

Unacceptable

Source: Risk Management Guide for DoD Acquisition

Criteria is extremely subjective and could be easily “gamed”

13



(A)

Marginal

(B)

Significant

(C)

Critical

(D)

Catastrophic

(E)


Cost impacts contained within the project

Cost impacts contained within the program

Cost impacts contained within the organization

Cost impacts affect external partners and could jeopardize program

Schedule Minimal or no impact

Impacts to critical path contained within the project

Impacts to critical path contained within the program

Impacts to critical path of more than one program in the organization

Impacts the critical path of external partners and could jeopardize program


Impacts to critical functions of the project

Impacts to critical functions of the program

Impacts to critical functions of the organization

Impacts to critical functions of external partners and could jeopardize the program

Criteria definitions are easy to assess and impact levels are consistent across criteria

14


Problem #2: Quantifying Qualitative Data

Engineers often feel more comfortable using numbers when performing analysis – Assign numbers to levels of impact– Use specific probabilities instead of ranges– Risk scores often determined by multiplying impact by probability

However, most risk analysis uses uncalibrated ordinal scales– Cannot perform math on uncalibrated scales– Gives a false sense of rigor– Decisions made on inaccurate data

Performing math on qualitative data may lead to inaccurate conclusions and a false sense of rigor

15


Solution: Do Not Include Numbers in Analysis

Avoid assigning numbers to levels of impact and probability– Use probability ranges

– Use letters to represent levels of impact and probability

Use the highest level of individual impacts to determine overall impact (don’t average impacts)

Use a risk matrix to determine overall level of risk

16


Example Qualitative Risk Analysis

Overall level of risk

is Red

17


Problem #3: Overemphasizing Risk Scores

If problem #2 is not overcome, this becomes a much larger problem

Even if numerical scores are not used, rankings could be incorrect– Risk tools can generate a list of top risks based on locations on the

risk matrix

– Cannot rely solely on the placement on the risk matrix

– Dependencies, political issues and timeliness factors also play a part

Overemphasis of risk scores to determine top risks results may result in poor decisions made by senior leaders

18


Solution:

Tailor list of top risks based on the audience

Consider all factors, not just likelihood and consequence, when creating the list

Do not use arbitrary thresholds to determine handling strategies

Aim to manage all yellow and red risks

19


Summary and Conclusion

Select the appropriate type of analysis based on the risk management maturity level of the organization

Typical challenges associated with qualitative analysis can be overcome:– Know your audiences

– Avoid using numbers to perform analyses

– Consider other factors when selecting list of top risks

Qualitative analysis can provide leaders with enough information to make sound decisions if implemented properly

20


Contact Information

Courtney LaneSenior Consultant

Booz Allen [email protected]

703-961-8009

1 INCOSE HRA Advanced Risk Management Conference 2007 Courtney Lane INCOSE HRA Risk Management Conference November 9, 2007 Its More Than Just Numbers:

Documents

risk management tools

risk scores

risk analysts

risk assessments

risk management maturity

overall level of project

performance slide

analysis subprocess