09 November 2007 Insights from Assessing the Risk Management Programs of Major Defense Acquisition Programs Peter Lierni, PMP, CISA Lierni © 2007.

09 November 2007

Insights from Assessing the Risk Management Programs of Major Defense Acquisition Programs

Peter Lierni, PMP, CISA

Lierni © 2007

Lierni © 2007

2

Contributors to Risk ManagementBody of Knowledge

International Council on Systems Engineering (INCOSE) Defense Acquisition University (DAU) Project Management Institute (PMI) Software Engineering Institute (SEI) Information System Audit and Control Association (ISACA) et al

Lierni © 2007

3 Risk, Issue, and Opportunity Management

Risks — future uncertainties that could have an adverse impact to cost, performance, schedule, etc.

“Apply to both products and processes”

Issues — “realized risks” that are now problems and impacting things such as cost, performance, schedule, etc.

“Issues could generate other risks”

Opportunities — future uncertainties that if realized could improve cost, performance, schedule, etc.

Distinguish between risk, issue, and opportunity management

Lierni © 2007

4

Questions a Mature Risk Management Program Should Be Able to Answer

What are the program’s risks? (e.g., technical, schedule, cost, etc.) What is the impact of these risks on program efforts? Have risk mitigation action officers been assigned? Have risk mitigation action due dates been assigned? How far has the program come with respect to accomplishment

of planned mitigation actions? How far has the program come with respect to meeting program

objectives? How does the program stand with respect to comparable efforts? To what extent has experience been incorporated into the

program?

Lierni © 2007

5

Essential Elements of A Mature Risk Management Program

Thorough risk assessment as part of Business Case Analysis (BCA) that at minimum addresses cost, schedule, and performance

Early and incremental and iterative use of Modeling and Simulation (M&S) and testing throughout the Systems Development Lifecycle (SDLC) to investigate various design options and system level requirement

Acquisition Strategy that addresses all major known program risks “Include risk mitigation strategies and assumptions, as well as rationales for

assumptions”

Experience from similar programs sought and incorporated to enable risk reduction before initiating program planning

Risk management well integrated with program’s SE approach

Technical baseline of system being acquired continually used to assess technical risk

Lierni © 2007

6

Key subcontractors/suppliers integrated into risk management planning, execution, monitoring, and control activities

“Most effectively accomplished through Integrated Product Team (IPT)” Well-chartered and executed decision-making bodies [e.g., Configuration

Control Board (CCB), IPTs, Risk Review Board (RRB)] Joint Government/Contractor collaborative relationship and feedback

mechanism in existence between all SE activities and risk management

“Risk data should be visible, accessible, and understandable to both” Risk Watch List and Program-level Risk Cube Program/technical reviews provide input into risk management process

and vice versa Evident how the Government/Contractor PMs use program/technical

reviews to assess risk

Essential Elements of A Mature Risk Management Program (Continued)

Lierni © 2007

7

Trade studies that consider risks associated with alternatives

Continuity in explanation of how risk is addressed in key program documentation [e.g., Acquisition Strategy Report (ASR), Request for Proposal (RFP), Program Management Plan (PMP), Systems Engineering Plan (SEP), Risk Management Program Plan, Test Evaluation Strategy (TES), etc.]

“Best efforts should be made to ensure documents reflect most current risk status and critical mitigating actions of the overall program”

Web-enabled Risk Management Information System (RMIS) to ensure awareness and accessibility of risk information (horizontally & vertically)

“Information in RMIS should be current, complete, clear, and cogent to best enable decision-making bodies. Particularly important with regards to traceability, monitoring, and control of mitigation plans”

Essential Elements of a Mature Risk Management Program (Cont.)

Best enabled by a core set of risk management principles and effective leadership

Lierni © 2007

8

Use Holistic Risk Management Approach to Navigate the Program

1. Change Management — ability to accommodate change that reduces adverse impact

“Risk management should be viewed as a component of change management”

2. Communication Effectiveness — ability to reduce the likelihood of misunderstandings among parties involved in business dealings

3. Risk Assessment — ability to set or determine risk amount and its potential impact

4. Risk Mitigation — ability to reduce adverse impacts of assessed risks

5. Performance Management — ability to manage change (and risk) quantitatively

6. Knowledge Management — ability to accumulate and apply knowledge for organizational benefit and growth

Lierni © 2007

9

Risk Management Is Inherently A Natural Part of Change Management

Change is constant with all programs Change results from known things “as planned” Change results from eventual unknowns “as unplanned”

Change management is the ability to accommodate change that reduces adverse impact

Lierni © 2007

10 Look and Risk & Change Three Ways

Change occurs to address risk (i.e., to eliminate something negative) or to realize something positive

Risk analysis should be performed on change being considered

“Technology X vs. Technology Y”

Risk mitigation plans may have to be implemented to address risks that could prevent the change from successfully being implemented

Lierni © 2007

11

Mechanisms to Accomplish Effective Risk Management

Effective communication is essential to managing risk

Risk management can best be accomplished through the use of CCB and IPT processes

“These forums when properly implemented provide a proven means for reducing the likelihood of miscommunication”

Lierni © 2007

12

Integrating Risk Management with Program Controls

Objective of a well-managed risk management program is to provide a tool for balancing cost, schedule, and performance goals within program funding

“Especially on programs with designs that approach or exceed state-of-the- art or have tightly constrained or optimistic cost, schedule, and performance goals”

Often there is lack of a linkage amongst the following processes such that they are used as effective tools to enable risk reduction:

Work Breakdown Structure (WBS) Integrated Master Plan (IMP)/Integrated Master Schedule (IMS) Earned Value Management (EVM) Performance Measures Risk Mitigation/Issue Recovery Plans Knowledge Management (KM)

Most effective when working together vs. alone

Lierni © 2007

13

Rationale for Integrating Risk Management with Program Controls

Increased vigilance Better communication Increased responsiveness Enhanced internal programmatic controls Improved application of Management Reserve (MR) Increased learning Better technical planning

All of these attributes when evident are enablers of program risk reduction

Lierni © 2007

14 WBS

Technical risk management should be based on individual product or specific critical processes (e.g., design, development, and test) affecting individual WBS elements

Risk assessments and mitigation activities should be conducted on individual WBS elements “Emphasize technology, product/process maturity or perceived quality

and deviations from the cost & schedule baseline”

IPTs should carefully review those sections of WBS that they are responsible for to identify, assess, and track technical risks

IPTs should primarily look for impact on cost and schedule, and the resulting effect on the overall product

Identified WBS-derived risks and associated mitigation plans should have related WBS element number for the risk specified

Implementing a risk mitigation plan is an important reason for a scope change and should be reflected in WBS “Updates should be done in a timely manner and reflected in updates to

IMS”

Lierni © 2007

15 IMP/IMS

Employ as tool for planning, executing, and tracking risk mitigation efforts

“Conduct Schedule Risk Assessments”

Ensure significant risks identified by Government in RFP are addressed in Contractor IMS in response to Government

Use IMP/IMS to enable risk management “Ensure staff responsible for IMP/IMS process work with IPTs to

regularly identify moderate-to-high risk tasks to ensure that specific risk reduction (handling) activities are properly reflected in IMS”

Have Program Manager (PM) regularly assess the status of risk management activities based on inclusion of risk mitigation activities in IMP/IMS

Ensure risk mitigation activities in IMP/IMS are flagged so that they are easy to call out from the other tasks in IMP/IMS

Lierni © 2007

16 IMP/IMS (Continued)

Do not manage IMS at exclusion of risk management

“An IMS summarized at too high a level often results in masking critical elements of the plan necessary to execute the program and fails to show the risk management approaches being used”

Review IMS for completeness and consistency with program staff responsible for IMS

“Work together to evaluate duration and logical relationships to ensure they will accomplish the desired risk mitigation”

Lierni © 2007

17 EVM

Ensure specific risk-handling actions are reflected in detailed work packages as part of performance baseline

Have IPTs monitor effectiveness of risk-handling actions by providing periodic comparisons of actual work accomplished in terms of cost and schedule with the work planned and budgeted

Analyze cost/schedule variances in work packages containing risk-handling actions to isolate root causes and gain insights into need to modify actions

Understanding root causes of cost/schedule variances in work packages containing risk-handling actions allows opportunity to improve technical planning

Lierni © 2007

18 Enable Vigilance

Lack of Measures-driven Approach to Risk Management

Measures-driven Approach to Risk Management Monitor areas of known risk (e.g.,

product, process, people, etc.)

Provides early detection of new risks before irrevocable impacts on cost/schedule occur

Use to assess effectiveness of risk-handling actions

Lierni © 2007

19 Think SIPOC

Program staff responsible for individual processes should employ the notion of Supplier, Input, Process, Output , Customer (SIPOC) from Six Sigma (∂) “Enables better communication and collaboration with other program staff

responsible for individual processes” Effective choreography should exist amongst all program staff

responsible for individual processes for integrated risk management to truly be realized

Measures should be developed with participation of Government/ Contractor stakeholders so that they:

Answer stakeholder’s question(s) Focus on the key thing(s) necessary to answer the question Reflect stakeholder’s vocabulary Are weighted according to what stakeholder believes is important

Lierni © 2007

20

Bad vs. Good Risk Management and Funding Availability to Mitigate Risk

Lack of a measures-driven approach to program risk management causes .... Bad Risk Management

Greater portion of program funds expended mitigating risk associated with

known and known/unknown risks

Less program funds available to mitigate any possible risk associated

with unknown/unknown risks

Greater portion of program funds available to mitigate any possible risk associated with

unknown/unknown risks

Less program funds expended mitigating risk associated with

known and known/unknown risks

As compared to a measures-driven approach to program risk management which causes .... Good Risk Management

Lierni © 2007

21 Evidence of Proper Risk Mitigation Planning

Mitigation Plans task-oriented with realistic and achievable actions

Planned and actual start/completion dates

Action officer accountable for over all status of mitigation plan

Individual(s) and/or organization(s) assigned responsibility to implement and report status of assigned tasks

Requisite resources (i.e., personnel, capital equipment, facilities, procured items) identified

Included in IMS, particularly risks with initial status of “red”

Funded with MR employed as necessary

Mapped to WBS to at least tier-three level

“Enables better EVM”

Quantifiable and/or tangible measures of success for closure criteria

Lierni © 2007

22

Evidence of Proper Risk Mitigation Planning (Continued)

Likelihood of risk realization as near-term, mid-term, or far-term event

Logical explanation of reduction to probability and/or impact Off-ramp (contingency plan) enabled by trigger point(s),

particularly for “red” and higher-rated “yellow” risks being tracked

“Contingency plans should be developed and implemented in same manner specified for risk mitigation plans”

Minutes to date available that reflect mitigation plan reporting status and outcomes

Risk mitigation plans with “risk burn-down” graphs

Lierni © 2007

Lierni © 2007

23

Prevent Organizational Knowledge from Being Lost

Systematically secure knowledge gained from outcomes of risk mitigation

“Certain knowledge and useful experiences otherwise gained could be lost to the detriment of future technical planning and improving risk reduction on the program, as well as the portfolio of programs within the organization”

Have PM continually seek out and capture lessons learned, particularly as root cause analysis is performed throughout the program

Use “knowledge gained over time” to improve processes, as well as entry and exit criteria of program/technical readiness reviews

Lessons Learned

“1) Repeatable 2) Traceable 3) Assignable 4) Measurable 5) Provides Benefit”

Knowledge Certainty Control Risk

Lierni © 2007

24 Risk Program Health Metrics

Employ metrics to assess overall risk management program effectiveness. Emphasize:

1. Trends2. Planning3. Accountability4. Communications effectiveness

Without the last three items, risk is less likely to be mitigated and issues recovered from

Lierni © 2007

25 Example Risk Program Health Metrics

# of Total Current Period vs. Total Prior Period Open Risks by Status (e.g., Red/Yellow/Green) by IPT

# Total Risks by IPT

# of Total Risks by Status

# of Total Risks by IPT by Status

# of Total Current Period vs. Total Prior Period Open Risk Aging (e.g., 1-30 days/31-60 days/61-90 days/ 91+ days) by IPT

# of Total Risks by Age

# of Total Risks by Age over Time for Current Period vs. Prior Period

# of Total Risks by Age by IPT

# of Total Open Risks by Specific Mitigation Plan Action Officer

# of Mitigation Plan Action Items (i.e., Tasks) Open/Closed vs. Total across All Plans

# of Mitigation Plan Action Items Open/Closed vs. Total across All Plans by IPT

Lierni © 2007

26 Example Risk Program Health Metrics (Continued)

# of Total Mitigation Plans Assigned and Not Developed

# of Total Mitigation Plans Assigned by IPT and Not Developed

# of Total Mitigation Plans Assigned by Current Status and Not Developed

# of Total Mitigation Plans Assigned by IPT by Current Status and Not Developed

# of Mitigation Plans developed and Not Reflected in IMS

# of Mitigation Plans Developed and Unfunded

# of Mitigation Plans Developed without Resources Identified for All Tasks

# of Mitigation Plans Developed without Due Dates Identified for All Tasks

# of Mitigation Plans Developed without Tasks Currently Reported on at latest IPT/RRB meeting (e.g., Risk Open/Closed/other Update)

Metrics best presented graphically!

Lierni © 2007

Lierni © 2007

27 Summary

Contact Information: [email protected]