James.taylor

0

Getting What you Paid for:Acquisition Risk Analysis

James TaylorOmnia Paratus

CorporationUsed with Permission

1

Omnia Paratus Corporation

Lamborghini Reventon

$1.6M

Kia Rio

$12,145

2


A recent poll showed that 68% of companies experienced financial losses directly related to Supply Chain disruptions

Most of the financial impacts were related to supplier performance that did not meet demand requirements, and delayed, damaged or misdirected shipments.

The majority of companies polled are in the early stages or have yet to think about integrating Risk Management into their Supply Chain

Supplier related risks are most often identified after contract award is a program issue, not a risk:– Supplier shipment delay – Supplier capacity exceeded– Unable to meet technical requirements – 1st Article or Flight Test failures– Parts damaged during shipment or rejected during quality inspection

Schedule delays caused by supplier performance can have an equal or greater financial impact to a program’s bottom line, but are often overshadowed budget impacts due to supplier cost overruns

3


Procurement analysis exists in varying degrees, although the most commonly used practice is Best Value Analysis (BVA)

Incumbent Supplier w/ Existing Capabilities

New Supplier

Score based on savings to program

~$300K

Technical Score based on:- New Capability to Supplier- Technical deviations req’d based on proposal- 1st Article & Flight Req’d based on Customer reqt’s

BVA considers not only cost, but other quantifiable and non-quantifiable factors supporting an investment decision– Utilizes weighting scales for analyzing “True” program value of supplier bids– Can include, but is not limited to, performance, producibility, reliability, maintainability, and

supportability enhancements– Intended to select the source offering the greatest overall benefit in response to the requirement

4


While a BVA attempts to provide insight to the least risky procurement, it does not specifically address or integrate impacts of risk

Technical Deviations required by Supplier

Design changes resulting from Technical deviations resulted in cost growth to original purchase order

Program Award fee lost resulting in Supplier delays cause IMS milestones to be missed

Orders placed were against original design and fulfilled by Incumbent suppler

Impact from production delays of 4 weeks

Fees required to get Incumbent supplier operations back up and minimize delay of deliveries to Customer

$390K overrun was a direct loss of company profit

Performing a sample analysis of a new supplier shows the potential risks that may be incurred when basing decisions on costs alone– Technical requirements outside current capabilities– Schedule impact due from potential delay of 1st Article Testing or Flight Test Requirements– Cost growth due to technical deviation’s required– Impact to Operations due to late supplier deliveries

5


Despite the value-add it offers to improving the acquisition process, risk management is seldom considered or implemented

Risk Management Is Not Difficult, Is Well Documented, and Training Materials Are Readily Available. So Why Is It “Hard”? – Seldom seems urgent It’s a “lower right quadrant” activity It’s often overcome by events It’s someone else’s job

– Requires careful thought People think it’s “easy” because it’s not difficult Fail to distinguish between perception and reality Skip the analysis and solve the wrong problem Determining what can be controlled, influenced, or changed

– Team participation Part of the culture Common understanding Training, support, and reinforcement

Risk Management

Low High

Low

H

igh

Importance

Urg

ency

6


Risks that impact large scale programs often directly originate from supplier performance throughout all phases of a program

The main challenges that confront these programs include:– Program Requirements – Technical requirements can have a tendency to increase unknown

potential for inhibiting program success or realization of full award fee are often over looked while developing scope of work and supplier selection process.

– Source Selection Analysis - A review of industry standard Best Value Analysis or Lowest Cost Alternative approach reveals gaps in several areas of Procurement Analysis skewing the perception of results derived, inherently selecting a supplier who may not adequately meet program requirements.

– Risk Identification – Most risks identified often have root causes stemming from supplier performance and/or capabilities, these risks tend to be identified after procurement award.

– Risk Impact Analysis – Procurement analysis does not provide insight into the potential cost and schedule impacts associated with selecting a supplier, impacts that could affect the success or ability of program operations, award fee and sustaining customer contracts.

– Mitigation Analysis – Developing a mitigation plan after a problem has occurred limits a program’s mitigation options resulting in exuberant mitigation costs that extend beyond program office and/or client budget.

7


Request for ProposalDevelopment Proposal Evaluation Contract Award

Integrating Risk Management directly into the acquisition analysis process is seamless and beneficial

Develop and issue RFPs based on a standard format; collect vendor proposals

Des

crip

tion

Evaluate proposals using a clear and structured evaluation mechanism and methodology

Activ

ities

Assess program requirements Assess supplier capabilities Assessment methodology &

criteria development Define risk parameters

RFP responses collection Technical and contractual /

procedural evaluation Risk identification Risk ratings defined

Supplier proposal risk analysis Identify potential mitigation plans Assess mitigation posture Integration of risk impact into

program budget and schedule

Out

com

e Detailed functional and technical requirements

Evaluation methodology Scoring model for RFP evaluation Vendor bidders list RFP document Risk rating scales & categories

Technical, contractual / procedural and commercial evaluation of proposals

List of potential risks and risk ratings within individual proposals

Pre-Mitigation Analysis Post-Mitigation Analysis and

Effectiveness Program level risk adjusted

cost and schedule analysis Supplier selection Contract Award

Analyze potential risk impact and mitigation posture relating to supplier proposal’s

Assess program requirements, supplier capabilities and evaluation criteria to establish RFP

Provide insight into where supplier risks affect the program and uncover their true impacts.

Compare supplier risk profile at proposal evaluation completion to determine outstanding risk exposure.

8


Acquisition Risk Analysis follows a typical Risk Management process and can be tailored to program specific needs

Risk Identificat

ion

Sets risk parameters for effective risk analysis

Examines risks to determine impacts by

and across alternatives

Identifies risks that may impact cost,

schedule, or performance

Develops measures for keeping risk at

acceptable levels

Uses results of risk analysis in assessing

alternatives

Risk Planning

• Collecting SME input, best practices, metrics, etc., to identify possible risks

• Documenting risk data into a central repository

1 Risk Identification

• Creating standard risk terms and definitions

• Developing processes, criteria, and scoring approach for risk

Risk Analysis

• Conduct probabilistic assessments of risk impacts

Risk Mitigation

• Identifying, evaluating, and selecting strategies to reduce risk

• Developing an actionable risk mitigation plan, with sound rationale (if applicable)

Applying Risk Results

• Using cumulative risk scores as a vehicle for ranking alternatives

2 3 4 5

9


Building qualitative definitions for risk ratings enables an objective, not subjective quantification of proposal risks The first step in Risk Planning involves defining standard risk terminology using many different resources, but

should be robust and accurate enough to reflect the program’s overall risk tolerance

Developing a robust, well-defined risk analysis process will produce results that reflect program needs; effectively communicates these needs across stakeholders; reduce personal bias in determining relative risk importance; and uncover potential impacts of great importance to the program

Risk criteria is not normative, and are based on available resources, time constraints, and amount/type of information solicited in generating the criteria

Performance Metrics

The analyst needs to ensure that the units of measure for risk impact reflect program needs. Critical program drivers might include life-cycle costs or metrics reliability (e.g., Mean Time Between Failure).

Order of MagnitudeIf impact thresholds are expressed in percentages (for example) the analyst needs to review those figures within the context of the overall anticipated program cost—e.g., 10% of $5 million program is substantially different than 10% of a $5 billion program.

Level of ImpactThe analyst needs to ensure that there is an adequate number of thresholds for evaluating the risk impact. The more levels of consequence that are utilized, the greater the insight into the need for increased data fidelity/quantity of data the analyst would need to consider.

Qualitative vs. Quantitative

Qualitative data is represented by probability values ranging from “Very Unlikely” to “Very Likely” , using stakeholder input to assess and weigh the importance of benefits, cost, and risk in relationship to the total analysis or evaluation of benefit, risk, and cost factors that cannot be quantified.

Quantitative data is represented by consequence values using linear numerical probabilities (e.g., 0.1, 0.3, 0.5, 0.7, and 0.9), nonlinear numerical probabilities (e.g., 0.05, 0.1, 0.2, 0.4, 0.8), cost estimates, metrics, ranges, or percentages.

Considerations for Defining Risk Criteria

10


A Supplier Management Maturity module is used to determine if supplier capabilities comply with program requirements

Framework used to assess each IPTs risk management maturity in terms of People, Process, Technology, and Governance

The optimal goal of the risk enhancement effort is to select supplier cable of bidding based on their “Best in Class” maturity

Regular training conducted to enhance skills and capabilities Dedicated organizational resources All staff is informed and capable of applying mid- to advanced concepts

Applied use of specific processes/tools

Dedicated team resources In-house core experts, formally

trained in basic skills

Dedicated resources do not exist

Limited to individuals who may have had little or no formal training

Minimal understanding or experience in applying basic concepts and principles

Minimal understanding of principles or language

Integral to informed decision-making by upper management

Active use encouraged and rewarded Part of the organizational philosophy to

achieving program success Top-down commitment by leadership

• Accepted as a program management function

Benefits recognized and expected Upper management requires

tracking and reporting Focus on mitigation effectiveness

versus reporting and status tracking

Upper management encourages, but does not require use

Application varied through out program

• Process may be viewed as additional overhead with variable benefits

• Minimal awareness Minimal upper management

involvement Tendency to continue with

existing processes even in the face of potential failure

Dedicated resources do not exist

State-of-the-art tools and methodologies

Distributed data environment that provides access to all program resources

Standardized and automated reporting capabilities

• Integrated set of tools and methodologies

Centralized data environment managed by dedicated team resources

• Customizable solutions tailored to program

• Few repeatable technological solutions in place

• No structured application Management and tracking tools

not in use Analysis not performed

Qualitative/quantitative analysis methodologies employed with emphasis on valid and reliable data sources Metrics used and reported, with consistent feedback for improvement External stakeholders actively participate in process Integration into organizational processes and decision-making

Formal processes integrated into different areas of the program

Active allocation and management of budgets

Metrics collected Key internal stakeholders actively

participate in process

Common processes defined and formally documented

Process effectiveness limited to a dedicated team

Qualitative analysis based on ill-defined rating system

Established decision-body forum

Inconsistent application of concepts or principles

Formal risk decision-making body does not exist

Risk reporting and/or metrics is minimal or does not exist

Formal, documented process does not exist

4 – Best in Class3 – High Performance2 – Functional1 – Minimal Capability

Peop

lePr

oces

sTe

chno

logy

Gov

erna

nce

NOTIONAL

People Process Tech Gov’t Metrics Quality Capacity Average Rating

3 4 4 1 2 2 1 2.42

2 3 - 3 3 1 1 1.85Supplier B

3 3 4 2 3 3 2 2.85Supplier C

3 3 4 4 3 3 2 3.14Supplier D

Supplier A

Supplier’s will be scored based on the maturity model to establish a list of qualified suppliers for proposal solicitation

NOTIONAL

11


Once proposals have been received a technical evaluation should be conducted for clarifications prior to identifying supplier risks

Proposal Clarification Items

Topic

General

Area

Staffing

Questions

Is the team available as indicated in the Proposal? Additionally, a further elaboration on local staff/ qualifications is

required

Ref.

P. 60

Approach

A further elaboration on the timeline is required, especially on the 9 days timeframe proposed for defining the architecture and standards

A show-case of the deliverables must be provided How realistic are the assumptions on available data and

Organization staff?

P. 46

Assumptions What is the impact on costs and timeline of a provision of a documentation in other languages?

P. 18

Phase 0: Kick-Off

Phase 1: Baseline

Phase 2: Best Practice Review

Alignment

Tools

Approach

How will Bidder ensure the alignment of the PM tools to the Organization’s PMO?

How will the automatic “systems” feed work (XML). Is it a requirement?

How will any incompatibilities of import functions affect the suggested timeline?

What if the Organization wants to use other EA tools, like ARIS? Further elaboration on “GEAS” layers required. A mapping to the

Organization framework layers is needed as well Explanation of the rationale for selecting Singapore, Australia and

Canada Is the access to those countries only via databases/benchmarks or

“real” contact?

P. 34

P. 36-38

P. 43

Phase 3: Definition of Architecture &

StandardsImperatives

No reference given, on how the Organization’s imperatives will be ensure – therefore, a further elaboration on the approach is required

P. 45

Answer Given Issue Resolved

All questions to a Bidder should be compiled and sent in a formal email / fax

All Bidders should be given ample and the same amount of time to respond to clarifications requested

12


The next step to the risk assessment process is to identify risks that may impact a program’s cost, schedule, or technical performance……

Risk Identificat

ion

The risk analysis focuses on risks that affect each supplier’s ability to ensure program success (i.e., ongoing, uninterrupted support to the operational forces).

Although identification of risk relies on the skill, experience, and insight of subject matter experts and risk personnel, the methods and tools for initiating the identification of risk may vary

As such utilizing risk categories such as these bolsters risk identification…

“Common” Risks

Scope of Work

Testing & Integration

Material Availability

Suppler Metrics Schedule Budget Constraints

Qualification Requirements

Supplier Capabilities

Performance Metrics

Resources

Lessons Learned

ManagementComponents Program Requirements

Sensitivity Analysis

13


Technical

…each risk is then reviewed against a set of Program-specific impact definitions, with the highest rating used to determine overall risk severity

Cost ( % over of cost target) Supplier Delivery Schedule

Existing technology does not exists

Existing technology exists, but has not been proven

Supplier has never built component before, Flight Test required

Supplier has never built component before, 1st article required

Supplier has never built component before

Supplier built similar component, Flight Test required

Supplier has built a component similar in nature, 1st article required

Supplier has built identical component for other Programs

Supplier has previously built component

Incumbent supplier

Level

Disastrous

Severe

Critical

Substantial

Significant

High

Moderate

Medium

Low

Minimal

Technical = High 3 Schedule = Critical 1 Cost = High 2

Consequence Rating = Critical

> 6.00%

5.01 – 6.00%

4.01 – 5.00%

3.01 – 4.00%

2.01 – 3.00%

1.01 – 2.00%

.76 – 1.00%

.51 - .75%

.26 - .50%

< .25%

>5 month slip in MRP requirements

4-5 month slip in MRP requirements




< 1 month slip in MRP requirements

Risk erodes 100% of schedule margin

Risk erodes 51 – 75% of schedule margin

Risk erodes 26 – 50% of schedule margin

Risk erodes < 25% of schedule margin

14


Risk #

WBS Impact

IMS Task ID Risk Description Risk

RatingCost Impact

($K's)

Schedule Impact (Days)

1 1.2.1.353

Lack of supplier staff to support delivery requirements

3 8 31.00$7,500 88

2 2.3.1.6 71 CCB is a development item 5 9 71.25 $8,500 118

3 4.3.5.1.1101

Supplier X's On Time Delivery rating subpar

2 10 28.50$9,500 150

4 6.3.3.788,97

Requirements of flux capacitor increase the need for Supplier X System Test & Eval staff and resources

4 6 32.25$5,500 52

5 1.3.1.570

Lack of technical capability in power generation / storage hardware may cost / schedule over runs

1 3 3.30$2,500 18

6 1.3.1.21.3.3.15 67,97

Tooling re-use approach may not be compatible with new technology

4 10 71.25$9,500 150

Prob. Conseq.

Risks are then integrated into the suppliers proposal to determine impacts of risks based on the risk ratings identified The risk database transposes risk rating into cost and schedule impacts in quantifiable dollar and

days

WBS and IMS task are later used to integrate supplier risk adjusted proposal into the Program’s Budget and IMS

15


Monte Carlo Analysis is used in calculating risks associated with supplier cost impacts to program budget….This chart example utilized Crystal

Ball to perform Monte Carlo, multiple types of software exists and can be used for this analysis

Pick a Confidence Level based on program maturity and requirements

This example highlights the 75th

Percentile– 75% of costs are below the line,

25% of costs are above– The 75% CL is $49,448

16


….as well as supplier risks that have the potential for disrupting a program’s Operations schedule.

The analysis example utilized SCRAM to run the schedule risks analysis

Confidence Level (CL) picked for cost is also used for schedule to determine how risk may impact a supplier’s delivery schedule

17


Schedule & Cost Risk Assessment Module (SCRAM) is a MS Project add-in and FREE for use on NASA projects.

SCRAM’s capabilities compare with that of Pertmaster, @Risk and Risk++

Extremely user friendly and reliable

Customizable aspects not available with other tools

Compatible will all MS Office Products

18


Once Monte Carlo simulations are complete and confidence levels selected, supplier bids can be compared based on potential risk impacts

Proposed $ 494,958 $ 465,135 $ 514,453

10% $ 507,645 $ 512,622 $ 547,565

20% $ 509,542 $ 523,002 $ 555,345

30% $ 510,834 $ 530,892 $ 561,676

40% $ 512,016 $ 537,855 $ 567,689

50% $ 513,090 $ 544,29 $ 573,385

60% $ 514,241 $ 551,640 $ 579,471

70% $ 517,449 $ 552,673 $ 585,912

80% $ 520,923 $ 565,114 $ 594,321

90% $ 523,007 $ 582,310 $ 607,304

100% $ 629,484 $ 589,685 $ 711,107

Supplier A Supplier B Supplier C

$700,000,000$600,000,000$500,000,000

Confidence Level ChosenSupplier B

Supplier A

Supplier C

19


$497,042 $482,331 $427,436

$0

$50,000

$100,000

$150,000

$200,000

$250,000

$300,000

$350,000

$400,000

$450,000

$500,000

$550,000

$600,000

$650,000

Supplier A Risk Adjusted Bid Supplier B Risk Adjusted Bid Supplier C Risk Adjusted Bid

Side-by-side risk exposure calculations provides leadership with comparative insights into supplier potential costs impacts

Prog

ram

Cos

t (in

thou

sand

s)

$39,100

Supplier Initial Cost

Risk Exposure

All costs reported at the 90% confidence interval.

Analysis will lead to a cumulative assessment of the total risk exposure and the potential impact to program

budget.

Component budget $500K

Potential Risk Impact $25K



20


0

1

2

3

4

5

6

7

8

6/2/10 6/6/10 6/10/10 6/14/10 6/18/10 6/22/10

Material Resource Planning requirement for this component is 04/25/2010

Risk adjusted delivery schedules are then compared to determine the potential risk impact to program’s operations

Supplier’s Initial ProposalSupplier A B C

Proposed Schedule 04/18/2010 04/04/2010 04/21/2010

Risk Adjusted Schedule 05/13/2010 07/18/2012 08/21/2012

0

1

2

3

4

5

6

7

8

9

5/3/10 5/7/10 5/11/10 5/15/10

0

0.5

1

.5

2

.5

3

.5

4

4.5

7/7/10 7/11/10 7/15/10 7/19/10 7/23/10 7/27/10 7/31/10 8/4/10 8/8/10

Illustrative IllustrativeIllustrative

Supplier A Supplier B Supplier C

21


The risk-adjusted cost and schedule results are then compared for awarding contracts on a risk averse path

Initial review of Supplier bids would indicate “C” as the supplier of choice

Based on Supplier B & C delivery metrics and potential risk, schedule impact could result in more than a 3 months past MRP requirements

Based on supplier C’s lack of technical capabilities and schedule risk to operations, risk impact could equate to ~$100K over component budget

Based on risk adjusted Cost & Schedule proposal analysis supplier “A” should receive program consensus for contract award based on least amount of risk exposure to the program

Supplier’s Risk Adjusted ProposalSupplier A B C

Risk Adjusted Cost $ 523,007 $ 582,310 $ 607,304

Risk Adjusted Schedule 05/13/2010 07/18/2012 08/21/2012

Supplier’s Initial Proposal

Supplier A B C

Proposed Cost $ 497,042 $ 482,331 $ 427,436

Proposed Schedule 04/18/2010 04/04/2010 04/21/2010

22


Given these potential benefits, a few key considerations are worth noting

In order to have a successful portfolio risk management process, it’s important that the constituent components of the program have sufficiently mature supply chain management and risk management processes.

Integration of acquisition risk analysis into a program’s budget and schedule is necessary to capture the magnitude of potential program risk impact by a single supplier

Identifying risks within a proposal enables forward looking program management that can be streamlined into existing risk database’s for future risk management planning and mitigation.

Qualitative risk analysis provides enhanced proposal evidentiary support and solid justification for awarding contracts

The success of a supply chain risk management program requires the consistent and active support of program leadership in order to be successful.

23


For more information on how acquisition risk analysis can be applied to your specific challenges, please:

Contact:

James TaylorHuntsville, [email protected]

James.taylor

Business

supplier delays

supplier cost overruns

incumbent supplier operations

supplier selection process

supplier design changes

late supplier deliveries

mitigation analysis

sample analysis