Joint Space Cost Council - NASA Joint Space Cost Council Background: Joint Space Cost Council (JSCC) !!!! Established by the Undersecretary of Defense for Acquisition, Technology,

JSCC Joint Space Cost Council

Joint Space Cost Council

Better EVMS Implementation Study: Integrating Identified Phase I Cost Impacts with

Phase II Government Value

August 2016

DRAFT Summary of Industry Survey Responses


Survey Background and

Phase I Results

2


Background: Joint Space Cost Council (JSCC)

Ø

Ø

Ø

Ø

Established by the Undersecretaryof Defense for Acquisition,Technology, and Logistics Supportto improve collaboration withoversight and service/agencylevelsFocus on cost credibility andrealism in estimates, budgets,schedules, data, proposals andprogram executionBroad participation across industryand governmentInitiatives consistent withgovernment and industry focus onAffordability

JSCC is an effective forum for government and industry collaboration to improve EVM implementation

3


Industry and Government Study Phases Include Government Value

Government-Industry collaboration through all phases of the survey and analysis

JSCC Industry Day (joint Government/ Industry participation)

Identification of 78 Industry Cost Areas

Industry Survey to assess cost areas as high, medium, low, no impact

Phase I Recommendation Report, focusing on high and medium cost impact areas

Joint Government/ Industry Implementation Plan

JSCC Government

Day (joint Government/

Industry participation)

Identification of EVM Products and Management Activities used by the Government

Government Survey assessed areas based on Value

Phase II Recommendation Report, focusing on PM value assessment areas

Phase I Recommendation Report available at: http://www.acq.osd.mil/evm/docs/JSCC%20Better%20EVM%20Implementation%20%20Recommendations%2015%20April%202015.pdf 4


Setting a Realistic Study Scope: Phase I

Ø The Phase I scope of the study was to identify the Delta Implementation CostImpact between EVM implemented on Government Programs and EVMimplemented on Commercial, Internal or Fixed Price Programs

Industry identified cost areas perceived as being Government driven and rated the impact high, medium, low or no impact

5


Cost Impact for Each Cost

Area

High

Medium

Low

None

Phase I Approach

Ø A Cost Impact Survey was completed by 5 major contractors (Ball Aerospace,Boeing, Lockheed Martin, Northrop Grumman, and Raytheon) for 46 differentNASA, NRO, and SMC space programs. It resulted in data on 78 specific CostAreas and more than 1,000 comments from Industry regarding their assessments.

6

Phase I Data Results

Phase I Survey

Potential Areas for Cost Impact Identified by

Industry


The Structure of the Phase I JSCC Survey

Similar Cost Areas were grouped into 15 Cost Drivers to help facilitate the survey

Survey included an assessment of 78

different Cost Areas

Survey based on 78 industry-identified cost areas – Respondents assessed the Cost Impacts for each area as High, Medium, Low or No Impact

7


Size of Programs that Participated in Phase I

8 8

0

5

10

15

20

< 50 $M < 100 $M < 500 $M < 1 $B >= 1 $B NotIdentified

Programs in JSCC Survey by $ Value

The survey responses included 46 programs - 17 greater than or equal to $1B as well as 7 in the $20M-$100M range


Cost Impacts of EVM and Stakeholders Identified in Phase I

Gov Program Mgmt40%

Contracting Officer8%

KTR EVM Process Owner12%

KTR Program Mgmt10%

Cost Estimators2%

DCMA19%

PARCA1%

NRO ECE4%

DCAA0%

Not Provided4%

Stakeholders for High and Medium Impacts

High13%

Medium14%

Low28%

No Impact45%

Survey Impacts944 High and

Medium Impacts

~73% of all survey data points (2,644 of the 3,588 answers) had Low to No cost

premium identified to comply with Government EVM requirements.

Of the ~27% identified as High and Medium Impacts Government Program Management was identified as Primary Stakeholder, followed by DCMA. Contractor EVM Process Owner and

Contractor Program Management also identified as “Significant” Stakeholders. 9


Phase I Synopsis from Data and Comments

Ø The cost impart of Implementing EVM on Government Programs is not significant.Industry is already performing the vast majority of EVM functions without anyGovernment Requirement.Many perceived cost impacts are actually a function of the Acquisition Environmentwith the Government and not related to the implementation of EVMThe highest value EVM cost impacts identified in Phase I are not consistent amongall Government Programs and there is no single Cost Area identified as High Impacton the majority of all surveyed Programs.The Government Program Manager is considered to be the biggest Stakeholder (asperceived by Industry) in driving EVM related cost impact (i.e.. Level of Reporting,Lack of Meaningful VAR Thresholds, Volume of IMS Tasks/Level of Detail,Additional Requirements beyond CDRLs, etc.)

Ø

Ø

Ø

Ø

EVM Experts from Government and Industry published Final Recommendations to reduce the Cost Impact of EVM on Government Programs and addressed the following three themes: 1)  The Control Account level (size and number) significantly impacts the cost of

EVM2)  Program volatility and lack of clarity in program scope as well as uncertainty

in funding may impact the cost of EVMS, just as any other ProgramManagement Discipline

3)  Volume of IBRs and Compliance/Surveillance reviews and inconsistentinterpretation of the 32 EIA 748 Guidelines impacts the cost of EVM

10


Phase II

Government Program Manager Assessment of

EVM Products and Management Activities

11


Focus of Phase II

Gov Program Mgmt40%

Contracting Officer8%

KTR EVM Process Owner12%

KTR Program Mgmt10%

Cost Estimators2%

DCMA19%

PARCA1%

NRO ECE4%

DCAA0%

Not Provided4%

Ø

Ø

Ø

Ø

Government PM is more

than twice as large as the 2nd largest

Stakeholder

In order to understand how the CostImpacts identified in Phase I areValue-Added, the JSCC performed asecond phase of the study to identifyvalue of EVM Products and EVM-Related Management ActivitiesJSCC focused on GovernmentProgram Managers

Government PM was thelargest Stakeholder identified inPhase I (in attributing the mostMedium and High CostImpacts)Government PM is the primarybeneficiary of those Productsand Activities

12


EVMProducts

and Processes

109

8

7

6

5

4

32

1

HIGHEST

LOWEST

EVMProducts

and Processes

109

8

7

6

5

4

32

1

HIGHEST

LOWEST

Setting a Realistic Study Scope: Phase II

Ø The scope of Phase II of the study was to identify the Government Valueof specific EVM Products and Management ActivitiesPhase II also included some inquiries regarding the current Assessmentof Quality of EVM Data and Timeliness of EVM Data

Ø

10 9

8

7

6

5

4

3 2

1

HIGHEST VALUE

LOWEST VALUE

Quality of EVM

Data

Government Value

Please provide your assessment of the value of the following EVM Products

and Management Activities:

1) EVM data by WBS2) EVM data by OBS3) Staffing Reports4) VARs5) IMS 6) Integrated Master Plan7) Contract Funds Status

Report8) Schedule Risk Analysis9) EVM Metrics10) IBR11) OTB/OTS12) Surveillance Reviews

Assessment

What is your assessment of the overall Data Quality of your Contractor's EVM-‐

related data?

Timeliness of EVM

Data

EVM Products

And Management

Activities What is your assessment of the Timeliness of Data

provided by the Contractor in order for you to use it to

assist in program management decisions?

13


109

8

7

6

5

4

32

1

HIGHESTVALUE

LOWESTVALUE

EVMProductsAnd

ManagementActivities

Phase II Approach

Ø An EVM Products and Management Activities Value Survey was completed byGovernment Program Managers (or their equivalent) for 32 different space programs. Thesurvey provided data on 12 specific EVM Products and Management Activities andincluded more than 400 comments from Program Managers regarding their assessments.

Products & Management Activities

EVM data by WBS

EVM data by OBS

Staffing (Manpower) Reports

Variance Analysis Reports

Integrated Master Schedule

Integrated Master Plan

CFSR

Schedule Risk Analysis

EVM Central Repository

EVM Metrics

Integrated BaselineReviewSurveillance Review

OTB & OTS

Phase II Survey

Phase II Data Results

14


Size of Programs that Participated in Phase II

Survey responses included 32 different programs with a wide range of Values – 19 greater than or

equal to 1 $B as well as 1 in the 50-100 $M range 15


Value Ratings – Converting to Net Promoter Score (NPS)

Net Promoter Score (NPS) was adopted as one way to interpret the results of Phase IIThe NPS metric was introduced in 2003 in the Harvard Business Review and adopted bynumerous companies* to differentiate between individuals who would actively “promote”a product or service and those less likely to exhibit value creating behavior.

1 2 3 4 5 6 7 8 9 10 0

Detractors Passives Promoters

Low Neutral High

ØØ

% Promoters - % Detractors = Net Promoter Score

Net Promoter Score takes into account Positive Impact of Promoters and the Negative Impact of Detractors to yield a Summary Score

*Charles Schwab, Apple, GE, Intuit, American Express, etc.16


Interpreting Phase II Data – Net Promoter Score (NPS)

% Promoters - % Detractors = NPS 16% – 62% = -46% NPS

% Promoters - % Detractors = NPS 61% – 5% = +56% NPS

Net Promoter Scores identify Strong Positive Responses as well as

Strong Negative Responses to Survey Questions

17


JSCC Phase II Overview of Data

0

1

2

3

4

5

6

>= +50%>= +25% &< +50%

> -‐25% & <+25%

<= -‐25% & >-‐50%

<= -‐50%:

NPS Breakout

44%

34%

22%

Organization

NASA

NRO

SMC

3%

28%

10%59%

Program Size

>=50 $M & <100 $M

>=100 $M & <500 $M

>=500 $M & <1 $B

>=1 $B

59%28%

13%

System Type

Space System

Ground System

Other

34%

22%

44%

Respondent

PM

Deputy PM

Other

29%

29%13%

29%

% Subcontract

<25%

>=25% & <50%

>=50% & <75%

>=75%

53%37%

10%

Responsibility

Multiple Contracts

Single Contract

Segment of Contract

-‐100% -‐75% -‐50% -‐25% 0% 25% 50% 75% 100%

EVM data by Work Breakdown Structure (WBS)

EVM data by Organizational Breakdown Structure (OBS)


Variance Analysis Reports (VARs)

Integrated Master Schedule (IMS)

Integrated Master Plan (IMP)

Contract Funds Status Report (CFSR)

Schedule Risk Analysis (SRA)

EVM Metrics

Integrated Baseline Review (IBR)

Surveillance Reviews

OTB or OTS

Products/Processes Net Promoter Scores

COMMENTS: 406

Most EVM Products and Management

Activities show NPS Value

leaning towards High

Value

18


Average Raw and Net Promoter Scores of EVM Products and Management Activities

EVM Products/Management Activities AVG Raw Score Promoters Detractors Passives NPSIntegrated Master Schedule (IMS) 8.7 75% 13% 13% 63%Contract Funds Status Report (CFSR) 8.7 61% 6% 33% 56%Integrated Baseline Review (IBR) 8.4 46% 4% 50% 42%EVM Metrics 8.4 53% 9% 38% 44%Variance Analysis Reports (VARs) 8.1 41% 19% 41% 22%Staffing (Manpower) Reports 8.0 48% 19% 32% 29%EVM data by Work Breakdown Structure (WBS) 7.9 44% 22% 34% 22%OTB or OTS 7.8 44% 25% 31% 19%Schedule Risk Analysis (SRA) 7.0 32% 36% 32% -‐4%Surveillance Reviews 6.4 22% 43% 35% -‐22%Integrated Master Plan (IMP) 5.9 24% 52% 24% -‐29%EVM data by Organizational Breakdown Structure (OBS) 5.5 15% 62% 23% -‐46%

Sorted by Average RAW Score

Even though there were some negative Net Promoter Scores, ALL average Raw Values were well above 6 (out of 10) except the IMP and EVM data by OBS

Structure. EVERY EVM Product or Management Activity received Promoters (values of 9 or 10) from the total population of Program Managers interviewed.

19


Phase II EVM Expert Working Group: Step 1

Ø A group of EVM Subject Matter Experts (SMEs) representing Government and Industrymet to review all data and comments received from the Phase II surveysAs a first step in generating Phase II analysis, SMEs performed the following functions:Ø

1) Consolidated comments from all surveys and developed a comprehensive list ofissues identified by Government Program Managers

2) Reviewed the Program Manager’s assessment of the current state of Data Qualityand Data Timeliness

3) Consolidated comments regarding Timeliness and Quality of EVM Data andprovided suggestions on improvements

Phase IIData Results

SUBJECT MATTER WORKING GROUP

GOVERNMENT INDUSTRY

DCMA BALL AEROSPACE

NGA LOCKHEED MARTIN

NASA RAYTHEON

NRO NORTHRUPGRUMMAN

PARCA

SMC (USAF)

CONSOLIDATION OF VALUE COMMENTS

SUGGESTIONS FOR INCREASED VALUE

20


Integrating Phase I and Phase II

Understanding the Value of the Cost Impact Identified in Implementing

EVM on Government Contracts

21


Phase II EVM Expert Working Group: Step 2

Ø In addition to consolidating value comments and generating suggestions,the SMEs performed a second step, of generating Cross Index of DirectRelationships between Phase I (Cost Impacts) and Phase II (Value of EVMProducts and Management Activities)

Phase IIData Results

SUBJECT MATTER WORKING GROUP

GOVERNMENT INDUSTRY

DCMA BALL AEROSPACE

NGA LOCKHEED MARTIN

NASA RAYTHEON

NRO NORTHRUPGRUMMAN

PARCA

SMC (USAF)

CONSOLIDATION OF VALUE COMMENTS

SUGGESTIONS FOR INCREASED VALUE

CROSS INDEX BETWEEN PHASE I

COST IMPACTS AND PHASE II VALUE ASSESSMENTS

22


The Phase I and Phase II Matrix

Ø The Matrix includes 78 Cost Areas (from Phase I) and 12Products/Management Activities (from Phase II)There are 936 specific intersections that contain an assessment of whether or not the Product or Management Activity has a DirectImpact on the Cost AreaExperts from Government and Industry identified DirectRelationships between Phase I Cost Impacts and Phase IIProducts and Management Activities using the followingstatement:

“the Customer Requirement for Product/Management Activity A(from Phase II) can directly influence Cost Area X (from Phase I)”

The completed Matrix provides a way to identify the Cost Impactsthat are generated by specific Government EVM Products andManagement Practices

Ø

Ø

Ø

Ø The Matrix shows that ALL Products and ManagementPractices directly influence Multiple Cost ImpactsThe Matrix shows that Cost Impacts are directly influencedby multiple Products and Management Practices

Ø The completed Phase I and Phase II

Matrix

23


Products & Management Activities

EVM data by WBS

EVM data by OBS


Variance Analysis Reports

Integrated Master Schedule

Integrated Master Plan

CFSR

Schedule Risk Analysis

EVM Central Repository

EVM Metrics

Integrated BaselineReviewSurveillance Review

OTB & OTS

Input and Output: Correlating Phase I Cost Impacts with Phase II Government Value Assessments

Phase I Data

Step 1

Step 1: Phase I identifies Cost Impact for 78 specific Cost Areas

and scores them as High, Medium, or Low.

Recommendations are made to reduce those identified Cost Impacts.

Phase I Recommendations

Phase II Data

Step 2

Step 2: Phase II identifies

Government Value for Specific EVM Products and Management Activities and

assesses them as 1 (Low) to 10 (High)

Phase II Recommendations

Step 3

Step 3: Government and Industry EVM Experts develop a matrix to correlate Cost Area Impacts to specific EVM

Products and Processes

Step 4

Step 4: Products and Management Activities are analyzed to understand maximum benefitand potential for future EVM trade studies

24


18895

50 53106

38 81 96164

177

99

76 80

129

61

93108

142

388

234

157 158

249

105

161211

316

0

100

200

300

400

500

600

700

800

900

EVM data by W

BS

EVM data by OBS

Staffin

g (M

anpo

wer) R

eports

Varia

nce An

alysis Re

ports

Integrated

Master S

ched

ule

Integrated

Master P

lan

CFSR

Sche

dule Risk

Analysis

EVM M

etrics

Num

ber o

f Cost Impa

cts

Shared Cost Impacts for EVM ProductsHigh Medium Low

19595

50 53106

38 81 96164

180

99

76 80

129

61

93108

142

402

234

157 158

249

105

161211

316

0

100

200

300

400

500

600

700

800

900

EVM data by W

BS

EVM data by OBS

Staffin

g (M

anpo

wer) R

eports

Varia

nce An

alysis Re

ports

Integrated

Master S

ched

ule

Integrated

Master P

lan

CFSR

Sche

dule Risk

Analysis

EVM M

etrics

Num

ber o

f Cost Impa

cts

Cost Impacts for EVM ProductsHigh Medium Low

Cost Impacts are Spread Across Multiple Reporting Requirements

Ø Although the study indicates the Government PM value for the IMP and EVM data by OBSas the lowest for all products, the majority of cost impacts required to develop these reportsare shared with other deliverables.

Virtually EVERY Phase I Cost Impact attributed to a Product is shared by other

Products or Management Activities

Shared Cost Impacts are a subset of all Cost Impacts

If any IPMR format were removed from contract, the results of the JSCC study indicate the cost savings are intangible

25


What Program Managers Telling Us about the Most Significant Cost Impacts of Implementing EVM on Government Programs

(as identified in Phase I of the JSCC Study)

Theme 1: The Control Account level (size and number) significantly

impacts the cost of EVM

Cost Impact IS VALIDATED* by Government Value of


Theme 2: Program volatility and lack of clarity in program scope as well as uncertainty in funding may impact the

cost of EVMS, just as any other Program Management Discipline

Cost Impact is NOT ATTRIBUTED* to EVM

Products and Management Activities

Theme 3: Volume of IBRs and Compliance/Surveillance reviews and inconsistent interpretation of the 32 EIA 748 Guidelines impacts

the cost of EVM

Cost Impact IS VALIDATED* by Government Value of the Integrated Baseline Review

(IBR)

These Cost Impacts ARE VALIDATED* by

Government Value of the Surveillance Review (SR)

*Assessment is based on EVM Expert WorkingGroup correlation of Cost Impact (Phase I) and

Government Value (Phase II)

26


JSCC Study Theme 1

Program Managers are aware that they have an Impact on the Size and Number of Control Accounts

Theme 1: The Control Account level (size and number) significantly

impacts the cost of EVM

5 Specific Phase I Recommendations were made to help Reduce these Cost Impacts

Cost Impact IS VALIDATED* by Government Value of


Phase II Recommendations will Recommend Ways to

Provide More Value

Government Program Managers score EVM

Data by WBS as High-‐to-‐ Medium Value

Government Program Managers score Metrics as High-‐to-‐Medium Value

Government Program Managers score

Associated EVM Products as High-‐to-‐Medium Value

Government Program Managers recognize the Need for Multiple CLINs


Government Value (Phase II) 27


JSCC Study Theme 1 Government Value vs Cost Impact

LOW GOVERNMENT VALUE HIGH GOVERNMENT VALUE

LOW COST IM

PACT

HIGH

COST IM

PACT

Phase I Recommendations should help Reduce the Existing Cost Impacts

Phase II Recommendations should help Increase the

Existing Value

All Associated EVM Products are Currently are Scored at High-to-Medium Value with a Medium-to-Low Cost Impact

28


JSCC Study Theme 2

Theme 2: Program volatility and lack of clarity in program scope as well as uncertainty in funding may impact the cost of EVMS, just as any other Program Management Discipline




Cost Impact is NOT ATTRIBUTED* to EVM

Products and Management Activities

These Cost Impacts are defined by Program

Volatility, Scope Changes, and Funding Issues

These Cost Impacts are Directly Related to the Cost of doing Business with the Federal Government

29


JSCC Study Theme 3

Without a Valid Process, there can be no Valid Data



Theme 3: Volume of IBRs and Compliance/ Surveillance reviews and inconsistent

interpretation of the 32 EIA 748 Guidelines impacts the cost of

EVM



Government Value of the Integrated Baseline Review

(IBR) Process

Phase II Recommendations will Recommend Ways to

Provide More Value

Government Program Managers score IBR as High-‐to-‐Medium Value

Government Program Managers recognize the Need for a Good

Performance Measurement Baseline

Government Program Managers recognize the Need to Understand Risk

in the Baseline


Government Value of the Surveillance / Compliance Review (CR/SR) Process

Government Program Managers score SR as Medium-‐to-‐High Value

Government Program Managers see High Value in EVM Data and Metrics

Government Program Managers identify need for Better EVM-‐Related

Data Quality

Government Program Managers indicate SRs should be performed every

Two Years

30


JSCC Study Theme 3 Government Value vs Cost Impact

Phase I Recommendations should help Reduce the Existing Cost Impacts

Phase II Recommendations should help Increase the

Existing Value

LOW GOVERNMENT VALUE HIGH GOVERNMENT VALUE

LOW COST IM

PACT

HIGH

COST IM

PACT

IBR is Currently are Scored at High-to-Medium Value and SR is Currently are Scored at Medium-to-High Value. Both are scored

as a Medium-to-Low Cost Impact

31


In Conclusion…

Ø  The Cost Impact of Implementing Earned Value Management on a Government Program does not appear to be significant Ø  Most EVM related functions are ALREADY being performed by Industry

There are Acquisition Environmental Issues (Funding Volatility, Scope Changes, etc.) that are often mistakenly considered to be Cost Impacts to EVM

Ø 

Ø  The Government PM considers EVM related products and processes to be value-added in their management toolkit The Cost/Value for EVM Products and Processes appears to be primarily High Value at Low Cost

Ø 

Ø  There is clearly room to improve Value on some Products and Processes – management to better understand what EVM-related data is actually providing and understanding how to use it better Phase II Recommendations will address the Increasing the Value of EVM Products and Processes Most Phase I Cost Impacts are spread across Multiple Products and Processes

Ø 

Ø Ø  Ineffective or poorly implemented management and contracting practices can generate

Cost Impacts to the Cost of Implementing EVM (Driving WBS to extremely Low Levels, Using Multiple CLINs and requiring Multiple Reports, Customizing Reporting Requirements, etc.)

The Recommendations from Phase I and Phase II should provide ways to generate even better efficiencies (more value for less cost) for implementing EVM on Government Programs

Ø 

32


Final Thought

The 1994 Coopers & Lybrand/TASC study estimated the cost of Government mandated C/SCSC (the predecessor to Earned Value Management) to be 0.9% of Value Added Costs of Government contracts.* Since that time, Earned Value Management (EVM) has become embraced by Industry as a best practice on all types of contracts including both Government and commercial efforts. As a result, since a contractor must use some type of management approach, the Government should only be concerned with any delta costs for implementing EVM on Government contracts and not all full costs of EVM (since Industry will use this approach to manage regardless of the type of contract).

Using this premise, all of the data in the JSCC study indicates that as a rule, the delta implementation cost of implementing EVM is significantly less than 0.9% identified in 1994. There may be some unique situations where this value may be breached, however, this is typically driven by Government Program Management in order to meet a specific programmatic need.

*According to Coopers and Lybrand/TASC, Value Added Costs = Total Costs - Material/Subcontract Purchases

33


Speaker Contact Information

Ivan Bembers Chief of the National Reconnaissance Office Earned Value Management Center of Excellence

Michelle Jones Cost and Acquisition Assessment Group Support Booz Allen Hamilton Email: [email protected]

Ed Knox Earned Value Management Center of Excellence Support Tecolote Research, Inc. Email: [email protected]

Jeff Traczyk Earned Value Management Center of Excellence Support

Stefanie Terrell NASA

34

Joint Space Cost Council - NASA Joint Space Cost Council Background: Joint Space Cost Council (JSCC) !!!! Established by the Undersecretary of Defense for Acquisition, Technology,

Documents