William.miller.pmc2010

Challenges in partnering on major research platforms and facilities

William L. Miller, Ph.D.AAAS Science and Technology Policy Fellow

National Science Foundation

Used with permission

PMC2010: Partnering challenges on large projects 2W. Miller

Scope

• High cost and complexity of major projects have drive partnering between U.S. agencies and with international entities.

How do partner processes and practices align? (Start with agencies)

What are the challenges and best practices for partnering?


Science platforms and facilities: a highly varied class…


Big science projects engage many stakeholders…

Large ScienceInfrastructure

ScienceEnterprise

TechnologyEnterprise

NationalPriorities

InteragencyPartnerships

InternationalCooperation

lots of interest in process, performance and outcomes…


Comparison of project practices at DOE, NASA and NSF

Organizations and portfolios studied:

DOE Office of Science “Science User Facilities”

NASA Science Missions Directorate (SMD)“Robotic Science Missions”

NSF Science & Engineering Directorates, OPP“Large Facilities” (MREFCs)

W. Miller, PRECONSTRUCTION PLANNING FOR LARGE SCIENCE INFRASTRUCTURE PROJECTS: A comparative analysis of practices and challenges at DOE, NASA and NSF, November 3, 2009

• Reviewed key agency guidelines, external reports & analyses…

• “Looked under the hood”: ~45 stakeholder interviews; site visits; direct observations…


Comparative Approach

• Framework for agency comparison along many dimensionsOrganizational structures

Development processes

Oversight and decision/approval

Funding and acquisition

etc….

• Identify partnering issues along these and other dimensions…


Comparative anatomy of agency science structures

FPD Federal Project DirectorOECM Office of Engineering and Construction MgmtOPA Office of Project AssessmentSRB Standing Review Board

LFO Large Facilities OfficeNSB National Science BoardS&E Science and engineering

Provides policyProvides independent review

$33.8B

$4.8B $4.5B

$17.8B

$2.0B

$6.5B

$5.3B

Budgets: FY 2009 plans (no ARRA), from FY2010 Requests

“Project assurance” bodies

DOESecretary

Program Offices

Laboratories

Projects

OPA

Programs

Centers

Projects

Science MissionsDirectorate

NSFDirector

Programs

Awardees

Projects

Directorates& Offices

NASAAdministrator

ChiefEngineer

NSB

SRBs PanelsDivisions

Divisions

UnderSec

Divisions

42 U.S.C.§1873b(“shall not operate

laboratories”)

Adm

inis

trat

ion

Prog

ram

sPr

ojec

ts

FPDs

Ofc of Science

ManagementSupport Office

OECM

Budget, Finance& Award Mgmt

LFO


Development process framework look at elements

Earlyacquisitions

Early operations

ProjectCloseout

TerminateFacility

Development

Conceptualization& Prioritization Construction DOperations

Begin majorInvestment

Conceptual

Preconstruction Planning*Preliminary

Finaldesign

Ongoing community input, horizon planning, reprioritization, and portfolio adjustment

*Also called definition, formulation, front-end planning…

• Plan Management/governance plans, WBS assemble project team• Design Goals, requirements iterative design bring to readiness• Invest R&D, necessary technologies bring to readiness• Estimate Effort, cost, schedule, reserves, risks refine to believability• Govern Progress oversight and decision-making

PMC2010: Partnering challenges on large projects 9W. MillerW.Miller - Big Projects & Sponsor Processes 9ProjSci Oct 22, 2009

ImplementationFormulation

ConceptStudies

Prelim Design & Tech Completion

Final Design & Fabrication*

Assembly, Integ & Test,

LaunchOperationsConcept &

Tech Devel

Pre-Phase A Phase A Phase B Phase C Phase D Phase E

KDP-A KDP-B KDP-C KDP-D KDP-ENASA-SMD Ref: NASA NPR 7120.5D

Development processes: alignment and terminology

MCR MDRSRR

PDR CDR SIR

ORR

FRRLRR PLAR

CERR

CDR Critical Design ReviewCERR Critical Events Readiness ReviewFRR Flight Readiness ReviewLRR Launch Readiness ReviewMCR Mission Concept ReviewMDR Mission Definition Review

ORR Operational Readiness ReviewPDR Preliminary Design ReviewPLAR Post-Launch Assessment ReviewSDR System Definition ReviewSIR System Integration ReviewSRR System Requirements Review

*elongated to visually align NASA, NSF & DOE equivalent events. Sequences on this chart do not represent typical or relative phase durations.

Key Decision Point (KDP) approvalsKDP-C Approve Implementation

CDR PDR FDR Ops

NSB Approved

Preliminary Design Final Design Construction OperationsHorizon planning and

Conceptual Design

Readiness

NSFRef: NSF 0738CDR Conceptual Design ReviewFDR Final Design ReviewPDR Preliminary Design ReviewOps Operations Review

ApprovalsPost-CDR Approve advance to ReadinessPost-PDR Approve submission to Nat. Science Board (NSB)Post-FDR Congress appropriates MREFC fundsPost-Ops Approve Operations start

IPR** CDR EIRPDR

FDRIPR/EIR**

ORR/RA

**CD-0 IPR and CD-3 EIR for >$750M projects

CDR Conceptual Design ReviewEIR External Independent Review (OECM)FDR Final Design ReviewIPR Independent Project Review (SC)ORR Operations Readiness ReviewPDR Preliminary Design ReviewRA Readiness Assessment

Critical Decision (CD) approvalsCD-0 Approve mission needCD-1 Approve Alternatives selection & cost rangeCD-2 Approve Performance baselineCD-3 Approve Construction startCD-4 Approve Operations start

DOE-SCRef: DOE O 413.3A

Execution

Pre-conceptualPlanning

Trans/CloseoutOperations

ConceptualDesign

CD-0 CD-1 CD-2 CD-4CD-3

Initiation Definition

PreliminaryDesign Final Design Construction

Decision

WLM Rev2.05052009

Review


Oversight: two modes of assessment

Conception PreconstructionPlanning Construction DOperations

Initiate Ready Finished

Review &Approve

Qualitativeassessment

(Reviews at major milestones)

Quantitative Tracking

-0.60

-0.40

-0.20

0.00

0.20

0.40

CV

SV

• EVM required by OMB for major projects

• Industry standards• Encourages baselining• Fuller assessment of

outcomes


Review-based oversight and decision/approval

Independent Review

Internal decisionsupport

Sign Off(e.g. Implement Project)

NASAPDR

by Standing Review Board

Program Mgmt Council

• Program Manager• Center Mgmt Council• Technical Authority• Project Manager

KDP-CDecision Authority (DA)

(AA for Cat 1, otherwise MDAA)Approve Implementation

(Enter Phase C – Final Design & Fab)

NSFPDR

by Review PanelNSF Director

MREFC* Panel• CFO, DDLFP• Directorate/Division• Program Officer

NSB Approvalfor inclusion in a future

budget in MREFC* construction account

DOEPDR

by OPA“Lehman Review”

(also: EIR by OECM)

Energy Systems Acquisition Advisory Board (ESAAB)

• OPA briefs AE, calls ESAAB• Federal Project Director

CD-2Acquisition Exec (AE)

(DepSec, US Sci, SC Dir/AD)Approve Performance Baseline

(Enter Final Design Stage)

* Major Research Equipment and Facilities Construction


Baseline establishment – with focus on cost

“Baseline”(what you were promised)

“Outcome”(what you got)

During construction, projects performance assessed relative tobaseline…

CostCost

DOE • “Bottom-up" Independent Cost Estimate (ICE) or Independent Cost Review as part of External Independent Review for CD-2.

NASA • Project’s bottom-up Life-Cycle Cost Estimate (LCCE).

• ICE prepared and owned by the Standing Review Board (SRB), (generally parametric, using same definitions as LCCE)

Project must reconcile ICE with LCCE at Preliminary Design Review.

NSF • Bottom-up cost estimate in proposal.

• NSF may acquire an independent top-down estimate for comparison.

Cost estimation toward the baseline:


Funding

MO&DAProgram Funds, project line-itemMission Operations and Data Analysis

ExecutionPre-conceptual

PlanningTrans/Closeout

OperationsConceptual

Design

CD-0 CD-1 CD-2 CD-4CD-3

Initiation DefinitionPreliminary

Design Final Design Construction

NSB ApprovedPreliminary

Design Final Design Construction OperationsHorizon planning and Conceptual Design

Readiness

R&RAProgram Funds (Research & Related Activity, R&RA) MREFCMajor Research Equipment and Facilities Construction

ConceptStudies

Prelim Design & Tech Completion

Final Design & Fabrication*

Assembly, Integ & Test,

LaunchOperationsConcept &

Tech Devel

Pre-Phase A Phase A Phase B Phase C Phase D Phase E

KDP-A KDP-B KDP-C KDP-D KDP-E

Formulation Implementation

Requires separate appropriation

Program Ops FundsLIC Prelim Engr & Design (PED)Program Funds

Two modes for intramural projects: Line Item Construction (LIC) orMajor Item of Equipment (MIE, no major construction)

Major Item of Equipment (MIE) LIC Construction

NASA-SMD

NSF

DOE-SC

• Separate funds support research (Research and Analysis, R&A)

• Same funds support research (Research & Related Activities, R&RA)

• Separate funds support research


Scope

How do partner processes and practices align? (Start with agencies)

What are the challenges and best practices for partnering? Try to “systematize”

Sources: • Interviews with ~45 stakeholders for study• “Lessons learned” and other reports


Some recent cases of major partnered projects

NASA DOE NSF International

Ulysses ▲ ESA

Cassini ▲ ESA, ASI

Tropical Rainfall Measuring Mission (TRMM) ▲ Japan/JAXA

Global Precipitation Measurement (GPM) ▲ Japan/JAXA

Fermi (operating) ▲ ▲

LHC (operating) ▲ ▲ CERN members

International Ocean Drilling Program (IODP) ▲ Japan/Mext

Atacama Large Millimeter Array ▲ Europe, Japan, Asia


Examples of interagency Joint Oversight Groups (JOG)

• DOE-NASA: FERMI, Joint Dark Energy Mission

• NSF-DOE: U.S. Large Hadron Collider (LHC), Deep Underground Science and Engineering Laboratory (DUSEL, in planning)


-Source: R. Staffin, 14Feb2006, FY06 presentation to HEPAP, www.er.doe.gov/hep/files/pdfs/HEPAPFeb142005Staffin.pdf

Partnering Levels of impedance (mis)matches

Strategicvaluation

Technicaldomains

Managementpractices

Capabilities& Practices

Better planning could involve assessment of COMPLEMENTARITY across capabilities, practices, lessons learned, etc…)


Challenges: Strategic Management Technical

• Valuation. How highly does each partner prioritize the project – and how much is it willing to spend based on that priority? What does it expect as an outcome (science, jobs, prestige) and when? Where does it fit in the partner's national plan?

• Goals and roles. Are the overall goals and specific objectives shared? Does each partner want a leadership or participatory role? How committed are the commitments?

• Approval and allocation. Which hoops does each partner have to jump through to obtain funding and approval? What are the pressures? What is the funding source and cycle? How long does prioritization and approval typically take?



• Role of project management. Is project management valued? Are managers well chosen, given sufficient authority? Who do they report to?

• Project definition and baselining. How are requirements, risks and baseline elements defined, documented and managed? Change control? How much emphasis placed on system I&T and operations planning?

• Budgeting. Can be large differences in costing – e.g. labor cost inclusion in project budgets. Can funds be used early, carried-over? Contribution approach (in kind, etc.)? How are contingency and reserve handled?

• Oversight. Partners may not perform reviews with the same rigor or frequency. Identified problems may be addressed with less urgency. What systems do partners use for tracking and quantifying performance?

• Cultural differences. Sense of urgency to obtain desired outcome? Consequences of (and solutions adopted for) cost overruns, late development, poor performance, outright failure?



• Complementarity. Synergies among partners in technical know-how relative to requirements of the project? Is at least one partner strong in each area? What are the gaps?

• Design process. What approaches are employed by each partner? How are software and IT systems developed?

• Systems engineering. To what degree is systems engineering valued by each partner? Is each partner familiar with establishing interfaces? With I&T planning?

• HR. How do partners obtain their skill contributions – via contractors, in-house staff, exchanges of personnel from member academic institutions…? Workforce stablity?


Synchronizing “On-ramps”: when does a project start?

DOE • Projects identified in a SC 20-year prioritized facility plan1

• Establish “mission need”, feasibility at CD-0 Definition phase

NASA • Strategic and Science Plans2 based on Decadal Surveys, roadmaps.

Two flavors:• Strategic missions are internally initiated (KDP-A) and managed• “PI-led” missions are competed in Phase A selected Phase B

NSF • Peer-reviewed unsolicited proposals, workshops, studies, etc.• Evolved concepts may be brought to development

“Mission-driven” – projects determined via strategically-defined goals and priorities

1. Facilities for the Future of Science, A twenty year outlook, DOE/SC-0078, Dec 2003; and Four Years Later: an Interim Report on Facilities for the Future…, Aug 2007. 2. NASA Strategic Plan, 2006; and NASA Science Plan 2007–2016

“Community-driven” – projects “bubble up” from the scientific disciplines

ConceptualizationConceptual Preliminary Final

Preconstruction Planning

?


Reported best practices – responses to challenges• Early engagement. Agree on clear goals, timeframe, and effective

membership rules and governance structure. Having a dominant partner may work best…

• Communication. Identify the appropriate interfaces at all levels. Maintain open, frequent and honest communication.

• Complementarity. Learn partner enterprises and realities. Identify technical domains required and respective partner(s) capabilities.

• Strong project management. Devise clear process for project management and oversight. Aim for integrated a single project management team, in place before funding begins and with effective budget authority (not just a coordinator).

• Budgeting. Aim to adopt standard costing and budgeting techniques.


AcronymsCD Critical Decision (DOE)

CDR Conceptual Design Review (DOE, NSF)Critical Design Review (NASA)

DDLFP Deputy Director for Large Facility Projects, head of LFO (NSF)

EIR External Independent Review (DOE OECM)

IPAO Independent Program Assessment & Oversight Office (NASA)

KDP Key Decision Point (NASA)

LFO Large Facilities Office (NSF)

MO&DA Mission Operations and Data Analysis funding account (NASA)

MREFC Major Research Equipment and Facilities Construction funding account (NSF)

NAR Non-Advocate Review (PDR, NASA)

OECM Office of Engineering and Construction Management (DOE)

OPA Office of Program Assessment (DOE/SC)

PED Preliminary Engineering and Design funding account (DOE)

PDR Preliminary Design Review

PNAR Preliminary Non-Advocate Review (MDR, NASA)

R&RA Research and Related Activities funding account (NSF)

R&A Research and Analysis funding account (NASA)

SC Office of Science (DOE)

SRB Standing Review Board (NASA)

SMD Science Missions Directorate (NASA)

William.miller.pmc2010

Technology

large projectsw

b science missionsdirectorates

aaas science

science platforms

b nasa

big science projects

partnering issues

project team design