2-5 November 2010 University of Southern California Los Angeles CA Systems Engineering Desk Reference An Aid for Cost Estimators Sherry Stukes Jet Propulsion.

2-5 November 2010

University of Southern CaliforniaLos Angeles CA

Systems Engineering Desk Reference

An Aid for Cost Estimators

Sherry StukesJet Propulsion Laboratory/

California Institute of Technology

Integrated Ground Data Systems

Ground Software Systems Engineering

4800 Oak Grove Drive MS 301-225

Pasadena CA818.393.7517 (o)805.402.8664 (c)

[email protected]

25th International Forum on COCOMO and

Systems Software Cost Modeling

Henry ApgarPresident

MCR Technologies LLC390 N Sepulveda Blvd

Suite 1050El Segundo CA

424.218.1616 (o)805.402.4232 (c)[email protected]

Copyright 2010. All rights reserved.

BackgroundSSCAG* Systems Subgroup Desk

Reference productRepresents industry “Best Practices”Useful to Cost EstimatorsOrganized into six sections

ContributorsNASA (JPL, LaRC, MSFC)MCR TechnologiesUSCDesign for Value

*SSCAG (Space Systems Cost Analysis Group) is an International working group comprised of member organizations that develop estimating products for the space industry. SSCAG currently has four active subgroups: Hardware, Software, Risk, and Systems, supported by members from industry, Government, and the academic community.

SAICUSAF (SMC, AFCAA)RaytheonModel Vendors (Galorath,

PRICE Systems)

2

Desk Reference Overview System Engineering Desk Reference

ContentDocument OverviewHow to Predict and Evaluate Systems Engineering and SoS

CostsDefinitionsRules of Thumb“Top 10”ToolsLessons Learned and Estimating Examples

SourcesRedStar LibraryConstellation ProgramModel vendor researchUniversity researchContractor organizations 3

Defining our TermsSystem of Systems (SoS) is a

collection of task-oriented or dedicated systems that pool their resources and capabilities together to obtain a new, more complex, 'meta-system' which offers more functionality and performance than simply the sum of the constituent systems.”

4

Assemblies

Subsystems

Systems

SOS

System Engineering can be considered to include the pure engineering efforts to ensure that a number of subsidiary elements function together properly but also project/program management, integration and test, missions assurance and ground support elements sometimes called “SEPM” or Systems Engineering and Project/Program. The definition can be adapted to lower levels, including subsystems and assemblies.

SoS HierarchyThe hierarchal aspect of

SoS is reflected in the fact that depending on how one defines system, almost any integration activity can be tagged as SoS

An example of SoS hierarchy is the crew launch system for the NASA Constellation Program. System of Systems” consists of

the Ares I launch vehicle system and the Orion crew capsule system

These systems themselves are comprised of multiple systems 5* Reference – materials submitted by

Andy Prince, NASA Marshall Space Flight Center.

Rules of Thumb Model Development

More parameters increase the explanatory power of the model, but too many parameters make the model too complex to use and difficult to calibrate.

Break the problem and analysis into phases over time; the right amount of granularity is important.

Let available data drive the application boundaries of the model.

Design the rating scale according to the phenomenon being modeled.

Some system characteristics are more likely to be cost penalties than cost savings.

Model Calibration All calibrations are local. Calibrations fix chronic errors in over- or underestimation. Be skeptical of data that you did not collect. For every parameter in the model, 5 data points are

required for the calibration. Don’t do more analysis than the data is worth. You need less data than you think, you have more data

than you think. Model Usage

A model is not reality. All models are wrong, but some of them are useful. Begin with the end in mind.

Requirements are king. Not all requirements are created equal. Reuse is not free. Operational Scenarios may come first, but

requirements will ultimately describe the system.

Don't double dip. Find your sea level. Nominal is the norm. If you're estimating a large project, personnel

capability is Nominal. Most of your off-Nominal cost drivers should

match your last project. If you're going to sin, sin consistently. Use a combination of models to estimate total

system cost. Avoid overlap between models. Estimate using multiple methods (analogy,

parametric, etc.). Estimation

Estimate early and often. Experts all disagree forever. Bound the options

they are given to evaluate.People are generally optimistic.

6

“Top 10” ToolsVersion SE Defined LCC

PhaseLevels WBS

DefinedSE Est. Method

Price ES System Engineering Process A-F

Subsystem, System

User Defined Imbedded Algorithm

Price True Planner

System Engineering A-D SoS EIA/ANSI 632 WBS

Academic - COSYSMO algorithm

SEER -H System Engineering and Integration A-F

Subsystem, System

User Defined Imbedded Algorithm

NAFCOM System Engineering & Integration A-E System Yes -

ConfigurableCER by mission type

SSCM07 System Engineering (w/PM) C,D System Yes – Defined CER

USCM Systems Engineering (w/in Program Level Cost)

A-D System Yes – Configurable

CER by Spacecraft type

NICM (both models)

Systems Engineering (estimated but not def.)

B,C,D thru L+30 System Yes - Defined CER in two

forms

COCOMOSoftware Engineering in the System Context (Waterfall WBS)

Waterfall and Mbase

SystemSoftware

Waterfall WBS and Mbase WBS

COCOMO algorithm

COSYSMOfor SE

Systems Engineering Effort A-D

System, Systems Eng

EIA/ANSI 632 WBS

academic COSYSMO algorithm

COSYSMOfor SoSE

Systems of Systems Engineering Effort C-E SoS DoD SEGuide

for SoSacademic COSYSMO algorithm

7

Cost Factors - NASA SOS Cost ModelingSystem-of-Systems Cost =

0.07411*(DDTE$)0.9993*(1.09585)M/LV

where DDTE$ = Total DDTE* Cost, in 2006$, M/LV = Manned or Launch Vehicle (Yes = 1, No

= 0)

Good Quality MetricsR2 = 93.1%SPE = 33.2%

System-of-Systems Actual Vs. Estimated

$0

$500

$1,000

$1,500

$2,000

$2,500

$3,000

$3,500

$0 $500 $1,000 $1,500 $2,000 $2,500 $3,000 $3,500

Actual (2006 $M)

Estim

ated

(200

6 $M

)

8*DDTE – Design Development Test and

Evaluation

9

Estimate Example – NASA

Factor-based approach - derived from real data (c. Smart, SAIC)

WBS Number

WBS Element

WBS Title WBS Description / Notes

1

SO

S

System of Systems Level Systems Engineering, Integration and Test and Top Level Program Management of Architecture

2 Space Segment Assets in orbit or interplanetary

2.1 Crew Expolration Vehicle (CEV)

2.1.3 Command Module (CM)2.1.3.2 CM PMP CM Prime Mission Product

(HW + SW)Orion Integration, Assy., & C/OOrion SE&IOrion PMOrion STOOrion GSE

2.1.4 Service Module (SM)2.1.4.1 SM SE IT PM Systems Engineering,

Integration and Test and Top Level Program Management of SM

2.1.4.2 SM PMP SM Prime Mission Product (HW + SW)

2.2 International Space Station2.2.1 ISS Communications2.2.2 TDRS Space Segment additional satellite(s) to

support CxP ISS mission3 Launch Segment

3.1 CLV3.1.1 First Stage SRM3.1.2 Upper Stage Improvements for

communicationg with ground and TDRSS.

3.1.2.2 Upper Stage Avioncis Software

3.1.2.1 Upper Stage Avionics Hardware

Comms to/from AF ROCC

Upper Stage Integration, Assy., and C/OUpper Stage SE&IUpper Stage PMUpper Stage STOUpper Stage GSE

4 Mission/Ground Ops4.1 Ground Stations Provides -S-band and UHF

air-to- ground4.1.1 Launch Head MILA/PDL = Merritt Island &

Ponce de Leon; Merritt has 14 antennae; Ponce de Leon has 3 antennae; both are dedicated to shuttle/CxP missions.

4.1.2 Wallops Island, VA 5m and 8m S-Band tracking; not dedicated site

4.1.3 New Boston, NH4.1.4 Argentia, Newfoundland

4.2 Deep Space Network (DSN) Stations

Canberra, Goldstone, Madrid

4.3 Range Safety Backuo provided by DoD ground-based radars

SP

AC

EM

ISS

ION

/GR

OU

ND

OP

SLA

UN

CH

Systems of Systems Cost

Total DDT&E Cost % of DDTE

Apollo $1,654.1 $18,449.0 8.97%Atlas II $234.6 $3,040.4 7.71%Brilliant Pebbles $77.8 $1,100.2 7.07%HST $113.3 $1,592.6 7.12%ISS $3,646.4 $36,027.4 10.12%Peacekeeper $265.6 $10,006.7 2.65%Pioneer Venus $24.9 $306.3 8.13%Saturn IB $325.1 $3,776.7 8.61%Saturn V $603.9 $10,333.9 5.84%Shuttle $597.8 $18,152.6 3.29%Skylab $334.1 $4,127.7 8.09%Titan IV $516.5 $5,178.0 9.97%

9* Reference – data collected and analyzed by Dr. Christian Smart, SAIC, under contract to NASA Marshall Space Flight Center.

RecommendationsCarefully consider the applicationConsider how SoS differs from traditional engineering systems

and how this affects the estimatorSupporting platforms are operationally, geographic, and managerial

independent, as well as network-centricNew acquisition concepts means we need new CERs, factors, and cost

driversImmaturity of concept means little cost data is currently available

SOS cost drivers are unique and require considerations beyond traditional systems estimating

Review available research and papersCurrent research by USC, DAU, SEI, NASA, Cranfield University

(UK)Available papers from USC, MIT, IEEE, INCOSE

10

Lessons LearnedInconsistent parameter definitions

Between models (need to adopt INCOSE standards and develop a mapping scheme)

Between historic data (collected before standards were established) and new CERs

Inconsistent WBS applicationsAre we double-counting the SOS costs by applying factors at each platform

level?Inconsistent platform applications

Do we use the same factors for hardware systems as well as software systems? Are space platform factors different from air and ground platform factors? Are manned-space platform factors different from unmanned-space platform factors?

Might be more useful if, in the near term, we rely on databases (and factors) rather than on statistical CERs

11

Publication ScheduleHighly motivated, volunteer workforceHosted in a collaborative work areaPeriodic “tag-up” teleconsCurrent Schedule

Review 1 – 31 October 2010Review 2 – 20 November 2010Materials to Editor – 30 November 2010Complete DR first draft – 31 December 2010

Seeking volunteer reviewersPublished Desk Reference will be available in

April 2011!12