Software Acquisition Best Practices 2004 Edition · Software Acquisition Best Practices: 2004 Edition Richard J. Adams Suellen Eslinger Karen L. Owens Mary A. Rich Software Engineering

Software Acquisition Best Practices:2004 Edition

Richard J. AdamsSuellen EslingerKaren L. Owens

Mary A. RichSoftware Engineering Subdivision

January 27, 2004

3rd OSD Conference on the Acquisition of Software-Intensive Systems

© 2003-2004 The Aerospace Corporation.

2

Acknowledgements

This work would not have been possible without assistance from thefollowing:

• Reviewers❖ Linda A. Abelson, Software Acquisition and Process Office❖ Peter Hantos, Software Acquisition and Process Office❖ John Cantrell, Software Architecture and Engineering Department

• Sponsor and Reviewer❖ Michael Zambrana, USAF Space and Missile Systems Center, Directorate

of Systems Engineering

• Funding Source❖ Mission-Oriented Investigative Experimentation (MOIE) Research Program

(Software Acquisition Task)

3

Outline

• Background and Definitions• Scope

❖ Software Acquisition Best Practices 2003 Reviewed❖ Scope of Software Acquisition Best Practices 2004

• Software Acquisition Best Practices 2004❖ Early Acquisition Life Cycle Phases❖ Evolutionary Acquisition

• Conclusion

4

• Definition: Best Practices are practices that people withrecognized expertise in the subject area have identified throughexperience as being significant contributors to project success

• Negative experience or positive experience may identify BestPractices❖ However, one must not be trapped by logical fallacies

• Note that Best Practices (both individually and collectively)❖ Have not necessarily undergone detailed study❖ Have almost never been analytically determined to be “best”❖ Never form an exhaustive set (There is always the possibility of more)❖ Are not static (They change with new experiences and new technologies)❖ Are dependent on the context and environment

Best Practices

PracticeNot A

PracticeA

5

Software Acquisition (SA) Best Practices

• Software Acquisition (SA) Best Practices are, therefore,practices that people with recognized software acquisitionexpertise have identified through experience as beingsignificant contributors to the successful acquisition ofsoftware-intensive systems

• The SA Best Practices presented derive from the researchteam’s collective experience in the acquisition of software-intensive space systems❖ Over 60 collective years of software acquisition experience spanning

approximately 20 years duration❖ Many additional years of experience in developing software,

managing software development projects, and leading softwareprocess improvement efforts

6

Characteristics of Space Systems (SS)

• Large software-intensive systems❖ SLOC order of magnitude: 105 onboard and 106 – 107 on the ground❖ Multi-satellite constellations❖ Multiple ground elements, frequently worldwide

• Complex combinations of hardware and software• Complex external and internal interfaces• Usually unprecedented• High reliability and integrity requirements• Developed by large teams of multiple contractors

Space Systems Software Acquisition BestPractices must support these characteristics.

7

Outline

• Background and Definitions• Best Practice Scope



• Conclusion

8

SS SA Best Practice Scope

• Single system development contract for a software-intensive system

• Pre- and post-contract award software acquisition activitiesfor the system development contract

• Full life cycle software acquisition activities spanning thecontract award boundary❖ Software Risk Management❖ Software Systems Acquisition

9

Software Acquisition Domain

Software Engineering DomainContractor

GOVERNMENT

Pre Contract Post ContractRFP

Proposal

DevelopmentContract

Establish Program BaselineObtain Contractual InsightObtain Contractual CommitmentSelect Capable Contractor TeamProvide Contract Management Tools

Perform Technical Product ReviewsPerform Software Process ReviewsManage the Development Contract

SS SA Best Practices for aSystem Development Contract

10

SS SA Best Practice Scope

• Software acquisition activities for the full DoD and NationalSecurity Space (NSS) acquisition life cycle

• Pre- and post-contract award software acquisition activitiesfor early DoD and NSS life cycle phases

• Evolutionary acquisition

DoD and NSS Acquisition Models*

JROCJROCICDICD

NSS Space Acq Policy 03-1

PHASE B (Design Phase)Risk Reduction & Design

Development

PHASE A (StudyPhase) Concept/Architecture Dev

PHASE C (Build, Test, Launch)Acquisition & Operations Support

BA

PDRPDRSDRSDR SRR SRR

Pre-Systems Acquisition Systems Acquisition Sustainment

C

PHASE BApproval

PHASE AApproval

PHASE CApproval

CDRCDR

Pre KDP-AActivities

TechnologyTechnologyDevelopmentDevelopmentApprovalApproval

SystemSystemDevelopment &Development &DemonstrationDemonstrationApprovalApproval

A B C

Productionand

Deployment

Operations andSupport

Low-RateLow-Rate Initial Prod Initial ProdApprovalApproval

DesignDesignReadinessReadiness Review Review

CDRCDR

ConceptRefinement

TechnologyDevelopment

System Development &Demonstration

DoDI 5000.2 (12 May 2003)

KeyKeyDecisionDecisionPoints:Points:

Milestones:Milestones:

Full RateProductionApproval

IOC

IOC

11* From National Security Space Acquisition Policy #03-01, 6 October 2003.

12

Outline




• Conclusion

13

Concept Refinement Best Practices(Pre KDP-A Activities)

Defining:• Program life cycle• Initial Government

architecture concepts• Initial Government cost

and schedule baselines• Executable program

evolutions• Global acquisition

strategy

JROCJROCICDICD

NSS Space Acq Policy 03-1

A

Pre-Systems Acquisition

PHASE AApproval

Pre KDP-AActivities


A

ConceptRefinement

DoDI 5000.2 (12 May 2003)



14

Best Practices for Defining theProgram Life Cycle

Use a software-friendlyacquisition model

• Avoid contract actions in themiddle of software developmentspirals (e.g., post System PDR)

• Develop firm basis for softwarecosting before MS B/KDP-B

• SDR level of maturity beforeMS B/KDP-B

• Selection of a single contractor atappropriate point in softwaredevelopment life cycle

• With or without production phase

ProgramLife Cycle

• Evolutionary acquisition is moresuited to large, complex software-intensive systems, such as spacesystems

Tailor the acquisition model forsoftware-intensive system

Choose software-friendly points inthe life cycle for contract actions

Example DoD and NSS Acquisition ModelsTailored for Software-Intensive Systems without Production

JROCJROCICDICD

NSS Space Acq Policy 03-1 (Adapted)

PHASE A/B Concept/Architecture Dev &

System Design

PHASE B/C Design, Development (Build, Test, Launch)& Operations Support

B/CA/B


Pre-Systems Acquisition Systems Acquisition Sustainment

PHASE B/CApproval

PHASE A/BApproval

Pre KDP-AActivities



A B C

Deployment Operations andSupport

LimitedLimitedDeploymentDeploymentApprovalApproval

DesignDesignReadinessReadiness Review Review

CDRCDR

ConceptRefinement

TechnologyDevelopment & System

Design

System Development &Demonstration

DoDI 5000.2 (12 May 2003) (Adapted)



CDRCDR


FullDeploymentApproval

IOC

IOC

15

16

Best Practices for Developing the InitialGovernment Architecture Concepts

Include software in evaluation ofarchitecture concepts

• Evaluate software evolution andgrowth capability

• Include software in life cycle costanalysis (COTS software refresh,legacy and new software re-engineering and maintenance)

GovernmentArchitecture

Concepts

Perform software-inclusivearchitecture trade studies

• Able to grow with each successiveevolution with little expected rework

• Able to integrate each successiveevolution with previous evolutions(and legacy system, as applicable)

• With system architecture trades• Identify and address critical HW/SW

architecture issues• Include major legacy components

and COTS software

Select a set of integratedHW/SW architecture concepts

17

Best Practices for Developing the InitialGovernment Cost and Schedule Baseline

Determine realistic SW effort & costestimates for each evolution

• Include COTS, reuse and newlydeveloped software

• Include tasks not reflected in costmodels (e.g., integration of SWcomponents costed separately, COTS)

• Include all software effort inschedule

• Never compress softwareschedule >20% off nominal*

Gov’t. Cost And Schedule

Baseline

Determine realistic SW scheduleestimates for each evolution

Determine realistic SW sizeestimates for each evolution

• Use Gov’t. HW/SW architecture concept• Include all SW functionality and

infrastructure needed• Use historical data from similar past

programs & early concept study data

* Software Program Manager’s Network, The ProgramManager’s Guide to Software Acquisition Best Practices,Version 2.31, p. 22.

18

Best Practices for DefiningExecutable Program Evolutions

Consider SW implications whendefining evolution capabilities

• Analyze feasibility of overlappingsoftware development schedulesfor closely spaced evolutions

• Avoid plans that require developingsubsequent evolutions on unknownsoftware baselines

• Analyze feasibility of COTS refreshand legacy SW upgrade schedules

ExecutableProgram

Evolutions

• Analyze feasibility of developing therequired software for each evolution

• Based on realistic software size,effort, cost and scheduleestimates

• Include software cost andschedule estimation risk

• Analyze feasibility of integrating thesoftware in each evolution with allprevious evolutions (and legacysystem(s), as applicable)

• Based on integrated hardware/software architecture

• Analyze impacts of COTS softwarerefresh and legacy softwareupgrades

Consider SW implications whendefining evolution schedules

19

Develop plans for computersystem technology insertion

Develop plans for evaluation ofcontractor software capability

• Perform a Government evaluationof contractor team softwarecapability

• Prior to or part of selection of asingle development contractor

GlobalAcquisition

Strategy

• Include COTS HW and SW refresh ineach successive evolution

• Understand new computer HW & SWtechnologies needed for eachevolution and study their readiness

• Plan for managing multiple baselines(operations and development)

• Plan for integrating softwaremaintenance actions on operationalevolutions into evolutions underdevelopment

Develop plans for softwaresupport

Best Practices for Developing theGlobal Acquisition Strategy

20

Concept/Technology/ArchitectureDevelopment (Phase A/B) Best Practices

Principal objective of Phase A/Bcontract(s)* is to develop theinformation needed for theGovernment to:

❖ Solidify the programdefinition to establish anexecutable program

❖ Update the globalacquisition strategy,including acquisition plansand products for this andall future evolutions

NSS Space Acq Policy 03-1 (Adapted)

PHASE A/B Concept/Architecture Dev &

System Design

B/CA/B

SDRSDR SRR SRR

Pre-Systems Acquisition

PHASE B/CApproval

PHASE A/BApproval



A B

DoDI 5000.2 (12 May 2003) (Adapted)



SDRSDR SRR SRR

TechnologyDevelopment & System

Design

* Space systems usually have multipleparallel contracts in this phase, withselection of a single development contractorin the next phase (B/C).

21

SS SA Best Practicesfor a Phase A/B Contract

Contractor

Software Acquisition DomainGOVERNMENT

Software Engineering Domain

Pre Contract Post Contract

Establish Requirements BaselineDevelop System Architecture ConceptReduce Software Risk

Manage the Phase A/B Contract

RFP

ProposalPhase A/BContract

22

Include software in Gov’t. systemperformance requirements

Contract for delivery of SW-inclusive reqs. specifications

• Require System and SegmentSpecifications as CDRL items

• Use System/SubsystemSpecification DID (DI-IPSC-81431a)

RequirementsBaseline

Best Practices for Establishing theRequirements Baseline

• Specialty engineering, especiallyRMA

• Key Performance Parameters• Open system architecture• Design for evolution and growth

23

Contract for softwarearchitecture trade studies

Contract for delivery of systemarchitecture

• Require system architecture as aCDRL item

• Require an integrated HW/SWarchitecture, defined by multiplearchitecture views

• Include newly developed, reuse andCOTS software

SystemArchitectural

Design

• With system architecture trades• Include major software legacy

components and COTS software

Best Practices for Developing theSystem Architectural Design

24

Contract for software productrisk reduction

Contract for software processrisk reduction

• Require delivery of SoftwareDevelopment Plan (DID DI-IPSC-81427a)

• Require compliance with robustsoftware development standard

• Enable contractor team to preparefor software capability evaluation

SW DevelopmentRisk Reduction

• Studies/prototyping of high riskareas for software, e.g.

•Mission processing algorithms•Mission planning concepts

• Simulation development• Increase readiness level of computer

HW and SW technologies

Best Practices for ReducingSoftware Development Risk

25

Best Practices for Managingthe Phase A/B Contract

• Government software acquisitionpersonnel with technical expertisein software product and processengineering must participate

Participate with contractor insoftware risk reduction

Ensure contractor(s) defineintegrated HW/SW architecture

• Software systems engineers(contractor and Government)must participate with contractorand Gov’t. systems engineers

• Software systems engineers(contractor and Government)must participate with contractorand Gov’t. systems engineers

Ensure contractor(s) definesoftware-inclusive reqs. specs.

Managing the Contract

26

Evolutionary Acquisition Strategy - 1

Global Acquisition

Strategy

A/B B/C (O&S)

B/CIOC

Increment 3

PreA

A/B B/C (O&S)

B/CA/B

IOC

Increment 1

A/B B/C (O&S)

B/CIOC

Increment 2

Ongoing or near term

Future planning

Feedback

Acquisition Planning

27

Evolutionary Acquisition Strategy - 2

PreA

A/B B/C (O&S)

B/CA/B

IOC

Future planning

A/B B/C (O&S)

B/CIOC

Increment 3

Global Acquisition

Strategy

Increment 1

A/B B/C (O&S)

B/CIOC

Increment 2

Ongoing or near term

Feedback

Feedback

Acquisition Planning

28

Update SW-inclusive programbaseline

Update definition of SW-friendlyevolutions

• Evolution capabilities, schedulesand integration strategies

• COTS software refresh and legacysoftware upgrades

• Software support strategy• Contractor team software capability

evaluations• Software technology insertion• Software transition to operations

Updated Global Acquisition

Strategy

• Software-inclusive systemrequirements

• Integrated HW/SW architecture• Realistic software size, effort, cost &

schedule estimates for eachevolution

Update software-specific plans

Best Practices for Updating theGlobal Acquisition Strategy

29

Software Acquisition RiskManagement

• Integrate software acquisition withthe system acquisition process

• From capability needsidentification through systemretirement

• Especially during earlyacquisition life cycle phases

• Continuous software acquisitionrisk management

• Across the entire acquisitionlife cycle

• Across all evolutions• Within each ongoing evolution

• Program level risk management andcontractor development riskmanagement are necessary but notsufficient

Software Systems Acquisition

Best Practices that Span theDoD and NSS Acquisition Life Cycle

Software Acquisition Risk ManagementSoftware Systems Acquisition

PHASE A/B PHASE B/CPre KDP-A

30

Outline




• Conclusion

31

Conclusion

• Software acquisition best practices do not guarantee success❖ They are not a panacea!

• Using best practices, however, can reduce risk in complexsoftware-intensive system acquisitions

• Evolutionary acquisition, in particular, is a complex strategythat requires careful planning and execution in order toachieve its anticipated benefits

• Software acquisition best practices will be most effectivelyimplemented if done in the context of a software acquisitionprocess improvement program❖ Based on experiences with software development

Section 804 of the FY03 Defense Authorization Actrequires the establishment of software acquisition

process improvement programs.

32

Back-Up Charts

• Software Acquisition Best Practices 2003• Acronym List• Author Contact Information

33

Perform software architecture-inclusive trade studies

Determine realistic, independentbaseline software estimates

• Size, effort, cost and schedule• COTS, reuse and newly developed• Tasks not reflected in cost models• Realism especially critical for

evolutionary acquisition

• Specialty engineering, esp. RMA• Key Performance Parameters• Open system architecture

ProgramBaseline

• With system architecture trades• Include major legacy components• Supports Government software

architecture baseline selection

Include software in systemperformance requirements

Best Practices forEstablishing the Program Baseline

34

Best Practices forObtaining Contractual Insight

• In addition to system reviews

ContractualInsight

• Highest risk reduction potential:• Plans (development, build,

transition)• Requirements & Architecture• Test plans, procedures & reports• Metrics reports• Delivery, installation & maintenance

documentation• Use electronic delivery

Require key software technical& management deliverables

Require software level technical& management reviews

Require timely electronicaccess to all software products

• Requirements• Architecture, Design• Implementation (including code)• Integration and Verification Testing• Intermediate and Final Products

35

Mandate compliance with robustfull life cycle SW dev. standard

• Include commitment in IntegratedMaster Plan (IMP)

ContractualCommitment

• For example, EIA/IEEE J-STD-016

Require contractor commitmentto Software Development Plan

Best Practices forObtaining Contractual Commitment

36

Evaluate software capability aspart of source selection

Capable Software Contractor Team

• Evaluate software capability ofofferor teams

• Individual team memberevaluation insufficient

• Evaluate software capability/processes as subfactor

• Under Mission Capability factor• Weight according to software

risk• Evaluate teams’ proposed software

processes• Corporate and past project

process evaluation insufficient

Best Practices for Selecting aCapable Software Contractor Team

• Suspect extremes of productivity,COTS & reuse, & low lines of code

Evaluate realism of cost andschedule bids

Evaluate software architecturewith system design

• Evaluate major HW/SW architectureissues (e.g., space-ground trades,reuse of legacy components)

37

Incentivize software quality,*not just cost and schedule

• Relate results and improvementactions directly to award fee

Tools forContract

Management

• Use award and incentive fee plans• Reward adherence to

• Defined software processes• Software process improvement

• Reward timely and adequateresponse to Government comments

• Reward low rework rates• Reward meeting RMA requirements

post delivery/launch

Mandate periodic team softwarecapability appraisals

Best Practices for Providing Toolsfor Contract Management

* Quality in this context is producing workproducts that do not require rework insuccessor activities.

38

Perform in-depth technicalreviews of software products

• Begin at the build level• Focus on areas of highest risk• Focus on early performance

analysis results and meeting KPPs

Technical Product Reviews

• IPTs, TIMs, working groups, peerreviews, etc.

• Software Level Technical Reviews• High risk/critical software products• Key software technical deliverables• Focus on areas of highest risk

Monitor software integrationand verification adequacy

Best Practices for PerformingTechnical Product Reviews

Include users/operators in alltechnical review activities

• Focus on operational suitability ofevolving software-intensive system

39

Best Practices for PerformingSoftware Process Reviews

SoftwareProcessReviews

• Review team’s adherence to definedsoftware processes

• Identify adherence deficiencies• Assist in deficiency correction

• Evaluate effectiveness of definedSW processes

• Identify process deficiencies• Assist with process

improvement• Level 2 & 3 CMMI®/CMM® adherence

for an individual team member maynot be sufficient*

Review effectiveness ofcontractor team’s SW processes

Perform periodic team softwarecapability appraisals

• During contract performance• Support for significant program or

award fee milestones

* CMM and CMMI are registered trademarksof Carnegie Mellon University.

40

Use incentive/award feesaggressively

• Especially RMA

Managing the Contract

• Motivate good software practices• Focus on quality

Ensure satisfaction of software–inclusive requirements

Best Practices forManaging the Development Contract

Apply proactive quantitativemanagement

• Ensure a comprehensive software/system metrics program balancedacross information categories

• Include leading quality indicators(e.g., rework)

• Perform cross-metric analysis• Earned value alone is insufficient

Perform periodic independentassessments

• Support for significant program oraward fee milestones

• Act aggressively on findings

41

Acronyms and Abbreviations - 1

Acq AcquisitionCDR Critical Design ReviewCDRL Contract Data Requirements ListCMM® Capability Maturity Model®

CMMI® Capability Maturity Model® IntegrationSM

COTS Commercial Off the ShelfDB DatabaseDev DevelopmentDID Data Item DescriptionDoD Department of DefenseDoDI DoD InstructionEIA Electronic Industries AllianceFY Fiscal YearGov’t. GovernmentGUI Graphical User InterfaceHW HardwareIEEE Institute of Electrical and Electronics Engineers

42


IMP Integrated Management PlanIPT Integrated Product TeamIOC Interim Operational CapabilityJ JointKDP Key Decision PointKPP Key Performance ParameterMOIE Mission-Oriented Investigation and ExperimentationMS MilestoneNSS National Security SpaceO&S Operations and SupportOSD Office of the Secretary of DefensePDR Preliminary Design ReviewRFP Request for ProposalRMA Reliability, Maintainability, AvailabilitySA Software AcquisitionSDP Software Development PlanSDR System Design Review

43


SLOC Source Lines of CodeSM Service MarkSRR System Requirements ReviewSS Space SystemSTD StandardSW SoftwareTIM Technical Interchange MeetingUSAF United States Air Force

44

Author Contact Information

• Richard J. Adams❖ Senior Engineering Specialist❖ Software Engineering

Subdivision, The AerospaceCorporation

❖ (310) 336-2907❖ [email protected]

• Suellen Eslinger❖ Distinguished Engineer❖ Software Engineering



• Karen L. Owens❖ Senior Engineering Specialist❖ Software Engineering



• Mary A. Rich❖ Principal Director❖ Software Engineering



Software Acquisition Best Practices 2004 Edition · Software Acquisition Best Practices: 2004 Edition Richard J. Adams Suellen Eslinger Karen L. Owens Mary A. Rich Software Engineering

Documents