Using Parametric Software Estimates During Program Support Reviews Chris Miller Office of the Deputy Director, Software Engineering and System Assurance SYSTEMS & SOFTWARE ENGINEERING Office of the Deputy Under Secretary of Defense for Acquisition and Technology US Department of Defense October 2008 Version 2.0
24
Embed
Using Parametric Software Estimates During Program Support Reviews Chris Miller Office of the Deputy Director, Software Engineering and System Assurance.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using Parametric Software Estimates During
Program Support Reviews
Chris Miller Office of the Deputy Director,
Software Engineering and System Assurance
SYSTEMS & SOFTWARE ENGINEERINGOffice of the Deputy Under Secretary of Defense
for Acquisition and TechnologyUS Department of Defense
October 2008
Version 2.0Version 2.0
Systems & Software Engineering – Chris Miller, October 2008 Slide 2
OUSD (AT&L) OrganizationMay 2006
USD, AcquisitionTechnology & Logistics
DUSD, Acquisition &Technology
Dir, Joint AdvancedConcepts
Dir, Systems andSoftware Engineering
Dir, PortfolioSystems Acquisition
IndustrialPrograms
Defense Procurementand Acquisition Policy
Small BusinessPrograms
Defense ContractManagement Agency
Defense AcquisitionUniversity
Systems & Software Engineering – Chris Miller, October 2008 Slide 3
Director, Systems &Software Engineering
Systems and Software Engineering
Director, Systems &Software Engineering
Deputy DirectorEnterprise Development
Deputy DirectorDevelopmental Test
& Evaluation
Deputy DirectorSoftware Engineering &
System Assurance
Deputy DirectorAssessments & Support
Systems & Software Engineering – Chris Miller, October 2008 Slide 4
Elements of a DoD Strategy for Software
• Support Acquisition Success
– Ensure effective and efficient software solutions across the acquisition spectrum of systems, SoS and capability portfolios
• Improve the State-of-the-Practice of Software Engineering
– Advocate and lead software initiatives to improve the state-of-the-practices through transition of tools, techniques, etc.
• Leadership, Outreach and Advocacy
– Implement at Department and National levels, a strategic plan for meeting Defense software requirements
• Foster Software Resources to meet DoD needs
– Enable the US and global capability to meet Department software needs, in an assured and responsive manner
Promote World-Class Leadership for Defense Software EngineeringPromote World-Class Leadership for Defense Software Engineering
Systems & Software Engineering – Chris Miller, October 2008
Notional View of Software Measurement
Software Engineering and Systems Assurance (SSA) initiatives • Program feasibility analysis using estimation models • Software Resources and Data Report: Feasibility Study• Revision of MIL-HDBK-881A to improve software guidance• Integration of software metrics with EVM to assess consistency of estimates
Concepts - Requirements - Arch/Design Development - MaintenanceConcepts - Requirements - Arch/Design Development - Maintenance
Determine methods of obtaining cost estimating
data
Determine methods of obtaining cost estimating
dataUse estimation tools,
techniques, processes, & practices
Use estimation tools, techniques, processes,
& practices
Generate SW appropriate
WBS
Generate SW appropriate
WBS
Earned Value Management (EVM)
for SW
Earned Value Management (EVM)
for SW
Link SW quality indicators to EVM
Link SW quality indicators to EVM
Systems & Software Engineering – Chris Miller, October 2008 6
Driving Technical Rigor Back Into Programs “Program Support Reviews”
• Program Support Reviews provide insight into a program’s technical execution focusing on:
– SE as envisioned in program’s technical planning
– T&E as captured in verification and validation strategy
– Risk management - integrated, effective and resourced
– Milestone exit criteria as captured in Acquisition Decision Memo
– Acquisition strategy as captured in Acquisition Strategy Report
• Independent, cross-functional view aimed at providing risk-reduction recommendations
The PSR reduces risk in the technical and programmatic execution of a program
The PSR reduces risk in the technical and programmatic execution of a program
Systems & Software Engineering – Chris Miller, October 2008 7
Program Support Review Activity (as of August 2008)
Systems & Software Engineering – Chris Miller, October 2008 8
DoD Software Performance:What We’re Seeing*
• Software issues are significant contributors to poor program execution– Schedule realism (compressed, overlapping)– Software requirements not well defined, traceable, testable– Immature architectures, COTS integration, interoperability,
obsolescence (electronics/hardware refresh)– Software development processes not institutionalized, planning
documents missing or incomplete, reuse strategies inconsistent– Software test/evaluation lacking rigor and breadth– Lessons learned not incorporated into successive builds– Software risks/metrics not well defined, managed
*Based on over 60 program reviews over the past 3 ½ years*Based on over 60 program reviews over the past 3 ½ years
Systems & Software Engineering – Chris Miller, October 2008 9
Software Engineering and System Assurance (SSA) Role
• SSA produced software estimates to support several PSR teams in 2007– Program A: SSA was asked to assess software schedule
feasibility prior to MS B– Program B: Significant software issues
• Opportunity for SSA to support program decision making by providing software estimates– Estimation activities aimed at gauging overall program feasibility
and quantifying magnitude of top program risks– Focus on support for engineering vs. budgeting decisions
Systems & Software Engineering – Chris Miller, October 2008 10
Program A
• PSR of aircraft program with ambitious schedule– Three years (36 months) from Milestone B to LRIP– Modifications needed to meet U.S. requirements
• Developed three software estimates– Software size (SLOC) estimates provided by program office– Re-estimated both new and reused code
» Based on reuse and code growth» Estimates for optimistic, typical, and pessimistic were identified
as “Min”, “Mid”, and “Max”– Used three parametric models
» COCOMO II, SLIM, and SEER-SEM– Most input parameters set at nominal
Systems & Software Engineering – Chris Miller, October 2008 11
Program A
• Estimation & Feasibility Analysis– Given an adjusted code size (ASIZE) of 918K to 1590K SLOCs– A range of 5375 to 9571 person months should be expected over a 62
to 74 calendar month schedule
– All three models forecasted 65 to 68 months (assuming a 50% confidence-level)
• Result: – Analysis revealed existing acquisition strategy was not feasible– Service added more schedule to acquisition strategy– DUSD(A&T) estimates change in acquisition strategy saved $5 billion
Calculated Estimate at Complete Range:$1,797K - $2,515K
Cumulative CPI = 0.79SW Metrics
Calculated Schedule Range:05/14/08 - 06/18/08
Calculated Estimate at Complete Range:$1,570K - $1,710K
SW MetricETC
SW MetricETC
Program Plan ETC Program Plan ETC
EV ETC EV ETC
Acquisition Oversight confidence increases as ETCs overlapi.e., as multiple measures indicate to same conclusion
Acquisition Oversight confidence increases as ETCs overlapi.e., as multiple measures indicate to same conclusion
Systems & Software Engineering – Chris Miller, October 2008
SRDR Feasibility Study
• SSA funded a STSC feasibility study on normalizing DCARC data
– Purpose of the study was to determine if the DCARC data set could be used to develop a parametric software estimation model
– Dr. Randall Jensen at Hill AFB is the creator of Sage and SEER-SEM
• Preliminary study results – Data was submitted using a wide variety of definitions that made it
difficult to use directly, but was successful in normalizing the data and validating the results
– The data collected in the SRDR forms allows a first-level parametric model to be constructed (lack of environment information prohibits the creation of a detailed model)
– Minimum development time projection model
Systems & Software Engineering – Chris Miller, October 2008
Minimum Software Development Time Projection Model
Source: Long, L. G. et al, Aerospace Corp Report,2004
Impossible Zone
Impossible Zone
Analysis of CSCI data:• Apparent Maximum Size of 200 KSLOC (per CSCI) • Apparent Minimum Schedule (Impossible Zone)• Environment & Complexity are primary drivers of variation• Apparent Maximum Schedule of 50 months
Analysis of CSCI data:• Apparent Maximum Size of 200 KSLOC (per CSCI) • Apparent Minimum Schedule (Impossible Zone)• Environment & Complexity are primary drivers of variation• Apparent Maximum Schedule of 50 months
Systems & Software Engineering – Chris Miller, October 2008
Updating MIL-HDBK-881A for Software Overview
• Revision objectives:– Revise to make handbook acceptable of software
engineering practice – Correct errors
• Notable changes: – Replace 'material item' with ' acquisition program' – Add words to make 'artifacts' equal to 'products'– Include words that make 'product-oriented' and 'DoDAF
architecture' views interchangeable WRT to creating a WBS• Results
– Compiled a Comment matrix– Drafted a new Appendix B (for software)
• During final review and clean up, the working group was made aware of 90’s effort (draft MIL-HDBK-171)
Systems & Software Engineering – Chris Miller, October 2008
MIL-HDBK-881A Next Steps
• The Software Cost Control working group goal is to provide an community-coordinated set of recommendations
• Reached out to industry members of NDIA to review the suggested changes– Ensure recommended changes are improvements from industry
perspective– Obtain additional examples to include in Appendix B– NDIA feedback is due on September 19th
• Submission of suggested changes to PA&E ARA will take place after collection of NDIA feedback and consolidate into a single baseline
Systems & Software Engineering – Chris Miller, October 2008 24
Questions/Discussion
Contact Information:
Chris MillerSSE/SSA Support Software Engineering & Systems Assurance ODUSD(A&T) Systems & Software [email protected]