1 FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann Contract #S-54493-G September 4, 2002 Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-Making
30
Embed
1 FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP) for the NASA Independent Verification and Validation Facility COTR: Kenneth McGill PI: Nancy Eickelmann.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
FY2001 CENTER SOFTWARE INITIATIVE PROPOSAL (CSIP)for the
NASA Independent Verification and Validation Facility COTR: Kenneth McGillPI: Nancy EickelmannContract #S-54493-G
September 4, 2002
Developing Risk-Based Financial Analysis Tools and Techniques to Aid IV&V Decision-
Making
2
Agenda
• Why we need ASK IVEY
• Consequences and Likelihood of Failure
• IV&V Yield
• Probability of IV&V Yield: Min, Max, Most Likely
• ROI and Magnitude of Return of IV&V
• What ASK IVEY can do
3
Why we need ASK IVEY
• NASA program managers are asked to quantify the ROI and evaluate the cost/benefit of applying IV&V technologies.
• This is a prediction of future events based on decisions and actions taken in the present.
• A point estimate is likely to be inaccurate, whereas a probability of yield has a history of providing a scope of potential yield and an extent of likelihood of expected yield.
4
Calculating ROI a Financial Analysis Prompt Map
Yes1 A
Financial Analysis Process Map
Create developmentcost framework:
Total CostCOQCOPQ
?
IV&V and IA analysis complete.Level of IV&V or IA designated
H i g h R i s k ( I V & V R e q u i r e d ) I n t e r m e d i a t e R i s k ( I A R e q u i r e d )
F i g u r e 1 S o f t w a r e R i s k
1 6 3 2 6 4 1 2 8 2 5 6
I V & V
I V & V
I V & V
I n s i g n i fi c a n t
M a r g i n a l
I A
I A
I A
9 6
T o t a l L i k e l i h o o d o f F a i l u r e B a s e d o n S o f t w a r e E n v i r o n m e n t
I A G r a v e
NPG 2820 IV&V Criteria
6
Likelihood of Failure
Factors contributing to probability of software failure
Weighting Factor
Likely- hood of failure rating
1 2 4 8 16Software team complexity
Up to 5 people at one location
Up to 10 people at one location
Up to 20 people at one location or 10 people with external support
Up to 50 people at one location or 20 people with external support
More than 50 people at one location or 20 people with external support
X2
Contractor Support
None Contractor with minor tasks
Contractor with major tasks
Contractor with major tasks critical to project success
X2
Organization Complexity*
One location Two locations but same reporting chain
Multiple locations but same reporting chain
Multiple providers with prime sub relationship
Multiple providers with associate relationship
X1
Schedule Pressure**
No deadline Deadline is negotiable
Non-negotiable deadline
X2
Process Maturity of Software Provider
Independent assessment of Capability Maturity Model (CMM) Level 4, 5
Independent assessment of CMM Level 3
Independent assessment of CMM Level 2
CMM Level 1 with record of repeated mission success
CMM Level 1 or equivalent
X2
Degree of Innovation
Proven and accepted
Proven but new to the development organization
Cutting edge X1
Level of Integration
Simple - Stand alone
Extensive Integration Required
X2
Requirement Maturity
Well defined objectives - No unknowns
Well defined objectives - Few unknowns
Preliminary objectives
Changing, ambiguous, or untestable objectives
X2
Software Lines of Code***
Less than 50K Over 500K Over 1000K X2
Total
Un-weighted probability of failure score
Table 1 Likelihood of Failures Based on Software Environment
7
IV&V YIELD
• Ultimately, the yield of an IV&V program is based upon the difference between the net resource flow with IV&V and without IV&V.
• If the resources saved (e.g., reduced rework) or returns gained (e.g., improved customer satisfaction or increased safety) are greater than the resources consumed to save/gain these resources, we have a net benefit.
• Should the resources saved be less than the resources consumed, we
have a net cost.
8
Cost of Poor Quality
• Defect Leakage– If discovered internally
• defect management• rework• retesting
– If discovered externally• technical support• complaint investigation• defect notification
9
Stephen Knox“Modeling the Cost of Software
Quality,”Digital Technical Journal, (Fall
1993)
0
10
20
30
40
50
60
Co
st a
s a
Pe
rce
nt o
f D
eve
lop
me
nt
1 2 3 4 5 SEI CMM Level
Prevention Appraisal Int Failure Ext Failure TCoSQ
•DIFFICULT TO MAINTAIN TOOLS & PRACTICES AT STATE OF THE ART
IV&VSOMEWHAT
UNPREDICTABLE
UNABLE TO ESTIMATE
NON-TECH %
IV&V
IV&V
IV&V
IV&V
NON-TECH UP TO 25%
NON-TECH UP TO 15%
NON-TECH UP TO 6-8%
NON-TECH UP TO 3-4%
MATURITY
12
Cost of Leakage GrowsOver Time
• Relative cost of fixing a problem found in design/coding, testing, or after release are:– 1:20:82 (Remus, 1983)– 1:13:92 (Kan, 1989)– 10:100:1000 (Coyle, 1999)
13
Cost of Rework in Each Phase• Reworkproduct design =
• There is some evidence to suggest organizations with increased maturity have reduced rework costs
• Knox: Percent of Budget to Rework:– Level 1: 55%
– Level 2: 45%
– Level 3: 35%
– Level 4: 20%
– Level 5: 6%
17
IV&V and Defect Leakage
• Application of IV&V can reduce leakage to subsequent phases
• The goal of the financial model is to propose a range of potential savings
• Specific parameters will need to be established empirically
18
Timing of benefits for IV&V
• Full In-Phase IV&V– prevention of errors starting at requirements - can
potentially bar any errors from leaking through
• Partial IV&V– prevention of errors at point of insertion - no errors from
this phase will leak
• Endgame IV&V– discovery of errors at the end of development - can
potentially bar any errors from leaking to deployment
• Audit Level IV&V
19
Rework and Return from IV&V By Maturity Level
Maturity Level IV&V Insertion
1 2 3 4 5
Full In-Phase IV&V(0% Rework)
0550,000(55% )
0450,000(45% )
0350,000(35% )
0200,000(20% )
060,000(6% )
IV&V @Requirements Only(22% Rework)
121,000429,000(43% )
99,000351,000(35% )
77,000273,000(27% )
44,000156,000(16% )
13,20046,800(47% )
IV&V @ DesignOnly(7% Rework)
38,500511,500(51% )
31,500418,500(42% )
24,500325,500(33% )
14,000186,000(19% )
4,20055,800(5% )
IV&V @Programming Only(7% Rework)
38,500511,500(51% )
31,500418,500(42% )
24,500325,500(33% )
14,000186,000(19% )
4,20055,800(5% )
End-Game IV&V,(28% Rework)
154,000396,000(40% )
126,000324,000(32% )
77,000273,000(27% )
56,000144,000(14% )
16,80043,200(4% )
20
Components to Return on Investment
• Cost of IV&V
• Expected Return– cost savings - measured as hours of rework
• Likelihood of Returns– how effective is the organization at minimizing
rework?– how effective will IV&V be?
21
Independence…
• An organization independent from the developers study the artifacts of software production [IEEE Std. 1012-1998].
• This requires:- Technical independence. Members of the IV&V team may not be
personnel involved in the development of the software. - .Managerial independence. The responsibility for IV&V belongs to
an organization outside the contractor and program organizations that develop the software.
- Financial independence. Control of the IV&V budget is retained in an organization outside the contractor and program organization that develop the software.
• IV&V is often perceived as testing the code after the development is completed …..NASA IV&V is full life cycle activities
22
IV&V is NOT SQA
• IV&V is a full life cycle set of acivities that are applied to defect prevention, defect detection, and certification. NASA IV&V conforms to IEEE Standard 1012-1998.
• IV&V and Software Quality Assurance (SQA) are not redundant activities. SQA as defined by DOD-Std 2168 defines 10 activities of SQA that are complemented by IV&V activities. There are 32 types of activities conducted by IV&V, of these 32, 22 are unique to IV&V and 10 are complemented by SQA.