Date: 02/09/10 Slide 1 Are Your Software Measures Useful? Or Are You Getting Your Money’s Worth From Your Measures? Used with Permission
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Date: 02/09/10 Slide 1
Are Your Software Measures Useful?
Or
Are You Getting Your Money’s Worth From Your Measures?
Used with Permission
Date: 02/09/10 Slide 2
Agenda
# A
ctiv
ities
• Why Measure?
• How Should Measures Be Selected?
• Are There MeasurementRequirements?
• Which Measures AreUseful?
Why Measure? (1 of 2)
• For the benefit of your current project– Use objective measurement data to plan, track, and correct
project– Verify that technical/quality requirements have been met
• For the benefit of your future projects (and the rest of your organization, too!)– Help create basis for planning future projects– Create a baseline of “typical” or “expected” performance– Provide a subset of your measures to your organization
Date: 02/09/10 Slide 3
Why Measure? (2 of 2)
• Because it’s required!– Comply with NPR 7150.2 measurement requirements– Tools can help meet these requirements
• Selection and evaluation of new technology– Develop a rationale for adopting/refining technologies– Assess the impacts of techniques
• Process Improvement– Determine the strengths and weaknesses of the current
process and product– Assess impacts of improvements
Date: 02/09/10 Slide 4
Why Measure?Measures Help You Manage!
Where am I?What should
I do now?
Point Counts
StaffingSchedule
RiskDefects
Changes
I’m on planor
I’m varying from plan
orI’m off plan and need help!!!!
Analyze ReportMeasure
Date: 02/09/10 Slide 5
How to Select Measures
• Preferred methods for guiding measurement selection:– The Goal-Question-Metric (GQM) method or – The Goal-Question-Indicator-Metric (GQIM) method
• First, decide what your measurement “goals” are:– What information are you trying to learn?– What decisions will require additional information?
• Next, what “questions” will provide the information to meet your goal– Each goal may require several questions
• Then decide what types of charts or “indicators” might show the answers to the questions– Data at points in time, trends, stoplight charts ….
• Now ---What “metrics” or measures are required for those charts?
Date: 02/09/10 Slide 6
An Example: Selecting Measures for Progress Tracking (1 of 3)
• Goals or Objectives– Keep cost and schedule within acceptable bounds– Monitor effort by process activity to assure correct staffing
• Possible Questions:– Will the project finish on schedule?– Can the project stay within its allotted budget without
reducing requirements?– Is there enough staffing in the right areas?
• Possible Indicators:– Would a stoplight chart tell me what I need?– If I see a problem, do I have enough other information to fix
it?
Date: 02/09/10 Slide 7
An Example: Selecting Measures for Progress Tracking (2 of 3)
• Typical measures – Planned and actual data for …– Event dates– Progress tracking points– Budget and staff effort– Effort for each process activity
• Selection rules– Require at least one measure of schedule progress and one measure
of cost or effort– Need process measures for some CMMI process areas– Start small – know how you are going to use the measures
Date: 02/09/10 Slide 8
An Example: Selecting Measures for Software Progress Tracking (3 of 3)
How will I know if I’m falling behind?Budget and ScheduleObjectives
Point Counting Status
0
200
400
600
800
1000
1200
1400
1600
1800
6/29
/200
4
7/29
/200
4
8/29
/200
4
9/29
/200
4
10/2
9/20
04
11/2
9/20
04
12/2
9/20
04
1/29
/200
5
2/28
/200
5
3/29
/200
5
4/29
/200
5
5/29
/200
5
6/29
/200
5
7/29
/200
5
8/29
/200
5
9/29
/200
5
Week Ending
Poin
ts
PlanActualBaselineUpperLower
As Of Nov-15-2005SDO Build 1 Test
Planned and Actual Progress Points
How will I know if I have enoughresources? The right ones?
0.00
5.00
10.00
15.00
20.00
25.00
30.00
Staf
f-Mon
ths
per M
onth
(FTE
s)
Planned / Mo 1.60 1.80 1.60 1.85 3.55 4.40 9.05 13.55 23.25 23.25 23.40 24.85 24.85 24.85 24.85 24.85 24.85 24.85
Projected / Mo 1.60 1.80 1.60 2.40 3.80 4.20 6.50 10.55 17.10 18.10 16.85 16.15 24.85 24.85 24.85 24.85 24.85 24.85
Actual / Mo 1.60 1.80 1.60 2.40 3.80 4.20 6.50 10.55 17.10 18.10 16.85 16.15
Nov-04
Dec-04
Jan-05
Feb-05
Mar-05
Apr-05
May-05
Jun-05 Jul-05 Aug-
05Sep-05
Oct-05
Nov-05
Dec-05
Jan-06
Feb-06
Mar-06
Apr-06
ABC Monthly Planned vs Actual/Projected Staffing-Project Funded OnlyBaselined: Oct-04 Actuals As Of: Oct-05
Planned and Actual Staffing
Scheduled Activities
Date: 02/09/10 Slide 9
Measurement Requirements(As Per NPR 7150.2)
• Required measurement areas for all software projects– Software Progress Tracking– Software Functionality– Software Quality– Software Requirements Volatility– Software Characteristics
• Additional NPR requirements for Class A and B projects– Process monitoring as required for CMMI Generic Practice 2.8– Data specified for Software Inspection/Peer Review Report– Data collected “on a CSCI basis”
Date: 02/09/10 Slide 10
Examples of Required Measures (1 of 2)
Date: 02/09/10 Slide 11
Software Progress Tracking
Software Functionality
Examples of Required Measures (2 of 2)
Date: 02/09/10 Slide 12
Software Volitility
Software Quality
Charts for Discussion
So----What Did the Software Managers Submit as
“Useful” Charts?
Date: 02/09/10 Slide 13
Progress Tracking:Development Progress
Analysis: Progress has fallen below threshold due to 1) SS1: More effort than planned to address DR/ERs 2) SS2: Lead developer delayed due to office move and change of development platform and 3) SS3 and SS$: No development staff available
Impact: 1) SS1 will defer some of their testing until release 2.5 2) SS2 will not implement all defect fixes as planned and 3) SS3 and SS4 will not be included in the April release as planned (Not required project deliverables)
Corrective Action: 1) Reschedule the non-implemented tasks and 2) Work with component leads to redefine the work for SS3 and SS4.
Date: 02/09/10 Slide 14
Date: 02/09/10
Progress Tracking:18 Month Staffing Chart
Analysis: Lost one of SS1 developers but received additional staffing this month. Staffing is still less than planned on other components
Impact: Additional staffing needs to come up to speed and this may affect the scheduled April release.
Corrective: Let the customers know of the staff changes.
Slide 15
Progress Tracking:Requirements Implementation Status
This chart was used in an MMR by the software manager of a FSW project.
Baselined L4 requirements have been through MRR and are being used by module developers. These will be verified in SW I&T.
Date: 02/09/10 Slide 16
Date: 02/09/10 Slide 17
Progress Tracking:Earned Value –Schedule Variance
Date: 02/09/10 Slide 18
Progress Tracking:Earned Value –Cost Variance
Status Tracking:Other Direct Costs (ODC)
Date: 02/09/10 Slide 19
Multiple Replans Due To Delayed Funding!
Date: 02/09/10
Progress Tracking:Overall RID Status
Have 103 of 449 RIDs left to be implemented. (Less than 25% of RIDs remain unimplemented)
Slide 20
Requirements Volatility
XX
X
Date: 02/09/10 Slide 21
Quality Example - Defect Data
Status: More than expected number of reports, but opening and closing at about the same rate until near the end.
Impact: None since we are not gaining or losing ground.Corrective Action: Additional attendees are now in the code walkthroughs and test inspections.
PLOG Status
0
20
40
60
80
100
120
140
160
180
11/8/2001 11/15/2001 11/22/2001 11/29/2001 12/6/2001 12/13/2001 12/20/2001 12/27/2001 1/3/2002
PLO
G C
ount
Opened Closed Active Expected R1 Expected R2
7-JanAs ofR2002.1 Example
Date: 02/09/10 Slide 22
Quality/Progress Tracking:Software Defect Status
Expect 1386 work hours to repair defects
- 25% increase from November- V&A and CA activities only- 8.2 hours on average to complete
(Based on institutional)- 169 defects are in these states
Days open are starting to increase again
Slide 23Date: 02/09/10
Progress Analysis: Defect Status by State
This chart was requested by line management to help determine the amount of work to be performed by the line organization. Defect life-cycle states above the line are to be worked by the implementing organization. States below the line are to be worked by the project.DPFR-Development Problem Failure Report; PFR-Problem Failure Report; PRS-Problem Reporting System
Date: 02/09/10 Slide 24
Quality Analysis:Defect Containment
This metric will be shown at the conclusion of each Formal Review. The purpose of this metric is to show all of the “out of phase” defects in the Project Work Products. There typically should be a very low percentage of out of phase RIDs, if the documents have been adequately reviewed prior to the formal reviews.
PDR CDR
Date: 02/09/10 Slide 25
Date: 02/09/10 Slide 26
Quality Analysis:Need Better Code Inspections?
Document
Total Number of RIDsSuspense Date
12/03/2008Suspense Date
01/21/2010 RIDs Open to DateRIDs Awaiting Concurrence RIDs Recommending Closure RIDs Closed
Requirement Document 60 59 1 0 0 0 59
System Design Document 13 8 5 5 0 0 8
Verification and Validation Plan 14 7 7 7 0 0 7
Quality/Progress:RID Monthly Status
Date: 02/09/10 Slide 27
Functionality: Deliveries as Promised?
The due dates for some milestones have been realigned.
Appear to be on track to meet the realigned schedule,
Date: 02/09/10 Slide 28
Date: 02/09/10 Slide 29
Technology Assessments: Are Inspections More Effective For Certain Defects?
• NASA software managers found a variety of measures useful
• They chose measures to:– Help them manage their projects---– Answer their questions----– Give them early indicators of possible problems– Improve their processes– Evaluate potential technologies
• Decide what you want to know and get busy measuring it!
Date: 02/09/10 Slide 30
Summary
Contact Information
Sally GodfreyMetrics Subgroup Lead, NASA Software Working Group
NASA/Goddard Space Flight CenterCode 585
Greenbelt, MD 20771
Date: 02/09/10 Slide 31
Back-Up Slides
Date: 02/09/10 Slide 32
Date: 02/09/10 Slide 33
Date: 02/09/10 Slide 34
Errors Found in System TestingNeed More Focus On Code Logic?
Process Document Peer Review Metric
Burn-down chart for the Requirements/Design Section Peer Review Comments. All 59 comments
were ready to verify as of 6/26/09. Date: 02/09/10 Slide 35
Requirements Stability Example
Requirements Volatility Status:• Requirements have begun to stabilize since the SRR in February. We are continuing to work with the
Project to resolve the remaining 5 TBDs before the PDR in August, one of which could cause significant impact.
Requirements Stability
0
100
200
300
400
500
600
700
800
Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05 Jul-05 Aug-05 Sep-05
Num
ber o
f Req
uire
men
ts
Changes 0 27 53 37 8 16 0 0
Additions 0 78 127 78 22 0 0 0
Deletions 0 0 0 53 0 0 0 0
Baseline 475 475 553 680 705 727 727 727 727 727 727 727
TBDs 125 125 66 10 5 5 5 5
Oct 04 Nov 04 Dec 04 Jan 04 Feb 04 Mar 04 Apr 04 May 04 Jun 04 Jul 04 Aug 04 Sep 04
Requirements Volatility Status:• Requirements have begun to stabilize since the SRR in February. We are continuing to work with the
Project to resolve the remaining 5 TBDs before the PDR in August, one of which could cause significant impact.
Requirements Stability
0
100
200
300
400
500
600
700
800
Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05 Jul-05 Aug-05 Sep-05
Num
ber o
f Req
uire
men
ts
Changes 0 27 53 37 8 16 0 0
Additions 0 78 127 78 22 0 0 0
Deletions 0 0 0 53 0 0 0 0
BaselineBaselineBaseline 475 475 553 680 705 727 727 727 727 727 727 727
TBDsTBDs 125 125 66 10 5 5 5 5
Oct 04 Nov 04 Dec 04 Jan 04 Feb 04 Mar 04 Apr 04 May 04 Jun 04 Jul 04 Aug 04 Sep 04
Status: Requirements have begun to stabilize since the SRR in February but 5 TBDs remain.Impact: One TBD could cause significant impact.Corrective Action: We are working closely with the Project to resolve all remaining TBDs before the PDR in August. Status is
reported weekly.
Date: 02/09/10 Slide 36