page 1 September 2009 Supporting the CMMI Metrics Framework thru Level 5 Supporting the CMMI Metrics Framework thru Level 5 EDS EDS - - Electronic Data Systems do Electronic Data Systems do Brasil Brasil Ltda Ltda . . M M á á rcio rcio Silveira Silveira
32
Embed
Supporting the CMMI Metrics Framework thru Level 5
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
page 1
03-23-05September 2009
Supporting the CMMI Metrics Framework thru Level 5Supporting the CMMI Metrics Framework thru Level 5
EDSEDS--Electronic Data Systems do Electronic Data Systems do BrasilBrasil LtdaLtda. .
MMáárciorcio SilveiraSilveira
page 2September 2009Supporting the CMMI Metrics Framework thru Level 5
AgendaAgendaAgenda
• Objective
• EDS Overall Process Improvement Strategy
• Measurement Elements of the CMMI Model
• M&A and High-Maturity Practices
• Processes and Procedures for M&A and High-Maturity
• Lessons Learned
• Q&A
page 3September 2009Supporting the CMMI Metrics Framework thru Level 5
ObjectiveObjective
This presentation shows the Software Engineering Framework that EDS utilizes at its Applications Development and Maintenance Centers in Brazil that supports the measurement requirements of the CMMI® model thru level 5. During the presentation you will learn the basics and advanced measurement requirements from the CMMI® model and see how EDS implemented the processes and procedures to support these requirements.
® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
page 4September 2009Supporting the CMMI Metrics Framework thru Level 5
EDS Overall Process Improvement StrategyEDS Overall Process Improvement StrategyEDS Overall Process Improvement Strategy
page 11September 2009Supporting the CMMI Metrics Framework thru Level 5
OPP & QPM ObjectivesOPP & QPM ObjectivesOPP & QPM Objectives• The purpose of Organizational
Process Performance (OPP) is to establish and maintain a quantitative understanding of the performance of the organization’s set of standard processes in support of quality and process-performance objectives by:
Selecting processes to be managed quantitatively
Establishing process performance measurements and quality and process performance objectives
Creating and maintaining Process performance baselines and models
• The purpose of Quantitative Project Management (QPM) is to quantitatively manage the project’s defined process to achieve the project’s established quality and process-performance objectives by:
Establishing and maintaining the project’s quality and process-performance objectives
Identifying suitable sub-processes that compose the project’s defined process based on historical stability and capability data found in process-performance baselines or models
Selecting the sub-processes of the project’s defined process to be statistically managed
Monitoring the project to determine whether the project’s objectives for quality and process performance are being satisfied, and identifying appropriate corrective action
Selecting the measures and analytic techniques to be used in statistically managing the selected sub-processes
Establishing and maintaining an understanding of the variation of the selected sub-processes using the selected measures and analytic techniques
Monitoring the performance of the selected sub-processes to determine whether they are capable of satisfying their quality and process-performance objectives, and identifying corrective action
Recording statistical and quality management data in the organization’s measurement repository
page 12September 2009Supporting the CMMI Metrics Framework thru Level 5
• SPs Goal 1: Establish Performance Baselines and Models
Supporting OPP Goal 1 Supporting OPP Goal 1 Supporting OPP Goal 1
SP 1.4 Manage Project Performance (more in Goal 2)SP 1.4 Manage Project Performance (more in Goal 2)SP 1.1 Establish the Project’s ObjectivesSP 1.2 Compose the Defined ProcessSP 1.3 Select the Sub-processes that w/be Statis. Managed
SP 1.1 Establish the Project’s ObjectivesSP 1.2 Compose the Defined ProcessSP 1.3 Select the Sub-processes that w/be Statis. Managed
Organization
Project
ProjectQuantitative
Management Plan
ProjectQuantitative
Management Plan
Project Defined Process(PDP)
Project Defined Process(PDP)
Project Tracking Reports (Local
Repository)
Project Tracking Reports (Local
Repository)
Project MetricsAnalysis Report
Project MetricsAnalysis Report
ProjectQuantitative
Management Plan
ProjectQuantitative
Management PlanPerformance and
QualityGoals and Objectives
Performance and Quality
Goals and Objectives
OrganizationalQuantitative Management
Approach
OrganizationalQuantitative Management
Approach
page 14September 2009Supporting the CMMI Metrics Framework thru Level 5
SP 2.2 Apply Statistical Methods to Understand VariationSP 2.3 Monitor Performance of the Selected Sub-processesSP 2.4 Record Statistical Management Data
SP 2.2 Apply Statistical Methods to Understand VariationSP 2.3 Monitor Performance of the Selected Sub-processesSP 2.4 Record Statistical Management Data
SP 2.1 Select Measures and Analytic TechniquesSP 2.1 Select Measures and Analytic Techniques
• It addresses establishment of process-performance and quality objectives for the organization. These objectives will be used by projects to establish project-level process-performance and quality objectives, and determine the measurements that will be used to support these objectives. Basically contains :
√ Quality goals and process-performance objectives
Avg Review Effort per Work Product Review CompletedOrg Avg Review Effort per Work Product Review CompletedLinear (Avg Review Effort per Work Product Review Completed)
Client Satisfaction Survey 2006 Summary
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Within schedule Meetrequirements
Quality Prompt, clearand effective
communications
Opencommunication
Englishproficiency
Changes effectivelymanaged.
Issues/Riskseffectivelymanaged.
Overall projectmanagement.
Appropriate skills Adequate level ofknowledge
Target
n=34
• Some examples :
Productivity:
Hours per FP per platform
KLOC per resource per month per platform
Quality Assurance:
Average number of non-conformances per audit
Average days for non-conformances closing
Defects:
Defect density (#defects / size * 100)
Defect containment (#defeitos detected in peer reviews/adosem revisões/#total defects) by deliverable/phase.
Estimates:
Effort variation by project/Service Request by platform/language
Duration variation by project/Service Request by platform/language
OPP Baselines and Models
Customer Satisfaction Index
page 26September 2009Supporting the CMMI Metrics Framework thru Level 5
T ic k e ts f r o m 2 0 0 7I -M R C h a r t o f E f f o r t
page 27September 2009Supporting the CMMI Metrics Framework thru Level 5
Process Performance Baselines (PPBs) Process Performance Baselines (Process Performance Baselines (PPBsPPBs) )
Name Description Main Characteristics
Design
Defect Density
It shows the expected range of Defect Density (number of Defects by Pages of Documentation * 100) for the External Reviews.
- Design Phase- Health Care- Design Work Products- Minor Enhancement
Testing Productivity
It shows the range of expected Appl.CRproductivity when a Test Case is created.
- Produce Phase- Health Care, Transportation- Test Case- Minor Enhancement
Effort Variation
It shows the range of expected effort variation that would normally be achieved by a Application CR following the minor modification process with similar characteristics.
- Main Phases (Design, Produce and Testing).
- Health Care, Transportation
- Primary Language (Cobol, Cool:Gen, Java, Shell Script)
- Minor Enhancement
Ticket Effort Resolution
It shows the behaviourof the ticket resolutionprocess.
- On-going support
- Ticket category
- Effort ticket resolution
page 28September 2009Supporting the CMMI Metrics Framework thru Level 5
Process Performance Models (PPMs) Process Performance Models (Process Performance Models (PPMsPPMs) )
Planning(Assignment)
Design
Produce
Testing
Non Dev
Ticket
Dev
Ticket
Predictability
80706050403020100
40
30
20
10
0
-10
Code line Qty _avg
Codi
ng
S 6.09834R-Sq 30.6%R-Sq(adj) 24.6%
Regression95% C I95% PI
Fitted Line Plot - Metavance Defect Team - only SEs, all PCVsCoding = 10.86 - 0.4746 Code line Qty _avg
+ 0.007352 Code line Qty _avg**2
Apr/2007 to Apr/2008
Productivity QualityGAD QMSLife Cycle
0.90.80.70.60.50.40.30.20.10.0
4
3
2
1
0
% Reused Test Cases
Test
Cas
e Cr
eati
on P
rodu
ctiv
ity
S 0.514988R-Sq 66.5%R-Sq(adj) 65.4%
Regression95% CI95% PI
Fitted Line PlotTest Case Creation Productivity = 0.4834 + 2.622 % Reused Test Cases
Pilot and After Pilot
3.02.52.01.51.0
150
100
50
0
-50
Work Product Creator Business Skill
Exte
rnal
Def
ect
Den
sity
S 20.7319R-Sq 70.6%R-Sq(adj) 66.2%
Regression95% PI
Fitted Line PlotDefect Density = 848.4 - 1236 Business Skill
+ 589.2 Business Skill**2 - 91.02 Business Skill**3
3.02.52.01.51.0
150
100
50
0
-50
Work Product Creator Business Skill
Exte
rna
l Def
ect
Den
sity
S 20.7319R-Sq 70.6%R-Sq(adj) 66.2%
Regression95% PI
Fitted Line PlotDefect Density = 848.4 - 1236 Business Skill
+ 589.2 Business Skill**2 - 91.02 Business Skill**3
Standard Process
Standard Process
Standard Process
Test Case Creation Reuse
Library Reuse
Knowledge Repository
Training Guideline
Knowledge Base
Business Skill
Business Skill Business Skill
Business Skill
Business Skill
Business Skill
Cheklist Mentoring
Standard Process
Standard Process
Standard Process
Standard Process
page 29September 2009Supporting the CMMI Metrics Framework thru Level 5
The SimulationThe SimulationIn order to evaluate the alternatives a simulation was performed, evaluating the impact on External Defect Density and Cost. Cristal Ball, a Monte Carlo simulator, was the tool used.
Option 1 Mean Cost: US$ 2.312 ( +7.8%)Option 1 Mean Defect Density: 35.2 (-36%)
Current Mean Cost: US$ 2143Current Mean Defect Density: 55.6
Option 2 Mean Cost: US$ 2.174 ( +1.4%)Option 2 Mean Defect Density: 54.4 (-2,2%)
Option 3 Mean Cost: US$ 2.184 ( +1.9%)Option 3 Mean Defect Density: 55.6 (0%)
The ImprovementThe ImprovementA model was developed and deployed on Health Care projects. The Project Manager will use it to analyze the benefits of perform an internal review to retain defects.
ProjectAppl. CR ID
PhaseSystem Area
Work Product Design documentPOD 50
WP Creator Business Skill 1.30Internal Reviewer Business Skill 3.70
Average
al Defects without Internal Review
Max (UCL)Average
Min (LCL)
The Project Manager decides to perform an internal review considering the possibility of retain 74% of the forecasted defects.
Analysis Result
Regression EquationDefect Density = 848.4 - 1236 (WP Creator Business Skill) + 589.2 (WP Creator Business Skill)**2 - 91.02 (WP Creator Business Skill)**3
50
Forecasted Total of External Defects without Internal Review
19
Forecasted Total of External Defects with Internal Review
19
Organizational Process Performance ModelDefect Density by Business Skill
ABC1234
Design Application Change Billing
Project Data
Confidence Intervals for Forecasted Total of External Defects
0
2
4
6
8
10
12
14
16
18
20
Forecasted Total of External Defectswithout Internal Review
Forecasted Total of External Defects withInternal Review
Before Improv ement (Apr to Sep, 2007) P ilot (Sep to Nov , 2007) A fter Improvement (F eb to A pr, 2008)
645750433629221581
120
90
60
30
0
Work Product Reviewed
Exte
rnal
De
fect
Densi
ty
__MR=38.9 __
MR=27.3__MR=5.3
UC L=127.0
UC L=89.2
UC L=17.4
LC L=0 LCL=0 LC L=0
Before Improv ement (A pr to Sep, 2007) P ilot (Sep to Nov , 2007) A fter Improv ement (F eb to A pr, 2008)
11
111
I-MR Chart of External Defect Density
Mean from 18.6 to 3,2 defects by 100PODs
88%
page 30September 2009Supporting the CMMI Metrics Framework thru Level 5
Lessons LearnedLessons LearnedLessons Learned1. M&A/OPP/QPM will force you to create a structured process for Strategic
Business Planning. If you don´t have it will be very difficult to implement M&A/OPP/QPM practices
2. Have a long-term strategy to store measurement data, not only M&A but also SPC data.
3. Have an integrated set of tools supporting your Software Engineering projects, otherwise metrics collection will be a nightmare.
4. Reduce the effort, as much as you can, for metrics collection process, if possible, collect at the same time that the task is being performed.
5. Do specialize people on SPC, Six Sigma otherwise levels 4 and 5 will be a very hard journey. Consider to have statisticians around.
6. Excel will not be sufficient in many cases for QPM/OPP.
7. Do start with few performance indicators and then sophisticate and add additional ones, please remember, they must be aligned to companies´sgoals/objectives. 100 performance indicators will not allow you to take any conclusion.
8. Keep these indicators stable, as much as possible, so that you can analyze them periodically.
9. Do not individualize data at personal level, particularly on QPM/OPP analysis where sub-process analysis may lead you to see detailed data on people.
page 31September 2009Supporting the CMMI Metrics Framework thru Level 5