Weighted Defect Removal Effectiveness: Method and Value

IBM Rational Software Conference 2009

Overview

Asking quality questions

Defect removal efficiency (DRE, CDRE)

Weighted DRE (DREw, CDREw) Demo

Answering quality questions

1

IBM Rational Software Conference 2009

Asking Quality Questions

How good was our testing?

How good is our software?

2

IBM Rational Software Conference 2009

Cumulative Defect Removal Efficiency (Simple Method)

“Cumulative defect removal efficiency =

defects found before release

defects found before and after release

By this formula, if 100 defects are found in a program during its

entire life -- in both development and in production -- and 90 of the

defects are found before release, then the cumulative defect removal

efficiency is considered to be 90 percent.”

-- T.C. Jones, IBM Systems Journal, 1978

IBM Rational Software Conference 2009

Work-In-Process Defect Removal Efficiency

defects found prior test

defects found prior and current test

How good was my testing?WIP DRE is retrospective.

IBM Rational Software Conference 2009

A DRE by Any Other Name

Defect Removal Effectiveness

Defect Fix Percentage

Defect Detection Effectiveness

Defect Detection Percentage

Defect Detection Rate

5

IBM Rational Software Conference 2009

What are Actual DREs?

6

-- data from table by Capers Jones, CrossTalk, 2008

IBM Rational Software Conference 2009

What are Actual CDREs?

7

< 80% 80-85% 85-90% 90-95% 95-99% >99%

-- based on Capers Jones data published 2008 by ITMPI

IBM Rational Software Conference 2009

Jones’ Simplifying Assumptions

All detection methods are equivalent

All fixes are good and singular

All defect causes are equivalent *

All defects are equivalent

8

-- T.C. Jones, IBM Systems Journal, 1978

IBM Rational Software Conference 2009

Defect Detection

CRITICAL XMAJOR XMINORCOSMETICINCONSEQUENTIAL

Development

CRITICALMAJORMINOR XCOSMETIC XINCONSEQUENTIAL  

IBM Rational Software Conference 2009

Work In Process CalculationsWIP DRE 67%

CRITICAL X  MAJOR X  MINOR XCOSMETIC  INCONSEQUENTIAL  

Development Acceptance

CRITICAL  MAJOR XMINOR XCOSMETIC X  INCONSEQUENTIAL    

WIP DRE 67%

IBM Rational Software Conference 2009

WIP DRE becomes DREWIP DRE 50% WIP DRE 50%

CRITICAL X    MAJOR X    MINOR X  COSMETIC   XINCONSEQUENTIAL    

Development Acceptance Production

CRITICAL   XMAJOR XMINOR XCOSMETIC X    INCONSEQUENTIAL      

WIP DRE 50% WIP DRE 50%

With Production counts, WIP DRE becomes DRE

IBM Rational Software Conference 2009

Cumulative Defect Removal Efficiency (CDRE)DRE 50% DRE 50%

CRITICAL X    

 CDRE 75% 

MAJOR X  

MINOR XCOSMETIC   XINCONSEQUENTIAL    

Development Acceptance Production

CRITICAL   XMAJOR X

CDRE 75% MINOR XCOSMETIC X  INCONSEQUENTIAL      

DRE 50% DRE 50%

IBM Rational Software Conference 2009

Are these test results equivalent ?????

CRITICAL X    MAJOR X    MINOR X  COSMETIC XINCONSEQUENTIAL    

Development Acceptance Production

CRITICAL XMAJOR XMINOR XCOSMETIC X    INCONSEQUENTIAL      

IBM Rational Software Conference 2009

Severity Weighting

“Obviously, it is important to measure defect severity levels as well as recording numbers of defects.” -- T. Capers Jones, 2008

15

IBM Rational Software Conference 2009

Weighted Defect Removal Effectiveness (DREw)

Critical x 5Major x 4 Minor x 3

 Cosmetic x 2 Inconsequential x 1

Keep It Simple!

(or use quantified potential business impact)

IBM Rational Software Conference 2009

Weighted Defect Removal Effectiveness (DREw)DREw 75%

CRITICAL 5  MAJOR 4  MINOR 3COSMETIC  INCONSEQUENTIAL  

9/12

5/9

CRITICAL  MAJOR 4MINOR 3COSMETIC 2  INCONSEQUENTIAL    

DREw 56%

IBM Rational Software Conference 2009

Weighted Defect Removal Effectiveness (DREw)DREw 60%

CRITICAL     

MAJOR  MINOR 3COSMETIC   2INCONSEQUENTIAL    

3/5

4/9

CRITICAL   5MAJOR 4

 MINORCOSMETIC  INCONSEQUENTIAL    

DREw 44%

IBM Rational Software Conference 2009

Cumulative DREw (CDREw)

CRITICAL 5    

 CDREw 86% 

MAJOR 4  

MINOR 3COSMETIC   2INCONSEQUENTIAL    

12/14

9/14

CRITICAL   5MAJOR 4

CDREw 64% MINOR 3COSMETIC 2  INCONSEQUENTIAL      

IBM Rational Software Conference 2009

Why Measure Work-In-Process Testing?

Consistent WIP DRE lends predictive value for product reliability from a stable process

Consistent (WIP) DREw lends predictive value for product releasability from a stable process

20

IBM Rational Software Conference 2009

Answering Quality QuestionsDev Int QA Alpha Beta Prod

CriticalMajorMinorCosmeticWeightedTotal

How good was

our testing?

How good is

our software?

21

How good was our testing?

How good is our software?