Weighted Defect Removal Effectiveness: Method and Value

IBM Rational Software Conference 2009

Overview

Asking quality questions

Defect removal efficiency (DRE, CDRE)

Weighted DRE (DREw, CDREw) Demo

Answering quality questions

1


Asking Quality Questions

How good was our testing?

How good is our software?

2


Cumulative Defect Removal Efficiency (Simple Method)

“Cumulative defect removal efficiency =

defects found before release

defects found before and after release

By this formula, if 100 defects are found in a program during its

entire life -- in both development and in production -- and 90 of the

defects are found before release, then the cumulative defect removal

efficiency is considered to be 90 percent.”

-- T.C. Jones, IBM Systems Journal, 1978


Work-In-Process Defect Removal Efficiency

defects found prior test

defects found prior and current test

How good was my testing?WIP DRE is retrospective.


A DRE by Any Other Name

Defect Removal Effectiveness

Defect Fix Percentage

Defect Detection Effectiveness

Defect Detection Percentage

Defect Detection Rate

5


What are Actual DREs?

6

-- data from table by Capers Jones, CrossTalk, 2008


What are Actual CDREs?

7

< 80% 80-85% 85-90% 90-95% 95-99% >99%

-- based on Capers Jones data published 2008 by ITMPI


Jones’ Simplifying Assumptions

All detection methods are equivalent

All fixes are good and singular

All defect causes are equivalent *

All defects are equivalent

8

-- T.C. Jones, IBM Systems Journal, 1978


Defect Detection

CRITICAL XMAJOR XMINORCOSMETICINCONSEQUENTIAL

Development

CRITICALMAJORMINOR XCOSMETIC XINCONSEQUENTIAL


Work In Process CalculationsWIP DRE 67%

CRITICAL X MAJOR X MINOR XCOSMETIC INCONSEQUENTIAL

Development Acceptance

CRITICAL MAJOR XMINOR XCOSMETIC X INCONSEQUENTIAL

WIP DRE 67%


WIP DRE becomes DREWIP DRE 50% WIP DRE 50%

CRITICAL X MAJOR X MINOR X COSMETIC XINCONSEQUENTIAL

Development Acceptance Production

CRITICAL XMAJOR XMINOR XCOSMETIC X INCONSEQUENTIAL

WIP DRE 50% WIP DRE 50%

With Production counts, WIP DRE becomes DRE


Cumulative Defect Removal Efficiency (CDRE)DRE 50% DRE 50%

CRITICAL X

CDRE 75%

MAJOR X

MINOR XCOSMETIC XINCONSEQUENTIAL


CRITICAL XMAJOR X

CDRE 75% MINOR XCOSMETIC X INCONSEQUENTIAL

DRE 50% DRE 50%


Are these test results equivalent ?????

CRITICAL X MAJOR X MINOR X COSMETIC XINCONSEQUENTIAL


CRITICAL XMAJOR XMINOR XCOSMETIC X INCONSEQUENTIAL


Severity Weighting

“Obviously, it is important to measure defect severity levels as well as recording numbers of defects.” -- T. Capers Jones, 2008

15


Weighted Defect Removal Effectiveness (DREw)

Critical x 5Major x 4 Minor x 3

Cosmetic x 2 Inconsequential x 1

Keep It Simple!

(or use quantified potential business impact)


Weighted Defect Removal Effectiveness (DREw)DREw 75%

CRITICAL 5 MAJOR 4 MINOR 3COSMETIC INCONSEQUENTIAL

9/12

5/9

CRITICAL MAJOR 4MINOR 3COSMETIC 2 INCONSEQUENTIAL

DREw 56%


Weighted Defect Removal Effectiveness (DREw)DREw 60%

CRITICAL

MAJOR MINOR 3COSMETIC 2INCONSEQUENTIAL

3/5

4/9

CRITICAL 5MAJOR 4

MINORCOSMETIC INCONSEQUENTIAL

DREw 44%


Cumulative DREw (CDREw)

CRITICAL 5

CDREw 86%

MAJOR 4

MINOR 3COSMETIC 2INCONSEQUENTIAL

12/14

9/14

CRITICAL 5MAJOR 4

CDREw 64% MINOR 3COSMETIC 2 INCONSEQUENTIAL


Why Measure Work-In-Process Testing?

Consistent WIP DRE lends predictive value for product reliability from a stable process

Consistent (WIP) DREw lends predictive value for product releasability from a stable process

20


Answering Quality QuestionsDev Int QA Alpha Beta Prod

CriticalMajorMinorCosmeticWeightedTotal

How good was

our testing?

How good is

our software?

21

How good was our testing?

How good is our software?

Weighted Defect Removal Effectiveness: Method and Value

Technology

dre wip dre

critical x majorxcdre

consistent wip dre

critical major minor3

defect causes

ibm systems journal

process calculations

cosmetic inconsequential912