Www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 1 – Defect Analysis and Removal 1INFO631 Week 1.

www.ischool.drexel.edu

INFO 631 Prof. Glenn Booker

Week 1 – Defect Analysis and Removal

1INFO631 Week 1

www.ischool.drexel.eduINFO631 Week 1 2

Motivation

• Much of software measurement is devoted to improving the quality of the product

• To do so, it helps to understand how, when, and why mistakes are made during the software life cycle, resulting in defects

• It is assumed that defects are queued for being fixed (removed) after they are detected


Defect Analysis Overview

• Defect analysis looks at:– When a defect was created (“injected”)– When a defect was found (“detection”)– What caused a defect (type of defect, and/or

orthogonal defect classification)– How a defect was found (“triggers”)

• Yes, there are often many terms for the same activity!


Defect Creation (Injection)

• Defects may be created, detected, and removed during every phase of the software life cycle, including during maintenance– Most are created during requirements

analysis, design, and coding– Testing and maintenance create relatively

few defects


Waterfall Life CycleRequirements

Analysis

High LevelDesign

Low Level(Detailed)

Design

Coding

Unit TestingComponent(Integration)

Testing

SystemTesting

Maintenance


Defect Detection

• Defects may be found:– During quality assurance activities between

phases of the life cycle (such as during major reviews and inspections); and often

– Within each phase (during preliminary reviews, peer reviews and testing activities)

• Here we mostly consider the former (between-phase activities)


Defect Detection

• Defects are found (discovered) by– Reviews and formal inspections

• For requirements analysis, high level design, low level design, and coding

– Testing• Unit testing• Integration testing• System testing• Acceptance testing


Origin of Defects & Type

• What kind of defects are made in each life cycle phase?– Requirements - incorrect specification;

missing requirements– High level design - design does not cover all

requirements, or is inflexible– Low level design - mismatch between HLD

and LLD; design does not cover all requirements


Origin of Defects & Type

– Coding - code errors – Integration – interface, compatibility problems– Unit testing - bad fixes – Component testing - bad fixes– System testing - bad fixes– Acceptance testing - bad fixes– Maintenance (enhancement) - bad

requirements, design, coding, testing


Defect Causal Analysis

• Regardless of when defects are created or discovered, they may be analyzed to determine their cause– Causes may range from the mundane to the

esoteric– Once identified, causes should be recorded

to support defect prevention activities


Types of Defect Causes

• Some causes may include:– Unclear requirements– Incorrect architecture – Incomplete design– Unfamiliarity with programming language– Typos– Inadequate understanding of interfaces– Inadequate understanding of standards (e.g.

TCP/IP, ODBC, SQL, etc.)


Types of Defect Causes

• and:– Poor documentation for legacy system– Design based on outdated requirements– Code written to outdated design– Poor design for future expansion (e.g.

hardwired constants, buried assumptions, etc.)

– Conflicting requirements– Bad fix of a previous problem– …and many more


Orthogonal Defect Classification

• ODC is a formal type of defect causal analysis

• Classify defects by the type of defect, and the life cycle phase when it was created

• This particular scheme is still somewhat experimental, but the basic concept is widely accepted


ODC - Defect Type

• A possible set of definitions:– Function error–affects capability, user, product or

hardware interfaces, global data structures– Assignment error – errors with initialization of

control blocks or data structure– Interface error – errors in interacting with other

components, modules, or or device drivers– Checking – errors in validating data and values


ODC - Defect Type

– Timing/serialization – errors in management of shared and real-time resources

– Build/package/merge – errors due to mistakes in library systems, change management, or version control

– Documentation – errors in publications and maintenance notes

– Algorithm – errors regarding efficiency or correctness that affect the task


Orthogonal Defect Classification

Type Phase CausedFunctional DesignInterface Low Level Design

(LLD) Checking LLD or Coding Assignment CodingTiming LLDBuild/package Library toolsDocumentation PublicationsAlgorithm LLD


Defect Trigger

• Is a condition that allows a defect to surface (become visible); examples include:

• Design conformance• Logic or data flow incomplete• Workload or stress (performance)• Boundary conditions (extremes)• Bug fix• Recovery timing• User code


Defect Removal Modeling

• Defect removal is critical to reducing (development or maintenance) cycle time and cost, and improving quality

• Michael Fagan and Capers Jones are noteworthy authors

• See, for example, “Fagan Inspections”


Defect Removal Model (DRM)

• Software development phase-based DRM covers– Defect injection into a phase – Defect removal during a phase– Defect carryover across phases– Effectiveness of defect removal within a

phase


Defect Removal Model

• DRM is a quality management tool• Provides insights into the defect removal process

and where this process might be improved• Used after a project is completed; provides post

mortem information about that project• If defect removal process is similar for a new

project, then the DRM can be used to improve the defect removal process of that new project



• Defects are created (injected) by incorrect requirements gathering, analysis, design, coding, or by bad fixes– Any given defect can be removed in the

phase in which it was injected or in a later phase

• Remove defect means fix bug, redo specs and/or design as needed


Defect Removal Effectiveness

• Phase based defect detection activities– Requirements analysis & inspection– Design inspection– Code inspection, Build verification testing– Unit, string, integration, regression, system

testing– Fix verification inspection & testing



Applies to each life cycle phase

Life Cycle Phase

Defects Injected (new mistakes)

NetDefectsfrom previous Phase

Defect Detection

(inspection)

Known Defects

Undetected Defects

Fixed Defects

Unfixed Defects + Bad Fix Defects

Net Defects

Bad Fix Defects



• For any given phase of defect removal

• Defects present at removal operation– Defects found during removal operation +

defects found later– May be determined ex post facto (after the

fact), or using a statistical prediction model

Removal Effectiveness = Defects found by removal operation

Defects present at removal operation________________________________ X 100



• Defect removal effectiveness = (# of defects found by inspection) /(# of defects originally present) *100

• Early detection percentage = (# of major inspection errors) /(# of major and minor errors) * 100


Defect Matrix Assumptions

• Defects are removed in the same life cycle phase when they are found

• No defects are knowingly left unfixed• No bad fixes

– Or at least they are blended into the number of defects created in that life cycle phase


Sample Defect Matrix—When Originated vs. When Found

When Originated (injected, or created)When Found/Fixed

Rqmts Top Lev. Design

Low Lev Design

Coding Unit Testing

Compon Testing

System Testing

Field Total

Rqmts Top Lev Design 49 681 730 Low Lev Design 6 42 681 729 Coding 12 28 114 941 1095 Unit Testing 21 43 43 223 2 332 ComponTesting 20 41 61 261 4 387 SystemTesting 6 8 24 72 1 111 Field 8 16 16 40 1 81 Total 122 859 939 1537 2 4 1 3465

1

0 0



Life cycle Phase(s)

Defects passed from Previous life cycle phases

Defects Created this Phase

Defects Found & removed this Phase

Defects passed to Next life cycle phase

Next = (Previous + Created) – FoundDRE = Found / (Previous + Created) * 100


High Level Design Effectiveness

• There are no requirements defects removed; 122 defects are passed to HLD

• High (Top) Level Design (I0) Inspection Effectiveness– Defects found and removed at I0: 730– Defects existing on step entry (escapes from

requirements phase: 122– Defects injected in current phase: 859– E(I0) = 730/(122+859) x 100 = 74%


Low Level Design Effectiveness

• Low Level Design (I1) Inspection Effectiveness– Defects found and removed at I1: 729– Defects existing on step entry (escapes from

requirements phase and I0): 122+859-730 = 251

– Defects injected in current phase: 939– E(I1) = 729/(251+939) x 100 = 61%


Code Inspection Effectiveness

• Code Inspection (I2) Effectiveness– Defects found and removed at I2: 1095– Defects present on step entry (escapes from

requirements phase, I0, and I1): 251+939-729 = 461

– Defects injected in current phase: 1537– E(I2)= 1095/(461+1537) x 100 = 55%


Unit Testing Effectiveness

• Unit Testing (UT) Effectiveness– Defects found and removed at UT: 332– Defects existing on step entry (escapes from

previous phases): 461+1537-1095 = 903– Defects injected in current phase (bad fixes):

2– E(UT) = 332/(903+2) x 100 = 37%

• Can follow the same pattern for the other testing phases and post-release


Summary Effectiveness Measures

• Overall Design & Coding Inspection Effectiveness– IE = (730+729+1095)/(122+859+939+1537) x 100 =

74%• Overall Effectiveness of all Testing activities

– TE = (332+387+111)/(903+2+4+1)x100 = 91%• Overall Defect Removal Effectiveness of the

development process (not including release)– DRE = (0+730+729+1095+332+387+111)/(122+859+

939+1537+2+4+1) x 100 = 3384/3464*100 = 97.7%


Rayleigh Model

• The Rayleigh Model describes the number of defects which will be discovered, by development phase

• It’s a special case of the Weibull family of distributions, which we’ll cover later


Rayleigh Model Assumptions

• Defect rate observed during development process is positively correlated with defect rate in field

• Given the same defect injection rate, if more defects are discovered and removed earlier, fewer will remain in later stages, leading to fewer defects in the field


Rayleigh Model Inspections

• I0 = High Level Design Inspection• I1 = Low Level Design Inspection• I2 = Code Inspection• UT = Unit Testing • CT = Component (Integration) Testing• ST = System Testing• GA = after General Availability (release or

Fielding of the system)


Rayleigh Model

Number of Defects

Development PhaseI0 I1 I2 UT CT ST GA

Each bar represents the number of defects found during that life cycle phase’s inspection activity


Desired Rayleigh Curve TrendsEffect of Early Defect Removal and Reducing Error Injection


Cost of Defect Removal by Phase

• Defect removal and rework is less costly the closer that the defects are found relative to the phase in which they are injected– Rework in the I0, I1, and I2 inspection levels can be

10 to 100 times less expensive than if it is done during formal testing

– Reviews can reduce number of defects reaching testing phases by factor of 10• These reductions cut testing costs by 50-80%, even

including the review costs


Cost Effectiveness of DRM

• Relative cost of fixing a problem found in design/coding, testing, or after release are: 1:20:82 (Remus, 1983) 1:13:92 (Kan, 1989)

• Cost of defect removal can be analyzed by inspection type, testing phase, defect severity, defect origin, etc.

Www.ischool.drexel.edu INFO 631 Prof. Glenn Booker Week 1 – Defect Analysis and Removal 1INFO631 Week 1.

Documents

defect type of defect

testing activities

defect detection defects

types of defect

defect causal analysis

defect prevention activities

lld design

kind of defects