Chapter09-Defect Classification and Analysis-SQM@NNTu

QuảnQuản lýlý chấtchất lượnglượng phầnphần mềmmềmSoftware Quality ManagementSoftware Quality Management

GV: Nguyễn Ngọc TúGV: Nguyễn Ngọc Tú

Email: [email protected]: [email protected]

Are we building the right software for the need ?

Are we building the software right ?

DEFECT CLASSIFICATION AND DEFECT CLASSIFICATION AND ANALYSIS ANALYSIS SQSSQS –– SQMSQM –– SQASQA –– SRESRE

Chapter 09

Hoa Sen University 2NNTu - SQM W2009 HoaSen University

“The software is done.We are just trying to get it to work”

Unknown

2

AIMSAIMS

� (actual/potential) defect� or quality� in

current and future products.

HoaSen University 3NNTu - SQM W2009

ContentsContents

� General Types of Defect Analyses

– Defect distribution analysis

� What: Distribution over defect types

� Where: Distribution over defect locations

� General observations about defect distribution

– Defect trend analysis and defect dynamics model

– Defect causal analysis

� DEFECT CLASSIFICATION AND ODC

– ODC concepts

– Defect classification using ODC: A comprehensive example

– Adapting ODC to analyze web errors

� DEFECT ANALYSIS FOR CLASSIFIED DATA

– One-way analysis: Analyzing a single defect attribute

– Two-way and multi-way analysis: Examining cross-interactions


IntroductionIntroduction

� Defect CA

– help both developers and testers to detect and remove potential defects

– help other project personnel to improve the development process, to prevent injection of similar defects and to manage risk better

� defect data

– from the main QA activities


General Types of Defect General Types of Defect AnalysesAnalyses

� A defect is discovered,

– various individual analyses can be performed.

� defect data are accumulated over time

– collective analyses can be performed

� General defect analyses:

– Questions: what/where/when/how/why?

– Distribution/trend/causal analyses.


General Types of Defect General Types of Defect Analyses Analyses [2][2]

� Analyses of classified defect data:

– Prior: defect classification.

– Use of historical baselines.

– Attribute focusing in 1-way and 2-way analyses.

– Tree-based defect analysis



� What– The identification and classification of the discovered defects can be

performed to identify what they are and classify them by some

consistent scheme

� Where– Where was the defect found or discovered? This information can be

used to provide valuable feedback to the development process through

defect distribution analysis.

� When– The identification of the exact time or associated development phase or

subphase when a defect is injected and when it is discovered is

important, because it provides information to analyze the overall defect

trend and serves as the basis for quality prediction into the future.



� Pre- or Post release– An important extension to the “when” question is whether a defect is a

pre-release defect or a post-release defect

� How and Why– How was the defect injected into the software, and why? These two

questions are closely related, both pertaining to the cause of the

discovered defects


Defect distribution analysisDefect distribution analysis

� can help us answer the what and where

questions– distribution of defects over different defect types

– can find out the distribution of defects over different areas or product

components

� typically deal with faults or defect fixes

instead of failures or errors

� Defect fixes– is typically used if actual fixing of discovered problem took place before

defect analyses were performed


Defect distribution analysisDefect distribution analysis

Type Description #failures

A permission denied 2079

B no such file or directory 14

C stale NFS file handle 4

D client denied by server configuration 2

E file does not exist 28,631

F invalid method in request 0

G invalid URL in request connection 1

H mod_mime_magic 1

I request failed 1

J script not found or unable to start 27

K connection reset by peer 0

All types 30,760


http://lyle.smu.edu/~tian/class/8317.09f/webM.pdf

What: Distribution over defect What: Distribution over defect typestypes

Type Errors %

.gif 12489 43.62

.class 4913 17.16

directory 4425 15.46

.html 3656 12.77

.jpg 1323 4.62

other 394 1.38

All 28631 100


Table 20.2 Characterizing web errors by file types

Where: Distribution over Where: Distribution over defect locationsdefect locations

DF= 0 1 2 3 4 5 6 7 8 9 10-19 20-

37

all

module

#

771 174 102 63 31 29 23 25 16 7 50 14 1295

% 58.8 13.4 7.9 4.9 2.4 2.2 1.8 1.9 1.2 0.5 3.9 1.1 100

DFsum 0 174 204 189 124 145 138 175 128 63 673 417 2367

% 0 7.4 8.6 8 5.2 6.1 5.8 7.4 5 2.7 28.4 17.

6

100


Table 20.3 Distribution of DF for a commercial product LS

General observations about General observations about defect distributiondefect distribution

DF= 0 1 2 3 4 5 6 7 8 9 10-19 20-49 >50 all

modul

e#

23 13

1

11

2

12

0

99 94 68 50 38 32 147 68 13 995

% 2.3 13.

2

11 12 9.9 9.4 6.8 5 3.8 3.2 14.8 6.8 1.3 100

DFsum 0 13

1

22

4

36

0

39

6

47

0

40

8

35

0

30

4

28

8

2109 2040 910 7824

% 1.6

7

2.8

6

4.6 5.1 6 5.2 4.5 3.9 3.7 3.1 26.96 26.07 11.6

3

100


Defect trend analysis and Defect trend analysis and defect dynamics modeldefect dynamics model

� defect data contains some timing

information

– Minimum: pre-release or post-release


Injection

phase

Removal Phase

req. spec. design coding testing post-rel all

phases

requirement 10 22 8 0 5 2 47

specification 10 20 2 0 1 33

design 52 120 32 5 209

coding 198 320 46 564

testing 58 7 65

post-release 2 2

all phases 10 32 80 320 415 63 920

Defect causal analysisDefect causal analysis

� two forms

– logical analysis

� is a deterministic analysis that examines the logical

link between the effects and the corresponding

causes, and establishes general causal relations

– statistical analysis

� is a probabilistic analysis that examines the

statistical link between causes and effects and

deduces the probable causal relations between the

two


Defect causal Defect causal analysis analysis [2][2]

� The effects

– the observed failures or discovered (or fixed) faults

� the corresponding causes

– are the faults that caused the failures or the errors

that caused the injection of the faults, respectively.

� causal relations

– are determined by the developers or code owners

– faults are typically determined through dedicated

defect causal analysis



� Root cause analysis

– is human intensive, and should be performed by experts with thorough knowledge about the product, the development process, the application domain, and the general environment

� Gilb inspection

� for all the critical defects



� Statistical analysis

– is based on empirical evidence collected either locally or from other similar projects

– can be fed to various models to establish the predictive relations between causes and effects

� employ various statistical models

– correlation analysis

– Eg. the number of defects per module may be closely

correlated to module control flow complexity


DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC

� various detailed information can be collected

and recorded regarding the problems or the

defects

� part of this information

– is usually derived from explicit or implicit root cause analysis

� Such information

– can be organized in a systematic way for further analyses

� more valuable and specific feedback: use statistical models


DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC [2][2]

� The systematic classification and analysis

of defect data bridge the gap between

causal analysis and statistical quality

control, and provide valuable in-process

feedback to the development or

maintenance process and help assure and

improve product quality


DEFECT CLASSIFICATION DEFECT CLASSIFICATION AND AND ODCODC [3][3]

� Orthogonal defect classification, or ODC,

developed initially at IBM (Chillarege et al.,

1992)

– is the most influential among general frameworks for software defect classification and analysis.

� to identify problematic areas, and to

improve overall software product quality


ODCODC

� Key elements of ODC– Aim: tracking/analysis/improve

– Approach: classification and analysis

– Key attributes of defects

– Views: both failure and fault

– Applicability: inspection and testing

– Analysis: attribute focusing

– Need for historical data


ODCODC conceptsconcepts

� ODC has a rich and extensive category of

defect attributes, stemming from both the

failure view and the fault view

� The attributes (8) – related to the former are typically completed by

software testers or inspectors who initially observed

problems and opened defect reports; while those

related to the latter are typically completed by the

software developers or system maintenance

personnel who fixed the reported problems and

updated the corresponding defect reports


ODCODC

� 8 attributes– Activity

� refers to the actual process step (code inspection, function

test, etc.) when defects are discovered.

– Trigger

� describes the environment or condition that had to exist to

expose the defects.

– Impact

� refers to either perceived or real impact on users.

– Target

� represents the high-level identity (i.e., design, code, ID, etc.)

of the entity that was fixed.


http://www.ibm.com/developerworks/rational/library/aug06/gu/index.htmlODC v5.11, IBM Center for Software Engineering

ODCODC

� 8 attributes– Type

� represents the nature of the actual correction that was made.

– Qualifier

� specifies whether the fix that was made was due to missing,

incorrect, or extraneous code or information.

– Source

� indicates whether the defect was found in code written in-

house, reused from a library, ported from one platform to

another, or outsourced to a vendor.

– Age

� identifies the history of the target (i.e., design, code, ID, etc.)

that had the defect.


http://www.ibm.com/developerworks/rational/library/aug06/gu/index.html


� failure view and information collected at

defect discovery – Defect impact,

� with attribute values covering functionality, reliability, etc.

– Defect trigger,

� with attribute values corresponding to the specific types of

testing

– Defect severity,

� with commonly used attribute values: critical, major, minor, or

or inspection activities or scenarios that triggered the defect

detection. some numerical scale.



� fault view collected at defect fixing– Defect type,

� with attribute values: function, interface, algorithm, timing,

etc.

– Number of lines changed for the fixing.

� Some additional causal analyses – Defect source,

� with attribute values: vendor code, new code, base code, etc.

– Where the defect was injected,

� located to subsystems, modules, or components.

– When the defect was injected,

� typically identified with the development phase.


Defect classification using Defect classification using ODCODC: : A comprehensive exampleA comprehensive example

� various defect attribute data according to

ODC were collected in the system testing

stage

� a defect is detected,

– a formal report (called Problem Tracking Report or PTR in IBM) is recorded and tracked until its final resolution


Defect classification using Defect classification using ODCODC: : A comprehensive A comprehensive example example [2][2]

Label Name Possible Values or Categories & Labels

imp impact

c=capability, im=implementation, in=installation, ma=maintenance,

mi=migration, p=performance, r=reliability, sec=security, ser=service,

std=standard, u=usability

t r i g trigger

i=installation, m=migration, s=stress, b=backup, c=communications, f=file

ilo, co=coexistence, e=exception, hc=hlw config., sc=slw config., a=ad-hoc,

ss=startup/shutdown, o=normal operation

sev severity range from 1 (highest) to 4 (lowest) in severity

wk week week detected, counted from the start of the project

ftypefix

type

o=other product, s=specification, hld=high-level design,

lld=low-level design, c=code, b=build process

act action a=add, d=delete, c=change

srccode

source

b=base, v=vendor, n=new, c=changed, i=incremental (added to old),

s=scaffolded, p=previous defect fix

injphase

injected

p=previous release, s=specification, hld=high-level design, Ild=low-level

design, c=coding, ut=unit test, ft=function test, st=system test, d=customer

usage



� Defect impact – “If this defect is not fixed, how will it impact the customer?”

– Pre-defined impact categories (possible answers) include

performance, reliability, etc.

� Defect trigger categories – closely resemble test scenario classes used for managing the

testing process for this product.

� Defect severity – can be 1 (critical problem), 2 (major problem), 3 (minor problem),

and 4 (minor inconvenience).

� The week – when the defect is detected, counted from the start of the project.



� The information collected at defect fixing – pertains to the actions taken by the developers to

locate, identify and correct the faults that caused

detected failures:

� Fix type: fix to design, code, etc.

� Number of lines changed for the fixing.

� Fix action: adding, deleting, or changing to design or code.

� Some simple causal analyses– Defect source: vendor code, new code, base code, etc.

– The development phase when the defect was injected: previous

release or waterfall like development phases in the current

release.


Adapting Adapting ODCODC to analyze web to analyze web errorserrors

� For web-based applications,

– ODC-like defect classification can be defined and relevant defect data can be extracted from existing web server logs for analysis (Ma and Tian, 2003)

� data collection is always a big hurdle that

requires developers and testers to devote

substantial time to analyze the defects and

report the findings


Adapting Adapting ODCODC to analyze web to analyze web errors errors [2][2]

� Attributes

– Defect impact� corresponds to web error type, which indicates what problem was

experienced by web users. It can be analyzed directly based on

information extracted from the error logs or from response code used in

web access logs (Kallepalli and Tian, 2001).

– Defect trigger� corresponds to specific usage sequences or referrals that lead to

problems recorded in the error logs. It can be analyzed by examining the

referral pair information that can be extracted from the access logs (Ma

and Tian, 2003).

– Defect source� corresponds to specific files or file types that need to be changed,

added, or removed to fix problems recorded in the error logs. It can be

analyzed by examining both the specific errors and referral pairs.


Adapting Adapting ODCODC to analyze web to analyze web errors errors [3][3]

� Various other attributes

– can also be adopted or adapted from the original

ODC attributes through a close examination of the

web environment and data availability. Such

adaptation to different environments can help

people analyze problems or issues of concern to

them and fulfill different purposes.


DEFECT ANALYSIS FOR DEFECT ANALYSIS FOR CLASSIFIED DATACLASSIFIED DATA

� Various techniques

� most obvious and most straightforward analyses– are to apply defect distribution and trend analyses

� one-way analysis– examines one attribute at a time, either its overall distribution or its trend

over time

� Two-way analysis – can be used to examine the crossinteraction of two attributes (Bhandari

et al., 1993).

� Higher-order analysis is also possible, – such as using tree-based modeling on all the ODC attributes (Tian and

Henshaw, 1994).


expected defect profile

OneOne--way analysis: Analyzing a way analysis: Analyzing a single defect attributesingle defect attribute

� For each defect attribute,

– the overall distribution of its values can beexamined.

– distribution of defects among the differentdefect impact areas is very non-homogeneous

� When similar distribution data are available

over time or different development phases,

– can trace them to perform defect trend analysis






Figure 20.2 gives us the trend of type E errors for the SMU/SEAS web site over 26 days

TwoTwo--way and multiway and multi--way analysis: way analysis: Examining crossExamining cross--interactionsinteractions

� analysis examines the interaction between

two attributes, and can be applied to all

the attributes in pair-wise fashion

� simplest form

– is the conditional analysis of an individual

attribute under the condition of another attribute

taking a specific value.


TwoTwo--way and multiway and multi--way analysis: way analysis: Examining Examining crosscross--interactions interactions [2][2]

Impact Severity

1 2 3 4

Capability 2 12 13 1

Documentation 0 1 14 10

Installability 0 6 6 4

Maintainability 0 6 19 7

Migration 0 0 0 1

Performance 1 1 3 0

Reliability 27 96 66 7

Security 1 3 3 0

Service 0 0 4 4

Standards 0 1 2 1

Usability 0 10 44 19


ReferenceReference

� [1]Chapter 20

� [1]Chapter 13

� [3]Chapter 5

� [4]Chapter 04

� ODC - http://www.research.ibm.com/softeng/ODC/DETODC.HTM IBM


Q/AQ/A


Chapter09-Defect Classification and Analysis-SQM@NNTu

Documents

single defect

analyze web

defect causal

defect trend

defect data

defect severity

error logs

defect types