Defect Prevention Training

Protection notice / Copyright notice

Defect Prevention Training Induction – Sep 2007

Version 2.0

Page 2 Sep-07Protection notice / Copyright notice

For Internal Use OnlyP&Q

Introduction

Defect Prevention is a process of improving quality and productivity by preventing the injection of defects into a software work product.

Definition: “…an activity of continuous institutionalized learning during which common causes of errors in work products are systematically identified and process changes eliminating those causes are made.” [Eickelmann]

SEI has identified ‘Causal Analysis and Resolution’ as Level 5 PA of CMMI



Objectives

Course ObjectiveTo enable participants understand and apply defect prevention concepts

Defect Prevention Objectives

Identify and analyze the causes of defects& Reduction in number of defect categories

Reduction in the extent of defect escape between phases

Reduction in frequency of common defects

Improvement in PCB values



Contents

Defects and Bugs (Examples)Origin of DefectsClassification of DefectsDefect ManagementDefect DetectionDefect Prevention Cycle



Objectives of Defect Prevention

• Establish practice of Root Cause Analysis within projects for Analysis

of Identified Defects

• Identify critical processes as part of root cause analysis

• Set goals for improving critical process (shift mean and narrow

variation)

• Reduce most frequent type of defects such as “ not following coding

guidelines”

• Analyze opportunities for improvement by conducting escape analysis.

• Use defect distribution data to drive process improvement activities

• Spread lessons learnt - Team Meetings, SEPG, Process Database



Responsibility

Project team is responsible for the DP activities pertaining to

the project life cycle activities &

Project Manager (at project level)

Project Quality Manager (at project level)

P&Q (at Org level)

SEPG (at Org level)



Defects and BugsDefects and Bugs



Defects and Bugs - Example

The Atlas-Agena spacecraft, destined for Venus, had to be blown up during launch because it became unstable about 90 miles up. (Malfunctioning rockets have to be destroyed to avoid crashes in populated areas). A missing hyphen in the flight plan resulted in the loss of the $18.5 million US spacecraft.



Defects and Bugs - Example

On January 15, 1990, 114 switching computers in the AT&T telephone network crashed because of a software flaw. 65 million subscribers were unable to use their phones. The problem arose when a switching computer in New York crashed, sending out a digital “out of service” message to nearby computers. Normally, other switches would route traffic around the disabled computer. However, a misplaced “break” in a C statement caused the nearby computers to go down as well. For the next 9 hours, the switches went down, rebooted themselves, and came back up, only to go back down immediately.



Effect of Software Error

•unreasonable added cost•lost time and effort•inconvenience and annoyance•death



Could these incidences of software errors been prevented?

YES!



Product and Process Defects

PRODUCT PROCESS

Definition

Types of Defects

Strategy for Handling Defect

Artifacts created during the life cycle of the project.

Complete set of activities needed to transform user requirements to a product.

Process Defects are related to tasks/activities: Non-adherence to standards Poor Documentation Schedule overrunTraining related

Product Defect are related to requirements : Functional and Non-Functional

Defect PreventionDefect Removal/Elimination

Product defect is always a result of Process Defect

Process defect is like a potential carrier of disease



Stages of a Software Cycle

ReviewsReviewsWhite BoxBlack Box

Stress/Load

White BoxBlack Box

Stress/Load

ProblemReports/CRs

ProblemReports/CRs

Defect Prevention - Feedback and Process adjustmentsDefect Analysis and Process Improvement

Defect Prevention - Feedback and Process adjustmentsDefect Analysis and Process Improvement

RequirementsRequirements DesignDesign CodingCoding TestingTesting MaintenanceMaintenanceProposalProposal



Origin of DefectsOrigin of Defects



Human Errors

Types of Errors

Omission

Ignorance

Commission

Typography

Knowledge

Information

External

More than 80% of software errors are human



Translation Errors

A requirement is often stated in terms of a solution

Focus on solution may hide the real requirement

The mismatch between the solution desired and the real requirement leads to translation errors

NEEDED TOLD URS DESIGN

TRANSLATION ERRORS

BUILD

Detected Bugs

Hidden Bugs



Design Errors

Errors that affect data integrityErrors that alter correctly stored dataIncorrect algorithm used to compute a value

Types of Errors

Some examples:

•Does each module in the system design exist in detailed design?•Are all assumptions explicitly stated? Are they acceptable?•Have the exceptional conditions been handled?•Are all data formats consistent with the system design?•Are the loop termination conditions properly specified?

Checklist

Mitigation of design

errors



Coding Errors

Exception handling

Incorrect Algorithm

Missing Functionality

Language pitfalls

Memory release

Omitted program sections

A programming error alters a program’s ability, in a negative sense, to completely and effectively meet the user’s requirement.



Testing Errors

Failure to notice a problem

Misreading the screen

Failure to execute a planned test

Failure to use the most ‘promising’ test cases.

Ignoring programmers suggestions

Corrupt data file used

Incorrect test cases

Concentration on trivial

Failure to report



0

20

40

60

80

100

120

140

RS Revi

ew

FS Rev

iew

Desig

n Rev

iew

Code Rev

Testin

g

Proble

m R

eport

Ideal

Acceptable

Costly

Disaster

Defect Detection as early as possible

High proportion defect reported by customer - Unacceptable

More defect detected in testing – Quality at High cost

Maximize defect detection during reviews- Quality at right

cost

Objectives of Defect Prevention



So where should the focus be?

On Proactive Defect Prevention

On Early Defect Detection

On Usage of Past Experience



Defect PreventionDefect Prevention



Defect Management Rules

•Fixing downstream is costly…

•Rework eats away resources...

•Pareto Rule - About 80% of the available rework comes from 20% of the defects

•Another Pareto Rule - About 80% of the defects come from 20% of the modules (and about half the modules are defect free)

•Peer Reviews catch 60% of the defects

•Perspective-based reviews catch 35% more defects than non-directed reviews (use of checklists)

•Disciplined personal practices can reduce defect introduction rates by up to 75%

•About 40-50% of user programs enter with trivial defects

Barry BoehmVictor Basili



Activities Performed during DP – Organization Level

• Defect Prevention Plan is prepared at SBU-level (by TC/Business Partner) – contains lessons learnt and improvement actions

• SPI (Software Process Improvement) Plan is an overall Plan for the Organization and bears reference to DP Plan as well

• DP Plan is shared with SEPG members and PQMs

• Lesson’s Learnt from SBU is disseminated through SEPG

• Improvement Action are tracked and DP Plan is updated accordingly



Activities Performed during DP – Project Level

•Kick-off/Start PES meeting – •Goal setting, • Identification of critical processes• Incorporation of Lessons Learnt from previous projects as preventive measures• Phase-wise Defect Distribution goal setting

• Defect reporting – Reviews & Testing

• Root Cause Analysis

• Action Implementation

• Information Dissemination



Causal Analysis Cycle

Test LogsLODs

Classify Defects (Type, Injected &

Detected Phase)

Identify Top 80% Defects for RCA

Prj RCAReport

And select all high impact defects

Perform Fishbone AnalysisUsing Potential Causes

Arrive at Root CauseAnd Action List

Application

Reviews/Testing

Preventive Feedback



Pareto Chart

What is a Pareto ChartBar chart arranged in descending orderBars on the left are more important than those on the right Separates the “vital few” from the “trivial many”

Uses of Pareto ChartBreaks a big problem into smaller piecesIdentifies most significant factors (80-20 rule)Shows where to focus effortsAllows better use of limited resources



Pareto Chart - Exercise

Participants to discuss possible Code Review Defects

Classify the defects under different categories/types and assign a number of defects against each

Prepare a Pareto Chart using Excel to focus on the most significant defects (80-20 rule)



Pareto Chart - Code Review

0

1

2

3

4

5

6

7

8

Logi

cal e

rror

Red

unda

nt c

ode

Rem

ove

debu

ggin

g co

mm

ents

Com

men

ts n

ot e

xhau

stiv

e

Impl

emen

tatio

n er

ror

Fun

ctio

nalit

y m

issi

ng

Cod

e re

adab

ility

Hea

der

inco

mpl

ete

Wro

ng p

aram

eter

s pa

ssed

Nam

ing

conv

entio

n no

t fol

low

ed

Nu

mb

er o

f D

efec

ts

0

20

40

60

80

100

120

% o

f D

efec

ts

No. of defects

Percentage

Pareto Chart (contd.)



Cause & Effect Diagram - Fishbone

What is a Cause & Effect Diagram?A graphic tool that helps identify, sort and display possible causes of a problem or quality characteristic

Benefits of CEDDetermination of root causesEncourages group participationIndicates possible causes of variationUses a orderly, easy-to-read formatIdentifies areas for collecting data



Cause & Effect Diagram

• Decide the “Effect” to examine

• Identify the main categories

• Identify as many causes or factors as possible and attach them as sub branches of the major branches

• Identify increasingly more detailed levels of causes by asking a series of why questions

• Look for causes that appear repeatedly. These may be root causes

• Identify and circle the causes that we can take action on



Driven toofast

PoorMileage

Method

MaterialsMan

Machinery

Wronggears used

Carburetor needsadjustment

Under inflated tires

PoorDriving habits

Wrong Octanegas used

ImproperLubrication

Poor Maintenance

Cause & Effect Diagram



Exercise on RCA

Make Groups

Assign a PM to each group

Brainstorm and prepare a cause and effect/fishbone analysis

Present the result

(20 mins)



Defect Estimation

Phases Defect DistributionRequirement 7%Design 14%Coding & UT 49%IT/ST 27%AT 3%

Proposed Goal9%

16%44%29%2%

Design14%

Coding & UT49%

IT/ST27%

AT3%

Requirement7% - Use Historical Data

- Focus on Business Objectives and Process Improvement - Set more Challenging Goals



Applicability of RCA

•Defects•Customer Feedback•Non-conformance (NC)•Process Capability Baselines•Major Issues (that impact cost, quality, schedule)



Example

Coding & UT

People Measurement

Hardware/Software/ Tools

Support/ Guidance

Process/Standards

Guidelines not followed

Guidelines notupdated

Not aware

DEFECTS AT END OF CODING & UT

Assets not available

Not CommunicatedLong OverdueNot Available

Not Trained/Inadequate resources

Not adequateTool not inspected

Data not adequate



Common Defect Types

Defect Types Examples

Function/Class/Object error is one that affects significant capability, end-user interfaces, product interfaces, interface with hardware architecture, or global data structure(s) and should require a formal design change.

Database design/modeling error, functionality not working, etc

Assignment error indicates a few lines of code, such as the initialization of control blocks or data structure.

Oversight during coding, initialization of parameters/variables, incorrect setting of variables, java script validation, etc

Interface/Messages corresponds to errors in interacting with other components, modules or device drivers via macros, call statements, control blocks or parameter lists.Checking addresses program logic, which has failed to properly validate data and values before they are used.

Incorrect validation, missing validation, error handling, return value not checked

Timing/serialization errors are those, which are corrected by improved management of shared and real-time resources. Build/package/merge describe errors that occur due to mistakes in library systems, management of changes, or version control. Incorrect packaging, Setup problem, etc



Common Defect Types (contd.)

Defect Types Examples

Documentation errors can affect both publications and maintenance notes. Unclear specifications, standards not followed,

redundant code, GUI errors, incorrect description, ambiguous description, etc

Algorithm errors include efficiency or correctness problems that affect the task and can be fixed by (re)implementing an algorithm or local data-structure without the need for requesting a design change.

Hard coded values used, data type mismatch, etc

External Environment errors that occur due to factors that are outside the application scope.

Test data, test drivers, other tool defects, support system, concurrent work, inherited from previous release, third party software dependency, etc.

Performance errors affect the performance of the system.

Memory not released, Web session timeout not handled properly, browser cache related problems, etc

Database errors are related to errors in database or scripts.

Integrity constraint violated, SQL statements not tuned, Error in SQL statement, etc

Trivial/MinorTypo/minor errors in documentation, rephrasing, extra information in document, etc



Common Root Causes

Major Defect Categories (from Fishbone) Root Cause

Support/Guidance (e.g. Management Support, Training, etc)Handover (Change Coordination)Inadequate training (QMS, Defect Prevention, technical)

Process/Standards Guidelines/Standards/Procedures not updatedInadequate Process for Handling Requirements/DesignChange in Requirements/Design

People Breakdown of communicationsLack of knowledge (domain/system/tool)Oversight

Hardware/Software/Tools Configuration related problemInadequate tools

Measurements Incorrect analysis of data



Escape Analysis

(Requirements defects getting slipped to next phases of Life cycle)

Phase Detected RequirementsDesignCode Review UT IT ST AT Total

Phase InjectedRequirements 10 9 1 1 2 3 26Design 16 3 1 1 2 23Code review 24 20 16 6 66UT 0IT 2 2ST 3 3AT 0Total 10 25 28 22 21 14 0 120

Slippage (from Requirements to Design = 9/19) 47%DRE of Requirements Phase (10/19) 53%Slippage (from Coding Phase to subsequent phases = 52/80) 65%DRE of Code Review = 28/80) 35%

Defects Found during Code Review 28Defects slipped from Coding (not considering those injected in IT & ST) 52Defects slipped into Coding Phase 4TOTAL (Coding + subsequent Phases) 80



Root Cause & Action Planning

Root Causes of Defects for <Month_1>, <Phase_1> <Month_n>, <Phase_n>

Special Causes:(Root causes of high impact defects)Common Causes:(Root causes of high occurrence defects)

Action PlanImplementation Technique

Monitoring technique

PriorityAssociated Risk

Impact on PCB *

Expected Date of Closure

Person Responsible

StatusActual Date of Closure

Team to be given training on Domain Knowledge

Workshop every Friday by each team member in turn

PM to ensure that training is held

HighDefects in software

Schedule Slippage

20th June 2004

PM Open



Some Lessons Learnt

1Integration Testing should be scheduled so that the core modules are initially tested.

2 Client should be given overview for SISL's P&Q processes

3Client responsibilities should be clearly communicated in the beginning of the project.

4Design documents should contain all the necessary validations to avoid validation error

5 Checklist should be used to avoid GUI errors6 Rigorous unit testing to be done to avoid logical errors.

7Basic level review and testing should be done at developer's level before handing over the code to the testers and reviewers.

8 Test cases should be formed with test data.



Recap…

Defects and Bugs (Examples)Origin of DefectsClassification of DefectsDefect ManagementDefect DetectionDefect Prevention Cycle



Any Questions?



Thank You

Defect Prevention Training

Documents

copyright notice version

internal use

pq protection

project level pq

analysis of identified

injection of defects

bugs page

causes of defects reduction