Verification and Validation

Software Inspections

Verification and Validation

Blekinge Institute of Technology

SE-371 79 Karlskrona

+46 455 38 50 00

www.bth.se/eng

Today

Placing of inspection in V&V practices

Overview of software inspections

Different reading techniques

Defect estimation

Inspections vs. Testing



+46 455 38 50 00

www.bth.se/eng

Historical background

Inspection technique more than 30 years old

Was developed by Michael Fagan in 1970’s

Is one of the software quality assurance techniques



+46 455 38 50 00

www.bth.se/eng

Inspection’s place in VV

Dynamic V&V

ReliabilityProcess

Validation

Static V&V



+46 455 38 50 00

www.bth.se/eng

Software Defects

On 4 June 1996, the

maiden flight of the

Ariane 5 launcher

ended in a failure.

The failure of the Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition sequence (30 seconds after lift- off). This loss of information was due to specification and design errors in the software of the inertial reference system.

The exception was due to a floating-point error: a conversion from a 64-bit integer to a 16-bit signed integer…

There was no explicit exception handler to catch the exception, so it followed the usual fate of uncaught exceptions and crashed the entire software, hence the on-board computers, hence the mission.



+46 455 38 50 00

www.bth.se/eng

Inspection Objectives

To find problems at the earliest possible point in the software development process

Ensure that necessary rework will be done

Verify that any rework done meets the predefined criteria



+46 455 38 50 00

www.bth.se/eng

Inspection Process Fagan

• System overview presented to inspection team• Documents are distributed to inspection team in

advance• Inspection takes place and discovered faults

are noted• Modifications are made to repair discovered

faults• Re-inspection may or may not be required



+46 455 38 50 00

www.bth.se/eng

Software Inspection Characteristics

Can be characterized by terms:

Objective

# of participants (subject of discussion)

Reading techniques

Participant’s roles

Inspection meeting

Other



+46 455 38 50 00

www.bth.se/eng

Modification to Fagan’s process

Active Design (1985)

Two Person Inspection (1989)

N-fold Inspection (1990)

Phased Inspections (1993)

Inspection without meeting (1993)

Gilb Inspection (1993)



+46 455 38 50 00

www.bth.se/eng

Different reading techniques

Ad hoc reading

Checklist

Stepwise abstraction

Scenario based reading

Perspective based reading

And

TCD inspections



+46 455 38 50 00

www.bth.se/eng

Checklist for Requirements Inspection

Clearness Is the requirement ambiguous?

Is the requirement measurable?

Uniqueness Is it possible to find duplicate requirements?

Numbering Does the requirement have a number?



+46 455 38 50 00

www.bth.se/eng

Checklist for Code Inspections

Data Flows Are all program variables initialized?

Have all constants been named?

Control Flows For a conditional statement is the condition correct?

Is each loop certain to terminate?

Interface Flows Are the parameters in the right order?

Do function calls have the correct number of parameters?



+46 455 38 50 00

www.bth.se/eng

Cost-benefit analysis

What do inspections cost?

About 10-15% of the development budget (Not included the start-up costs) - Tom Gilb

Example of costs: Planning; Training; Inspecting; Analysis Other?

What are the benefits?

42% of all defects result from lack of traceability from code to design…

50-90% of the defects are caught by the inspections.



+46 455 38 50 00

www.bth.se/eng

Inspections Vs. Testing

To think about:

Which of the V&V technique should be applied and when?

The trade off between inspection and testing?

How to combine the inspections and program testing to reach the best results?



+46 455 38 50 00

www.bth.se/eng

V&V of Inspections

How we can improve?

How do we know that inspections work for us?



+46 455 38 50 00

www.bth.se/eng

Inspection Metrics

# of errors found

Inspection pace

# of documents inspected

Document size

Other?



+46 455 38 50 00

www.bth.se/eng

Defect Content Estimations

Objective:

Improve estimation of remaining fault content after review.

Motivation:

Process control is essential.

Reviews are an important means for control.

Fault content estimations from reviews give regular feedback on the quality status.



+46 455 38 50 00

www.bth.se/eng

Reviews and estimation

Normally, the number of found defects are documented in a protocol from the review, but no estimation of remaining fault content is performed.

Capture-recapture has been proposed as a way of estimating remaining number of faults.

Different statistical methods can be applied in the estimation process.



+46 455 38 50 00

www.bth.se/eng

Basic idea of CRC

Reviewer1

Reviewer2 Reviewer3

Reviewer1

Reviewer2Reviewer3

Is there more defects remaining in case 1 or case 2?

Case 1 Case 2



+46 455 38 50 00

www.bth.se/eng

Types of estimation methods

Defect contentestimation methods

from review

Historicalmethods

Directmethods

Experience-based

methods

Capture-recapturemethods

Curve-fitting

methods



+46 455 38 50 00

www.bth.se/eng

Method : Jack-knife method

Assumption: Detection ability the same for

all reviewers.

Defect number

Pro

bab

ilit

y fo

rd

ete

ctio

n

0

0,2

0,4

0,6

0,81

1 2 3 4 5 6 7 8 9

10

Reviewer 1

Reviewer 2

Reviewer 3

Reviewer 4



+46 455 38 50 00

www.bth.se/eng

Method : Maximum-Likelihood

Assumption: Detection ability the same for all

defects.

Defect number

Prob

abili

ty fo

rde

tect

ion

00,10,20,30,40,50,60,70,80,9

1

1 2 3 4 5 6 7 8 9 10

Reviewer 1

Reviewer 2

Reviewer 3

Reviewer 4



+46 455 38 50 00

www.bth.se/eng

Graphic estimation methods

Previous methods are statistical.

Proposal: sort data and fit function

Two approaches:

Detection profile method,

exponential decreasing function

Cumulative method,

exponential increasing function

Defect number (k)

Num

ber

of

defe

ct

occ

urr

ence

s

(mk)

0

2

4

6

8

1 2 3 4 5 6 7 8 9

10

Defect number (k)

Cum

ula

tive

num

ber

of

defe

ct

occ

urr

ence

s

(Mk)

0

10

20

30

40

1 2 3 4 5 6 7 8 9 10



+46 455 38 50 00

www.bth.se/eng

Detection profile method

Main idea:

sort data: the defect found by most reviewers is defect number 1 and so forth,

fit an exponential function,

estimate (last time the function is above 0.5)

Defect number (k)

Num

ber o

fde

fect

occu

rren

ces

(mk)

0

2

4

6

8

1 2 3 4 5 6 7 8 9

10



+46 455 38 50 00

www.bth.se/eng

Cumulative method

Main idea:

sort data: same numbering, plot cumulative,

fit an exponential function (C maximum)

estimate (C - number of defects found) => likely to overestimate

Defect number (k)

Cum

ulat

ive

num

ber

ofde

fect

occu

rren

ces

(Mk)

0

10

20

30

40

1 2 3 4 5 6 7 8 9 10

C



+46 455 38 50 00

www.bth.se/eng

Some general conclusions

Software Inspections are important QA technique

Software companies can benefit from calculating the trade-off between inspections and testing

There exist methods to estimate the defect content in the software artifact, though further research is required



+46 455 38 50 00

www.bth.se/eng

Reading Reference Books:

Steven. R. Rakitin, “Software Verification and Validation For Practitioners and Managers”, Chapters 5,6.

I. Sommerville, “Software Engineering”, Excerpt: Verification & Validation, Addison-Wesley, 7-th addition, pp. 521-527.

Articles:

A. Aurum, H. Petersson and C. Wohlin, “State-of-the-Art: Software Inspections after 25 Years”, Software Testing Verification and Reliability, 2001.

H. Petersson and T. Thelin, ”A Survey of Capture-Recapture in Software Inspections”, Proc. First Swedish Conf. on Software Engineering Research and Practice, October 2001.

S. Biffl, “Using Inspection Data for Defect Estimation”, IEEE Software, November/ December 2000, pp. 36-43.

Verification and Validation

Documents

person inspection

gilb inspection

inspection teamdocuments

inspection characteristicscan

discovered faultsreinspection

nfold inspection

entire software

requirement ambiguous