PezzeYoung-Ch02-Framework.pdf

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 1

A Framework for Testing andAnalysis

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 2

Learning objectives

Introduce dimensions and tradeoff between

test and analysis activities

Distinguish validation from verification

activities

Understand limitations and possibilities of test

and analysis

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 3

Verification and validation

Validation:

does the software system meets the user's real

needs?

are we building the right software?

Verification:

does the software system meets the

requirements specifications?

are we building the software right?

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 4

Validation and Verification

Actual

Requirements

SW

SpecsSystem

Validation VerificationIncludes usability

testing, user

feedback

Includes testing,

inspections, static

analysis

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 5

Verification or validation depends onthe specification

Unverifiable (but validatable) spec: ... if a user

presses a request button at floor i, an available

elevator must arrive at floor i soon...

1 2 3 4 5 6 7 8

Example: elevator response

Verifiable spec: ... if a user presses a request

button at floor i, an available elevator must

arrive at floor i within 30 seconds...

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 6

Validation and Verification Activities

Actual Needs and

Constraints

Unit/

ComponentSpecs

System Test

Integration Test

Module Test

User Acceptance (alpha, beta test )

Re

vie

w

Analysis /

Review

Analysis /

Review

User review of external behavior as it is

determined or becomes visible

Unit/

Components

Subsystem

Design/SpecsSubsystem

System

Specifications

System

Integration

DeliveredPackage

validation

verification

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 7

You cant always get what you want

Correctness properties areCorrectness properties are undecidable undecidable

the halting problem can be embedded in almost

every property of interest

Decision

Procedure

Property

Program

Pass/Fail

ever

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 8

Getting what you need ...Perfect verification of

arbitrary properties by

logical proof or exhaustive

testing (Infinite effort)

Model checking:

Decidable but possibly

intractable checking of

simple temporal

properties.

Theorem proving:

Unbounded effort to

verify general

properties.

Precise analysis of

simple syntactic

properties.

Typical testing

techniques

Data flow

analysis

Optimistic

inaccuracy

Pessimistic

inaccuracy

Simplified

properties

optimistic inaccuracy: we mayaccept some programs that donot possess the property (i.e.,it may not detect allviolations).

testing

pessimistic inaccuracy: it isnot guaranteed to accept aprogram even if the programdoes possess the propertybeing analyzed

automated program analysistechniques

simplified properties: reducethe degree of freedom forsimplifying the property tocheck

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 9

Example of simplified property:Unmatched Semaphore Operations

synchronized(S) {

...

...

}

Static

checking for

match is

necessarily

inaccurate ...

if ( .... ) {

...

lock(S);

}

...

if ( ... ) {

...

unlock(S);

}

Java prescribes a

more restrictive, but

statically checkable

construct.

original problem simplified property

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 10

Some Terminology

Safe: A safe analysis has no optimisticinaccuracy, i.e., it accepts only correctprograms.

Sound: An analysis of a program P with respectto a formula F is sound if the analysis returnstrue only when the program does satisfy theformula.

Complete: An analysis of a program P withrespect to a formula F is complete if theanalysis always returns true when the programactually does satisfy the formula.

(c) 2007 Mauro Pezz & Michal Young Ch 2, slide 11

Summary

Most interesting properties are undecidable,thus in general we cannot count on tools thatwork without human intevention

Assessing program qualities comprises twocomplementary sets of activities: validation(daes the software do what it is supposed todo?) and verification (does the system behaveas specificed?)

There is no single technique for all purposes:test designers need to select a suitablecombination of techniques