chapter6-part1

Ch. 6 1

Verification

Ch. 6 2

Outline• What are the goals of verification?• What are the main approaches to

verification?– What kind of assurance do we get

through testing?– How can testing be done systematically?– How can we remove defects

(debugging)?• What are the main approaches to

software analysis?– informal vs. formal

Ch. 6 3

Need for verification

• Designers make mistakes even if they are skilled and follow sound principles

• Everything must be verified, every required quality, process and products– even verification itself…

Ch. 6 4

Properties of verification

• May not be binary (OK, not OK)– severity of defect is important– some defects may be tolerated

• May be subjective or objective– e.g., usability

• Even implicit qualities should be verified– because requirements are often incomplete– e.g., robustness

Ch. 6 5

Approaches to verification

• Experiment with behavior of product– sample behaviors via testing– goal is to find "counterexamples"– dynamic technique

• Analyze product to deduce its adequacy– analytic study of properties– static technique

Ch. 6 6

Testing and lack of "continuity"

• Testing samples behaviors by examining "test cases"

• Impossible to extrapolate behavior of software from a finite set of test cases

• No continuity of behavior– it can exhibit correct behavior in

infinitely many cases, but may still be incorrect in some cases

Ch. 6 7

Verification in engineering

• Example of bridge design• One test assures infinite correct

situations

Ch. 6 8

procedure binary-search (key: in element; table: in elementTable; found: out Boolean)

beginbottom := table'first; top := table'last; while bottom < top loop

if (bottom + top) rem 2 ≠ 0 then middle := (bottom + top - 1) / 2;

end if;if key ≤ table (middle) then

top := middle;else

bottom := middle + 1;end if;

end loop;found := key = table (top);

end binary-search

Ch. 6 9

procedure binary-search (key: in element; table: in elementTable; found: out Boolean)

beginbottom := table'first; top := table'last; while bottom < top loop

if (bottom + top) rem 2 ≠ 0 then middle := (bottom + top - 1) / 2;

else middle := (bottom + top) / 2;

end if;if key ≤ table (middle) then

top := middle;else

bottom := middle + 1;end if;

end loop;found := key = table (top);

end binary-search

if we omit thisthe routineworks if the elseis never hit!(i.e. if size of table is a power of 2)

Ch. 6 10

Goals of testing

• To show the presence of bugs (Dijkstra, 1987)

• If tests do detect failures, we cannot conclude that software is defect-free

• Still, we need to do testing– driven by sound and systematic

principles

Ch. 6 11

Goals of testing (cont.)

• Should help isolate errors– to facilitate debugging

• Should be repeatable– repeating the same experiment, we

should get the same results• this may not be true because of the effect

of execution environment on testing• because of nondeterminism

• Should be accurate

Ch. 6 12

Theoretical foundations of testing

Ch. 6 13

Definitions (1)• P (program), D (input domain), R

(output domain)– P: D R (may be partial)– Let OR be the requirement on output

values of P as stated in P’s specification

• Correctness defined by OR D R– P(d) correct if <d, P(d)> OR– P correct if all P(d) are correct

Ch. 6 14

Definitions (2)• FAILURE

– A manifest symptom of the presence of an error.– P(d) is not correct

• may be undefined (error state) or may be the wrong result

• ERROR (DEFECT)– anything that may cause a failure

• typing mistake• programmer forgot to test “x = 0”

• FAULT– incorrect intermediate state entered by program

Ch. 6 15

Definitions (2.1)• FAILURE

– Occurs only if a fault happens during execution

• ERROR (DEFECT)– P(d) does not satisfy the output

requirements

• FAULT– Occurs only if the programs contains an

error

Ch. 6 16

Definitions (3)

• Test case t – an element of D

• Test set T– a finite subset of D

• Test is successful if P(t) is correct• Test set successful if P correct for all

t in T

Ch. 6 17

Definitions (4)

• Ideal test set T– if P is incorrect, there is an element of

T such that P(d) is incorrect

• if an ideal test set exists for any program, we could prove program correctness by testing

Ch. 6 18

Test criterion• A criterion C defines finite subsets of D

(test sets)– C 2D (the set of all finite subsets of D)

• A test set T satisfies C if it is an element of CExampleC = {<x1, x2,..., xn> | n 3 i, j, k, ( xi<0 xj=0

xk>0)}<-5, 0, 22> is a test set that satisfies C<-10, 2, 8, 33, 0, -19> also does<1, 3, 99> does not

Ch. 6 19

Properties of criteria (1)• C is consistent

– for any pairs T1, T2 satisfying C, T1 is successful iff T2 is successful• so either of them provides the “same”

information

• C is complete– if P is incorrect, there is a test set T of

C that is not successful• C is complete and consistent

– identifies an ideal test set– allows correctness to be proved!

Ch. 6 20

Properties of criteria (2)

• C1 is finer than C2– for any program P

• for any T1 satisfying C1 there is a subset T2 of T1 which satisfies C2

Ch. 6 21

Properties of definitions

• None is effective, i.e., no algorithms exist to state if a program, test set, or criterion has that property

• In particular, there is no algorithm to derive a test set that would prove program correctness– there is no constructive criterion that

is consistent and complete

Ch. 6 22

Empirical testing principles

• Attempted compromise between the impossible and the inadequate

• Find strategy to select significant test cases– significant=has high potential of

uncovering presence of error

Ch. 6 23

Complete-Coverage Principle

• Try to group elements of D into subdomains D1, D2, …, Dn where any element of each Di is likely to have similar behavior– D = D1 D2 … Dn

• Select one test as a representative of the subdomain

• If Dj Dk for all j, k (partition), any element can be chosen from each subdomain

• Otherwise choose representatives to minimize number of tests, yet fulfilling the principle

Ch. 6 24

Complete-Coverage Principle

example of a partition

Ch. 6 25

Testing in the smallWe test individual modules• BLACK BOX (functional) testing

– partitioning criteria based on the module’s specification

– tests what the program is supposed to do

• WHITE BOX (structural) testing– partitioning criteria based on module’s

internal code– tests what the program does

Ch. 6 26

White box testing

derives test cases from program code

Ch. 6 27

Structural Coverage Testing

• (In)adequacy criteria – If significant parts of program

structure are not tested, testing is inadequate

• Control flow coverage criteria– Statement coverage– Edge coverage– Condition coverage– Path coverage

Ch. 6 28

Statement-coverage criterion

• Select a test set T such that every elementary statement in P is executed at least once by some d in T– an input datum executes many

statements try to minimize the number of test cases still preserving the desired coverage

Ch. 6 29

Exampleread (x); read (y);if x > 0 then

write ("1");else

write ("2");end if;if y > 0 then

write ("3");else

write ("4");end if; 

Ch. 6 30

Exampleread (x); read (y);if x > 0 then

write ("1");else

write ("2");end if;if y > 0 then

write ("3");else

write ("4");end if; 

{<x = 2, y = 3>, <x = - 13, y = 51>, <x = 97, y = 17>, <x = - 1, y = - 1>}covers all statements

{<x = - 13, y = 51>, <x = 2, y = - 3>} is minimal

Ch. 6 31

Weakness of the criterion

if x < 0 then x := -x;

end if;z := x;

{<x=-3} covers allstatements

Ch. 6 32

Weakness of the criterion

if x < 0 then x := -x;

end if;z := x;

{<x=-3} covers allstatements

it does not exercise the case when x is positiveand the then branch isnot entered

Ch. 6 33

Edge-coverage criterion

• Select a test set T such that every edge (branch) of the control flow is exercised at least once by some d in Tthis requires formalizing the concept of the control graph, and how to construct it– edges represent statements– nodes at the ends of an edge represent entry into

the statement and exit

Ch. 6 34

G G1 2 G1

G1

G1

G 2

I/O, assignment, or procedure call

if-then-else if-then

while loop

two sequential statements

Control graph construction rules

Ch. 6 35

Simplification

a sequence of edges can be collapsed into just one edge

. . .n n nnn k-1 k1 2 3

n1n

k

Ch. 6 36

beginread (x); read (y);while x ≠ y loop

if x > y then x := x - y;

else y := y - x;

end if;end loop;gcd : = x;

end;

Exemple: Euclid's algorithm

Ch. 6 37

Weaknessfound := false; counter := 1;while ((not found) and counter < number_of_items) loop

if (table[counter] = desired_element) then found := true;

end if;counter := counter + 1;

end loop;if found then

write ("the desired element is in the table");else

write ("the desired element is not in the table");end if;  

Ch. 6 38

Condition-coverage criterion

• Select a test set T such that every edge of P’s control flow is traversed and all possible values of the constituents of compound conditions are exercised at least once– it is finer than edge coverage

Ch. 6 39

Weakness if x ≠ 0 then

y := 5; else

z := z - x; end if;if z > 1 then

z := z / x; else

z := 0; end if; 

Ch. 6 40

Weakness

{<x = 0, z = 1>, <x = 1, z = 3>}

 if x ≠ 0 then

y := 5; else

z := z - x; end if;if z > 1 then

z := z / x; else

z := 0; end if; 

Ch. 6 41

Weakness

{<x = 0, z = 1>, <x = 1, z = 3>} causes the execution of all edges, but fails to expose the risk of a division by zero

 if x ≠ 0 then

y := 5; else

z := z - x; end if;if z > 1 then

z := z / x; else

z := 0; end if; 

Ch. 6 42

Path-coverage criterion

• Select a test set T which traverses all paths from the initial to the final node of P’s control flow– it is finer than previous kinds of

coverage– however, number of paths may be too

large, or even infinite (see while loops)• additional constraints must be provided

Path-coverage Criterion

For loop <= 20 times what would be the number of uniquePaths through the program?

Control-flow graph

Ch. 6 44

Problemmain (){ int i, j, k, match;

cin >> i >> j >> k; cout << i << j << k;

if (i <= 0 || j <= 0 || k <= 0 || i+j <= k || j+k <= i || k+i <= j) match = 4; else if !(i == j || j == k || k == i) match = 3;else if (i != j || j != k || k != i) match = 2;else

match = 1;cout << match << endl;

}

Ch. 6 45

Problem1) If, for an assignment of values to the input variables i, j, and

k, the output variable match will assume a correct value upon execution of the program, we can assert that the program is correct for this particular test case.

2) And if we can test the program for all possible assignments to i, j , and k, we will be able to determine its correctness.

3) Assume an ordinary integer variable inC++ to be a value in the range −32,768 to +32,767 (i.e., 216).

Ch. 6 46

The infeasibility problem• Read( x ); z := 2 * x; if( z = 4 ) then statements

• If( x > 0 ) then if( x < 0 ) then statements

Ch. 6 47

The infeasibility problem• Syntactically indicated behaviors

(statements, edges, etc.) are often impossible– unreachable code, infeasible edges, paths,

etc.

• Adequacy criteria may be impossible to satisfy – manual justification for omitting each

impossible test case– adequacy “scores” based on coverage

• example: 95% statement coverage

Ch. 6 48

Further problem

• What if the code omits the implementation of some part of the specification?

• White box test cases derived from the code will ignore that part of the specification!

Ch. 6 49

Further problem1) Write a program that takes three positive integers

as input and determine if they represent three sides of a triangle, and if they do, indicate what type of triangle it is.

Ch. 6 50

Further problem

To be more specific, it should read three integers and setflag as follows:1) If they represent a scalene triangle, set the flag to 1.2) If they represent an isosceles triangle, set the flag to 2. 3) If they represent an equilateral triangle, set the flag to 3.4) If they do not represent a triangle, set the flag to 4.