1 Software Unit Test Coverage And Test Adequacy Hong Zhu, Patrick A. V. Hall, John H.R. May Presented By: Arpita Gandhi.

1

Software Unit Test Coverage

And

Test Adequacy

Hong Zhu, Patrick A. V. Hall, John H.R. May

Presented By:

Arpita Gandhi

2

Survey recent research on software test criteria in the last two decades

Put it into a uniform framework

Address issues like Defining a test criterion

Understanding different test adequacy criteria

Whether they are worth the cost

Future directions for this subject

Goal

3

Importance today

Critical feature in the software development cycle

Identify correctness, completeness Quality assurance Verification and validation Reliability estimation

http://www.aptest.com/resources.html

4

Previous Work-Test Adequacy

A software test adequacy criterion is a predicate thatdefines what properties of a program must be exercised to

constitute a thorough test.

Proposed test criteria :Reliability- producing consistent resultsValidity – producing meaningful results

Not practicably applicable

Reliability and validity closely related and not independent.

5

Notions associated with a test adequacy criterion

It is a stopping rule: Determines if sufficient testing is done that it can stop. E.g. execution of all statements for statement

coverage

It provides measurements of test quality: Degree of adequacy E.g. Percent of code coverage

Current Work-Test Adequacy

6

Specify a software testing requirement Determine test cases to satisfy requirement

Determine observations that should be made during testing

Control the cost of testing Avoid redundant and unnecessary tests

Help assess software dependability

Build confidence in the integrity estimate

Uses of Test Adequacy Criteria

7

Categories of Adequacy Criteria

Based on: Source of information to specify testing requirements

Specification based Program based Combined Interface based

Underlying testing approach Structural Fault based Error based

8

Structural Testing Criteria

Program-Based

MutationPathBranchStatement

Specification-BasedSpecification-Based

Control Flow Based Data Flow Based Dependence coverage

Structural Testing

9

Statement Coverage Execute every statement at least once The simplest , weakest type of testingint* p = NULL;

if (condition) {

p = &variable; *p = 1; } *p = 0;

void function(const char* string1, const char* string2) {

if (condition || strcmp(string1, string2) == 0) ... } Useful link:http://www.bullseye.com/lineCoverage.html

Control flow – based testing

10

Control flow-based testing

Branch Coverage

Exercise all control statements in the program Subsumes statement coverage All combinations of control transfers may not be

checked

if (condition1 && (condition2 || function1( ))) statement1;

else statement2;

11

Path Coverage Execute all paths from program’s entry to exit Subsumes branch coverage

Number of paths is exponential to the number of branches

Many paths are impossible to exercise if (success)

statement1;

statement2;

if (success)

statement3;

Control flow-based testing

12

Criteria Inclusion Hierarchy

Branch Coverage

Statement Coverage

Path Coverage

Control flow-based testing

13

Program-Based Specification-BasedSpecification-Based

Control Flow Based Data Flow Based Dependence coverage

Structural Testing

Simple Definition-UseCoverage

Testing for structured and dynamic data

Inter-procedural testing

Program-Based

Data flow-based testing

14

Definition Occurance – Where a value is bound to the variable. (x=1)

Use Occurance- Where the value is referred (a=x) C-use :if used compute a value or as an output (n=x+1)

Global c-use: Value is assigned in a block other than the one in which it is being used.

Local c-use P-use: if used to decide whether a predicate is true (x>0)

Definition-Clear path w.r.t x: for all nodes in the path,there is no definition occurance of x

Data flow-based testing

15

Simple definition use coverage

If variable x has a global definition in node i, All definitions Criterion : test data covers some path which goes from i to a node or edge

at which the value assigned to x in node i is used All Uses Criterion : test data covers at least one path which goes from i to each

node and edge at which the value assigned to x in node i is used All-du-paths: test data covers all paths which go from i to each node and edge

at which the value assigned to x in node i is used are exercised There may be infinitely many such paths

Data flow-based testing

16

2

1

3

4

5

67

89

Data flow-based testing

17

Criteria Inclusion Hierarchy

All-paths

All-du-paths

All-uses

All-c/some-p-uses All-p/some-c-uses

All-definitions All-c-usesAll-p-uses

Data flow-based testing

18

For structured and dynamic data

Limitations of testing criteria seen so far: No distinction between atomic (integers) and structured

data arrays and records)

A[0l = A[il;A[i] = A[j];A[jl = A[0l;

There is a DU path from each statement to the next.The actual DU path should involve A [0] inthe first and last statement. Proposed solution: Treat structured data elements as independent data May lead to infinite du paths to be tested.

Data flow-based testing

19

Test data-dependence among procedure interfaces

Types of data dependence Direct

Definition occurs in procedure P,

use in directly called procedure Q Indirect

Considers multiple levels of calls and returns Definition occurs in P , use in G.

Example:

Class Q extends P, R extends Q

Inter-procedural data-flow testing

Data flow-based testing

f(P)

f(Q)

f(R)

20

Fault-based testing criteriaError Seeding

Estimate the number of faults that remain Measure quality of software testing

r = # artificial faults detected # of artificial faults f = # of not seeded errors detected

Estimated no. of inherent faults = (1/r)*f

Applicable to any testing method

Dependent on how faults are introduced

21

Program Mutation Testing

Mutant: A program with a planted fault Execute mutants on each member of test set Compare results Mutation Adequacy Score =D/N

D=No. of dead mutants

N = No. of non equivalent mutants

Fault-based testing criteria

c = a – b; c = a + b;

R1 R2

If (R1 = R2): mutant is alive otherwise it is killed.

22

Variants of Program Mutation Testing Weak Mutation Testing

proposed to improve efficiency mutate and test components

Firm Mutation Testing Select portion of program , subset of parameters and mutate them. Compare original and changed versions Less expensive than strong mutation testing,more

efficient than weak mutation testing No basis to select area of program code, parameters

Fault-based testing criteria

23

Criteria Inclusion Hierarchy

Firm Mutation Testing

Weak Mutation Testing

Strong Mutation Testing

Fault-based testing criteria

24

Perturbation Testing

Tests the robustness of a program Predicted fault tolerance = # of faults detected

total # of executions A perturbation function is applied to change the data stateExample:

int perturbation (int x) { int changedX; changedX = x + 50; return changedX; }

Fault-based testing criteria

25

main(){int x; x = getVal(); if (x > 0) printf(“X is positive”); else printf(“X is negative”);}

main(){int x; x = getVal(); x = perturbation(x); if (x > 0) printf(“X is positive”); else printf(“X is negative”);}

Fault-based testing criteria

Original program Fault injected program

Perturbation Testing

26

Firm Mutation Testing

Weak Mutation Testing

Strong Mutation Testing

Branch Coverage

Statement Coverage

Summary

All-paths

All-du-paths

All-uses

All-c/some-p-uses All-p/some-c-uses

All-definitionsAll-c-usesAll-p-uses