Topic 12Summer 2003 1 ICS 52: Introduction to Software Engineering Lecture Notes for Summer Quarter, 2003 Michele Rousseau Topic 12 Partially based on.

Topic 12 Summer 2003 1

ICS 52: Introduction to Software Engineering

Lecture Notes for Summer Quarter, 2003

Michele RousseauTopic 12

Partially based on lecture notes written by Sommerville, Frost, Van Der Hoek, Taylor & Tonne. Duplication of course material for any commercial purpose

without the written permission of the lecturers is prohibited


Verification and Validation

Informal Requiremen

ts

Informal Requiremen

ts

Formal Specificatio

n

Formal Specificatio

n

Software Implementati

on

Software Implementati

on

Validation

Verification

Verification: is implementation consistent with requirements specification?Validation: does the system meet the customer’s/user’s needs?


Software Quality: assessment by V&V

Software process must include verification and validation to measure product qualities • correctness, reliability, robustness• efficiency, usability, understandability • verifiability, maintainability• reusability, portability, interoperability,• real-time, safety, security, survivability, accuracy

Products can be improved by improving the process by which they are developed and assessed


Testing Terminology

Failure: Incorrect or unexpected output, based on specifications• Symptom of a fault

Fault: Invalid execution state• Symptom of an error• May or may not produce a failure

Error: Defect or anomaly or “bug” in source code• May or may not produce a fault


1: input A,B

2: A>0?

3: C :=0 4: C := A*B

5: B>0?

6: X := C*(A+2*A) 7: X := A+B

8: output X

Examples: Faults, Errors, and Failures

Suppose node 6 should be X:= C*(A+2*B)

• Failure/Fault-less error: - Suppose the inputs are

(A=2,B=1) – the executed path

will be (1,2,4,5,7,8) which will not reveal this fault because 6 is not executed

- Suppose the inputs are (A=-2,B=-1) – the executed path will be (1,2,3,5,6,8) which will not reveal the fault because C = 0

Need to make sure proper test cases are selected

• the definitions of C at nodes 3 and 4 both affect the use of C at node 6- executing path (1,2,4,5,6,8) will reveal the failure, but only if B != 0 - (e.g. Inputs (A=1,B=-2) )


Software Testing

Exercising a system [component] • on some predetermined input data• capturing the behavior and output data• comparing with test oracle• for the purposes of

» identifying inconsistencies » verifying consistency between actual results and specification

to provide confidence in consistency with requirements and measurable qualities

to demonstrate subjective qualities

» validating against user needs Limitations

• only as good as the test data selected• subject to capabilities of test oracle


Definition to execution (Levels of testing)

System Testing • Defined at Requirements -> Run after integration testing

Integration Testing• Defined at Design -> Run after Module Testing

Module Testing• Defined at Design -> Run after Unit Testing

Unit Testing• Defined at Implementation -> Run after Implementation of

each unit Regression Testing (testing after Change)

• Defined throughout the process -> Run after modifcations


Requirements Analysis

Architectural (Preliminary)

Design

Algorithmic (Detailed)

Design

Implemen- tation

Requirements Specification

Module Interface

Specification

Module Design

Specification

Source Code

System Testing

Integration Testing

Module Testing

Unit Testing

Problem Definition

System Test Plan

Integration Test Plan

Module Test Plan

Unit Test Plan

Behavior verification

against oracle

MaintenanceRS/MIS/ MDS/SC

Modification

Regression Testing

Regression Test Plan

Testing-oriented Life Cycle Process


How to make the most of limited resources?

Fundamental Testing Questions

Test Criteria: What should we test? Test Oracle: Is the test correct? Test Adequacy: How much is enough? Test Process: Is our testing effective?


Practical Issues

Purpose of testing• Fault detection• High assurance of reliability• Performance/stress/load• Regression testing of new versions

Conflicting considerations• safety, liability, risk, customer satisfaction,

resources, schedule, market windows and share

Test Selection is a sampling technique• choose a finite set from an infinite domain


Test Criteria

Testing must select a subset of test cases that are likely to reveal failures

Test Criteria provide the guidelines, rules, strategy by which test cases are selected• actual test data• conditions on test data• requirements on test data

Equivalence partitioning is the typical approach• a test of any value in a given class is equivalent to a

test of any other value in that class• if a test case in a class reveals a failure, then any

other test case in that class should reveal the failure• some approaches limit conclusions to some chosen

class of errors and/or failures


Test Adequacy

Coverage metrics• when sufficient percentage of the program

structure has been exercised Empirical assurance

• when failures/test curve flatten out Error seeding

• percentage of seeded faults found is proportional to the percentage of real faults found

Independent testing• faults found in common are representative of

total population of faults


Functional and Structural Testing

Functional Testing• Test cases selected based on specification• Views program/component as black box

Structural Testing• Test cases selected based on structure of

code• Views program /component as white box

(also called glass box testing)


Black Box vs. White Box Testing

SELECTED INPUTS

RESULTANT OUTPUTS

INTERNAL BEHAVIOR

DESIRED OUTPUT

SOFTWARE DESIGN

“BLACK BOX” TESTING

“WHITE BOX” TESTING

SELECTED INPUTS

RESULTANT OUTPUTS

DESIRED OUTPUT


Structural (White Box) Testing

Use source code to derive test cases Build flow graph of program State test requirements in terms of graph

coverage Various types of coverage identified Choose test cases that guarantee different types of

coverage• All-Statements coverage• All-Branches coverage• All-Edges• All-Paths


Graph representation of control flow and data flow relationships

Flow Graphs

Control Flow• The partial order of statement execution, as

defined by the semantics of the language Data Flow

• The flow of values from definitions of a variable to its uses


123456789

101112131415

function P return INTEGER isbegin

X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop

X := X – 10;exit when X = 10;

end loop;if (Y < 20 and then X mod 2 = 0) then

Y := Y + 20;else

Y := Y – 20;end if;return 2*X + Y;

end P;

A Sample Program to Test


2,3,4 5

6

9´

10

12

14

T T

F

F 9 T

F

7

TF

9a 9b

P’s Control Flow Graph (CFG)

function P return INTEGER isbegin

X, Y: INTEGER;READ(X); READ(Y);while (X > 10) loop

X := X – 10;exit when X = 10;

end loop;

12345678

9101112131415

if (Y < 20 and then X mod 2 = 0) then

Y := Y + 20;else

Y := Y – 20;end if;return 2*X + Y;

end P;


All-Statements Coverage

Select test cases such that every node in the graph is visited• Also called node coverage

» Guarantees that every statement in the source code is executed at least once

Selects minimal number of test cases

1 3 7 82 4 5 6 9 10


2,3,4 5

6 10

12

14

TF

9

T

F

7

TF

At least 2 test cases needed

Example all-statements-adequate test set:(X = 20, Y = 10)

<2,3,4,5,6,7,9,10,14>(X = 20, Y = 30)

<2,3,4,5,6,7,9,12,14>

All-Statements Coverage of P


All-Branches Coverage

Select test cases such that every branch in the graph is visited

» Guarantees that every branch in the source code is executed at least once

More thorough than All-Statements coverage• More likely to reveal logical errors

1 3 7 82 4 5 6 9 10


2,3,4 5

6 10

12

14

TF

9

T

F

7

TF


Example all-branches-adequate test set:(X = 20, Y = 10)

<2,3,4,5,6,7,9,10,14>(X = 15, Y = 30)

<2,3,4,5,6,7,5,9,12,14>

All-Branches Coverage of P


All-Edges Coverage

Select test cases such that every edge in the graph is visited

» Takes complex statements into consideration – treats them as separate nodes

More thorough than All-Branches coverage• More likely to reveal logical errors


2,3,4 5

6

9b

10

12

14

T T

F

F9a

T

F

7

TF


Example all-edges-adequate test set:(X = 20, Y = 10)

<2,3,4,5,6,7,9a,9b,10,14>(X = 5, Y = 30)

<2,3,4,5,9a,12,14>(X = 21, Y = 10)

<2,3,4,5,6,7,5,6,7,5,9a,9b,12,14>

All-Edges Coverage of P


All-Paths Coverage

Path coverage• Select test cases such that every path in the

graph is visited• Loops are a problem

» 0, 1, average, max iterations

Most thorough… …but is it feasible?


2,3,4 5

6

9b

10

12

14

T T

F

F9a

T

F

7

TF

Infinitely many test cases needed

Example all-paths-adequate test set:(X = 5, Y = 10)(X = 15, Y = 10)(X = 25, Y = 10)(X = 35, Y = 10)…

All-Paths Coverage of P


2,3,4 5

6

9b

10

12

14

T

F

9aT

FY

X

X

Y

Y X

X

Y

YXX

X

TF

7

TF X

X

P’s Control and Data Flow Graph


Subsumption of Criteria

C1 subsumes C2 if any C1-adequate testT is also C2-adequate• But some T1 satisfying C1 may detect fewer

faults than some T2 satisfying C2


all-statements

all-edges

boundary-interiorloop testing

min-maxloop testing

all-paths

all-defs

all-uses

all-DU-paths

all-p-uses all-c-uses

C1 C2

subsumes all-branches

Structural Subsumption Hierarchy


Equivalence partitioning

Identify the set of all inputs Identify possible bases for subdividing the input Domain:

•size, order, structure•correctness•stated requirements & your smarts

Divide input set into (sub)domains using the basis•“Equivalence partitions”• Subdomains may overlap (may not be a partition)

Select representative test cases from each subdomain•one test case may suffice, more is better


Example

1 float homeworkAverage(float[] scores) {

2 float min = 99999;

3 float total = 0;

4 for (int i = 0 ; i < scores.length ; i++) {

5 if (scores[i] < min)

6 min = scores[i];

7 total += scores[i];

8 }

9 total = total – min;

10 return total / (scores.length – 1);

11 }


Possible Bases

Array length• empty array• one element• 2 or 3 elements • 4 or more elements

Input domain: float[]Basis: array length

one

small

emptylarge

} subdomains, eq. partitions


Possible Bases

Position of minimum score• Smallest element first• Smallest element in middle• Smallest element last

Input domain: float[]Basis: position of minima

somewhere in middlefirst last


Possible Bases

Number of minima• Unique minimum• A few minima• All minima

Input domain: float[]Basis: number of minima

all data equal1 minimum2 minima


Equivalence partitions & white-box testing

The basis - equivalence partition - test case approach works for structural testing, too.

A basis could be nodes, edges, or paths.

Each node, edge, or path defines a partition of the input.


Challenges

Structural testing can cover all nodes or edges without revealing obvious faults• No matter what input, program always returns 0

Some nodes, edges, or loop combinations may be infeasible• Unreachable/unexecutable code

“Thoroughness”• A test suite that guarantees edge coverage also

guarantees node coverage…• …but it may not find as many faults as a

different test suite that only guarantees node coverage


Limitations of Structural Testing

Interactive programs Listeners, event-driven programs Concurrent programs or tasks; threads Exceptions Self-modifying programs (assembly language) Objects loaded over the network Super-class constructors Asynchronous garbage collection


Summary of Structural Testing

Conceptually simple Relies on access to source Not applicable in all situations Acts in ignorance of specifications


Ensure that changes made during maintenancedo not destroy existing functionality

Regression Testing

Permanent suite of test cases• Saves effort creating test cases• Provides record of existing functionality

Add new test cases and delete obsolete ones when necessary


Cost reduction:Select minimal subset of

regression test suite that tests the changes

Selective Regression Testing

Analyze relationship between the test cases and the software entities they cover

Use information about changes to select test cases for new version


= modified entity

System Under Test

Test 2Test 1 Test 3

= exercises relation

= entity

Simple Example


System Under Test

Test 3

= exercises relation= entity

= modified entity

Selective Regression Testing


Cost of running necessary tests


RetestAll

Cost-IneffectiveSelective

Retest


Cost

Cost-EffectiveSelective

Retest

Cost of running unnecessary tests

Cost of additional analysis

Cost of additional analysis

The Cost-Effectiveness Tradeoff


The Reality of Software Testing

One of the most widely-used, practical class of techniques for revealing faults in software

An area of software engineering showing the greatest gap between theory and practice

Topic 12Summer 2003 1 ICS 52: Introduction to Software Engineering Lecture Notes for Summer Quarter, 2003 Michele Rousseau Topic 12 Partially based on.

Documents