1 Software Testing. 2 Software Life Cycle Sommerville, 1992: D evelopment efforts are typically distributed as follows: Specifications / Design 30% -

1

Software Testing

2

Software Life Cycle

Sommerville , 1992: Development efforts are typically distributed as follows:Specifications / Design 30% - 40%

Implementation 15% - 30%

Testing 25% - 50%

3

Remarks by Bill Gates17th Annual ACM Conference on Object-Oriented Programming, Seattle, Washington, November 8, 2002

“… When you look at a big commercial software

company like Microsoft, there's actually as much

testing that goes in as development. We have as many

testers as we have developers. Testers basically test all

the time, and developers basically are involved in the

testing process about half the time…

… We've probably changed the industry we're in. We're

not in the software industry; we're in the testing

industry, and writing the software is the thing that

keeps us busy doing all that testing.”

4

Remarks by Bill Gates (cont’d)

“…The test cases are unbelievably expensive;

in fact, there's more lines of code in the test

harness than there is in the program itself.

Often that's a ratio of about three to one.”

“… Well, one of the interesting questions is,

when you change a program, … what portion

of these test cases do you need to run?“

5

“We also went back and say, OK, what is the state-of-the-art in terms of being able to prove programs, prove that a program behaves in a certain way? …And although a full general solution to that is in some ways unachievable, for many very interesting properties this idea of proof has come a long way in helping us quite substantially just in the last year…

“We use model-checking… You describe the constraints, such as ‘nobody should acquire the lock if they've already acquired it’ … then it's trying to go through to determine is there some path through the program that violates the constraints. … The initial domain we applied this in was in device drivers… “

Remarks by Bill Gates (cont’d)

6

The V-model of development

Requirementsspecification

Systemspecification

Systemdesign

Detaileddesign

Module andunit codeand tess

Sub-systemintegrationtest plan

Systemintegrationtest plan

Acceptancetest plan

ServiceAcceptance

testSystem

integration testSub-system

integration test

Implementation and unit testing

7

From V to Y: automatic code generation

Implementation Unit-test

AutomaticCode-generation

Detailed design

Specification

...

...

...

...

8

Some popular testing categories

Black box / white box

Static / dynamic

9

Specification-based testing Reflects true

intention of testing

Does not propagate errors from previous versions

Regression testing Does not need a

specification Easy to

implement Finds subtle

errors

Testing methodologies

10

How shall we check the I/O relation ? Manually (specification-based) Table of expected results ( a simple

version of non-temporal specification-based)

Compare results to previous version (Regression testing)

Input OutputSystem under test

Black Box Testing(Behavioral testing)

11

Testing Input-Output relationships only Pros

This is what the product is about. Implementation independent.

Cons For complicated products it is hard to identify

erroneous output. It is hard to estimate whether the product is

error-free. Practically: Choosing input with high probability of

error detection is very difficult (e.g. division of two numbers).

Black Box Testing(Behavioral testing)

Input Output

12

White Box Testing(Operational Testing)

Testing how input becomes output (including algorithms) Pros

Easier to detect errors. Enables to find better tests (direct the tests) The only way to check coverage.

Cons Implementation weaknesses are not necessarily

those of the product. Code is not always available

13

Dynamic testing (Run your program) Predefined tests

Good for Regression Testing (comparing an old version against a new one)

Testing the product under extreme conditions (stree-testing)

Random tests “real life” tests

Static testing (Inspect your code) Code analyzers (e.g., tools like lint and purify) Inspection (code review) Proofs (by tools, or by mathematical arguments):

“formal methods” (incl. Model-checking).

Static and Dynamic Testing

15

Special Testing Methods

.

Stress TestingA product that will work under heavy load

(e.g, on-line banking system) should be tested under increasing load - much heavier than expected.

16

Code analysis Unreachable code Objects declared and never used Parameters type/number mismatch Variable used before initialization Variable is assigned to twice, without

using the first value Function results not used Possible array bound violations

Static Testing

17

Code inspection Self - The default choice.

Subtle errors and micro-flaws may be overlooked.

Wrong conceptions propagate to review…

Code review by others - Very efficient !

Static Testing

18

One more quote…

Dijkstra:

“Testing can only prove the existence of bugs, not their absence…”

19

In general – it is undecidable, i.e. can’t be done. In most cases it is possible, but with manual

assistance – the same way we would prove a math theorem.

In some cases properties of software can be proved automatically.

Chances for errors increase with length of text Write short code (e.g, divide into more functions). Provide short proofs for correctness (even if they are

informal).

… So why not try to prove correctness?

20

Estimate how clean is your software

Error Implantation (For measuring the effectiveness of testing) Introduce errors. See how many of them are detected. This gives us an “educated guess” about

testing quality.

21

Estimate how much of the software’s behavior is covered

Coverage is a mean to estimate how rigorous is the testing effort

We can use coverage information in order to guide the process of test generation (some times even automatically)

22

Statement CoverageExample 1

int a, b, sum;int list1[10] = {0,1,2,3,4,5,6,7,8,9};int list2[10] = {9,8,7,6,5,4,3,2,1,0};

cin >> a >> b;if (a >= 0 && a <= 9) sum = list1[a];if (b >= 0 && b <= 9) sum = sum + list2[b];cout << sum << "\n";

23

Statement CoverageExample 1

if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)

sum = sum + list2[b]; Statement coverage may be achieved by

__ test case(s):

24

Statement CoverageExample 1

if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)

sum = sum + list2[b];

But statement coverage may not cater for all conditions... ... such as when a and b are beyond

the array size.

25

Branch CoverageSame Example 1

if (a >= 0 && a <= 9)

sum = list1[a];

if (b >= 0 && b <= 9)

sum = sum + list2[b];

Branch coverage may be achieved by __ test cases.

26

Branch coverage: Example

switch (x){

case 1: x = 1; break;

case 2: switch (y){

case 1: x = 3; break;

case 2: x = 2; break;

otherwise: x = 1; }

otherwise: 4;}

branch coverage may be achieved by __ test cases

28

if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)

sum = sum + list2[b];

else sum = 0;

Path coverage may be achieved by __ test cases:

Path CoverageSame Example 2

29

Path coverage

subsumes Branch coveragesubsumes Statement coverage

But can we always demand path coverage?

Subsumption Relationships

30

Branch CoverageExample 3

if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)

sum = sum + list2[b];

else sum = 0;

...

if (z >= 0 && z <= 9)

sum = sum + list26[b];

else sum = 0;

Branch coverage may be achieved by __ test cases

31

Path CoverageSame Example 3

if (a >= 0 && a <= 9)

sum = list1[a];

else sum = 0;

if (b >= 0 && b <= 9)

sum = sum + list2[b];

else sum = 0;

...

if (z >= 0 && z <= 9)

sum = sum + list26[b];

else sum = 0;

Path coverage may be achieved by __ test cases

32

Statement CoverageExample 4

sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;

Statement coverage may be achieved by __ test cases:

33

Branch CoverageSame Example 4

sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0;

Branch coverage may be achieved by __ test cases:

34

Loop coverage

Skip the loop entirely Only 1 pass through the loop 2 passes through the loop n–1, n and n+1 passes through the loop,

where n is the maximum number of allowable passes

35

Loop CoverageSame Example 4

sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];else sum = 0;

Loop coverage may be achieved by __ test cases

36

Path CoverageSame Example 4

sum = 0;

while (a >= 0 && a <= 9) {

sum = list1[a];

a = a + 1;

};

if (b >= 0 && b <= 9)

sum = list2[b];

else sum = 0; Path coverage may be achieved by __ test

cases

37

Path coverage

subsumes Branch coveragesubsumes Statement coverage

Subsumption Relationships

Path coverage

subsumes Loop coverage

But can we always demand path coverage?

38

Path CoverageExample 5

sum = 0;while (a >= 0 && a <= 9) { sum = list1[a]; a = a + 1; };while (b >= 0 && b <= 9) { sum = list2[a]; b = b + 1; };...while (z >= 0 && z <= 9) { sum = list26[z]; z = z + 1; };

Path coverage may be achieved by __ test cases

39

Path Coverage is not EverythingExample 1

z = x + y; Path coverage may be

achieved by 1 test case x = 8, y = 0

Cannot detect z = x - y;

x = 2, y = 2 Cannot detect

z = x * y;

x = 8, y = 9 Cannot detect z = 8 + y;

Cannot detect z = x + 9;

We need 2 test cases: x = 8, y = 9 x = 29, y = 18.

40

Path Coverage is not EverythingExample 2

int a[10];

if (b > 0)

a[b] = 1;

else …

Same path with b = 5 and b = 12 behave differently

41

Condition coverageExample

if (b1 ||(b2 && b3))

a = 1;

else …

Sub-expression coverage may be achieved by __ test cases

Every sub-expression in a predicate should be evaluated to TRUE and FALSE

42

Multiple condition coverageExample

if (b1 ||(b2 && b3))

a = 1;

else …

Multiple condition coverage may be achieved by __ test cases

Every Boolean combination of sub-expressions in a predicate should be evaluated to TRUE and FALSE

43

Condition/Branch coverage (MC / DC)(Modified Condition/Decision Coverage)

if (!b1 || b2)

a = 1;

else …

MC/DC coverage may be achieved by __ test cases

Union of Condition coverage and Branch coverage

This is an industry standard, introduced by Boeing

Modified – due to some constructs not present in C / C++.Decision coverage = Branch coverage

44

Specification-based testing

We will see an example of a system for specification-based testing of real-time applications.

It is a Run-time verification tool. The testing system is called “Logic Assurance” It monitors the specification, and can also

intervene in the execution of the program.

45

Monitoring

Logic Assurance

SystemUnderDevelopment

Environment Report State

control

Optional

Report stateEvent Reporting

=

control

Overview

46

The user should:

1. Specify rules:

Formalize parts of the specification by writing rules in the specification Language.

The rules should refer to events, the System Under Development (SUD)'s variables, external data etc..

47

A typical specification language: The LA Language (LAL)

Derived from Temporal Logic and C, LAL enables specification of:

Event order. Relative frequency ("fairness"). Timing demands. Logical and mathematical operators. More...

48

Examples (1/4):

1. Using event names, time directives and messages:

OPEN_DOOR follows CLOSE_DOOR after not more than 10 sec ?: message("Open door is late");

49

...Examples (2/4)

2. Logical operators and functions:

when time>20: time([CLOSE_DOOR])> time([OPEN_DOOR])+10?: message("CLOSE_DOOR is early");

50

...Examples (3/4)

3. User-defined functions are used to enhance the language and enable external control:

if JOB_DONE(10) then HEAT(3,5) < 30?: REDUCE_HEAT(5);

51

...Examples (4/4)

4. Locators are used to scan the log and retrieve event index:

[2nd SEND s.t. Time>=10, packet=5] > 0 ?: REDUCE_HEAT(5);

52

2 .Report events:

From inside the SUD by using the LA Report Facility.

From outside the SUD by using black-box techniques (e.g. OS events)

From the environment (Sensors, etc.)

53

The LA Report Facility:

The following directive should be inserted where the event occurs in the program:

LA_report_event (

int identifier,

float time-stamp,

user additional data...

)

54

LA will:

1. Keep an event log.

2. Analyze the rules in real time* (during execution) using the LA Analyzer.

* Off-line analysis is also possible

55

3. When a rule is violated, do one of the following:

Report the exact place and time the rule was violated.

Invoke a user-defined function.

56

Demo

A basic communication protocol ("Stop & Wait"):

2 2 23

time-out time-out

Rules:1. Resend packet after time-out…2. Do not resend packet that was acknowledged……

send

1

Ack 1

57

# event time message index arguments

-- ------ ------- ---------- ------- ------------

1: send 10 A 1 7, 10, 21..

2: Tout 15 A 1 8, 9 ,10..

3: send 16 B 2 20,30,21.. :

Rule 1, event 3: failed to resend packet

‘A’ sent at 10, Ack expected at 13..

Tout for A activated at 15, must send A again..

‘B’ sent at 16, Ack expected at 19..

.

Incoming events: messages:

user screen:

58

Advantages of real time monitoring:

Tests can be planned, maintained, expanded and applied throughout the development process.

Problems can be detected sooner.

The product is ‘tied’ to the specification.

59

A generic tool and methodology.

By receiving input from different sources, it enables testing of: Multiprocessor systems Dynamic environment systems.

Advantages of real time monitoring:

60

Enables temporal tests.

Enables smarter tests by using information from inside the program.

Advantages of real time monitoring: