Testing 4 White

8/9/2019 Testing 4 White

1/54

1

Structural Testing& Mutation

Filippo RiccaDISI, Università di Genova, Italy

[email protected]


2/54

22

White vs. Black box testing! A white box testing is based upon

explicit knowledge of the SUT and itsstructure

! Called also structural testing! Statement coverage testing is an example

! The aim is to create enough testcases to ensureevery statement is executed at least once

! A black box testing approach will device

testcases without any knowledge of theSUT or any aspect of its structure! Called also functional testing! Usually testcases are generated starting from

requirements/specifications

Inputs Outputs

Inputs Outputs


3/54

3

What is adequacy?

! Fault detection! what if no faults are found?

!

How good this test suite is? Has P beentested thoroughly?, or Is T adequate?! Two possibilities:

! Test Coverage and program mutation 3

Test suite TP


4/54

Test Coverage

4


5/54

55

Test Coverage! Coverage can be based on:

! source code! model

! control flow graph! (extended) finite state machines! data flow graph

! requirements checklist! ...


6/54

66

Coverage: what to measure?! For any coverage measure, we need:

! A coverage unit : an element with the properties:! We can count the total number of units in the software! We can identify which units were “hit” during a single

execution run

! This means that we can determine the percentageof units hit during one or more execution runs

! 89%


7/54

7

Control flow coverage! Statement coverage! Branch coverage (also called decision coverage)

! Minimum coverage specified by the IEEE unit teststandard

! Multiple Condition coverage! Covers combinations of condition in decisions

! If (x=6) &&(y=7) then! Path coverage

! 100% path coverage impossible in practice! Loops ...


8/54

8

Control Flow graphint proc(int a, int b, int x){

if ((a>1) && (b==0)) // 1{x = x/a; // 3

}if ((a==2)||(x>1)) // 4{

x = x+1; // 5

}return x; // 7}

a > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


9/54

9

Statement Coverage! Criterion: All statements must be covered

during test execution! This is the weakest form of coverage

! Some branches may be missed

Procedure:! Find paths that cover all statements! Choose input data that will result in the

selected paths


10/54

10

Statement Coverage! The following path is sufficient

for statement coverage:

1 – 3 – 4 – 5 – 7

int proc(int a, int b, int x)

Possible input:

a = 2, b = 0, x = 4

a > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


11/54

11

Not covered branchesa > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


12/54

12

Branch Coverage! Criterion: At any branch point, each branch

must be covered during test execution! The true and false branch of a 2-way if statement!

Each case in a switch statement! Loops! While, for, goto, ….

Procedure:! Find paths that cover all branches! Choose input data that will result in the

selected paths

Branch coverage includes statement coverage!


13/54

13

Branch Coverage! The following paths are

sufficient for branchcoverage:

1 – 2 – 4 – 5 – 71 – 3 – 4 – 6 – 7

! Possible input:

1.

a = 2, b = 2, x = -12. a = 3, b = 0, x = 1

a > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


14/54

14

Multiple Condition Coverage! Criterion:

! Every atomic (i.e. does not include AND or OR)condition must be true and false at some pointduring test execution

! In a compound logical statement (i.e. includes AND and OR), every combination of atomicconditions must be covered during test execution

! Achieving multiple condition coverage alsosatisfies statement and branch coverage


15/54

15

Multiple Condition CoverageNeed cases where1. a > 1 is true and b = 0 is true2. a > 1 is true and b = 0 is false3. a > 1 is false and b = 0 is true4. a > 1 is false and b = 0 is false5. a = 2 is true and x > 1 is true6. a = 2 is true and x > 1 is false7. a = 2 is false and x > 1 is true8. a = 2 is false and x > 1 is false

int proc(int a, int b, int x){

if ( (a>1) && (b==0) ) {

x = x/a;}if ( (a==2) || (x>1) ) {

x = x+1;}return x;

}


16/54

16

Multiple Condition Coverage

Possible input:a = 2, b = 0, x = 2 [1][5]a = 2, b = 1, x = 0 [2][6]a = 0, b = 0, x = 2 [3][7]a = 0, b = 1, x = 0 [4][8]

1. a > 1 is true and b = 0 is true

2. a > 1 is true and b = 0 is false

3.

a > 1 is false and b = 0 is true4. a > 1 is false and b = 0 is false

5. a = 2 is true and x > 1 is true

6. a = 2 is true and x > 1 is false

7. a = 2 is false and x > 1 is true

8. a = 2 is false and x > 1 is false


17/54

17

Multiple Condition Coverage! Multiple condition coverage

covers all branches andstatements.

! Input values:a = 2, b = 0, x = 2 a = 2, b = 1, x = 0 a = 0, b = 0, x = 2 a = 0, b = 1, x = 0

! Paths covered1 – 3 – 4 – 5 – 7

1 – 2 – 4 – 5 – 71 – 2 – 4 – 5 – 71 – 2 – 4 – 6 – 7

a > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6Equal


18/54

18

All Paths Coverage! Criterion:

! All paths through the code must be covered!

This is typically infeasible when loops arepresent! A version of this coverage with loops is to treat loops

as having two paths:1. The loop is executed (normally, once)2. The loop is skipped

! Some paths may also be infeasible because there is nocombination of data conditions that permit a path to betaken! See example below


19/54

19

All Paths Coverage! Set of all paths:

1 – 2 – 4 – 6 – 71 – 3 – 4 – 6 – 7

1 – 2 – 4 – 5 – 71 – 3 – 4 – 5 – 7

! Input values:a = 0, b = 1, x = 0 a = 3, b = 0, x = 0 a = 2, b = 1, x = 0 a = 2, b = 0, x = 0

a > 1 ANDb = 0

a == 2OR x>1

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


20/54

2020

Comparison! From the previous two examples, we

can see that:! Multiple condition coverage does not imply

all paths coverage! All paths coverage does not imply multiple

condition coverage


21/54

21

Infeasible paths! Set of paths:

1 – 2 – 4 – 6 – 71 – 3 – 4 – 6 – 7

1 – 2 – 4 – 5 – 71 – 3 – 4 – 5 – 7

! To be able to take this path, wewould have to have:

! a 4which is logically impossible!

a > 1

a > 4

x! x/a

x ! x+1

false

true

true

false

1

2

5

3

7

4

6


22/54

22

Code Coverage Tools

! Clover:! http://www.cenqua.com/clover

! Emma:! http://emma.sourceforge.net! http://www.eclemma.org

! Coverlipse:! http://coverlipse.sourceforge.net

! Cobertura:! http://cobertura.sourceforge.net

! ....

22


23/54

23

Emma

! Open-source tool! Supports class, method, and line coverage

! “Fractional” line coverage supported, but notbranch coverage

! Only part of a line covered

! Standalone version!

http://emma.sourceforge.net! Eclipse plugin EclEmma also available

! Installing the plug-in In Eclipse:! Help -> install new software -> http://update.eclemma.org/


24/54

24

Emma coverage report

Testcases are written in Junit!


25/54

25

Emma source code annotations


26/54

26

Fractional line coverage Only partof conditionalexecuted

Loop incrementnot executed

" green for fully covered lines," yellow for partly covered lines and" red for lines that have not been executed at all


27/54


28/54

Output EMMA

Coverage = 100%


29/54

Program Mutation

29


30/54

30

What is program mutation?P is alterated in several istances (automatically)

! For several reasons:! Computing test adequacy of a testsuite! Improving/completing a testsuite! Detecting new errors


31/54

31

Mutation terms

! Mutant: a copy of the original program with a small change (seeded fault)

! Mutation operator: applied to make change (automatically) to the original program

! Es. + --> *! Mutant killed: if its behaviors/outputs differ

from those of the original program! Live otherwise

31


32/54

32

Mutation operator

32


33/54

33

Method-level operator(examples)

33

Mutant operator In P In mutant

Variable replacement z=x*y+1; x=x*y+1;z=x*x+1;

Relational operatorreplacement

if (xy)if(x


34/54

34

Class-level operator(examples)

! Access modifier change:The AMC operator changesthe access level for instance

variables and methods toother access levels

! Hiding variable deletion:The IHD operator deletes a

hiding variable, a variable ina subclass that has the samename and type as a variablein the parent class 34


35/54

3535

Mutant score

testcase

Mutant killed: if its behaviors/outputs differ from those of the original program


36/54

36

Test adequacy using mutation! Mutant/Mutation score can be also computed

for the entire TestsuiteMS(T)=number killed mutants/number mutants

! MS(T)=1 means that all the mutant are killed=> the testsuite is 100% adequate

! Best case ...! In same cases it is not possible to reach

MS(T)=1! Mutants equivalent to P!

! Ex. x+y --> x+y+036 Automatic tools should avoid it!


37/54

37

Computing Adequacy …

Mutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67%

FooFoo’Foo’’Foo’’’

…

MS(FooTest)=?

Tools Usage


38/54

38

Test enhancement using mutation

! One has the opportunity to enhance a test set Tafter having assessed its adequacy

! If the mutation score (MS) is 1, then some othertechnique, or a different set of mutants, needs to beused to help enhance T

! If the mutation score (MS) is less than 1, then thereexist live mutants. Each live mutant needs to bedistinguished from P

! Adding testcases to the testuite!!!!!

38


39/54

39

Error detection using mutation

! There is no guarantee that tests derived todistinguish live mutants will reveal a yetundiscovered error in P

! Nevertheless, empirical studies have found to be themost powerful of all enhancement techniques

! The next simple example illustrates how “test

enhancement using mutation” detects errors

39


40/54


41/54

41

Foo (2)

! Next we execute each mutant against tests in T

41

int foo(int x, y) {return (x+y);

}

M1: int foo(int x, y) {return (x-0);

}

M2: int foo(int x, y) {return (0+y);

}

M3:

Test (t) foo(t) M1(t) M2(t) M3(t)

t1 1 1 1 0t2 -1 -1 -1 0

Live Live Killed

int foo(int x, y) {return (x-y);

}

T={ t1: , t2: }

Testsuite is not adequate!MS(T)= 1/3

mutants


42/54


43/54

43

Tools for mutation testing: features • A typical tool for mutation

testing offers the followingfeatures:

# A selectable palette of mutationoperators

# Management of test set T# Ex. Junit

# Generation of mutants# Execution of the program under test

and mutants against T# Report Mutation score (adequacy)

MuClipse


44/54

44

Tools Free for mutation testing

! Proteum is the first mutation testing tool that implementsall mutation operators designed for the ANSI Cprogramming language

! ! Java (muJava) is a mutation system for Java programs! MuClipse is an Eclipse plugin verion of ! Java (muJava)

! Jumble is a mutation system for Java programs withJUnit tests. Jumble can be used from the command lineor as an Eclipse plugin

! The bytecode is mutated! Others at: http://www.mutationtest.net/twiki/bin/view/Resources/WebHome

44


45/54

45

Jumble installation1. Download Jumble from:

! http://jumble.sourceforge.net

2. Unzip the downloaded file in a folder:

jumble_1_1_03. Copy or Move the folder jumble ( jumble_1_1_0 >

eclipseplugin > plugin-export ) to your Eclipse pluginsfolder:

! Eclipse >plugins, or Eclipse > dropins > eclipse >

plugins4. Restart Eclipse

45


46/54

46

Run default mutation testing! If :

! and your test are inthe same package

! and your test’s name follows thispattern: Test.java,

! then you can runJumble

1. Right click on 2. Choose Jumble > Jumble Class3. Look at the Console view to see

the mutant score46

class to be mutated


47/54

47

How Jumble work?! class "Foo“! JUnit tests "FooTest"! Jumble starts by running the unit tests (in

FooTest.class) on the unmodified Foo class to check that they all pass, and to measure the time taken byeach test

! Then it will mutate Foo in various ways and run thetests again to see if they detect the mutation

! It continues this process until all mutations of Foohave been tried

! It provides the output47


48/54

48

Jumble Output

! Overall, 67% (8/12) of the mutations were killed by the unittests, which means that they probably need to be improved … 48

Mutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67%

Jumble found 12 different mutants of Foo

if (C) decision on line 31 was mutated to if (!C)unit tests kill the mutant (indicated by a '.')


49/54

49

Jumble mutations! Conditionals : Jumble replaces each condition with its negation

! the condition x > y would mutate to become !(x > y)! Arithmetic Operations : Jumble replace arithmetic operations

! A fixed table is used!

+ --> -, * --> /! Increments : Increments, decrement are mutated

! for example, i++ becomes i--! Constants : Jumble can change the value of literal constants

! for example, 0 --> 1! Return Values : Jumble can change return values

! for example, return X --> return 0! Switch Statements : Jumble can mutate each case of a switch

! swapping49


50/54

public class Calc {int tot;

Calc(){tot=0;

}

int somma ( int x) {

tot=tot + x;return tot;

}

int sottrai ( int x) {tot=tot - x;

return tot;

}int moltiplica ( int x) {

tot=tot * x;return tot;

}}

public class CalcTest {Calc c;

@Beforepublic void setUp(){

c= new Calc();}

@Testpublic void testSomma() {

Assert. assertEquals (c.somma(3), 3);}

@Testpublic void testSottrai() {

Assert. assertEquals (c.sottrai(0), 0);}

@Testpublic void testMoltiplica() {

c.somma(2); Assert. assertEquals (c.moltiplica(3), 6);

}}

Jumble Example


51/54

51

Jumble output

Mutating CalcTests: CalcTest

Mutation points = 7, unit testtime limit 2.31s...M FAIL: Calc:15: - -> +...Score: 85%

public class Calc {int tot;

Calc(){tot=0;

}

int somma ( int x) {tot=tot + x;

return tot;}

int sottrai ( int x) {tot=tot + x;

return tot;}

int moltiplica ( int x) {tot=tot * x;return tot;

}}

Mutant non killed


52/54

public class CalcTest {Calc c;

@Before

public void setUp(){c= new Calc();}

@Testpublic void testSomma() {

Assert. assertEquals (c.somma(3), 3);}

@Testpublic void testSottrai() {

Assert. assertEquals (c.sottrai(3), -3);}

@Test

public void testMoltiplica() {c.somma(2); Assert. assertEquals (c.moltiplica(3), 6);

}}

Mutating CalcTests: CalcTestMutation points = 7, unit test

time limit 2.31s.......Score: 100%

Test enhancement


53/54

53

Possible exercises at the exam! Given a piece of code:

! Build the CFG! Find paths that cover all statements, branches, conditions,

or paths! Choose input data that will result in the selected paths! See exercise “Delete rows from array” at

! http://www.site.uottawa.ca/~awilliam/seg3203/Coverage.ppt

! Enhance a test set T after having assessed itsadequacy

! Given P, T (Junit tests) and a set of Jumble mutations! Foo:33: negated conditional

! Error detection using mutation! Ex. Foo 53


54/54

5454

References(used to prepare these slides)

! Slides of the book “Foundations of software testing”by Aditya P. Mathur

! http://www.cs.purdue.edu/homes/apm/foundationsBook/InstructorSlides.html

! Slides of Alan Williams, University of Ottawa! http://www.site.uottawa.ca/~awilliam/seg3203 /

! Glover, “Don’t be fooled by the Coverage Report”,IBM developer works article

! http://www-128.ibm.com/developerworks/java/library/j-cq01316! Slides of Cu Nguyen Duy, Software Analysis and

Testing 2009-2010! http://selab.fbk.eu/swat/program.ml?lang=en

! Wikipedia

Testing 4 White

Documents