Transcript
8/9/2019 Testing 4 White
1/54
1
Structural Testing& Mutation
Filippo RiccaDISI, Università di Genova, Italy
ricca@disi.unige.it
8/9/2019 Testing 4 White
2/54
22
White vs. Black box testing! A white box testing is based upon
explicit knowledge of the SUT and itsstructure
! Called also structural testing! Statement coverage testing is an example
! The aim is to create enough testcases to ensureevery statement is executed at least once
! A black box testing approach will device
testcases without any knowledge of theSUT or any aspect of its structure! Called also functional testing! Usually testcases are generated starting from
requirements/specifications
Inputs Outputs
Inputs Outputs
8/9/2019 Testing 4 White
3/54
3
What is adequacy?
! Fault detection! what if no faults are found?
!
How good this test suite is? Has P beentested thoroughly?, or Is T adequate?! Two possibilities:
! Test Coverage and program mutation 3
Test suite TP
8/9/2019 Testing 4 White
4/54
Test Coverage
4
8/9/2019 Testing 4 White
5/54
55
Test Coverage! Coverage can be based on:
! source code! model
! control flow graph! (extended) finite state machines! data flow graph
! requirements checklist! ...
8/9/2019 Testing 4 White
6/54
66
Coverage: what to measure?! For any coverage measure, we need:
! A coverage unit : an element with the properties:! We can count the total number of units in the software! We can identify which units were “hit” during a single
execution run
! This means that we can determine the percentageof units hit during one or more execution runs
! 89%
8/9/2019 Testing 4 White
7/54
7
Control flow coverage! Statement coverage! Branch coverage (also called decision coverage)
! Minimum coverage specified by the IEEE unit teststandard
! Multiple Condition coverage! Covers combinations of condition in decisions
! If (x=6) &&(y=7) then! Path coverage
! 100% path coverage impossible in practice! Loops ...
8/9/2019 Testing 4 White
8/54
8
Control Flow graphint proc(int a, int b, int x){
if ((a>1) && (b==0)) // 1{x = x/a; // 3
}if ((a==2)||(x>1)) // 4{
x = x+1; // 5
}return x; // 7}
a > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
9/54
9
Statement Coverage! Criterion: All statements must be covered
during test execution! This is the weakest form of coverage
! Some branches may be missed
Procedure:! Find paths that cover all statements! Choose input data that will result in the
selected paths
8/9/2019 Testing 4 White
10/54
10
Statement Coverage! The following path is sufficient
for statement coverage:
1 – 3 – 4 – 5 – 7
int proc(int a, int b, int x)
Possible input:
a = 2, b = 0, x = 4
a > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
11/54
11
Not covered branchesa > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
12/54
12
Branch Coverage! Criterion: At any branch point, each branch
must be covered during test execution! The true and false branch of a 2-way if statement!
Each case in a switch statement! Loops! While, for, goto, ….
Procedure:! Find paths that cover all branches! Choose input data that will result in the
selected paths
Branch coverage includes statement coverage!
8/9/2019 Testing 4 White
13/54
13
Branch Coverage! The following paths are
sufficient for branchcoverage:
1 – 2 – 4 – 5 – 71 – 3 – 4 – 6 – 7
! Possible input:
1.
a = 2, b = 2, x = -12. a = 3, b = 0, x = 1
a > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
14/54
14
Multiple Condition Coverage! Criterion:
! Every atomic (i.e. does not include AND or OR)condition must be true and false at some pointduring test execution
! In a compound logical statement (i.e. includes AND and OR), every combination of atomicconditions must be covered during test execution
! Achieving multiple condition coverage alsosatisfies statement and branch coverage
8/9/2019 Testing 4 White
15/54
15
Multiple Condition CoverageNeed cases where1. a > 1 is true and b = 0 is true2. a > 1 is true and b = 0 is false3. a > 1 is false and b = 0 is true4. a > 1 is false and b = 0 is false5. a = 2 is true and x > 1 is true6. a = 2 is true and x > 1 is false7. a = 2 is false and x > 1 is true8. a = 2 is false and x > 1 is false
int proc(int a, int b, int x){
if ( (a>1) && (b==0) ) {
x = x/a;}if ( (a==2) || (x>1) ) {
x = x+1;}return x;
}
8/9/2019 Testing 4 White
16/54
16
Multiple Condition Coverage
Possible input:a = 2, b = 0, x = 2 [1][5]a = 2, b = 1, x = 0 [2][6]a = 0, b = 0, x = 2 [3][7]a = 0, b = 1, x = 0 [4][8]
1. a > 1 is true and b = 0 is true
2. a > 1 is true and b = 0 is false
3.
a > 1 is false and b = 0 is true4. a > 1 is false and b = 0 is false
5. a = 2 is true and x > 1 is true
6. a = 2 is true and x > 1 is false
7. a = 2 is false and x > 1 is true
8. a = 2 is false and x > 1 is false
8/9/2019 Testing 4 White
17/54
17
Multiple Condition Coverage! Multiple condition coverage
covers all branches andstatements.
! Input values:a = 2, b = 0, x = 2 a = 2, b = 1, x = 0 a = 0, b = 0, x = 2 a = 0, b = 1, x = 0
! Paths covered1 – 3 – 4 – 5 – 7
1 – 2 – 4 – 5 – 71 – 2 – 4 – 5 – 71 – 2 – 4 – 6 – 7
a > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6Equal
8/9/2019 Testing 4 White
18/54
18
All Paths Coverage! Criterion:
! All paths through the code must be covered!
This is typically infeasible when loops arepresent! A version of this coverage with loops is to treat loops
as having two paths:1. The loop is executed (normally, once)2. The loop is skipped
! Some paths may also be infeasible because there is nocombination of data conditions that permit a path to betaken! See example below
8/9/2019 Testing 4 White
19/54
19
All Paths Coverage! Set of all paths:
1 – 2 – 4 – 6 – 71 – 3 – 4 – 6 – 7
1 – 2 – 4 – 5 – 71 – 3 – 4 – 5 – 7
! Input values:a = 0, b = 1, x = 0 a = 3, b = 0, x = 0 a = 2, b = 1, x = 0 a = 2, b = 0, x = 0
a > 1 ANDb = 0
a == 2OR x>1
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
20/54
2020
Comparison! From the previous two examples, we
can see that:! Multiple condition coverage does not imply
all paths coverage! All paths coverage does not imply multiple
condition coverage
8/9/2019 Testing 4 White
21/54
21
Infeasible paths! Set of paths:
1 – 2 – 4 – 6 – 71 – 3 – 4 – 6 – 7
1 – 2 – 4 – 5 – 71 – 3 – 4 – 5 – 7
! To be able to take this path, wewould have to have:
! a 4which is logically impossible!
a > 1
a > 4
x! x/a
x ! x+1
false
true
true
false
1
2
5
3
7
4
6
8/9/2019 Testing 4 White
22/54
22
Code Coverage Tools
! Clover:! http://www.cenqua.com/clover
! Emma:! http://emma.sourceforge.net! http://www.eclemma.org
! Coverlipse:! http://coverlipse.sourceforge.net
! Cobertura:! http://cobertura.sourceforge.net
! ....
22
8/9/2019 Testing 4 White
23/54
23
Emma
! Open-source tool! Supports class, method, and line coverage
! “Fractional” line coverage supported, but notbranch coverage
! Only part of a line covered
! Standalone version!
http://emma.sourceforge.net! Eclipse plugin EclEmma also available
! Installing the plug-in In Eclipse:! Help -> install new software -> http://update.eclemma.org/
8/9/2019 Testing 4 White
24/54
24
Emma coverage report
Testcases are written in Junit!
8/9/2019 Testing 4 White
25/54
25
Emma source code annotations
8/9/2019 Testing 4 White
26/54
26
Fractional line coverage Only partof conditionalexecuted
Loop incrementnot executed
" green for fully covered lines," yellow for partly covered lines and" red for lines that have not been executed at all
8/9/2019 Testing 4 White
27/54
8/9/2019 Testing 4 White
28/54
Output EMMA
Coverage = 100%
8/9/2019 Testing 4 White
29/54
Program Mutation
29
8/9/2019 Testing 4 White
30/54
30
What is program mutation?P is alterated in several istances (automatically)
! For several reasons:! Computing test adequacy of a testsuite! Improving/completing a testsuite! Detecting new errors
8/9/2019 Testing 4 White
31/54
31
Mutation terms
! Mutant: a copy of the original program with a small change (seeded fault)
! Mutation operator: applied to make change (automatically) to the original program
! Es. + --> *! Mutant killed: if its behaviors/outputs differ
from those of the original program! Live otherwise
31
8/9/2019 Testing 4 White
32/54
32
Mutation operator
32
8/9/2019 Testing 4 White
33/54
33
Method-level operator(examples)
33
Mutant operator In P In mutant
Variable replacement z=x*y+1; x=x*y+1;z=x*x+1;
Relational operatorreplacement
if (xy)if(x
8/9/2019 Testing 4 White
34/54
34
Class-level operator(examples)
! Access modifier change:The AMC operator changesthe access level for instance
variables and methods toother access levels
! Hiding variable deletion:The IHD operator deletes a
hiding variable, a variable ina subclass that has the samename and type as a variablein the parent class 34
8/9/2019 Testing 4 White
35/54
3535
Mutant score
testcase
Mutant killed: if its behaviors/outputs differ from those of the original program
8/9/2019 Testing 4 White
36/54
36
Test adequacy using mutation! Mutant/Mutation score can be also computed
for the entire TestsuiteMS(T)=number killed mutants/number mutants
! MS(T)=1 means that all the mutant are killed=> the testsuite is 100% adequate
! Best case ...! In same cases it is not possible to reach
MS(T)=1! Mutants equivalent to P!
! Ex. x+y --> x+y+036 Automatic tools should avoid it!
8/9/2019 Testing 4 White
37/54
37
Computing Adequacy …
Mutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67%
FooFoo’Foo’’Foo’’’
…
MS(FooTest)=?
Tools Usage
8/9/2019 Testing 4 White
38/54
38
Test enhancement using mutation
! One has the opportunity to enhance a test set Tafter having assessed its adequacy
! If the mutation score (MS) is 1, then some othertechnique, or a different set of mutants, needs to beused to help enhance T
! If the mutation score (MS) is less than 1, then thereexist live mutants. Each live mutant needs to bedistinguished from P
! Adding testcases to the testuite!!!!!
38
8/9/2019 Testing 4 White
39/54
39
Error detection using mutation
! There is no guarantee that tests derived todistinguish live mutants will reveal a yetundiscovered error in P
! Nevertheless, empirical studies have found to be themost powerful of all enhancement techniques
! The next simple example illustrates how “test
enhancement using mutation” detects errors
39
8/9/2019 Testing 4 White
40/54
8/9/2019 Testing 4 White
41/54
41
Foo (2)
! Next we execute each mutant against tests in T
41
int foo(int x, y) {return (x+y);
}
M1: int foo(int x, y) {return (x-0);
}
M2: int foo(int x, y) {return (0+y);
}
M3:
Test (t) foo(t) M1(t) M2(t) M3(t)
t1 1 1 1 0t2 -1 -1 -1 0
Live Live Killed
int foo(int x, y) {return (x-y);
}
T={ t1: , t2: }
Testsuite is not adequate!MS(T)= 1/3
mutants
8/9/2019 Testing 4 White
42/54
8/9/2019 Testing 4 White
43/54
43
Tools for mutation testing: features • A typical tool for mutation
testing offers the followingfeatures:
# A selectable palette of mutationoperators
# Management of test set T# Ex. Junit
# Generation of mutants# Execution of the program under test
and mutants against T# Report Mutation score (adequacy)
MuClipse
8/9/2019 Testing 4 White
44/54
44
Tools Free for mutation testing
! Proteum is the first mutation testing tool that implementsall mutation operators designed for the ANSI Cprogramming language
! ! Java (muJava) is a mutation system for Java programs! MuClipse is an Eclipse plugin verion of ! Java (muJava)
! Jumble is a mutation system for Java programs withJUnit tests. Jumble can be used from the command lineor as an Eclipse plugin
! The bytecode is mutated! Others at: http://www.mutationtest.net/twiki/bin/view/Resources/WebHome
44
8/9/2019 Testing 4 White
45/54
45
Jumble installation1. Download Jumble from:
! http://jumble.sourceforge.net
2. Unzip the downloaded file in a folder:
jumble_1_1_03. Copy or Move the folder jumble ( jumble_1_1_0 >
eclipseplugin > plugin-export ) to your Eclipse pluginsfolder:
! Eclipse >plugins, or Eclipse > dropins > eclipse >
plugins4. Restart Eclipse
45
8/9/2019 Testing 4 White
46/54
46
Run default mutation testing! If :
! and your test are inthe same package
! and your test’s name follows thispattern: Test.java,
! then you can runJumble
1. Right click on 2. Choose Jumble > Jumble Class3. Look at the Console view to see
the mutant score46
class to be mutated
8/9/2019 Testing 4 White
47/54
47
How Jumble work?! class "Foo“! JUnit tests "FooTest"! Jumble starts by running the unit tests (in
FooTest.class) on the unmodified Foo class to check that they all pass, and to measure the time taken byeach test
! Then it will mutate Foo in various ways and run thetests again to see if they detect the mutation
! It continues this process until all mutations of Foohave been tried
! It provides the output47
8/9/2019 Testing 4 White
48/54
48
Jumble Output
! Overall, 67% (8/12) of the mutations were killed by the unittests, which means that they probably need to be improved … 48
Mutating FooTests: FooTestMutation points = 12, unit test time limit 2.02s.. M FAIL: Foo:31: negated conditionalM FAIL: Foo:33: negated conditionalM FAIL: Foo:34: - -> +M FAIL: Foo:35: negated conditional......Score: 67%
Jumble found 12 different mutants of Foo
if (C) decision on line 31 was mutated to if (!C)unit tests kill the mutant (indicated by a '.')
8/9/2019 Testing 4 White
49/54
49
Jumble mutations! Conditionals : Jumble replaces each condition with its negation
! the condition x > y would mutate to become !(x > y)! Arithmetic Operations : Jumble replace arithmetic operations
! A fixed table is used!
+ --> -, * --> /! Increments : Increments, decrement are mutated
! for example, i++ becomes i--! Constants : Jumble can change the value of literal constants
! for example, 0 --> 1! Return Values : Jumble can change return values
! for example, return X --> return 0! Switch Statements : Jumble can mutate each case of a switch
! swapping49
8/9/2019 Testing 4 White
50/54
public class Calc {int tot;
Calc(){tot=0;
}
int somma ( int x) {
tot=tot + x;return tot;
}
int sottrai ( int x) {tot=tot - x;
return tot;
}int moltiplica ( int x) {
tot=tot * x;return tot;
}}
public class CalcTest {Calc c;
@Beforepublic void setUp(){
c= new Calc();}
@Testpublic void testSomma() {
Assert. assertEquals (c.somma(3), 3);}
@Testpublic void testSottrai() {
Assert. assertEquals (c.sottrai(0), 0);}
@Testpublic void testMoltiplica() {
c.somma(2); Assert. assertEquals (c.moltiplica(3), 6);
}}
Jumble Example
8/9/2019 Testing 4 White
51/54
51
Jumble output
Mutating CalcTests: CalcTest
Mutation points = 7, unit testtime limit 2.31s...M FAIL: Calc:15: - -> +...Score: 85%
public class Calc {int tot;
Calc(){tot=0;
}
int somma ( int x) {tot=tot + x;
return tot;}
int sottrai ( int x) {tot=tot + x;
return tot;}
int moltiplica ( int x) {tot=tot * x;return tot;
}}
Mutant non killed
8/9/2019 Testing 4 White
52/54
public class CalcTest {Calc c;
@Before
public void setUp(){c= new Calc();}
@Testpublic void testSomma() {
Assert. assertEquals (c.somma(3), 3);}
@Testpublic void testSottrai() {
Assert. assertEquals (c.sottrai(3), -3);}
@Test
public void testMoltiplica() {c.somma(2); Assert. assertEquals (c.moltiplica(3), 6);
}}
Mutating CalcTests: CalcTestMutation points = 7, unit test
time limit 2.31s.......Score: 100%
Test enhancement
8/9/2019 Testing 4 White
53/54
53
Possible exercises at the exam! Given a piece of code:
! Build the CFG! Find paths that cover all statements, branches, conditions,
or paths! Choose input data that will result in the selected paths! See exercise “Delete rows from array” at
! http://www.site.uottawa.ca/~awilliam/seg3203/Coverage.ppt
! Enhance a test set T after having assessed itsadequacy
! Given P, T (Junit tests) and a set of Jumble mutations! Foo:33: negated conditional
! Error detection using mutation! Ex. Foo 53
8/9/2019 Testing 4 White
54/54
5454
References(used to prepare these slides)
! Slides of the book “Foundations of software testing”by Aditya P. Mathur
! http://www.cs.purdue.edu/homes/apm/foundationsBook/InstructorSlides.html
! Slides of Alan Williams, University of Ottawa! http://www.site.uottawa.ca/~awilliam/seg3203 /
! Glover, “Don’t be fooled by the Coverage Report”,IBM developer works article
! http://www-128.ibm.com/developerworks/java/library/j-cq01316! Slides of Cu Nguyen Duy, Software Analysis and
Testing 2009-2010! http://selab.fbk.eu/swat/program.ml?lang=en
! Wikipedia
top related