Transcript
MUTATION TESTING
Woot?EXAMPLE 1
public void testMethod() {
Something something = new Something();
something.executeSomeMethod();
}
EXAMPLE 2
public void testConstructor() {
Something something = new Something();
}
LITTLE HISTORYFIRST PROPOSED in 1971 by Richard Lipton
A. Jefferson Offutt and R. H. Untch “Mutation 2000: Uniting the Orthogonal”
FIRST DEFINED in 1978
R. A. De Millo, R. J. Lipton, F. G. Sayward “Hints on test data selection”
FIRST IMPLEMENTED in 1980 by Timothy Budd
T. A. Budd, “Mutation Analysis of Program Test Data”
PROBLEM
SOLUTION
EXPECTATIONS
TESTS - TDD?
TESTS - NO TDD?
TEST
MUTATION
TESTING MUTATED CODE...
EXPECTED RESULT
SOMETHING WRONG?
FLOW1. Run tests2. Mutate production code3. Run tests again4. Generate report
KILL THE MUTANT!
TESTS RESULTS?PASSED (SURVIVED) FAILED (KILLED)
other results applicable (e. g. timeout, memory error)
EXAMPLEORIGINAL
if (i >= 0) {
return "foo";
} else {
return "bar";
}
MUTATED
if (i > 0) {
return "foo";
} else {
return "bar";
}
conditionals boundary mutation
COMMON MUTATORS #1CONDITIONALS MUTATORS
< -> <=
<= -> <
> -> >=
>= -> >
== -> !=
!= -> ==
MATH MUTATORS
+ -> -
- -> +
>> -> <<
<< -> >>
* -> /
/ -> *
COMMON MUTATORS #2RETURN VALUES MUTATOR
returns true if original returns false
returns false if original returns true
returns 0 if original returns int != 0
returns 1 if original returns 0
METHOD CALLS MUTATOR
removes void method call
removes non-void method call, returns default value for the return type
replaces members with null in constructors
SURVIVORS STORIES
HOW TO GET STARTED?PITEST for JAVA http://pitest.org
Parasoft INSURE++ for C/C++ https://www.parasoft.com/product/insure/
MutPy for Python https://pypi.python.org/pypi/MutPy/0.4.0
Maven: mvn org.pitest:pitest-maven:mutationCoverage
PERFORMANCE100 classes
10 tests / class
1 ms / test
tTOTAL = 100 x 10 x 1ms = 1s
10 mutations / class
tALL = 10 x 100 x 1s ~ 17min
tSMART = 10 x 100 x (10 x 1ms) = 10s
1000 classes
10 tests / class
1 ms / test
tTOTAL = 1000 x 10 x 1ms = 10s
10 mutations / class
tALL = 10 x 1000 x 10s ~ 28h
tSMART = 10 x 1000 x (10 x 1ms) ~ 2min
real life example: 481 test cases fully mutated on 16GB i7 -> 03:24 min
ATTRIBUTIONSHands Up - Avel Chuklanov - https://unsplash.com/photos/9cx4-QowgLc (CC0 1.0)
Bridge - Franc - https://thenounproject.com/search/?q=bridge&i=122542 (CC BY 3.0 US)
Cargo Truck - No way - https://thenounproject.com/search/?q=truck&i=154864 (CC0 1.0)
Philosoraptor - OnlyOneKenobi - http://wallpaperswide.com/philosoraptor-wallpapers.html
Chainsaw - James Keuning - https://thenounproject.com/search/?q=chainsaw&i=11328 (CC0 1.0)
Refugees - Gerald Wildmoser - https://thenounproject.com/search/?q=drown&i=208787 (CC BY 3.0 US)
Resurrection - Isabel Martinez Isabel - https://thenounproject.com/search/?q=survived&i=202842 (CC BY 3.0 US)
Pac Man - iconoci - https://thenounproject.com/search/?q=pac+man&i=9215 (CC BY 3.0 US)
Stopwatch - Nick Holroyd - https://thenounproject.com/search/?q=stopwatch&i=12666 (CC BY 3.0 US)
Zombie analysis - clement127 - https://www.flickr.com/photos/clement127/16089737804/ (CC BY-NC-ND 2.0)
in order of appearance
THANK YOU!Mateusz Bryła
top related