Can you trust your tests?

CanYou Trust Your Tests?

2016 Vaidas Pilkauskas & Tadas Ščerbinskas

An Introduction to Mutation Testing

Thanks to co-author

Tadas ŠčerbinskasVilniusRB organizer

@tadassce

Van Halen

Band Tour Rider

Van Halen’s 1982 Tour Rider

Agenda

1. Test quality & code coverage2. Mutation testing in theory3. Mutation testing in practice

Prod vs. Test code quality

Code has bugs.

Tests are code.

Tests have bugs.

Test quality

Readable

Focused

Concise

Well named

“Program testing can be used to show the presence of bugs, but never to

show their absence!”- Edsger W. Dijkstra

Code Coverage

Types of Code Coverage

Lines

Branches

Instructions

Cyclomatic Complexity

Methods

& more

Lines

String foo() { return "a" + "b";}

assertThat(a.foo(), is("ab"))

Lines

String foo(boolean arg) { return arg ? "a" : "b";}

assertThat(a.foo(true), is("a"));

Branches

String foo(boolean arg) { return arg ? "a" : "b";}

assertThat(a.foo(true), is("a"));

assertThat(a.foo(false), is("b"));

SUCCESS: 26/26 (100%) Tests passed

Can you trust 100% coverage?

Code coverage can only show what is not tested.

For interpreted languages 100% code coverage is equivalent to compilation.

Code Coverage can be gamed

On purpose or by accident

Mutation testing

Mutation testing

Changes your program code and expects your tests to fail.

What exactly is a mutation?

Boolean isFoo(int a) { return a == foo;}

Boolean isFoo(int a) { return a != foo;}

Boolean isFoo(int a) { return true;}

Boolean isFoo(int a) { return null;}

>>>

Terminology

Applying a mutation to some code creates a mutant.

If test passes - mutant has survived.

If test fails - mutant is killed.

Failing is the new passing

array = [a, b, c]

max(array) == ???

// testassertThat(Max.of(0), equalTo(0)); ✘

// testassertThat(Max.of(0), equalTo(0)); ✔

// implementationpublic static int of(int... integers) { return 0;}

// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✘

// implementationpublic static int of(int... integers) { return 0;}

// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔

// implementationpublic static int of(int... integers) { return integers[0];}

// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✘

// implementationpublic static int of(int... integers) { return integers[0];}

// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔

// implementationpublic static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max;}

Coverage

Mutation// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔

// implementationpublic static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max;}

Mutation// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔

// implementationpublic static int of(int... integers) { int max = integers[0]; for (int i : integers) if (true) max = i; return max;}

Mutation// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔

// implementationpublic static int of(int... integers) { return integers[integers.length - 1];}

Mutation// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔assertThat(Max.of(2, 1), equalTo(2)); ✘

// implementationpublic static int of(int... integers) { return integers[integers.length - 1];}

Mutation// testassertThat(Max.of(0), equalTo(0)); ✔assertThat(Max.of(1), equalTo(1)); ✔assertThat(Max.of(1, 2), equalTo(2)); ✔assertThat(Max.of(2, 1), equalTo(2)); ✔

// implementationpublic static int of(int... integers) { int max = integers[0]; for (int i : integers) if (max < i) max = i; return max;}

Baby steps matter

Tests’ effectiveness is measured by number of killed mutants by

your test suite.

It’s like hiring a white-hat hacker to try to break into your server and making sure you detect it.

What if mutant survives

● Simplify your code● Add additional tests

● TDD - minimal amount of code to pass the test

Challenges

1. High computation cost - slow2. Equivalent mutants - false negatives3. Infinite loops

Equivalent mutations

// Originalint i = 0;while (i != 10) { doSomething(); i += 1;}

// Mutantint i = 0;while (i < 10) { doSomething(); i += 1;}

Infinite Runtime

// Originalwhile (expression) doSomething();

// Mutantwhile (true) doSomething();

● Let’s say we have codebase with:● 300 classes● around 10 tests per class● 1 test runs around 1ms● total test suite runtime is about 3s

Is it slow?

Let’s do 10 mutations per class● We get 3000 (300 classes * 10

mutations) mutations● runtime with all mutations is 150 minutes (3s * 3000)

Speeding it up

Run only tests that cover the mutation● 300 classes● 10 tests per class● 10 mutations per class● 1ms test runtime● total mutation runtime

10 tests * 10 mutations * 1 ms * 300 classes = 30 s

Speeding it up

During development run tests that cover only your current changes

● Continuous integration● TDD with mutation testing only

on new changes● Add mutation testing to your

legacy project, but do not fail a build - produce a warning report

Usage scenarios

Tools

● Ruby - Mutant● Java - PIT● And many tools for other

languages

Pit● Active project● Good reporting● No Scala support :(

Pit

PIT originally stood for Parallel Isolated Test.

Now it stands for PIT.

Test Selection

● Line coverage● Test execution speed● Test naming convention

Running PIT

● Command line tool● Maven● Ant● Gradle

PIT supports many mocking frameworks

● Mockito● EasyMock● JMock● PowerMock● JMockit

Maven usage

mvn org.pitest:pitest-maven:mutationCoverage

With SCM plugin

mvn org.pitest:pitest-maven:scmMutationCoverage \

-Dinclude=ADDED,UNKNOWN -DmutationThreshold=85

Configuration

● Dependency distance● Limit mutation number per class● Activate/deactivate mutators

● Extension points - Java SPI

What about TDD?

● Influences code style● Helps to spot dead features● Puts emphasis on minimal code

to pass the test

Summary

Code coverage highlights code that is not tested.

It shows which code you have executed in tests.

Summary

Mutation testing highlights code that is tested.

It shows which code you have asserted in tests.

How much we suck

Vaidas Pilkauskas

@liucijus

● Vilnius JUG co-founder

● Vilnius Scala leader

● Coderetreat facilitator

● Mountain bicycle rider

● Snowboarder

About me

Q&A

Credits

A lot of presentation content is based on work by these guys

● Markus Schirp - author of Mutant● Henry Coles - author of PIT● Filip Van Laenen - working on a book