Live andLetDie Robust TestingtheFancyWay - improuv · Mutation Testing ☺ Generate a mutant by applying one well-defined change (mutation operation) to your production code (byte

© improuv GmbH Agile Leadership. http://improuv.com

Live and Let Die

Robust Testing the Fancy Way

Seb Heglmeier

Daniel Zappold

© improuv GmbH Agile Leadership | http://improuv.com

About us

improuv is an agile consultancy. We support enterprises on their agile journey.

Our expertise is Scrum, Kanban, and Lean & Agile software development.

Sebastian Heglmeier

@SebHeglmeier

github.com/CandleLightDoener

Daniel Zappold

@dzappold

github.com/dzappold


Mutation Testingin a Nutshell


A common problem:

OMG we have a BUG in production!!

• How could that happen?

• I thought there‘s enough tests!

• Are your tests any good at all?


Philosophical question

Quis custodiet ipsos custodes?

…

Who will guard the guards?


A real life problem

• Research at University of Gothenburg & Volvo [2]: car parking algorithm not

as robust against as expected when sensor measurements are a bit off


A real life problem

• Research at University of Gothenburg & Volvo [2]: car parking algorithm not

as robust against as expected when sensor measurements are a bit off


“Covered” doesn’t mean “well tested”

source: www.pitest.org


So …

How did they find these bad real-life scenarios

which were not covered by tests?


Mutation Testing ☺Generate a mutant by applying one well-defined change (mutation

operation) to your production code (byte code) and let your test suites run

against it. If none of your tests turns red, it means that this change was

not covered. This might be bad (it depends).

Note: there’s many mutants per class (typically 1 to 3 mutants per LOC)

Original code

4 mutants


Examples for typical Mutation Operators

Original Mutated

• Conditionals

• Change slightly < <=

• Negate == !=

• Remove if(a==b) if(true), if(false)

• Math Mutators?

• Negate + -

& |

• Return values

• Negate or replace some Object null

null throw RuntimeException

• Skip void calls someVoidMethod() …[not called]


Mutation Testing..why like this and not

differently?

a) stupid small mistake

b) Murphy’s law (mistakes couple)

c) big problem

� Typical Mutation Operators mimic exactly these small mistakes

source: wikipedia


History of Mutation Testing

• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize) as a

better way to measure the quality of your tests

• 1st implementation tool was a PhD work in 1980

More than 40 years old and you call that „fancy“?

• Recent availability of massive computing power has led to a resurgence

• Test Automation and Byte Code Manipulation speeden up things further


Example: Gilded Roseone badly written method which updates

the quality of (magical) items based on their

respective dates of expiry.


Some mutants survived

Number of

generated mutants

per line

This means: we could just alter the code like this .. and get away with it?


Mutation Test Results

KILLED

• A mutant is killed if a test fails (detecting the mutated code)

• This proves the mutated code is properly tested

SURVIVED

• A mutant didn‘t trigger a failing test ...

TIMED OUT

• The mutant caused the program loop, get stuck


What to do with survivors?

• Manual analysis of the surviving mutant

• Is the mutant’s behaviour equivalent to the orginal code? -> That’s OK!

• Otherwise: try to write red test and fix it

[3]


Conclusion on Mutation Testing

• It’s very powerful in detecting hidden defects even in well tested code

• It is computationally expensive (not that big of an issue I find)

• Equivalent mutants detection is still “research in progress” and involves

manual work (depending on your code: that could be a killer)

• Proper TDD would help avoid these kinds of (potential) bugs

• BUILDING quality in (instead of TESTING quality in)


References for Mutation Testing

• [1]

https://www.researchgate.net/publication/4200521_Is_mutation_an_appro

priate_tool_for_testing_experiments

• [2]

https://www.researchgate.net/publication/274079891_Early_Verification_a

nd_Validation_According_to_ISO_26262_by_Combining_Fault_Injection_an

d_Mutation_Testing

• [3] https://www.st.cs.uni-saarland.de/publications/files/gruen-mutation-

2009.pdf

• PIT Mutation Testing: http://pitest.org

• Gilded Rose: https://github.com/emilybache/GildedRose-Refactoring-Kata


Property Based Testingin a Nutshell


Are you familiar with something like this

• Losing overview with too many (similar) tests?

• Did we choose proper examples for our tests in order to fulfill our

specification?


What is Property Based Testing?

• Moving from examples to properties


Why Property Based Testing?

• Readability, understandability

• One property replaces many tests

• Better testing

• Lots of combinations you‘d never test by hand

• Less time spent on diagnosis

• Failures minimized automagically (since junit-quickcheck 0.6 (11.03.2016) also for java)


Gilded Rose

• Let‘s look at the requirements to spot a first property


Gilded Rose – Quality is Never Negative

• Let‘s write our first property based test


Gilded Rose – Ups ...

• Our test fails ... but was shrunken to ...


Gilded Rose – Wrong Input Data

• Limit your input


Gilded Rose – Respect Max Quality

• Limit your test data to correct input


Should be valid for every item


Gilded Rose – Quality is Never Negative

Independent from Item Name

• .


Gilded Rose – Writing Own Generator


Gilded Rose – Writing Own Generator


Coverage after first property.


Potential Pitfalls

• Using wrong test data set

• Re-implementing a function

• Can be difficult to determine what properties you should use

• Forces you to think carefully about what a function is doing

• Some generators are complex


Conclusion on Property Based Testing

• Writing good properties can be hard

• Writing good generators can be even harder

• PBTs are more general

• One proberty-based test can replace many example-based tests.

• PBTs can reveal overlooked edge cases

• Nulls, negative numbers, weird strings, etc.

• PBTs ensure deep understanding of requirements

• Property-based tests force you to think! �/☺

• Example-based tests are still helpful though!

• Easier to understand for newcomers, better for TDDing


References for PBT

• JUnit Quickcheck: https://github.com/pholser/junit-quickcheck

• Title Slide: http://www.teachthis.com.au/products/view-

resource/link/Properties-of-Living-Things/id/4008/

• Gilded Rose: https://github.com/emilybache/GildedRose-Refactoring-Kata

Live andLetDie Robust TestingtheFancyWay - improuv · Mutation Testing ☺ Generate a mutant by applying one well-defined change (mutation operation) to your production code (byte

Documents