David Pryor. Mutation-Based Testing Same basic goal as Code Coverage Evaluate the tests Determine how much code exercised Mutation testing goes beyond.

• Slide 1

David Pryor Slide 2 Mutation-Based Testing Same basic goal as Code Coverage Evaluate the tests Determine how much code exercised Mutation testing goes beyond checking which lines of code were executed Goal: Distinguish statements that are simply executed from those that are fully tested History Theory published in 1971 by a student Computationally Infeasible Technological advances: 90s present Still not mainstream, but gaining popularity 2 Slide 3 Basic Procedure Requirements Complete working program Tests written and passing Make a small change(mutation) to the source code of the program Run the tests If a test fails: The test killed this mutant If all tests still pass: Redundant/Unnecessary code Tests are incomplete 3 Slide 4 Example test functions This test achieves 100% code coverage testAbs1 will never fail Even if abs(-3) == -3 Mutation testing can detect this testAbs1 will initially pass It will still pass on mutated code It failed to kill the mutant The test is inadequate 4 Slide 5 Example test functions testAbs2 passes initially On the mutated function, it fails abs(-5) != 5 Test is supposed to fail It shows the test is robust enough to catch errors such as this mutation It killed the mutant 5 Slide 6 Mutation Operators Types of mutations to be applied Types of changes to make to the code Defined by the testing framework Chosen by the user Goals Introduce errors Simulate common coding bugs Ensure the testing of all possible circumstances Traditional Mutation Operators Simple Common Included in most mutation frameworks 6 Slide 7 Dropped Statement Removes a single statement from the program Unnecessary code Needs to be selective Many possible mutants 7 Slide 8 Arithmetic Operator Replacement Swaps arithmetic operators +, -, *, /, % Some frameworks have set translations + always becomes * Others allow any/random swaps 8 Slide 9 Boolean Relation Replacement Swaps boolean relations ==, !=,, >= Tightly constrained form Only mutates to/from similar relations < toto >= == to != 9 Slide 10 Boolean Expression Replacement Replaces an entire expression with true or false Unnecessary Code Code paths that arent sufficiently tested 10 Slide 11 Variable Replacement Replaces a variable reference with a different variable reference The new variable must be accessible and defined in the same scope Not trying to create compiler errors False positives Unnecessary/Duplicate variables 11 Slide 12 Non-Traditional Operators Lots of Operators out there Bit operation / Binary operators Shift/Rotate replacement Increments / Decrements Invert Negatives Should be able to define your own for customized testing Ideally: Minimize false positives Reasonable number of created mutants 12 Slide 13 Replace inline constants Replaces an inline constant Numeric or string Non-Deterministic Tests for responses more than behavior Requires many tests that may not be helpful Used in Heckle(Ruby) 13 Slide 14 Object-Oriented Operators Encapsulation Changing access modifiers Inheritance Hiding variables Method overriding Parent / super actions Polymorphism Lots of operators here Basic Idea: change something between the parent and the child in the usage of an object 14 Slide 15 Concurrency Operators Modify sleep/wait calls Mutual exclusion and semaphores Change boundaries of the critical section Change synchronization details Switch concurrent objects Others Goal is the same Evaluate the adequacy of the tests 15 Slide 16 Computation Time The biggest problem/roadblock with Mutation testing Theory has existed for 40+ years Computationally infeasible for use in industry for a long time Frameworks can automate almost everything, but: Every mutant has to be run against the entire test suite Many mutants and a large test suite cause immense testing times 16 Slide 17 Computation Time Estimation T = Time to run test suite against the code base M = # of Mutation Operators in use (3-20+) N = # of Mutants per Operator (depends on code base size) Mutation testing causes time to increase from T to T*M*N Minimum time increase of a factor of 30 For small code base and few operators Total time increases very quickly Only time needed to run tests, not compilation If T = 1 minute, T*M*N can become hours or days If T = 1 hour, T*M*N can become weeks or longer 17 Slide 18 Addressing Computation Time Need to spend less time on mutation testing Variety of methods that fall into three categories Do Less Test fewer mutants and mutation operators Need to be careful Fewer may result in poor tests slipping through Do Faster Increase the speed of execution Do Smarter Eliminate mutants and mutation operators that do not provide meaningful results 18 Slide 19 Source code or Byte code? Can perform mutations on the source code itself, but: Large code bases result in lots of slow disk reads Have to compile EVERY mutant Instead, compile the original source once Mutate the compiled byte code Much faster Can be difficult to back-trace the byte code to the source code to show the mutants that were created 19 Slide 20 Weak vs. Strong Testing Two conditions for killing a mutant Testing data should cause a different program state for the mutant than for the original For example, a test results in: valid = false done = true The difference in state should result in a difference in output and be checked by the test In this case: the test should check result Weak Testing: Only satisfy the first condition Strong Testing: Both conditions 20 Slide 21 Weak vs. Strong Testing Weak assures that the tests cause a difference Not assured that they check the difference Not as thorough Strong is ideal Computationally expensive Must always run to the end Weak can stop as soon as it detects a difference in state 21 Slide 22 Incremental Analysis Currently experimental Most useful for long-term projects with large code bases, which use mutation testing over and over Basic Idea: save state and results of tests and code Only re-run those tests and mutants for which relevant code has changed Decide which to skip based on changes made Not perfected yet can be tricked by odd behavior 22 Slide 23 Selective Mutation Goal: Eliminate some mutants or operators that are not necessary Mutants Remove duplicates caused by multiple operators Remove likely duplicates those that will probably cause duplicate results Some amount of error here Mutation Operators Some pairs of operators might produce many of the same mutants An operator might produce a subset of mutants from a different operator Detection can be difficult 23 Slide 24 Coverage Based Test Selection Typical mutant only affects a single statement / line Typical test suite has many tests that do not execute this line No need to run these tests on the mutant Only run tests that exercise the mutated code Optimize the running order of tests 24 Slide 25 Other Problems and Design Considerations Equivalent Mutants This mutant is functionally equivalent to the original No test that calls this code could ever distinguish the two Some mutants cant be killed Can sometimes be detected automatically and filtered out Not all can be detected Requires human effort to determine if equivalent Not always an easy task 25 Slide 26 Other Problems and Design Considerations Mutant Infinite Loops Some mutations can cause infinite loops in the mutants Statement deletion removed the only way out of this loops Solution: Time the un-mutated code If the mutant takes significantly longer than this time Probably an infinite loop Timeout after the un-mutated time, plus some padding 26 Slide 27 Other Problems and Design Considerations Complex Bugs Mutation only makes small changes What if the test cases miss a large, complex error? Mutation doesnt create complex mutants Coupling Effect Hypothesis Tests that detect simple errors are sensitive enough to detect more complex errors Supported by empirical data Testing for simple errors helps to find the more complex ones 27 Slide 28 Fuzz Testing / Fuzzing Completely unrelated to Mutation testing Often confused with Mutation Involves generating random data to use as input to a program Test security/vulnerability See if the program crashes Fuzzing modifies input Checks program behavior Mutation modifies source code Checks test case results Deterministic 28 Slide 29 Tools and Environments Java MuJava Bacterio Javalanche Jumble PIT Jester C/C++ Insure++ Fortran Mothra PHP Mutagenesis Ruby Heckle Mutant C# Nester 29 Slide 30 Using Mutation Testing in Industry Use Mutation from the beginning of a project Dont use it with dangerous methods Look for high quality tools/environments Speed optimizations Reporting/Coverage information Configurable Operators 30 Slide 31 Benefits of Mutation testing Evaluation of tests / test suite More than code coverage: are the tests adequate? Mutation Score: % of non-equivalent mutants killed Evaluation of code Find unreachable or redundant code Find bugs that were hidden through inadequate tests Future: Automatic Test Generation Create dummy tests, use Mutation to revise Repeat until all or most non-equivalent mutants killed Still experimental, but promising 31 Slide 32 Questions 32 Slide 33 References Alexander, R. T., & Bieman, J. M. (2002). Mutation of java objects. IEEE Int. Symp. Software Reliability Engineering, Retrieved from http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf Offutt, A. J. (n.d.). A practical system for mutation testing: Help for the common programmer. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdf Offutt, A. J., & Untch, R. H. (n.d.). Mutation 2000: Uniting the orthogonal. Retrieved from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf Ma, Y. S., Kwon, Y. R., & Offutt, A. J. (n.d.). Mujava: An automated class mutation system. Retrieved from http://www.cs.gmu.edu/~offutt/rsrch/papers/mujava.pdf Bradbury, J. S., Cordy, J. R., & Dingel, J. (n.d.). Mutation operators for concurrent java(j2se 5.0). Retrieved from http://www.irisa.fr/manifestations/2006/Mutation2006/papers/14_Final_version.pdf Pit mutation testing. (n.d.). Retrieved from http://pitest.org/ 33