White-box Unit Test Generation with Microsoft IntelliTest

Budapest University of Technology and EconomicsDepartment of Measurement and Information Systems

Budapest University of Technology and EconomicsFault Tolerant Systems Research Group

1

White-box Unit Test Generationwith Microsoft IntelliTest

Dávid Honfi

2

Focus

module B

module Cservice

Unit testing

module A

Code-based (White-box)

White-box test designpublic int M1(int a, int b){ if(a == 0) { Console.WriteLine(ERROR_MSG); return -1; } if(b > a) return b*a+5; else return (a+b)/2;}

Let’s cover all the statements!

a==0

a!=0 && b>a a!=0 &&

b<=a a==0 a!=0 &&

b<=aa!=0 & b>a

a 0 1 2b 0 0 3

What’s missing?

Usually encoding observed outputs Where can it be employed?oBasic, common bugs (e.g., uncaught exceptions)oViolation of assertions or contractsoDeviations from existing outputs (regression)

Automation with code exploration

test case = input + expected output

Symbolic execution: the idea

Static program analysis technique from the ’70s Application for test generation

o Symbolic variables instead of normal oneso Constraints forming for each path with symb. variableso Constraint solving (e.g., SMT solver)o A solution yields an input to execute a given path

New century, new progress:o Enough computing power (e.g., for SMT solvers)o New ideas, extensions, algorithms and tools

5

Existing tools using symbolic execution

.NET: Microsoft IntelliTest (a.k.a. Pex) x86 binary: Microsoft SAGE JavaoNASA Symbolic PathFinderoCATGo…

JavaScript: Jalangi C: KLEE

IntelliTestMicrosoft

DEMO 1.

The basic use of IntelliTest

public int M1(int a, int b){ if(a == 0) { Console.WriteLine(ERROR_MSG); return -1; } if(b > a) return b*a+5; else return (a+b)/2;}

Details of IntelliTest (Dynamic SE)

Source code

Concrete input values

Symbolic variables

Constraint solver Constraints

Concrete execution

Symbolic execution

a==0

a:=0

b:=0

a,b

Transformationa!=0

a:=1

b:=0

public int M1(int a, int b){ if(a == 0) { Console.WriteLine(ERROR_MSG); return -1; } if(b > a) return b*a+5; else return (a+b)/2;}

Details of IntelliTest (Dynamic SE)

Source code

Concrete input values

Symbolic variables

Constraint solver Constraints

Concrete execution

Symbolic execution

a!=0 &&

b<=a

a!=0 && b>a

a:=1

b:=0

a:=2

b:=3

11

Parameterized Unit Testing Idea: Using test methods as specifications

o Easy to understand, easy to check, etc.o But: too specific (used for a code unit), verbose, etc.

Parameterized Unit Test (PUT)oWrapper method for method/unit under testoMain elements• Inputs of the unit• Assumptions for input space restriction• Call to the unit• Assertions for expected results

o Serves as a specification Test generators can use it

12

Example: Parameterized Unit Testing

void ReduceQuantityPUT(Product prod, int soldCount) {// AssumptionsAssume.IsTrue(prod != null && prod.Quantity > 0);Assume.IsTrue(soldCount > 0);int oldQuantity = prod.Quantity;// Calling the UUTint newQuantity =

StorageManager.ReduceQuantity(prod,soldCount);// AssertionsAssert.IsTrue(newQuantity >= 0);Assert.IsTrue(newQuantity < oldQuantity);

}

/// The method reduces the quantity of the specified/// product. The product is known to be NOT null with/// quantity larger than 0, also the sold amount is/// always more than zero. The method has effects on/// the database, and returns the new quantity of the/// product. If the quantity would be negative, the/// method reduces the quantity to zero.int ReduceQuantity(Product prod, int soldCount) { … }

13

Example: Parameterized Unit Testing

void ReduceQuantityPUT(Product prod, int soldCount) {// AssumptionsAssume.IsTrue(prod != null && prod.Quantity > 0);Assume.IsTrue(soldCount > 0);int oldQuantity = prod.Quantity;// Calling the UUTint newQuantity =

StorageManager.ReduceQuantity(prod,soldCount);// AssertionsAssert.IsTrue(newQuantity >= 0);Assert.IsTrue(newQuantity < oldQuantity);

}

/// The method reduces the quantity of the specified/// product. The product is known to be NOT null with/// quantity larger than 0, also the sold amount is/// always more than zero. The method has effects on/// the database, and returns the new quantity of the/// product. If the quantity would be negative, the/// method reduces the quantity to zero.int ReduceQuantity(Product prod, int soldCount) { … }

DEMO 2.

Parameterized Unit Testing with IntelliTest

Challenges of SE-based techniques1. Exponential growth of execution paths2. Complex arithmetic expressions3. Floating point operations4. Compound structures and objects5. Pointer operations6. Interaction with the environment7. Multithreading8. …

15

T. Chen et al. „State of the art: Dynamic symbolic execution for automated test generation”. Future Generation Computer Systems, 29(7), 2013


16


SEViz

17

Visualizing symbolic execution SEViz: Symbolic Execution VIsualiZer Goal: Help identifying test generation problems

o Long execution pathso Unsolvable constraintso…

Solution: Symbolic execution tree with metadataMapping to source code

Calls to constraint solver

1

No Yes

End of execution paths

No test Failed test Passed test

1 11 1 1 1

SHAPE BORDER COLOR

18

Visualizing symbolic execution

Additional metadatao Sequence numbero Path conditiono Incremental path condition (based on parent)o Status: can have siblings or noto Source code mapping (if available)

DEMO 3.

Visualizing test generation with SEViz


20


SEViz

Automated isolation

21

Existing solutions Stubbing and mocking (faking)

o Fixed values and checks for all DSE executionso Not suitable for test generation

Parameterized mockingo Interaction with DSE is possible• More relevant test cases• Custom behavior in mocks: e.g., state change of objects

o Introduces complexity for users of DSE• Requires large amount of time and effort• Not trivial task in case of complex structures

o Fakes cannot be generated under certain conditions

22

Approach for automated isolation Automated isolation on source code level

1. Abstract syntax tree transformations in the SUT2. Parameterized sandbox synthesization

SUT

C1

C2

Environment

i(C1,C2)

i(C2,E)

i(C2,C1)

i(C1,E)

SUT

C1’ C2’

Environment

SB i'(C2,E)

i'(C1,E)

i(C2,C1)

i(C1,C2)

Replaced external objects

Replaced external

calls

Sandbox

23

Example of AST transformationpublic class WeekendNotifier { public bool IsWeekendNear() { DateTime date = DateTime.GetNow(); date.AddDays(2); if(date.GetDay() == "Saturday") return true; return false; }}

public class WeekendNotifier { public bool IsWeekendNear() { DynamicFake date = Fake.DateTimeGetNow(); Fake.DateTimeAddDays(2,date); if(Fake.DateTimeGetDay(date) == "Saturday") return true; return false; }}

24

Example of parameterized sandboxpublic static class Fake {

public DynamicFake DateTimeGetNow() { // Return a state container object instead of the original return new DynamicFake(); }

public void DateTimeAddDays(int days, DynamicFake date) { // TODO: State change of date using DSE }

public int DateTimeGetDay(DynamicFake date) { // Obtaining return value from DSE return DSEEngine.ChooseValue<int>(); }

}

25

Summary White-box test generation: symbolic execution IntelliTest, a state-of-the-art test generator

o Dynamic Symbolic Execution (DSE)o Parameterized Unit Tests

DSE has several challenges, e.g.:o Test generation problem indentificationo Environment dependencies

My researcho SEViz: A tool for visualizing symbolic executiono Automated isolation with source code transformation

26

Resources

IntelliTesthttps://www.visualstudio.com/en-us/docs/test/developer-testing/intellitest-manual

https://www.codehunt.com/ SEViz [ICST’15 Tool Paper]

https://ftsrg.github.io/seviz Automated isolation

https://pp.bme.hu/eecs/article/view/9768

https://www.visualstudio.com/en-us/docs/test/developer-testing/intellitest-manual

https://www.visualstudio.com/en-us/docs/test/developer-testing/intellitest-manual

https://www.codehunt.com/

https://www.codehunt.com/

https://ftsrg.github.io/seviz




White-box Unit Test Generation with Microsoft IntelliTest

Software