A Methodology for Applying Concolic Testing

IT 16 071

Examensarbete 15 hpSeptember 2016

A Methodology for Applying Concolic Testing

Manuel Cherep

Institutionen för informationsteknologiDepartment of Information Technology

Teknisk- naturvetenskaplig fakultet UTH-enheten Besöksadress: Ångströmlaboratoriet Lägerhyddsvägen 1 Hus 4, Plan 0 Postadress: Box 536 751 21 Uppsala Telefon: 018 – 471 30 03 Telefax: 018 – 471 30 00 Hemsida: http://www.teknat.uu.se/student

Abstract

A Methodology for Applying Concolic Testing

Manuel Cherep

Concolic testing is a technique that combines concrete and symbolic execution inorder to generate inputs that explore different execution paths leading to bettertesting coverage. Concolic testing tools can find runtime errors fully automaticallyusing available type specifications. The type specifications in a function define the typeof each input. However, most specification languages are never expressive enough,which can lead to runtime errors caused by malformed inputs (i.e. irrelevant errors).Moreover, logic errors causing a program to operate incorrectly without crashingcannot be reported automatically. A universal methodology for any programminglanguage is proposed. Preconditions force the concolic execution to generate wellformed inputs before testing a function. On the other hand, postconditions lead to aruntime error when a program operates incorrectly, helping to find logic errors. Theresults obtained using the concolic testing tool CutEr, in the functional programminglanguage Erlang, show how a program is only tested using well formed inputs speciallygenerated to try to violate the defined postconditions.

Tryckt av: Reprocentralen ITCIT 16 071Examinator: Olle GällmoÄmnesgranskare: Justin PearsonHandledare: Konstantinos Sagonas

Resumen en espanol

Las pruebas de software, conocidas como “testing”, son el metodo predominanteen la industria para asegurar que un programa es correcto y fiable. Comoresultado de la creciente complejidad del software existen errores cada vez masdifıciles de encontrar, lo que requiere tecnicas de “testing” mas sofisticadas.

Para mejorar la cobertura y fiabilidad del software se han desarrollado her-ramientas y tecnicas de “testing” automatico. Algunas de estas tecnicas usan“testing” aleatorio donde los valores de entrada de los programas se generanaleatoriamente. El problema que tienen dichas tecnicas es que diferentes val-ores de entrada pueden estar “testeando” repetidamente un mismo compor-tamiento del codigo. Esto significa que valores de entrada aleatorios no garan-tizan la cobertura de diferentes ramas de ejecucion. Ademas, estas tecnicasrequieren que se escriban manualmente propiedades de los programas que vana ser “testeados”.

“Concolic testing” es una tecnica que combina la ejecucion concreta y simbolicade un programa. El objetivo es generar valores de entrada que ejercitan difer-entes ramas de ejecucion. Un programa se ejecuta de forma concreta y al mismotiempo se recogen restricciones simbolicas durante la ejecucion que forman uncamino de restriccion. Las restricciones en dicho camino se niegan y se resuel-ven sistematicamente usando un demostrador de teoremas, generando ası nuevosvalores de entrada que ejercitan una rama de ejecucion diferente.

“Concolic testing” ha ganado popularidad en los lenguajes de programacionimperativos como C y Java, sin embargo su uso en lenguajes de programacionfuncional es mas reciente. CutEr, una herramienta que aplica “concolic testing”en Erlang es la primera herramienta que aplica dicha tecnica en un lenguaje deprogramacion funcional.

Las herramientas de “concolic testing” son completamente automaticas ylos valores de entrada de los programas se generan utilizando informacion detipos disponible. Sin embargo, la mayorıa de los lenguajes de especificacion detipos no son lo suficientemente expresivos. Si esto sucede, una herramienta de“concolic testing” puede generar valores que no respetan completamente el tipoesperado para despues mostrarlos como valores que provocan errores en tiempode ejecucion.

En este trabajo se propone una metodologıa universal, la primera en “con-colic testing”, que garantiza el uso de valores de entrada bien formados y queayuda a encontrar errores en la logica de los programas.

2

Acknowledgements

I am extremely grateful to my supervisor Konstantinos Sagonas at UppsalaUniversity for his valuable feedback sharing his expertise.

I would also like to thank my reviewer Justin Pearson at Uppsala Universityfor his great feedback and dedication.

Last, thank you to Aggelos Giantsios at National Technical University ofAthens for always answering my questions regarding CutEr.

3

Contents

1 Introduction 51.1 Motivation and Goals . . . . . . . . . . . . . . . . . . . . . . . . 61.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Background 72.1 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Concolic Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 Erlang . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Methodology 113.1 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Preconditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.3 Postconditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

4 Concolic Testing in Erlang 214.1 CutEr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2 Limited Type Language . . . . . . . . . . . . . . . . . . . . . . . 214.3 A More Complex Example . . . . . . . . . . . . . . . . . . . . . . 23

5 Results 275.1 Average Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 275.2 Rally Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6 Discussion 326.1 CutEr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326.2 Multiple Postconditions . . . . . . . . . . . . . . . . . . . . . . . 326.3 Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

7 Conclusions and Future Work 34

8 Bibliography 35

A Source Code 37

4

Chapter 1

Introduction

Testing [1] is the predominant method in industry to ensure software correctnessand reliability. More sophisticated software techniques are necessary as a resultof the growing complexity of software that make errors difficult to find.

Software systems nowadays have thousands of lines of code with multipledifferent execution paths. Therefore, it is infeasible for a test engineer to man-ually write tests covering all the possible execution paths, which results in poorsoftware reliability.

In order to have better testing coverage and improve software reliability,automated software testing tools and techniques have been created. Some ofthese techniques use random testing where inputs are generated randomly, suchas property-based testing [2, 3, 4, 5]. The problem with such techniques is thatmany different inputs may be repeatedly testing the same behaviors, and ran-dom inputs do not guarantee covering different execution paths [6]. Moreover,it requires manually writing properties for the program to be tested.

Concolic testing is a technique that combines concrete and symbolic exe-cution of a program [7]. The goal is to generate inputs that exercise all thedifferent execution paths. The program is executed concretely and at the sametime symbolic constraints are collected during the symbolic execution generat-ing a path constraint. The path constraint is then negated and solved using aconstraint solver generating a new input that will exercise a different executionpath.

Concolic testing has gained popularity in imperative programming languagessuch as C and Java [8, 9, 10], but it has recently been applied to functionallanguages [11]. CutEr [11], a concolic testing tool for Erlang, is the first toolapplying concolic execution to a functional language.

Concolic testing tools are fully automatic, inputs are generated utilizing datatype specification available. However, most type specification languages are notexpressive enough, which can lead a concolic testing tool to generate malformedinputs (i.e. inputs that did not respect completely the expected type), laterreported as inputs leading to runtime errors. Moreover, concolic testing toolsrequire more information in order to find logic errors that do not necessarily

5

lead to runtime errors.A universal methodology is proposed which guarantees testing programs

with well formed inputs, finding logic errors in the results.

1.1 Motivation and Goals

The aim of this project is to define a methodology for applying concolic testingindependently from the programming language and tool used. The goal of thisthesis is to design a methodology which guarantees testing functions with wellformed inputs and helps to find logic errors hidden in the program. The functionunder test can only be executed with inputs that satisfy all the preconditions.Furthermore, postconditions are provided to find logic errors when one of themis violated.

There are different tools for concolic testing implemented in different pro-gramming languages with different characteristics. However, common patternscan be extrapolated in order to find a universal methodology that can be appliedto all of them.

For the sake of illustration the functional programming language Erlang andthe tool CutEr [11] are being used.

1.2 Contributions

The main contribution of this thesis is providing guidelines for testing a programusing concolic testing. The designed methodology is an important contribution,because it is the first one for applying concolic testing. Although it has beeninspired by property-based testing [2, 3, 4, 5] and CutEr [11]. This methodologysolves the two recurrent problems when using concolic testing tools. It showsreal examples where the program is only tested with inputs that satisfy all theconstraints (i.e. well formed inputs). Furthermore, it shows how to introduceassertions that can lead to finding errors in the logic of a program.

6

Chapter 2

Background

Testing coverage measures the range of different behaviors tested (i.e. the num-ber of execution paths exercised). A higher testing coverage means that a pro-gram has been more thoroughly tested.

2.1 Unit Testing

Unit testing is a method by which different individual components or unitsof a program, which are collections of functions or procedures, are tested in-dependently. A program is divided into different units that are then testedindependently.

In this method it is necessary to specify input values for every unit that isgoing to be tested. The specification of such inputs can be done manually butthis does not ensure that all possible execution paths would be exercised duringthe testing. It is usually an arduous work and the testing coverage is low.

Automatically generating values for the inputs would reduce the effort ofwriting these values manually and it might increase the testing coverage.

2.2 Concolic Testing

Concolic testing [7] is a technique that combines concrete and symbolic executionof a program to generate inputs that exercise all the different execution paths.The goal is to achieve high testing coverage. The combination of concrete andsymbolic execution running simultaneously is called concolic execution.

In concolic execution, the concrete execution is the normal execution of theprogram. Symbolic execution [12, 13, 14] collects symbolic constraints overthe symbolic inputs of the program at each conditional branch. The symbolicexecution has to be done without altering the concrete execution of the programwhich is accomplished by adding code that collects the constraints, also knownas instrumentation. The program is executed with a symbolic value (i.e. avariable has a symbol associated with it instead of a value) for each variable

7

that depends on inputs to the program. During the execution the same path isfollowed until there is a conditional expression where the execution follows oneof the possible branches based on variables that have symbolic values. At thisconditional expression, with the given symbolic values it is possible to createa symbolic constraint that describes the possible input values that lead theexecution of the program to follow one branch or another (i.e. determines anexecution path). At the end of the concolic execution, the conjunction of all thesymbolic constraints collected at each branch point is called path constraint. Apath constraint describes the input values that lead the concrete execution tofollow an execution path.

In a concolic testing tool, concrete random input values are generated toexecute the program that is going to be tested. During this first concrete exe-cution the symbolic constraints are collected creating a path constraint. Eachconstraint in the path is then negated and solved systematically using a con-straint solver, if it is feasible, generating new test inputs that will exercise anunexplored execution path in the next iteration. This process is repeated untilall feasible execution paths in the program have been exercised. In order tounderstand this better a concolic execution will be explained using the exampleshown in Figure 2.1.

-spec foo(number(), number()) -> ok.

foo(X, Y) ->

case bar(X) =:= Y of

true ->

erlang:error("RUNTIME ERROR");

false ->

ok

end.

-spec bar(number()) -> number().

bar(Z) ->

Z + 1.

Figure 2.1: A running example

Figure 2.2 shows the control-flow graph of the function foo in the previousexample. The pink node corresponds to the entry point of the function; theblue node represents the condition expression; the green node represents a resultpoint (i.e. the end of an error-free path); the red node corresponds to an errorin the execution.

8

foo/2(X, Y)

X+1 == Y

FAIL ok

falsetrue

Figure 2.2: Control flow graph for the function foo in the example

An example with inputs and its corresponding path constraints for a concolicexecution of the function foo in Figure 2.1 is shown in Table 2.1. It is assumedthat the first input values are {X �→ 2, Y �→ 7}. The constraints collectedduring the concrete execution of the function with the given inputs result in thepath constraint < X0 + 1 �= Y0 >, where X0 and Y0 are the symbolic values ofX and Y respectively. This concrete execution does not fail. The last constraintin the path constraint is negated (i.e. the only constraint in this example) andthe resulting path constraint is < X0 + 1 = Y0 >. This path constraint issolved generating values for the next concrete execution. Assuming the valuesgenerated are {X �→ 1, Y �→ 2}, the concrete execution reveals the error in thecode. If the path constraint had more than one constraint then it would benecessary to iterate again, negating the next constraint and solving it.

Input Path Constraint{X �→ 2, Y �→ 7} < X0 + 1 �= Y0 >{X �→ 1, Y �→ 2} < X0 + 1 = Y0 >

Table 2.1: Inputs and path constraints of a concolic execution of function foo

2.3 Erlang

Erlang [15] is a concurrent functional programming language designed for pro-gramming fault-tolerant distributed systems.

Erlang is dynamically typed which means that the type checking is doneat runtime. Type checking is the verification of the constraints of types in aprogram. Figure 2.3 shows an example of a function that fails because types donot match. This function is not rejected by the compiler but fails at runtime.

9

foo() ->

%% Bar is a string

Bar = "Hello World!",

%% This will fail because Bar is not a integer

integer_to_list(Bar).

Figure 2.3: A function containing a runtime error

Erlang achieves fault-tolerance having a process supervising the behavior ofanother process. If the supervised process fails, the supervising process must beable to detect it and take over to handle the error. While other languages tryto make error-free programs, Erlang assumes that errors will happen.

The recommended way of programming in Erlang is non-defensive program-ming. A function should crash if there is an unexpected behavior instead ofhandling every possible scenario. This programming style leads to more cleanand compact code. Figure 2.4 shows examples of defensive and non-defensiveprogramming, where the specification of the function represents the expectedtype.

-spec non_defensive_add1(number()) ->

number().

non_defensive_add1(Value) ->

Value+1.

-spec defensive_add1(any()) ->

number() | value_is_not_number.

defensive_add1(Value) when is_number(Value) ->

Value+1;

defensive_add1(Value) ->

value_is_not_number.

Figure 2.4: Non-defensive and Defensive functions

The Erlang philosophy is “let it crash”. Failing processes should crash im-mediately and another supervising process will detect the crash and correct theerror.

10

Chapter 3

Methodology

There are many different programming paradigms with different characteris-tics. However, it is possible to extrapolate common problems mentioned in thefollowing chapter to define a methodology that can be applied to all of them.

3.1 Problem

Concolic testing is a powerful testing technique to find runtime errors. Thereare errors in a program, also known as logic errors, that cause the program tooperate incorrectly but do not lead the program to crash. Since these kindsof errors do not lead the program to crash during runtime, a concolic testingtool will not find them automatically unless more information is provided. Onthe other hand, some functions require the inputs to always be well formed orotherwise these functions would crash. It is more convenient a concolic testingtool that reports runtime errors caused only by well formed inputs.

Figure 3.1 shows a function that aims to calculate the average of two integernumbers. The two inputs are given as strings, but it is implicitly assumed thatthese strings can be converted to integers and the function will crash if thiscondition is not satisfied. Moreover, the function contains a logic error dueto operator precedence. This example is a simplified version of real exampleswhere strings are used to represent inputs that are telephone numbers, e-mailaddresses, etc.

11

-spec average(string(), string()) -> number().

average(A, B) ->

A_Int = list_to_integer(A),

B_Int = list_to_integer(B),

%% (A_Int+B_Int)/2 is the correct code

A_Int+B_Int/2.

Figure 3.1: Average function containing a logic error

The concolic execution of the function in Figure 3.1 is going to generatevalues for the two inputs and the only constraint is that both values have to bestring. Assuming the values generated are A = “ ” and B = “ ” (i.e. the emptystring) the execution of the function would lead to a runtime error. However,this cannot be considered a wrong implementation nor a false positive. It is nota false positive because it is true that the function would crash with the giveninput. However, this scenario should not occur since it violates our implicitcondition that both inputs must be convertible to integer. This situation is notinteresting and it is better if it is not reported.

On the other hand, assuming both inputs are well formed (i.e. strings con-vertible to integer), the concolic execution would not lead to a runtime errorbecause even though the function operates incorrectly, it does not crash.

3.2 Preconditions

The preconditions can solve the problem of testing a function with inputs thatare not well formed. A precondition is a condition that must be fulfilled be-fore continuing the execution of the original function under testing. If all thenecessary preconditions are satisfied that means that the inputs are well formed.

One way of satisfying preconditions is to force inputs to satisfy them (e.g.if a list should have an special last element, attach it before continuing withthe test). Figure 3.2 shows another way a precondition can be satisfied in orderto continue executing the original function under testing, which in this case iscalled testme.

12

test(Input) ->

case precondition(Input) of

true ->

%% Input is well formed

%% Call the function to be tested

testme(Input);

false ->

%% Skip input

ok

end.

Figure 3.2: Test using a precondition

It is important to create tests that do not modify the original function.Therefore, the code shown in Figure 3.2 is a new function created specially forconcolic testing of this program unit. This will be explained in more detail inSection 3.4.

Figure 3.3 shows the control-flow graph of the function in Figure 3.2. Ayellow node represents variable bindings and calls to other functions.

test/1(Input)

precondition/1(Input)

testme/1(Input) ok

falsetrue

Figure 3.3: Control flow graph for the function test with a precondition

This way the precondition ensures that testme will only be called with wellformed inputs, assuming that the precondition is implemented correctly. In casethe input is not well formed the precondition will not be satisfied, returningfalse. The concrete execution will not lead to a runtime error, it would followthe execution path of the branch false, which returns normally. This pathconstraint is negated and solved with a constraint solver, generating inputs thatwould lead the concrete execution to follow the other branch true. Takingadvantage of the constraint solver, well formed inputs are generated to test thefunction testme.

13

-spec test_average(string(), string()) -> ok.

test_average(A, B) ->

case precondition(A, B) of

true ->

average(A, B),

ok;

false ->

%% Skip input

ok

end.

-spec precondition(string(), string()) -> boolean().

precondition(A, B) ->

is_list_integer(A) andalso is_list_integer(B).

-spec is_list_integer(string()) -> boolean().

is_list_integer([H]) when H >= $0, H =< $9 ->

true;

is_list_integer([H|T]) when H >= $0, H =< $9 ->

is_list_integer(T);

is_list_integer(_) ->

false.

Figure 3.4: Testing function average using a precondition

Figure 3.4 shows a function test average created to test the function inFigure 3.1. This test uses a precondition that returns true if both inputs canbe converted from string to integer, meaning that the inputs are well formed;otherwise it returns false. Figure 3.5 shows the control-flow graph of thefunction test average.

test average/2(A, B)

precondition/2(A, B)

average/2(A, B) ok

falsetrue

Figure 3.5: Control flow graph for the function test average with a precondition

14

3.3 Postconditions

The postconditions can contribute to finding logic errors in a program. A post-condition is a condition that must always be true after the execution of theoriginal function under test. The idea of a postcondition is similar to a prop-erty in property-based testing [2, 3, 4, 5].

Figure 3.6 shows how a postcondition must be satisfied after executing theoriginal function under testing, which in this case is called testme.

test(Input) ->

Result = testme(Input),

case postcondition(Input, Result) of

false ->

%% Runtime error

erlang:error("Postcondition failed");

true ->

ok

end.

Figure 3.6: Test using a postcondition

The code shown in Figure 3.6 is a new function created specially for thetest. This will be explained in more detail in Section 3.4. Figure 3.7 shows thecontrol-flow graph of the function in Figure 3.6.

test/1(Input)

Result = testme/1(Input)

postcondition/2(Input, Result)

ok FAIL

falsetrue

Figure 3.7: Control flow graph for the function test with a postcondition

15

The function testme is called and the result saved, collecting the necessaryconstraints. The concrete execution will lead to a runtime error if the post-condition is not satisfied or as a consequence of the execution of the functiontestme. It is assumed that the postcondition is implemented correctly and it issatisfied during the first concrete execution. The path constraint collected willbe negated and solved to generate an input that can violate the postcondition,which is only possible if there is a logical error in the function testme. Oth-erwise, the constraint solver will not be able to generate inputs to violate thepostcondition.



Result = average(A, B),

case postcondition(A, B, Result) of

false ->

%% Runtime error


true ->

ok

end.

-spec postcondition(string(), string(), number()) -> boolean().

postcondition(A, B, Result) ->

%% Assuming the input is well formed at this point



max(A_Int, B_Int) >= Result.

Figure 3.8: Testing function average using a precondition

Figure 3.8 shows a function test average created to test the function inFigure 3.1. The postcondition of the function average is that the result hasto be less or equal than the maximum of both inputs. Figure 3.9 shows thecontrol-flow graph of the function test average.

16


Result = average/2(A, B)

postcondition/3(A, B, Result)

ok FAIL

falsetrue

Figure 3.9: Control flow graph for the function test average with a postcondition

3.4 Unit Testing

The methodology presented consisting of preconditions and postconditions ismodular (i.e. preconditions and postconditions are independent). Sometimesthe preconditions are not necessary and can be omitted, for instance when afunction is programmed defensively and it should also be tested with inputsthat are not well formed. The postconditions can also be omitted if necessary.

Being able to test a function without modifying it is very important. There-fore each test is a new function wrapping up the function that is going to betested. The test function should have the same arity as the function to be tested,both having the same input specification. The preconditions and postconditionsare included in the test function.

Figure 3.10 shows a test applying the methodology with both preconditionsand postconditions testing the function testme. Figure 3.11 shows the corre-sponding control-flow graph.

17

test(Input) ->

case precondition(Input) of

true ->

Result = testme(Input),

case postcondition(Input, Result) of

false ->

%% Runtime error


true ->

ok

end;

false ->

%% Skip input

ok

end.

Figure 3.10: Test using the methodology

test/1(Input)

precondition/1(Input)

okResult = testme/1(Input)

postcondition/2(Input, Result)

ok FAIL

falsetrue

falsetrue

Figure 3.11: Control flow graph for the function test with the methodology

18

-module(foo).

-spec average(string(), string()) -> number().

average(A, B) ->



A_Int+B_Int/2.



case precondition(A, B) of

true ->

Result = average(A, B),

case postcondition(A, B, Result) of

false ->

%% Runtime error


true ->

ok

end;

false ->

%% Skip input

ok

end.

-spec precondition(string(), string()) -> boolean().

precondition(A, B) ->

is_list_integer(A) andalso is_list_integer(B).

-spec is_list_integer(string()) -> boolean().

is_list_integer([H]) when H >= $0, H =< $9 ->

true;

is_list_integer([H|T]) when H >= $0, H =< $9 ->

is_list_integer(T);

is_list_integer(_) ->

false.

-spec postcondition(string(), string(), number()) -> boolean().

postcondition(A, B, Result) ->

%% The input is well formed at this point



max(A_Int, B_Int) >= Result.

Figure 3.12: Testing function average using the methodology

19


precondition/2(A, B)

okResult = average/2(A, B)

postcondition/3(A, B, Result)

ok FAIL

falsetrue

falsetrue

Figure 3.13: Control flow graph for the function test average with the method-ology

Figure 3.12 shows the functions average and test average which have beenused in previous sections. On this occasion the example is presented completelyfollowing the methodology. Figure 3.13 shows the corresponding control-flowgraph.

The function test average is going to be executed in Section 5.1 afterpresenting a concolic testing tool in the same chapter.

It is also important to emphasize that the same logic can be applied usingmore than one precondition and postcondition, it has been kept simple for thesake of illustration. However, in more complex functions it could be necessaryto use many preconditions and postconditions. The structure would be exactlythe same. All the preconditions have to be satisfied in order to execute thefunction under test. On the other hand, in case any of the postconditions isviolated, a runtime error must occur.

20

Chapter 4

Concolic Testing in Erlang

Erlang is a dynamically typed programming language, type checking is doneat run-time instead of compile-time. Erlang comes with a type specificationlanguage to define types that are later used for documentation or by testingtools.

4.1 CutEr

CutEr [11] is a Concolic Unit Testing tool for Erlang, implemented mostly inErlang with a small part in Python.

The heuristic used by CutEr in the concolic execution to explore differentexecution paths is based on path coverage. All execution paths form a searchtree with a certain depth, which can be set as a boundary to stop the concolicexecution. For each execution path the first decision node whose reversed labelhas not been visited yet, is explored. The exploration continues until all possibleexecution paths have been explored, or when a certain depth in the search treehas been reached. The constraint solver currently used by CutEr is Z3 [16].

Erlang type specification language is supported by CutEr. CutEr considerstype specifications as preconditions of program inputs. These are considered asadditional constraints that are never negated to avoid breaking a precondition.

4.2 Limited Type Language

A subset of the type specification language in Erlang is shown in Figure 4.1.Erlang specifications are never checked during compilation. However, type spec-ifications can be used to document function interfaces or provide informationfor testing tools [17].

21

Type :: any()

| none()

| pid()

| port()

| reference()

| []

| atom()

| Bitstring

| float()

| Fun

| Integer

| List

| Tuple

| Union

| UserDefined

Bitstring :: <<>>

| <<_:M>>

| <<_:_*N>>

| <<_:M, _:_*N>>

Fun :: fun()

| fun((...) -> Type)

| fun(() -> Type)

| fun((TList) -> Type)

Integer :: integer()

| Erlang_Integer

| Erlang_Integer..Erlang_Integer

List :: list(Type)

| maybe_improper_list(Type1, Type2)

| nonempty_improper_list(Type1, Type2)

| nonempty_list(Type)

Tuple :: tuple()

| {}

| {TList}

TList :: Type

| Type, TList

Union :: Type1 | Type2

Figure 4.1: Erlang Type Language

22

The language, as any other type language, is limited and it cannot representall possible constraints for all types. An illustrative example [11] is the followingtype declared in the calendar module of the standard library:

-type date() :: {Year::non_neg_integer(), Month::1..12, Day::1..31}.

which is used in some functions of this module. CutEr reports that these func-tions will fail with the input 42,4,31, which according to the specification iscorrect. It is not a relevant error since it is not a real date, but CutEr cannotknow that.

This issue was already faced in the example shown in Figure 3.12. Thefunction average should take strings that can be convertible to integer, but thespecification language is not expressive enough. Therefore, it is only specifiedthat the inputs have to be strings.

CutEr generates inputs that satisfy the specifications, but if they are not ex-pressive enough then some of the inputs could be not well formed. This problemneeds to be addressed for the automatic use of the tool and the preconditionsexplained in Section 3.2 can help CutEr to always generate well formed inputs.

4.3 A More Complex Example

The next problem that is going to be tested is known as the rally problem [18].The goal is to calculate the minimum number of moves necessary for a car tofinish the track without exceeding the speed limit. A track is divided in sectionunits, each one of them with a specific speed limit. After a move the speed canbe increased or decreased. The speed determines the number of moves, for each10 km/h the car moves 1 unit. Speed can be increased or decreased by multiplesof 10, and there is a maximum acceleration and maximum brake speed. Thismeans that a car cannot accelerate (brake) at once more than the maximumacceleration (braking) speed.

The preconditions required for this problem are: the end of the track isindicated with the element {0,0}, the maximum number of units in a track is10.000, speed is always a positive multiple of 10 smaller than 250.

Appendix A shows a correct implementation to solve the rally problem.Figure 4.2 shows part of the code containing an error, because the speed chosenis always the limit of the current part of the track. This is wrong, because thereis a maximum acceleration that cannot be violated.

23

-spec rally(speed(), speed(), track()) -> moves().

rally(MaxA, MaxB, Track) ->

{Moves, _AccSpeed} = rally(MaxA, MaxB, Track, 0, 0, []),

Moves.

-spec rally(speed(), speed(), track(),

speed(), moves(), [speed()]) -> {moves(), [speed()]}.

rally(_MaxA, _MaxB, [], _CurrentSpeed, Moves, Acc) ->

{Moves, Acc};

rally(_MaxA, _MaxB, [{0,0}], _CurrentSpeed, Moves, Acc) ->

{Moves+1, Acc};

rally(MaxA, MaxB, Track = [{_N,SpeedLimit} | _SubTrack],

CurrentSpeed, Moves, Acc) ->

%% Correct code:

%% HighestSpeed = CurrentSpeed + MaxA,

%% SpeedCandidate =

%% case HighestSpeed >= SpeedLimit of

%% true ->

%% SpeedLimit;

%% false ->

%% HighestSpeed

%% end,

SpeedCandidate = SpeedLimit, %% Wrong code

{NewTrack, NewSpeed} =

optimal_speed(Track, MaxB, SpeedCandidate),

NewAcc = Acc ++ [NewSpeed],

rally(MaxA, MaxB, NewTrack, NewSpeed, Moves+1, NewAcc).

Figure 4.2: Function rally with a bug

24

Figure 4.3 shows functions test rally precondition and test rally cre-ated to test the rally implementation. Preconditions related to the speed andthe maximum number of units in a track are going to be satisfied automaticallywith the types specified (i.e speed, units, track and moves). CutEr understandsthis language and it is going to generate inputs that satisfy this preconditions.On the other hand, the precondition that the last element of a track must be{0,0} is not cover with the specified types. One way of satisfying a precondition,as mentioned in Section 3.2, is to force inputs to satisfy it. In this scenario, alltracks have the precondition that their last element has to be {0,0}, which it isimpossible to specify using the type language in Erlang. Therefore, each trackis forced to satisfy that precondition appending the required last element beforecalling the original function under test.

The function test rally precondition in Figure 4.3 guarantees that thefunction rally is going to be called with well formed inputs. On the other hand,the function test rally is also setting a postcondition. The speed differencebetween the previous speed and the next one can be at most the maximumacceleration.

25

-spec test_rally_precondition(speed(), speed(), track()) -> ok.

test_rally_precondition(MaxA, MaxB, Track) ->

%% Precondition: The last element of the track must be {0,0}

WFTrack = Track ++ [{0,0}],

rally(MaxA, MaxB, WFTrack),

ok.

-spec test_rally(speed(), speed(), track()) -> ok.

test_rally(MaxA, MaxB, Track) ->

%% Precondition: The last element of the track must be {0,0}

WFTrack = Track ++ [{0,0}],

{_Moves, Acc} = rally(MaxA, MaxB, WFTrack, 0, 0, []),

%% Postcondition: Cannot accelerate more than MaxA

InitSpeed = 0,

case postcondition(MaxA, InitSpeed, Acc) of

false ->


true ->

ok

end.

-spec postcondition(speed(), speed(), [speed()]) -> boolean().

postcondition(_MaxA, _PreviousSpeed, []) ->

true;

postcondition(MaxA, PreviousSpeed, [Speed | Rest]) ->

case PreviousSpeed + MaxA >= Speed of

true ->

postcondition(MaxA, Speed, Rest);

false ->

false

end.

Figure 4.3: Code to test function rally

26

Chapter 5

Results

The results in this chapter were obtained using the following system require-ments:

• CutEr version 0.1: Concolic testing tool [11].

• Z3 version 4.4.2: Constraint solver [16].

• Erlang/OTP version 18.7.2: Programming language [15].

• Python version 2.7.10: Programming language [19].

5.1 Average Function

At this point, CutEr can be used to test the function in the example shownin Figure 3.1. First the original function is tested automatically and then it istested implementing the proposed methodology. In order to define a differentdepth, explain in the search tree than the default one the option -d is used.

Firstly, running CutEr on the original function average:

$ cuter foo average ’[\"0\", \"0\"]’

where cuter is the name of the script provided with the tool and its argumentsare the name of the module, the function to test and an input for the givenfunction. In case the input of the function is not well formed, it will be reportedby CutEr. The output as a result of executing this command is:

Testing foo:average/2 ...

<more dots and x’s here deleted>

=== Inputs That Lead to Runtime Errors ===

#1 foo:average("", "")

#2 foo:average("-", "")

#3 foo:average("+", "")

#4 foo:average([45,0], "")

27

#5 foo:average([43,0], "")

#6 foo:average("9", "")

#7 foo:average("-0", "")

#8 foo:average([0], "")

#9 foo:average("+0", "")

#10 foo:average("-9", "")

#11 foo:average("+9", "")

As expected, CutEr reported errors that are strings not convertible to inte-ger. The inputs lead to runtime errors, but they are irrelevant to us becausesuch strings should never be inputs. On the other hand, other more interest-ing errors (e.g. foo:average("2", "1")) are not found because there is nopostcondition to help CutEr.

Running CutEr on the function test average in Figure 3.4 (i.e. only usinga precondition) generates the following output:

$ cuter foo test_average ’[\"0\", \"0\"]’ -d 100

Testing foo:test_average/2 ...

..x..x.xxxxxxxxx.x..xxx.xx.xxx..

xxxxxxxxxxxxxx..xxxx..xxxxxxxxxxxx

..xxx.xxxxx.xxxxxxxx.xxxxxxxxxxxxxxxxx

..xxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxx

..xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxx.xxx.xxxxx...

No Runtime Errors Occured

No runtime errors occurred because the function under test is only executedwith well formed inputs (i.e. strings convertible to integer). However, there isa logic error in the average function that has not been reported yet.

Finally, running CutEr on the function test average with the whole imple-mented methodology seen in Figure 3.12.

$ cuter foo test_average ’[\"0\", \"0\"]’ -d 50

Testing foo:test_average/2 ...

....xxxxxxxxxxxxxxxxxxxxxxxxxxx.

xxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxx..xxxxxx

xxxxxxxxxxxxxxx..xxxxxxxxxxxxxxx

xxxxxxxxxxxx..xxxxxxxxxxxxx.x.xx

xxxxxxxxx..xxxxxxxxxxxxxxxxxxxx.

.xxxxxxxxxxxxxxxxxxx.xxxxxxxxxxx

xxxxxxxxxxxxxxxxxx..xxxxxxxxxxxx

xxxxxxxx..xxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxx..

28

foo:test_average("9", "9")

.xxxxxx.xxxxx...xxxxxxxxxxxxxxxx


#1 foo:test_average("9", "9")

CutEr reports a runtime error that is particularly interesting because it isviolating the postcondition. Executing the input reported in an Erlang shell:

Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4]

[async-threads:10] [hipe] [kernel-poll:false]

Eshell V7.3 (abort with ^G)

1> foo:average("9","9").

13.5

2> foo:test_average("9","9").

** exception error: "Postcondition failed"

in function foo:test_average/2 (foo.erl, line 17)

the function average returns 13.5 which is wrong because the correct result is9. Executing the test with the given input generates an exception because thepostcondition failed.

5.2 Rally Program

Firstly, running CutEr on the original function rally generates the followingoutput:

$ cuter rally rally ’[20, 10, [{5, 20},{0,0}]]’ -d 80

Testing rally:rally/3 ...

.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xx.xxxxxx.xxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxx.x.xxx

rally:rally(240, 10, [{1,240}])

xxxxxxx.xxxxxxx.

rally:rally(40, 50, [{1,60}])

xxxxx


#1 rally:rally(240, 10, [{1,240}])

#2 rally:rally(40, 50, [{1,60}])

CutEr reports two inputs that lead to runtime errors. However, a closerlook reveals that one of the preconditions is not met: the last element of a trackmust be {0,0}. The reported inputs are irrelevant because the inputs are notwell formed.

29

Running CutEr on the function test rally precondition in Figure 4.3generates the following output:

$ cuter rally test_rally_precondition ’[20, 10, [{5, 20}]]’ -d 80

Testing rally:test_rally_precondition/3 ...

.xx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

xxxxxxxxx..x..xxxxxxxxxxxxxxxxxxxxxxx


No Runtime Errors Occured

No runtime errors occurred now that the function is only executed with wellformed inputs. However, there is a logic error in the rally function that has notbeen reported yet.

Finally, running CutEr on the function test rally with the whole imple-mented methodology.

$ cuter rally test_rally ’[20, 10, [{5, 20}]]’ -d 80 -s 4

Testing rally:test_rally/3 ...

.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

x.xxxxxxxxxxxxxxxxx

rally:test_rally(10, 100, [{18,180}])

xxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxx

xxxxxxxxxxxxxxxxxxx



#1 rally:test_rally(10, 100, [{18,180}])

<more inputs here deleted>

CutEr reports a runtime error that is violating the postcondition. In order toexecute the rally function with the reported input it is necessary to append thelast element {0,0}. CutEr does not report the input with {0,0} as last element,because the precondition was met explicitly in the test. Executing the inputreported in an Erlang shell:

30

Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:4:4]

[async-threads:10] [hipe] [kernel-poll:false]

Eshell V7.3 (abort with ^G)

1> rally:rally(10, 100, [{18,180}, {0,0}]).

2

2> rally:test_rally(10, 100, [{18,180}]).

** exception error: "Postcondition failed"

in function rally:test_rally/3 (rally.erl, line 112)

Executing the test with the given input generates an exception because thepostcondition failed.

31

Chapter 6

Discussion

The discussions in this chapter focus on the current limitations of CutEr, mul-tiple postconditions and integration in existing projects.

6.1 CutEr

CutEr version 0.1 is still under heavy development. A concolic testing tool suchas CutEr needs to execute symbolically as much code as possible in order tohave more constraints to generate inputs that exercise different execution paths.At this moment, some built-in functions (BIFs) in Erlang are not supportedsymbolically. The examples shown in this report were prepared to only usefunctions that CutEr can execute symbolically in order to get the most out ofit. This limitation made impossible to test more complex programs becausesome functions were not supported symbolically. As a result of this thesis somebugs in CutEr were reported.

CutEr is a powerful testing tool that can find corner cases that would bevery difficult to find using other testing techniques. However, the scope of thisthesis is to show a methodology to provide the tool with more information tofind logic errors avoiding not well formed inputs. A complete demonstration ofthe power of CutEr cannot be covered in this project.

6.2 Multiple Postconditions

The examples shown in this thesis contain only one postcondition. However,one may need many postconditions to fully test an unit. Each postconditionadds more complexity to the search tree of constraints and slows the execu-tion. In such a case it is more efficient to create different tests testing differentpostconditions.

32

6.3 Integration

One of the great advantages of using concolic testing tools such as CutEr is thatthey work out of the box. On the other hand, other testing tools (e.g. property-based testing tools) are not fully automatic requiring more testing expertise.The methodology proposed in this thesis only requires knowledge of the pro-gramming language used. Preconditions and postconditions are written usingstandard functions without having to create special generators or properties asin the case of property-based testing tools.

Furthermore, unit tests can become concolic tests easily. It is only necessaryto modify an existing unit test to take arguments that are going to be generatedby the concolic testing tool. Thus, a unit test becomes a concolic test with ahigher testing coverage.

6.4 Related Work

The problem of testing functions with well formed inputs was already addressed[11] in a similar way with a previous validation of the inputs. To the best of myknowledge there is no other previous work defining a methodology in concolictesting. However, property-based testing techniques [2, 3, 4, 5] follow a simi-lar structure. In property-based testing inputs are randomly generated with amanually written generator. On the other hand, in the proposed methodologypreconditions only validate whether an input is well formed or not. However,properties in property-based testing are similar to postconditions in the method-ology here proposed.

33

Chapter 7

Conclusions and FutureWork

Concolic testing is a powerful technique that combines concrete and symbolicexecution to generate inputs that exercise different execution paths increasingtesting coverage.

Concolic testing tools are automatic but may report irrelevant errors if theinputs used to test are not well formed. Moreover, logic errors may not bereported without providing more information.

In this thesis a methodology is proposed where preconditions always guar-antee that the inputs used to test a function are well formed. The functionunder testing is only executed with inputs that satisfy all the preconditions.Furthermore, logic errors can be found setting postconditions that produce aruntime error in case the result of the function under testing does not satisfyany postcondition.

The results obtained with the concolic testing tool CutEr show that irrel-evant errors (i.e. inputs not satisfying all the preconditions) are not reported.Moreover, CutEr is able to find logic errors reporting inputs that violate a de-fined postcondition.

Once CutEr supports more built-in functions (BIFs) it would be necessaryto apply the described methodology in a bigger code base. It would be partic-ularly interesting to study the performance of the proposed design in a biggercode base, since preconditions and postconditions make the search tree growincreasing its complexity.

Another challenge would be to obtain results applying this methodology indifferent programming languages. Each programming language is different andthe methodology may not cover all the necessities. Moreover, there are specificfeatures in every language where a more specific design could lead to findingmore errors in a code base.

34

Chapter 8

Bibliography

[1] B. Beizer, Software testing techniques. Dreamtech Press, 2003.

[2] K. Claessen and J. Hughes, “Quickcheck: a lightweight tool for randomtesting of haskell programs,” ACM SIGPLAN notices, vol. 46, no. 4, pp.53–64, 2011.

[3] C. Runciman, M. Naylor, and F. Lindblad, “Smallcheck and lazy small-check: automatic exhaustive testing for small values,” in ACM SIGPLANnotices, vol. 44, no. 2. ACM, 2008, pp. 37–48.

[4] “Erlang QuickCheck.” [Online]. Available: http://www.quviq.com/products/erlang-quickcheck/

[5] M. Papadakis and K. Sagonas, “A proper integration of types and func-tion specifications with property-based testing,” in Proceedings of the 10thACM SIGPLAN workshop on Erlang. ACM, 2011, pp. 39–50.

[6] A. J. Offutt and J. H. Hayes, “A semantic model of program faults,” inACM SIGSOFT Software Engineering Notes, vol. 21, no. 3. ACM, 1996,pp. 195–200.

[7] K. Sen, “Concolic testing,” in Proceedings of the twenty-secondIEEE/ACM international conference on Automated software engineering.ACM, 2007, pp. 571–572.

[8] K. Sen, D. Marinov, and G. Agha, “Cute: a concolic unit testing engine forc,” in ACM SIGSOFT Software Engineering Notes, vol. 30, no. 5. ACM,2005, pp. 263–272.

[9] K. Sen and G. Agha, “Cute and jcute: Concolic unit testing and explicitpath model-checking tools,” in CAV, T. Ball and R. B. Jones, Eds., 2006,pp. 419–423.

35

[10] P. Godefroid, N. Klarlund, and K. Sen, “Dart: directed automated randomtesting,” in ACM SIGPLAN Notices, vol. 40, no. 6. ACM, 2005, pp. 213–223.

[11] A. Giantsios, N. Papaspyrou, and K. Sagonas, “Concolic testing for func-tional languages,” in Proceedings of the 17th International Symposium onPrinciples and Practice of Declarative Programming. ACM, 2015, pp.137–148.

[12] T. Ball, “Abstraction-guided test generation: A case study,” MicrosoftResearch, Tech. Rep. MSR-TR-2003-86, 2003.

[13] D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majum-dar, “Generating tests from counterexamples,” in Proceedings of the 26thInternational Conference on Software Engineering. IEEE Computer Soci-ety, 2004, pp. 326–335.

[14] C. Csallner and Y. Smaragdakis, “Check’n’crash: combining static check-ing and testing,” in Proceedings of the 27th international conference onSoftware engineering. ACM, 2005, pp. 422–431.

[15] J. Armstrong, “Erlang,” Communications of the ACM, vol. 53, no. 9, pp.68–75, 2010.

[16] “Z3 SMTv2 Guide.” [Online]. Available: http://rise4fun.com/z3/tutorial/guide

[17] T. Lindahl and K. Sagonas, “Practical type inference based on success typ-ings,” in Proceedings of the 8th ACM SIGPLAN international conferenceon Principles and practice of declarative programming. ACM, 2006, pp.167–178.

[18] “Car Rallying.” [Online]. Available: http://uva.onlinejudge.org/index.php?option=onlinejudge&page=show problem&problem=900

[19] G. Rossum, “Python tutorial,” CWI (Centre for Mathematics andComputer Science), 1995.

36

Appendix A

Source Code

I. rally.erl

-module(rally).

-compile(export_all).

%% Brief explanation of the algorithm:

%%

%% 1. Choose the highest speed possible.

%%

%% 2. Moves all the units at that speed. Check if it is breaking

%% any limit. If a limit is broken then step 4.

%%

%% 3. Start braking as much as possible to see if it is possible to

%% continue without breaking limits (i.e. If there is enough

%% "time" to decrease the speed). If a limit is broken then

%% step 4.

%%

%% 4. Try next candidate speed (i.e. current speed minus 10).

%% Go back to step 2.

%%

%% 5. Once the speed chosen does not break any limit, the car

%% moves. Go back to step 1 until the track is compelted.

-type speed() :: 10 | 20 | 30 | 40 |

50 | 60 | 70 | 80 |

90 | 100 | 110 | 120 |

130 | 140 | 150 | 160 |

170 | 180 | 190 | 200 |

210 | 220 | 230 | 240.

37

-type units() :: 1..10000.

-type track() :: [{units(), speed()},...].

-type moves() :: non_neg_integer().

-spec rally(speed(), speed(), track()) -> moves().

rally(MaxA, MaxB, Track) ->

{Moves, _AccSpeed} = rally(MaxA, MaxB, Track, 0, 0, []),

Moves.

-spec rally(speed(), speed(), track(),

speed(), moves(), [speed()]) -> {moves(), [speed()]}.

rally(_MaxA, _MaxB, [], _CurrentSpeed, Moves, Acc) ->

{Moves, Acc};

rally(_MaxA, _MaxB, [{0,0}], _CurrentSpeed, Moves, Acc) ->

{Moves+1, Acc};

rally(MaxA, MaxB, Track = [{_N,SpeedLimit} | _SubTrack],

CurrentSpeed, Moves, Acc) ->

HighestSpeed = CurrentSpeed + MaxA,

SpeedCandidate =

case HighestSpeed >= SpeedLimit of

true ->

SpeedLimit;

false ->

HighestSpeed

end,

{NewTrack, NewSpeed} =

optimal_speed(Track, MaxB, SpeedCandidate),

NewAcc = Acc ++ [NewSpeed],

rally(MaxA, MaxB, NewTrack, NewSpeed, Moves+1, NewAcc).

-spec optimal_speed(track(), speed(), speed()) ->

{track(), speed()}.

optimal_speed(Track, MaxB, Speed) ->

NumOfMoves = Speed div 10,

case move(Track, NumOfMoves, Speed) of

overlimit ->

optimal_speed(Track, MaxB, Speed-10);

SubTrack ->

case braking(SubTrack, MaxB, Speed) of

true ->

{SubTrack, Speed};

false ->

optimal_speed(Track, MaxB, Speed-10)

end

end.

38

-spec braking(track(), speed(), speed()) -> boolean().

braking([],_MaxB, _Speed) ->

true;

braking(Track, MaxB, Speed) ->

SpeedBraking = Speed - MaxB,

case SpeedBraking =< 10 of

true ->

true;

false ->

NumOfMovesBraking = SpeedBraking div 10,

case move(Track, NumOfMovesBraking, SpeedBraking) of

overlimit ->

false;

SubTrack ->

braking(SubTrack, MaxB, SpeedBraking)

end

end.

-spec move(track(), units(), speed()) -> track() | overlimit.

move([{0,0}], _NumOfMoves, _Speed) ->

[];

move([{N,V} | SubTrack], NumOfMoves, Speed) ->

case Speed =< V of

true ->

if N =:= NumOfMoves ->

SubTrack;

N > NumOfMoves ->

[{N-NumOfMoves, V} | SubTrack];

true ->

MovesLeft = NumOfMoves-N,

move(SubTrack, MovesLeft, Speed)

end;

false ->

overlimit

end.

39

A Methodology for Applying Concolic Testing

Documents