Graph Methods for Generating Test Cases with Universal and Existential Constraints (Talk @ ICTSS 2015)

Graph Methods for GeneratingTest Cases with Universal and

Existential Constraints

Sylvain Hallé, Edmond La Chanceand Sébastien Gaboury

Université du Québec à Chicoutimi, Canada

http://www.liflab.ca

November 23rd, 2015

Example:

How to test the system in such a way that we cover all possible interactions between two inputs?

can each be in position 1, 2 or 3, ,

Naïve method: enumerate all combinations of values

All combinations of values for any two inputs are present

How many test cases are there?

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3

1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3

In fact, this table verifies all combinations of 3 inputs exactly once... but combinations of 2 appear more than once

Can we observe them all by doing fewer tests?

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3

1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3

Here's how:

Each combination of values for 2 given parameters appears the same number of times (here, once)

We call this an orthogonal array (OA)

3 2 1 2 1 3 1 3 2

1 2 3 1 2 3 1 2 3

1 1 1 2 2 2 3 3 3

How many testcases this time?

Pairwise testingTesting strategy that identifies interactions between two input parameters, and tests all combinations of values for the selected pairs

Intuition: some problems occur only through an interaction between many (e.g. 2) parameters

By varying each parameter individually, it isunlikely to generate the pair causing the error

We can use orthogonal arrays to get the test casesto try

Exemple of an interaction:

The error occurs only when altitude_adj == 0 AND volume < 2.2

if (altitude_adj == 0) { // do something if (volume < 2.2) { faulty code! BOOM! } else { good code, no problem }} else { // do something else}

10 effects, each with 2 possible values (on/off)

All combinations:210 = 1024 tests

How many tests to coverall interactions of 3effects?

There are = 120 triplets of effects

For each triplet, we have to test 23 = 8 combinations of values. We need a maximum of 960 tests.

But each test exercises three triplets; hence we need no more than 320 test to cover everything...

0 1 1 0 1 0 1 0 1 0

...and in fact, each test covers much more than three triplets

103( )

Quel est le nombre minimum detests?

Actually, only 13 tests are necessary!

Is it an orthogonalarray?

0 0 0 0 0 0 0 0 0 01 1 1 1 1 1 1 1 1 11 1 1 0 1 0 0 0 0 11 0 1 1 0 1 0 1 0 01 0 0 0 1 1 1 0 0 00 1 1 0 0 1 0 0 1 0 0 0 1 0 1 0 1 1 1 01 1 0 1 0 0 1 0 1 00 0 0 1 1 1 0 0 1 10 0 1 1 0 0 1 0 0 10 1 0 1 1 0 0 1 0 01 0 0 0 0 0 0 1 1 10 1 0 0 0 1 1 1 0 1

Covering arrayAn array CA(N, k, v, t) such that:

The previous array is CA(13, 10, 2, 3)

The array contains N lines

We consider k parameters

Each parameter can have v values

Each combination of values for t parameters occurs at least once in the array

t-way test case generation

Back to our example:

But...

can each be in position 1, 2 or 3, ,

must always be greater than

and can never be at position

1 at the same time

Back to our example:

But...

can each be in position 1, 2 or 3, ,

must always be greater than

and can never be at position

1 at the same time

Universal constraints

Universal constraintConstraint on the possible values for a test case

Every test case must follow the constraint to be considered valid

We wish to find the smallest covering array CA(N, k, v, t), for a given set of constraints

May lead to a smaller or a larger covering array, depending on the constraints

Example: mutually exclusive parameters (subscript/superscript)

Back to our example:

Also...

The test suite shouldinclude a case where allinputs are different

At least one test should try and at the same position

can each be in position 1, 2 or 3, ,

Back to our example:

Also...

The test suite shouldinclude a case where allinputs are different

At least one test should try and at the same position

can each be in position 1, 2 or 3, ,

Existential constraints

Existential constraintConstraint on the possible values for a test case

At least one test case must follow the constraint for the test suite to be considered valid

We wish to find the smallest covering array CA(N, k, v, t), for a given set of constraints

May lead to a smaller or a larger covering array, depending on the constraints

Same as before

Why use existential constraints?

Three scenarios (at least):

See the paperb

Extending an existing test suite (add onecondition for each existing test)Relaxed t-way (Mone combination sufficesM)MC/DC testing

Let D1, D2, ..., Dk be the domains for k parameters named p1, p2, ..., pk

Let Φ = {φ1, φ2, ..., φm} be a set of m arbitrary Boolean expressions, whose ground terms are of the form pi = d, or pi = pj, for d ∈ Di

Let Σ ⊆ D1 × D2 × ... × Dk be a set of value assignments for each parameter

We call Σ a Φ-way covering if, for every φ ∈ Φ, there exists an assignment σ ∈ Σ such that σ |= φ

Φ is the set of existential constraintsΣ is the set of test cases

Back to our example:

Also...

The test suite shouldinclude a case where allinputs are different

At least one test should try and at the same position

can each be in position 1, 2 or 3, ,

Back to our example:

Also...

The test suite shouldinclude a case where allinputs are different

At least one test should try and at the same position

D1 = D2 = D3 = {1,2,3}

Back to our example:

Also...

The test suite shouldinclude a case where allinputs are different

p2 = p3

D1 = D2 = D3 = {1,2,3}

Back to our example:

Also...

p1 ≠ p2 ∧ p2 ≠ p3 ∧ p1 ≠ p3

p2 = p3

D1 = D2 = D3 = {1,2,3}

Any t-way problem can be converted into a Φ-way problem...

Let I = {S ⊆ [1,k] : |S| = t}

Let DS = Di

Define Φ as:∩

S ∈ I {d1,...,dn} ∈ DS

∏i ∈ S

∩

pi = di∧i ∈ S

How do we find a Φ-way covering?

Two solutions based on reductions to graph problems

Graph colouring

Hypergraph vertex covering

Reduction to graph colouring

Create the graph G = <V,E> such that:

Let κ : V → [1,n] be a colouring of G with n colours. Define σκ,i as any variable assignment satisfying:

Then Σ = σκ,i is a Φ-way covering (theorem)

V = ΦE = { (φ,φ') : φ ∧ φ' is unsatisfiable }

φ∧φ ∈ κ-1(i)∩

i ∈ [1,n]

Informally...

Create one vertex per constraint

Link pairs of vertices that cannot be true at the same time

Find a colouring for this graph

Create a test case for each colour (take the conjunction of all conditions for nodes ot this colour)

Example with

a = 0 ∧ b = 0, a = 0 ∧ b = 1, a = 1 ∧ b = 0,a = 1 ∧ b = 1, a = 0 ∧ c = 0, a = 0 ∧ c = 1,a = 1 ∧ c = 0, a = 1 ∧ c = 1, b = 0 ∧ c = 0,b = 0 ∧ c = 1, b = 1 ∧ c = 0, b = 1 ∧ c = 1 }

Φ = {

a=0 ∧ b=0

a=1 ∧ b=0

a=1 ∧ b=1a=1 ∧ c=0

a=1 ∧ c=1

a=0 ∧ b=1

b=1 ∧ c=0

b=1 ∧ c=1

b=0 ∧ c=0

b=0 ∧ c=1a=0 ∧ c=0

a=0 ∧ c=1

Pairwise testing for a, b, c ∈ {0,1}

a=1,b=0,c=1

a=1,b=0,c=0

a=1,b=1,c=0

a=1,b=1,c=1

a=0,b=1,c=1

Testcases

Some remarks:

The conjunction of all conditions of the same colour must be satisfiable. Counter-example:

Classical t-way problems (and many others) always satisfy this constraint

The minimum number of tests is the chromatic number of G

⇒ "Guaranteed" lower bound

a=0 ∨ b=0 a=1 b=1

1.

2.

3.

Reduction to hypergraph vertex covering

Create the hypergraph G = <V,E> such that:

Let Σ ⊆ V be a vertex covering of G. Then Σ is a Φ-way covering (theorem)

V = D1 × D2 × ... × Dk

E = {S ⊆ V : there exists φ ∈ Φ such that s ∈ S iff s |= φ}

Works all the time!

Can even deal with universalconstraints (remove verticesthat violate any of them)

Example with

a = 0, b = 0, a ≠ 0 ∨ b ≠ 0 }Φ = {

a=0,b=0

Testcases

a=0, b=0

a=0, b=1

1

a=1, b=0

2

3

a=1, b=1

3

3

a=1,b=1

a : 0, 1, 2b : 0, 1c : 2, 3, 4, 5

Once a != bOnce a < b

Always !(a > b) || c == 2

Both techniques implemented in a test case generation tool

Only known tool to handle existential AND universal constraints

Uses a simple extension of the input format used by PICT

Domains

Existential constraints

Universalconstraints

t=2AllPairs QICT

t >2

EA

Vertexcover

TCases

JennyForbiddentuples

Colouring

5

10

15

20

25

30

35

40

45

50

55

2 3 4 5 6 7 8 9

Test

siz

e

n

Test size with t:2, D:3

QICTColouring

AllPairsTCases

Jennyhypergraph

For pairwise test case generation...

With t = 3...

20

30

40

50

60

70

80

90

100

110

120

3 4 5 6 7 8

Test

siz

e

n

Test size with t:3, D:3

ColouringTCases

JennyHitting

Fewer toolsremain...

With universal constraints...

With existential constraints...

Still fewertools...

Hypergraph6060118

TCases158203254

n567

Hypergraph6566110

n567

Last onestanding!

Existential constraints in test case generation arise in a variety of situations, but are not addressed by existing tools/research

Finding an optimal test suite can be reduced to finding the optimal solution of two well-known graph problems

Empirically, existing heuristics for these problems perform at least as well as avariety of existing tools

https://bitbucket.org/ sylvainhalle/gcases

Take-home points...