Heuristics for Improving Model Learning Based Testing Muhammad Naeem Irfan VASCO-LIG, Computer Science Lab, Grenoble Universities, 38402 Saint Martin d’Hères.

Heuristics for Improving Model Learning Based

Testing

Muhammad Naeem IrfanVASCO-LIG, Computer Science Lab,

Grenoble Universities, 38402 Saint Martin d’Hères

1

Introduction

Formal models of such components is missing Methods to construct the model from source code

are available Source code for components (black boxes) is

most often missing Construct the model by sending inputs and

observing outputs

Component Based Software Engineering

2

Black Box

Introduction

Reverse engineering models

Learner

Tester

Observe outputs for inputs

Learn model

Learned Model

Test model

3

Angluin Style Learning

Complexity O( |I|² nm + |I| m n²) I : the size of the input set n : the number of states in the actual machine m : the length of the longest counterexample

Method by Shahbaz to process the counter exampleO( |I|² n + |I| m n²)

Observe outputs

Send inputs

Input set I

Perform test

Oracle

“OK” or Counterexample

Learnt Model

Algorithm

4

Problem Statement

The length of counterexample m is important parameter for the complexity

Oracle is of low quality How we can reduce relying on the quality of counterexample Process the counterexample more efficiently

Observe outputs

Send inputs

Input set I

Perform test

Oracle

“OK” or Counterexample

Learnt Model

Algorithm

5

Conjecture

Black Box

Learning Black Box

q0q0 q2

q2

q1q1

b/y a/x

a/x

b/xa/x

b/y

q0

a b

ε x y

a x y

b x y

Observation Table

Discriminating Sequences

States

Eq

uivalen

t S

tates

a b a b a a b

x y x y x x y

x y x y x x x

Counterexample:

Conjecture:

Black Box:6

Discriminating Sequences Discriminating Sequences

a b

ε x y

a x y

b x y

a b

ε x y

a x y

b x y

abaababaabbabaabbaab

Processing the Counterexample

Counterexample: ab b a b a a ba

States

Eq

. Sta

tes

Observation Table(Previous Method)

Observation Table(New Method)

7

Difference between Algorithms

Add suffixes to Discriminating Sequences until new state is identified

Observation Table after processing counterexample with previous method

a b ab aab baab abaab babaab

ε x y xy xxx yxxx xyxxx yxyxxx

a x y xx xxx yxxx xxxxx yxyxxx

aa x x xx xxx xxxx xxxxx xxxxxx

b x y xy xxx yxxx xyxxx yxyxxx

ab x y xy xxx yxxx xyxxx yxyxxx

aaa x x xx xxx xxxx xxxxx xxxxxx

aab x y xx xxx yxxx xxxxx yxyxxx

a b ab

ε x y xy

a x y xx

aa x x xx

b x y xy

ab x y xy

aaa x x xx

aab x y xx

Observation Table after processing counterexample

Processing the Counterexample

a b a b a a b

2149

8

Edinburg Concurrency Workbench Results

Examples TestsPrev

MethodCounterExample

Prev Method

Buff3 259 259 4 4

Peterson-2 340 374 6 5

VM 392 448 2 2

VMnew 1638 1521 8 6

Sched2 553 790 1 1

CSPROT 3094 3757 7 5

System1 5936 9328 5 3

Gain for number of tests conducted Lose for the number of counter

examples processed

CWB Examples Tests

Counterexamples

9

Comparison of two Methods

New method less number of discriminating sequences

Worst case complexity is same

Previous method processing all the discriminating sequences

Gain for processing the Counterexample from Random Walk method

10

Perspectives

Find efficient heuristics to bring down the average complexity of algorithm Process the counterexamples

Processing sparse tables

How to deal with the abstractions to be made on actual interactions with black box components

How to perform random walks to get the counterexample

11

Questions

12