Volker Fortströer. How improve quality control for reactive systems? Availability of a specification of system behavior ◦ Rarely precise & reliable.

Seminar: Software Quality and Safety

Learning By TestingVolker Fortströer

Volker Fortströer – Learning by Testing 2

How improve quality control for reactive systems?

Availability of a specification of system behavior◦ Rarely precise & reliable because of last minute

changes etc. But a specification of the system is needed

to test and to verify the behavior Automatically generated and maintained

reference models can help to get precise system specification

Motivation [1]

Volker Fortströer - Learning by Testing 3

Model generation from source code is possible but not practicable ◦ hardware systems, closed source components

An automated method is needed to generate a model of a (reactive) system

Motivation [2]

Volker Fortströer - Learning by Testing 4

Reactive systems Regular languages Finite-state automaton Learning an automaton Angluins Algorithm L* Optimizations to L*

Overview

Volker Fortströer - Learning by Testing 5

Systems which creates output on different input and may change internal states◦ i.e.: telecommunication systems

Preconditions:◦ The system responses have to be deterministic◦ The same input results in the same end state (output)◦ All input is accepted regardless of the system’s internal

state Conclusion: such a system can be seen as an

input/output automaton This automaton can be learned through

algorithms by creating a deterministic automaton

Reactive Systems

Volker Fortströer - Learning by Testing 6

A deterministic finite-state automaton (DFA) over Σ is a structure A = (Q, δ, q0, F).◦ Q: non-empty finite set of states◦ q0 є Q: initial state◦ F ⊆ Q: set of final states◦ δ: Q х Σ → Q is the transition function

Run of A: execution of A with input w◦ w: a1 … an є Σ*◦ Acceptance of w if A stops in a state q є F◦ The set of accepted words build a regular language

Deterministic Finite-State Automaton

Volker Fortströer - Learning by Testing 7

Let L be a language over the alphabet Σ. L ⊆ Σ* is called regular iff. either

◦ L is generated by a regular grammar or◦ L is accepted by a finite automaton or◦ L can be described trough a regular expression

Regular languages

Volker Fortströer - Learning by Testing 8

Algorithm L* invented by Angluin (87)◦ determining an initially unknown regular language

L◦ Two information sources are needed to learn the

regular language L: Membership Oracle (MO) checks if a sequence w is in

L Equivalence Oracle (EO) checks if an acceptor for L

is equivalent to the unknown L◦ Construct a equivalent minimal DFA which

accepts L

Learning a Finite Automata [1]

Volker Fortströer - Learning by Testing 9

Basic idea behind L*:◦ Systematically explore feasible strings using the

MO◦ Construct the transition table◦ maintain during exploration

A set S of state-access strings A set E of explored states A function T for mapping (S ∪ S ∙ A) ∙ E → {0,1}

T is kept in the observation table (OT) A is the input-alphabet

Learning a Finite Automata [2]

Volker Fortströer - Learning by Testing10

◦ Gradually build a DFA M (hypothesis)◦ If M seems to be stable make an equivalence

query If answer is correct then we have found an

equivalent DFA Otherwise we get a counterexample (which can be

used for refining DFA M)

Learning a Finite Automata [3]

Volker Fortströer - Learning by Testing

Assume a reactive system like a telecommunication system

One switch and some phones Phone actions (input):

◦ on-hook (↑)◦ off-hook (↓)◦ perform a call (→)

Switch actions (output):◦ initiated◦ cleared◦ hookswitch (signal received)

11

Application of L* [1]

Volker Fortströer - Learning by Testing

different (simple) scenarios:◦ S1: 1 physical device A

AI = {A↑, A↓}

AO = {initiatedA, clearedA, [hookswitchA]}◦ S2: 2 physical device A, B

AI = {A↑, A↓, B↑, B↓}

AO = {initiatedA,B, clearedA,B, [hookswitchA,B]} depending on input through devices the switch

produces output corresponding to its internal states

⇒ I/O-Automaton

12

Application of L* [2]

Volker Fortströer - Learning by Testing

An input / output automaton over Σ is a structure S = (Σ, AI, AO, →, s0).◦ Σ: non-empty finite set of states◦ AI: finite set of input actions

◦ AO: finite set of output responses

◦ →: transition relation → ⊆ Σ х AI х AO* х Σ

◦ s0 є Σ: initial state An input / output automaton can be transformed

in a deterministic finite-state automaton◦ Divide input words into single symbols◦ Create auxiliary-states for transitions

13

Input / Output Automaton

Volker Fortströer - Learning by Testing14

Represent a I/O-automaton as a DFA◦ Because a large input alphabet results in

inefficient runtime of L* Membership Oracle:

◦ This queries can be answered by the system itself◦ Problems:

Interface to the system is needed for invoking the input und capturing the response

Expensive in time because of timeouts

Adaptations to L* for learning a reactive system [1]

Volker Fortströer - Learning by Testing

Equivalence Oracle:◦ Learning a black-box system

⇒ one can never be sure that the whole system behavior was learned

◦ Using an approximation: Checking consistency within a fixed look ahead from

all states

15

Adaptations to L* for learning a reactive system [2]

Volker Fortströer - Learning by Testing16

Performing an oracle query takes relatively much time◦ L* has to wait for response of tested system

An obviously improvement is reducing the number of queries

Some queries can be answered with the result of other queries◦ i.e.: counterexample of EQ

Other queries can be filtered because of the properties of the learned language

Optimizing the runtime of L*

Volker Fortströer - Learning by Testing17

If every state is an accepting state the corresponding language is prefix-closed◦ All prefixes v of every word w in language L are

member of L Filter 1 (positive prefix):

∃σ2 ∈ A*. T(σ1;σ2) = 1 ⇒ MO(σ1) = true Filter 2 (negative prefix):

∃σ2 ∈ prefix(σ1). T(σ2) = 0 ⇒ MO(σ1) = false

Prefix-Closure

Volker Fortströer - Learning by Testing18

Every input produces always the same output

Filter 3 (Input Determinism):∃x ∈ AO, y ∈ A, σ2, σ3 ∈ A*. σ1 = σ2;x;σ3 ∧ T(σ2;y;σ3 ) = 1 ∧ x ≠ y ⇒ MO(σ1) = false

Filter 4 (Output Completion): ∃a ∈ AI, x ∈ AO. σ2;x ∈ prefix(σ1) ∧ T(σ2;a) = 1 ⇒ MO(σ1) = false

Input Determinism

Volker Fortströer - Learning by Testing19

Reactive systems exhibit often a high degree of parallelism

Sometimes different components of one type are interchangeable◦ i.e.: a device A behaves like device B◦ Independence:

actions of device A and B can be performed in different order

◦ Symmetry:an action performed by A can also be performed by B

Independence of Events [1]

Volker Fortströer - Learning by Testing

A membership query for σ1 is true if an equivalent σ2 is already in OT

Let Cpo,sym be the set of all equivalent sequences up to the given partial order and symmetry

Filter 5 (Partial Order): ∃σ2 ∈ Cpo,sym(σ1). T(σ2) = 1 ⇒ MO(σ1) = true

20

Independence of Events [2]

Volker Fortströer - Learning by Testing

The latter filter depends strongly on the concrete application domain,but there exist an pattern for realization:

1. An expert specifies an independence relation2. Abstraction by replacing concrete identifiers

with generic place holders3. σ is inspected if it contains independent

subparts4. All re-orderings are computed5. Generic place holders are replaced again by

concrete identifiers

21

Independence Pattern

Volker Fortströer - Learning by Testing

remember the two scenarios S1, S2

Number of Membership Queries:

relatively great effect other scenario with 80 states has a total

factor of 459.5 after applying the filters

22

Conclusion

States

no filter

1 & 2

Factor 3 & 4

Factor 5 Factor total Factor

S1 4 108 30 3.6 15 2.0 14 1.1 7.7

S2 12 2431 593 4.1 218 2.7 97 2.2 25.1

Volker Fortströer - Learning by Testing

Thanks for your attention!

References:◦ Hardi Hungar, Oliver Niese, Bernhard Steffen: Domain-Specific

Optimization in Automata Learning. CAV 2003: 315-327◦ Therese Berg, Bengt Jonsson, Martin Leucker, and Mayank Saksena (SVV

2003), Mumbai (India): Insights to Angluin's learning. Electr. Notes Theor. Comput. Sci. 118: 3-18 (2005).

23