Effective Test Suites for ! Mixed Discrete-Continuous Stateflow Controllers

.lusoftware verification & validationVVS

Effective Test Suites for !Mixed Discrete-Continuous

Stateflow Controllers Reza Matinnejad Shiva Nejati Lionel Briand SnT Center, University of Luxembourg

Thomas Bruckmann Delphi Automotive Systems, Luxembourg

Cyber Physical Systems (CPSs) Combination of computations (algorithms) and physical dynamics (differential equations)

2

Physical world Computation

Testing (Typical) Software

3

X = 10, Y = 30

Z = 20

Algorithms

Fail Pass Z = 10

Testing (CPS) Software

4

Algorithms + Differential Equations

Fail Z = 20

X = 10, Y = 30 S1(t) S2(t)

S3(t) Pass

Z = 20 S3(t)

S1t

S2t

S3t

S3t

Software Testing Challenges (CPS)

• Mixed discrete-continuous behavior (combination of algorithms and continuous dynamics)

•  Inputs/outputs are signals (functions over time)

• Simulation is inexpensive but not yet systematically automated

• Partial test oracles

5

Generating effective test suites for Software used in !

Cyber-Physical Systems

6

Our Goal

Simulink/Stateflow

• A data flow-driven block diagram language

• Is widely used to develop Cyber Physical Systems

• Is executable

7

Stateflow

• A Statechart dialect integrated into Simulink

• Captures the state-based behavior of CPS software

• Has mixed discrete-continuous behavior

8

Generating effective test suites for mixed discrete-continuous

Stateflow controllers

9

Our Goal

Discrete Behavior What we typically think of software models

10

On

Off

On

Off

Speed < 10 Speed > 10

Discrete-Continuous Behavior What software models are actually being built using Stateflow

11

On

Off

CtrlSig

On

Off

Speed < 10 Speed > 10

tCtrlSig

tCtrlSig

Generating effective test suites for mixed discrete-continuous

Stateflow controllers

12

Our Goal

Test Suite Effectiveness (1) •  Test suite size should be small because

•  Test oracles cannot be fully automated

•  Output signals need to be inspected by engineers

13

ModelSimulation

InputSignals

OutputSignal(s)

S3t

S2t

S1t

S3t

S2t

S1t

Test Case 1

Test Case 2

Test Suite Effectiveness (2) •  Test suites should have a high fault revealing power

•  Small deviations in outputs may not be recognized/important

•  Test inputs that drastically impact the output signal shape are likely to have a higher fault revealing power

14

Test Output 1

TimeTime

CtrlSig

Faulty Model OutputCorrect Model Output

Test Output 2

Test Generation Algorithms!!

15

Our Approach

Test Generation Algorithms •  Input-based Test Generation:

•  Input Diversity Algorithm

•  Coverage-based Test Generation:

•  State Coverage Algorithm

•  Transition Coverage Algorithm

•  Output-based Test Generation:

•  Output Diversity Algorithm

•  Failure-based Algorithm

16

Input Diversity • Maximizing distances among input signals

17

Test Case 1

Test Case 2

Input Signal 1 Input Signal 2

S1t

S1t

S2t

S2t

Distance Between Signals

18

Time

Signal

Test Generation Algorithms •  Input-based Test Generation:

•  Input Diversity Algorithm

•  Coverage-based Test Generation:

•  State Coverage Algorithm

•  Transition Coverage Algorithm

•  Output-based Test Generation:

•  Output Diversity Algorithm

•  Failure-based Algorithm

19

Structural Coverage

• Maximizing the number of states/transitions covered

20

State Coverage Transition Coverage

1

4

2

3

1

4

2

3

Test Generation Algorithms •  Input-based Test Generation:

•  Input Diversity Algorithm

•  Coverage-based Test Generation:

•  State Coverage Algorithm

•  Transition Coverage Algorithm

•  Output-based Test Generation:

•  Output Diversity Algorithm

•  Failure-based Algorithm

21

Output Diversity • Maximizing distances among output signals

22

Test Case 1

Test Case 2

Output Signal

S3t

S3t

Failure-based Test Generation

23

Instability Discontinuity

0.0 1.0 2.0-1.0

-0.5

0.0

0.5

1.0

Time

Ctr

lSig

Output

• Maximizing the likelihood of presence of specific failure patterns in output signals

0.0 1.0 2.0Time

0.0

0.25

0.50

0.75

1.0

Ctr

lSig

Output

We developed our failure-based test generation algorithm using!

Meta-Heuristic Search

24

The Alternative Choice

25

Our ApproachExisting WorkTechnique

ModelChecking

- Require precisely definedoracles (user-specified assertions)

- Have been largely appliedto time-discrete models

- State-explosion problem!

- No need for automated test oracles

- Applicable to time-continuousand non-linear models

- Our algorithms are black-boxrandomized search: - non-memory intensive - can be parallelized

26

Failure-based Test Generation using Meta-Heuristic Search

Input Signals

Slightly Modifying Each Input Signal

Fitness Functions Capturing the Likelihood

of Presence of Failure Patterns in the Output Signals

Repeat

Until maximum resources spent

S Initial Candidate Solution

Search Procedure

R Tweak (S)

if Fitness (R) > Fitness (S)

S R

Return S

Output Stability !Fitness Function

• Sum of the differences of signal values for consecutive simulation steps

27

stability(sgo

) =kP

i=1|sg

o

(i ·�t)� sgo

((i� 1) ·�t)|

0.0 1.0 2.0-1.0

-0.5

0.0

0.5

1.0

Time

Ctr

lSig

Output

Output Continuity !Fitness Function

28

• Maximum of the minimum left or right derivatives for all the simulation steps

0.0 1.0 2.0Time

0.0

0.25

0.50

0.75

1.0

Ctr

lSig

Output

continuity(sg

o

) =

K�1max

i=1(min(|LeftDer(sg

o

, i)|, |RightDer(sgo

, i)|))

Comparing the!Test Generation Algorithms!

!

29

Evaluation

Research Questions

•  RQ1 (Fault Revealing Ability)

•  RQ2 (Fault Revealing Subsumption)

•  RQ3 (Test Suite Size)

30

Experiment Setup • Three Stateflow models: two industrial and one publicly

available case study

31

75 (faulty models) * 100 (algorithm runs) *6 (generation algorithms) * 5 (different test suite sizes) =

225,000 test suites (in total)

Test Suite(size=3,5, 10,25,50)

{1.Fault

Seeding2.Generation

AlgorithmSF FaultySF

{75 75

Research Question 1!Fault Revealing Ability

How does the fault revealing ability of our proposed test generation algorithms

compare with one another?

32

1.0

0.0

0.5

Input Diversity

OutputDiversity

FaultRevealing

Rate

RQ1: Fault Revealing Ability

33

1.  Output-based and coverage-based algorithms outperformed the input diversity algorithm

2.  Output-based algorithms outperformed the coverage-based algorithms

3.  Overall, output stability algorithm performed the best

Research Question 2!Fault Revealing Subsumption

Is any of our generation algorithms subsumed by other algorithms?

34

RQ2: Fault Revealing Subsumption

35

•  For each of the 75 faulty models, we identified the best generation algorithm(s) for different test suite sizes (5, 10, 25, and 50)

Fault 1State Coverage

Transition Coverage

Output Diversity

Output Stability

Output Continuity

Fault 2 Fault 3 Fault 4

RQ2: Fault Revealing Subsumption (2)

36

1.  The coverage-based algorithms found the least number of faults

2.  Coverage-based algorithms are subsumed by output diversity algorithm when the test suite size increases (size = 25 , 50)

Research Question 3!Test Suite Size

What is the impact of the size of test suites generated by our generation algorithms on

their fault revealing ability?

37

RQ3: Test Suite Size

38

1.  The fault revealing rates for output stability/continuity is very high for small test suites(size = 3,5) for Instability/Discontinuity failures

2.  For Other failures, the ability of output diversity in revealing failures rapidly increases as the test suite size increases

DiscontinuityInstability Others

0.0

0.5

1.0

3 5 10 25 50

Test Suite Size

Faul

t Rev

ealin

g Ra

te M

ean

3 5 10 25 50 3 5 10 25 50

Output StabilityOuput Continuity State Coverage

Transition CoverageOutput Diversity

Lessons Learned

39

Lesson 1!Coverage-based algorithms are less

effective than output-based algorithms •  The test cases resulting from state/transition coverage

algorithms cover the faulty parts of the models

•  97% state coverage and 81% transition coverage

•  Cover faulty parts for 73 (out of 75) fault-seeded models

• However, they fail to generate output signals that are sufficiently distinct from the oracle signal, hence yielding a low fault revealing rate

40

Lesson 2!Combining Output-based Algorithms

41

•  We suggest to divide the test suite size budget between output-based algorithms:

Output Continuity Output Stability Output Diversity

CoCoTest

42

.lusoftware verification & validationVVS

Effective Test Suites for !Mixed Discrete-Continuous

Stateflow Controllers Reza Matinnejad (reza.matinnejad@uni.lu) Shiva Nejati Lionel Briand SnT Center, University of Luxembourg

Thomas Bruckmann Delphi Automotive Systems, Luxembourg

Lesson 1!Combing Output-based Algorithms

•  We suggest to divide the test suite size budget between output stability, output continuity, and output diversity:

1.  Allocate a small part of the test budget to output continuity

2.  Share the rest of the budget between output stability and output diversity, by giving output diversity a higher share

44

Input / Output Vectors

45

0 5 10

50

150

250

Fuel

Lev

el S

enso

r

Fuel

Lev

el0 5 10

100.0

91.43

84.43

75.6270.01

66.1961.21

56.6654.3252.81

50

100

Time (s) Time (s)

Study subjects

46

Publicly AvailableName No. of

InputsHierarchical

States ParallelismNo. of States

SCPCASS

NoNo

2342

1316

2 No1 No

GCS Yes 8 10 0 Yes

No. of Transitions

2553

27

• SCPC: Supercharger Clutch Position Controller

• ASS: Auto Start Stop Control

• GCS: Guidance Control System

Fault Revealing Rate (FRR)

47

FRR(SF ,TS ) =

(1 91iq

ˆdist(sgi, gi) > THR

0 81iqˆdist(sgi, gi) <= THR

•  FRR based on gi, output of the fault-free model, sgi, output of the fault-seeded model, and a threshold THR:

1.  For continuous dynamic systems, the system output is acceptable when the deviation is small and not necessarily zero

2.  It is more likely that manual testers recognize a faulty output signal when the signal shape drastically differs from the oracle.

RQ3: Test Suite Size

48

1.  The fault revealing rates for output stability/continuity is very high for small test suites for Instability/Discontinuity

2.  For “Other” failures, the ability of OD in revealing failures rapidly increases as the test suite size increases

Discontinuity

SCTC

ODOSOC* *+ +

--

Instability Others

0.0

0.5

1.0

3 5 10 25 50

**

* *

+

++

--

Test Suite Size

FRR

Mea

n

+-

-

3 5 10 25 50 3 5 10 25 50

* * *

-

--

+

+

+ - - -*

* *

+

++

+

+*

* *---