A Novel Test Data Generation Approach Based Upon Mutation Testing by Using Artificial Immune System for Simulink Models

Applying the meta-heuristic algorithms

for mutation-based Test data generation

for Simulink models

Le Thi My Hanh, Nguyen Thanh Binh, Khuat Thanh Tung

DATIC Laboratory

University of Science and Technology

Danang

December 2014

Contents

Introduction

Related work

Mutation testing for Simulink

Test data generation

Experimentation

Conclusion

2

1. Introduction Simulink

Mutation testing

2. Related work

3. Mutation testing for Simulink Mutation operators

Mutant generation and execution - MuSimulink

4. Test data generation Random test data generation

Mutation-based test data generation

5. Experimentation

6. Conclusion

Simulink

Simulink

Mutation testing

3

Block diagram environment

multi-domain simulation, model-based design

Being widely used for developing embedded systems,

control systems…

aerospace, automobile and electronics systems

Provides a graphical editor, customizable block libraries

A Simulink model consists of two main elements: blocks and lines

blocks are the functional units, used to generate, manipulate and

output signals

blocks are connected by lines that provide the mechanism to transfer

signals

It is integrated with Matlab, enabling to incorporate Matlab

algorithms into models and export simulation results to Matlab for

further analysis

Introduction

Related work



Experimentation

Conclusion

Software testing

4

Important activity in software process in order to assure the

quality

Discovering faults

Detecting faults as soon as possible in software process to

reduce development costs

Detecting faults in design phase

Several testing techniques: black-box, white-box...

Mutation testing: popular and easy to automate

Mutation testing for Simulink models

Simulink

Mutation testing

Introduction

Related work



Experimentation

Conclusion

Mutation testing

5

Fault-based testing technique

Measuring the quality of a test set according to its ability

to detect specific faults

Faults are inserted into the original program

Mutation operators: representing common faults usually found in

program

Faulty program is called mutant

Example

Original

if (a > 5000) ...

Mutant

if (a < 5000) ...

Simulink

Mutation testing

Introduction

Related work



Experimentation

Conclusion

Mutation testing

6

Process

Generating mutants

Executing the original program and the mutant on the same

test data

Comparing the output of the original program and the output

of the mutant

If they are different, the mutant is killed

If they are not different, the mutant is alive

Simulink

Mutation testing

Introduction

Related work



Experimentation

Conclusion

Mutation testing

Introduction

Related work



Experimentation

Conclusion

Simulink

Mutation testing

7

Concepts

Killed mutant

There exists a test case distinguishing the original program and the

mutant

Alive mutant

There is no test case distinguishing the original program and the mutant

Equivalent mutant

The outputs of the original program and the mutant are always the same

Mutation score (MS)

The score of the number of killed mutants over the total number of non

equivalent mutants

Measuring the quality of the set of test cases

Mutation testing

Introduction

Related work



Experimentation

Conclusion

Simulink

Mutation testing

8

Disadvantages

determining equivalent mutants

generating mutants

cost of executing mutant

test data generation

Related work

Introduction

Related work



Experimentation

Conclusion

9

Mutation testing

• For some programming languages: C, Fortran, Ada, Java, SQL, . . .

• Three mutation operators for Simulink: add, multiply, assign for signals

carried on inputs (Zhan và Clark)

Mutation testing tools

• Mothra (Fortran), MuJava (Java), Proteum (C), SQLMutation (SQL), ...

Cost reduction techniques

Do fewer

• Selective mutation

• Mutant sampling

Do faster

• Schema-based Mutation Analysis

• Separate compilation

• Compilation bottleneck

Do smarter

• Weak mutation

• Distributed Architectures: SIMD (Krauser - 1991), MIMD (Offut - 1992)

Objectives

Introduction

Related work



Experimentation

Conclusion

Objectives

Mutation operators

MuSimulink Tool

10

Defining a set of mutation operators for Simulink

Basing on common faults committed by designers

Developing a mutation testing tool for Simulink - MuSimulink

Automatic generation of mutants

Executing mutants

Improving the cost by applying parallelism

Generating test data

Improving test data by meta-heuristic algorithms

Mutation operators for Simulink

Introduction

Related work



Experimentation

Conclusion

Fault Classes

Mutation operators

MuSimulink Tool

11

Operators Description

TRO Types Replacement Operator

VCO Variable Change Operator

VNO Variable Negation Operator

CCO Constant Change Operator

CRO Constant Replacement Operator

SCO Statement Change Operator

SSO Statement Swap Operator

DCO Delay Change Operator

ROR Relational Operator Replacement Operator

AOR Arithmetic Operator Replacement Operator

ASR Arithmetic Sign Replacement Operator

LOR Logical Operator Replacement Operator

BRO Block Removal Operator

MuSimulink Tool

Introduction

Related work



Experimentation

Conclusion

Fault Classes

Mutation operators

MuSimulink Tool

12

MuSimulink tool

Generating mutants

Translating model into an intermediate form (oriented graph)

Applying a set of mutation operators to generate mutants

Storing mutants in a Mutants Description Table - MDT

o Each element in this table is a record describing a mutant

Generating random test data

Executing the original model and each mutant on the test data

Analysing the results

List of killed mutants and list of alive mutants

Computing mutation score


13

Automatic test data generation

killing the most mutants

Two approaches

Random generation

do not take care mutants

basing only upon the input specification

Mutation-based generation

generated test data should kill the most mutants

basing on mutants

Applying the meta-heuristic algorithms

Introduction

Related work



Experimentation

Conclusion

Random test data generation



14

Process The original model

Determining the inputs Desc. of input domains

Random generation

Executing the original model

on test data

Executing the mutant

models on test data

Verifying if the test data

kills mutants?

Marking the test data

Introduction

Related work



Experimentation

Conclusion




15

Using meta-heuristic algorithms

Genetic Algorithm (GA)

Simulated Annealing (SA)

Artificial Immune System (AIS)

Introduction

Related work



Experimentation

Conclusion



Using Genetic Algorithm

16

Gen

etic

Alg

ori

thm

Individual representation

Set of test cases

Individual selection

Roulette Wheel Selection Technique

Retaining the best individual in each generation

Crossover operator

Two points crossover

Mutation operator

Single point mutation: replacing the test case at

randomly chosen locus by another (random generated test case)

Fitness Function

The total number of killed mutants by all the tests combined into the individual

Father

Mother

Child 1

Child 2

Introduction

Related work



Experimentation

Conclusion



Using Simulated Annealing

17

Sim

ulat

ed A

nnea

ling

Solution representation

Set of test cases

Search process

Being controlled by gradually changing the value of a parameter called temperature, from high to low

Depending on the current temperature

Selecting the best solution

Mutating the current solution to get a new

solution

New solutions with lower cost are always accepted

The probability of accepting a worse state is given by the equation: 𝑒−𝛿/𝑇

𝛿 = 𝑓 𝑛𝑒𝑤𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛 − 𝑓 𝑐𝑢𝑟𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛

T : current temperature

Cost function f = 1 – Mutation Score

Introduction

Related work

Mutation operators for Simulink

Mutant generation and execution


Experimentation

Conclusion



Using Artificial Immune System

18

Art

ific

ial I

mm

une

Syst

em Inspired by

human immune

system

B-cells produce antibodies

Interaction between antibody and antigen - affinity

Four main immunological theories

negative selection theory

immune network theory

danger theory

Clonal selection theory

Selecting n of the highest affinity antibodies

The number of clones for an antibody is proportional to its affinity

The mutation is inversely proportional to the affinity

Introduction

Related work



Experimentation

Conclusion



Using Artificial Immune System

19

Mapping between AIS and mutation testing

AIS Mutation testing

B-cell Model under test

Antigen Mutant

Antibody Test data

Affinity Mutation score

Clonal Selection Evolving test data

Memory cells The memory set to save test data which is

able to kill mutants

Introduction

Related work



Experimentation

Conclusion



Experimentation

20

Configuration

CPU: Intel Core 2 Quad CPU Q6600 , 2.4 GHz

Ram: 2GB

OS: Windows 7

Using 5 Simulink models

Model No of Input Var No of Mutants

SmplSw 2 92

Quadratic_v1 2 161

RandMdl 3 188

Tiny 3 144

Quadratic_v2 3 140

Introduction

Related work



Experimentation

Conclusion

Comparing approaches

The improvement of genetic algorithm

The effectiveness of the algorithms



Introduction

Related work



Experimentation

Conclusion

21

Model No. of

mutants

Mutation Score (%) Time (s)

AIS GA SA AIS GA SA

SmplSw 92 96.74 95.65 94.57 2772.85 14348.81 10474.15

Quadratic_v1 161 88.82 86.33 87.58 8619.93 21445.35 19943.68

RandMdl 188 94.68 86.17 88.30 6629.87 23370.20 21821.83

Tiny 144 93.06 86.81 88.19 4451.38 12474.87 10360.35

Quadratic_v2 140 81.43 79.29 75.71 12670.05 34858.63 32120.53

The best mutation scores of experimentations The average execution time

The effectiveness of the algorithms



Introduction

Related work



Experimentation

Conclusion

22

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

SmplSw Quadratic_v1 RandMdl Tiny Quadratic_v2

%

Comparing the execution time of approaches

GA SA AIS

Hybrid Artificial Immune Genetic Algorithm



23

Individual representation Each individual contains only a test data: v = {a1, a2,…, an},

n is the number of variable and/or inport of Simulink model and ai is the value of ith input data.

Genetic operators Selection operator and mutation operator are the same as the GA

Crossover operator Single point crossover between two parent individuals to create two offspring

Only one child individual is created and its value will be the arithmetic mean of values in two parent individuals

If the number of inputs of Simulink model is one

This crossover always occur in the each generation so we do not use crossover probability.

The fitness for each individual the fitness computed for each individual by basing on alive mutants in that

generation. The fitness function for each individual is the total number of killed mutants by that individual

per the total number of alive mutants in current genetic generation.

test data that are able to kill at least one mutant which is not killed by any individual else will be added to memory set.

Introduction

Related work



Experimentation

Conclusion

The effectiveness of the proposed hybrid algorithm



Introduction

Related work



Experimentation

Conclusion

24

Test

Set


HAIGA GA AIS HAIGA GA AIS

1 90.97 86.81 88.89 7168.9 10009.9 2937.8

2 93.06 88.89 93.06 12802.1 17632.1 4020.5

3 93.06 86.81 93.06 7629.6 9839.5 3130.1

The empirical results of Tiny model

Test Set

HAIGA GA AIS

nG pSz mP nTCpi pSz nG cR mR nIs pSz nISP nC rPS

1 20 500 0.5 25 20 20 0.9 0.5 20 500 20 2 20

2 20 1000 0.5 40 25 20 0.9 0.5 20 1000 20 2 20

3 10 1000 0.5 40 25 10 0.9 0.5 10 1000 20 2 20

Parameter values of the algorithms

The effectiveness of the proposed hybrid algorithm



Introduction

Related work



Experimentation

Conclusion

25

Test

Set


HAIGA GA AIS HAIGA GA AIS

1 90.97 86.81 88.89 7168.9 10009.9 2937.8

2 93.06 88.89 93.06 12802.1 17632.1 4020.5

3 93.06 86.81 93.06 7629.6 9839.5 3130.1

The empirical results of Tiny model

Test Set

HAIGA GA AIS

nG pSz mP nTCpi pSz nG cR mR nIs pSz nISP nC rPS

1 20 500 0.5 25 20 20 0.9 0.5 20 500 20 2 20

2 20 1000 0.5 40 25 20 0.9 0.5 20 1000 20 2 20

3 10 1000 0.5 40 25 10 0.9 0.5 10 1000 20 2 20

Parameter values of the algorithms

0

20000

40000

60000

80000

100000

120000

140000

160000

1 2 3 4 5 6 7 8 9 10 11

The number of interpreted mutants

GA HAIGA AIS

Conclusion

26

Mutation testing

An effective technique for evaluating the quality of test cases

Costly

big number of mutants

Set of mutation operators

Automatic framework for Simulink

Generating mutants, Generating test data, Executing mutants,

Analysing results

Introduction

Related work



Experimentation

Conclusion

Conclusion

27


Using meta-heuristic algorithms

Genetic Algorithm (GA)

Simulated Annealing (SA)

Artificial Immune System (AIS)

the time of test data generation of AIS is the least, and GA is the most

the AIS is stable and it has also significantly improved the number of killed mutants compared to the GA and SA methods

Improving Genetic Algorithm

Hybrid Artificial Immune Genetic Algorithm

The number of killed mutants is greater than that of conventional genetic algorithm and even AIS.

Has less execution time than conventional genetic algorithm, but more than AIS.

Introduction

Related work



Experimentation

Conclusion

Conclusion

28

Ongoing and future work


Improving fitness function to determine test data which kills a

specific mutant

Applying more algorithms

Introduction

Related work



Experimentation

Conclusion

A Novel Test Data Generation Approach Based Upon Mutation Testing by Using Artificial Immune System for Simulink Models

Documents