Test-Case Coverage Analysis of State-Based Automata ... - TUHH · Test-Case Coverage Analysis of State-Based Automata for Fuel-Cell Systems in Commercial Aircraft July 11, 2014 supervised

Bachelor Thesis

Sascha Lehmann

Test-Case Coverage Analysis ofState-Based Automata for Fuel-Cell

Systems in Commercial Aircraft

July 11, 2014

supervised by:Prof. Dr. S. SchuppProf. Dr.-Ing. F. ThieleckeDipl.-Ing. J. Grymlas

Hamburg University of Technology (TUHH)Technische Universität Hamburg-HarburgInstitute for Software Systems21073 Hamburg

Contents |

Contents

1. Introduction 1

2. Background and Motivation 32.1. Basics of the fuel cell system . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2. Functionality of the MFFCS model . . . . . . . . . . . . . . . . . . . . . . 32.3. Research Background of the virtual system . . . . . . . . . . . . . . . . . 6

3. Given Test-Case Coverage Concepts and Analysis Methods 73.1. Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.2. Basic state and transition coverage . . . . . . . . . . . . . . . . . . . . . . 83.3. (N-)switch coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.4. Path coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.5. Branch coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.6. Hybrid automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4. Development of the Hierarchical TCA method 134.1. Formulation of structuring and processing algorithms - Part I (General) . 154.2. Transfer of the research results to the MATLAB system . . . . . . . . . . 17

4.2.1. Horizontal state transition diagrams . . . . . . . . . . . . . . . . . 204.2.2. Vertical state transition diagrams . . . . . . . . . . . . . . . . . . . 214.2.3. Graphical function flow charts . . . . . . . . . . . . . . . . . . . . . 224.2.4. Decision constructs in physical block code . . . . . . . . . . . . . . 23

4.3. Formulation of structuring and processing algorithms - Part II (Matlab) . 254.4. Workflow of the TCA program . . . . . . . . . . . . . . . . . . . . . . . . 28

5. TCA Application and Coverage Data Acquisition 31

6. Evaluation 33

7. Conclusion 417.1. Future prospect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

A. User manual 43

B. Complete secondary path table 45

C. Listings 49

iii

List of Figures |

List of Figures

2.1. The basic structure of the MFFCS model . . . . . . . . . . . . . . . . . . 5

3.1. Transformation of a physical controller into a hybrid automaton . . . . . . 10

4.1. Looped structure of a horizontal state transition diagram . . . . . . . . . . 204.2. Serial structure of a horizontal state transition diagram . . . . . . . . . . . 204.3. Structure of a vertical state transition diagram [nested and parallel states] 224.4. Flow chart function for a basic mass flow evaluation . . . . . . . . . . . . 224.5. Air valve code (excerpt from decision constructs) . . . . . . . . . . . . . . 234.6. The TCA workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

6.1. The development of state coverage during the primary path segments . . . 336.2. Coverage comparison for total, best and worst covering test cases . . . . . 39

List of Tables

5.1. Coverage for primary simulation paths without events . . . . . . . . . . . 315.2. Coverage for secondary simulation paths with failure events (the full table

can be found in the appendix) . . . . . . . . . . . . . . . . . . . . . . . . . 32

6.1. Examples of activated and non-activated relevant Stateflow states . . . . . 346.2. Examples of activated and non-activated relevant Simscape states . . . . . 366.3. Total test suite coverage data . . . . . . . . . . . . . . . . . . . . . . . . . 37

List of Algorithms

1. DetermineExistingNSwitches . . . . . . . . . . . . . . . . . . . . . . . . . . 152. DetermineExistingPaths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163. AssignLogVariablesInCode . . . . . . . . . . . . . . . . . . . . . . . . . . . 254. MergeConditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265. DetermineActiveTime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266. DetermineTraversedPaths . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

v

Abstract |

Abstract – Based on a given model of a multifunctional fuel cell system (MFFCS), weperform coverage analysis steps on certain parts of the model throughout the simulationwith a predefined set of test cases. Afterwards, we evaluate and assess the test case suitebased on the coverage results, and identify the basis for possible improvements suchas covering inactive states, extending the state classification and using less generalizedcontrol units. Furthermore, we analyze the coverage intersection of different test cases,and come to the conclusion that all tests are built up on a similar control sequence andreach an overlapping ratio of up to 97%. As the result, we are able to provide advice ontest suite reduction as well as improvements for the model itself.

vii

1. Introduction |

1. Introduction

Aircraft systems engineering has been a major field of research over the past decades.Back then, the airplane was strictly divided into mostly autonomous parts with definitelydedicated functions. Although a certain degree of communication between these partswas given, they could mostly be reviewed as separate components which then contributeto the whole system. In order to increase the efficiency of future aircraft, it becameincreasingly important to not only provide functional blocks for each task, but also todistribute each task among different blocks. The use of specialized units for every singlefunction needs too many resources for a proper optimization in terms of size, mass andconsumption of resources. Therefore it was necessary to modify these units that servedistinct purposes, and transform them to multifunctional systems. One of those partswhich are subject to modern research are fuel cells, and we will dedicate the thesis to amodel which represents one approach to implement these cells in the architecture of acomplete airplane.The concept of a fuel cell is comparatively adaptable when it comes to its integration

into a system. Therefore the cells are used for a variety of tasks: The main area ofapplication is still its function as the auxiliary power unit (=APU), which is neededto provide electricity during ground operations. Furthermore it is possible to use theexhaust of the fuel cells to reduce the combustibility of the gases inside the main tanks.It is also possible to use them as electrical power supply in case of emergency or evenas another way to provide another water source for flight operations [1]. Ways to utilizea fuel cell in an aircraft architecture as well as the optimization and integration of thealready developed functions are part of current studies [2].

The basis of this thesis is a hierarchical model of the major system parts of an airplane,which addresses specifically the functional integration of fuel cells into a complete aircraftarchitecture. Consistent with the complexity of flight machines nowadays regarding thecomponent extension and interconnection, the model consists of a higher three-digitrange of states and transitions, together with multi-branched function calls to calculatedomain parameters dependent on feedback information. With such an extent of complexstructures, it is not a trivial task anymore to choose a suitable set of test cases to stimulateand test all implemented model functions and classified scenarios. For that reason, theneed for fundamental computer science methods arises, at which two of them - the testcase generation as well as the evaluation - are of particular importance.The first one describes the automated generation of a number of test cases from scratch,

using basic principles like gray- or white-box testing, model-based testing as an exampleof black-box tests, or methods like propagating state invariants and transition conditionsbackwards in order to obtain boundary input values and traverse certain paths and modelsections. The other one consists of a coverage verification of an already given set of testcases inside the model environment, allowing one to make a statement about the scopeand reachability of every case after it has been applied to the model in a simulation run.

1

| 1. Introduction

While the former method results in a complete set of test cases that definitely fulfillsthe coverage requirements but is not adapted to real conditions, the latter one buildsup on realistic and use-oriented test cases created with background knowledge about thesystem application. This second technique is used to indicate the flaws and uncoveredareas of the test case suite, so that the system developer can use that information torecombine and extend the set of tests.

The latter method will be the topic of this thesis, leading through the different phasesranging from model changes for run-time logging, model structure analysis and the actualsimulation, to the evaluation of coverage data and final information on best covering testcases as well as uncovered parts of the model. For that purpose, an insight into theexisting ways of coverage analysis will be given, and we will discuss to which extent theycan be realised and applied to the model. The first part of our implementation provides anautomated way to retrieve and store the activation data of every single state, transition aswell as condition branches on code level. With the raw logging data from the simulationphase, the second part of our program restructures the information into an interpretabletable form, allowing us to systematically review the coverage ratio of the model and itssubsystems. The results that we obtained from this analysis made it possible to identifya set of different failures of the modelled multifunctional fuel cell system (=MFFCS) onboth the test suite side and the model side, ranging from unactivated or unintentionallyentered states to fluctuating effects of numerical calculation. Information on these failureswere finally transmitted to the model creator to precisely locate the failure sources andacquire suitable fixing solutions.

We start with an overview of the project background and information on fuel cell systemsin general as well as the model in chapter 2. It is followed by an in-depth look at thedifferent types of testing and test case coverage in chapter 3, in which we also discuss theapplication of hybrid automata principles. In chapter 4, we lead through the differentdevelopment phases of our test case coverage analysis methods (in the following alsoreferred to as TCA), before we finally apply these methods to the model and evaluatethe results in chapter 5 and 6. We then finish the thesis with our conclusion in the finalchapter 7.

2

2. Background and Motivation |

2. Background and Motivation

We will begin this thesis with a closer look on the model and its underlying fuel cell sys-tem as the basis of our analysis approach. We will gather the ways in which the differentcomponents interact, understand their function for the complete system and see whichareas of application are covered with the model. A fundamental understanding of thesystem itself is necessary to recognize the importance of a systematic look at the compo-nent activity, which we will be getting into with the following chapters. Furthermore wewill get basic insights into the research environment which this model is embedded in.

2.1. Basics of the fuel cell system

Fuel cell systems are energy converters which operate using hydrogen as the main reactantof its combustion processes. So they base on chemical energy like every other fuel systems,but are capable to covert that energy into direct current electricity without additional,external conversion steps [3]. For that reason, there is no need for electric generatorsto transform the kinetic energy as it is done from kerosene combustion. The productsbesides the electrical component are heat as well as water, which can be transferred toanother circular flow to be used for different purposes there.The main advantage of the fuel cell system is the weight of its fuel, as hydrogen is lighter

than the conventionally used kerosene. Besides that,H2 provides a higher efficiency factorof energy generation compared to the currently established energy sources in an aircraft.However, one has to keep in mind that a gaseous instead of a liquid fuel either needs tobe kept under high pressure or in a larger tank than fuels like kerosene in order to containa comparable amount of energy. By using specifically designed tanks and materials, it ispartly possible to store hydrogen in its low-temperature liquid form [4].Although these cells are also considered to be able to completely power an aircraft

at some point, they are planned to be only used as a support energy supply in currentprojects, and the MFFCS model base of our work considers this application as one aspect.It may still take some additional decades of research on this comparatively new field tofinally replace the conventional power supply in an aircraft. Nevertheless, the scalabilityof these cells already allows a utilization in many fields of application, ranging from air-,ground- and underwater-vehicles to power generation for buildings [3, p.373]. Consideringthe products of its combustion process, it may also limit the emission values, which is acritical point of modern air traffic [5].

2.2. Functionality of the MFFCS model

The model that we base our analysis techniques on describes all major subsystems ofthe airplane architecture. This involves the division into the cargo and cabin area on theone hand, and the underlying technical components on the other hand. There we find

3

| 2. Background and Motivation

the hydrogen tank connected to the two fuel cell systems, which form the base of themultifunctional integration. The tanks for the fuel, which powers the conventional engineparts of the aircraft, are segmented into three parts in the left and right wing as well asthe center area. On system level, the modelled aircraft is built up from five components.The environmental control system and the cooling system mostly influence the pressureand temperature values of the involved mass flows. The water system provides the basicfluid elements and might for example be additionally filled with water from the fuel cellsystem outputs. Finally, the cargo fire suppression system as well as the fuel tank inertingsystem serve the purpose of using the produced gases from the fuel cells to reduce thecombustion probability inside the tanks and to extinguish a possible fire in the cargoarea.

The top-level condition and event controller simulates entire flight missions under normaland emergency conditions. It triggers failure events such as a cargo fire or engine failureat algorithmically determined points of time within a set flight or ground operationphase. For that purpose, it first simulates flight missions during a normal cycle andthen switches to alternative paths which are based on these ideal runs, but include singleor combined deviations from the norm. The evaluation blocks of the model analyzethe value development in both normal and critical situations and gives feedback on theoverall functionality of each system.The considered flight missions are divided into a set of sectors, each consisting of

multiple phases for the different operations during a mission. The initial phase of eachcase is the off state, followed by a start-up part in which all systems are activated, anda gate part, in which the aircraft is moved to the gate to allow passenger boarding andcargo loading. The following ground operation consists of typically three different parts:In the push-back phase, the aircraft is moved backwards from the gate. Afterwards, theaircraft moves to the flight start point of the runway in the taxi-out, and uses the MES(= main engine start) phase to power up the engines for the following flight mission. Apotential failure event that could occur during ground operation is a cargo fire, whichcauses the mission to be immediately aborted.

In case of a flawless positioning on the runway, the flight operation phases can be initiated.The normal process involves five steps, which are the takeoff, the climb flight up to thedestined height, followed by the cruise to the destination airport, and finally the descentphase to lower the flight height for the fifth phase, which is the landing procedure. Themajority of potential failure events take place in one of these phases.Besides the cargo fire already mentioned during ground operation, a total engine flame

out (=TEFO) can occur. This describes a complete engine breakdown, which is normallybypassed with a ram air turbine in conventional aircraft systems. In our case, the fuel cellsystem works as a replacement and provides emergency power in this situation. Reactingto a cargo fire, the fuel cells supply inert gas to the cargo bay to extinguish that fire. If aTEFO takes place, the aircraft will not be able to complete the mission as intended andis forced to perform a gliding flight (also called the volplane) while aiming for the nextairport before switching over to an adapted descent and landing phase.

4

2. Background and Motivation |

After the flight operation part has been completed, the aircraft will go through anotherset of ground actions before the mission is finished. During a taxi-in phase, the airplanewill move to its final position on the taxiway. If an emergency situation occurred duringflight, the evacuation process takes place now. Otherwise the aircraft will move backto the gate and eventually shutdown the system, or return to the push-back phase foranother flight mission. Those cycles are described with each of the test cases that we aregoing to analyze in the following chapters.

The most important aspect for our analysis is the systematic composition of the MF-FCS system, as it defines the way we can access the activation data for the coverageevaluation. The model is implemented as a hierarchical Simulink system. This systemcontains Stateflow chart objects representing the state transitions diagrams and physicalSimscape domain blocks reacting to input parameters through if-else-constructs. Most ofthe functions used in that model are implemented in MATLAB code, specifically Level-2-MATLAB-S-Functions. These are function blocks which can be integrated into theSimulink signal flow via their input and output interfaces, and perform calculations ondefined or frequent time steps using the MATLAB code API. Figure 2.1 shows the generalstructure of the model, with placeholders representing the different domains used.

Stateflow controller chart

Simscape physical block set

Simscape physical block 1

Simscape physical block 2

. . .

Simulink block set

Simulink block 1

Simulink block 2

. . .

Top

-levele

vent

/cond

itioncontrolle

r Simulink subsystem 1

Simulink subsystem 2 . . .

MFFCS model

Figure 2.1.: The basic structure of the MFFCS model

5

| 2. Background and Motivation

2.3. Research Background of the virtual system

The development and simulation of the MFFCS system is embedded into a project incooperation with industrial partners. The so called ‘VIPER’ platform (= Virtual Inte-gration Platform for Education and Research) aims to provide an environment in whichnew software and hardware components can be tested regarding its integration into thearchitecture of an aircraft. The core of the modelling work is given with the ViRACmodel (= Virtual Research Aircraft), which is designed to evaluate flight dynamics andaircraft systems during predefined flight missions. One of the applications of the ViRACsystem is the MFFCS model, used to create a well-founded base for the field of aircraftfuel cell integration. This field of research is comparatively young, which makes the modelwe are analysing a foundation for future integration approaches and system extensions.With the MFFCS, the FST institute provides a model that serves as an executable

specification in this project. From a practical point of view, it is also the basis for theintegration test of real-world physical components and control software. Because of the‘building block system’ behind the model as well as the strict definition of the input andoutput signals of each subsystem, it is possible to replace any component with real partsand evaluate the way it works together with the rest of the system.The model is additionally used to determine time parameters and ranges of environ-

mental values for the prospective real implementation. An example of this are the timeperiods a state may remain active before the whole process is considered to be stuck oroverextended. Finally, the hierarchical system design will help to expand the function-ality of the multifunctional fuel cells to additional application areas and adapt it to newsystem requirements from both the research and industrial point of view.

6

3. Given Test-Case Coverage Concepts and Analysis Methods |

3. Given Test-Case Coverage Conceptsand Analysis Methods

A great variety of different coverage analysis approaches has been developed in youngerhistory since it became increasingly important to systematically test a software system.Although some of these tests were only meant to be applied to software code, they havebeen expanded over time, and are not restricted to code anymore, but can be appliedto a whole set of abstract systems. These systems range from UML class diagramsand flow charts up to real physical systems, which have to be remodelled before theanalysis application. Both the code aspect as well as the model-based approaches willbe important for a solution to our coverage task, and will therefore be the topic of thischapter. We will start with the basics of state transition constructs and lead to differentlevels of coverage. As a great part of the model consists of physical systems, we will alsodiscuss the remodelling aspect and how it can be applied to the MFFCS model.

3.1. Fundamentals

In order to understand the relevant coverage techniques for this project, we need to be-come aware of the general structure of state- and transition-based constructs at first.The most fundamental elements in the model are the individual states S and transitionsT . A state S is a defined location inside a state system and describes one possible situ-ation of the machine (for example, ‘Startup’, ‘Active’, and ‘Shutdown’). Each transitionT = {S1S2} is defined by a source and destination state, and can either be unconditionedand therefore be triggered at the directly following time step, or bound to conditionswhich need to be fulfilled for triggering.

With a composition of these two basic elements, we get to the level of a whole finite statemachine (=FSM). It is defined by a finite amount of states, and every following transitiononly depends on the active state and the input parameters for the machine. For thatreason, a finite state machine can produce different outputs for the same input, as thenext triggered transition and therefore the output is set by the current state and thetransition conditions. Furthermore, there must be one initial state that is activated by adefault transition on system start. There are attributes that additionally hold for someFSMs like those we can find in the MFFCS model: they are completely specified andminimal, meaning that there are neither outgoing transitions with overlapping conditionsfrom a single state, nor duplicated states. Moreover, all states should be connected tothe machine by at least one transition [6, p.179].

To assess the effectiveness of a test system, we need to understand the principles ofsoftware testing. The normal procedure leads through the following steps: at first a setof different input parameter combinations is chosen. For each of them, the expected

7

| 3. Given Test-Case Coverage Concepts and Analysis Methods

output values are then manually calculated or looked up in the system specification, andprepared for comparison. Finally, the program is executed with every set, and the outputvalues of the system are compared to the formerly determined reference values.The evaluation of that process can either be performed by direct debugging during

execution, or partly trace logging for a further failure analysis after execution (an exampleof such a trace-based system can be found in the automatic debugging work in [7]).The flow of a program can be represented by a flow chart for that purpose, containingfunctional as well as predicative elements [8], which can then be analyzed with graphtheory principles. In other cases - and as we will find it in the fuel cell model as well -the program section may already be modelled on a graphical level instead of bare code.In that case we can skip the remodelling part and directly start the evaluation process.

From a model access perspective, we consider the difference between white- and black-boxtesting, depending on whether it is necessary or possible to get to the underlying designor not. In our case, the model is provided as a whole and therefore we can assume insightinto the system and apply white-box related methods. This means that we do not onlyneed to evaluate the state and transition activations from outside the system through thereaction to certain input signals, which would imply that we rebuild a ‘perceived’ copyof the system structures solely with these pairs of input and output data. In fact, we areable to directly look inside the model parts while the simulation is being executed, andto evaluate the activity from within the system. This complete white-box foundationallows us to tap any of the signals and data flows to trace them for future processing.

3.2. Basic state and transition coverage

State and transition coverage are the most common and basic methods of coverage anal-ysis. They focus on the individual components of a finite state machine, and check theiractivation independently of the surroundings. This provides a solid base for startinga coverage analysis, but one has to keep in mind that it will only help to identify themost obvious problems such as uncovered states or transitions without providing anycontextual information. Nevertheless, it will be an important tool for our analysis of theMFFCS model, as we will see in the following chapters.

3.3. (N-)switch coverage

For a satisfactory coverage analysis, it is not sufficient to just focus on the single statesand transitions. To achieve a complete system coverage with a test suite that complieswith the requirement that all possible input parameter combinations produce definedand verified outputs, it is also necessary to analyze the potential switches. Only withthat knowledge we can be sufficiently certain that the system will work as intended,independently of the way the data traverses through the net. In that case, a switch isdefined as a state or junction at which the next triggered transition is selected depending

8


on one or more conditions. If one of those outgoing transitions is covered without anerror, we can assume that the corresponding sub-path entry point is valid.But the switches do not have to be restricted to only one transition. T. S. Chow

formulated a more generalized form of these components, which he called ‘n-switches’.They base on the single switch concept, but extend this concept to paths of switches witha defined length of n. He derived that 0 -switches can be solely used to detect operationerrors, while 1 -switches add the detection of a certain set of transfer errors. The fullcovering n-1 -switch sets finally allow the complete detection of all operation and transfererrors as well as unconnected states, in which the system is supposed to be at maximumn-1 -distinguishable. For minimal finite state machines, this implies that n equals thenumber of states [6].

A problem that derives from all switch coverage attempts is the existence of infeasiblepaths. Those paths are defined as not being accessible with any possible set of inputparameter values. This can be the result of unsatisfiable transition conditions leading intothese paths, or may happen if the path represents a sort of error handling routine whichis only triggered if an internal component or function shows a situational malfunction.Because of the complexity and the necessary effort to predict, classify and identify thosepaths as well as their influence on the system reliability, infeasible paths are still animportant subject to current research [9].

3.4. Path coverage

The analysis regarding path coverage provides an even more extended assurance forvalid system behavior. A path in a diagram is defined as a continuous sequence ofstates starting at a declared entry point of the system and ending at either a definedexit transition or a junction point without further outgoing connections. For the lack ofa defined exit point of a state transition diagram, it is not suitable to apply the pathcoverage principles there. This will become important as soon as we consider systemsthat can be described as a flow diagram, such as code-based programs.The paths inside a diagram can be represented in two different ways. We can deduce

them from a Boolean matrix that contains information on whether a connection betweena row’s and a column’s state exists [10]. A problem with that representation occurs ifwe have multiple transitions between two junctions, as this matrix can only indicate onetransition between each pair of states. The other possibility is to directly enumeratethem in a sort of list, which would increase the memory usage because of multiple stateredundancy.Regarding path coverage, we have to deal with the same problem of infeasible paths

as during n-switch analysis. It will be hardly possible to distinguish between a path thatis not covered by a given test suite, and a path which cannot be accessed at all. For thescope of the MFFCS model, we will only be able to determine the test suite coverage ofthose paths, but will not prove the general reachability in a model composition context.

9

| 3. Given Test-Case Coverage Concepts and Analysis Methods

3.5. Branch coverage

Generally spoken, branch coverage is completely equal to the principles of the 0 -switchcoverage. However, in our case we will use that term when we talk about the method thatis not directly applied to finite state machines in the form of state transition diagrams.Branches in the sense of this method will be the ones that we find in conditional if-else-based code structures. So when we talk about branch coverage in the context ofthis thesis, we will mean the coverage technique applied to the underlying code of thephysical blocks, especially the ones located in the Simscape environment.

3.6. Hybrid automata

A hybrid automaton describes a system which consists of discrete control states connectedby conditional transitions, and a set of continuous flow variables, which follow physicalspecifications and can take an infinite amount of different values. In its most basicform described by Alur, Courcoubetis et al in 1995, it can be specified as a 6-tupleH = (Loc, V ar, Lab,Edg,Act, Inv) [11, p.2]. The Loc parameter contains a set of controlstates, Var consists of all continuous flow variables, Lab serves the synchronization ofmultiple automata, and Edg describes all transitions between the states. Act and Invare activity and invariant assignments to each of the states. This concept has beenexpanded to a 9-tuple in the following years with additional information on events andjump conditions. For our purposes, this system model would be useful, as it allows

Offx = −0.1xx ≥ 18

Onx = 5−0.1x

x ≤ 22

+

−

Controller

Plant

Sensor x < 19 x > 21

x = 20

Referencesignal

Feedbackerror

Controlsignal

Outputsignal

Measuredoutput

Figure 3.1.: Transformation of a physical controller into a hybrid automaton

10


additional coverage techniques like a reachability analysis of this infinite space of real-value states.

A general scheme for such a controlled system is displayed in Figure 3.1. On the lefthand side, we see the most basic concept of a closed-loop control system, consisting ofa controller that feeds the control path with a control signal, and that output is finallymeasured and fed back into the reference signal.A system with that control purpose can be transformed into a hybrid automaton. It

combines the functions of the controller, the plant, and the sensors into a state transitionnet with defined continuous parameter development over time dependent on the currentlyactive state. The example of a hybrid automaton on the right hand side of Figure 3.1represents a thermostat as one application of a controlled system, and is taken fromHenzinger’s fundamental research work ‘The Theory of Hybrid Automata’ [12]. It consistsof two states ‘On’ and ‘Off’, which are activated as soon as the temperature showsa certain deviation from the desired value of 20 degrees Celsius. The MFFCS modelcontains a great set of control systems, which could potentially be modelled as suchhybrid automata, ranging from simple valves and sensors to more complex compressorsand purge modules. However, there are two limitations that keep us from using theseprinciples for our approach.

The first one is the system complexity. In order to transform a component into a hybridautomaton with reasonable effort, we would want to describe the parameter developingin each state as a linear term. But as the output signal of one element traverses througha whole net of other components before it returns to the initial controller, it is notpossible for us to reduce that environment to linear equations. A further expansionof the hybrid automaton to non-linear systems can be found in a consecutive work byHenzinger, dealing with their algorithmic analysis [13]. But even if we were able to doso, we would still need to find an automated way to transform each component into itshybrid automaton form. For that reason, it would be more realistic to perform a valuecoverage analysis if the provided model was already designed from ground up followingthe principles of hybrid automata. As this is not the case here, we will concentrate onthe other introduced coverage techniques to assess the system.

11

4. Development of the Hierarchical TCA method |

4. Development of the Hierarchical TCAmethod

In this section, we will take the insights and methods of the previous chapter, evaluatethem in terms of possible applications to the airplane model, and finally transfer theacquired pseudo code algorithms to a high-level language code compatible with the giveninterfaces of the model.

One important factor is the question of the general extent of the model that these conceptscan be applied to. Summing up the results of the last chapter, we need to transfer asubsystem to either a finite state machine or a flow chart in order to apply an object-based coverage analysis to it. To accomplish this, we need to discuss whether the modelparts are transformable to that effect or if the effort would extend the benefits of such atransformation.The control components satisfy the requirements of a finite state machine, as they

contain a finite set of states and transitions, where the triggering of the latter ones onlydepends on the input to the system and the active state itself. The usability of themethods for the controller parts of the model should be clear at this point, consideringthe fact that they are already represented through state transition diagrams. Being thesource of every process within the system, the main focus should lie on these automata,as they not only allow evaluating the pure coverage, but also the rough development ofall variables involved.

A validation of variable value ranges within the physical model components will not besubject to our research and implementation in the scope of this thesis. This decisionis based on the former discussion on hybrid automata, which revealed that feedbackcontrolled sub-parts of a model could be transformed into an automaton, making ittheoretically possible to apply coverage analysis methods to it sufficiently. Nevertheless– due to the strong interconnection of all considered physical function blocks – it is hardlypossible to transform the affecting surroundings of one of those functional units into a(linear) equation in order to calculate the value pattern of the controlled parameters.Another factor against the inclusion of the physical component of the system is the

effort that we would need to put into the whole model in order to obtain usable results.All physical components are implemented as code blocks with if-else-constructs decidingabout the development of the parameters. If we would want to analyze those partsthrough the hybrid automata approach, we would have to split every block into a netof discrete control state and continuous flow transitions (see Figure 3.1 in chapter 3 forinformation on the procedure). As the valve is only the most trivial component among thephysical blocks, it is obvious that adapting a model of this complexity would exceed allboundaries of time, even if it was possible to transfer the surrounding to one single linearequation (whose possibility we eliminated in the previous paragraph). The systematicsof hybrid automata are rather applicable if the underlying model was built up on these

13

| 4. Development of the Hierarchical TCA method

principles from ground up as well, which is not the case with the given model.

Hence, the coverage analysis can only be applied to the controllers behind the systemsand those code parts behind an object which are either self-contained (this means theyregulate themselves through feedback and therefore can be viewed as an independentcomponent) or contain exclusive decisions which can be interpreted as states. Takingthese arguments into consideration, our test case analysis method provides coverageinformation on the following systems:

1. Controlling structures that regulate the behavior of all physical parts in scope

2. The top-level controller that stimulates the model with global event / value impulses

3. Components that consist of decision constructs within these physically modelledparts

The first two of these systems, basically including all directly accessible finite statemachines in the model, will be analyzed regarding a set of different coverage methods.We will first look at the single states and transitions, then consider their combination tocertain n-switches, and finally examine the automaton as a whole in terms of coverage.

We will divide our algorithm formulation into two separate parts. This is necessarybecause we will have to deal with two different types of algorithms. The first one ismore general and useful for all state transition diagrams or flow charts independentlyof the software environment and model they are applied to. They are the topic of thenext section. But when we eventually transfer our research results to the MATLABsystem, we have to take decisions on some fundamental implementation designs, whichthen depend on the constraints and limitations of that system. There are, for instance,various different ways to divide decision code constructs into log sections, and we wouldhave to develop algorithms for all of them without insight in the system that we applythese algorithms to. For that reason, we will deal with that set of algorithms after theresult transfer.

14


4.1. Formulation of structuring and processing algorithms -Part I (General)

The need for processing algorithms regarding general finite state machines arises at thepoint during which we do not consider single states and transitions in an isolated formanymore, but their compositions. That way, they will form structures of paths (in case offlow charts) or n-switches (for state transition diagrams), and it is our task to determineall possible ones in order to compare the list of actually traversed paths and switcheswith them. The focus will be on state transition diagrams at first.

For the sets of n-switches, we want to be flexible enough to freely choose the path depth,and include a path to that set if and only if its length exactly matches the selected pathdepth. To avoid overlapping of the path sets for different values of n, we decide to directlydiscard the paths that end before the defined length is reached. The value n is thereforenot the maximum path length, but the exact one. Pseudo code of the algorithm thatfits our requirements is shown in Algorithm 1. All algorithmic representations of ourfunctions contain a ‘data’ section, which indicates the data that is directly accessible bythe algorithm without the need to provide it as an input argument. In order to go througha net of transitions with initially unknown length, we will refrain from implementing aniterative approach, but use a recursive function. Each branch of it terminates as soon aswe either reach the destined path length, or come to a dead end state with no furtheroutgoing transitions. The final decision we need to make defines whether we will focus onthe states to determine our paths, or if we focus on the connecting transitions betweenthem. We use the latter approach especially for one reason: as soon as we get the loggedactivity data from the model with partly parallel executed states, we need to consider

Algorithm 1: DetermineExistingNSwitchesInput: N-switch path (P), N-switch depth (n)Data: A finite set of transitions (Tset) and states (Sset)Result: Set of existing N-switches (Pset)

1 if P is empty then2 foreach {transition T = {S1 S2}|T ∈ Tset, {S1, S2} ∈ Sset} do3 Pnew = ”;4 Pnew.add(T);5 DetermineExistingNSwitches(Pnew, n);

6 else7 Sdest = destination state of last P element;8 foreach {transition T = {Sdest x}|T ∈ Tset, x ∈ Sset} do9 Pnew = P;

10 Pnew.add(T);11 if size(Pnew) == n then12 Pset.add(Pnew);13 else14 DetermineExistingNSwitches(Pnew, n);

15


the logged transitions anyway to identify which next state belongs to which path whenmore than one change occurs simultaneously. Therefore we will define the paths by theirtransitions and not by the involved states to get distinct results.

For the complete paths in the flow charts, we can take our first algorithm as a basis, buthave to adapt some lines to break the restriction caused by defined lengths and deal withthe different definition of exit points. Every flow chart consists of several junction points,which are connected with transitions, and with each execution of that chart, a completeset of transitions forming a path is executed, depending on the defined conditions of eachtransition. Each of the paths starts at a default transition which defines the input of thechart. So for the first execution of our path determination algorithm, we have to definethose default transitions as the first path element and afterwards start the recursivepassing through the junction net. An end point of the net is defined by one or multiplejunctions without any outgoing transition. The chart execution automatically stops atthat point. To transfer the same principle to our algorithm, we need to determine thesize of the set of outgoing transition from the currently processed junction, and in casethat the size is equal to zero, add the current set of transitions as a new path to the finalpath set. Otherwise we start a new recursion step with each outgoing transition as thenew last element of the temporary path variable. All of these aspects are combined tothe code in Algorithm 2.

With these techniques, we are now able to determine single states and transitions as wellas all existing paths and n-switches that share the same hierarchical level. At this point,we have a sufficient base to analyze the structure of all graphical elements in the MFFCSmodel. All remaining decisions on state recognition on a source code level and the way

Algorithm 2: DetermineExistingPathsInput: path (P)Data: A finite set of transitions (Tset) and junctions (Jset)Result: Set of existing paths (Pset)

1 if P is empty then2 foreach {transition T = {ChartInput S2}|T ∈ Tset, S2 ∈ Sset} do3 Pnew = ”;4 Pnew.add(T);5 DetermineExistingPaths(Pnew, n);

6 else7 Jdest = destination junction of last P element;8 if |Tnext| > 0, with Tnext = {T = {Jdest x}|T ∈ Tset, x ∈ Jset} then9 foreach {transition T ∈ Tnext} do

10 Pnew = P;11 Pnew.add(T);12 DetermineExistingPaths(Pnew, n);

13 else14 Pset.add(Pnew);

16


we process our logging data will be made after we acquaint ourselves with the limitationsof the MATLAB, Simulink, Stateflow and Simscape systems.

4.2. Transfer of the research results to the MATLAB system

The main point of the MATLAB program approach is that most modelling work is basedon matrices (as the name ‘MATrix LABoratory’ already implies). Therefore there aretwo possible ways to deal with the analysis of the MFFCS.On the one hand, we could try to dispense with the use of MATLAB structures and

data types as much as possible while still assuring that the interfaces between the TCAcode and the original model can be sufficiently provided. This method would lead toan increase of separate S function blocks (as each of these blocks can only handle eitherMATLAB, C, C++, or Fortran code without combining them), and will also increase theamount of data type conversions, as only the basic types (Integer, Double and Boolean)can be used in a port communication between C-S-function and MATLAB-S-functionblocks. Another deficiency is the functional restriction that comes with using only thesebasic types without further structuring, and with the necessity of using the port con-nections between the single blocks. The dimension of each port needs to be definedwithin the code and cannot be changed at simulation run time. The advantages of thismethod would be a greater degree of code freedom within the C-S-function block itself.The provided data can be structured with any data structure that is possible in C-code.So besides being able to handle diagram data matrices, we could also use ‘structs’ tocreate objects for states, transitions, junction etc. and connecting them with pointers.A creation of net and tree structures would be possible that way, although we will seethat relying on matrices and tables will be a better solution for our approach.

The other method would be to choose a data structure which is compatible with MAT-LAB code and fully relies on MATLAB-S-function blocks without using C-code. Apossible approach can be the use of ‘tables’ to structure the data, either creating tablesfor each chart or one complete table of all states, one for transitions and so on. They canbe globally accessible or passed as arguments to the individual functions. Using one tablefor a chart with multiple active states accessing it at the same time may of course causeproblems in a multiprocessing environment. This situation cannot occur in MATLABcode during our analysis though, as it is only a single process as long as the function-ality is not extended with the ‘Parallel Computing Toolbox’. Besides that, intersectionof data access can not occur with state or transition tables because the relation of aMATLAB-S-functional part of a chart and its corresponding entries in the chart are 1:1and therefore distinct. It is not possible that one state has two or more correspondingdata rows in one of the tables.A clear disadvantage of this method arises from the data structure restrictions within

MATLAB code. As trees, linked lists as well as other pointer-based structures are notpossible without a major workaround (the only way to achieve a pointer-like structure bynow is the definition of a class that inherits from the base ‘handle’ class). Therefore the

17


system will be restricted to matrix structures such as tables, structures, and cells. Butif the task can be accomplished using one of those structures, no data type conversionsare necessary regarding the structure variables, as even the MATLAB GUI elements arecapable of receiving their data from them.We decided to use the table and matrix structure for our approach. It is considered

to be the more practical solution here, as we can analytically and graphically review thedata without the need of converting the data structures to a suitable form beforehand.Furthermore, we are then allowed to process the data with the already optimized matrixoperations from the MATLAB system instead of relying on a tree or object structure,which is still not fully and directly supported by that software.

As our next step, we have to choose a suitable logging technique to access the state andtransition activation data. The Stateflow environment itself provides a feature to signalthe currently active state at an additional output. Although this ability is sufficientenough for some kinds of finite state machines, we cannot use it for our purposes in theMFFCS model. The reason for this is the fact that the function can only handle oneactive state at a time. Such a state arrangement with only one active state at any givenpoint of time is called ‘exclusive OR state decomposition’.In our case, some general higher-level states split up into multiple lower-level states,

which are activated simultaneously. This kind of arrangement in contrast is called ‘par-allel AND state decomposition’. In a case where a state chart contains at least one ofthese latter decompositions, the provided feature will not work anymore, as a simulationstart up will directly be aborted with a corresponding error message. This means that weneed to implement our own functionality to log the activity of all states and transitions.

As we already explained the occurrence of parallel state activations, it is not possible todefine one single log variable, which each active state assigns its ID to. So for this purpose,we can choose between two different solutions, and we will see that it is advantageousto use both of these methods for our different areas of application. A similarity of bothsolutions is that we can either lead all obtained state data to new outputs and processor log them with additional methods outside of the chart on the Simulink level, or wecan directly call functions within each state and transition committing the necessary IDsto identify the component. The called function will then find the corresponding dataset entry to the state and chart ID and update the values of it. Besides that, the twotechniques follow different approaches, and we will evaluate both of them regarding theirusability for our model.The first possible method implies that we only log states on the lowest vertical level

of each chart. The data set would be sufficient enough to derive information for allactivated states, as each sub-state has exactly one explicit parent state, so we can con-clude its activity from the activation of the child state as well. Furthermore, we can alsoget information on the activated transitions, but the parallel execution sets one majorlimitation for that. There could be a situation in which four states are completely inter-connected and two states are active, while the other two will be active on the followingtime step. In that case, we would not be able to determine which pairs of states were

18


the consecutive ones. Our closer look at the different categories of diagrams in the nextsection will still reveal that we have the set of physical blocks where we need to applyexactly this method to be able to log the states.

19


The other method is more extensive regarding memory usage, but allows us to reconstructall sorts of state and transition triggering in a simulation. For this solution, we willdefine a separate logging variable for each single state and each transition. During thesimulation execution, the components will then set or reset their exclusive variable onthe entry and exit points of a state or during the triggering of a transition. As mentionedbefore, we can then either directly process the data within the state, or perform thatdata processing step outside of the chart. For the MFFCS model, we are interested inall events in each chart, so this log method will be the one to choose for the upcomingcoverage analysis.

The final steps to consider for the technique transfer to MATLAB are the actual appli-cation areas inside the model, where we can distinguish between four different diagramstructures.

4.2.1. Horizontal state transition diagrams

The first group of relevant state structures consists of those which are arranged horizon-tally. They are the most basic way of a state transition diagram, and the state activationas well as transition triggering is completely sequential (this kind of execution is calledOR-exclusive in the Stateflow environment). This means that exactly one state is ac-tive at every point of time, and there is no possibility that two transition are traversedsimultaneously, which would lead to two different states in the following time step.

We can find two different variations of these horizontal diagrams in the MFFCS model.A simplified example diagram of the first form is shown in Figure 4.1. It represents

Standby

. . .

Startup

. . .

Shutdown

. . .

Operation

. . .

[Air_Supply_mode == 2]

[Air_Supply_mode == 1]

[shutdown_finished]

[startup_finished] [Air_Supply_mode == 1]

Bootup_Compressor

. . .

Bootup_CompressorTimeout

. . .

Vent_Air_Channels

. . .

Shutdown_Compressorand_Valves

. . .

Shutdown_Compressoror_Valves_Timeout

. . .

Shutdown_Procedurefinished

. . .

[m_dot >= 0.02] [m_dot < 0.02]

after(10, sec)

[Air_V alve_1_cur <= 2]

after(10, sec)

[Air_Supply_reset ∼= 0]

after(10, sec)

[Air_Supply_reset ∼= 0]

Figure 4.1.: Looped structure of a horizon-tal state transition diagram

Figure 4.2.: Serial structure of a horizontalstate transition diagram

20


the top-level activity diagram for one of the air supply systems, but can be found in apredominant fraction of all control charts. A diagram with this structure is activated atsimulation start through a default transition and stays active for the whole simulationtime without an exit transition. In practice of our coverage analysis, it means that anormal path coverage technique is not applicable for a looped diagram in this form, sowe focus on the variations of n-switch coverage as well as basic state and transitioncoverage for this part. With that method and a successively increased value for n, wecan freely decide about how many loop iterations we want to include at the most bydefining the maximum path length.The other sort of horizontal diagram is defined by a serial interconnection of the states.

An example for this can be seen in Figure 4.2, showing the shutdown sequence of an airsupply controller. This is important for our examination for two reasons: the serialstructure and the timeout states. The serial composition would allow us to directlyapply path coverage methods to the subsystem, but as all of these systems are single-stranded, it will not provide additional information on top of the data from normalstate and transition coverage analysis in this case. Therefore, we will stick to the sameanalysis principles for both of these diagrams. Only the timeout states need a specialtreatment. They should not be counted to the set of normal states but to a differentgroup, as it should be the norm that these states are not reached by any possible inputdata from realistic flight missions. In fact, they should only be activated in case thatone of the underlying physical components shows erroneous behavior, which cannot beforced through general environment parameters.

4.2.2. Vertical state transition diagrams

The second group among the state transition diagrams is defined through a vertical stateconnection. These so called AND-parallel decompositions can be found in every controlstructure of the fuel cell model, and among other things guarantee that both controlfunctions and control evaluation can work simultaneously. An example of this structureis shown in Figure 4.3, which shows a simplified part of the general top-level controlstructures of the model. The active state is divided into two active sub-states thatrun in parallel and are therefore displayed in a dashed representation. In addition, thisdiagram demonstrates that this parallel execution is possible in both a horizontal and avertical manner. The states ‘Active_1’ and ‘Active_2’ are executed simultaneously onthe same level, while ‘Active_2’ together with either ‘Normal’ or ‘Fault’ are nested andalso executed at the same time.Regarding the applicable coverage techniques, we need to keep in mind that the number

of executions and execution times are directly dependent on each other. For our basicapproach, this condition does not have an influence as long as we only consider theminimum activity of each state. However, as soon as we also define boundaries for themaximum number of executions to prevent unnecessary execution duplications inside atest case suite, we need to treat higher-level states with nested sub-states in a differentway. A constraint that dictates a maximum one time state activation, for example, wouldlead to ‘Active_2’ being activated just once and therefore only one of its sub-states would

21


ActiveActive_1

Monitoring

Active_2

Normal

Fault

[Inert_Fault == 1]

[Inert_Reset ∼= 0]

Figure 4.3.: Structure of a vertical state transition diagram [nested and parallel states]

be activated in the simulation process. This problem also arises in horizontal diagramsin the case that some serial path needs to be executed before we reach a split intomultiple states. For our examination, we will use the same techniques as described forthe horizontal diagrams to evaluate the model coverage. The parallel execution is onlya limiting factor in our choice of how to log the state activity, which we discussed in thebeginning of this section.

4.2.3. Graphical function flow charts

Beside the state transition diagrams, some part of the functionality is modelled as flowcharts. One example of this is shown in Figure 4.4. It is the graphical representationof an evaluation function, which is called inside the operational power state of the airsupply system and guarantees that the mass flow always lies above a given minimumvalue. This kind of diagram always starts with a default transition into the system, andends at the first junction without output transitions. Branched paths can be achievedthrough conditional transitions at a junction, as we can see in the diagram.We need to consider a different set of coverage analysis techniques for this sort of dia-

grams. These charts are built up on the same principles as software code itself (containingif-else-branches, while- and for-loops etc.), so we can use the established methods fromthat field of application. As we cycle through the whole flow chart with each function

function Mass_Flow_SP_Evaluation

{m_dot_fcs_sp_temp = lambda_O2/xi_O2 ∗ M_O2 ∗ (2n) ∗ (l_fc1 + l_fc2)/4/F ;}

[m_dot_fcs_sp_temp > 0.008]

{m_dot_fcs_sp = m_dot_fcs_sp_temp;}

{m_dot_fcs_sp = 0.008;}

Figure 4.4.: Flow chart function for a basic mass flow evaluation

22


execution, and with the only difference being the path that we traverse, we can rely onthe path coverage technique here. Another aspect to evaluate is the actual execution ofeach of these functions, which is known as function coverage in the scope of code coverageanalysis. We can assume the activation of a function, if at least one of its transitions hasbeen activated once, so we refer to that coverage value in the evaluation chapter as well.

4.2.4. Decision constructs in physical block code

The last major part of the model that we will include in our analysis affects the underlyingcode of the physical system components in the model. To assert the state coverage ofa system, it is not only necessary to look at the super control structures, but also toevaluate the decision parts inside the function blocks that directly influence the changeof physical parameters. One of these components is the set of valves, which control themass flow in every subsystem. We can see an excerpt from the code behind this blockin Figure 4.5, which is reduced to the decision parts for the parameters. They can becreated in two different ways, in which one of them applies the value at the decidedbranch directly to a variable as shown in line 2, and the other one is a more commonform with separate condition evaluation and parameter assignments as apparent fromline 4 to 13.

Our task at this point is to reproduce all of the given if-else-structures with correspondinglog variable assignments, which we need to direct to an additional output afterwards.This implies that we process the conditions of a branch and generate a single conditionthat will describe the branch as a state. Every branch in such a construct is exclusivelyexecuted, meaning that we will not observe a situation in which two branch conditionson the same level are true at the same time. This exclusiveness is not only limited toour model, but a general constraint of the principles behind the physical blocks, whichrequire an explicit and defined value for each variable at every given time step. We

1 . . .2 dir = if A.p >= B.p, 1 else − 1 end;3 . . .4 if p_u <= p_d then5 m_dot_g == { 0 , ’kg/s’ };6 else if (p_d/p_u) > A.B_lam then7 m_dot_g == dir * Area_C_d * T_u^-0.5 * c_4 * (p_u - p_d);8 else if (p_d/p_u) > B_cr then9 m_dot_g == dir * Area_C_d * T_u^-0.5 * p_u *

10 sqrt( c_1 * ( (p_d / p_u)^c_5 - (p_d / p_u)^c_2 ) );11 else12 m_dot_g == dir * Area_C_d * T_u^-0.5 * p_u * c_3;13 end14 . . .

Figure 4.5.: Air valve code (excerpt from decision constructs)

23


need to understand that all variable assignment in the code are not done sequentially,but simultaneously, so that multiple assignments are invalid. In the same manner, it isnecessary to assign a value to the variable in each state, as they are not memorizing theirprevious value, so that each of the constructs needs to be closed with an ‘else’-statement.This behavior limits our approach regarding a unified acquisition of log data, as wecannot log each of the branches individually without changing the construct structures.Therefore, we will cover each of the if-else-constructs with a single logging variable anduse the exclusive execution to assign the state value of the currently active branch toit. For that, we have to keep in mind that an assignment in each if-, elseif- or else-condition will also violate the precondition of explicit variable values. It takes place fornested conditions, in which the variable assignments of the parent and child conditionwill coincide. Considering this exception, the physical blocks are the ones where we aregoing to apply the log method which only includes the lowest-level conditions and derivethe super conditions from them.

24


4.3. Formulation of structuring and processing algorithms -Part II (Matlab)

Now that we discussed the restrictions and requirements of the systems we are workingwith, we can formulate suitable algorithms for the remaining parts of the model. Thelast of the four structural components we reviewed, dealt with the conditions within thephysical block code parts.We decided that due to the limitation to activate branches only exclusively - which

means there are no parallel executions of different branches within the same if-else-construct - we have to reduce the log variable assignments to the leaf nodes of a conditiontree. A piece of pseudo code which considers this point is shown in Algorithm 3. Wealso distinguish between long form and short form conditions, where the former one canbe extended, while the latter one needs to be duplicated and adapted.In order to interpret the branches of such a conditional construct as states, we have

to merge all conditions down to that individual leaf node to one single expression whichallows us to identify the meaning of that branch. This implies that we combine all prede-cessor conditions as a conjunction because they are all active at that point and thereforetrue, and add the negated conditions of all previous paths as the current condition is onlyreached when those have been proven false. Therefore, the condition tree is required tobe chronologically sorted, so that we can consider all paths left of our current conditionto be neglected.An algorithmic form of this idea can be found in Algorithm 4. With both the log

variables for each relevant branch and our new list of merged conditions to describe eachof these branches, we are now able to treat them as states just like we do in all statetransition diagrams. Once again, there is still a difference between those diagrams, asour physical system is limited to exclusive-OR decomposition, so we do not have to dealwith parallel executed branches here.

Algorithm 3: AssignLogVariablesInCodeInput: Original source code file (SRC)Data: A tree representation of conditions (Cset)Result: Log-enabled source code file (SRCLog)

1 Determine all if-else-constructs in source code;

2 foreach {if-else-construct IEC with conditions C ∈ Cset} do3 Define single log variable;4 if IEC is in short form then5 Duplicate construct with assignment to log variable;

// logvar1 = if cond1, result1 elseif cond2, result2 else result3 end;6 else7 foreach {condition C ∈ IEC | C has no nested conditions} do8 Insert log variable assignment into the original construct;

25


Algorithm 4: MergeConditionsInput: Source code file (SRC)Data: A tree representation of conditions (Cset)Result: A list with one merged condition for each branch (MergedCset)

1 Determine all if-else-constructs in source code;

2 foreach {if-else-construct IEC with conditions C ∈ Cset} do3 Determine condition tree for the IEC;4 foreach {condition C ∈ IEC | C has no nested conditions} do5 trueCond = all conditions of the path of C;6 falseCond = all top-level conditions left to the path of C;7 mergedCond = ’NOT(falseCond(1) OR falseCond(2) OR . . . ) AND8 trueCond(1) AND trueCond(2) AND . . . ’;9 MergedCset.add(mergedCond);

Another point we need to think about is how we should handle the logging data wereceive from the simulation. As we decided to log in raw form of a list with only the timeand data values to relieve the simulation process from additional load and performancedegradation, we have to restructure that data in a way that allows us to use it for thecoverage analysis. There are two major areas of interest.At first we want to get the data about the number of activations and the total active

time of each state. We will add a log entry each time a state either gets activated ordeactivated instead of logging each time step. This allows us to easily get those twovalues by basic matrix operations. For the number of activations, we just have to countthe number of entries with a data value equal to 1. To get the active time, we willduplicate and shift the new data table by one entry and expand them with the start andend time of that simulation. After determining the indices of the non-shifted table inwhich the state gets activated, we reduce both tables to these lines, perform a matrixsubtraction and finally calculate the sum of that vector of time differences. These stepscan be processed quite fast, as MATLAB provides highly optimized functions for matrixcalculations. Summing up this cycle, we obtain the pseudo code in Algorithm 5.

Algorithm 5: DetermineActiveTimeInput: Logging data set(Lset)Result: Active time (Tact)

1 timeData = {start time, Lset time column};2 shiftedTimeData = {Lset time column, end time};3 activationData = {initial status, Lset value column};4 activatingLogs = activationData.find(value == 1); // entries where the state gets activated

5 Reduce both time vectors to activatingLogs ;

6 timeDifferences = shiftedTimeData - timeData;7 Tact = sum(timeDifferences);

26


Algorithm 6: DetermineTraversedPathsInput: Logging data sets(Lsets), Path length (n)Data: A finite set of transitions (Tset) and states (Sset)Result: Set of used paths (Pset)

1 Append transition data to each log entry;

2 unionList = ”;3 foreach {Logging data set Lset | Lset ∈ Lsets} do4 unionList.append(Lset);

5 Sort list by time value;

6 foreach {Logdatarow L = {T ime, TransitionID, Source,Destination}|7 L ∈ unionList, {Source,Destination} ∈ Sset} do8 usedPath = ”;9 usedPath(1) = L.TransitionID;

10 for i = 2 to n− 1 do11 destination = L.Destination;12 Find first L ∈ unionList with L.Source == destination;13 if L exists then14 usedPath(i) = L.TransitionID;15 else16 break;

17 if usedPath.length == n-1 then18 Pset.add(usedPath);

The second and last aspect to consider is the determination of the paths that haveactually been taken during the simulation. We already stated that our logging data onlyconsists of activation values with the corresponding time step. So the most reasonablemethod to find paths within that data is to merge all the transition logging tables fromone chart, sort them by their time value and extend the entries with information aboutthe source and destination state of that transition. We can then iterate through that list,and each time take the current row as a starting point, determine its destination state,find the chronologically next transition with this state as its source, and repeat thatprocedure until we either find a path end, or reach the destined length of an n-switch.Our algorithm for that task will therefore look as shown in Algorithm 6, and the obtainedlist of used paths only needs to be compared to the list of existing paths in a last stepto calculate the ratio of path and n-switch coverage.

27


4.4. Workflow of the TCA program

Summing up the necessary process steps for the model,General Preparation

Simscape Log Setup Stateflow Log Setup

Data Postprocessing

Model Preprocessing

Unlocking Library Links

Acquire Simscape Code Data

Gather Chart and PhysicalBlock Information

Choose Model Parts to Log(set through GUI)

BackupSimscapeCode

Adapt CodeFor Logging

Transfer Branchesto State Structure

Rebuild PhysicalBlock Library

Compose State /Transition Data

Structure

Add LoggingVariable Calls toAll SF Objects

Connect Logging BlocksInside the Model

Simulation Execution

Transfer Raw Logging Data toTCA Data Structure

Determine Execution Timesand Activated Parts

Gather Coverage Dataon Each Logical Level

Output Data

Figure 4.6.: The TCAworkflow

it leads to a function graph as shown in Figure 4.6. Thedifferent steps can be divided into three major cate-gories.

The first one - called the model preprocessing step -contains all relevant changes that need to be applied tothe model in order to enable a decent logging function-ality. This may require that all existing library links ofchart and subsystem objects get deactivated. The linkdeactivation implies that we have direct access to thereferenced libraries, as it would be impossible to set upthe logging functions for the base blocks otherwise. Itis a necessary step considering the fact that the internalobject search functions will only be able to find maskedblocks in case that their parent block is really containedin the model and not just referenced. Through thisdeactivation, the model will contain copies of the li-brary referenced blocks. This step also allows us toapply changes to the interconnection of blocks underthe mask or inside the chart without the need to followthe references to the library and perform changes there.Through the next steps of the general preparation,

the system will search for underlying code files of thephysical components as well as gather handles to allcharts and physical blocks to acquire significant infor-mation about them. The user will then be able to usethe GUI in order to choose those blocks and code filesthat need to be tracked in the following logging andanalysis part. The complete preprocessing workflow incode form is provided in the appendix with the ‘pre-processModel’ function.

In the next step, we will distinguish between physicalSimscape blocks and Stateflow charts regarding the waythe logging function is set up. The Simscape path leadsto the code level of the system, requiring automaticallyhard coded changes to the referenced files to recreate the involved if-else-structures andmap them to log variables. While iterating through these file passages, we will save thedifferent conditions of the branches and generate state representations of them. Whenall changes are finally applied, we need to rebuild the physical block library so that theprevious changes get integrated into the model itself.

28


The Stateflow path on the other hand requires us to structure the state and transitioninformation from a graphical level. As we already obtained complete lists of all charts inthe general preparation step, we need to acquire all states and transitions inside them andrestructure their information inside the prospective coverage data structure. The secondstep then consists of adding logging variable calls to each of the states and transitions toallow the system to recognize activation and deactivation processes.In a final preprocessing step, we connect copies of the designed logging block to each

of the newly created outputs, which allows us to save the raw (de)activation data to theworkspace and afterwards to a file. To reduce the data size as well as the time needed toiterate through the data, we designed the logging block to be triggered by input change,so that only those relevant time steps are actually logged.

The next part involves the normal simulation run, which is not influenced through theadditional functions in the way it is executed. As the logging process works independentlyfor each simulation run, the tasks can still be distributed to a cluster system for parallelexecution.Certainly, it would be possible to process the state and transition changes directly

and update the data tables at this point while the simulation is running. However, thesefunction calls are forced to be extrinsic in order to access the global tables (this means thatthe functions are not precompiled, but transferred to the MATLAB interpreter). As aresult, such a ‘real-time’-based coverage tracking causes major performance degradation,executing the simulation at less than 1% of the initial speed. We will show later on inthe evaluation section why we can save a huge amount of time with postprocessing.

The third section of the diagram shows the different steps that are applied to the loggeddata as postprocessing. The data is logged in raw form, which means that they areenumerated in a timeseries object with the data values and a time stamp using theSimulink signal logging function. Therefore, we need to iterate through these data setsand update parameters like ‘number of activations’ and ‘total activation time’ respec-tively. After this step, we have provided a complete data structure with all necessaryinformation to determine the actual coverage values. The corresponding code for thisstep regarding the primary simulation paths is appended as the ‘createPrimaryTables’function. Lastly, the program will iterate through the data structure on all levels andsum up the coverage information for each section. The top-level code for this processis the ‘evaluateActivityData’ function shown in the appendix. We transfer the resultingstructure to a graphical interface at last, allowing the user to review and evaluate thecoverage of both the complete model and selected parts.

29

5. TCA Application and Coverage Data Acquisition |

5. TCA Application and Coverage DataAcquisition

In the following, the previously developed and MATLAB compatible code blocks formodel preprocessing and data postprocessing will be applied to the MFFCS model inorder to collect coverage data and evaluate the results in terms of effectiveness andpossible optimization.The first simulation process in focus will be primary paths through the model. These

paths provide input stimuli throughout the whole simulation time and are designed tosimulate normal system behavior. This means in particular that all parameter precondi-tions like air pressure and temperature lie within regular ranges and no fault events forany of the involved components are triggered. The simulation results of these runs willthen provide a basis for the secondary path simulations, which we will deal with in thenext section. As each of the considered paths is again divided into several segments, andcoverage data can be acquired between each simulation call, we will not only be able toget the overall coverage of these primary paths, but also intermediate results to evaluatethe time periods of greater coverage.The coverage data for the complete model with all four different primary paths is shownin Table 5.1, which is divided into the state transition diagram section (=STD) as wellas the function flow chart (=FN) and Simscape code (=SSC) parts. By providing the‘total’ object amounts and the covered parts in both absolute and relative form inside thetable, we can get a basic understanding of the scope and size of the model. The primarypaths 1 and 2 are the most extensive ones among this first set of simulation runs, beingdivided into 13 segments for the different phases of a flight mission with both groundand flight operation. The first path represents a complete flight mission with all relevantstages. This means that it leads through the different phases of initial ground operationbefore take-off, including software- and engine-startup as well as machine positioning,the complete set of flight operation steps (for example, climb, cruise and descent flight),and the final ground operations after landing. The second path is similar to the first onein many respects, with the difference that the system is not led through the shutdown

Primarysim paths

STD FN SSC

state transition 1-switch 2-switch function path branch(total: 697) (total:837) (total:959) (total:1184) (total:17) (total:49) (total:1917)

path 1 484 (70%) 449 (54%) 281 (29%) 195 (16%) 15 (88%) 17 (35%) 1355 (71%)path 2 447 (64%) 392 (47%) 228 (24%) 159 (13%) 15 (88%) 17 (35%) 1353 (71%)path 3 413 (59%) 365 (44%) 199 (21%) 132 (11%) 15 (88%) 15 (31%) 1233 (64%)path 4 32 (5%) 24 (3%) 3 (<1%) 2 (<1%) 0 (0%) 0 (0%) 1115 (58%)

Table 5.1.: Coverage for primary simulation paths without events

31

| 5. TCA Application and Coverage Data Acquisition

Secondarysim paths

STD FN SSC


path 1 518 (74%) 478 (57%) 306 (32%) 205 (17%) 16 (94%) 19 (39%) 1399 (73%)path 2 518 (74%) 481 (57%) 309 (32%) 208 (18%) 16 (94%) 19 (39%) 1384 (72%)path 3 518 (74%) 478 (57%) 306 (32%) 205 (17%) 16 (94%) 19 (39%) 1397 (73%)path 4 518 (74%) 478 (57%) 306 (32%) 205 (17%) 16 (94%) 19 (39%) 1400 (73%)path 5 518 (74%) 478 (57%) 306 (32%) 205 (17%) 16 (94%) 19 (39%) 1397 (73%)

path 6 472 (68%) 426 (51%) 246 (26%) 159 (13%) 15 (88%) 17 (35%) 1330 (69%)path 7 470 (67%) 424 (51%) 245 (26%) 158 (13%) 15 (88%) 17 (35%) 1320 (69%)path 8 469 (67%) 422 (50%) 242 (25%) 156 (13%) 15 (88%) 17 (35%) 1328 (69%)path 9 463 (66%) 416 (50%) 237 (25%) 150 (13%) 15 (88%) 17 (35%) 1321 (69%)path 10 462 (66%) 414 (49%) 234 (24%) 148 (13%) 15 (88%) 17 (35%) 1327 (69%)

Table 5.2.: Coverage for secondary simulation paths with failure events (the full tablecan be found in the appendix)

process after landing, but will return to the gate state afterwards for the preparation ofanother flight mission.

The primary paths 3 and 4 are shorter than the previously described complete flightmissions, as they will be restricted to ground operation. The third path describes thescenario in which the systems are booted up, and perform a shutdown directly afterwards.The fourth path approaches the same system sections, but it will already initiate theshutdown during the boot up procedure. One aspect we need to be clear on is that thebasic coverage of these shorter paths will be rather low compared to the other paths, asthe execution will not advance into flight related sections. A reduction of the total setof states and transitions to those which are potentially addressed by a test case could beuseful in this situation, but will not be subject to our examination here.

As seen in Table 5.1, the basic path with the most expanded coverage is the first one.Being the most general and comprehensive of these primary scenarios, it will provide thebase for the majority of the given set of secondary paths in the next step.

These secondary paths take up the greatest part in the complete suite of test cases with128 different scenarios. They can be divided into different sections, ranging from outbreakof fire in the cargo area, extreme condition deviation from the given norm to electricityunder-supply, to only mention a few. Table 5.2 shows a selection of these simulationpaths, five of them with the most and five with the least coverage. They will be used inthe evaluation to showcase the differences in coverage as well as determining the mostcritical coverage sections.

32

6. Evaluation |

6. Evaluation

For our evaluation, we will start in detail with the most notable aspects of our analysis,and move on to the whole picture later on. The first important aspect that we canidentify from the two tables of the Data Acquisition chapter is the similarity of thedifferent secondary paths regarding their coverage ratio. This could lead to two differentinterpretations: The state activations are either well distributed among the test cases,so they cover same sized, different parts of the diagram, or they all basically cover thesame subset of states and transitions. Our final evaluation of the whole test suite willreveal that the latter interpretation is actually true.A first hint for this statement can be found in Figure 6.1, which illustrates the coverage

ratio values of each segment of the primary paths, representing the different phases ofthe flight mission. From the complete paths 1 and 2, we can see that there are mainlytwo phases where the coverage value significantly changes, one at the beginning and onetowards the end. The most significant of them is obviously the start phase, as the coveragechanges from 4.45% to 50.79% between the first two phases. Earlier we pointed out thatcaused by the completeness of the first primary path, nearly all secondary paths use thatone as their base. So as most states and transitions are activated during initial groundoperation and take off in that primary path, the following secondary segments start withthe same high coverage ratio and only influence the specialized paths as emergency modeor a fire in the cargo area. This explains the comparatively small differences between thecoverage values of the best and worst secondary paths we have seen in Table 5.2 before.

0 1 2 3 4 5 6 7 8 9 10 11 12 130

20

40

60

80

100

4.45

50.79

Path segment

StateCoverage[%

]

Primary path 1Primary path 2Primary path 3Primary path 4

Figure 6.1.: The development of state coverage during the primary path segments

33

| 6. Evaluation

As we could see in the previous chapter, none of the given test cases reached a completecoverage of the model. We need to consider a number of different aspects related to themodel composition and function to explain that behavior.The first point pertains the possibility of activation for the states and transitions con-

tained in the diagrams. In the MFFCS model, it is intended that a certain subset ofstates deals with those situations where a timeout of a functional state occurs. As wealready pointed out earlier, these timeout situations should not be reachable at normalsystem behavior paired with any possible combination of input and environment param-eters. A timeout state should only be activated in case that one of the related physicalcomponents shows failure behavior, which cannot be forced by any parameter that ismodifiable through test cases.But even when we only consider the number of activated functional states or transi-

tions, we can observe that a complete coverage is not reached in any case. One reasonfor that is an insufficient division of the states into different categories. This means thatbesides the already identified and alternatively handled timeout states, there are stillstates inside the functional set which are only activated on system fault, so the sameprinciples as explained before would have to be applied here.A manual exclusion of all those states from the group of functional components reduces

the set of unintentionally activated or inactive states to a small amount of elements thatreally indicate flaws of the model. This remaining subset is shown in Table 6.1 andTable 6.2 and it is divided into groups that demonstrate the different behaviors we canidentify.

We will start with a look at the Stateflow related failures. The first state in Table 6.1 ispart of the MFFCS main controller and belongs to the group of the intentionally inactivetimeout indication states. On normal system processing, the corresponding functional‘Start_Ground_Power_Mode’ state is restricted to run for 30 seconds at most, butboth extensive primary paths 1 and 2 force the system to stay in that state for a longertime and therefore trigger the transition to the timeout component. The situation isespecially relevant because this functional state is also being entered during the startupprocedure, but without triggering the timeout transition. So this timeout only occurs

RelevantStateflow states

state information

model component state name activations

state 1 ’MFFCS_Controller’ ’Start_Ground_Power_Mode_Timeout’ 2

state 2 ’Model_Controller’ ’Takeoff_vB’ 0state 3 ’Model_Controller’ ’Climb_vB’ 0state 4 ’Model_Controller’ ’Descent_vB’ 0state 5 ’Model_Controller’ ’Landing_vB’ 0

state 6 ’AirSupply2_Controller’ ’Cargo_Inerting’ 0state 7 ’AirSupply1_Controller’ ’Emerg_Power’ 0

Table 6.1.: Examples of activated and non-activated relevant Stateflow states

34

6. Evaluation |

after a certain sequence of actions, which most probably involves the activation of thetank inerting system. With the insight from our discussion on the reachability of timeoutstates, we can see that this failure is not caused by insufficient coverage of our test cases,but entirely model-related, so the model itself needs to be adapted to fix this.

A case in which we can indeed observe a deficient coverage is given with the next fourstates 2-5, all part of the top-level event and condition controller. We have been provideda reduced test suite for this analysis, and as we can see, the whole ‘version B’ branch ofevent triggering states is not covered by that set. This contains a completely alternativeflight mission, ranging from takeoff to the landing process. An interesting observationat this point is the fact that nevertheless nearly all functional states are reached, whichmeans for our analysis that this alternative branch is not important in terms of stateand transition coverage. We can assume that this scenario only changes the input andoutput values of the physical system without changing the controlling behavior of thefinite state machines. More precisely, this ‘version B’ branch contains alternative climband sink rates for the flight mission, and affects the dynamic controllers on the physicallevel regarding their value change rates. So the higher-level control automata are mostlyindependent of those changes, and this independence is given in our test cases.

A third sort of failure that could be revealed with our analysis approach regards stateswhich will not be executed at all. Of course we are not able to directly identify andclassify non-triggered states as either not connected to the net through transitions, orunreachable because of an unsatisfiable transition condition. But the knowledge fromour analysis - which involves the activation numbers and times as well as path andswitch information - makes it possible to locate these flaws for further investigation. Inthis specific case, shown with the states 6 and 7, the problem could be traced back tothe base library object which this subsystem is derived from. Both of the air supplycontrollers are equal in their structure and composition, but their corresponding physicalcomponents serve different purposes.As long as the flight mission follows the normal cycle, the two fuel cell systems are used

as the auxiliary power unit. The first one is additionally used to inert the tank, and incase of emergency it can be switched to cargo inerting as well. The second system on theother hand will be used to provide power in emergency mode. So while ‘Cargo_Inerting’is activated in some test cases for the first fuel cell system, ‘Emerg_Power’ is nevertriggered at all, and vice versa for the second system.This demonstrates that although their basic function is the same, each of the fuel

cell systems shows some specific behavior under exceptional circumstances. Hence, thegeneral and non-specified design of the controller, which we can find in the library, leadsto unused parts inside the state transition diagram, and therefore reduces the possiblemaximum ratio of coverage.

Using our approach regarding if-else-constructs in source code, we are also able to identifycoverage related failures and get insight into the function of the precompiled, closedSimscape systems as shown in Table 6.2. The states 1 and 2 both show a number of

35

| 6. Evaluation

activations that lies above 45 million times during the execution of the four primaryand 128 secondary paths. Analyzing the activation sequences that we obtained from theraw state activity logging data, we can locate certain phases in a flight mission in whichthe condition changes nearly each time step (∆t = 0.01s), especially during takeoff andlanding, whereas there are other phases of the simulation in which a condition stays trueor false for hundreds of seconds during flight. This seemingly abnormal developmentis most probably caused by numerical deviation when the pressure difference is closeto 0 and the calculated value fluctuates around that point. Nevertheless, this behaviordoes not affect the function of the system, so that the priority to adapt the underlyingmathematical equations may be lower compared to the other analysis results.

The second point are those branches inside the physical Simscape blocks which are nevertraversed at any point of the simulation. The four states 3-6 are the core conditionsof the regulative part of an air valve, which can be found inside numerous componentsof the system. We can see that the first two states are entered nearly six thousandtimes during the simulation, which is a normal behavior considering that the pressurelies within laminar ranges. In contrast, the last two states are never executed in any ofthe test cases.At this point, we need to assess that situation from two different perspectives. From a

testing point of view, it is detrimental for a functionality analysis because we can neverevaluate the effects of the parameter assignments inside those decision branches. Butconsidering the functional quality of the model, it is a positive characteristic as thesetwo states represent the transition region to overcritical pressure as well as the criticalpressure section itself. So when these states are never entered, it indicates that thesubsystem operates in normal mode. There are other valves in the model which reachthe transition region a few times, but none of them ever enters the overcritical state,allowing us to consider the mass flow situation to be secure.

Although it is not directly related to the results of our analysis, we were also able todetect two missing parts of the model during our systematic look at the existing statesand transitions. The first one concerns the ‘H2_Tank_Controller’, where we analyzedthe timeout and fault components and discovered that a state for triggering the pressureevaluation inside the tank was not implemented. As a result, the system would directly

RelevantSimscape states

state information

model component state name activations

state 1 ’FCS2_H2_Pump3’ ’dir_flow > 0’ 45,791,431state 2 ’FCS2_H2_Pump3’ ’dir_flow <= 0’ 45,791,459

state 3 ’FCS1_Air_Valve1’ ’p_u <= p_d’ 5,848state 4 ’FCS1_Air_Valve1’ ’(p_d / p_u) > A.B_lam’ 5,741state 5 ’FCS1_Air_Valve1’ ’(p_d / p_u) > B_cr’ 0state 6 ’FCS1_Air_Valve1’ ’(p_d / p_u) <= B_cr’ 0

Table 6.2.: Examples of activated and non-activated relevant Simscape states

36

6. Evaluation |

change from ‘Pressure_OK’ to ‘Pressure_Fault’ state when the value exceeds a boundarywithout providing a transition period in which that value is further evaluated beforedeclaring a system fault. The other peculiarity affects the custom Simscape ‘H2_Valve’components inside the fuel cells and the H2 tank. They should also be divided intolaminar, transition, and overcritical sections like the air valves we looked at before, buta check and the corresponding mass flow value assignments of the laminar region cannotbe found in the source code. This laminar region needs to be implemented in order todefine the component behavior under normal use conditions.

Finally, we evaluate the test case suite as a whole. A complete top-level overview of allcoverage techniques we applied can be found in Table 6.3. For both the basic state andtransition coverage in the state transition diagrams (= STD), we divided the set intofunctional and timeout components to reduce the influence of those parts that shouldnever be triggered at all, and to separately evaluate the timeout states under the oppositeview point ‘less is more’. We can see that among the states that we want to be entered,nearly 85% is already a really high value considering the fact that still quite some of theremaining states are only fault related and should not be traversed as well. A furthermanual categorization of the states revealed that actually 82 out of the 96 uncoveredfunctional states would have to be counted to the timeout and fault category in fact.The same principle applies to the transitions, and by addressing these categorizationproblems, we would obtain a coverage ratio of above 98% for that part, but still needto deal with the previously discussed abnormal states. Besides that, we determined thefirst few n-switch coverage values, with n ∈ {0, 1, 2}, and just as expected, the totalnumber of possible sub-paths increases in value, while the number of covered sub-pathsis decreased. We have to keep in mind that all of the test cases represent real flightmissions, and with increasing path length we will get more paths like ‘Off → Active →

Coverage criteria Statistic values

total covered ratio

STD total state coverage 697 529 75.90%functional state coverage 624 528 84.62%timeout state coverage 73 1 1.37%

total transition coverage 837 511 61.05%functional transition coverage 770 510 66.23%timeout transition coverage 67 1 1.49%

0-switch coverage 837 511 61.05%1-switch coverage 959 339 35.35%2-switch coverage 1184 228 19.26%

FN function coverage 17 16 94.12%function transition coverage 138 75 54.35%

function path coverage 49 19 38.78%

SSC code branch coverage 1917 1419 74.02%

Table 6.3.: Total test suite coverage data

37

| 6. Evaluation

Off→ Active’ or ‘Watch→ Off→ Active’, expecting a system restart throughout a flightmission, which is not possible during a takeoff-flight-landing cycle. Furthermore, thesesets of n-switches include all timeout transitions as well, with the consequence that thosenon-triggered transitions are part of many identified paths, especially with higher valuesof n.

Regarding the graphical function flow charts in the model (= FN), we analyzed them fromtwo different perspectives, one being the function call itself, and the second one beingthe possible paths inside a function. We see that all functions except one are executed atleast once, and the last one is related to the problem of generalized control charts from alibrary that we discovered before, in which one of these flow charts will only be activatedin a state whose function is not used by that instance of the controller. The low functiontransition coverage as well as the function path coverage values are, in contrast, based onthe problem that the transitions and junctions inside a flow chart are not categorized. InStateflow, both transitions and junctions do not have a name parameter, so there is nodirect way of indicating functional and fault paths in the diagram. A manual evaluationof these flow charts revealed that all paths which indicate proper system behavior aretraversed.

The last table row shows the coverage value of all branches in the physical Simscape codeblocks (= SSC). With a coverage of three quarters of the system, nearly all function-relevant branches are entered at least once, and the ones which are never activated aremainly critical conditions or value boundary related assignments. The latter ones areonly implemented to make sure that in case of an erroneous behavior of an upstreamcomponent, negative values are set to 0 to keep the parameters in a realistic and possiblerange.

The diagram in Figure 6.2 demonstrates how the total test suite coverage compares tothe one secondary test case with the smallest and the one with the greatest coverageability. It once again points out that all of the flight and ground missions are built up onnearly the same base phases and that - despite of the differences of the input parametersand conditions in each case - each mission traverses through a similar sequence of states.We can see that even the test case of least coverage is able to enter 66.28% of the statesand trigger 49.46% of transitions, which is around 80% of the set of components that isentered during the whole test suite execution. So when we look at the flight missions froma systematic model view, there is room for improvement regarding a reduction of testcases maintaining the current coverage values. Regarding state coverage, the completesecondary path table in the appendix shows that there are up to 45 test cases coveringthe same amount of states (in this example, all of them enter 503 states). A furtherlook into the logging data reveals that these covered states are indeed identical ones. Asimilar situation can be found in the other coverage data sets as well, so this redundancyis not limited to state coverage.While the states and transitions as well as all three analyzed n-switch coverages stillshow a slight improvement of coverage when we compare the best single test case to

38

6. Evaluation |

state

state-fun

ctional

state-tim

eout

transition

transition-functional

transition-timeout

0-switch

1-switch

2-switch

function

function-transition

function-path

code-branch

0

20

40

60

80

100

20

40

60

80

ratiovalue[%

]

worst case best case total test suite

Figure 6.2.: Coverage comparison for total, best and worst covering test cases

the complete set, all function related viewpoints reveal that only one single test case issufficient to cover the same transitions and paths inside the flow charts which are alsocovered by the total test suite.On the source code level of the physical components, we can observe a similar branch

coverage of the three cases as well. While the worst case covers 69.22% of all branches,the best test case reaches a value of 72.30%, and the test case suite finally covers 74.02%of the physical system. So as the best case already has a share of over 97% of the totalreached coverage, it is obvious that this suite can also be optimized and reduced fromthis point of view.

39

7. Conclusion |

7. Conclusion

On a systematic, state-based level, our coverage analysis approach allowed us to identifyall remaining failures of the model regarding state and transition activation. We havealso been able to get insight into the cycle of the state transition diagrams and flowcharts, which led us through the evaluation of paths and n-switches and made it possibleto determine the reasons why and to which extent the system cannot be completelycovered. We found out that this limitation is based on one of the fundamental principlesbehind the test suite, claiming that all test cases are based on a realistic flight mission,so that situations like intermediate, complete system resets do not occur. To reach themaximum coverage values for paths and n-switches, we would therefore need to considerthese unrealistic cases as well. On the other hand, we saw that a large number of testcases barely contribute to the total achieved coverage value. This fact shows that thereis a need to optimize the test cases in both directions, in case a sufficient coverage ofboth the components and paths is set as the main focus.

It was also possible to systematically look into the processes inside all custom createdphysical components using our logging technique at code level. We discovered thatnumerical calculations caused rapid fluctuation for some of the decision branches in manyblocks, especially regarding the stream directions in phases of low pressure gradients closeto 0. But we were also able to confirm the demands that were made for some parametersas long as the different parameter ranges were modelled as separate branches of an if-else-construct. That way we could determine that the pressure values of all valves rarelyentered a range beyond normal laminar pressure and never reached the overcritical state.Nevertheless, a complete analysis of the value coverages inside and between the physicalblocks was not possible for us, as the model structure is too complex to automaticallytransform the single components into hybrid automata.

7.1. Future prospect

The previous paragraph showed that it is necessary to change the model itself in orderto perform any further analysis techniques that go beyond the level of state, transitionsand decision branches. If a systematic and state-based analysis of the parameter valuereachability is aimed for, each of the physical components needs to be remodelled bythe principles of a hybrid automaton, which involves the transformation of each possibleinfluence on the value into a separate control state. Our valve example in chapter 3already demonstrated the effort one needs to put into the remodelling process, and thisexample is still one of the most basic ones, while the MFFCS model contains more com-plex constructs. Another approach might be to leave the code section and tap the signalsbetween the physical blocks on Simulink level. The signal data may then be classifiedin either even distributed ranges or predefined sectors based on realistic categories (forexample, low, normal, and critical pressure), which can be interpreted as states in our

41

| 7. Conclusion

approach. This would allow us to include the set of Simulink blocks without source codeaccess into our considerations as well.A further categorization of the states and transitions is another aspect that needs to

be addressed in the future. As our simulation results revealed, it is not only necessaryto separate the timeout components from the rest of the system, but we would need toextend this division for fault states and evaluation blocks in the state transition diagramsas well as for erroneous paths in flow charts. The results should then be one set of statesthat we aim a complete coverage for, and another set of states which are not triggeredon normal system behavior with any possible input parameter value or event. Those setscan finally be used to analyze the total coverage on the one hand, and on the other handto easily identify system failures leading to timeout or even fault states.As a third point, we might reconsider the real time integration of our analysis methods

into the simulation execution in the long term. For this model scale, we had to relyon a MATLAB code implementation to assure realistic development time frames. Aswe pointed out before, the MATLAB function calls paired with the need to randomlyaccess data table entries caused a major slowdown of the complete execution. A dif-ferent solution for this would involve the creation of additional interfaces between theMATLAB-based model and C-code regarding (parallel) structure access. It is then pos-sible to use the C-S-Function blocks for the calculations, which are not interpreted likeMATLAB code, but precompiled and therefore provide immense performance improve-ments. Basically all steps included in the data postprocessing step could be executedduring simulation this way, while ensuring a still acceptable increase of simulation time.

We have to remark that our work only covers the analysis part of the optimization cycle,but neither the definite source of the failure (as we can only review the automata level)nor a solution to fix the fault in most cases. This means that another aspect for futurework has to be the integration of our analysis results into the model and test case suite.At the time of writing, the creator of the provided model for example was able to locatethe source of the ground power mode timeout (see chapter 6 for the failure details) inanother subsystem connected to the ground power state. The reason could be identifiedas a too general definition of one state, which caused the system to wait for multipleoperations before signalizing a success, while only the first operation is related to theground power mode state. So the ground power mode state should not need to waitfor the following operations (for example, disabling heater and dryer systems), which isthe case in the current implementation. A more differentiated division of the states andmode signals is then considered to solve the problem. As the flight mission phase thatcontains this failure is only followed by the system shutdown after landing in a normalmission, it did not show an effect on the system, which made it difficult to recognizeduring the simulation runs. Our analysis approach proved to be a beneficial addition inthose cases, as it treats every state and transition separate of the context it is integratedto.

42

A. User manual |

A. User manual

Step 1: Add the code base directory of the TCA program to the MATLAB path.

Step 2: Run the model preprocessing step with the following command:preprocessModel(modelName,modelPath);

[e.g., preprocessModel(’mffcs_mil’, ’E : \MFFCS_MIL_Test_Coverage \’)]

Step 3: Select the Stateflow chart blocks that should be logged via the GUI:

Step 4: Select the Simscape code files to log, as well as their corresponding physicalblocks in the model:

43

| A. User manual

Step 5: The MATLAB command window provides information on each execution phase.[e.g., Deactivating link for block 14: mffcs_mil/ViTEST_Suite/

Test_Interface/ECOS_Mode/Ambient_Temperature ... done.]

Step 6: Wait until the script execution has finished.The model is now prepared to log the selected signals on execution.

Step 7: Restructure the obtained raw logging data with the following command:processRawLoggingData(rawDataPath, TCADataTemplate);

[e.g., processRawLoggingData(’E : \TCA_Logging_Data \cov_data \’,TCALogStructure)]

→ The template can be found in the model directory after preprocessing.

Step 8: Initiate the evaluation of the new data structures by typing:evaluateActivityData(structuredDataPath);

[e.g., evaluateActivityData(’E : \TCA_Logging_Data \cov_data \TCAData \)’]

Step 9: The data structure tables can now be opened via the MATLAB workspace(to be viewed, edited, and sorted in the Variables window)

44

B. Complete secondary path table |

B. Complete secondary path table

Secondarysim paths

STD FN SSC


path 1 462 (66%) 414 (49%) 234 (24%) 148 (13%) 15 (88%) 17 (35%) 1327 (69%)path 2 469 (67%) 422 (50%) 242 (25%) 156 (13%) 15 (88%) 17 (35%) 1328 (69%)path 3 472 (68%) 426 (51%) 246 (26%) 159 (13%) 15 (88%) 17 (35%) 1330 (69%)path 4 463 (66%) 416 (50%) 237 (25%) 150 (13%) 15 (88%) 17 (35%) 1321 (69%)path 5 470 (67%) 424 (51%) 245 (26%) 158 (13%) 15 (88%) 17 (35%) 1320 (69%)path 6 473 (68%) 428 (51%) 249 (26%) 161 (14%) 15 (88%) 17 (35%) 1322 (69%)path 7 498 (71%) 455 (54%) 289 (30%) 196 (17%) 16 (94%) 19 (39%) 1361 (71%)path 8 497 (71%) 453 (54%) 287 (30%) 195 (16%) 16 (94%) 19 (39%) 1362 (71%)path 9 497 (71%) 453 (54%) 287 (30%) 195 (16%) 16 (94%) 19 (39%) 1359 (71%)path 10 497 (71%) 453 (54%) 287 (30%) 195 (16%) 16 (94%) 19 (39%) 1364 (71%)




45

| B. Complete secondary path table

Secondarysim paths

STD FN SSC







46

B. Complete secondary path table |

Secondarysim paths

STD FN SSC





path 121 504 (72%) 465 (56%) 287 (30%) 192 (16%) 15 (88%) 17 (35%) 1380 (72%)path 122 504 (72%) 465 (56%) 287 (30%) 192 (16%) 15 (88%) 17 (35%) 1380 (72%)path 123 504 (72%) 465 (56%) 287 (30%) 192 (16%) 15 (88%) 17 (35%) 1380 (72%)path 124 504 (72%) 465 (56%) 287 (30%) 192 (16%) 15 (88%) 17 (35%) 1370 (71%)path 125 504 (72%) 465 (56%) 287 (30%) 192 (16%) 15 (88%) 17 (35%) 1370 (71%)path 126 504 (72%) 462 (55%) 284 (30%) 189 (16%) 15 (88%) 17 (35%) 1372 (72%)path 127 503 (72%) 468 (56%) 287 (30%) 193 (16%) 15 (88%) 17 (35%) 1376 (72%)path 128 504 (72%) 470 (56%) 290 (30%) 195 (16%) 15 (88%) 17 (35%) 1370 (71%)

47

C. Listings |

C. Listings

1 function preprocessModel (modelName , modelPath )2 % PREPROCESSMODEL Setup the model components f o r a c t i v i t y l o g g i n g .3 % The func t i on i d e n t i f i e s the e x i s t i n g char ts , code f i l e s and ph y s i c a l4 % blocks , a l l ow s a p a r t i a l s e l e c t i o n o f them through the GUI, and f i n a l l y5 % se t s up the l o g g i n g connec t ions to adapt the model f o r a c t i v i t y l o g g i n g6 % purposes . The model w i l l then d e l i v e r the l o g data when i t i s execu ted .7 global TCALogStructure ;8

9 % −− I n i t i a l i z e a l l Maps and Se t s as empty s t r u c t u r e s10 TCASSCDataSets = struct ( ) ;11 TCALogStructure = struct ( ) ;12 TCALogStructure . SSC = struct ( ) ;13 TCALogStructure .STD = struct ( ) ;14

15 cd (modelPath ) ;16 load_system (modelName) ;17

18 % −− Enable the model f o r s i g n a l l o g g i n g in t o da t a s e t format o b j e c t s19 set_param(modelName , ’ S igna lLogg ing ’ , ’ on ’ ) ;20 set_param(modelName , ’ SignalLoggingSaveFormat ’ , ’ Dataset ’ ) ;21

22 % −− Unlock r e l e v an t l i b r a r y l i n k s23 unlockLibraryLinksByType (modelName , ’ SubSystem ’ ) ;24

25 % −− Search f o r a l l S t a t e f l ow char t o b j e c t s i n s i d e the model26 fpr intf ( ’ Search f o r Sta t e f l ow char t s . . . ’ ) ;27 r t = sfroot ;28 m = rt . find ( ’− i s a ’ , ’ Simulink . BlockDiagram ’ , ’−and ’ , ’Name ’ ,

modelName) ;29 s ta t e f l owChar t s = m. find ( ’− i s a ’ , ’ S ta t e f l ow . Chart ’ ) ;30 s tate f lowChartPaths = sta t e f l owChar t s . get ( ’ Path ’ ) ;31

32 % −− Erase dup l i c a t e d e n t r i e s caused by d e t e c t i n g both char t s and33 % subchar t s wi th the prev ious search34 for i =1:1 :numel( s ta t e f l owChar t s )35 subCharts = s ta t e f l owChar t s ( i ) . find ( ’− i s a ’ ,

’ S ta t e f l ow . AtomicSubchart ’ , ’−or ’ , ’− i s a ’ ,’ S ta t e f l ow . AtomicBox ’ ) ;

36 subChartPaths = sprintf ( ’%s/%s ’ , subCharts . get ( ’ Path ’ ) ,subCharts . get ( ’Name ’ ) ) ;

37 s tate f lowChartPaths = setd i f f ( state f lowChartPaths , subChartPaths ) ;38 end39 fpr intf ( ’%i found . \ n ’ , numel( s tate f lowChartPaths ) ) ;40

41 % −− Search f o r a l l Simscape o b j e c t s i n s i d e the model42 fpr intf ( ’ Search f o r Simscape b locks . . . ’ ) ;43 s imscapeBlockPaths = find_system (modelName , ’ LookUnderMasks ’ , ’ a l l ’ ,

’ BlockType ’ , ’ SimscapeBlock ’ ) ;44 fpr intf ( ’%i found . \ n ’ , numel( s imscapeBlockPaths ) ) ;

49

| C. Listings

46 % −− Search f o r Simscape code f i l e s in the s p e c i f i e d package d i r e c t o r y47 fpr intf ( ’Get Simscape code f i l e s . . . ’ ) ;48 s imscapeCodeFi les = getFo lderAndSubfo lderF i l e s ( ’+Simscape ’ , ’ ∗ . s s c ’ ,

[ ] ) ;49 fpr intf ( ’%i found . \ n ’ , numel( s imscapeCodeFi les ) ) ;50

51 % −− Open GUI to s e l e c t a l l c ha r t s t h a t shou ld be l o gged52 i f ∼isempty ( s tate f lowChartPaths )53 [∼ , s e l e c t edS ta t e f l owChar t s ] = LogSe l e c t i on ( state f lowChartPaths ,

’ S ta t e f l ow chart b lock ’ ) ;54 else55 s e l e c t edS ta t e f l owChar t s = [ ] ;56 fpr intf ( ’No s t a t e f l ow char t s a v a i l a b l e . \ n ’ ) ;57 end58

59 % −− Open GUI to s e l e c t a l l Simscape code f i l e s to l o g60 i f ∼isempty ( s imscapeCodeFi les )61 [∼ , s e l e c t edS imscapeCodeFi l e s ] = LogSe l e c t i on ( s imscapeCodeFi les ,

’ Simscape code f i l e ’ ) ;62 else63 s imscapeCodeFi les = [ ] ;64 fpr intf ( ’No simscape code f i l e s a v a i l a b l e . \ n ’ ) ;65 end66

67 % −− Extrac t the names from the f u l l Simscape code f i l e paths68 numOfFiles = numel( s e l e c t edS imscapeCodeFi l e s ) ;69 se lectedSimscapeCodeFi leNames = ce l l ( numOfFiles , 1 ) ;70 for i =1:1 : numOfFiles71 [∼ , se lectedSimscapeCodeFi leNames { i } ,∼ ] =

f i l eparts ( s e l e c t edS imscapeCodeFi l e s { i }) ;72 end73

74 % −− Ignore a l l s imscape b l o c k s t ha t are not connected to one o f75 % the p r e v i o u s l y chosen code f i l e s ( they w i l l not be l o gged )76 for i =1:1 :numel( s imscapeBlockPaths )77 completePath = get_param( s imscapeBlockPaths { i } , ’ ReferenceBlock ’ ) ;78 [∼ , r e f e r enceB lock ,∼ ] = f i l eparts ( completePath ) ;79 cor re spStateTab le = find ( strcmp ( selectedSimscapeCodeFileNames ,

r e f e r enc eB lo ck ) ,1 ) ;80 i f ( isempty ( co r re spStateTab le ) )81 s imscapeBlockPaths { i } = [ ] ;82 end83 end84 s imscapeBlockPaths ( cell fun (@isempty , s imscapeBlockPaths ) ) = [ ] ;85

86 % −− Graph i ca l l y s e l e c t Simscape b l o c k s from the acqu i red code su b s e t87 i f ∼isempty ( s imscapeBlockPaths )88 [∼ , s e l e c t edS imscapeBlocks ] = LogSe l e c t i on ( simscapeBlockPaths ,

’ Simscape func t i on block ’ ) ;89 else90 s e l e c t edS imscapeBlocks = [ ] ;91 fpr intf ( ’No simscape b locks us ing p r ev i ou s l y chosen simscape code

f i l e s . \ n ’ ) ;92 end

50

C. Listings |

93

94 % −− Preprocess a l l s e l e c t e d Simscape b l o c k s95 backupSimscapeCodeFiles ( ’+Simscape ’ ) ;96 for i =1:1 :numel( s e l e c t edS imscapeCodeFi l e s )97 % −− add l o g v a r i a b l e ass ignments to the Simscape code f i l e98 SSCDataSet = adaptSimscapeCodeFile ( s e l e c t edS imscapeCodeFi l e s { i }) ;99 i f (∼isempty ( SSCDataSet ) )

100 i f ( isempty ( fieldnames (TCASSCDataSets ) ) )101 TCASSCDataSets = SSCDataSet ;102 else103 TCASSCDataSets (end+1) = SSCDataSet ;104 end105 end106 end107

108 ssc_build ( ’ Simscape ’ ) ;109 close_system (modelName , 1) ;110 load_system (modelName) ;111 fpr intf ( ’ Generating Simulink l i b r a r y . . . done . \ n ’ ) ;112 SSCDataStructure = setupSimscapeLogging (modelName ,

se l ec tedS imscapeBlocks , TCASSCDataSets ) ;113

114 % −− Get S t a t e f l ow API o b j e c t s f o r the s e l e c t e d char t s115 numOfTopLevelCharts = numel( s e l e c t edS ta t e f l owChar t s ) ;116 s ta te f l owChartObjec t s = get_param( s e l e c t edSta t e f l owChar t s , ’ Object ’ ) ;117 topLeve lChartsCel lArray = ce l l ( numOfTopLevelCharts , 1 ) ;118 for i =1:1 : numOfTopLevelCharts119 topLeve lChartsCel lArray { i } = state f l owChartObjec t s { i } . find ( ’− i s a ’ ,

’ S ta t e f l ow . Chart ’ ) ;120 end121 topLeve lState f l owChart s = cell2mat ( topLeve lChartsCel lArray ) ;122

123 % −− Preprocess a l l s e l e c t e d S t a t e f l ow char t s124 STDDataStructure = composeState f lowLoggingStructure (modelName ,

topLeve lState f l owChart s ) ;125 prepareLogg ingForState f l owState s (modelName , topLeve lState f l owChart s ) ;126 prepareLogg ingForState f l owTrans i t i ons (modelName ,

topLeve lState f l owChart s ) ;127 prepareLogg ingForState f lowFunct ions (modelName ,

topLeve lState f l owChart s ) ;128 se tupState f lowLogg ing (modelName , topLeve lState f lowCharts ,

STDDataStructure ) ;129

130 % −− Assign the data s t r u c t u r e s to the g l o b a l l o g g i n g s t r u c t131 TCALogStructure . SSC = SSCDataStructure ;132 TCALogStructure .STD = STDDataStructure ;133

134 % −− Save the crea t ed s t r u c t u r e to a f i l e as i t i s needed f o r the data135 % pos t p ro c e s s i n g a f t e r the s imu la t i on136 save ( [ modelPath ’ TCALogStructure . mat ’ ] , ’ TCALogStructure ’ ) ;137

138 close_system (modelName , 1) ;139 end

51

| C. Listings

1 function createPr imaryTables ( fo lderPath , TCALogStructure )2 % CREATEPRIMARYTABLES Create s t r u c t u r ed data t a b l e s f o r a l l primary paths .3 % The func t i on i t e r a t e s through a l l i d e n t i f i e d primary path segments ,4 % re s t r u c t u r e s t h e i r l o g g i n g data , and saves in t e rmed ia t e coverage data5 % t a b l e s a f t e r each segment so we do not have to proces s them again6 % for the secondary paths , which base on those primary path segments .7 % −− Find a l l primary path segments in the chosen f o l d e r8 s ea r chAtt r ibute = ’ cov_data_1_∗_1.mat ’ ;9 matchFi les = dir ( f u l l f i l e ( fo lderPath , s ea r chAtt r ibute ) ) ;

10 numOfPrimaryPaths = numel( matchFi les ) ;11 mkdir ( fo lderPath , ’TCAData ’ ) ;12

13 % −− Process a l l primary paths14 for i =1:1 : numOfPrimaryPaths15 TCADataStructure = TCALogStructure ;16 fpr intf ( ’ c r e a t i n g t ab l e s f o r primary path %i . . . \n ’ , i ) ;17 s ea r chAtt r ibute = sprintf ( ’ cov_data_1_%i_ ∗ .mat ’ , i ) ;18 matchFi les = dir ( f u l l f i l e ( fo lderPath , s ea r chAtt r ibute ) ) ;19 matchNames = {matchFi les . name} ;20 numOfPrimaryPathSegments = numel( matchFi les ) ;21 matchFullNames = cell fun (@(x ) f u l l f i l e ( fo lderPath , x ) , matchNames ,

’ UniformOutput ’ , f a l s e ) ;22

23 % −− Sort a l l f i l e matches by t h e i r segment number24 [∼ , baseMatchNames ,∼ ] = cell fun (@(x ) f i l eparts ( x ) , matchNames ,

’ UniformOutput ’ , f a l s e ) ;25 matchNamesSplitted = cell fun (@(x ) s trsp l i t (x , ’_ ’ ) ,

baseMatchNames , ’ UniformOutput ’ , f a l s e ) ;26 entryNumbers = cell fun (@(x ) str2double ( x (end) ) ,

matchNamesSplitted ) ;27 indexAndNumberArray = [ ( 1 : numOfPrimaryPathSegments ) ’

entryNumbers ’ ] ;28 sortedIndexAndNumberArray = sort rows ( indexAndNumberArray , 2) ;29 sortedMatchElementIndices = sortedIndexAndNumberArray ( : , 1 ) ;30 sortedMatchFullNames = matchFullNames ( sortedMatchElementIndices ) ;31

32 % −− Process a l l primary path segments33 for j =1:1 : numOfPrimaryPathSegments34 fpr intf ( ’ c r e a t i n g tab l e f o r path segment %i . . . ’ , j ) ;35 % −− Load the segment l o g data in t o memory36 primaryPathSegment = load ( sortedMatchFullNames{ j }) ;37 % −− Res truc ture the raw l o g g i n g data38 TCADataStructure = analyseRawLoggingData (TCADataStructure ,

primaryPathSegment . cov_data ) ;39 % −− Save the crea t ed data s t r u c t u r e to a f i l e40 TCADataStructureName = sprintf ( ’TCADataStructure_1_%i_%i .mat ’ ,

i , j ) ;41 save ( sprintf ( ’%s /TCAData/%s ’ , fo lderPath ,

TCADataStructureName ) , ’ TCADataStructure ’ ) ;42 fpr intf ( ’ done . \ n ’ ) ;43 end44 fpr intf ( ’ c r e a t i n g t ab l e s f o r primary path %i . . . done . \ n ’ , i ) ;45 end46 end

52

C. Listings |

1 function eva luateAct iv i tyData ( fo lde rPath )2 % EVALUATEACTIVITYDATA Determine coverage va l u e s from pre s t r u c t u r ed t a b l e s .3 % The func t i on i t e r a t e s through the t a b l e f i l e s o f a de f ined f o l d e r ,4 % determines the t r a v e r s ed n−sw i t ch e s and path , and combines t ha t5 % informat ion wi th the a l r eady g iven s t a t e and t r a n s i t i o n a c t i v a t i o n6 % va lue s in order to prov ide d i f f e r e n t coverage r a t i o s . A l l f i n a l t a b l e s7 % for the d i f f e r e n t s imu la t i on paths are then merged f o r a complete8 % overview , and the coverage in format ion o f each path i s added to a9 % summary t a b l e f o r a genera l view .

10

11 % −− Find a l l data s t r u c t u r e s in the chosen f o l d e r12 s ea r chAtt r ibute = ’TCADataStructure_∗_∗_∗ .mat ’ ;13 matchFi les = dir ( f u l l f i l e ( fo lderPath , s ea r chAtt r ibute ) ) ;14 numOfDataStructures = numel( matchFi les ) ;15

16 % −− I n i t i a l i z e the s t r u c t u r e t a b l e s17 TCAMergedStructure = [ ] ;18 TCAGlobalCoverageTable = struct ( ) ;19 TCAGlobalCoverageTable . SimRunEvaluation = struct ( ) ;20 TCAGlobalCoverageTable . Tota lTestSu i teEva luat ion = struct ( ) ;21

22 % −− Evaluate a l l data s t r u c t u r e s23 for i =1:1 : numOfDataStructures24 [∼ , f i leName ,∼ ] = f i l eparts ( matchFi les ( i ) . name) ;25 fpr intf ( ’ eva lua t ing data tab l e %s . . . ’ , f i leName ) ;26

27 % −− Determine the path and segment i n d i c e s28 f i l eNameParts = s trsp l i t ( f i leName , ’_ ’ ) ;29 pathLevel = str2double ( f i l eNameParts (2 ) ) ;30 pathNum = str2double ( f i l eNameParts (3 ) ) ;31 pathSegmentNum = str2double ( f i l eNameParts (4 ) ) ;32

33 % −− Load the s t r u c t u r e in t o memory34 TCAPathDataTable = load ( sprintf ( ’%s/%s ’ , fo lderPath ,

matchFi les ( i ) . name) ) ;35 TCADataStructure = TCAPathDataTable . TCADataStructure ;36

37 % −− Perform the sw i t ch and path coverage ana l y s i s s t e p s38 TCADataStructure = analyseSwitchCoverage (TCADataStructure ) ;39 TCADataStructure = analysePathCoverage (TCADataStructure ) ;40

41 % −− Determine the coverage va l u e s and r a t i o s42 TCADataStructure = dete rmineSta t i s t i cData (TCADataStructure ) ;43

44 % −− Add a coverage data row o f t h i s s imu la t i on path to the g l o b a l45 % overv iew t a b l e46 TCAGlobalCoverageTable . SimRunEvaluation =

addEntryToGlobalCoverageTable (TCAGlobalCoverageTable . SimRunEvaluation ,TCADataStructure . S t a t i s t i c s , pathLevel , pathNum ,pathSegmentNum) ;

53

| C. Listings

48 save ( sprintf ( ’%s/%s ’ , fo lderPath , matchFi les ( i ) . name) ,’ TCADataStructure ’ ) ;

49

50 % −− Merge the a c t i v a t i o n data in t o the summary t a b l e in case t ha t51 % the curren t segment i s the f i n a l one o f a path52 s ea r chAtt r ibute = sprintf ( ’ TCADataStructure_%i_%i_ ∗ .mat ’ ,

pathLevel , pathNum) ;53 matchPathFiles = dir ( f u l l f i l e ( fo lderPath , s ea r chAtt r ibute ) ) ;54 numOfSegments = numel( matchPathFiles ) ;55 i f (pathSegmentNum == numOfSegments )56 i f ( isempty ( TCAMergedStructure ) )57 TCAMergedStructure = TCADataStructure ;58 else59 fpr intf ( ’ merging s t r u c t u r e s . . . ’ ) ;60 TCAMergedStructure =

mergeTwoDataTables (TCAMergedStructure ,TCADataStructure ) ;

61 end62 end63 fpr intf ( ’ done . \ n ’ ) ;64 end65 save ( sprintf ( ’%s/%s ’ , fo lderPath , ’ TCAMergedStructure ’ ) ,

’ TCAMergedStructure ’ ) ;66

67 % −− Determine coverage data o f the complete t e s t s u i t e68 TCAMergedStructure = dete rmineSta t i s t i cData (TCAMergedStructure ) ;69 TCAGlobalCoverageTable . Tota lTestSu i teEva luat ion =

addEntryToGlobalCoverageTable (TCAGlobalCoverageTable . Tota lTestSuiteEvaluat ion ,TCAMergedStructure . S t a t i s t i c s , ’ a l l ’ , ’ a l l ’ , ’ a l l ’ ) ;

70

71 save ( sprintf ( ’%s/%s ’ , fo lderPath , ’ TCAPrimaryGlobalCoverageTable ’ ) ,’ TCAGlobalCoverageTable ’ ) ;

72 end

54

Bibliography |

Bibliography

[1] J. Grymlas and F. Thielecke, Virtual Integration and Testing of MultifunctionalFuel Cell Systems in Commercial Aircraft, vol. 6(2):2013 of SAE Int. J. Aerosp.Germany: SAE International, 2013.

[2] H. P. Lüdders, H. Strummel, and F. Thielecke, “Model-based development of multi-functional fuel cell systems for more-electric-aircraft,” CEAS Aeronautical Journal,vol. 4, no. 2, pp. 151–174, 2013.

[3] F. Barbir, “Pem fuel cells: Theory and practice,” 2013.

[4] D. Schultheiß, Permeation barrier for lightweight liquid hydrogen tanks. PhD thesis,Univ, Augsburg, 2007.

[5] A. Lücken, T. Kut, H. Langkowski, S. Dickmann, and D. Schulz, “Concept analysisof an electrical fuel cell integration in modern aircraft,” in IEEE 4th InternationalConference on Clean Electrical Power (ICCEP 2013) : Alghero, Italy, 11 - 13 June2013, pp. 590–594, Piscataway and NJ: IEEE, 2013.

[6] T. S. Chow, “Testing software design modeled by finite-state machines,” IEEE Trans-actions on Software Engineering, vol. 4, no. 3, pp. 178–187, 1978.

[7] K. Jacoby and H. Layton, “Automation of program debugging,” in the 1961 16thACM national meeting (T. C. Rowan, ed.), pp. 123.201–123.204, 1961.

[8] C. Böhm and G. Jacopini, “Flow diagrams, turing machines and languages with onlytwo formation rules,” Communications of the ACM, vol. 9, no. 5, pp. 366–371, 1966.

[9] S. R. Vergilio, J. C. Maldonado, and M. Jino, “Infeasible paths in the context of dataflow based testing criteria: Identification, classification and prediction,” Journal ofthe Brazilian Computer Society, vol. 12, no. 1, pp. 73–88, 2006.

[10] J. C. Miller and C. J. Maloney, “Systematic mistake analysis of digital computerprograms,” Communications of the ACM, vol. 6, no. 2, pp. 58–63, 1963.

[11] R. Alur, C. Courcoubetis, M. Halbwachs, T. A. Henzinger, P.-H. Ho, X. Nicollin,A. Olivero, J. Sifakis, and S. Yovine, “The algorithmic analysis of hybrid systems,”Theoretical computer science : theory of natural computing, vol. 138, no. 1, pp. 3–34,1995.

[12] T. A. Henzinger, “The theory of hybrid automata,” in Logic in Computer Science,1996. LICS ’96. Proceedings., Eleventh Annual IEEE Symposium on, pp. 278–292,1996.

55

| Bibliography

[13] T. A. Henzinger, P.-H. Ho, and H. Wong-Toi, “Algorithmic analysis of nonlinearhybrid systems,” IEEE transactions on automatic control : a publication of theControl Systems Society, vol. 43, no. 4, pp. 540–554, 1998.

56

Hintergrund:

Schon seit mehreren Jahren besteht in der Luftfahrtforschung der Trend, Flugzeuge in ihrer

Lärm- und Abgasemission deutlich zu reduzieren. Ein Teil dieser Forschung beschäftigt sich

damit, Brennstoffzellen aufgrund ihrer positiven Eigenschaften in die Systemarchitektur von

Verkehrsflugzeugen zu integrieren.

Damit die Integration

einer Brennstoffzelle

sowohl ökonomisch als

auch ökologisch vorteil-

haft ist, muss diese

Technologie jedoch mul-

tifunktional an Bord ein-

gebunden werden. Die-

ses bedeutet, dass die

Brennstoffzelle nicht nur

als ein reiner Ersatz für

die Hilfsgasturbine zur

Versorgung des Flug-

zeuges mit elektrischer

Energie im Bodenbetrieb

betrachtet werden kann.

Vielmehr muss dieses

System zusätzliche Aufgaben an Bord eines Flugzeuges übernehmen, um so weitere Systeme zu

substituieren. Hierbei könnte die Brennstoffzelle auch für die Notenergieversorgung im Fehler-

fall verwendet werden. Zudem kann das Abgas der Brennstoffzelle getrocknet und in die Kraft-

stofftanks geleitet werden, um so die Entflammbarkeit des Gasgemisches in den Tanks zu redu-

zieren. Das bei der Trocknung abgespaltene Wasser könnte dazu beitragen, dass die Wassertanks

an Bord kleiner und somit auch leichter dimensioniert werden können.

Am Institut für Flugzeug-Systemtechnik werden Konzepte für die Systemautomation eines sol-

chen multifunktionalen Brennstoffzellensystems entwickelt. Diese beinhaltet die Regelung,

Steuerung und Überwachung aller Systemkomponenten basierend auf einer hierarchischen

Struktur. Das Zusammenwirken aller Module der Automation mit den beteiligten Systemen und

Komponenten bis hin zum Cockpit wird in einer virtuellen Integrationsumgebung getestet. Für

die Simulation von Testszenarien wurde ein Gesamtsystemmodell aufgebaut, das neben den

Brennstoffzellenstacks die Luft- und Wasserstoffversorgung sowie die Kühlung der Brennstoff-

zellen und das Inertisierungssystem enthält. Für die Visualisierung gesamter Flugmissionen wird

zudem ein virtuelles Flugzeug verwendet, in das ein multifunktionales Brennstoffzellensystem

integriert ist.

Bachelorarbeit

für Sascha Lehmann

Analyse der Testabdeckung einer zustandsbasierten

Systemautomation für ein Brennstoffzellensystem

in Verkehrsflugzeugen

Aufgabenstellung:

Im Fokus dieser Bachelorarbeit steht die Analyse der Testabdeckung, die für eine zustandsba-

sierte Systemautomation eines multifunktionalen Brennstoffzellensystems im Rahmen virtueller

Integrationstests erreicht werden kann. Die einzelnen Module der Systemautomation sind in Mat-

lab/Stateflow implementiert und kommunizieren auf Basis eines hierarchischen Konzepts. Durch

die Simulation verschiedener Testszenarien kann die Systemautomation mit einer Abfolge von

Testvektoren stimuliert werden. Das Ziel dieser Bachelorarbeit besteht darin, eine Methode zu

implementieren, mit der die Abdeckung des zustandsbasierten Teils der Systemautomation durch

die verwendeten Testszenarien analysiert und bewertet werden kann.

Im Einzelnen sind folgende Schritte zu bearbeiten:

1. Literaturrecherche und Einarbeitung in die Theorie und Methoden der Analyse der Test-

abdeckung

2. Einarbeitung in ein vorhandenes Matlab/Stateflow Modell der Systemautomation eines

multifunktionalen Brennstoffzellensystems

3. Auswahl einer Methode für die Analyse der Testabdeckung der Systemautomation im

Rahmen virtueller Integrationstests

4. Implementierung der Analyse-Methode

5. Durchführung der Analyse

6. Bewertung der Testabdeckung

7. Zusammenfassung und Dokumentation der Ergebnisse

Über die Ergebnisse der Arbeit ist im Rahmen eines Kolloquiums zu berichten.

Alle im Rahmen der Arbeit gewonnenen Erkenntnisse und Ergebnisse sowie zugänglich gemach-

te Informationen sind vertraulich zu behandeln; sie bleiben Eigentum des Instituts für Flugzeug-

Systemtechnik und des Instituts für Softwaresysteme und dürfen ohne deren Erlaubnis weder

Dritten zugänglich gemacht, noch in irgendeiner Form gewerblich genutzt werden.

Eigenständigkeitserklärung

Hiermit versichere ich, dass die vorliegende Bachelorarbeit selbstständig verfasst und keine

weiteren Hilfsmittel als die entsprechend angegebenen benutzt wurden. Diejenigen

Textabschnitte, die nach ihrem Wortlaut oder Sinn einem anderen Werk entnommen wurden,

sind mit entsprechenden Quellenangaben referenziert.

Datum / Unterschrift

Test-Case Coverage Analysis of State-Based Automata ... - TUHH · Test-Case Coverage Analysis of State-Based Automata for Fuel-Cell Systems in Commercial Aircraft July 11, 2014 supervised

Documents