Software engineering and standardisation section (TEC-EME) 1 Automatic test case generation from a formal model LTG technology evaluation based on a B.

Software engineering and standardisation section (TEC-EME)

1Automatic test case generationfrom a formal model

LTG technology evaluation based ona B model of

a part of ATV onboard software

Julien ORSAT – Stagiaire from the University of Franche-Comté (France)

08/03 – 23/07/2004

Supervisor: Philippe CHEVALLEY

2Outline

B modelling for Leirios Test Generator

ATV software case study: Gyra functional unit

Gyra B formal model

LTG automatic test case generation

Conclusion

3Outline

B modelling for Leirios Test Generator B formal notation LTG requirement traceability LTG syntax restrictions

ATV software case study: Gyra functional Unit

Gyra B formal model

LTG automatic test case generation

Conclusion

4B formal notation

Used as input language for LTG Other supported input format : Statecharts/Statemate, UML

Part of the B method Code generation from the model Proof obligations to ensure correctness Several refinement steps from the model to the target code Created in the 90’s by J-R Abrial Applied in railway transports and smart card industry

LTG input: 1st step of the B method (“abstract machine”)

Text format Statecharts and UML are graphically used

5Some B concepts

Sets ; could be considered as base types GYRO_ID = {G1, G2, G3, G4} GYRO_MODE = {COARSE, FINE}

Relations ; to define sets of ordered pairs gyra_mode : GYRO_ID --> GYRO_MODE

Substitutions ; to modify variable values gyra_mode := gyra_mode <+ {(G3 |-> FINE)}

Operations ; to describe system behaviour Used by LTG to compute test cases

6B operation example

A B operation is considered atomic <+ is the relation overriding operator

cmd_report <-- setGyroMode(gyro, mode) =

PRE /* precondition */

gyro : GYRO_ID &

mode : GYRO_MODE & …

THEN /* body */

IF condition THEN

cmd_report := REJECTED

ELSE

cmd_report := ACCEPTED ||

gyra_mode := gyra_mode <+ { (gyro |-> mode) }

END

END

ProfileB reserved wordCommentaryBoolean expressionSubstitution

7LTG requirement traceability

Provided by LTG not part of B notation

Uses tags in the model Based on B commentaries IF (...) THEN

cmd_report := REJECTED /*@REQ: R52 @*/ELSE ...

Allows automatic traceability report generation Test case -> covered requirements Requirement -> covering test cases

Requirement coverage criteriafor generated test case evaluation

8LTG restrictions to the B notation

Syntax Real numbers

Difficulty to compute expected results

Sequences list-like data structures

“Such as” set description { element | predicates }

Single B machine only No machine inclusion, promotion …

9Outline

B modelling for Leirios Test Generator

ATV software case study: Gyra functional unit

Gyra Functional Unit ATV software structure Case study choice explanation

Gyra B formal model

LTG automatic test case generation

Conclusion

10Gyra Functional Unit

High level services to usea 4 GYRoscope Assembly (GYRA)

Hardware control Monitoring algorithms Processing algorithms

Designed and developed by EADS 89 requirements 13 Ada83 packages 7 000 code lines

Managed by an internalfinite state machine (FSM)

11ATV software structure

Gyra Functional Unit is part of software layer 3

LN 2 library

LN 1 library

LN 3:…

LN 3:Gyra FU

Device#n

Hardware

Software layers

28 FUs

… … …

LN 3:…

Device#1

Hardware interface

Communications

Applications

12Case study choice explanation

ATV spacecraft Uses current space software techniques

(software under development)

Gyra application Easy access to Gyra specification documents

Focus on the FSM Heavily used in ATV: 28 FSMs, 989 transitions Gyra FSM has an average size (38 transitions) FSMs are difficult/expensive to test

“Transition testing is one of the most, if not the most, demanding tasks of the product validation” EADS test plan

Would be valuable to automate test case generation

13Outline

B modelling for Leirios Test Generator

ATV software case study: Gyra functional Unit

Gyra B formal model Modelled Finite State Machine Requirement traceability

Automatic test case generation

Conclusion

14Modelled Finite State Machine

OFF_N

ON_N

ON_4ON_1

ON_2

ON_13

ON_14

ON_3

ON_24

ON_12

ON_23

ON_34

Transitions are activated by commands Redundant commands are accepted (requirement 16241) FSM modes are groups of FSM states (OFF mode, ON mode)

15Modelled FSM key figures

12 states

22 graphically represented transitions

+ 14 different reflexive transitions would be 102 more transitions

on the picture distinct from

command redundancy transitions

750 line model

NB: separated model for Survival modes(3 states, 2 transitions – 150 lines)

ON_x (11 states)

Example: GYRA_TR_MODE

According to EADS:1 transition

Modelling choice:11 different

transitions

ON_x (11 states)ON_x (11 states)

16Requirement traceability

Traceable Gyra requirements: 43 / 49

Non traceable requirements (6) : Initial values of variables (3),

No LTG tag for INITIALISATION part of the model High level requirements, refined by others (2) FSM mode shall be computed from FSM state (1)

That is to say not stored in memory – code inspection is required

17Outline

B modelling for Leirios Test Generator

ATV software case study: Gyra functional unit

Gyra B formal model

LTG automatic test case generation Test generation criteria Generated test cases Ada test program generation

Conclusion

18Test generation criteria

Model coverage criteria: 4 possibilities Decision coverage (DC) Condition/Decision coverage (C/DC) Modified condition/Decision coverage (MC/DC) Multiple condition coverage (MCC)

An LTG test case is made of 1. Preambule 2. Body + Identification 3. Postambule

With Gyra, test programs may end in any FSM state no need for a postambule

Softwarecriticality

19Generated test cases

179 functional abstract test cases generated = 179 operation activation sequences XML format (HTML and RTF translators)

Comparison to EADS test plan Chaining of several tests (8 large test sequences)

difficult to compare

Traceable requirements covered by LTG: 37/43 Difficulty with generic requirements (5)

Multiple tags for one requirement Were those requirements fully covered ?

1 requirement was not covered: some behaviours, related to 1 B operation, were not reached different generation criteria should solve the problem.

HTML example

20EADS test plan coverage

Gyra FSM state and transition coverage

OFF_N

ON_N

ON_4ON_1

ON_2

ON_13

ON_14

ON_3

ON_24

ON_12

ON_23

ON_34

21Coverage comparison

State coverage EADS test plan: 9/12 LTG: 12/12

Graphically represented transition coverage

EADS test plan: 14/22 LTG : 22/22

Reflexive transition coverage EADS: 13/14, tested on one state LTG: 13/14, tested on every state

uncovered transition is relatedto uncovered B operation

ON_x (11 states)

Example: GYRA_TR_MODE

In EADS test plan:1 test case

Our choice:11 test cases

22

Required to generate test programs from test cases:

Tagged test program pattern Relation mapping between B operations and Ada services

Test program generation on a simple model Test program example

Not enough time to generate test programs

Ada test program generation

23Outline

B modelling for Leirios Test Generator

ATV software case study: Gyra functional unit

Gyra B formal model

LTG automatic test case generation

Conclusion

24

179 Test Cases

LTG Test CaseGenerator

Test CaseGeneration

Test GenerationCriteria

LTGeXecutable

Script Builder

Test Execution Environment Test Driver Generation

Test DriverSchemas

49 Gyra functional requirements

B Model (750 lines)FunctionalModeling

LTGModel Animator

ModelValidation

LEIRIOS Test Generator™

done

to do

24

25Conclusion

Technology evaluation: B notation Particularly suited to model finite state machines Difficulty to model real time aspects like scheduling Added requirement traceability Some restrictions on the notation

Automatically generated test cases on the case-study 15 hour computation on a desktop PC, for 179 test cases Coverage is similar or better compared to EADS test plan

LTG product stability, documentation quality to be raised

Transition from research prototypeto commercial product is in progress

26Perspectives

Short-term perspectives Generate and execute Gyra FSM test programs

2 possibilities: by Estec or by EADS Structural coverage study Fault injection in the Gyra implementation to evaluate test

efficiency Generate tests for Gyra algorithms

Long-term perspectives LTG input format comparison for space projects

B notation Statecharts UML

Software engineering and standardisation section (TEC-EME)

27Automatic test case generationfrom a formal model

LTG technology evaluation based ona B model of

a part of ATV onboard software

Julien ORSAT – Stagiaire from the University of Franche-Comté (France)

08/03 – 23/07/2004

Supervisor: Philippe CHEVALLEY