Mdd test qa_test2014_bryan_bakker

MDD and its impact on

testing...

A nanotech case study

Bryan Bakker

bryan.bakker@sioux.eu

@Bryan_Bakker

Contents

Intro

MDD

MDD and Testing

Vacuum redesign

Experiences

© Sioux 2014 | Confidential | 2

About Bryan Bakker

Test Expert

Certifications: ISTQB, TMap, Prince2

Member of ISTQB Expert Level on Test Automation

Tutor of several test related courses

Domains: medical systems, professional security

systems, semi-industry, electron microscopy

Specialties: test automation, integration testing, design

for testability, reliability testing

© Sioux 2014 | Confidential | 3

About Sioux

HERENTALS NEDERWEERT

EINDHOVEN

UTRECHT MOSCOW

© Sioux 2014 | Confidential | 4

DANANG

About FEI

World leader in electron

microscopes

Light microscope: 1000x

200nm (limited by the

wavelength of light)

Electron microscope: 4Mx

0.05 nm

Nm = a billionth of a meter (10-9

meter)

© Sioux 2014 | Confidential | 5

About FEI

© Sioux 2014 | Confidential | 6 2 nm2 nm 0. 5 nm0. 5 nm

Atomic structur of Ge

(Germanium). Distance

is 0.5 nm

Breast cancer cell.

Magnification 5.000x Salmonella bacteria.

Magnification 80.000x

What is a model?

These models are not meant:

© Sioux 2014 | Confidential | 7

What is a model?

Simplification of the reality

By definition wrong, but can be useful

Do not try to put everything in the model

© Sioux 2014 | Confidential | 8

Traditional model

© Sioux 2014 | Confidential | 9

Traditional

Model

Source

code

Is the model

correct?

Is the

source code

correct?

ASD (Analytical Software

Design)

Model Driven Development technology

Component Based Development

Models are verified mathematically at design time

Extensive model checker:

Deadlock

Live-lock

Starvation

Race conditions, etc

© Sioux 2014 | Confidential | 10

Model

© Sioux 2014 | Confidential | 11

Model:

- Precise

- Complete

- Correct

Model verification

© Sioux 2014 | Confidential | 12

Model:

- Precise

- Complete

- Correct

Formal model

and verification

Generate

formal model

Design

errors

Code generation

© Sioux 2014 | Confidential | 13

Model:

- Precise

- Complete

- Correct

Formal model

and verification

Source code:

- Java

- MISCRA C

- C++

- C#

Generate

formal model

Design

errors

Generate defect

free source code

from verified model

Guaranteed

equivalence

What is covered?

Interface model:

Specification of interfaces of components

“What”

Design model:

Implementation of interfaces

Internal implementation

“How”

Also the outside interfaces are specified

Interfaces to not-modelled components

© Sioux 2014 | Confidential | 14

Handwritten vs. generated

© Sioux 2014 | Confidential | 15

Interface

InterfaceInterface

Interface

Interface Interface

Handwritten vs. generated

© Sioux 2014 | Confidential | 16

Interface

Interface Interface

Handwritten code

Generated

code

Part of the model

Scope of model

Typically used to model behaviour described in:

State machines

Sequence diagrams

Algorithms

© Sioux 2014 | Confidential | 17

State machines

© Sioux 2014 | Confidential | 18

Large (encapsulated) state machines can be modelled

Sequence diagrams

© Sioux 2014 | Confidential | 19

Scope of model

Typically used to model behaviour described in:

State machines

Sequence diagrams

Algorithms

What is modelled?

All normal functionality (good weather)

All exceptional functionality (bad weather)

All illegal behaviour

© Sioux 2014 | Confidential | 20

MDD and Testing

© Sioux 2014 | Confidential | 21

Implementation

Component TestDesign

Integration TestArchitecture

System

RequirementsSystem Test

User

RequirementsAcceptance Test

Wish, need

Release

Formal

methods

MDD and Testing

Formal methods typically:

Applied at Design level

Only in Software

For part of the software

Not for legacy and 3rd party software

© Sioux 2014 | Confidential | 22

Implementation

Component TestDesign

Integration TestArchitecture

System

RequirementsSystem Test

User

RequirementsAcceptance Test

Wish, need

Release

Formal

methods

MDD and Testing

Part of the component tests are

not needed anymore

(Software Engineer)

Test activities of Test Engineer:

From integration level and up

Are still needed

But:

MDD has impact on quality of deliverables

© Sioux 2014 | Confidential | 23

Implementation

Component TestDesign

Integration TestArchitecture

System

RequirementsSystem Test

User

RequirementsAcceptance Test

Wish, need

Release

Formal

methods

Microscope Vacuum redesign

Vacuum software + firmware redesign

ASD has been used

Functionality tested manually (by experts)

Vacuum is critical part of system

verify reliability separately

AutoIT used for test automation

Freeware, mainly GUI tool

Only scripting used (no recording)

© Sioux 2014 | Confidential | 24

Test automation

First steps: quick and dirty

Fast feedback

Automation via

GUI

© Sioux 2014 | Confidential | 25

Test automation

© Sioux 2014 | Confidential | 26

UI-Controls and graphics used for

Test actions

Test verifications

Reliability tests

System tests on vacuum system

Executed for long time (days)

Not only software but whole system is tested

Test automation

© Sioux 2014 | Confidential | 27

Testing experiences

Testing is still necessary

Integration (with other SW, with HW)

System testing

Model incorrect Rubbish in, rubbish out

New insights, so model will change

Functionality must still be verified (and errors are found)

Programming errors or integration issues:

In handwritten code

On boundary between generated and handwritten

code (hard to analyse)

Not in generated code!

© Sioux 2014 | Confidential | 28

Testing experiences

Reliability of generated code: very high!

But crashes can still occur... (but not in generated code)

Reliability tests

HW failures/wear detected much earlier

Days instead of months

Integration and system test

not hampered by reliability issues

big improvement, less cycles

© Sioux 2014 | Confidential | 29

Testing experiences

Models are much easier to review, understand and adapt

than code

Design discussions are held early in project/sprint (as it

should be!) defects prevented!

Refactoring (relatively) simple and reliable

(less regression)

Fits well in agile approach

© Sioux 2014 | Confidential | 30

Product Risk Analysis

© Sioux 2014 | Confidential | 31

3

3

15

15

Impact

L

ike

liho

od

Comp1

Comp2 Comp3

9

9

Product Risk Analysis

© Sioux 2014 | Confidential | 32

3

3

15

15

Impact

L

ike

liho

od

Comp1

Comp2 Comp3

9

9

III I

IV II

Could test Must test

“Won´t test” Should test

Product Risk Analysis

© Sioux 2014 | Confidential | 33

3

3

15

15

Impact

L

ike

liho

od

Comp1

Comp2 Comp3

9

9

Product Risk Analysis

© Sioux 2014 | Confidential | 34

3

3

15

15

Impact

L

ike

liho

od

Comp1

Comp2 Comp3

9

9

MD

D

Conclusions

Component tests (white box) could be skipped

Internal working is correct

Although functionality can be incorrect

Integration and system testing a lot smoother

Less interface issues

Less reliability issues

Reliability testing reliability measurement

Integration and system test more efficient (less cycles)

Architecture and design takes more effort

More re-designs with ASD are planned

© Sioux 2014 | Confidential | 35

Questions

© Sioux 2014 | Confidential | 36

37 © Sioux 2014 | Confidential |

www.sioux.eu

bryan.bakker@sioux.eu

+31 (0)40 26 77 100