Bart Knaack - The Truth About Model-Based Quality Improvements

© Logica 2008. All rights reserved

The Truth about Model Based Improvements

Bart Knaack

Agenda

• Introduction

• The future of testing

• What is a model?

• Different ways of Model Based Quality Improvement

• Conclusions

• What next

Bart Knaack

• Bart Knaack, Senior Test Advisor, Logica, The Netherlands

• 15 years experience in IT, of which 12 in testing

• Father of 2 kids (age 6 and 8)

[email protected]

The future of testing

• Crystal ball called Academia

• The past as the mirror of the future

• Trends and threats

But:

• Testing is not quality improvement

• Testing is not LEAN

What is a model?

A model is an abstraction or conceptual object used in the creation of a predictive formula (wikipedia)

A model in science is a physical, mathematical, or logical representation of a system of entities, phenomena, or processes.

Abstract representation of the reality focussing on a limited set of aspects

Programs are models too!!

Model for this presentation

Behavioral models: Models that somehow define the (intended) behavior of a system

•UML models•Finite State Machines•Algebraic models•Graphical model•Textual model

Model Based Quality Improvement

No. 725 March 2015 Testing a Virtual Juggling Program

Different ways of Model Based Quality Improvement?

• Model Based testing

–Online

–Offline

• Model Metrics and Code Conformance

• Model checking the models

• Applying Model checking techniques.

Model Based testing (Online)

• Definition: Directly coupling the test generation model to the SUT and directly executing the tests.

• Project: IRIS DOMAIN: Insurances

SUT: Actuarial calculation engine.

Torx modelling language for modelling the formulae.

Result: 5 Additional errors were found of which 3 were significant. Development and test-time for upcoming changes and fixes was reduced by 50%.

Model Based testing (Online)

• Advantages:

–Quick (re)testing of systems

• Disadvantages:

–Model SUT adaptor needs to be set up.

–Model errors need to be taken into account

–Not all systems and tests are suitable for this approach.

Model Based testing (Offline)

• Definition: Tests are generated by a Testgenerator and executed manually on the SUT.

• Project: CCBS Domain: Telecom.

SUT: 5ESS Telephony switch, new feature

Result: 10 features Modeled.1200 Tests designed of which 75% specific on feature interactions. 20% tests failed due to modelling errors.

Model Based testing (Offline)

•Advantages: – Quick adaption of testcases

– Models can be used for rapid prototyping.

– Quick reponse on Changes

•Disadvantages: – Most methods do not support the selection of testcases.

– Testexecution still manual

Model Metrics and code conformance

• Definition: Using Metrics to determine when models are good enough.

• Project:Siu Wai Tang Domain: Banking and Finance

SUT: Electronic Payments System.

Result: Improved Insight in elevated problem areas; redefinition of 3 models based on this insight.

Model Metrics

•Advantages: – Early stage warnings on model quality.

– Multidimensional view on models.

•Disadvantages: – No hard figures to adhere to.

– Statistical co-relation instead of direct cause-and-effect.

Model checking the models

• Definition: Requirements checking by checking the models that form the requirements.

• Project: Bos project Domain: Industry

SUT: ‘Waterkering’ (lock) protocols.

Result: Error free software (after 20 years of operation).

What is Model Checking?

•Statespace search

•Validation of a negative scenario.

•Structural traversal.

Model checking the models

•Advantages:– Finding CONCEPTUAL errors at an early stage.

– Finding DIFFERENT types of errors

•Disadvantages:– Model needs to be implemented in a formal manner

– Model checker needs to be available for the used paradigm.

Applying Model checking techniques.

• Definition: CODE is used model. Model checking techniques used for faultfinding

• Project: FeaVer Domain: Telecom

SUT: transaction based system (V5 interface)

5000 lines of code 1000 installments over 10 years.

Result: 35 bugs found (25 race conditions)

Applying Model checking techniques.

•Advantages:– Quick (re)test functionality

– Model errors are REAL errors

– Errors are found that are next to impossible to find using ‘traditional’ methods.

•Disadvantages: – Not all languages are supported by modelcheckers.

– Stubs and drivers need to be implemented to FEED the system.

Conclusions

• Modelling by itself can be quality improving.

• A lot can be gained from modelling and model improvement initiatives.

• Model Based Testing can help in t ackling the problems in complex systems.

BUT:

• New skills

• Academic image

• No standards

What next ?

• Integrated Toolsupport

• Simplify

• Educate

• Not included: Model Based development.