David J. Murray-Smith, School of Engineering, Rankine Building, University of Glasgow, Scotland, U.K. Keynote Tutorial: Methods for Testing and Validation.

David J. Murray-Smith,

School of Engineering, Rankine Building,

University of Glasgow,

Scotland, U.K.

Keynote Tutorial: Methods for Testing and Validation of Simulation Models for Engineering Applications

E-mail: [email protected]

Keynote Tutorial: Model quality, testing and validation 0

The presentation

1. Introduction

2. Model testing and validation issuesUncertainties, modelling errors, testing, verification and validation.

Model quality measures and model improvements.

3. Model managementLibraries of sub-models and development of generic models.

Model version control and model documentation.

4. Educational implications

5. ExamplesExternal validation and upgrading of linear and nonlinear

helicopter models for handling qualities and flight control studies.

6. Discussion and Conclusions


• To provide a basis for design.• To assist in human decision making.• To explain complex system behaviour.• For use within fault detection systems etc..• For simulator development (e.g. for operator training

or for engineering development applications).

The purpose of models in engineering


1. Conceptual models allow investigation of performance limitations at an early stage of the design process for normal and abnormal operating conditions.

2. Fully developed and proven models can provide information about key parameter sensitivities and inter-dependencies – useful for design decisions and optimisation.

3. Full models allow virtual prototypes to be created before any hardware prototype is available so identifying necessary design alterations at an early stage, avoiding expensive changes later on.

Design benefits with fit-for-purpose models


Models in design: some pointers

“Improved modelling of physical and manufacturing processes will improve our ability to predict the behaviour, costs and risks of future systems, and dramatically reduce the development timescale”. UK Office of Science and Technology, Technology Foresight Panel Report (1995).

“Verification, validation and accreditation” (VV&A). “Smart procurement” methods. Concept of “the model as a specification” (as promoted by

T.S. Ericsen, US Office of Naval Research).


Level of model quality necessary is of critical importance for any given application ........ an inappropriate model is less than useless as it may delay the project and lead to cost escalation

Balance needs to be found between model accuracy and the cost of developing the model.

Rigorous consideration of model quality is most common in applications involving safety critical issues (e.g. aeronautical engineering, automotive engineering etc.)

Levels of model quality


Current Problems with Models

Models are often used with very little systematic testing.

Model documentation is often minimal and is not recognised as a vital part of the model development process.

In many organisations models are passed from project to project and end up being used in ways that were never intended by the original developer of the model.

Current problems with models


Example: Hydro-turbine system modelling

.

Figure from Bryce, G.W., Foord, T.R., Murray-Smith , D.J. and Agnew, P., ‘Hybrid simulation of water turbine governors’, Simulation Councils Proceedings, 6(1), 35-44, 1976


Photographs: © D. J. Murray-Smith


Pipeline geometry



First simplification



Model comparisons(((Real-time approximation: continuous line Non-real-time model: dashed line)

Figure from Bryce, Fo Murray-Smith and Agnew, Simulation Councils Proceedings, Vol.6, No. 1, 35-44.



Second approximation F



Model comparisons for second approximation

( Real-time approximation: continuous line Non-real-time model: dashed line)



Site test and simulation comparisons



Model predictions


a) Model quality, uncertainties and modelling errors. b) Testing, verification and validation of models.

c) External validation methods. d) Model quality measures in external validation.

Part 2: Issues of model testing and validation


Tests only deal with a small number of cases. General statements about validity are impossible. All testing must be carried out in the context of the

application and especially the precise range of operating conditions for that application.

Should start from a well-understood case, even if much simplified; then move incrementally to testing for less certain situations for that application.

The more complex the model the harder the problem of quality assessment becomes: measures of model performance become harder to define and visualisation becomes more difficult.

Testing of models: fitness-for-purpose


Establishing the useful range of the dynamic model for a specific application.

Estimating the limits of accuracy of the model (usually both for steady-state and transient conditions) in terms the magnitude of expected errors in model predictions.

Aspects of model quality


Sources of errors and uncertainties

• Incorrect modelling assumptions

• Errors in a priori information (e,g, model parameter values)

• Errors in numerical solutions of model equations

• Errors in experimental procedures and in the measurements used for model testing.


Internal verification and external validation

Internal Verification – establishing that a computer–based model is consistent with the underlying mathematical model.

i.e. “Is the simulation model right?” External Validation – demonstrating that a final

(nonlinear)) model is adequate for the intended application.

i.e. “Is it the right simulation model?”

Note that checks of identified linear models are sometimes referred to as “external verification”.


External validation

Need to distinguish between:

Functional validation: where model is assessed in terms of how well it mimics input-output behaviour of the real system.

Physical/Theoretical validation where the model is based on theory and intermediate variables in model and system are compared. Approximations and assumptions are investigated within this process.


Approaches to external validation

Holistic approaches (e.g. subjective opinion of an expert on the real system such as an operator).

Model component approaches (e.g. each sub-system tested independently and compared with corresponding components of the real system).


Methods for external validation

Methods involving direct comparisons of response data from model and real system.

Methods based on system identification and parameter estimation.

Methods involving parameter sensitivity analysis. Methods based on inverse models and inverse

simulations.

Whatever method is used, data for external validation must be appropriate for the intended application. Careful experimental design is essential.


Methods for system and model comparisons

Graphical comparisons Quantitative measures System identification methods Expert opinion

Different approaches can provide different kinds of insight.


Plots of simulated and measured responses against an independent variable (often time).

Plots of simulated values against the corresponding measured values (should be 45 degree line).

Different graphical methods may emphasise different aspects of the simulation model performance so there are possible benefits from combining different approaches.

Methods of system/model comparison: some examples


Typical time history comparisons


Another time history comparison: a multi-input multi-output case

The original version of this figure was published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

BO-105 helicopter flight test data, DLR SIMH simulation model


Examples of simple cost functions

Mean absolute error

Mean absolute percent error

Weighted error


Theil’s Inequality Coefficient (TIC)


Use of quantitative measures:an example


Polar diagram form of display

From Smith, M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and validation issues in a generic model of electro-optic sensor systems ‘ J. Defense Modeling and Simulation, 4(1), 17-27, 2007


Issues of identifiability in external validation

Test input design is important since inputs must excite the system and model over an appropriate range of frequencies and amplitudes.

The concept of identifiability is central to issues of test input design external validation and is thus very important for external validation.

Structural identifiability relates to situations where a model may have an excess of parameters so that some specific parameters cannot be estimated uniquely for any possible experimental design (e.g. Bellman, R. and Åström, K.J., Mathematical Biosciences, 7, 329-339, 1970). Structural identifiability is also important for external validation.


Numerical unidentifiability arises when a structurally identifiable model is being used with data that is inappropriate for the application.

Numerical identifiability investigated from parameter information matrix M, the related dispersion matrix D and the parameter correlation matrix P. All depend on the sensitivity matrix X where:

Inputs may maximise the overall accuracy of all parameter estimates or may be chosen to maximise accuracy of specific parameter estimates.

Issues of numerical identifiability and test input design in model validation

11

1

jjii

ijij

mm

mp


Upgrading of simulation models

Following comparison of model and system behaviour usually need to analyse discrepancies and propose upgrades for model.

Changes must be evaluated systematically on a physical basis – with further iterations in the development cycle.

Parametric changes usually considered first, before structure . Sometimes possible to associate model deficiencies with specific

state variables model (e.g. correlation of output error with a state variable may help identify problem source).

Correlation of model errors with derivatives of state variables may suggest that a higher-order description would be more useful.

Optimisation tools (including evolutionary computing methods such as GA and GP) may be useful but should be used along with physical knowledge and understanding of the model.


Part 3: Model management a) Libraries of sub-models and generic models. b) Model version control and model documentation.


Generic and re-usable sub-models

Generally accepted that system design should be based on use of generic descriptions and re-usable sub-models.

Examples of the generic approach may be found in automotive engineering, gas turbines etc. Issues inevitably arise in the external validation of generic models – one approach is discussed in Smith, M.I., Murray-Smith, D.J. and Hickman, D., ‘Verification and validation issues in a generic model of electro-optic sensor systems’ J. Defense Modeling and Simulation, 4(1), 17-27, 2007


Model documentation and version control

Extra costs of creating good documentation should be more than balanced by the resulting re-usability of models. Version control processes should ensure that changes are fully documented.

Documentation and version control well developed in software engineering field. Same principles should be applied to the model development process.


Items for documentation

Purpose of model and the intended application. Full description of model and corresponding computer

code. List of all assumptions and approximations used. Details of all tests carried out on the real system to provide

information for model development. Details of the internal verification process. Details of the external validation process, with statements

about why model was accepted or rejected and information about usable range for model.


Part 4: Educational implications


Modelling and Simulation in Engineering Education

Engineering students encounter mathematical and computer-based modelling repeatedly in their university education.

Emphasis is most often on development of models from physical principles and on using models/simulations in place of experiments on real systems.

Modelling and simulation in engineering education


Modelling and Simulation in Engineering Education

Issues of model quality and fitness-for-purpose are seldom emphasised

in the teaching of modelling and simulation.

Model validation is neglected in education. The teaching of

system modelling and simulation should include much more on

model validation methods.

Model testing should become second nature for students.

Documentation, model re-use and libraries of models must be

given much more emphasis (especially in more advanced teaching).

Model quality and testing issues in engineering education


Part 5: Examples Drawn from external validation and upgrading of helicopter models for flight control system design.


Plant model limitations often impose serious performance limitations within control system, especially in systems with high-performance requirements.

Particularly important to have highly accurate plant models for the part of the frequency range close to the “cross-over” region in the frequency domain.

Model limitations in control


Examples from aircraft and helicopter flight control system design

• There are many well documented aircraft flight control examples illustrating problems of model quality and model limitations in integrated system design.

• Problems are often identified during initial testing of hardware. These lead to development of improved models and corresponding control design changes.

• The later in the design cycle these changes have to be made the more costly they are and the greater the delays to the project.


Helicopter model requirements

Must perform well over a defined range of frequencies.

Must perform well over a defined range of manoeuvre amplitudes.

For flight control design model must perform especially well close to open-loop gain and phase cross-over frequencies (as with control applications involving other types of system).

Photograph: © D. Murray-Smith


Nominal model: simulation and flight data compared for same test inputs. (Westland Lynx helicopter in 300 ft. quick-hop manoeuvre)

From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990.

SS

Common test inputs used for system identification and “external verification”of the identified model (BO-105 flight data)

The original versions of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

...........and flight testing


Simulation, identification and “external verification” results(BO-105 flight test data, DLR SIMH model)

The original version of these figures were published by the Advisory Group for Aerospace Research and Development, North Atlantic Treaty Organisation (AGARD/NATO) in AGARD Advisory Report 280 ‘Rotorcraft System Identification’, September 1991

Yv

Lv

Nv

Lp

NpL

r

Nr

L δ lat

N δ lat

L δ ped

N δ ped

2ζω0

Normalised HELISTAB values

Values estimated from flight data using system identification methods.

Estimated and theoretical parameter values of the identified model for lateral/directional characteristics(SA-330 Puma helicopter flight test data : 80 knots straight and level flight)

Assessment of a theoretical nonlinear model for a Puma helicopter

Parameter values for two different flight conditions, showing parametric trends from a physically-based nonlinear simulation model (HELISTAB) and the trends in estimates from flight tests involving system identification of separate linearised models for each flight condition.

From Bradley, R., Padfield, G.D., Murray-Smith, D.J. and Thomson, D.G., ‘Validation of helicopter mathematical models’, Transactions of the Institute of Measurement and Control’, 12(4), 186-196, 1990.


Good vehicle models are essential for design of high-bandwidth full-authority active flight control systems.

Published examples show that the achievable performance of flight control systems have, in some cases, been overestimated in initial design studies, usually because of limitations of the flight mechanics model of the vehicle (see e.g. Tischler, M. B. Advances in Aircraft Flight Control Systems, Taylor & Francis, London 1996).

Although control systems can be made robust to compensate for poor accuracy this is usually at the expense of performance. Improved modelling procedures and improved models can offer significant benefits. Otherwise, problems may not be apparent until the flight testing stage, leading to costly redesign, extended flight test programmes and delays in certification.

Summary of model quality issues for helicopter flight control system design


Part 5: Discussion and conclusions


Discussion: Model quality in design

The more demanding the design specification the more important is the fitness-for-purpose of models used in design.

More attention needs to be given to the external validation of models for the specific application in question.

a) Establishing the useful range of the model.

b) Estimating the accuracy of the model within that range.

Model validation is part of the model building process and external validation techniques need to be applied repeatedly.

Models should be retained, maintained and updated throughout the whole life-cycle of the system that they represent.


• More attention should be given to the fitness-for-purpose of models used in design, especially for demanding applications.

• Current methods for external validation are time-consuming and difficult to apply in many situations. More effort should be devoted to improving validation methods.

• Techniques of version control and rigorous documentation should borrowed from software engineering and applied to the model development process. Re-use of proven models should be made easier and more comprehensive model documentation should be available within model libraries.

• Issues of model quality should be given far more attention within engineering education.

Recommendations


Conclusions

With suitable structure and parameter values and rigorous external validation models that are fit-for-the-purpose of a given application can be developed (iteratively of course).

Good model management can reduce the cost of design and development.

There are no quick answers: a systematic approach is essential, moving incrementally from well-understood cases to less well known situations.

Educational and cultural changes are needed as well as improved management of the modelling, simulation and design processes within most organisations.


David J. Murray-Smith, School of Engineering, Rankine Building, University of Glasgow, Scotland, U.K. Keynote Tutorial: Methods for Testing and Validation.

Documents

model testing

model documentation

model quality measures

model improvements

inappropriate model

model accuracy

models models

model development process