Fundamental concepts and terminology of Verification and
Validation
Fundamental concepts and terminology of Verification and
ValidationVerification is the process that checks whether
mathematical model was implemented accurately enough.Validation is
the process that checks that simulation is close enough to
reality.This terminology is not universally accepted.1979 figure
from Oberkampf and Roy
In order to simulate reality by some calculation, we usually go
through two stages. First we postulate a conceptual model of
reality, which usually is a mathematical model consisting of a set
of algebraic equations and/or differential equations and/or
integral equations.
We then implement the solution of these equations, sometimes by
hand, more often by writing software that will perform the
solution. These two processes are error prone, so we need to check
that they were executed properly.
The process of checking that the mathematical model is being
solved accurately enough is called verification in our community.
The process of comparing the solution to test measurements in order
to check if we simulate reality closely enough is called
validation.
These terminology is not universally accepted, and is not even
well known to most engineers and scientists. However, it is in the
process of being established.1Paper Helicopter model
qualificationWhen dropped, paper helicopter starts autorotating,
which slows down its fall, like a helicopter when it loses
power.Has very simple math model: Newtons second law
F=maF=Weight-drag
Differential equation
.
2VerificationIn order to verify the computational model and its
implementation (e.g. software) we need to test it against
analytical solutions or special cases where the solution of the
differential equations is known.For the paper helicopter we will
first demonstrate with analytical solution, and then with a
numerical solution..
Analytical solution of paper helicopter velocity.
4Verifying the analytical solutionWe start by checking that at
t=0 the velocity is zero at that at infinity the velocity goes to
Vs. Then we substitute solution into the differential equation to
check that it is satisfied:
Top hat questionWhat conditions can we apply to check if the
expression for dV/dt is correct?We know its value at t=0.We know
its value at t=infinity.Both.Neither.Numerical solutionNumerical
solution may allow us to relax some of our assumptions, such as
constant drag coefficient (may be function of speed).The simplest
way to integrate a differential equation is with Euler explicit
algorithm with constant time step
Can get also height without the challenge of analytical
integrationNow need to implement in software.Will use Matlab and
verify against analytical solution.
Matlab codefunction [v,va,h,t]=paper_copter(v0,c,g,T,n)%
Calculates trajectory of paper helicopter both % analytically and
by Euler explicit integration.dt=T/n;
vs=sqrt(g/c);t=linspace(0,T,n+1);va=vs*(1-exp(-2*vs*c*t))./(1+exp(-2*vs*c*t));v(1)=v0;h(1)=0;for
i=1:n v(i+1)=(g-c*v(i)^2)*dt+v(i); h(i+1)=v(i)*dt+h(i);end
g=32.2;c=1;T=2;v0=0.;
ButIf we check Matlab code for negative (upward) initial
speeds.g=32.2;c=1;T=0.5;
v0=-5.7;[v,va,h,t]=paper_copter(v0,c,g,T,10); v -5.7000 -5.7145
-5.7373 -5.7731 -5.8295 -5.9187 -6.0602 -6.2866 -6.6526 -7.2554
-8.2775It is flying into space!What error did I stumble on?
Code verificationIncludes numerical algorithm testing and
software quality assurance.However, there are several errors that
may create false impression that code is deficient.
Model validationThree stages according to Oberkampf and
Trucano
CalibrationModel often includes parameters that need to be
estimated from measurements.For paper helicopter we can measure air
density and area.We estimate drag coefficient from final speed
measurement.We may introduce calibration parameter of fake initial
condition to account for period when it falls without spinning.
Distinguishing between alternative modelsConsider the
possibility that a model for paper helicopter where drag is
proportional to speed instead of square of speed is suggested.That
is, Newtons second law isSolution
Top Hat questionWhich of the following is a valid difference
between linear and quadratic modelLinear model does not suffer from
sign problem when helicopter flies up.Linear model approaches
steady state faster.In linear model mass has larger effect on
steady state speed.
Equivalent calibrationWe will assume that we measure the steady
state speed, and use it for calibration.For quadratic model
For linear model
With errors not be easy to distinguish
Anti-optimization for model discriminationPrinciple: Large
errors are easier on debugging than small ones.Example: Comparing
finite element and finite strip calculations of buckling loads for
stiffened panels I observed 0.3% difference.I knew that the
difference should be at most 100 times smaller.I did not have a
chance of getting any attention from software developers for this
problem.I played with problem parameters until the difference grew
to 100%.It became clear that finite-element solution was wrong, and
it took developer one day to find the error.
Anti-optimization for composite failure criteriaVan Wamelen A.,
Haftka, R.T., and Johnson, E.R., "Optimal Design of Laminated
Specimens to Evaluate Competing Composite Failure Criteria,"
Proceedings American Society for Composites, 8th Technical
Conference, Technomic Publishing Company, 1993,
pp.1045-1055Designed laminate with ply angles that caused two
failure criteria that usually give close results to lead to factor
of 2 difference.Test validated Tsai-Wu failure criterion, against
Hart-Smith criterion.However, developer of competing criterion
claimed boundary failure.
Anti-optimization for drag modelsMaximize ratio of quadratic and
linear speeds
What can we do to improve experimental conditions for measuring
difference?