06/13/22 1 Summary of Experimental Uncertainty Assessment Methodology F. Stern, M. Muste, M-L. Beninati, F. Stern, M. Muste, M-L. Beninati, W.E. Eichinger W.E. Eichinger
Feb 12, 2016
04/22/23 1
Summary of Experimental Uncertainty Assessment
Methodology
F. Stern, M. Muste, M-L. Beninati, W.E. EichingerF. Stern, M. Muste, M-L. Beninati, W.E. Eichinger
Table of Contents IntroductionIntroduction Test Design PhilosophyTest Design Philosophy DefinitionsDefinitions Measurement Systems, Data-Reduction Equations, Measurement Systems, Data-Reduction Equations,
and Error Sourcesand Error Sources Uncertainty Propagation EquationUncertainty Propagation Equation Uncertainty Equations for Single and Multiple Uncertainty Equations for Single and Multiple
TestsTests Implementation & RecommendationsImplementation & Recommendations
Introduction
Experiments are an essential and integral tool for Experiments are an essential and integral tool for engineering and scienceengineering and science
Experimentation:Experimentation: procedure for testing or determination of a procedure for testing or determination of a truth, principle, or effecttruth, principle, or effect
True values are seldom known and experiments have errors True values are seldom known and experiments have errors due to instruments, data acquisition, data reduction, and due to instruments, data acquisition, data reduction, and environmental effectsenvironmental effects
Therefore, determination of truth requires estimates for Therefore, determination of truth requires estimates for experimental errors, i.e., uncertaintiesexperimental errors, i.e., uncertainties
Uncertainty estimates are imperative for risk assessments in Uncertainty estimates are imperative for risk assessments in design both when using data directly or in calibrating and/or design both when using data directly or in calibrating and/or validating simulation methodsvalidating simulation methods
Introduction
Uncertainty analysis (UA): rigorous methodology for Uncertainty analysis (UA): rigorous methodology for uncertainty assessment using statistical and engineering uncertainty assessment using statistical and engineering conceptsconcepts
ASME (1998) and AIAA (1999) standards are the most ASME (1998) and AIAA (1999) standards are the most recent updates of UA methodologies, which are recent updates of UA methodologies, which are internationally recognized internationally recognized
Presentation purpose: to provide summary of EFD UA Presentation purpose: to provide summary of EFD UA methodology accessible and suitable for student and methodology accessible and suitable for student and faculty use both in classroom and research laboratories faculty use both in classroom and research laboratories
Test design philosophy Purposes for experiments: Purposes for experiments:
Science & technologyScience & technology Research & developmentResearch & development Design, test, and product liability and acceptanceDesign, test, and product liability and acceptance InstructionInstruction
Type of tests:Type of tests: Small- scale laboratorySmall- scale laboratory Large-scale TT, WTLarge-scale TT, WT In-situ experimentsIn-situ experiments
Examples of fluids engineering tests:Examples of fluids engineering tests: Theoretical model formulationTheoretical model formulation Benchmark data for standardized testing and evaluation of facility biasesBenchmark data for standardized testing and evaluation of facility biases Simulation validationSimulation validation Instrumentation calibrationInstrumentation calibration Design optimization and analysisDesign optimization and analysis Product liability and acceptanceProduct liability and acceptance
Test design philosophy
Decisions on conducting experiments: governed Decisions on conducting experiments: governed by the ability of the expected test outcome to by the ability of the expected test outcome to achieve the test objectives within allowable achieve the test objectives within allowable uncertainties uncertainties
Integration of UA into all test phases should be a Integration of UA into all test phases should be a key part of entire experimental programkey part of entire experimental program Test descriptionTest description Determination of error sourcesDetermination of error sources Estimation of uncertaintyEstimation of uncertainty Documentation of the resultsDocumentation of the results
Test design philosophy D E F IN E P U R P O S E O F T E S T A N D
R E S U LT S U N C E R TA IN T Y R E Q U IR E M E N T S
U N C E R TA IN T YA C C E P TA B L E ?IM P R O V E M E N T
P O S S IB L E ?
D E T E R M IN E E R R O R S O U R C E SA F F E C T IN G R E S U LT S
Y E SN O
N O
Y E S Y E S
Y E S
N O
S E LE C T U N C E R TA IN T Y M E T H O D
E S T IM AT E E F F E C T O FT H E E R R O R S O N R E S U LT S
- M O D E L C O N F IG U R AT IO N S (S )- T E S T T E C H N IQ U E (S )- M E A S U R E M E N T S R E Q U IR E D- S P E C IF IC IN S T R U M E N TAT IO N- C O R R E C T IO N S T O B E A P P L IE D
- D E S IR E D PA R A M E T E R S (C , C , ... .)D R
D E S IG N T H E T E S T
- R E F E R E N C E C O N D IT IO N- P R E C IS IO N L IM IT- B IA S L IM IT- TO TA L U N C E R TA IN T Y
D O C U M E N T R E S U LT S
N O T E S T
C O N T IN U E T E S T
IM P L E M E N T TE S T
S O LV E P R O B LE M
R E S U LT SA C C E P TA B L E ?
M E A SU R E-M E N T
S Y S T EMP RO BLEM ?
N O
P U R P O S EA C H IE V E D ?
Y E S
N O
S TA R T T E S T
E S T IM AT EA C T U A L D ATAU N C E R TA IN T Y
Definitions
Accuracy:Accuracy: closeness of agreement between closeness of agreement between measured and true valuemeasured and true value
Error:Error: difference between measured and true valuedifference between measured and true value
Uncertainties (Uncertainties (UU):): estimate of errors in estimate of errors in measurements of individual variables measurements of individual variables XXii ( (UUxxii) or ) or results (results (UUrr) obtained by combining ) obtained by combining UUxxii
Estimates of Estimates of UU made at made at 95% confidence level95% confidence level
Definitions
= to tal e rro r = b ias e rro r = p recis ion error
M A G N ITU D E O F XF
RE
QU
EN
CY
OF
OC
CU
RR
EN
CE
X
X
X
tru e
X tru e
k
k+ 1 k
k+ 1
k+ 1XX k
( a ) tw o re a d in g s
(b ) in f in i te n u m b e r o f re ad in g s
Bias errorBias error : fixed, : fixed, systematicsystematic
Bias limitBias limit B: estimate B: estimate of of
Precision errorPrecision error : : randomrandom
Precision limit Precision limit PP: : estimate of estimate of
Total error:Total error:
Measurement systems, data reduction equations, & error sources Measurement systems for individual variables Measurement systems for individual variables XXii: :
instrumentation, data acquisition and reduction procedures, instrumentation, data acquisition and reduction procedures, and operational environment (laboratory, large-scale and operational environment (laboratory, large-scale facility, in situ) often including scale modelsfacility, in situ) often including scale models
Results expressed through data-reduction equationsResults expressed through data-reduction equationsrr = = rr((XX11, , XX22, , XX33,…, ,…, XXjj))
Estimates of errors are meaningful only when considered Estimates of errors are meaningful only when considered in the context of the process leading to the value of the in the context of the process leading to the value of the quantity under considerationquantity under consideration
Identification and quantification of error sources require Identification and quantification of error sources require considerations of: considerations of:
Steps used in the process to obtain the measurement of the quantitySteps used in the process to obtain the measurement of the quantity The environment in which the steps were accomplishedThe environment in which the steps were accomplished
Measurement systems and data reduction equations Block diagram showing elemental error sources, individual Block diagram showing elemental error sources, individual
measurement systems, measurement of individual variables, measurement systems, measurement of individual variables, data reduction equations, and experimental resultsdata reduction equations, and experimental results
r = r (X , X ,......, X ) 1 2 J
1 2 J
M EASUREM ENTOF INDIVIDUALVARIABLES
INDIVIDUALM EASUREM ENTSYSTEMS
ELEM ENTALERROR SOURCES
DATA REDUCTIONEQUATION
EXPERIM ENTALRESULT
XB , P
1
1 1
XB , P
2
2 2
XB , P
J
J J
rB , P
r r
Error sources Estimation assumptions: 95% confidence level, Estimation assumptions: 95% confidence level,
large-sample, statistical parent distributionlarge-sample, statistical parent distribution
MODEL FIDELITY ANDTEST SETUP:- As built geometry- Hydrodynamic deformation- Surface finish- Model positioning
TEST ENVIRONMENT:- Calibration versus test- Spatial/temporal variations o f the flow- Senso r insta llation/location- Wall inte rfe rence- F luid and fac ility conditions
DATA ACQUISITION ANDREDUC TION:- Sampling, filtering, and statistics- Curve fits- Calibrations
SIMULATION TECHNIQUES:- Instrumentation interference- Scale effects
CONTRIBUTIONSTO ESTIMATEDUNCERTAINITY
Uncertainty propagation equation Bias and precision errors in the measurement of Bias and precision errors in the measurement of XXii
propagate through the data reduction equation propagate through the data reduction equation rr = = rr((XX11, , XX22, , XX33,…, ,…, XXjj) resulting in bias and precision errors in the ) resulting in bias and precision errors in the experimental result experimental result rr
A small error (A small error (XiXi) in the measured variable leads to a small ) in the measured variable leads to a small error in the result (error in the result (rr) that can be approximated using ) that can be approximated using Taylor series expansion of Taylor series expansion of rr((XXii) about ) about rrtruetrue((XXii) as) as
The derivative is referred to as sensitivity coefficient. The The derivative is referred to as sensitivity coefficient. The larger the derivative/slope, the more sensitive the value of larger the derivative/slope, the more sensitive the value of the result is to a small error in a measured variablethe result is to a small error in a measured variable
iXitrueir dX
drXrXri
)()(
Uncertainty propagation equation Overview given for derivation of equation describing the error Overview given for derivation of equation describing the error
propagation with attention to assumptions and approximations used to propagation with attention to assumptions and approximations used to obtain final uncertainty equation applicable for single and multiple obtain final uncertainty equation applicable for single and multiple tests tests
Two variables, Two variables, kkth set of measurements (th set of measurements (xxkk, , yykk))
x k
k
k k k
ktrue true
true
k
k
k
k
k
k
r r x , y )= (
x
r
x
r
r
x
x
r
yy
y
y
y
),( yxrr
kk xxtruek xx
kk yytruek yy
2Ryyyrxx
xrrr truektruektruek
kkkkk yyyxxxtruekr rr
The total error in the The total error in the kkth determination of th determination of rr
yrxr yx ; sensitivity coefficientssensitivity coefficients
Uncertainty propagation equation We would like to know the distribution of We would like to know the distribution of r r (called the parent (called the parent
distribution) for a large number of determinations of the result distribution) for a large number of determinations of the result rr
A measure of the parent distribution is its variance defined as A measure of the parent distribution is its variance defined as
N
krN kr N 1
22 1lim
yxxxyxyxr yxyxyxyx 22 222222222
Substituting (1) into (2), taking the limit for N approaching infinity, Substituting (1) into (2), taking the limit for N approaching infinity, using definitions of variances similar to equation (2) for using definitions of variances similar to equation (2) for ’s and ’s and ’s’s and and their correlation, and assuming no correlated bias/precision errorstheir correlation, and assuming no correlated bias/precision errors
’’s in equation (3) are not known; estimates for them must be made s in equation (3) are not known; estimates for them must be made
Uncertainty propagation equation Defining
estimate for
estimates for the variances and covariances (correlated bias errors) of the bias error distributions
estimates for the variances and covariances ( correlated precision errors) of the precision error distributions
xyyxyyxxxyyxyyxxc SSSbbbu 22 222222222
cr KuU
equation (3) can be written asequation (3) can be written as
To obtain uncertainty To obtain uncertainty UUrr at a specified confidence level ( at a specified confidence level (CC%), a %), a
coverage factor (coverage factor (KK) must be used for ) must be used for uucc::
For normal distribution, For normal distribution, K K is theis the t t value from the Studentvalue from the Student t t distribution.distribution.
For For N N 10, 10, t t = 2 for 95% confidence level = 2 for 95% confidence level
2cu 2
r222 ,, xyyx bbb
xyywx SSS ,, 2
Valid for any type of error Valid for any type of error distributiondistribution
Uncertainty propagation equation Generalization for Generalization for JJ variables in a result variables in a result rr = = rr((XX11, , XX22, , XX33,…, ,…, XXjj))
J
i
J
i
J
ikikkiii
J
i
J
i
J
ikikkiiir PP BB = U
1
1
1 1
22
1
1
1 1
222 22
ii X
r
AUDCAU
DC DD ,,,21 2
222
222
222
222
1
22
1
222
AAD
UUDD
DDD
J
iii
J
iiiC
PBA
CPBUCPBCPB
DC
PBUD
sensitivity coefficientssensitivity coefficients
Example:Example:
Uncertainty equations for single and multiple tests
Measurements can be made in several ways:Measurements can be made in several ways:
Single test (for complex or expensive experiments): one set Single test (for complex or expensive experiments): one set of measurements (of measurements (XX11, , XX22, …, , …, XXjj) for ) for rr
According to the present methodology, a test is considered a single test if the entire According to the present methodology, a test is considered a single test if the entire test is performed only once, even if the measurements of one or more variables are test is performed only once, even if the measurements of one or more variables are made from many samples (e.g., LDV velocity measurements)made from many samples (e.g., LDV velocity measurements)
Multiple tests (ideal situations): many sets of Multiple tests (ideal situations): many sets of measurements (measurements (XX11, , XX22, …, , …, XXjj) for ) for rr at a fixed test condition at a fixed test condition with the same measurement systemwith the same measurement system
Uncertainty equations for single and multiple tests
The total uncertainty of the result The total uncertainty of the result
P + B = U 2r
2rr
2
BBr r : same estimation procedure for single and : same estimation procedure for single and multiple testsmultiple tests
PPr r : determined differently for single and multiple : determined differently for single and multiple teststests
Uncertainty equations for single and multiple tests: bias limits
BBrr : :
J
i
J
i
J
ikikkiiir BBB
1
1
1 1
222 2
ii X
r
BBii: estimate of calibration, data acquisition, data reduction, conceptual : estimate of calibration, data acquisition, data reduction, conceptual bias errors for bias errors for XXi.i.. Within each category, there may be several elemental . Within each category, there may be several elemental sources of bias. If for variable sources of bias. If for variable XXii there are there are JJ significant elemental bias significant elemental bias errors [estimated as errors [estimated as (B(Bii))11, , (B(Bii))2,2, … … (B(Bii))JJ], the bias limit for ], the bias limit for XXii is calculated as is calculated as
BBikik: estimate of correlated bias limits for : estimate of correlated bias limits for XXi i and and XXkk
L
kiik BBB1
Sensitivity coefficientsSensitivity coefficients
2
1
2
k
J
kii BB
Uncertainty equations for single test: precision limits
Precision limit of the result (end to end):Precision limit of the result (end to end):rr tSP
tt: coverage factor (: coverage factor (t t = 2 for = 2 for NN > 10) > 10)
SSrr: the standard deviation for the : the standard deviation for the NN readings of the result. readings of the result. SSrr must be must be determined from determined from NN readings over an appropriate/sufficient time interval readings over an appropriate/sufficient time interval
Precision limit of the result (individual variables):Precision limit of the result (individual variables):
)P ( P 2ii
J
=1ir
iii StP the precision limits for the precision limits for XXii
Often is the case that the time interval is inappropriate/insufficient and Often is the case that the time interval is inappropriate/insufficient and PPii’s ’s or or PPrr’s must be estimated based on previous readings or best available ’s must be estimated based on previous readings or best available informationinformation
Uncertainty equations for multiple tests: precision limits The average result:The average result:
tt: coverage factor (: coverage factor (t t = 2 for = 2 for NN > 10) > 10)
: standard deviation for : standard deviation for MM readings of the result readings of the result
Precision limit of the result (end to end):Precision limit of the result (end to end):
M
kkrM
r1
1
MtSP r
r 2/1
1
2
1
M
k
kr M
rrS
The total uncertainty for the average result:The total uncertainty for the average result:
Alternatively can be determined by RSS of the Alternatively can be determined by RSS of the precision limits of the individual variablesprecision limits of the individual variables
rS
rP
22222 2 MSBPBU rrrrr
Implementation Define purpose of the testDefine purpose of the test
Determine data reduction equation: Determine data reduction equation: rr = = rr((XX11, , XX22, …, , …, XXjj) )
Construct the block diagramConstruct the block diagram Construct data-stream diagrams from sensor to resultConstruct data-stream diagrams from sensor to result Identify, prioritize, and estimate bias limits at individual variable levelIdentify, prioritize, and estimate bias limits at individual variable level
Uncertainty sources smaller than 1/4 or 1/5 of the largest sources are neglectedUncertainty sources smaller than 1/4 or 1/5 of the largest sources are neglected
Estimate precision limits (end-to-end procedure recommended)Estimate precision limits (end-to-end procedure recommended) Computed precision limits are only applicable for the random error sources that were Computed precision limits are only applicable for the random error sources that were
“active” during the repeated measurements“active” during the repeated measurements Ideally Ideally MM 10, however, often this is no the case and for 10, however, often this is no the case and for MM < 10, a coverage factor < 10, a coverage factor tt = =
2 is still permissible if the bias and precision limits have similar magnitude.2 is still permissible if the bias and precision limits have similar magnitude. If unacceptably large If unacceptably large PP’s are involved, the elemental error sources contributions must ’s are involved, the elemental error sources contributions must
be examined to see which need to be (or can be) improvedbe examined to see which need to be (or can be) improved
Calculate total uncertainty using equation (4)Calculate total uncertainty using equation (4) For each For each r,r, report total uncertainty and bias and precision limits report total uncertainty and bias and precision limits
Recommendations Recognize that uncertainty depends on entire testing process and that Recognize that uncertainty depends on entire testing process and that
any changes in the process can significantly affect the uncertainty of any changes in the process can significantly affect the uncertainty of the test resultsthe test results
Integrate uncertainty assessment methodology into all phases of the Integrate uncertainty assessment methodology into all phases of the testing process (design, planning, calibration, execution and post-test testing process (design, planning, calibration, execution and post-test analyses)analyses)
Simplify analyses by using prior knowledge (e.g., data base), Simplify analyses by using prior knowledge (e.g., data base), concentrate on dominant error sources and use end-to-end calibrations concentrate on dominant error sources and use end-to-end calibrations and/or bias and precision limit estimationand/or bias and precision limit estimation
Document: Document: test design, measurement systems, and data streams in block diagramstest design, measurement systems, and data streams in block diagrams equipment and procedures usedequipment and procedures used error sources considerederror sources considered all estimates for bias and precision limits and the methods used in their estimation all estimates for bias and precision limits and the methods used in their estimation
(e.g., manufacturers specifications, comparisons against standards, experience, etc.)(e.g., manufacturers specifications, comparisons against standards, experience, etc.) detailed uncertainty assessment methodology and actual data uncertainty estimatesdetailed uncertainty assessment methodology and actual data uncertainty estimates
References AIAA, 1999, “Assessment of Wind Tunnel Data Uncertainty,” AIAA AIAA, 1999, “Assessment of Wind Tunnel Data Uncertainty,” AIAA
S-071A-1999.S-071A-1999. ASME, 1998, “Test Uncertainty,” ASME PTC 19.1-1998.ASME, 1998, “Test Uncertainty,” ASME PTC 19.1-1998. ANSI/ASME, 1985, “Measurement Uncertainty: Part 1, Instrument ANSI/ASME, 1985, “Measurement Uncertainty: Part 1, Instrument
and Apparatus,” ANSI/ASME PTC 19.I-1985.and Apparatus,” ANSI/ASME PTC 19.I-1985. Coleman, H.W. and Steele, W.G., 1999, Coleman, H.W. and Steele, W.G., 1999, Experimentation and Experimentation and
Uncertainty Analysis for EngineersUncertainty Analysis for Engineers, 2, 2ndnd Edition, John Wiley & Sons, Edition, John Wiley & Sons, Inc., New York, NY.Inc., New York, NY.
Coleman, H.W. and Steele, W.G., 1995, “Engineering Application of Coleman, H.W. and Steele, W.G., 1995, “Engineering Application of Experimental Uncertainty Analysis,” AIAA Journal, Vol. 33, No.10, Experimental Uncertainty Analysis,” AIAA Journal, Vol. 33, No.10, pp. 1888 – 1896.pp. 1888 – 1896.
ISO, 1993, “ISO, 1993, “Guide to the Expression of Uncertainty in Guide to the Expression of Uncertainty in MeasurementMeasurement,", 1st edition, ISBN 92-67-10188-9.,", 1st edition, ISBN 92-67-10188-9.
ITTC, 1999, ITTC, 1999, Proceedings 22Proceedings 22ndnd International Towing Tank Conference International Towing Tank Conference, , “Resistance Committee Report,” Seoul Korea and Shanghai China.“Resistance Committee Report,” Seoul Korea and Shanghai China.