Measurement Advisory Committee Summary - Attachment 3

8/2/2019 Measurement Advisory Committee Summary - Attachment 3

1/70

NCSL INTERNATIONALNCSL INTERNATIONALSERVING THE WORLD OF MEASUREMENT SI NCE 196 1SERVING THE WORLD OF MEASUREMENT SI NCE 196 1

ANSI/NCSL Z540.3ANSI/NCSL Z540.3Requirements for the Calibration ofRequirements for the Calibration of

Measuring and Test EquipmentMeasuring and Test Equipment

Sub Clause 5.3 Assessor TrainingSub Clause 5.3 Assessor Training

Steve Doty, NSWC Corona, 171 Committee Chair, Working Group OneSteve Doty, NSWC Corona, 171 Committee Chair, Working Group One ChairChair

Del Caldwell, CCG, 171 Working Group One CoDel Caldwell, CCG, 171 Working Group One Co--ChairChair

Dennis Jackson, NSWC Corona, 171 Committee MemberDennis Jackson, NSWC Corona, 171 Committee Member


2/70

NCSL INTERNATIONALNCSL INTERNATIONALSERVING THE WORLD OF MEASUREMENT SI NCE 196 1SERVING THE WORLD OF MEASUREMENT SI NCE 196 1

Sub Clause 5.3 Assessor TrainingSub Clause 5.3 Assessor Training1. Welcome and overview of the session Steve Doty (5 minutes)

2. Introduction and overview of Z540.3 / Perspectives on cal lab

compliance Del Caldwell (45 minutes)

3. Assessing cal labs to Z540.3, sub-clause 5.3 requirements / Appendix Fintroduction and use Steve Doty (45 minutes)

Break - 15 minutes

4. Measurement uncertainty concepts for Z540.3 / Probability of falseacceptance, concept and compliance Dennis Jackson (115 minutes)

5. Wrap-up / Broad Q&A / Availability of Supplemental NCSLI resources[Handbook and bibliography; RP-1 (2009 rev); RP-3 (2007 rev); RP-12(2009 rev); and RP-18 (2009) new] Steve Doty (15 minutes)


3/70

3NCSL INTERNATIONALNCSL INTERNATIONAL

SERVING THE WORLD OF MEASUREMENT SINCE 19 61SERVING THE WORLD OF MEASUREMENT SI NCE 196 1

Overview

Calibrat ion Process Uncertaint y

Probabi li t y of False Accept (PFA)

PFA Est im at ion Compliance Methods

Guard Band Compliance Methods

Test Uncert aint y Rat io


4/70



Z540.3 PFA Requirement

Where calibrations provide for verification thatmeasurement quantities are within specifiedtolerances, the probability that incorrect acceptancedecisions (false accept) will result from calibrationtests shall not exceed 2% and shall be documented.Where it is not practicable to estimate this

probability, the test uncertainty ratio shall be equalto or greater than 4:1.

NOTE: Achieving these requirements may involve

adjustment and management of calibration systemparameters such as: measurement reliability,calibration intervals, measurement uncertainty,calibration tolerances, and/or guard bands.


5/70



Themes The object ive of a measurement is to est imate t he tr ue value

Measurement error represents the error in using t he measurement t oest imat e t he t rue value

The obj ect ive of calibrat ion is t o est imate and possibly correct for t heUUT bias

The difference (or deviation) bet w een t he UUT measurement /indicat ion and the CALSTD measurement / indicat ion est imates t heUUT bias

The error in estim ating t he UUT bias is called t he calibr ation processerror

The calibration process error includes all t hose measurement errorsw e w ouldnt correct for

The calibr ation process error i s characterized and reported as t hecalibr ation process uncertaint y

I ncorrect calibr ation test ing decisions are caused by calibr ationprocess errors

A false accept occurs when the UUT bias is out of t olerance, but t he

est imate of t he UUT bias ( t he deviat ion) is not


6/70



TestInstrument

MeasurementSource

Measures

Measurement = True Value + Error

Measurement Error Model

Error = Measurement True Value

The objective of measurement is to estimate the true value

The Measurement Error represents the error in using the

Measurement to estimate the True Value


7/70



Voltmeter 10.0 VoltSource

Measures

Measurement = 10.1 v

Measurement = True Value + Error= 10.0 v + 0.1 v

Error = Measurement True Value= 10.1 v 10.0 v

= 0.1 v

Measurement Error Example


8/70



Measurement Uncertainty puts limits on measurement errors 2 Standard Uncertainties ( 2 u) contains about 95% of the

Measurement Errors

Measurement Uncert aint y


9/70



Measurement Uncert aint y Budget

Repeatability

Resolution

Reproducibility

Environment

Location Operators

Correct ion Factors

Setup/ Ancillary Equipm ent (Cables, et c.)

Calibrat ion Cert if icate


10/70



General Calibrat ion Scenario

The CALSTD Measurement is used to approximate the TrueValue

The Deviation is used to estimate the UUT Bias

UnitUnder Test

(UUT)

CalibrationStandard(CALSTD)

Measurement

MeasurementSource

Measurement

UUT Error = UUT Measurement True Value

Deviation = UUT Measurement CALSTD Measurement


11/70



Calibrat ion Error Model

Deviation = UUT Measurement CALSTD Measurement

= UUT Bias + Calibration Process Error

UnitUnder Test

(UUT)


MeasurementMeasurement

Source

Measurement

The objective of calibration is to estimate the UUT Bias

The Calibration Process Error represents the error in using theDeviation to estimate the UUT Bias


12/70



Calibrat ion Error Example

Deviation = 0.09 V

= UUT Measurement CALSTD Measurement

= UUT Bias + Calibration Process Error= 0.11 V + (- 0.02 V)

UnitUnder Test

(UUT)


MeasurementMeasurement

Source

Measurement


13/70



Calibration Process Uncertainty puts limits on CalibrationProcess Errors

2 Standard Uncertainties ( 2 u) contains about 95% of the

Calibration Process Errors

Calibrat ion Process

Uncertainty


14/70



Calibrat ion Process

Uncert aint y Budget Calibrat ion Standard

Repeatabi li t y ( UUT and CALSTD)

Resolution

Reproducibility

Calibrat ion Cert if icate

Unit Under Test (UUT)

Repeatabi li t y ( UUT and CALSTD)

Resolution

Reproducibility

Setup/ Ancillary Equipm ent (Cables, et c.)


15/70



Calibrat ion Scenarios

UUT(Measuring Device)

Measured Value: y

CALSTD(Measurement Source)

Indictated Value: x

True Output Value: T

Measures

UUT(Measurement Source)

Indictated Value: yTrue Output Value: T

CALSTD

(Measuring Device)

Measured Value: x

Measures

1

2


16/70


17/70



Summary

There are four major calibrat ion scenarios:1. UUT measures CAL 3. UUT and CAL measure art i fact

2. CAL measures UUT 4. UUT and CAL are compared

During a calibrat ion t he UUT is compared t o t he CAL using t hedeviation

Generally, Deviat ion = UUT measurement CAL measurement The deviat ion cont ains t he UUT bias and t he calibrat ion

process error

The point of t he calibrat ion is t o determine the UUT bias

The calibrat ion process error represents t he error in using t hedeviation t o est imate the UUT bias


18/70



True I n Tolerance(Calibrat ion Tolerance Test ing Obj ect ive)

A Unit Under Test (UUT) is truly in tolerance if:

Lower Spec < UUT Bias < Upper Spec

0Lower Spec (-L) Upper Spec (L)

UUT Bias

The UUT Bias is unknown


19/70



True Out Of Tolerance

A Unit Under Test (UUT) is truly out of tolerance if:

UUT Bias < Lower Spec or UUT Bias > Upper Spec


UUT Bias

The UUT Bias is unknown


20/70



Observed I n Tolerance(Calibrat ion Test ing in Pract ice)

A Unit Under Test (UUT) is observed in tolerance if:

Lower Spec < Deviation < Upper Spec


Deviation

The Deviation is the observed difference between the UUT and the CALSTD


21/70



False Accept s

Deviation

0- L L

UUT Bias

Cal

Proc

Error

False Accept (FA):

The Deviat ion is observed in to lerance [ -L < Deviat ion < L ] The UUT Bias is out of t olerance [ Bias > L or Bias < -L ] The decision t o accept t he UUT is incorrect


22/70



Probabil i t y of False Accept

Probabi li t y of False Accept (PFA):

PFA = Pr( [ Observed I n Tolerance] and [ True Out Of Tolerance] )

= Pr( [ -L < Deviat ion < L] and [ Bias > L or Bias < -L] )

Deviation

0- L L

UUT Bias

Cal

Proc

Error

PFA is the probability of making an incorrect acceptance decision


23/70



I nput s Needed t oCalculat e PFA

Probabili t y of False Accept (PFA) :

PFA = Pr( [ Observed I n Tolerance] and [ True Out Of Tolerance] )

= Pr( [ -L < Deviat ion < L] and [ Bias > L or Bias < -L] )

The Tolerance Lim it s ( -L, L) The Calibration Procedure for the UUT

The Measurement Uncert aint y for t he Calibrat ion Process The calibration process uncertainty is a 17025 and Z540.3

requirement This requires an uncertainty analysis for each calibration procedure

The Observed Test Point Measurement Reliabili t y for t he UUT Measurement reliability is obtained from calibration history data Using equipment level measurement reliability provides an upper

bound on PFA


24/70



PFA Examples

( )

( )Bias

L A

ABiased

BiasL

A

ABiased

deddedf

deddedfPFA

Bias

Bias

+=

,

,

,

,


25/70



PFA Tool


26/70



Summary

Calibrat ion t est ing determ ines if a UUT is out of t olerance

True out of t olerance means t he UUT bias is outside t hetolerances

The UUT bias is unknow n and is est imated using t he deviation(UUT meas CALSTD meas)

An observed in t olerance means t he deviat ion is inside t hetolerances

The calibrat ion process error represents t he error in using t hedeviation t o est imate the UUT bias

Calibrat ion process error can cause a tr ue out of t olerance to beobserved as in t olerance

A false accept means the UUT is observed in t olerance w hen t hebias is out of t olerance

The probabili t y of a false accept (PFA) is the probabilit y ofmaking a w rong acceptance decision dur ing a calibrat ion t est


27/70




Where calibrations provide for verification thatmeasurement quantities are within specified

tolerances, the probability that incorrect acceptancedecisions (false accept) will result from calibrationtests shall not exceed 2% and shall be documented.Where it is not practicable to estimate thisprobability, the test uncertainty ratio shall be equalto or greater than 4:1.

NOTE: Achieving these requirements may involve

adjustment and management of calibration systemparameters such as: measurement reliability,calibration intervals, measurement uncertainty,calibration tolerances, and/or guard bands.


28/70



PFA Compl iance MethodsPFA Est imat ion

Method 1, Uncondit ional - Test Point Populat ion Dat a:Provides an uncondi t ional PFA est imate w hich is a direct

determination of compliance to t he Standard. Method 2, Uncondit ional - M&TE Populat ion Dat a:

Provides a conservat ive uncondi t ional PFA est imateusing measurement reliabilit y data at t he M&TE modeland manufacturer level.

Method 3, Condit ional Acceptance Subpopulat ion:Provides a condit ional PFA est imate w here t hesubpopulat ion includes calibrat ion t est s that result inacceptance at t he test point level.

Method 4, Condit ional Bayesian:Determines condit ional PFA for m easurement result ; ifcondi t ional PFA is OK then uncondi t ional PFA OK.


29/70



I nformat ion RequirementsPFA Est imat ion

X-X-XBayesian

X-X

X

(if used)X

Acceptance

Subpopulation

XX-X

(if used)X

M&TE

Population

X-XX

(if used)X

Test Point

Population

Calibration

ProcessMeasurement

Uncertainty

M&TE

OverallMeasurement

Reliability

M&TE

Test PointMeasurement

Reliability

M&TE

Acceptance

Limits

M&TE

Test

Tolerances

I nformat ion Requirements

Method


30/70


31/70



PFA Est im at ion Method 1

Examples

PFA E i i M h d 2


32/70



PFA Est im at ion Method 2Uncondit ional M&TE

Populat ion Data

M&TE Measurement Reliability

Shows compliance to 2% requirement


33/70




Examples

PFA E t i t i M th d 3


34/70



PFA Est im at ion Method 3Condit ional Acceptance

Subpopulation



35/70




Examples


36/70




Condit ional Bayesian

Calibration Result (Deviation)



37/70




Examples

PFA E i i M h d


38/70



PFA Est imat ion Method

Comparisons

Met hod 1, Uncondit ional - Test Point Populat ion Data: Directdeterm inat ion of compliance, need t est point reliabilit y.

Method 2, Uncondit ional - M&TE Population Data:Conservat ive est imate (larger t han Method 1) , uses calibrat ionint erval m easurement reliabilit y data ( generally available).

Met hod 3, Condit ional Acceptance Subpopulat ion:

Conservat ive est imate ( larger t han Method 1) , appropriate fororganizat ions t hat w ork w it h PFA condit ioned on acceptance,need test point reliabilit y.

Met hod 4, Condit ional Bayesian: Very conservat ive est imate( larger than Method 1), need t est point r eliabilit y.

All methods require calibration process uncertainty


39/70




Where calibrations provide for verification thatmeasurement quantities are within specified

tolerances, the probability that incorrect acceptancedecisions (false accept) will result from calibrationtests shall not exceed 2% and shall be documented.Where it is not practicable to estimate thisprobability, the test uncertainty ratio shall be equalto or greater than 4:1.

NOTE: Achieving these requirements may involveadjustment and management of calibration systemparameters such as: measurement reliability,calibration intervals, measurement uncertainty,calibration tolerances, and/or guard bands.


40/70



Alt ering PFA

PFA may be alt ered t hrough adjustment ofcalibrat ion system cont rols including:

Measurement reliabil it y

Calibration intervals

Calibrat ion process uncert aint y

Calibrat ion t olerances

Guard bands

Guard Band use may low er t he probabilit y of

making false accepts dur ing a calibrat ion t est


41/70



Guard Bands

Guard Bands provide bet t er t est decision lim it s (acceptance lim it s)

Deviation

Deviat ions just inside the specif icat ion can be caused by t heCalibrat ion Process error Guard Bands lower t he probabilit y of m aking f alse accepts The determination of t he guard bands depends on how big

the Calibrat ion Process Error could be

AcceptanceAcceptance

0- L L- A A

UUT Bias

Cal

Proc

Error


42/70



Guard Band Methods

Method 1, Based on Uncondit ional PFA Est imation - Test PointPopulat ion Data: Calculat es guard bands based on uncondi t ional PFA.

Method 2, Based on Uncondit ional PFA Est imation - M&TE Populat ionData: Calculates guard bands based on conservat ive PFA estimatefrom M&TE equipment level m easurement reliabilit y.

Method 3, Based on Condi t ional PFA Estimat ion AcceptanceSubpopulation: Calculat es conservat ive guard bands based oncondit ional PFA.

Method 4, Based on Condi t ional PFA Est imat ion Bayesian:Calculat es guard bands based on condit ional Bayesian PFA(Condit ion is deviat ion at acceptance limi t ) .

Method 5, Based on the Expanded Calibr at ion Process Uncert aint y:Uses guard bands based on 95% expanded calibration process

uncertainty. Method 6, Based on t he Test Uncert aint y Rat io: Uses guard bands

based on the Test Uncert aint y Rat io w hich meets t he PFA requirementat t he worst case t est point measurement reliability.

I nformat ion Requirements


43/70



I nformat ion RequirementsGuard Band Methods

X--XTUR

X--XExpanded

Uncertainty

X(X)XXBayesian

X(X)XXAcceptance

Subpopulation

XX-X

M&TE

Population

X-XXTest Point

Population

Calibration

Process

MeasurementUncertainty

M&TE

Overall

MeasurementReliability

M&TE

Test Point

MeasurementReliability

M&TE

Test

Tolerances

I nformation Requirements

Method

Guard Band Method 1


44/70



UUT Bias

Cal Proc

Error

Deviation


Error Distribution

for Cal Process

Test Point Populat ion ( Method 1) approach f or Guard Bands:

Use uncert aint y in formation on t he Calibrat ion Process and t he UUT t estpoint populat ion data t o calculate t he probabil it y of a false accept ( PFA)

Choose a guard band w hich gives an acceptably low PFA

Error Distribution

for UUT Bias

0- L L- A A

Guard Band Method 1Uncondit ional - Test Point

Populat ion Data

Guard Band Method 1


45/70



a aUncondit ional - Test Point

Populat ion Data

Acceptance limits that meet the 2% requirement

Test Point Measurement Reliability


46/70


47/70



M&TE measurement reliabilit y is t he probabilit y an M&TEpasses all t he st eps of a calibrat ion procedure.

M&TE

TestSteps

ICP

Calibration

Tes

ts

M&TE Populat ion Data

Guard Band Method 2


48/70



M&TE Populat ion Data (Method 2) approach for Guard Bands:

M&TE Measurement Reliabilit y is the probabilit y of being in

tolerance (t he probabilit y all t he I CP steps pass) . UUT Bias Uncert aint y can be calculated fr om the Test Point

Measurement Reliabil ity ( t he probabil it y a single I CP test point setpasses).

A conservat ive est imate of t he UUT Bias Uncert aint y can becalculated fr om the M&TE Measurement Reliabilit y.

Use the Calibrat ion Process Uncert aint y and t he Conservat ive UUTBias Uncert ainty t o calculate a conservative probabilit y of a false

accept. Choose a guard band w hich gives an acceptably low conservat ive

PFA.

This guarantees at least as low of a non-conservat ive PFA.

Uncondit ional M&TE

Populat ion Data

Guard Band Method 2


49/70




Uncondit ional M&TE

Populat ion Data

M&TE Measurement Reliability

Guard Band Method 2


50/70



Guard Band Method 2Examples

After applying guard bands, the PFA meets the 2% requirementfor Scenarios 2 and 3

Guard Band Method 3Condit ional Acceptance


51/70



Acceptance Subpopulat ion (Method 3) approach forGuard Bands:

This chooses t he guard band based on a condi t ional version ofPFA ( CPFA)

This condit ional probabilit y answ ers t he quest ion:

What percentage of t he acceptances is expected to be st ill out -of-tolerance?

The methodology is essent ially t he same as Methods 1 and 2,t hough because of t he use of CPFA, the guard bands aresomew hat larger

Condit ional Acceptance

Subpopulation

Guard Band Method 3C dit i l A t


52/70



Condit ional Acceptance

Subpopulation


Guard Band Method 3


53/70



Examples

After applying guard bands, the PFA meets the 2% requirementfor Scenarios 1, 2, and 3

Guard Band Method 4


54/70



- L L

UUT Bias

Deviation

- A A


Conditional Distribution for UUT Bias

Given Deviation is at Acceptance Limit

The Bayesian approach ( Method 4) for Guard Bands:

Use uncertaint y inf ormat ion on t he Calibration Process t o calculatethe condit ional probabilit y of a false accept

Assumes calibrat ion result (deviation) is at t he acceptance lim it Choose a guard band w hich gives an acceptably low condit ional PFA


Guard Band Method 4


55/70





Guard Band Method 4


56/70



Examples


57/70

Guard Band Method 5Expanded Calibrat ion


58/70



Expanded Calibrat ion

Process Uncertaint y


Guard Band Method 5E l


59/70



Examples

Guard Band Method 6


60/70



The Test Uncert aint y Rat io (Met hod 6) approach f or Guard Bands:

PFA is driven by t he calibrat ion process uncert aint y, the test pointmeasurement reliabilit y, and t he tolerance limit s

The TUR is obtained f rom t he calibration process uncert ainty and t hetolerance lim it s

For a given TUR, a guard band can be chosen w hich meets t he PFArequirement for any measurement reliabilit y value

The guard band is a funct ion of t he TUR and the tolerance limit s

Using t his guard band obviates t he need t o obtain measurementreliabil i t y dat a w hile providing smaller guard bands t han Method 5


( )( )54.0log38.0%95%2 04.1

= TUReULA

Guard Band Method 6T t U t i t R t i


61/70





Guard Band Method 6E m l


62/70



Examples

Guard Band MethodComparisons


63/70



Comparisons

Method 1, Based on Uncondit ional PFA Est imation - Test PointPopulat ion Data: Smallest guard bands, need test point reliabili t y.

Method 2, Based on Uncondit ional PFA Est imation - M&TE Populat ionData: Fairly small guard bands, uses calibr ation int erval m easurementreliabilit y data ( generally available).

Method 3, Based on Condi t ional PFA Estimat ion AcceptanceSubpopulation: Appropriate for organizat ions t hat w ork w it h PFAcondit ioned on accept ance.

Method 4, Based on Condi t ional PFA Est imat ion Bayesian:Larger guard bands, need test point reliabil it y.

Method 5, Based on the Expanded Calibr at ion Process Uncert aint y:Largest guard bands, doesnt r equire test point reliabil it y.

Method 6, Based on t he Test Uncert aint y Rat io: Smallest of t he guardbands t hat do not require t est point r eliabilit y.

All methods require calibration process uncertainty

ANSI Z540.3 Test ing


64/70



ANSI Z540.3 Test ing

ANSI Z540.3 w as adopted July 2007 and is beingused by US indust ry

Changes caused by Z540.3 Calibrat ion procedures are based on Probabili t y of False

Accept (PFA) rat her t han Test Accuracy Rat io ( TAR)

Test point s in calibrat ion procedures must havePFA < 2%

A Test Uncert aint y Rat io (TUR) of 4:1 can be used w henPFA calculat ion is not pract icable

Calculat ion of PFA and TUR both require est imat ion of t hecalibrat ion process uncert aint y

The Test Accuracy Rat io ( TAR) is not used in ANSI Z540.3



65/70




The rat io of t he unit under t est specificat ions t o thecalibrat ion process uncert aint y is called t he TestUncertaint y Rat io (TUR)

Generally, the requirement for good test ing is to have:

A TUR greater t han 4.0 helps keep the probabili t y of baddecisions to an acceptable level.

This is usually referred t o w it h rat io jargon as 4 : 1 (4 t o 1)

Test Uncert aint y Rat io Example


66/70



est U ce ta t y at o a p e

Test at 10.0 volt s w it h 0.1 volt specif icat ions:

Upper Test Specif icat ion = 10.1 volt sLow er Test Specif icat ion = 9.9 volt s

The calibrat ion process has 0.025 volt 95% measurementuncertainty:

Upper 95% Uncertaint y = + 0.025 volt sLow er 95% Uncertaint y = 0.025 volt s

Applying t his to t he TUR equation, we get :

Test Accuracy Rat io


67/70



y

Much of industry has hist orically used t he Test AccuracyRat io (TAR) rat her t han t he TUR

The Test Accuracy Rat io is the rat io of t he unit under

test specif icat ions t o the calibrat or specif icat ions

The 4:1 requirement for t he TAR is t he same as for t heTUR.

Since t he calibrator specif icat ions are based oncalibrat or uncertainties rather t han calibration processuncert aint ies, TAR and TUR can be very dif ferent


68/70

TAR vs TUR


69/70



The calibrat or specif icat ion used in t he TAR is usuallybased on t he calibrator measurement uncert ainty

The calibrat ion process uncert aint y used in t he TUR

includes: The Calibrator (CALSTD) measurement uncertainty

The Calibration setup uncertainty (connections, etc.)

The UUT nonbias uncertainty(Repeatability, Resolution, Reproducibility)

I f t he UUT nonbias uncert ainty is large, t he TUR couldbe much w orse than the TAR

TAR and TUR w ill be close to t he same if :

The calibrator specifications are based on 95% measurementuncertainties

The UUT nonbias uncertainty is a small part of the calibrationprocess uncertainty

Summary


70/70



y

TAR is not TUR

TAR is not used in ANSI Z540.3

The TUR can be used w hen PFA calculat ion is not

practicable. The est imat ion of t he calibrat ion process uncert aint y is

t he hardest part of PFA compliance methods

The calculat ion of TUR requires t he est imat ion of t he

calibrat ion process uncert ainty PFA compliance methods should rarely be not

pract icable if TUR can be calculated

A 4:1 TUR meet s t he requirement s of t he St andard

The Probabil it y of False Accept (PFA) is the preferredmeasure of t est quality

Measurement Advisory Committee Summary - Attachment 3

Documents