Understanding the Law of Propagation of Uncertainties · Uncertainty expression Relative uncertainty: 5 mW m-2 nm-1 ±0.2 % i.e. uncertainty expressed as a percentage Absolute uncertainty:

http://www.emceoc.org

Understanding the Law of Propagation

of UncertaintiesEmma Woolliams

4 April 2017

The Law of Propagation of

Uncertainties

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Things you might already “know”

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Has a sensitivity coefficient

Adding in quadrature (% or units)

Averages reduce by 1 n

This term is to do

with correlation

Building on that starting point

• The Law of Propagation of Uncertainties

• How many readings you should average

• What correlation is

Has a sensitivity coefficient

Adding in quadrature (% or units)

Averages reduce by

This term is to do with correlation

1 n

2c

2

2

1

1

1 1

2 ,

n

iii

n n

i ji ji j i

u y

fu x

x

f fu x x

x x

At the end of this module, you should

understand

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

THE LAW OF PROPAGATION

OF UNCERTAINTIES

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

At the end of this module, you should

understand

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

Sensitivity Coefficients

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

How sensitive is

my result to

this?

How sensitive is

my result to

this?

Do an

experiment

to find out!

Lamp power supply:

What is the sensitivity of the irradiance to the lamp current?

0.985

0.99

0.995

1

1.005

1.01

1.015

250 500 750 1000 1250 1500 1750 2000 2250 2500

Sig

nal ra

tio

wavelength / nm

Ratio 27.21 A / 27.19 A

Measured ratio0.2 A10 %

PTFE Sphere phase transition

6 °C

1.5 %

How sensitive is

my result to

this?

Calculate it from

the

measurement

equation

Analytical sensitivity coefficients

ixi

fc

x

1 2 3, , ,y f x x x

The local slope:

Translating a change in one

parameter to a change in

the other

When it works well

S

light

1V

V

S light darkV V V

S

dark

1V

V

Very simple case

When it works well

2FEL 0 45 cal

s 2use

E dL

d

2

s FEL 0 45 cal s

3

use use use

22

L E d L

d d d

Inverse square law leads to a 2

When it doesn’t work well

• Because you can’t write a relationship

• Because it’s too difficult to differentiate

• Because it’s a program not an equation

How sensitive is

my result to

this?

Model

it!

Radiance through double aperture

system

2 2

1 2-d 2

2 2 2 2 2 2 2 2

1 2 1 2 1 2

2

4

r rg

r r x r r x r r

1A g L

By differentiating …

22 2 2 2 2

1 2 1 2

2 2 2

1 2

4r r d r r

r r d

2 2

1 2-d 2

2 2 2 2 2 2 2 2

1 2 1 2 1 2

2

4

r rg

r r x r r x r r

2 2 1

1 22g r r

2 2 2

1 2 1 2

1

2 2 2

1 2 2 1

2

2 2

1 2

4 21

4 21

4

r r r rg

r

r r r rg

r

r r dg

d

Or by “modelling”…

Sensitivity to atmospheric model choice

Tuz Gölü CEOS comparison

2010

Difference between continental

and desert model lead up to 3%

difference in 𝜌 values

Monte Carlo simulation

Sensitivity Coefficients

How sensitive is

my result to

this?

Experimentally

Mathematically

Numerically

ixi

fc

x

Vary in the lab and see

what happens

Vary in the

model and

see what

happens

All are acceptable

Adding in Quadrature

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Why quadrature?

Uncertainty Error

Describes the spread Difference to the

(unknowable) true

value

It’s very unlikely

that all will be

maximum in

same direction!

Combining

distributions

leads to the

quadrature rule

Uncertainty expression

Relative uncertainty:

5 mW m-2 nm-1 ± 0.2 %

i.e. uncertainty expressed as a percentage

Absolute uncertainty:

5 mW m-2 nm-1 ± 0.01 mW m-2 nm-1 i.e.

uncertainty expressed in the native

measurement units

Uncertainty expression

Relative uncertainty:

5 mW m-2 nm-1 ± 0.2 %

i.e. uncertainty expressed as a percentage

Absolute uncertainty:

5 mW m-2 nm-1 ± 0.01 mW m-2 nm-1 i.e.

uncertainty expressed in the native

measurement units

Adding in quadrature for absolute

models

y BA C 1B

y y

CA

y

2 2 2 2Au y u u uB C

Adding in quadrature for absolute

models

y BA

1A

y

22 2 21Au u By u

1B

y

2 2 2Au y u Bu

What behaves like an absolute model?

Distance = Measured + aperture thickness

Signal = Light signal – Dark signal

Uncertainty expression

Relative uncertainty:

5 mW m-2 nm-1 ± 0.2 %

i.e. uncertainty expressed as a percentage

Absolute uncertainty:

5 mW m-2 nm-1 ± 0.01 mW m-2 nm-1 i.e.

uncertainty expressed in the native

measurement units

Adding in quadrature for relative

modelsy BA

y y

A AB

2 2 2

u y u

Ay

B

B

A u

y y

B BA

2 2

2 2 2y yu y BA

BAu u

Adding in quadrature for relative

modelsy BA

1

BA A

y y

2 2 2

u y u

Ay

B

B

A u

2

y A y

BB B

2 2

2 2 2Ay y

BB

u uA

y u

Adding in quadrature for relative

models

2y A B

2

1y y

BA A

2 2 2

2u y u A B

BA

u

y

3

2 2y A y

B BB

2 2

2 2 22A

y yB

By u u

Au

What behaves like a relative model?

• Most radiometric equations!

• Anything where uncertainties are in %

Gain = Radiance / Signal

2FEL 0 :45 cal

s 2use

E dL

d

s s

FEL FEL

L L

E E

s s

0 :45 0 :45

L L

s s

use use

2L L

d d

2 2 22

s 0 :45 useFEL

s FEL 0 :45 use

2u L u u du E

L E d

At the end of this module, you should

understand

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

HOW MANY READINGS

SHOULD BE AVERAGED?

At the end of this module, you should

understand

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• What the ‘other term’ – the correlation bit means

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

Applying the Law of Propagation of

Uncertainties to an average

Averaging three readings

1 2 3M

3

E E EE

M M M

1 2 3

1

3

E E E

E E E

2 2 2

2 2 2 2

M 1 2 3

1 1 1

3 3 3u E u E u E u E

1 2 3 iu E u E u E u E

2

2 2

M

13

3iu E u E

2 2

M 2

3

3iu E u E

2 2

M

1

3iu E u E

2

2

M3

iu Eu E

M3

iu Eu E

Applying the Law of Propagation of

Uncertainties to an average

Averaging n readings

M 1

n

iiE E n

M 1

i

E

E n

2 2

M 1

n

iiu E u E n

2

2 2

M

1iu E n u E

n

2 2

M

1iu E u E

n

2

2

M

iu Eu E

n

M

iu Eu E

n

Average light – average dark

s light, dark,

1 1

1 1N M

i j

i i

V V VN M

dark,

dark,

1, 1, ,M

jV

j

Vc j

V M

light dark

2 2

2 V V

V

u uu

N M

light, dark,

2 2

2 2 2

1 1

1 1i j

N M

V V V

i j

u u uN M

light,

light,

1, 1, , N

iV

i

Vc i

V N

Taking enough readings

Applying the Law of Propagation of

Uncertainties to an average

Averaging three readings

1 2 3M

3

E E EE

M M M

1 2 3

1

3

E E E

E E E

2 2 2

2 2 2 2

M 1 2 3

1 1 1

3 3 3u E u E u E u E

1 2 3 iu E u E u E u E

2

2 2

M

13

3iu E u E

2 2

M 2

3

3iu E u E

2 2

M

1

3iu E u E

2

2

M3

iu Eu E

M3

iu Eu E

What is the uncertainty associated

with each reading?

1 2 3 iu E u E u E u E

Uncertainty

Describes the spread

Standard

deviation?

The dangers of taking a standard deviation

of a small number of readings

Reading numberValu

es a

nd c

um

ula

tive S

t d

ev

The dangers of taking a standard deviation

from a small number of readings

What you should do

Use more readings!

• Take more measurements today

M

iu Eu E

n

Better estimate of

uncertainty

More to average

What you should do

Use more readings!

• Have a facility commissioning phase where you

take more measurements

M

iu Eu E

n

Better estimate of

uncertainty

Today’s!

From standard deviation of 10: iu E

Today two readings, 2n

What you should do

Use more readings!

• Compare data from different days

Even if there is a drift:

e.g Use standard deviation of the

difference

between two readings

What you should do

Use more readings!

• Compare data from different wavelengths

(smooth out the bumps)

What you should do

Increase the estimate

• Increase standard uncertainty

Will be in revised GUM

2

light2

light,mean

1

3

sNu

N N

What you should do

Use more readings!

• Take more measurements today

• Have a facility commissioning phase

• Compare data from different days

• Compare data from different wavelengths

Increase the estimate

• Increased standard uncertainty

KNOWING WHEN TO STOP!

Use more readings!

But only when it’s worth it!

The metre: 1791 - 1799

Pierre-

Françoise-André

Méchain

Jean-Baptiste-

Joseph

Delambre

The Measure of all Things - Ken Alder

The uncertainties don’t keep dropping

forever

ADEV

Line Fit

Lower Bound

Upper Bound

AlaVar 5.2

Allan STD DEV

Produced by AlaVar 5.2

time / s 100 1000 10000

0.0001

0.001

The Allan Deviation – software

ADEV

Lower Bound

Upper Bound

AlaVar 5.2

Allan STD DEV

Produced by AlaVar 5.2

time / s 100 1000

1E-5

0.0001

www.alamath.com

At the end of this module, you should

understand

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

CORRELATION

At the end of this module, you should

understand

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

The Law of Propagation of

Uncertainties

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Truei iE E S R

Correlation: Type A and Type B

methods

Correlation occurs when there is something in common between the

different quantities, or between the measured values being combined.

Type A: From the data Type B: From knowledge

This is where the

correlation comes

from!

Systematic

Effects!

Correlation: Type A and Type B

methods

Correlation occurs when there is something in common between the

different quantities, or between the measured values being combined.

Type A: From the data

1

1,

1

ni i

i X Y

X X Y Yr x y

n s s

, ,u x y u x u y r x y

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Correlation? http://www.tylervigen.com/

Spurious correlations

Truei iE E S R

Correlation: Type A and Type B

methods

Correlation occurs when there is something in common between the

different parameters, or between the measurements being combined.

Type B: From knowledge

This is where the

correlation comes

from!

Systematic

Effects!

Calculate covariance

Remove covariance

Averaging partially correlated data

Truei iE E S R 1 2 3M

3

E E EE

True 1 2 3M

3 3

3 3 3

E R R RSE

2

2 2

M3

iu Ru E u S

Averaging Partially correlated data

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Truei iE E S R

2 2

i iu E u S u R

2, ;i ju E E u S i j

Averaging partially correlated data

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

1 2 1 3 1 4

2 3 2 4

3 4

, , , , , ,

, , , ,

, ,

u E E u E E u E E

u E E u E E

u E E

Averaging partially correlated data

Truei iE E S R 1 2 3

M3

E E EE

M M M

1 2 3

1

3

E E E

E E E

2, ;i ju E E u S i j

2 2

i iu E u S u R

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

2 2

2 2 2

M

1 13 3

3 3iu E u S u R

1 2 2 3

1 3 21 12

3 3u S

2

213 2

3u S

2 2

2 2 2

M

1 13 6 3

3 3iu E u S u R

2

2 2

M3

iu Ru E u S

Systematic and random effects: Lamp

measured 5 times (continued)

Systematic effects Random effects

Reference calibration Noise

Alignment Lamp current fluctuation

Lamp current setting

Temperature sensitivities

,S u S ,i iR u R

Truei iE E S R

2 2 2

i iu E u S u R

2

2 2

M

iu Ru E u S

n

Averaging partially correlated data

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

1 2 1 3 1 4

2 3 2 4

3 4

, , , , , ,

, , , ,

, ,

u E E u E E u E E

u E E u E E

u E E

Covariance Matrix

21 1 2 1

22 1 2 2

21 2

, ,

, ,

, ,

1 2

1

2

n

E n

n n n

u E u E E u E E

u E E u E u E E

u E E u E E un E

n

U

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

1 2 1 3 1 4

2 3 2 4

3 4

, , , , , ,

, , , ,

, ,

u E E u E E u E E

u E E u E E

u E E

Covariance Matrix Version

1 2y

n

f f f

x x x

C

21 1 2 1

22 1 2 2

21 2

, ,

, ,

, ,

1 2

1

2

n

E n

n n n

u E u E E u E E

u E E u E u E E

u E E u E E un E

n

U

Covariance matrix

with absolute

variance (squared

uncertainty) and

covariance

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

Covariance matrix

21 1 2 1

22 1 2 2

21 2

, ,

, ,

, ,

1 2

1

2

n

E n

n n n

u E u E E u E E

u E E u E u E E

u E E u E E un E

n

U

T 1 1i i i iE E S R s r

2 2 2 2 2

,2 2

( )

( ).

i i i

E ij

i j

E u S u R u s u r i jU

E E u S u s i j

Negative sensitivity coefficients

y BA 1A

y

1

B

y

22 2 21Au u By u

2 2 2Au y u Bu

E.g. Signal = Light - Dark

Negative sensitivity coefficients

y BA 1A

y

1

B

y

2 2 2 2 1 1 ,Bu y u AA u u B

E.g. Signal = Light - Dark

2 2 2 2 ,u y u u AB BA u

Correlation

reduces

uncertainty

Positive sensitivity coefficients

y BA 1A

y

1

B

y

2 2 2 2 1 1 ,BAu y u u u A B

E.g. Distance = measured + thickness

2 2 2 2 ,u y u u AB BA u

Correlation

increases

uncertainty

But I don’t know what the covariance is!

• Think of the worst-case scenario

Does correlation increase or decrease the

uncertainty?

Treat as “all random” and “all systematic”

At the end of this module, you should

understand

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

CONCLUSIONS AND

SUMMARY

At the end of this module, you should

understand

• The Law of Propagation of Uncertainties

• Why the sensitivity coefficient is central to the law

• How to determine the sensitivity coefficients

• What ‘adding in quadrature’ really means

• How many readings you should average

• What the uncertainty associated with an average is

• How you know you have enough readings

• When you’ve taken too many readings

• What correlation is

• What the uncertainties are for averaging partially correlated data

• How to estimate covariance from numerical and experimental data

• How to deal with not knowing what the correlation is

Sensitivity Coefficients

2 1

2 2c

1 1 1

  2 ,

n n n

i i ji i ji i j i

f f fu y u x u x x

x x x

How sensitive is

my result to

this?

Experimentally

Mathematically

Numerically

Taking enough readings, but not too

many

M

iu Eu E

n

Better estimate of

uncertainty

More to average

ADEV

Line Fit

Lower Bound

Upper Bound

AlaVar 5.2

Allan STD DEV

Produced by AlaVar 5.2

time / s 100 1000 10000

0.0001

0.001

Allan Deviation

Correlation

Truei iE E S R

Correlation occurs when there is something in common between the

different parameters, or between the measurements being combined.

Type B: From knowledge

This is where the

correlation comes

from!

Systematic

Effects!

Calculate covariance

Remove covariance