Insights on observation residual approaches for model and ...Introduction Residual Estimation of System Error Curiosities of Estimates from Residual Statistics What do we get from

IntroductionResidual Estimation of System Error

Curiosities of Estimates from Residual StatisticsWhat do we get from 4d-Var Residual Diagnostics?

Recast Sequential Approach for Q into Variational LanguageClosing Remarks

Insights on observation residual approaches formodel and observation error estimation

Ricardo Todling

Global Modeling and Assimilation Office, NASA

10th Adjoint Workshop1-5 June 2015

Roanoke, West Virginia

In collaboration with: Yannick Tremolet

Discussions with: S. E. Cohn, D. P. Dee, M. Fisher, & M. Leutbecher

Ricardo Todling Insights on error estimation 1 / 24




Outline

1 Introduction

2 Residual Estimation of System Error

3 Curiosities of Estimates from Residual Statistics

4 What do we get from 4d-Var Residual Diagnostics?

5 Recast Sequential Approach for Q into Variational Language

6 Closing Remarks





Residual Statistics from Sequential Filtering Approach

Introduction







Observation-minus-Background (OmB) residiuals

dbo = yo −Hxb

OmB error covariance matrix

< dbo(dbo)T >= HBHT + R

Sample error covariances of OmB residuals have traditionally beenused to estimate and model time-independent background errorcovariances (Schlatter 1974, Hollingsworth & Lonnberg 1986, Dee& da Silva 1999, Franke 1999, and others, going back to the workof T. Kailath in the 70s).

The expression above holds independently of optimality (though itrelies on assumptions about background and observation errors).







Other available residuals:

dao = yo −Hxa dab = Hxb −Hxa

which lead to the following cross-covariances:

< dao(dbo)T > = R + O(∆K)

< dab(dbo)T > = HBHT + O(∆K)

< dab(dao)T > = HAHT + O(∆K)

where ∆K is the deviation of the sequential filter gain from optimality.






General Remarks on Residual Statistics

Only under the assumption of optimality, ∆K = 0, thecross-covariances become covariances and the expressionsabove can be used to estimate observation, background, andanalysis error without concerns (Desroziers et al. 2005).

The validity of replacing the expectation operator, < • >,with the typical time average operator might not always bejustifiable.Derivation of residual cross-covariances corresponding tothose of the variational approach must be inferred carefully totry minimizing, if not possibly avoiding altogether, introducingtime correlations.






Residual Estimation of System Error





Residual Statistics from Sequential Smoothing ApproachIllustration: Application to L96

Residual Statistics from Sequential Smoothing Approach

Residuals from a sequential lag-1 smoother can be used to derive anestimate of the model error covariance∗:

dso = yo −HMxs dsa = Hxa −HMxs

which lead to the following cross-covariances:

< dso(dbo)T > = HQHT + R + O(∆G)

< dsa(dso)T > = HQHT + O(∆K) + O(∆G)

where ∆G is the deviation of the sequential smoother gain fromoptimality.

∗ see Todling (2014; QJRMS).






Illustration in L96

Examples for simple linear and nonlinear models serve as illustration ofthe method. For L96, with stochastically random Gaussian noise withcovariance of the form below . . .

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

site

site

True Q covariance

−0.1

−0.05

0

0.05

0.1

0.15

0.2

Specified model error covariance






Illustration in L96Residual statistics from lag-1 smoothers implemented for two assimilationmethodologies, namely the EKF and an adaptive OI, is shown to obtainreasonable estimates of Q.

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

site

site

sto: Regularised EKF Q estimate

−0.1

−0.05

0

0.05

0.1

0.15

0.2

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

site

site

sto: AOI Q estimate

−0.1

−0.05

0

0.05

0.1

0.15

0.2

EKF AOI

From Todling (2014; QJRMS)





Residual Statistics: Model or Observation Error?Residual Statistics: Lesson from OSSEs

Curiosities of Estimates from Residual Statistics






Model or Observation Error?Still examining the L96 application, when the residual diagnostic is usedto estimate R ≈< dao(dbo)T >, the derived estimates have a considerablesignature of the model error.

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

site

site

sto: Regularised EKF R estimate

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

site

site

sto: AOI R estimate

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

EKF AOIFrom Todling (2015; QJRMS)






Actual vs Prescribed OmB Residual Statistics: Original

t race(v*v)/ p ( 0.4761)trace(HPH+ R)/ p ( 0.4984)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800

sto: Regularised EKF MC50

cycle

actu

al v

s p

resc

rib

ed v

aria

nce t race(v*v)/ p ( 0.4933)

trace(HPH+ R)/ p ( 0.4935)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800

sto: Adapt ive OI MC50

actu

al v

s p

resc

rib

ed

var

ian

ce

EKF AOI






Actual vs Prescribed OmB Residual Statistics: Second Pass

t race(v*v)/ p ( 0.4922)trace(HPH+ R)/ p ( 0.4929)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800


cycle

actu

al v

s p

resc

rib

ed v

aria

nce t race(v*v)/ p ( 0.5372)

trace(HPH+ R)/ p ( 0.5057)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 100 200 300 400 500 600 700 800


cycle

actu

al v

s p

resc

rib

ed v

aria

nce

EKF AOI






Model or Observation Error?Not knowing any better - that is, that the error in the L96 application isdue to errors in the model - using the R estimate as a “correction” to thespecified observation error covariance leads to undesirable consequences.For example, re-estimation of R brings new estimates further away fromtrue value.

diag(R)diag(Re)diag(Reff)

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0 5 10 15 20 25 30 35 40


diag(R)diag(Re)diag(Reff)

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0 5 10 15 20 25 30 35 40


EKF AOI

From Todling (2015; QJRMS)






Lesson from OSSEs

Somewhat analogous to what we just seen are various estimates ofinter-channel correlations obtained from operational systems and OSSE’sbuild off these systems. Funny enough, OSSE residual statistics mightsuggest observations to be largely correlated when in fact they are not.Examples from GEOS-5 are given below.

0 20 40 60 80 100 1200

20

40

60

80

100

120GMAO OPS: AIRS AQUA

?1.0

?0.8

?0.6

?0.4

?0.2

0.0

0.2

0.4

0.6

0.8

1.0

0 20 40 60 80 100 1200

20

40

60

80

100

120GMAO (perfect) OSSE: AIRS AQUA

?1.0

?0.8

?0.6

?0.4

?0.2

0.0

0.2

0.4

0.6

0.8

1.0

0 20 40 60 80 100 1200

20

40

60

80

100

120GMAO OSSE: AIRS AQUA

?1.0

?0.8

?0.6

?0.4

?0.2

0.0

0.2

0.4

0.6

0.8

1.0

From GEOS-5 OPS From early OSSE (perfect obs) From tuned OSSE

Inspired by comment make by Anna Shlayeva at the Workshop on Correlated Obs. Errors, Reading U.K., 2014.OSSE results from exps in Errico et al. (2013; QJRMS)Results here from Todling (2015; QJRMS)





What do we get from 4d-Var Residual Diagnostics?





4d-ResidualsConsider SC 4d-Var in its 4d-PSAS form:

J(δx0) =1

2δxT0 B

−1δx0 +1

2(db −Hδx0)TR−1(db −Hδx0) ,

For

db ≡ [(db0)T (db1)T · · · (dbK )T ]T

da ≡ [(da0|0)T (da1|0)T · · · (daK |0)T ]T

dba ≡ db − da

it is simple to show that

< dba(db)T > = HBHT (HBH + R)−1 < db(db)T >”opt”

= HBHT .





4d-ResidualsWhen the evolution of forecast errors within the var-window is taken into account(i.e., considering how model error also propagates in the window), we really have

< db(db)T >= HPf0|−1H

T + R

where R is given by

R`,m ≡ R`δ`,m + H`|−1

`−1∑i=0

m−1∑j=0

M`,`−i|−1Q`−i,m−jMTm,m−j|−1

HTm|−1 ,

When B = Pf0|1, somewhat unsurprisingly, one finds that

< da(db)T >≈ R

This means:The cross covariance of “OmA” and “OmB” from 4d-Var give an estimate of aneffective observation error covariance.Indeed, only at initial time do we the sought out observation error covariance,i.e., < da0(db0)T >≈ R.





Brief argumentPreliminary estimation in the IFS

Recast Sequential Approach for Q into Variational Language






Residual Q estimation: sequential to variationalMotivated by the sequential approach, we consider two loosely connected 4d-Varproblems:

one being double the window size of the otherthink of the short-window as the filterthink of the long-window as the smootherfurthermore, each long window cycle is started from the short-window analysis

and derive expressions that relate the two problems to allow extracting information onthe model error covariance:

(< wI1

(dbI1)T >

)`,m

opt=

−1∑i=ks

−1∑j=ks

H`M`,`−i|ks−1Q`−i,m−jMTm,m−j|ks−1H

Tm + O(∆M`,m)

< wI1(dbI1

)T >opt= HQHT

wheredbI1

is the OmB residual in the first half of the short or long-window problems

wI1is an incremental residual differencing the OmA of the short-window with

that of the long-window problem within the first half of the long-window






Residual Q estimation: sequential to variational

T 0 0.2 0.4 0.6 0.8 1 1.2 1.4

1000

900

850

700

500

400

300

RH errors

Northern Hemisphere

sigbsigosigb2siga06siga12sigq

Q 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

1000

850

700

500

250

100

50

30

10

Tv errors (K)

Northern Hemisphere


U 0 0.5 1 1.5 2 2.5 3

1000

850

700

500

250

100

50

30

10

U errors (m/s)

Northern Hemisphere


V 0 0.5 1 1.5 2 2.5 3

1000

850

700

500

250

100

50

30

10

V errors (m/s)

Northern Hemisphere






Closing Remarks





Closing Remarks

Many of the insights on residual diagnostics presented here havebeen appreciated by others - largely in this audience.

However, I hope to have provided a few illustrations highlightingwhat we know from theory: that in actuality, background, modeland observations errors are inseparable.

It is possible to estimate model error covariances from residualstatistics using smoother ideas, but the entanglement withbackground and observations errors, particularly in variationalapproaches, make it really hard to obtain conclusive results.

Remark related to words on “Historical Overview”:

Recently, Grewal & Andrews (2010: IEEE Control Systems Magazine, 69-78)provides a nice review of the use of Kalman filtering in Aerospace. It seemsunfortunate, though, that these authors are not aware of the earlier review ofMcGee & Schmidt (1985: NASA TM 86847).


Insights on observation residual approaches for model and ...Introduction Residual Estimation of System Error Curiosities of Estimates from Residual Statistics What do we get from

Documents