Lorenz 1963 Multi-model Ensemble Test-bed

Lorenz 1963 Multi-model Ensemble Test-bed

Christine Johnson

NAEFS Teleconference September 2006

© Crown copyright Page 2

Research Aim

The Met Office is currently developing a real-time multi-model ensemble using the TIGGE data.

• Which methods should be used to calibrate and combine the multi-model ensemble?

• Evaluate the benefits of the multi-model ensemble.

Investigate within an idealized framework using the Lorenz 1963 model.


Lorenz 1963 multi-model ensemble testbed

Model 1

Model 2

Near =3/2Near =/2• The 1963 Lorenz

equations, with a dimensional scaling so that the fast-timescale is similar to the synoptic timescale of the atmosphere.

• Add ‘seasonally’ dependent biases that are sinusoidal and dependent on , which varies from 0 to 2 in one ‘model-year’ (300 days).

• Generate 8-member, 15-day ensembles every 12 hours.

Truth


Ensemble members

TruthControlMember

• The models give different forecasts.

• The true state tends to lie within the ensemble.

• The ensemble spread is representative of the forecast error.


RMS Errors

ControlMeanSpreadClimate

• The deterministic (control) forecast error is the same as that for climate, by 15 days.

• The ensemble mean is more accurate by the end of the forecast.

• The ensemble spread is too small at the end of the forecast.


Calibration and Combination Data

Use previous forecasts (in the blue box ) together with observations, to calibrate and combine the new forecast (red).

Use a 50 day moving calibration window.


Calibration Method

Choose the values of 0 and 1 to minimize the mean square error (MSE) over the calibration window (linear regression).

Choose the value of so that the ensemble variance is equal to the MSE of the bias-corrected ensemble mean.

1 0

'

' '

BC

VI

x x x

x x

x x


Rank Histograms

No bias correction or variance inflation

With bias correction

With bias correction and variance inflation

Note: Rank i means that i ensemble members are larger than the verification

Most ensemble members are smaller than the verification: negative bias.

Ensemble is under-dispersive, so either all the members are larger or all the members are smaller than the verification.

There is no bias, and the verification lies within the ensemble.


Impact of the position of the window

What is the impact of the position of the calibration window, relative to the forecast?

The best result is with a CENTRED window.

It is difficult to correct the errors at LONG LEAD TIMES.


Bias coefficients

Bias coefficients show the general seasonal variation.

Bias coefficients have more variability at longer lead times and therefore are more sensitive to the calibration sample.


Use of reforecast data

Following Hamill et al 2004, use 20 years of forecast data with a 50 day window centred on the forecast to be verified.

Raw data

With bias correction

Bias correction gives a reduction in the errors at long lead times, but not as large as for the centred moving window.


Summary: Calibration

Bias coefficients exhibit large fluctuations at long lead times due to flow-dependent errors.

We can only expect realistic moving-window bias correction to correct the errors at short lead times.

Reforecast data can be used to correct the seasonally varying component at long lead times. A centred moving-window is needed to correct the flow dependent component.


Combination Method

2

1

2

( ) ( , )

weight given to ensemble knormal distributionensemble mean

ensemble variancenumber of single-model ensembles

M

k k k kk

k

k

k

k

p x w f x

wfx

M

Assume that the multi-model pdf is formed by the sum of the pdfs of each single-model (Raftery et al 2005).

Estimate the model-dependent weights, wk.


Use of the model-dependent weights wk

Multi-model ensemble mean:

Multi-model ensemble probabilities (Stefanova and Krishnamurti 2002):

Where and are the single-model means and probabilities respectively.

MM k kk

x w x

MM k kk

p w pkx kp


Combination methods summary

Note: For a fair comparison, the multi-model ensembles has the same number of members as the single model ensembles (8 – member).


Bayesian model averaging

In a similar way to a mixture-density problem, estimate the weights and variances using the expectation-maximization algorithm

The BMA estimated weights are sensitive to the sample due to ill-conditioning. Hence it is important to add prior information to the log-likelihood function, in the form of a beta-distribution on w (Fraley and Raftery, 2005):

No prior

With prior

( ) (1 )k k kp w w w


The model-dependent weights

RMS-based

Multiple-regression

BMA

• All three methods give similar estimates for the model dependent weights.

• The weights show the ‘seasonal’ variation, despite having been bias-corrected.


Combination methods results

Resolution

Reliability

The simple combination gives an improvement in RESOLUTION.

The model-dependent weights gives an improvement in RELIABILITY.

Single-model with bias correction & variance inflation

It is difficult to determine the best combination method.


Impact of the position of the window

RMS error of Mean Reliability

Realistic window

Centred window

The model-dependent weights have more impact in improving the RMS errors and reliability when using a centred window.

The multiple-regression method gives slightly better results using the ideal centred window.


Summary: combination

Model combination gives improvements in resolution

Model-dependent weights give improvements in reliability.

All 3 methods give similar estimates for the weights.


Conclusions and future plans

The results show that model combination gives the most improvements to the ensemble resolution. Calibration and model-dependent weights give improvements in the reliability.

In the Met Office multi-model ensemble: Apply Kalman-filter type bias correction. Use the most simple method (skill-based)

for estimating model-dependent weights. Consider spatial correlations.


References

Hamill, T. M., J. S. Whitaker and X. Wei, 2004: Ensemble reforecasting: Improving medium-range forecast skill using retrospective forecasts. Mon. Weather Rev., 132, 1434-1447.

Fraley C. and A. Raftery, 2005: Bayesian regularization for normal mixture estimation and model-based clustering. Technical report 486, Dept. Statistics. Univ. Washington.

Raftery A., T. Gneiting, F. Balabdaoui and M. Polakowski , 2005: Using Bayesian model averaging to calibrate forecast ensembles. Mon. Weather Rev., 133, 1155-1174.

Stefanova L. and T. N. Krishnamurti, 2002: Interpretation of seasonal climate forecast using Brier skill score, the Florida state university superensembles and the AMIP I dataset. J. Climate, 15, 537-544.

Lorenz 1963 Multi-model Ensemble Test-bed

Documents

ensemble variance

ensemble mean

biascorrected ensemble

ensemble spread

realtime multimodel

i ensemble members

centred window

negative bias