JAN | Feb-Mar-Apr Mar-Apr-May Apr-May-Jun May-Jun-Jul IRI’s monthly issued probability forecasts of seasonal global precipitation and temperature We issue.

JAN | Feb-Mar-Apr Mar-Apr-May

Apr-May-Jun May-Jun-Jul

IRI’s monthly issued probability forecasts of seasonal global precipitation and temperature

We issue forecasts at four lead times. For example:

Forecast models are run 7 months into future. Observed dataare available through the end of the previous month (end ofDecember in example above). Probabilities are given for thethree tercile-based categories of the climatological distribution.

“Now”| Forecasts made for:

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 | || ||| ||||.| || | | || | | | . | | | | | | | | | |

Rainfall Amount (mm)

Below| Near | Below| Near | Above Below| Near |

The tercile category system:Below, near, and above normal

(30 years of historical data for a particular location & season)

Forecasts of the climate

Data:

33% 33% 33%Probability:

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200

Rainfall Amount (mm)

Below| Near | Below| Near | Above Below| Near |

20% 35% 45%

Example of a typical climate forecastfor a particular location & season

Probability:

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200

Rainfall Amount (mm)

Below| Near | Below| Near | Above Below| Near |

5% 25% 70%

Example of a STRONG climate forecastfor a particular location & season

Probability:

Below Above

Historically, the probabilities of above and below are 0.33. Shifting the mean by half a standard-deviation and reducing the variance by 20% (red curve) changes the probability of below to 0.15 and of above to 0.53.

Historical distribution Forecast distribution

The probabilistic nature of climate forecasts

30

12

30

24

12

24

24

10

24

10

FORECAST SST

TROP. PACIFIC: THREE (multi-models, dynamical and statistical)

TROP. ATL, INDIAN (ONE statistical)

EXTRATROPICAL (damped persistence)

GLOBAL ATMOSPHERIC

MODELS

ECPC(Scripps)

ECHAM4.5(MPI)

CCM3.6(NCAR)

NCEP(MRF9)

NSIPP(NASA)

COLA2

GFDL

ForecastSST

Ensembles3/6 Mo. lead

PersistedSST

Ensembles3 Mo. lead

IRI DYNAMICAL CLIMATE FORECAST SYSTEM

POSTPROCESSING

MULTIMODELENSEMBLING

PERSISTED

GLOBAL

SST

ANOMALY

2-tiered OCEAN ATMOSPHERE

30

GFDL has 10 to PSST

FORECAST SST

TROP. PACIFIC: THREE scenarios: 1) CFS prediction 2) LDEO prediction 3) Constructed Analog prediction TROP. ATL, and INDIAN oceans CCA, or slowly damped persistence EXTRATROPICAL damped persistence

MULTIPLEGLOBAL

ATMOSPHERICMODELS

ECPC(Scripps)

ECHAM4.5(MPI)

CCM3.6(NCAR)

NCEP(MRF9)

NSIPP(NASA)

COLA2

GFDL

IRI DYNAMICAL CLIMATE FORECAST SYSTEM

PERSISTED

GLOBAL

SST

ANOMALY

2-tiered OCEAN ATMOSPHERE

Atmospheric General Circulation Models Used in the IRI's Seasonal Forecasts, for Superensembles

 Name Where Model Was Developed Where Model Is Run

NCEP MRF-9 NCEP, Washington, DC QDNR, Queensland, Australia

ECHAM 4.5 MPI, Hamburg, Germany IRI, Palisades, New York

NSIPP NASA/GSFC, Greenbelt, MD NASA/GSFC, Greenbelt, MD

COLA COLA, Calverton, MD COLA, Calverton, MD

ECPC SIO, La Jolla, CA SIO, La Jolla, CA

CCM3.6 NCAR, Boulder, CO IRI, Palisades, New York

GFDL GFDL, Princeton, NJ GFDL, Princeton, NJ

Collaboration on Input to Forecast Production

Sources of the Global Sea Surface Temperature Forecasts 

Tropical Pacific Tropical Atlantic Indian Ocean Extratropical Oceans

NCEP Coupled CPTEC Statistical IRI Statistical Damped PersistenceLDEO Coupled Constr Analogue 

----- -----

Much of our predictability comes about due to

Strong trade winds

Westward currents, upwelling

Cold east, warm west

Convection, rising motion in west

Weak trade winds

Eastward currents, suppressed upwelling

Warm west and east

Enhanced convection, eastward displacement

NDJ Precipitation Probabilities associated with El Nino

Empirical Approach (a good baseline by which to judge the more advanced methods “honest”)

NDJ

Combining the Predictions of Seasonal Climate

by Several Atmospheric General Circulation

Models (AGCMs) into a Single Prediction.

What is Done at IRI

Goals in Combining

To combine the probability forecasts of severalmodels, with relative weights based on the past performance of the individual models

To assign appropriate forecast probability distribution: e.g. damp overconfident forecasts

toward climatology

One Choice: Multiple Regression

Multiple regression uses the predictions of 2 or more models,along with the corresponding observations, to derive a set ofmodel weights that minimizes squared errors of the weightedpredictions.

Good feature: The actual skills of each model are taken into account, andalso the overlap in sources of predictability. So, if two models have high skillin the same way or for the same reason, or if two good models are nearly identical, both of them do not get the same high weight.

Bad features: (1) Random variations in the training sample are taken asseriously as the predictable part of the variability. This leads to overfitting,and poor results in real forecasts. The more predictors compared to thenumber of independent cases, the more severe this problem is. (2) Forecastsfor conditions outside the range of the training sample may be dangerous.

Note: A model’s weight does not necessarily describe its skill.In fact, negative weights can be seen for skillful models.

Another Choice: Simple Skill Weighting

Here, the weights given to 2 or more models are determinedby each model’s individual skill in forecasting observations. Theskill may be a correlation or other measure. The weights arenormalized so that their sum is 1.

Good features: (1) The actual skills of each model are takeninto account. (2) Overfitting is not as serious a problem hereas it is in multiple regression. (3) Final forecast is always withinthe range of the individual forecasts.

Bad feature: Duplication in the sources of skill is not checked.(Colinearity ignored.) So, if two good models are almost identical,both will be weighted strongly, even though the second one doesnot add much skill. A medium-skill model that has a uniquesource of skill will not influence the forecast enough.

Note: A model’s weight is easily interpreted as being its skill.

Combine climatology forecast (“prior”) and an AGCM forecast, with its

evidence of historical skill, to produce weighted (“posterior”) forecast probabilities, by maximizing the

historical likelihood score.

Methods Used at IRI:(1) Bayesian Combination (2) Canonical Variate Combination

Bayesian Combination

Six GCM Precip. Forecasts, JAS 2000

RPSS Skill of Individual Models: JAS 1950-97

Probabilities and Uncertainty

Pkt (x) Pk (x)

Pkt (y) mkt

m

1/3

1/3

1/3

6/24

8/24

10/24

Above Normal

Near-Normal

Below Normal

ClimatologicalProbabilities

GCMProbabilities

x2

y2

x1

y1

Tercile boundaries are identified for the models’ own climatology, by aggregating all years and ensemble members. This corrects overall bias.

k = tercile numbert = forecast time

m = no. ens members

Bayesian Model Combination

Combine climatology forecast (“prior”) and an AGCM forecast, with its

evidence of historical skill, to produce weighted (“posterior”) forecast probabilities, by maximizing the

historical likelihood score.

The multi-year product of the probabilities that were hindcast for the category

that was observed.

(Could maximize other scores, such as RPSS)

Prescribed, observed SST used to force AGCMs.Such simulations used in absence of ones usingtruly forecasted SST for at least half of AGCMs.

1

( ) log PN

ktt

L w

Aim to maximize thelikelihood score

k=tercile category t=year number

(.333)c j jktjkt

c j

w w PP

w w

1. Calibration of each model, individually, against climatology

2. Calibration of the weighted model combination against climatol

k=tercile category (1,2, or 3)t=year number

j=model number (1 to 7) w=weight for climo (c) or for model j

where wMM uses wj proportional to results of the first step above

PMMkt= weighted linearcomb of Pjkt over all j,normalized by Σ(wj)

Optimizelikelihood

score

Optimizelikelihood

scoreMMc

MMktMMckt

final

ww

PwwP

*

)333(.*

Algorithm used to maximize the designated score:Feasible Sequential Quadratic Programming (FSQP)

“Nonmonotone line search for minimax problems”

C M : TOTAL NUMBER OF CONSTRAINTS.C ME : NUMBER OF EQUALITY CONSTRAINTS.C MMAX : ROW DIMENSION OF A. MMAX MUST BE AT LEAST ONE AND GREATERC THAN M.C N : NUMBER OF VARIABLES.C NMAX : ROW DIMENSION OF C. NMAX MUST BE GREATER OR EQUAL TO N.C MNN : MUST BE EQUAL TO M + N + N.C C(NMAX,NMAX): OBJECTIVE FUNCTION MATRIX WHICH SHOULD BE SYMMETRIC ANDC POSITIVE DEFINITE. IF IWAR(1) = 0, C IS SUPPOSED TO BE THEC CHOLESKEY-FACTOR OF ANOTHER MATRIX, I.E. C IS UPPERC TRIANGULAR.C D(NMAX) : CONTAINS THE CONSTANT VECTOR OF THE OBJECTIVE FUNCTION.C A(MMAX,NMAX): CONTAINS THE DATA MATRIX OF THE LINEAR CONSTRAINTS.C B(MMAX) : CONTAINS THE CONSTANT DATA OF THE LINEAR CONSTRAINTS.C XL(N),XU(N): CONTAIN THE LOWER AND UPPER BOUNDS FOR THE VARIABLES.C X(N) : ON RETURN, X CONTAINS THE OPTIMAL SOLUTION VECTOR.C U(MNN) : ON RETURN, U CONTAINS THE LAGRANGE MULTIPLIERS. THE FIRSTC M POSITIONS ARE RESERVED FOR THE MULTIPLIERS OF THE MC LINEAR CONSTRAINTS AND THE SUBSEQUENT ONES FOR THEC MULTIPLIERS OF THE LOWER AND UPPER BOUNDS. ON SUCCESSFULC TERMINATION, ALL VALUES OF U WITH RESPECT TO INEQUALITIESC AND BOUNDS SHOULD BE GREATER OR EQUAL TO ZERO.

Circumventing the effects of sampling variability

• Sampling variability appears to be an issue: noisy weight distribution with large number of zero weights and some unity weights

• Bootstrap the optimization, omitting contiguous 6-year blocks of the 48-yr time series– yields 43 samples of 42 years

– shows the sampling variability of the likelihood over subsets of years

– We average the weights across the samples

Example

• Six GCMs’ Jul-Aug-Sep precipitation simulations

• Training period: 1950–97

• Ensembles of between 9 and 24 members

Model Weights – initially, by individual model

Climatological Weights – Multi-model

Model Weights – after second (damping) step

Model Weights – step 2, and Averaged over Subsamples

For more spatially smooth results, theweighting of each grid point is averaged

with that of its 8 neighbors, using binomial weighting.

X X XX X XX X X

Climatological Weights

Combination Forecasts of July-Sept PrecipitationAfter first stage only

After second (damping) stage

After spatial smoothingAfter sampling subperiods

ReliabilityJAS Precip., 30S-30N

Above-Normal Below-Normal

Forecast probabilityForecast probability

Obse

rved r

ela

tive

Freq.

Obse

rved r

ela

tive

Freq.

Bayesian

Pooled

(3-model)from Goddard et al. 2003

IndividualAGCM

RPSS Precip.

from Roberson et al. (2004):Mon. Wea. Rev., 132, 2732-2744

RPSS 2-m

Temp.from Roberson et al. (2004):

Mon. Wea. Rev., 132, 2732-2744

Conclusions - Bayesian

• The “climatological” (equal-odds) forecast provides a useful prior for combining multiple ensemble forecasts

• Sampling problems become severe when attempting to combine many models from a short training period (“noisy weights”)

• A two-stage process combines the models together according to a pre-assessment of each against climatology

• Smoothing of the weights across data sub-samples and spatially appears beneficial

IRI’s forecasts use also a second consolidationscheme, whose result is averaged with

the result of the Bayesian scheme.

1. Bayesian scheme

2. Canonical Variate scheme

Canonical Variate Analysis (CVA)

A number of statistical techniques involve calculating linear combinations (weighted sums) of variables. The weights are defined to achieve specific objectives:

• PCA – weighted sums maximize variance

• CCA – weighted sums maximize correlation

• CVA – weighted sums maximize discrimination

Canonical Variate Analysis

Let X be a set of centered explanatory variables, with variance-covariance matrix Sxx. Let Y be a set of (non-centered) indicator variables that define the group membership of a set of observations, with cross-products matrix Syy. In our case, membership is with respect to tercile. Solve the eigenproblem:

1 1 2xx xy yy xy r S S S S I a 0,

which is identical to the CCA problem, but with Y as the set of indicator variables rather than continuous values on the dependent variables. The loadings, a, maximize ratio of between-group variance to total variance, represented by the canonical correlation, r. A discrimination is maximized w.r.t. tercile group membership.

Canonical Variate Analysis

The canonical variates are defined to maximize the ratio of the between-category (separation between the crosses) to the within-category (separation of dots from like-colored crosses) variance.

Conclusion

IRI is presently using a 2-tiered prediction system.

It is interested in using fully coupled systems also,and is looking into incorporating those.

Within its 2-tiered system it uses 4 SST predictionscenarios, and combines the predictions of 7

AGCMs.

The merging of 7 predictions into a single one uses

two multi-model ensemble systems: Bayesian andcanonical variate. These give somewhat differing solutions, and are presently given equal weight.

Plan for Improvement of the Basic Forecast Product

IRI plans to issue forecasts as probabilitydensity functions (pdfs) instead of just forthe 3 tercile-based categories.

From the pdfs, users can construct proba-bilities for any categories desired.

Skill results for IRI real-timeclimate forecasts from 1997-2001

Real-timeForecastSkillGoddard etal. 2003,Bull. Amer.Meteorol. Soc.,84, 1973-1796.

Real-timeForecastSkillGoddard etal. 2003,Bull. Amer.Meteorol. Soc.,84, 1973-1796.

Real-timeForecastSkillGoddard etal. 2003,Bull. Amer.Meteorol. Soc.,84, 1973-1796.

Real-timeForecastSkillGoddard etal. 2003,Bull. Amer.Meteorol. Soc.,84, 1973-1796.

Perfe

ct re

liabil

ity

RELIABILITY DIAGRAM

Climate Information System: A USER centric perspective

Issues aboutrelating toour users

Some Common Customer Requests

1. More accurate climate forecasts (tighter pdfs) (we can at least deliver most reliable ones)

2. Forecasts of full pdf in physical units

3. Forecasts of the weather within climate

4. Forecasts for subseasonal periods

5. Climate change nowcast