Assessing Predictability with a Local Ensemble Kalman Filter

Assessing Predictability with a Local Ensemble Kalman Filter

DAVID KUHL,* ISTVAN SZUNYOGH,� ERIC J. KOSTELICH,# GYORGYI GYARMATI,@ D. J. PATIL,@

MICHAEL OCZKOWSKI,& BRIAN R. HUNT,** EUGENIA KALNAY,� EDWARD OTT,�� AND

JAMES A. YORKE##

*Department of Meteorology, University of Maryland, College Park, College Park, Maryland�Department of Meteorology, and Institute for Physical Science and Technology, University of Maryland, College Park,

College Park, Maryland#Department of Mathematics and Statistics, Arizona State University, Tempe, Arizona

@ Institute for Physical Science and Technology, University of Maryland, College Park, College Park, Maryland&Francis Marion University, Florence, South Carolina

** Institute for Physical Science and Technology, and Department of Mathematics, University of Maryland, College Park,College Park, Maryland

�� Institute for Research in Electronics and Applied Physics, and Departments of Physics and Electrical and Computer Engineering,University of Maryland, College Park, College Park, Maryland

## Institute for Physical Science and Technology, and Departments of Mathematics and Physics, University of Maryland,College Park, College Park, Maryland

(Manuscript received 22 April 2005, in final form 13 July 2006)

ABSTRACT

In this paper, the spatiotemporally changing nature of predictability is studied in a reduced-resolutionversion of the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS), astate-of-the-art numerical weather prediction model. Atmospheric predictability is assessed in the perfectmodel scenario for which forecast uncertainties are entirely due to uncertainties in the estimates of theinitial states. Uncertain initial conditions (analyses) are obtained by assimilating simulated noisy verticalsoundings of the “true” atmospheric states with the local ensemble Kalman filter (LEKF) data assimilationscheme. This data assimilation scheme provides an ensemble of initial conditions. The ensemble meandefines the initial condition of 5-day deterministic model forecasts, while the time-evolved members of theensemble provide an estimate of the evolving forecast uncertainties. The observations are randomly dis-tributed in space to ensure that the geographical distribution of the analysis and forecast errors reflectpredictability limits due to the model dynamics and are not affected by inhomogeneities of the observationalcoverage.

Analysis and forecast error statistics are calculated for the deterministic forecasts. It is found thatshort-term forecast errors tend to grow exponentially in the extratropics and linearly in the Tropics. Thebehavior of the ensemble is explained by using the ensemble dimension (E dimension), a spatiotemporallyevolving measure of the evenness of the distribution of the variance between the principal components ofthe ensemble-based forecast error covariance matrix.

It is shown that in the extratropics the largest forecast errors occur for the smallest E dimensions. Sincea low value of the E dimension guarantees that the ensemble can capture a large portion of the forecasterror, the larger the forecast error the more certain that the ensemble can fully capture the forecast error.In particular, in regions of low E dimension, ensemble averaging is an efficient error filter and the ensemblespread provides an accurate prediction of the upper bound of the error in the ensemble-mean forecast.

Corresponding author address: Dr. Istvan Szunyogh, Institute for Physical Science and Technology, University of Maryland, CollegePark, College Park, MD 20742-2431.E-mail: [email protected]

1116 J O U R N A L O F T H E A T M O S P H E R I C S C I E N C E S VOLUME 64

DOI: 10.1175/JAS3885.1

© 2007 American Meteorological Society

JAS3885

1. Introduction

In dynamical systems theory, predictability is oftencharacterized by the largest Lyapunov exponent of thesystem. This characterization is based on studying theevolution of initially small perturbations to a nonlineartrajectory, assuming that a numerically computed, suf-ficiently long trajectory can explore the small neighbor-hood of all possible states of the system (e.g., Ott 2002).Such a characterization may not apply for finite timeforecasts and is especially inappropriate when the di-mensionality of the dynamics is so high that explorationof the attractor by a typical trajectory takes a very longtime. This is the case for a high-dimensional weatherprediction model that mimics the evolution of the at-mosphere.

Patil et al. (2001) introduced the ensemble dimension(E dimension) [originally called bred vector dimension(BV dimension)] to characterize the spatiotemporallychanging complexity of the dynamics for a physicallyextended large system, such as a state-of-the art nu-merical weather prediction model. The E dimension isa local, spatiotemporally evolving measure of complex-ity (Patil et al. 2001; Oczkowski et al. 2005). The calcu-lation of this measure is based on the singular valuedecomposition of an ensemble-based estimate of theanalysis (or forecast) error covariance matrix in a localregion. Heuristically, the E dimension measures theevenness of the distribution of the variance between theprincipal components of the ensemble-based estimateof the forecast error covariance matrix. The lowest pos-sible value of the E dimension, which is one, occurs whenthe estimated variance is confined to a single spatialpattern of uncertainty. The highest possible value of theE dimension, which is equal to the number of ensemblemembers N, occurs when the variance is evenly distrib-uted between N independent patterns of uncertainty.

Patil et al. (2001) applied the E dimension diagnosticto operational forecast ensembles of the National Cen-ters for Environmental Prediction (NCEP). They foundan intriguing relationship between the regions of low Edimensionality and the magnitude of the ensemble per-turbations: the lowest-dimensional regions were oftenthe regions of largest estimated forecast uncertainties.Patil et al. (2001) hypothesized that there was a largepotential for analysis and forecast improvements in theregions of low E dimensionality due to the simple struc-ture of potential analysis and forecast error patterns inthose area. Most importantly, this result motivated thedevelopment of the local ensemble Kalman filter(LEKF; Ott et al. 2004) data assimilation scheme.

While the results of Patil et al. (2001) with the NCEPforecast ensembles were encouraging, they could not be

considered conclusive due to some important limita-tions of the ensemble used in the study. Most impor-tantly, there were only five independent ensemblemembers available for the calculation. Second, theNCEP ensembles were initialized with the breeding al-gorithm (Toth and Kalnay 1993, 1997), which tends toforce the initial ensemble perturbations toward a fewdominant error patterns (e.g., Szunyogh et al. 1997).These limitations of the Patil et al. (2001) study moti-vated Oczkowski et al. (2005) to repeat the calculationsof Patil et al. (2001) with much larger ensembles. Ocz-kowski et al. (2005), who also employed local energeticsdiagnostics to identify the atmospheric dynamical pro-cesses that led to the development of local low dimen-sionality, confirmed the earlier result that local low di-mensionality was often the result of strong local insta-bilities that led to the rapid growth of simple errorpatterns.

The study of Oczkowski et al. (2005) was also basedon a bred-vector ensemble. As mentioned earlier, themain problem with this approach is that extreme lowdimensionality tends to occur in the initial ensemble asa result of the ensemble generation technique. Themain goal of the present study is to investigate the rolethat changes in the complexity of the local dynamicsplay in predictability, using an ensemble of initial per-turbations that has high E dimension and is consistentwith the estimated analysis uncertainties. To achievethis goal, we take advantage of our previous work totest an implementation of the LEKF on the NCEPGlobal Forecast System (GFS) model (Szunyogh et al.2005, SEA05 hereafter). We investigate the evolutionof the E dimension and the role it plays in predictabilityin forecasts started from analysis ensembles of SEA05.For a 40-member bred-vector ensemble, the typical val-ues of the E dimension vary between 5 and 25 (Ocz-kowski et al. 2005), but for a 40-member LEKF en-semble the E dimension is never smaller than 25 and istypically larger than 30 (SEA05).

We carry out experiments for the perfect model sce-nario: a “true” nonlinear trajectory is generated by along integration of the model from a realistic NorthernHemisphere winter initial condition. Then, imperfect(perturbed) initial conditions are obtained by assimilat-ing simulated noisy observations of the true states withthe LEKF data assimilation system. An important fea-ture of the hypothetical observing network is that theobservations are randomly distributed. Thus, unlike areal observing network, the simulated observing net-work may be assumed to have no effect on the geo-graphical distribution of the analysis and forecast un-certainties (provided that the observational network isnot too sparse). Here, the focus is on the spatiotempo-

APRIL 2007 K U H L E T A L . 1117

ral evolution of the forecasts and the forecast uncer-tainties started from the analyses of SEA05. Althoughthe unique features of the LEKF algorithm make theclose relationship between local dimensionality, errorgrowth, and skill of the ensemble to capture the spaceof forecast uncertainties especially transparent, we be-lieve that our results could be reproduced with any suit-ably formulated ensemble-based Kalman filter scheme(e.g., Anderson 2001; Bishop et al. 2001; Houtekamerand Mitchell 2001; Evensen 2003; Keppenne and Rie-necker 2002; Whitaker and Hamill 2002). In addition,we hope that our results help strengthen the theoreticalfoundation of the operational practice of using smallensembles to predict the evolution of uncertainties inhigh-dimensional operational numerical weather pre-diction models (e.g., Kalnay 2003).

The analysis–forecast system used in our experi-ments, as well as the experimental design, is describedbriefly in section 2. Section 3 investigates the geo-graphical distribution and typical evolution of the fore-cast errors. This section also provides a detailed ac-count of a case of explosive error growth. Section 4investigates the relationship between the E dimension,forecast error growth, and the skill of the ensemble intracking the space of the spatiotemporally evolvingforecast uncertainties. Section 5 is a summary of ourmain conclusions.

2. Experimental design

The LEKF scheme is a model-independent algorithmto estimate the state of a large spatiotemporally chaoticsystem (Ott et al. 2004). The term “local” refers to animportant feature of the scheme: it solves the Kalmanfilter equations locally in model grid space. More pre-cisely, the state estimate at a grid point P is obtainedindependently from the state estimate at the other gridpoints, considering the observations and the back-ground state only from a local cube centered at P. TheLEKF scheme also provides an estimate of the analysisuncertainty at P and generates an ensemble of analysisperturbations that represent the estimated uncertaintyat P. When the LEKF is applied to the assimilation ofobservations of a perfect model, we use a 4% multipli-cative variance inflation (Anderson and Anderson1999) at each analysis step to increase the estimatedanalysis uncertainty to compensate for the loss of en-semble variance due to sampling errors and the effectsof nonlinearities. In addition to the variance inflationcoefficient, the scheme has two tunable parameters: thenumber of grid points in the local cube and the numberof ensemble members.

Here, as well as in SEA05, the LEKF is implemented

on a reduced-resolution version of the 2001 operationalimplementation of the NCEP GFS model. With theexception of the resolution, which is reduced to T62 inthe horizontal direction and to 28 levels in the verticaldirection, the model we use is identical to the full op-erationally implemented version of the 2001 NCEPGFS (detailed documentation of the model can be foundonline at http://www.emc.ncep.noaa.gov/modelinfo).

A time series of true states was generated by a 60-dayintegration of the model starting from the operationalNCEP analysis at 0000 UTC 1 January 2000. The twocomponents of the horizontal wind vector and the tem-perature were observed at all model levels, and theassociated surface pressure was also observed. The as-sumed observational errors were normally distributedwith zero mean and standard deviations of 1 m s�1, 1 K,and 1 hPa, respectively. Initially, observations weregenerated at all 17 848 horizontal gridpoint locations.Then, reduced observational networks were created bygradually removing observational locations at ran-domly selected grid points. This approach was appliedto construct three additional observational networksthat take vertical soundings of the atmosphere at 2000,1000, or 500 fixed locations every 6 h.

In what follows, we investigate the subsequent evo-lution of the distribution of the forecast errors. Most ofthe results presented here are for a configuration of theLEKF that consists of a 40-member ensemble, 7 � 7 �� gridpoint local cubes (� is the number of vertical gridpoints in the cube and changes with altitude; see SEA05for details), and 2000 simulated vertical soundings. Wenote that seven grid points is equivalent to a distance of13.4° in the meridional direction and to a distance of13.1° in the zonal direction. The initial ensemble per-turbations are generated by adding random noise to theoperational NCEP background forecast, truncated tothe resolution used in this paper, at 0000 UTC 1 Janu-ary 2000. The distribution of the random noise is iden-tical to that of the simulated observations. That is, ex-cept for the effects of statistical fluctuations and trun-cation errors, the initial background is identical to theoperational NCEP background at 0000 UTC 1 January2000, and in the initial estimate of the background errorcovariance matrix the error variance is about the sameas the observational error variance, while the errors ofthe different variables at the different gridpoint loca-tions are uncorrelated.

Datasets

A state estimate is obtained every 6 h by assimilatingthe simulated observations with the LEKF scheme. De-terministic forecasts are started from the 0000, 0600,1200, and 1800 UTC ensemble mean analyses each day.


An ensemble of forecasts is also started every 12 h,using the analysis ensemble provided by the LEKF asthe initial conditions. Forecast error statistics are gen-erated by comparing the deterministic forecasts to thetrue states. (The only exceptions are the results pre-sented in section 4c, where the ensemble-mean forecastis compared to the true states.) The forecast error sta-tistics are computed for the 40-day period that starts atthe 15th day along the true trajectory. We refer to timeusing the 40-day period as reference, that is, the firstforecast that we verify starts at 0000 UTC on day 1, andthe last forecast we verify starts at 1200 UTC on day 40.The model outputs are processed on a 2.5° � 2.5° reso-lution grid. We present error statistics in the followingformats:

• Snapshots of errors are presented by mapping thedifference between the forecast and the true state onthe grid.

• �aps of the time-mean absolute error are generatedby first computing the absolute value of the differ-ence between the forecasts and the true states at thegrid points, then computing the 40-day mean of theabsolute values.

• The error for a geographical region is obtained bycomputing the root-mean-square (rms) of the errorover all grid points in the geographical region. Plotsshowing time series of the errors are based on thisinformation. Errors are shown for three geographicalregions: NH extratropics (30°N–90°N), Tropics(30°S–30°N), and SH extratropics (90°S–30°S).

• The spectrally filtered errors for a geographical re-gion are obtained by first spectrally filtering the grid-point values along each latitude, based on the zonalwavenumbers, then computing the rms over the re-gion.

• The time-mean absolute error for a geographical re-gion is obtained by computing the 40-day mean of theroot-mean-square error for the given geographicalregion.

3. Evolution of the forecast errors

The simulations in SEA05 found that the largestwind and temperature analysis errors were in the mainregions of deep convection in the Tropics, while thesmallest analysis errors were found in the midlatitudestorm track regions. Figure 1 illustrates the rapidchange in the geographical distribution of the errors asthe forecasts progress, showing the time mean of theforecast errors for the meridional component of thewind vector at the 500-hPa level (the figure shows thetime mean over all 160 forecast cycles). There seems to

be a relationship between the errors in the region ofdeep convection and the early amplification of the er-rors in the North Pacific storm track region. Then theerrors propagate westward along the upper-tropo-spheric waveguides. Although a clear indication of rap-idly growing errors in the North Atlantic and SouthernHemisphere storm track regions can be seen first at the48-h forecast lead time, the storm track regions becomethe location of the dominant error patterns in the ex-tratropics by the 72-h forecast lead time.

a. Dependence on the geographical region

The difference between the error growth character-istics in the extratropics and the Tropics becomes ob-vious by investigating the time evolution of the root-mean-square forecast errors for the different geo-graphical regions (shown by closed squares in Figs. 2and 3). The most striking difference between the extra-tropics and the Tropics is in the functional dependenceof the error growth on the forecast lead time. (Noticethat although the vertical scale in Fig. 2 is logarithmic,the vertical scale in Fig. 3 is linear.) In the extratropics,the root-mean-square of the forecast error is approxi-mately an exponential function of the forecast lead timefor the first 72 h, that is, zf(t) � zaert, where the scalarr denotes the exponential error growth rate. Afterabout 72 h, the error growth starts slowing down, indi-cating an initial stage of nonlinear error saturation. Incontrast, in the Tropics, the root-mean-square of theforecast error, zf(t), is a linear function of the forecastlead time, that is, zf(t) � bt � za , where za � zf(0) is theroot-mean-square analysis error and the scalar b is thelinear error growth rate.

We obtain estimates of the parameters za and r bycalculating their values for the curves that best fit theforecast errors for the first 72 h in the least squaressense. Although the initial errors are very slightly largerin the SH extratropics (not shown) than in the NHextratropics (0.42 versus 0.39 m s�1), the forecast errorsgrow a bit more slowly in the SH (not shown) than inthe NH extratropics; the error doubling time T � r�1 ln2is 38.5 h in the SH extratropics versus 34.7 h in the NHextratropics.

The shape of the error growth curves indicates thatthe magnitude of the errors in the first 72 h is governedby the differential equation

dzf�t�dt � rzf�t, zf�0 � za �1

in the extratropics and by the differential equation

dzf�t�dt � b, zf�0 � za �2

in the Tropics.

APRIL 2007 K U H L E T A L . 1119

FIG. 1. Time-mean absolute error in forecasts of the meridional wind component at the 500-hPa pressure level atdifferent forecast lead times.


Fig 1 live 4/C

Interestingly, the functional dependence of the errorgrowth is independent of the spatial scale in both re-gions: except for the zonal mean term (k � 0), theinitial error grows exponentially for all wavenumberranges in the extratropics (Fig. 2), while the error growslinearly for all wavenumber ranges in the Tropics (Fig.3). The linear error growth rate b and the initial expo-nential growth rate r are larger for the wavenumberranges k � 1–10 and k � 11–20 than for the range k �21–40. Also, the errors tend to start saturating earlierfor the smaller scales (larger wavenumbers).

b. Dependence on the LEKF parameters

We have carried out experiments to test the sensitiv-ity of the forecast results to the free parameters of theanalysis scheme (results are not shown). We find that,within a reasonable range of the parameters, the fore-cast errors depend only weakly on the parameters.More precisely, the small initial differences betweenthe analyses for 5 � 5 � �, 7 � 7 � �, and 9 � 9 � � localcubes show negligible growth in the forecast phase.Likewise, for a 5 � 5 � � local region size, the advan-tage of the 80-member ensemble filter over the 40-member ensemble filter is negligible in the first 72 h.Since the dominant errors grow exponentially in theextratropics, our result shows that differences in theanalysis due to changes of the free parameters haveonly a very small projection on the dominant instabili-ties. This indicates that, when the parameters of theLEKF scheme are chosen from a reasonable range, thescheme can efficiently remove the growing error com-ponents. This is a nontrivial result since the schemecorrects errors that were growing before the analysistime, while the forecast errors are governed by errors

that are growing after the analysis time. An importantpractical consequence of the weak sensitivity to the tun-able parameters is that it greatly increases the general-ity of our predictability assessment.

c. Dependence on the number of observations

In sharp contrast to the aforementioned weak sensi-tivity to the tunable parameters, the observational den-sity has a significant influence on the accuracy of theforecasts. Increasing the number of observations sub-stantially improves the accuracy of the forecasts in allgeographical regions (results are not shown).

In the Tropics, the improvement is essentially con-stant in time, due to a weak dependence of the linearerror growth rate on the number of observations. Thisresult suggests that increasing the number of observa-tions in the Tropics leads to a reduction of the magni-tude of the forecast errors, but it does not change thecharacteristics of error growth. Likewise in the extra-tropics, the influence of the observational density onthe exponential error growth rate is modest, althoughthe error growth is slightly faster for the higher obser-vational density (Table 1).

d. Temporal variability of the forecast errors

Among the three geographical regions considered inthis paper, the temporal variability of the forecast er-rors is highest in the NH extratropics and lowest in theTropics (Fig. 4). The high variability in the NH extra-tropics is due to episodes of unusually large forecasterrors. The first such episode is a pattern of extremelylarge errors in forecasts started between 1200 UTC onday 4 and 0000 UTC on day 7. We find (results not

FIG. 3. Dependence of the time-mean forecast error on theforecast lead time for the meridional wind component at the 500-hPa level in the Tropics. The evolution of the forecast error isshown for different ranges of the zonal wavenumber k.

FIG. 2. Dependence of the time-mean forecast error on theforecast lead time for the meridional wind component at the 500-hPa level in the NH extratropics. The evolution of the forecasterror is shown for different ranges of the zonal wavenumber k.

APRIL 2007 K U H L E T A L . 1121

shown) that improving the accuracy of the analysis, byadding more observations and/or increasing the en-semble size, leads to minuscule reductions in the fore-cast errors at these verification times. This indicatesthat the unusually large forecast errors in this case aremore associated with low predictability of the atmo-spheric states than with the accuracy of the analyses.An inspection of the atmospheric flow regimes revealsthat the relatively low predictability of these states isassociated with the rapid amplification of errors in thepresence of an unusually strong jet stream in the NorthAtlantic storm track region (further details on thisevent are provided in sections 3e and 4b).

The second episode involves a pattern of unusuallylarge analysis errors between about day 16 and day 24,which lead to a proportionally elevated level of forecasterrors at the associated verification times. An inspec-tion of the spatiotemporal evolution of the errors forthis period (not shown) reveals that the relatively largeerrors are due to exceptionally large analysis error inthe region of Indonesia that later propagate into theNH extratropics. The visible propagation of the time-mean forecast errors from the Tropics to the extratrop-ics shown in Fig. 1 is associated with this episode.

e. A case of explosive error growth

To gain a better understanding of the processes thatlead to the explosive error growth in the aforemen-tioned first episode, we select the forecast started at1200 UTC on day 6 for further inspection. Maps of theforecast errors show that the explosive error growth atthe 36-h lead time occurs in a very localized region offthe coast of Newfoundland (Fig. 5).

For the next 24 h, the dominant error pattern is char-acterized by an eastward-propagating, rapidly amplify-ing dipole structure. This structure and the fast propa-gation speed indicate that the dominant error patterntakes the shape of a packet of synoptic-scale Rossbywaves. This conclusion can be confirmed by calculatingthe packet envelope of the forecast errors for the 4- to9-wavenumber range with a Hilbert transform-based

method (Zimin et al. 2003, 2006). Using the techniqueof Zimin et al. (2006), Fig. 6 depicts an amplifying east-ward-extending envelope of errors. An inspection ofthe vertical cross section of the errors (not shown) alsoconfirms that the error growth starts in the jet layerwith an overestimation of the wind speed in the core ofthe jet and a small distortion of the upper-troposphericwave near the core of the jet. Although downstreamdevelopment [an initial divergence of the ageostrophicfluxes that triggers a baroclinic energy conversion; seeOrlanski and Chang (1993) and Orlanski and Sheldon(1995)] leads to the development of a closed low asso-ciated with the upper-tropospheric wave, the largestforecast errors occur further downstream, near theleading edge of the wave packet shown in Fig. 6. Suchpropagation of the dominant errors was documentedand analyzed in detail in Persson (2000), Szunyogh etal. (2000, 2002), Zimin et al. (2003), and Hakim (2005)and was foreseen long ago by the pioneers of numericalweather prediction (Rossby 1949; Charney 1949; Phil-lips 1990).

FIG. 4. Time series of the root-mean-square forecast error fordifferent forecast lead times. Shown is the forecast error for themeridional wind component at the 500-hPa level.

TABLE 1. NH extratropics root-mean-square analysis error, za,and error doubling time for the meridional wind component at the500-hPa level at different observational densities. While these val-ues are slightly different for the other model variables, they showthe same tendencies.

No. of soundings Rms analysis error Error doubling time

All locations 0.29 m s�1 33.3 h2000 locations 0.39 m s�1 34.7 h1000 locations 0.48 m s�1 36.7 h

500 locations 0.64 m s�1 38.9 h


In our example of rapid error growth, the atmo-spheric instability that drives the propagation of theerrors is a growing uncertainty in the characteristics(phase and amplitude) of finite amplitude waves gen-erated by an earlier downstream baroclinic develop-

ment. (Here the term “instability” is used in the math-ematical sense, i.e., it refers to a growing uncertainty inthe solution due to an uncertainty in the initial condi-tion.) The potential importance of an instability pro-cess, in which an earlier baroclinic or barotropic insta-

FIG. 5. Time evolution of the errors in the forecast started at 1200 UTC on day 7. Shown are the errors (color shades) and the“true” state of the geopotential height of the 500-hPa pressure level.

APRIL 2007 K U H L E T A L . 1123

Fig 5 live 4/C

bility leads to uncertainties in the characteristics of thedeveloping finite-amplitude waves, was first pointedout by Snyder (1999). That the dominant errors propa-gate along the upper-tropospheric waveguides (Fig. 1and related discussion earlier) suggests that this may bethe most important instability in the model solutions(forecasts). The importance of this instability process,in which temporal evolution and spatial propagationplay equally important roles, reinforces our view thatthe atmosphere should always be approached as a spa-tiotemporally chaotic system.

4. The role of local dimensionality

SEA05 found that the efficiency of the LEKF algo-rithm was inversely proportional to the E dimension.More precisely, a strong negative correlation was found

between the gridpoint values of the time-mean E di-mension and the gridpoint values of the time mean ofthe explained variance. The explained variance mea-sures the portion of the error that is captured by theensemble. In what follows, we investigate the relation-ship between E dimension, explained variance, and themagnitude of forecast errors.

a. E dimension, explained variance, and forecasterror

While the choice of the coordinates of the state vec-tor does not affect the state estimates, it has a profoundeffect on the singular value decomposition (SVD) ofthe error covariance matrices. Thus, the choice of co-ordinates has an important effect on such SVD-baseddiagnostics as the E dimension. We follow the strategyof Oczkowski et al. (2005) and transform the ensemble

FIG. 6. Time evolution of the wave packet envelope of errors in the forecast started at 1200 UTC on day 6. The wave packet envelopeis calculated based on errors in the prediction of the meridional component of the wind vector in the zonal wavenumber range from4 to 9. Notice the change in the color scheme between the 36- and 48-h forecast lead times.


Fig 6 live 4/C

perturbations so that the square of the Euclidean normof the transformed perturbations has dimensions of en-ergy. The local state vector is defined by all gridpointvariables in a local volume that contains a 5 � 5 hori-zontal grid (at 2.5° resolution in both direction) and theentire model atmosphere in the vertical direction.

This definition of the local state vector differs fromthat used for the calculation of the E dimension inSEA05. There, the local volume was defined by thelocal volume used in the LEKF algorithm, in whichonly a few model levels were included in the verticaldirection and the number of model levels in the verticallayers was height dependent. The rationale for thischange is that in SEA05 the goal was to evaluate theassumptions made in the implementation of the LEKFon the NCEP GFS; here, the goal is to study the role oflocal dimensionality in shaping the local predictability.

As expected based on the results of SEA05, the Edimension is typically higher in the Tropics (Fig. 7) thanin the extratropics for the entire 5-day forecast range.While the E dimension decreases with increasing fore-cast time over the entire globe, the decrease of thedimension is much faster in the storm track regionsthan elsewhere. One may wonder whether this effect isassociated with an inherent property of the model dy-namics or arises from an unexpected collapse of theensemble due to some unforeseen problem with theensemble generation technique. To answer this ques-tion, we apply the explained variance diagnostic (seeSEA05) to the forecast error and the forecast en-semble. The explained variance diagnostic measuresthe portion of the forecast error that lies in the spacespanned by the evolving ensemble perturbations. (For-mally, it is calculated by projecting the forecast error onthe space of the ensemble, then taking the square of theprojection, which is finally normalized by the square ofthe forecast error to obtain the measure). In the ex-treme cases, when the ensemble perfectly captures thespace in which the forecast error evolves, the explainedvariance is one, and when the forecast error falls en-tirely outside of the ensemble space, the explained vari-ance is zero. The close relationship between the typicalregions of low dimensionality and the typical regions ofhigh explained variance can be deduced subjectively bycomparing Figs. 7 and 8. This observation motivates usto assess the relationship between the two quantities ina more quantitative way. In addition, we would like toknow whether such a strong relationship exists only forthe temporal means of the two quantities or whetherone is also present for the spatiotemporally evolvingfields. To achieve these two objectives, we study thejoint probability distribution of the E dimension andthe explained variance in the NH extratropics (Fig. 9)

and the Tropics (Fig. 10). (The joint probability distri-bution for the SH extratropics is similar to that for theNH extratropics, thus it is not shown.)

The joint probability distribution function is obtainedby counting the number of cases when a pair of valuesfor the E dimension and the explained variance fallsinto a bin defined by a small interval E of the E di-mension and a small interval EV of the explained vari-ance. Then the number of cases is normalized by E �EV � n and the bin is color shaded based on theresult. The total sample size n is equal to the total num-ber of grid points in the given geographical region timesthe number of verification times, 160, on which thesample is based. This normalization ensures that theintegral of the plotted values over all bins is equal toone.

The most important common feature of the jointprobability for the NH extratropics and Tropics is thatthe smaller the E dimension, the larger the possiblesmallest value of the explained variance. In otherwords, the lower the E dimension, the higher the con-fidence we can have that the ensemble captures theactual forecast error. In addition, as the forecast timeincreases, the lowest possible value of the E dimensiondecreases, and the lowest values of the E dimensionbecome an increasingly sharper predictor of a high ex-plained variance. We also note that the boundary be-tween the NH extratropics and the Tropics is not sharp:when the two figures are merged (not shown) there isno visible jump in the probability distribution, becausethe high E dimension end of the distribution for the NHextratropics and the low E dimension end of the distri-bution for the Tropics are populated by values from thetransient region between the two areas.

What makes the close relationship between low Edimension and high explained variance potentiallyvaluable from a forecasting point of view in the extra-tropics is that fast error growth always leads to low Edimension. (We note that the opposite is not true, theforecast error can be small for a case of low E dimen-sion at any forecast time.) That is, we can have thehighest confidence in the ability of the ensemble topredict the space of possible errors, when the errors arethe largest. This property of the ensemble is illustratedby Figs. 11 and 12. Figure 11 shows the joint probabilitydistribution for the analysis and forecast errors and theexplained variance in the NH extratropics. It can beseen that as the forecast lead time increases, the en-semble captures an increasingly larger portion of theforecast errors for the cases of large errors. This can beexplained by the fact that the fast error growth alwaysleads to low E dimension, that is, to high explainedvariance (Fig. 12).

APRIL 2007 K U H L E T A L . 1125

FIG. 7. Time-mean E dimension at different forecast lead times.


Fig 7 live 4/C

FIG. 8. Time-mean explained variance at different forecast lead times.

APRIL 2007 K U H L E T A L . 1127

Fig 8 live 4/C

FIG. 9. Joint probability distribution of the E dimension and the explained variance in the NH extratropics. Thebins are defined by E � 0.2 and EV � 0.005.


Fig 9 live 4/C

FIG. 10. Joint probability distribution of the E dimension and the explained variance in the Tropics. The binsare defined by E � 0.2 and EV � 0.005.

APRIL 2007 K U H L E T A L . 1129

Fig 10 live 4/C

FIG. 11. Joint probability distribution of the explained variance and the magnitude of the error in theforecast of the meridional component of the wind at the 500-hPa level in the extratropics. The bins are definedby E � 0.005 and ER � 0.4, where ER is the interval for the forecast error.


Fig 11 live 4/C

FIG. 12. Mean E dimension for the bins shown in Fig. 11.

APRIL 2007 K U H L E T A L . 1131

Fig 12 live 4/C

The picture is very different for the Tropics (Figs. 13and 14). In this region, the magnitude of the forecasterror is more directly related to the magnitude of theanalysis error due to the linear nature of the errorgrowth. Since the analysis errors are smaller for thelower E dimensions, the forecast errors are also smallfor the low E dimensions. (We can start seeing a shift ofthe larger errors toward the smaller E dimensions onlyafter 72 h.) Thus the highest explained variance occursfor relatively small errors.

b. Local low dimensionality and explosive localerror growth

So far we have shown that there is a close statisticalrelationship between E dimension, explained variance,and forecast error. Here we illustrate this close rela-tionship using the example of the explosive forecasterror growth described in section 3e. In this case, theoverlap between the regions of large errors and lowdimensionality is almost perfect (Fig. 15), especially atand after the 36-h forecast lead time. Likewise, the ex-plained variance rapidly grows in the regions of rapidlydecreasing dimensionality, where the explained vari-ance exceeds 90% at and beyond the 24-h forecast leadtime (Fig. 16).

c. Local low dimensionality and the spread–skillrelationship

It has been long thought that the spread (the secondmoment) of a suitably prepared ensemble forecast canbe used as a predictor of the skill of the ensemble-meanforecast (Leith 1974). It has also been observed, how-ever, that the positive correlation between the spreadand the forecast error is disappointingly small; even inthe perfect model scenario, the correlation was foundto be less than 0.5 (Barker 1991). The theoretical ex-planation for this result was provided by Houtekamer(1993) and Whitaker and Loughe (1998) using a simplestochastic model of the spread–skill relationship: alarge correlation can be expected only when the tem-poral variability of the forecast (or analysis) error islarge. This rule explains the behavior of the spread–skill relationship for the LEKF system shown in Fig. 17:(i) initially the correlation increases due to the increas-ing variability of the forecast errors as the forecast timeincreases (see Fig. 4); (ii) the correlation peaks at alevel slightly below 0.5 at the 72-h forecast lead time inall three geographical regions; and (iii) the maximumvalue of the correlation is the largest in the NH extra-tropics, the region where variability of the forecast er-rors is the largest. The low initial correlation can beexplained by the fact that an ensemble-based data as-

similation system, such as the LEKF, is designed toremove that part of the analysis error that is success-fully captured by the ensemble. The only surprising fea-ture in Fig. 17 is the relatively high initial correlation inthe Tropics. The only plausible explanation for this isthat in the Tropics, the location of the dominant analy-sis errors is better captured by the ensemble than thestructure of the errors. This result reinforces our earlierconjecture, drawn in section 3c, that the assimilation ofobservations in the Tropics reduces the magnitude ofthe errors in the state estimation but does not changedrastically the structure of the errors. This indicatesthat there is no strong relationship between errors atthe different grid points in the Tropics.

The joint probability distribution function for the en-semble spread and the error in the ensemble-meanforecasts is shown in Fig. 18. This figure indicates thatthe ensemble spread is typically smaller than the errorin the ensemble mean. This finding is not unexpected,since as was shown earlier (e.g., Fig. 8), part of theforecast error is not captured by the ensemble. (Forshort forecast lead times, the ensemble-mean forecastand the forecast started from the analysis mean arenearly identical due to the nearly linear initial evolutionof the ensemble perturbations.) In addition, the en-semble spread predicts the upper bound of the errormost reliably at locations where the E dimension is thesmallest (Fig. 19). In contrast to the case of the singledeterministic forecast, where the largest errors occurfor the smallest E dimensions, the errors in the en-semble-mean forecast are relatively small in the regionsof the smallest E dimensions. This is due to the efficienterror-filtering effects of ensemble averaging in regionswhere the ensemble efficiently captures the space ofuncertainties, that is, in regions of high explained vari-ance.

5. Conclusions

In this paper, we assess atmospheric predictabilitywith the help of a state-of-the-art numerical weatherprediction model (at a reduced resolution) and the localensemble Kalman filter data assimilation scheme. Ourexperimental design addresses the issue of determiningthe degree to which uncertainty in the knowledge of theinitial state influences the predictability of a high-dimensional, spatiotemporally chaotic system. We as-sume that the numerical model provides a perfect rep-resentation of the true atmospheric dynamics. Ourmain findings are as follows:

• For this specific choice of the model and data assimi-lation system, the forecast errors grow exponentially


FIG. 13. Joint probability distribution of the explained variance and the magnitude of the error in the forecastof the meridional component of the wind at the 500-hPa level in the Tropics. The bins are defined by E � 0.005and ER � 0.4.

APRIL 2007 K U H L E T A L . 1133

Fig 13 live 4/C



Fig 14 live 4/C

in the extratropics and linearly in the Tropics. Asexponential growth has been found in many previousstudies that considered different types of uncertain-ties in the knowledge of the true initial conditions,the dominance of exponentially growing featuresseems to be an important property of predictability inthe extratropics. Our earlier research indicates thatthese dominant instabilities are closely related to thesynoptic-scale local generation and propagation ofthe eddy kinetic energy. Since these processes can bewell simulated by the models, there are good reasonsto believe that exponentially growing instabilitiesdominate real atmospheric dynamics in the extra-tropics. The linear growth of errors in the Tropics isa more unique result of our experiments. While thisresult may be an artifact of the model dynamics,which rely heavily on parameterized physical pro-

cesses in the Tropics, we tend to believe that the realatmosphere behaves similarly.

• The explained variance is always highest for the low-est E dimension, independently of the geographicalregion and the forecast lead times. (As was shown inSEA05, this guarantees that the analysis errors arethe smallest for the smallest E dimension indepen-dently of the geographical region.)

• In the extratropics, large forecast errors gradually be-come more likely to occur in regions of low E dimen-sion as the forecast time increases. Thus, the en-semble gradually becomes more likely to capture alarge portion of the forecast error as the forecast timeincreases. The larger the forecast error, the larger theportion of the forecast error that the ensemble cap-tures with high certainty.

• Since the ensemble captures a larger portion of the

FIG. 15. Shown are the E dimension (color shades) and the geopotential height forecast error at the 500-hPa level in the forecastsstarted at 1200 UTC on day 6.

APRIL 2007 K U H L E T A L . 1135

Fig 15 live 4/C

forecast error with high certainty in the regions oflow E dimension, in those regions ensemble averag-ing becomes an efficient error filter and the ensemblespread provides an accurate prediction of the upperbound of the error in the ensemble-mean forecast.

• In the Tropics, due to the linear error growth, themagnitude of the forecast error is closely tied to themagnitude of the analysis error. Since the analysiserrors are small for the smallest E dimensions, theforecast errors are also small for the smallest E di-mensions. In our experiments, this pattern startsbreaking up beyond a forecast lead time of 72 h.

Do these results have any practical use when theforecast model is not perfect? First of all, it is safe toassume that the local dimensionality of the true atmo-sphere is higher than in our global forecast model. This

FIG. 16. Shown are the E dimension (color shades) and the explained variance (contours) in the forecasts started at 1200 UTC onday 6. The contour interval is 0.1 and values smaller than 0.7 are not shown.

FIG. 17. Correlation between ensemble spread and error in theensemble-mean forecast as a function of forecast time.


Fig 16 live 4/C

FIG. 18. Joint probability distribution of the ensemble spread and the magnitude of the error in the ensemble-mean forecast of the meridional component of the wind at the 500-hPa level in the NH extratropics. The widthof the bins is 0.005 for the ensemble spread and 0.4 for the forecast error.

APRIL 2007 K U H L E T A L . 1137

Fig 18 live 4/C



Fig 19 live 4/C

would degrade the ability of the model-based ensembleto capture the space of forecast uncertainties. We notethat, in principle, the LEKF algorithm could be used toestimate the effect of forecast errors on the E dimen-sion. The extension of the LEKF algorithm described inBaek et al. (2006) provides an estimate of the modelerrors in addition to the estimate of the state. Moreprecisely, it provides an estimate of the augmentedstate, where the state is augmented by the parametersthat describe the model errors. The E dimension couldbe determined by using the augmented state to definethe local background covariance matrix. It is yet to beseen, however, whether the model errors can be effi-ciently parameterized for a complex weather predictionmodel, such as the NCEP GFS.

Local low dimensionality is a property that eventu-ally breaks down with increasing forecast lead time.Eventually, predictability is completely lost, and thepredictive value of the ensemble becomes the same asthat of a set of randomly drawn samples from the muchlarger set of climatologically realizable states of themodel. The larger the magnitude of the initial ensembleperturbations, the earlier the breakdown of local lowdimensionality occurs. For instance, Oczkowski et al.(2005) observed such breakdowns at forecast lead timesof as little as 24 to 48 h when investigating the evolutionof a set of bred vectors. In our experimental design, themagnitude of the analysis uncertainty is small (presum-ably an order of magnitude smaller than in an opera-tional weather analysis), so our results are not affectedby an overall breakdown of local low dimensionality inthe first 120 h of model integration. Our plan is toinvestigate the process of the breakdown of low dimen-sionality in a future paper for both simulated and realobservations.

Acknowledgments. The authors thank one of theanonymous reviewers for insightful comments. D. K.was supported by a Rising Star Fellowship from theNational Institute for Aerospace, Hampton, Virginia.This work was also supported by a National Oceanicand Atmospheric Administration THORPEX Grant,the Army Research Office, a James S. McDonnell 21stCentury Research Award, the NPOESS IntegratedProgram Office (IPO), the Office of Naval Research(Physics), and the National Science Foundation(Grants 0104087 and PHYS 0098632). E. J. K. grate-fully acknowledges support from the National ScienceFoundation (Grant DMS-0408102).

REFERENCES

Anderson, J. L., 2001: An ensemble adjustment filter for dataassimilation. Mon. Wea. Rev., 129, 2884–2903.

——, and S. L. Anderson, 1999: A Monte Carlo implementationof the nonlinear filtering problem to produce ensemble as-similations and forecasts. Mon. Wea. Rev., 127, 2741–2758.

Baek, S.-J., B. R. Hunt, E. Kalnay, E. Ott, and I. Szunyogh, 2006:Local ensemble Kalman filtering in the presence of modelbias. Tellus, 58A, 293–306.

Barker, T. W., 1991: The relationship between spread and forecasterror in extended-range forecasts. J. Climate, 4, 733–742.

Bishop, C. H., B. J. Etherton, and S. Majumdar, 2001: Adaptivesampling with the ensemble transform Kalman filter. Part I:Theoretical aspects. Mon. Wea. Rev., 129, 420–436.

Charney, J. G., 1949: On a physical basis for numerical predictionof large-scale motions in the atmosphere. J. Meteor., 6, 371–385.

Evensen, G., 2003: The ensemble Kalman filter: Theoretical for-mulation and practical implementation. Ocean Dyn., 53, 343–367.

Hakim, G. J., 2005: Vertical structure of midlatitude analysis andforecast errors. Mon. Wea. Rev., 133, 567–578.

Houtekamer, P. L., 1993: Global and local skill forecasts. Mon.Wea. Rev., 121, 1834–1846.

——, and H. L. Mitchell, 2001: A sequential ensemble Kalmanfilter for atmospheric data assimilation. Mon. Wea. Rev., 129,123–137.

Kalnay, E., 2003: Atmospheric Modeling, Data Assimilation, andPredictability. Cambridge University Press, 341 pp.

Keppenne, C. L., and M. M. Rienecker, 2002: Initial testing of amassively parallel ensemble Kalman filter with the Poseidonisopycnal ocean general circulation model. Mon. Wea. Rev.,130, 2951–2965.

Leith, C. E., 1974: Theoretical skill of Monte Carlo forecasts.Mon. Wea. Rev., 102, 409–418.

Oczkowski, M., I. Szunyogh, and D. J. Patil, 2005: Mechanisms forthe development of locally low-dimensional atmospheric dy-namics. J. Atmos. Sci., 62, 1135–1156.

Orlanski, I., and E. K. M. Chang, 1993: Ageostrophic geopotentialfluxes in downstream and upstream development. J. Atmos.Sci., 50, 212–225.

——, and J. P. Sheldon, 1995: Stages in the energetics of baroclinicsystems. Tellus, 47A, 605–628.

Ott, E., 2002: Chaos in Dynamical Systems. 2d ed. CambridgeUniversity Press, 490 pp.

——, and Coauthors, 2004: A local ensemble Kalman filter foratmospheric data assimilation. Tellus, 56A, 415–428.

Patil, D. J., B. R. Hunt, E. Kalnay, J. A. Yorke, and E. Ott, 2001:Local low dimensionality of atmospheric dynamics. Phys.Rev. Lett., 86, 5878–5881.

Persson, A., 2000: Synoptic-dynamic diagnosis of medium rangeweather forecast systems. Proc. Seminars on Diagnosis ofModels and Data Assimilation Systems, Reading, UnitedKingdom, ECMWF, 123–137.

Phillips, N. A., 1990: Dispersion processes in large-scale weatherprediction. WMO-No. 700, World Meteorological Organiza-tion, 126 pp.

Rossby, C.-G., 1949: On the dispersion of planetary waves in abarotropic atmosphere. Tellus, 1, 54–58.

Snyder, C., 1999: Error growth in flows with finite-amplitudewaves or coherent structures. J. Atmos. Sci., 56, 500–506.

Szunyogh, I., E. Kalnay, and Z. Toth, 1997: A comparison ofLyapunov and optimal vectors in a low-resolution GCM. Tel-lus, 49A, 200–227.

——, Z. Toth, R. E. Morss, S. J. Majumdar, B. J. Etherton, andC. H. Bishop, 2000: The effect of targeted dropsonde obser-

APRIL 2007 K U H L E T A L . 1139

vations during the 1999 Winter Storm Reconnaissance Pro-gram. Mon. Wea. Rev., 128, 3520–3537.

——, ——, A. V. Zimin, S. J. Majumdar, and A. Persson, 2002:Propagation of the effect of targeted observations: The 2000Winter Storm Reconnaissance Program. Mon. Wea. Rev.,130, 1144–1165.

——, E. J. Kostelich, G. Gyarmati, D. J. Patil, B. R. Hunt, E.Kalnay, E. Ott, and J. Yorke, 2005: Assessing a local en-semble Kalman filter: Perfect model experiments with theNCEP global model. Tellus, 57A, 528–545.

Toth, Z., and E. Kalnay, 1993: Ensemble forecasting at NMC: Thegeneration of perturbations. Bull. Amer. Meteor. Soc., 74,2317–2330.

——, and ——, 1997: Ensemble forecasting at NCEP and thebreeding method. Mon. Wea. Rev., 125, 3297–3319.

Whitaker, J. S., and A. F. Loughe, 1998: The relationship betweenensemble spread and ensemble mean skill. Mon. Wea. Rev.,126, 3292–3302.

——, and T. H. Hamill, 2002: Ensemble data assimilation withoutperturbed observations. Mon. Wea. Rev., 130, 1913–1924.

Zimin, A. V., I. Szunyogh, D. J. Patil, B. R. Hunt, and E. Ott,2003: Extracting envelopes of Rossby wave packets. Mon.Wea. Rev., 131, 1011–1017.

——, ——, ——, ——, and ——, 2006: Extracting envelopes ofnonzonally propagating Rossby wave packets. Mon. Wea.Rev., 134, 1329–1333.


Assessing Predictability with a Local Ensemble Kalman Filter

Documents