Seamless precipitation prediction skill comparison between ...

Quarterly Journal of the Royal Meteorological Society Q. J. R. Meteorol. Soc. (2016) DOI:10.1002/qj.2928

Seamless precipitation prediction skill comparison between twoglobal models

Matthew C. Wheeler,a Hongyan Zhu,a Adam H. Sobel,b Debra Hudsona and Frederic Vitartc

aBureau of Meteorology, Melbourne, Victoria, AustraliabDepartment of Applied Physics and Applied Mathematics, Lamont-Doherty Earth Observatory, Columbia University, New York, NY,

USAcEuropean Centre for Medium-Range Weather Forecasts, Reading, UK

*Correspondence to: M. C. Wheeler, Bureau of Meteorology, GPO Box 1289, Melbourne, Victoria 3001, Australia.E-mail: [email protected]

The prediction of precipitation by two ocean–atmosphere ensemble systems is comparedwith observations and each other over a broad range of time-scales. The systems are the 2015version of the Australian Bureau of Meteorology’s Predictive Ocean–Atmosphere Modelfor Australia (POAMA) with a T47 atmosphere, and the 2011 version of the EuropeanCentre for Medium-range Weather Forecasts’ (ECMWF) monthly system with a variableatmospheric resolution of T639 to T319. To facilitate the comparison across a seamlessrange of time-scales, verification against observations is performed using data averaged overtime windows equal in length to the forecast lead time, from 1 day to 4 weeks. In addition tothis ‘actual’ skill, potential skill is computed by taking one ensemble member as truth andcomputing how well the other members forecast that member.

Overall, ECMWF shows higher actual skill than POAMA across all time-scales and inboth the Tropics and extratropics, as expected given its greater sophistication. ECMWF isparticularly more skilful than POAMA in the Tropics for the shorter leads. Consistentbetween the two systems, however, is that as lead time and averaging window aresimultaneously increased the near-equatorial skill remains approximately constant, whereasit drops in all other latitude bands. As a result, both systems show much higher skill in theTropics than extratropics for the 1 week time-scale and beyond, with that skill concentratedover the equatorial Pacific.

Although potential skill in both systems is almost everywhere higher than their actualskill, there remains a strong similarity in the spatial patterns of potential and actual skillfor the longer time-scales. Within-model comparisons of potential and actual skill showlargest differences for POAMA in the Tropics at short lead times, and largest differences forECMWF in the Southern Hemisphere high latitudes (50–70◦S). The implications of thesefindings are discussed.

Key Words: seamless prediction; seamless verification; prediction skill; potential skill; POAMA; ECMWF; GPCPprecipitation

Received 20 January 2016; Revised 20 September 2016; Accepted 26 September 2016; Published online in Wiley OnlineLibrary

1. Introduction

Previously, Zhu et al. (2014) examined the skill of the PredictiveOcean–Atmosphere Model for Australia (POAMA) for predictingprecipitation over a large range of time-scales (from 1 day to∼1 month). It was shown that the highest skill, as measured bythe correlation of the model ensemble mean with observations,was found in the extratropics at lead times of a few days or less,but as the lead time increases extratropical skill decreased rapidlywhereas tropical skill did not, so that at lead times of a week or

more the highest skill was found in the Tropics. At a finer levelof detail, skill was found to also vary regionally and seasonally,depending on the varying influence of predictable modes andphenomena.

Zhu et al. (2014) also advocated the usefulness of anew ‘seamless’ verification approach whereby the verificationaveraging window is increased at the same rate as the forecastlead time. Comparisons with skill from persistence allowed forfurther physical understanding of the sources of skill as wellas for the identification of model system deficiencies. However,

c© 2016 Royal Meteorological Society

M. C. Wheeler et al.

Table 1. Summary of the coupled ocean–atmosphere forecast systems and hindcasts used in this analysis.

System abbreviation Atmospheric model resolution Ensemble size Start dates used Hindcast period used

ECMWF T639 L62 (≤10 d) T319 L62 (>10 d) 15 1 February, 1 May, 1 August, 1 November 1997–2008POAMA T47 L17 33 As above As above

Although more hindcasts were available, for a meaningful comparison we chose to use only hindcasts from the exact same dates from each system and only used thosefor which the GPCP observations were also available. All skill calculations were performed on the coarser ∼2.5◦ grid of the POAMA system.

questions remained on whether the results were model dependent,especially given the relatively low atmospheric resolution (T47) ofPOAMA.

Questions also remained on how the results of Zhu et al. (2014)would look with model potential (or perfect) skill, calculated bytaking one of the ensemble members as an example reality andseeing how well the mean of the remaining members forecaststhat member. This is a commonly computed statistic (e.g. Beckeret al., 2013; Boer et al., 2013; Holland et al., 2013) that is a measureof the upper bound of realizable skill in the hypothetical modelworld, thereby providing further information about a modellingsystem, although it has its limitations (Kumar et al., 2014).

In this article we therefore extend the results of Zhu et al.(2014) in two ways: (i) by including a higher-resolutionglobal prediction system in the comparison of skill; and (ii)by including computations of model potential skill in thecomparison.

As in Zhu et al. (2014), one of the models used is theBureau of Meteorology’s POAMA version 2M. POAMA wasoriginally developed as a seasonal prediction system and iscompetitive with other international models on that range(Barnston et al., 2012; Langford and Hendon, 2013), with asophisticated ensemble generation strategy (Hudson et al., 2013).However, its atmospheric resolution is relatively low with anapproximately 250 km grid size and 17 vertical levels. Wetherefore compare its skill with a system employing much higheratmospheric resolution: that of the European Centre for Medium-range Weather Forecasts (ECMWF) ‘monthly’ forecasting system(Vitart, 2004, 2014), with a variable horizontal resolution of30–60 km and 62 vertical levels. By comparing hindcasts fromvery different systems, the generality of the previous results ontropical versus extratropical skill can be tested.

Comparisons are also made with model potential skill, bothbetween the models and with the actual skill. These comparisonswith potential skill require careful interpretation since potentialskill is not only related to the existence of a predictable signal,but is also related to the spread of the model ensemble, whetheror not this spread is realistic or not. However, we argue thatthe locations and time-scale of largest difference between actualand potential skill can help guide future system development,although defining the pathway to system improvement isdifficult.

2. Data and method

2.1. POAMA-2M forecast system

POAMA version 2M (Hudson et al., 2013) currently producesthe Bureau of Meteorology’s operational monthly and seasonalforecasts, and is expected to be replaced in 2017. Severalimprovements incorporated in this version of POAMA have madeit applicable for forecasting the weekly time-scale as well (Hudsonet al., 2013). These improvements include the use of perturbedatmosphere and ocean initial conditions generated from a coupledbreeding scheme (which samples initial condition uncertainty andwas found to improve forecast reliability compared to using asimple time-lagged ensemble; Hudson et al., 2013), as well asthe use of three different model configurations so as to sampleforecast uncertainty due to model errors.

The atmospheric component of this POAMA system is aspectral model with resolution T47 (∼250 km grid) and 17

vertical levels. The ocean component has a zonal resolutionof 2◦ and a varying meridional resolution of 0.5◦ –1.5◦ with 25vertical levels. The unperturbed initial conditions are providedby separate data assimilation schemes for the ocean (Yin et al.,2011) versus the atmosphere and land (Hudson et al., 2011). Forthe atmosphere and land, these initial conditions are createdby nudging towards the 40-year ECMWF Re-Analysis (ERA-40:Uppala et al., 2005) for 1980 to August 2002, and to the Bureau’soperational global numerical weather prediction (NWP) analysisthereafter.

Perturbations to the initial conditions of the central memberare generated using a coupled breeding scheme. Ten perturbedstates are produced, giving 11 different initial states that are inputto three different configurations of the model, differentiatedby their use of a different convective parametrization or fluxcorrection, providing a 33-member ensemble (Hudson et al.,2013). This description applies to both the hindcasts and real-timeforecasts.

As described in Zhu et al. (2014), this modelling systemcontains some representation of the main sources of pre-dictability of weather and climate (out to several seasons),but it does not include the observed increasing atmosphericconcentration of CO2 (it is fixed at 345 ppm; Hope et al.,2015), and also uses constant climatologies for ozone andsea ice.

Although POAMA-2M hindcasts are available from six startdates per month for the period 1981–2014, we only use thehindcasts from start times on the 1st of the month for themonths of February, May, August and November for 1997–2008so as to match the available starting times from the ECMWFhindcasts, and the available years of observations (Table 1).Our analysis here of four starts per year for 12 years equatesto only 48 hindcasts in total, which is far fewer than the 117(in austral summer) and 108 (winter) hindcasts analysed inZhu et al. (2014). However, comparison of the results with Zhuet al. (2014) gives us confidence that these new results are stillmeaningful.

2.2. ECMWF monthly forecast system

The ECMWF monthly system we analyse is the version thatwas operational from November 2010 to May 2011, namely theIntegrated Forecast System (IFS) cycle 36R4 (Forbes et al., 2012).This system employed a variable atmospheric resolution throughthe forecast cycle and was only coupled to an ocean after day 10(Vitart et al., 2008). The atmospheric model was integrated for thefirst 10 days with a T639 resolution (∼30 km grid) and 62 verticallevels with persisted sea-surface temperature (SST) anomalies. Atday 10 the horizontal resolution was lowered to T319 (∼60 kmgrid) and coupled to an ocean model with an approximate 1◦by 1◦ horizontal resolution with equatorial refinement and 29vertical levels. During the first 10 days the uncoupled oceanmodel was forced by fluxes provided by the atmosphere forecastmodel. The ocean initial conditions at day 0 came from the nearreal-time component of the operational ocean analysis system(Balmaseda et al., 2007) and during the first 10 days the ocean wasconstrained by the same persisted SST anomalies as used for theatmosphere.

Unperturbed atmospheric initial conditions were taken fromthe ECMWF Interim Re-Analysis (ERA-Interim: Dee et al.,2011). An additional 14 ensemble members were created by

c© 2016 Royal Meteorological Society Q. J. R. Meteorol. Soc. (2016)

Seamless Precipitation Prediction Skill in Two Global Models

Figure 1. Schematic of the time window and lead time definitions used in this analysis, consistent with Zhu et al. (2014). The horizontal axis represents forecast timefrom the initial condition. The expression ‘1d1d’ refers to an averaging window of 1 day at a lead time of 1 day. Similarly, ‘2d2d’ represents an averaging window of2 days at a lead time of 2 days, and so on. Note that 1d1d is what is often called ‘day 2’ in other publications, and 1w1w is what is often called ‘week 2’.

perturbing the ocean and atmosphere initial states. Perturbationsto the atmosphere were produced using the singular vectormethod (Buizza and Palmer, 1995) and by randomly perturbingthe tendencies in the atmospheric physics during the modelintegrations (Palmer, 2001). The ensemble of ocean initialconditions was produced by applying a set of random wind stressperturbations during the ocean data assimilation (Vialard et al.,2005). This ECMWF system, and its similar versions, has beendocumented to well represent and predict many of the importantsources and examples of weather and climate variability (Bechtoldet al., 2008; Vitart et al., 2010; Webster et al., 2011). However,like POAMA it used a fixed atmospheric CO2 concentrationand ozone climatology. For sea ice, this ECMWF system tookadvantage of its known persistence (Blanchard-Wrigglesworthet al., 2011): observed sea ice anomalies were computed withrespect to the past 5-year climatology, and these anomalies werepersisted from the initial condition up to day 15 and then reducedtowards zero over the next 30 days. Table 1 lists the hindcast datesused.

2.3. Observations

To verify the model hindcasts we use the Global PrecipitationClimatology Project (GPCP) daily precipitation with 1◦ resolution(Huffman et al., 2001). The GPCP data are a blendedproduct derived from both station observations and satellitemeasurements. The satellite data are sourced from bothgeostationary and polar-orbiting platforms. For verification, weinterpolate the GPCP analyses and ECMWF forecasts to thePOAMA atmospheric model grid (∼2.5◦ grid). Our analysistherefore concentrates on scales of ∼250 km and above, providinga reasonable representation of synoptic-scale weather. Knownproblems exist in the GPCP data at high latitudes (Bolvinet al., 2009); however, our results and survey of the literaturegive us enough confidence to show results to a latitudeof 80◦.

2.4. Measure of skill

We measure skill/accuracy using correlation of the ensemblemean precipitation forecast anomalies with the observed GPCPanomalies (hereafter CORa). These correlations are computedover time (i.e. using data from the 48 different start times), andare computed separately for each grid point and each lead time.Anomalies are formed for the observations and hindcasts byremoving their respective climatologies. The model climatologyis lead-time dependent to account for model drift (Stockdale,1997). Further details of the calculation are provided in Zhu et al.(2014).

2.5. Forecast time window definition

We take the approach of widening the time-averaging windowof the forecast and verifying observational data when looking atlonger lead times following Zhu et al. (2014). A schematic of thisapproach and the terminology we use to label it is provided inFigure 1. Our intention is to provide a seamless transition fromweather to climate in this verification. Note that ‘1d1d’ is what iscommonly referred to as ‘day 2’ in other publications and ‘1w1w’is commonly referred to as ‘week 2’. The longest window and leadtime combination we consider is 4 weeks (i.e. 4w4w). 4w4w isroughly equivalent to ‘month 2’. We also study the intermediatewindow/lead times of 2d2d, 4d4d, and 2w2w, providing a totalof six different time-scales. This approach of varying both thetime-averaging window and lead time is similar to that usedpreviously by Rodwell and Doblas-Reyes (2006).

3. Results

3.1. Skill comparison between POAMA and ECMWF

Maps of CORa for the window/lead time combinations of 1d1d,1w1w and 4w4w are displayed in Figure 2 using all 48 hindcast starttimes (Table 1). Higher positive values indicate greater agreementbetween the predicted and observed anomalies, i.e. higher skill.Most obvious in these maps is that the ECMWF system showsmostly higher skill than POAMA, especially at the shorter leadtimes. Despite this skill difference, however, the overall resultfrom Zhu et al. (2014) still holds for each system: an overall shiftof the areas of highest skill from the extratropics to Tropics as thelead time is increased.

Compared to the CORa maps presented in Zhu et al. (2014;their Fig. 5), the POAMA maps in Figures 2((b), (d), (f)) havesimilar correlation values, but more spatial noise. The greaternoise can be explained by the fewer years and fewer start datesused in the current work so as to match what was available fromthe ECMWF system. A simple calculation of statistical significancesuggests that the error bars (i.e. noise) for the correlationspresented here would be about 1.5 times larger than those inZhu et al. (2014). Comparisons of the correlations at individualgrid points should therefore be avoided. Another difference toZhu et al. is that the maps in this article are presented usingdata from all seasons, rather than being separated into australsummer versus boreal summer. We did this as there were too fewECMWF hindcasts to make conclusions from seasonally stratifiedresults. Despite these computational differences, the same generalconclusions can be made; that is, a shift of the areas of highestskill from the extratropics to Tropics with lead time.

We now compare the POAMA correlation maps (Figures 2(b),(d), (f)) with those from the ECMWF system (Figures 2(a), (c),



60°N

1d1d, ECMWF(a)

30°N

EQ

30°S

60°S

0 60°E

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

120°E 180 120°W 60°W 0

(b)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1d1d, POAMA

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

(c)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1w1w, ECMWF

30°N

EQ

30°S

60°S

0 60°E 120°E 180 120°W 60°W 0

(d)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1w1w, POAMA

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

(e)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

4w4w, ECMWF

60°N

30°N

EQ

30°S

60°S

0 60°E 120°E 180 120°W 60°W 0

(f)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

4w4w, POAMA

60°N

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

Figure 2. Maps of CORa actual skill for precipitation for (a, b) 1d1d, (c, d) 1w1w, and (e, f) 4w4w, for (a, c, e) ECMWF and (b, d, f) POAMA, using data for the years1997–2008.

(e)). Overall the spatial patterns of CORa are similar. Perhapsthe strongest similarity between POAMA and ECMWF is at the4w4w time-scale with both showing a large area of high skill(CORa > 0.7) in the equatorial Pacific. As explained by Zhu et al.(2014), this skill is derived from the response of precipitationto the slowly-varying SST anomalies of the El Nino–SouthernOscillation (ENSO). Both of the modelling systems have beendesigned to capture ENSO-associated variability (through use ofcoupled ocean–atmosphere models that can resolve the essentialdynamics of ENSO, such as ocean equatorially-trapped waves;Neelin et al., 1998), so it is expected that they both show this.However, it is interesting that the much higher model resolutionof ECMWF provides only a modest increase in skill for 4w4w.Another similarity between ECMWF and POAMA is the poorskill (CORa < 0.3) for all time-scales in the subtropical dry zonesover Africa, the eastern Atlantic, and eastern Pacific.

Perhaps the greatest difference between the POAMA andECMWF correlation maps is at the shorter 1w1w and 1d1d

time-scales, with ECMWF showing higher skill than POAMAin most locations. This can be explained by the much higheratmospheric resolution of the ECMWF model, its arguablymore realistic physical parametrizations, and its likely reducedinitial shock. In the case of the physical parametrizations,Bechtold et al. (2008) reported that improvements in the ECMWFparametrizations of convection and diffusion reduced biases inthe model climate, increased the level of activity of mesoscale tosynoptic perturbations to more realistic levels, and consequentlyimproved the model simulation and prediction of many fields andphenomena, including the Madden–Julian Oscillation (MJO).This improved MJO, for example, is likely the source of some ofthe additional prediction skill displayed for the ECMWF systemthrough the tropical Indo-Pacific, where the MJO impact onconvection and precipitation is large (Madden and Julian, 1972;Wheeler and Hendon, 2004). Reduced initial shock is expectedfor the ECMWF system due to its use of atmospheric initialconditions derived from a system using a similar atmospheric



(a) (b)

Figure 3. Zonally averaged CORa actual skill for precipitation, for the different time window/lead combinations in (a) ECMWF and (b) POAMA.

model (i.e. ERA-Interim), whereas for POAMA they are derivedusing models that are more different to POAMA (i.e. ERA-40 andthe Bureau’s operational NWP analysis). It is of note that a slightlyearlier version of the ECMWF system also came out as ‘noticeablybetter’ in a comparison of precipitation prediction skill withcontemporary systems from Japan and the USA, as presented inLi and Robertson (2015). Using weekly-average precipitation andtwo different measures of skill, the ECMWF system was found tobe noticeably better in both measures.

Further comparison of the ECMWF and POAMA skill isprovided in Figure 3, showing the zonally averaged CORa forall six time-scales (reminiscent of similar zonally averaged skillfigures, but for monthly averages, in Chen et al. (2010) and Kumaret al. (2011)). Once again, the generally higher skill for ECMWF(Figure 3(a)) is apparent. For example, the zonally averaged CORafor POAMA is ∼0.4 at the Equator (for all time-scales), whereasfor ECMWF it varies from ∼0.55 for the 4-week time-scale to∼0.65 for the 1-day time-scale. Of particular note in this figure isthe somewhat different latitudinal profile for the 1d1d and 2d2dcurves in the two systems: POAMA shows a dip in skill for tropicallatitudes from 20◦S to 20◦N, whereas the ECMWF system doesnot have this dip. That is, the ECMWF system displays generallyjust as much skill in the Tropics as extratropics even at thesevery short time-scales. This was somewhat of a surprise given ourprevious results in Zhu et al. (2014) based on just POAMA, andthe work of Ebert et al. (2003) based on an analysis of severalthen-operational numerical weather prediction models, whichshowed lower short-range skill (as measured by the bias score andequitable threat score) in the Tropics.

Another view of the tropical versus extratropical skill isprovided in Figure 4, showing the variation of CORa in differentlatitude bands as a function of lead time. For POAMA the short-range skill is clearly higher in the extratropical latitude bands thanthe Tropics, while more skilful prediction emerges in the Tropicsafter the 4-day lead time. For ECMWF, there is higher skill inthe Tropics than extratropics at the longer lead times, but the10◦S–10◦N band is almost as skilful as the extratropical bandsat the shorter lead times as well. Despite this difference, in bothmodels there is a clear emergence of more skilful prediction in the10◦S–10◦N band for the 1-week time-scale (i.e. a 1-week averageat a lead of 1 week) and beyond. This shows the uniqueness andimportance of near-equatorial atmosphere–ocean dynamics forthese longer time-scales.

3.2. Potential skill/predictability

We now consider model potential (or perfect) skill (orpredictability), following the definition of Buizza (1997), as hasbeen used in numerous studies (e.g. Rodwell and Doblas-Reyes,2006; Becker et al., 2013; Boer et al., 2013; Holland et al., 2013). Foreach prediction system, one ensemble member is selected as reality(i.e. replacing observations), and the remaining members are usedfor the computation of the ensemble mean. The potential skill is

0.8

(a)

(b)

ECMWF

1d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N

2d2d 4d4d 1w1w 2w2w 4w4w

CO

Ra

0.70.80.60.40.30.20.1

0

0.8

POAMA

1d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N


CO

Ra

0.70.80.60.40.30.20.1

0

Figure 4. Zonally averaged CORa actual skill for precipitation over specifiedlatitude ranges versus forecast window/lead times from 1d1d to 4w4w for (a)ECMWF and (b) POAMA. Note that the spacing of the time-scale is based on thelogarithm of the lead time.

then computed in the same way as the actual skill by correlatingthe new ensemble mean with the selected member. This is repeatedfor all members and then averaged. The potential skill thereforeanswers the question of how well the model can predict itsfuture given the growing spread of the ensemble members. Highpotential skill is therefore an indicator of the existence of alarger predictable signal (in the model) and/or of lower ensemblespread. Usually, but not always, potential skill is higher thanactual skill (Kumar et al., 2014), and the difference between themcan be interpreted as a deficiency in the system and used as afocus for system improvement, such as in the model, in the initialconditions, or in the ensemble spread. Observational errors mayalso contribute to a difference, since observational errors wouldtend to make the skill computed against observations (i.e. thecalculated actual skill) be artificially low, but would not affect the



60°N

1d1d, ECMWF, potential skill

30°N

EQ

30°S

60°S

0 60°E

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

120°E 180 120°W 60°W 0

(a)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1d1d, POAMA, potential skill

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

(b)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1w1w, ECMWF, potential skill

30°N

EQ

30°S

60°S

0 60°E 120°E 180 120°W 60°W 0

(c)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

60°N

1w1w, POAMA, potential skill

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

(d)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

4w4w, ECMWF, potential skill

60°N

30°N

EQ

30°S

60°S

0 60°E 120°E 180 120°W 60°W 0

(e)

–0.9 –0.7 –0.5 –0.3 0 0.3 0.5 0.7 0.9

4w4w, POAMA, potential skill

60°N

30°N

EQ

30°S

60°S

0 060°E 120°E 180 120°W 60°W

(f)

Figure 5. As in Figure 2, but for potential skill.

potential skill. Whatever the cause, where the potential skill ismuch higher than actual skill is deserving of additional attentionfor future research and development, a view that is consistent withthe final statement in Kumar et al. (2014), who said: ‘the rightcontext for the use of perfect skill may be as a model diagnostictool’.

Figure 5 presents maps of the potential skill computed inthis manner for 1d1d, 1w1w and 4w4w, using the same colourscale as the maps of actual skill in Figure 2. As expected, thepotential skill is almost everywhere higher than actual skill.In addition, the potential skill of the ECMWF system almosteverywhere exceeds that for POAMA. Further, at the 1w1w and4w4w time-scales the spatial patterns (but not magnitudes) ofpotential and actual skill are quite similar for both ECMWF andPOAMA; all maps for 1w1w and 4w4w show highest skill in thenear-equatorial Pacific and patterns elsewhere are also similar.In contrast, substantial differences in the spatial patterns appearat the 1d1d time-scale; ECMWF shows slightly lower potentialskill in the Tropics than extratropics but POAMA shows highest

potential skill in the Tropics, and the potential skill maps do notshow the very low values of skill in the subtropical dry zones thatwere apparent in actual skill. Further comparisons are providedbelow.

Figure 6 shows zonally averaged potential skill for both theECMWF and POAMA systems as a function of latitude at differentlead times, in the same format as Figure 3. Comparing the panelsof Figure 6 with Figure 3, it is interesting that the shape andmagnitude of the line plots for the POAMA potential skill aresimilar to the ECMWF actual skill, whereas the shape (but not themagnitude) of the line plots for the ECMWF potential skill is morelike the POAMA actual skill. We will discuss this similarity ofPOAMA potential skill and ECMWF actual skill when discussingFigure 9 below.

We next show the potential skill for the different latitudebands as a function of forecast time-scale (Figure 7). Like foractual skill, potential skill in the 10◦S–10◦N band is clearly higherthan potential skill in the extratropics for the longer time-scales.However, the emergence of higher skill in the Tropics occurs



(a) (b)


0.8

1

0.6

0.4

0.2

0

ECMWF potential skill

1d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N


CO

Ra

0.8

1

0.6

0.4

0.2

0

POAMA potential skill

1d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N


CO

Ra

(a)

(b)


earlier for potential versus actual skill; for ECMWF it occursfor the 4d4d forecast and for POAMA it occurs for 1d1d. Thisis further evidence that tropical short-range predictability ofprecipitation may be potentially higher and closer to that of theextratropics than we previously thought (Ebert et al., 2003; Zhuet al., 2014).

The numerical difference between potential and actual skill ispresented in Figure 8 as a function of forecast time-scale andlatitude band. As discussed above, where the difference (or gap)

0.5

0.4

0.3

0.2

0.1

0

ECMWF potential minus actual skill

POAMA potential minus actual skill

1d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N


CO

Ra

0.5

0.4

0.3

0.2

0.1

01d1d

70°S–50°S50°S–30°S30°S–10°S

10°S–10°N

window/lead

50°N–70°N10°N–30°N30°N–10°N


CO

Ra

(a)

(b)

Figure 8. As in Figure 4 but for potential skill (Figure 7) minus actual skill(Figure 4).

is largest is deserving of additional attention for future researchand development. For ECMWF the largest gap is for the shortertime-scales in the Southern Hemisphere high latitudes (70–50◦S).For POAMA the largest gap is in the Tropics, especially near theEquator (10◦S–10◦N). For the latter we think this is a case wherePOAMA actual skill can be substantially improved through greaterattention to the parametrization of cumulus convection (alongthe lines of what has been done for ECMWF; Bechtold et al.,2008) and higher atmospheric model resolution. This conclusionis supported by the fact that ECMWF actual skill in the Tropicsis already much higher than POAMA actual skill, so this sets anachievable goal for model improvement. For the ECMWF system,we know that some of the gap in the high latitudes can be explainedby insufficient ensemble spread (personal communication withECMWF, 2016). But there is no reason to think that deficientspread can explain the larger gap in the Southern Hemisphere



–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

r = 0.35b = 1.49

1d1d, ECMWFA

ctua

l ski

ll

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

r = 0.23b = 0.49

1d1d, POAMA

Act

ual s

kill

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


r = 0.23b = 0.44

1d1d, ECMWF vs POAMA

EC

MW

F a

ctua

l ski

ll

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

r = 0.37 r = 0.49b = 0.54 b = 0.62

r = 0.32b = 0.43

r = 0.55b = 0.75

r = 0.68b = 0.72

r = 0.65b = 0.99

1w1w, ECMWF

Act

ual s

kill

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

1w1w, POAMAA

ctua

l ski

ll

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


1w1w, ECMWF vs POAMA

EC

MW

F a

ctua

l ski

ll

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

4w4w, ECMWF

Act

ual s

kill

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Potential skill

4w4w, POAMA

Act

ual s

kill

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1


4w4w, ECMWF vs POAMA

EC

MW

F a

ctua

l ski

ll

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

Figure 9. Scatterplots of precipitation CORa actual skill (y-axis) versus potential skill (x-axis) for all grid points from 60◦S–60◦N in Figures 2 and 5. Left column(a, d, g) is for ECMWF actual versus potential skill, middle column (b, e, h) is for POAMA actual versus potential skill, and right column (c, f, i) is for ECMWFactual skill versus POAMA potential skill. Top row (a–c) is for 1d1d (1-day time-scale), middle row (d–f) for 1w1w (1-week time-scale), and bottom row (g–i) is for4w4w (4-week time-scale). Green symbols for land points and blue for ocean points. The thick red line is the line of identical skill. The thin pink line is the linearleast-squares regression line for the plotted points, ‘r’ is the pattern correlation coefficient, and ‘b’ is the slope of the regression line.

than Northern Hemisphere. This therefore suggests that there isstill more room for improvement of actual skill in the SouthernHemisphere than Northern Hemisphere for this ECMWF systemthrough further improvements in the observing system and initialconditions (e.g. Fig. 1 of Kirtman et al. (2013) and referencestherein). But until improvements in actual skill are made, wecannot say for sure.

Finally, we further analyse the relationship of actual andpotential skill in Figure 9, which presents scatter plots of actualskill versus potential skill following the work of Kumar et al.(2014). We put potential skill on the axis of abscissas (or x-axis) since it has less noise than the actual skill due to the use ofeffectively more samples in its calculation. Three different forecasttime-scales are shown (1d1d, 1w1w and 4w4w). The comparisonswithin each individual model are in the left two columns andthe mixed-model combination of POAMA potential skill versusECMWF actual skill is in the right column. Additionally, points

representing land points are coloured in green, and ocean pointsin blue.

Similar to Kumar et al. (2014; their Fig. 2), who analysedmonthly forecasts of sea-surface temperature, most points in theindividual model plots (i.e. the left two columns) fall on theside of the diagonal (i.e. line of identical skill) that indicatesactual skill in each system is almost everywhere less than potentialskill. The plots also indicate a linear relationship between actualand potential skill which gets stronger with increasing lead time(the 4w4w points have a pattern correlation coefficient of 0.65for ECMWF and 0.68 for POAMA). These positive correlationsbetween actual and potential skill show that while there need notbe a relationship between them from a statistical perspective (asargued by Kumar et al. (2014)) in practice there is a relationshipin these prediction systems. Further knowledge on the sourcesof skill is provided by the distinction between ocean and landpoints. Ocean points have higher skill than land points, especially



at the longer leads, further demonstrating that it is slow processesassociated with the ocean, and ENSO in particular, that providethe greatest predictability at these longer time-scales.

A final interesting comparison is made in the far right column((c), (f), (i)) of Figure 9 of ECMWF actual skill versus POAMApotential skill. Compared to the within-model comparisons inthe left two columns (Figures 9(a), (b), (d), (e), (g), (h)), thecloud of points is now more centred on the line of identical skill,although with more scatter. Noting that for a perfectly reliableprediction system the potential skill should match the actual skill,we hypothesize that the POAMA potential skill and ECMWFactual skill are likely a closer estimate of the true predictabilityof the climate system than either the POAMA actual skill (whichis obviously lower than the true predictability since ECMWF isbetter) or the ECMWF potential skill (which is likely too high dueto insufficient ensemble spread). However, testing this hypothesisis not possible without having a more perfect prediction systemand many matching hindcasts and observations with which totest it.

4. Conclusions

Our goal in this work has been to further the understandingof the relative predictability and prediction skill of tropical andextratropical precipitation across a broad range of time-scales.

As expected, the ECMWF system with its much higherresolution atmosphere achieves higher actual skill than POAMAin almost all locations, especially at the short to medium range(1d1d to 1w1w; Figures 2 and 3). Somewhat surprising to us,however, is that at the short range the ECMWF tropical skill is ofsimilar magnitude to its extratropical skill. This was unexpectedgiven our previous work and that of others such as Ebert et al.(2003). It seems that the increased resolution of the ECMWFatmospheric model, its improved physical parametrizations, andits likely reduced initial shock, have provided a larger relativeimprovement in the Tropics. Consistent with this, the ECMWFsystem has been documented to well simulate and predict tropicalweather phenomena, such as the MJO (Vitart, 2014), convectivelycoupled equatorial waves (Bechtold et al., 2008), and tropicalcyclones (Vitart et al., 2010).

One effect of this much higher short-range tropical skill forECMWF is that unlike POAMA, there is no clearly definedcrossover time-scale when near-equatorial skill switches frombeing worse to better than that in the extratropics. However,one consistency between the actual skill in both systems isthat by our ‘seamless’ verification measure, the 10◦S–10◦N skillremains approximately constant with forecast time-scale whereasit substantially reduces with time-scale for all other latitudes(Figure 4). Also consistent between ECMWF and POAMA is thatfor the 1-week time-scale and beyond, zonally-averaged equatorialskill is clearly superior to that in the extratropics.

Model-estimated potential (or perfect) skill calculations havealso been presented. As expected (but not guaranteed; Kumaret al., 2014), potential skill for both POAMA and ECMWF isalmost everywhere higher than their own actual skill (Figure 5vs. Figure 2). Such differences in potential versus actual skillrequire careful interpretation. Possible explanations include: (i)room for improvement in actual skill; (ii) too little ensemblespread resulting in too high potential skill; and/or (iii) errorsin the verifying observations which artificially reduce actualskill. For POAMA the largest differences occur in the Tropicsfor the shorter time-scales. Noting that the actual short-rangetropical skill of POAMA is also inferior to that of ECMWF,this difference is best interpreted as ‘room for improvement’ (ofactual skill) for POAMA. For ECMWF, on the other hand, thelargest difference is for the shorter lead times in the SouthernHemisphere high latitudes. It is difficult to tell which of the threepossible explanations are best in this case. Whatever the cause,this result points to these regions of the Southern Hemisphere asdeserving of most attention for this ECMWF system.

Finally, we note that the POAMA potential skill is of a similarmagnitude and distribution to the ECMWF actual skill (Figure 6versus Figure 3, and Figure 9), whereas the ECMWF potentialskill is generally much higher and POAMA actual skill is lower.As explained above, this leads us to hypothesize that the ECMWFactual skill and POAMA potential skill are likely a closer estimateof the true predictability of the climate system than either thePOAMA actual skill or ECMWF potential skill.

Acknowledgements

The Managing Climate Variability Program is acknowledgedfor their support of POAMA and its products. Hongyan Zhureceived funding support from the Australian Climate ChangeScience Program, and Adam Sobel was supported by the Officeof Naval Research (N00014-415 12-1-0911). Thanks to HarryHendon and Eunpa Lim for providing internal reviews of thisarticle and to two anonymous reviewers in the formal journalreview process.

References

Balmaseda M, Vidard A, Anderson D. 2007. The ECMWF System-3 OceanAnalysis System. ECMWF Technical Memorandum 508. ECMWF: Reading,UK.

Barnston AG, Tippett MK, L’Heureux ML, Li S, DeWitt DG. 2012. Skill of real-time seasonal ENSO model predictions during 2002–11: Is our capabilityincreasing? Bull. Am. Meteorol. Soc. 93: 631–651.

Bechtold P, Koehler M, Jung T, Doblas-Reyes P, Leutbecher M, Rodwell M,Vitart F. 2008. Advances in simulating atmospheric variability with theECMWF model: From synoptic to decadal time-scales. Q. J. R. Meteorol.Soc. 134: 1337–1351.

Becker EJ, van den Dool HM, Pena M. 2013. Short-term climate extremes:Prediction skill and predictability. J. Clim. 26: 512–531.

Blanchard-Wrigglesworth E, Armour KC, Bitz CM, DeWeaver E. 2011.Persistence and inherent predictability of Arctic sea ice in a GCM ensembleand observations. J. Clim. 24: 231–250.

Boer GJ, Kharin VV, Merryfield WJ. 2013. Decadal predictability and forecastskill. Clim. Dyn. 41: 1817–1833.

Bolvin DT, Adler RF, Huffman GJ, Nelkin EJ, Poutiainen JP. 2009. Comparisonof GPCP monthly and daily precipitation estimates with high-latitude gaugeobservations. J. Appl. Meteorol. Climatol. 48: 1843–1857.

Buizza R. 1997. Potential forecast skill of ensemble prediction and spreadand skill distributions of the ECMWF ensemble prediction system. Mon.Weather Rev. 125: 99–119.

Buizza R, Palmer TN. 1995. The singular-vector structure of the atmosphericglobal circulation. J. Atmos. Sci. 52: 1434–1456.

Chen M, Wang W, Kumar A. 2010. Prediction of monthly-mean temperature:The roles of atmospheric and land initial conditions and sea surfacetemperature. J. Clim. 23: 717–725.

Dee DP, Uppala SM, Simmons AJ, Berrisford P, Poli P, Kobayashi S, AndraeU, Balmaseda MA, Balsamo G, Bauer P, Bechtold P, Beljaars ACM, vande Berg L, Bidlot J, Bormann N, Delsol C, Dragani R, Fuentes M, GeerAJ, Haimberger L, Healy SB, Hersbach H, Holm EV, Isaksen L, KallbergP, Kohler M, Matricardi M, McNally AP, Monge-Sanz BM, Morcrette J-J,Park B-K, Peubey C, de Rosnay P, Tavolato C, Thepaut J-N, Vitart F. 2011.The ERA-Interim reanalysis: Configuration and performance of the dataassimilation system. Q. J. R. Meteorol. Soc. 137: 553–597.

Ebert EE, Damrath U, Ergen W, Baldwin ME. 2003. The WGNE assessment ofshort-term quantitative precipitation forecasts. Bull. Am. Meteorol. Soc. 84:481–492.

Forbes RM, Tompkins AM, Untch A. 2012. ‘A new prognostic bulkmicrophysics scheme for the ECMWF forecast model’. In EGU GeneralAssembly Conference Abstracts, 22-27 April, 2012. Vienna, Austria. Vol. 14,p. 12155.

Holland MM, Blanchard-Wrigglesworth E, Kay J, Vavrus S. 2013. Initial-valuepredictability of Antarctic sea ice in the Community Climate System Model3. Geophys. Res. Lett. 40: 2121–2124, doi: 10.1002/grl.50410.

Hope P, Lim E-P, Wang G, Hendon HH, Arblaster JM. 2015. Contributors tothe record high temperatures across Australia in late spring. In ExplainingExtremes of 2014 from a Climate Perspective, Herring SC, Hoerling MP,Peterson TC, Stott PA. (eds.) Bull. Amer. Meteor. Soc. 96(12): S149–S153.American Meteorological Society: Boston, MA.

Hudson D, Alves O, Hendon HH, Wang G. 2011. The impact of atmosphericinitialisation on seasonal prediction of tropical Pacific SST. Clim. Dyn. 36:1155–1171.

Hudson D, Marshall AG, Yin Y, Alves O, Hendon HH. 2013. Improvingintraseasonal prediction with a new ensemble generation strategy. Mon.Weather Rev. 141: 4429–4449.

Huffman GJ, Adler RF, Morrissey M, Bolvin DT, Curtis S, Joyce R, McGavockB, Susskind J. 2001. Global precipitation at one-degree daily resolution frommulti-satellite observations. J. Hydrometeorol. 2: 36–50.



Kirtman B, Anderson D, Brunet G, Kang IS, Scaife AA, Smith DM. 2013.Prediction from weeks to decades. In Climate Science for Serving Society,Asrar GR, Hurrell JW. (eds.): 205–235. Springer: Dordrecht, Netherlands,doi: 10.1007/978-94-007-6692-1_8.

Kumar A, Chen M, Wang W. 2011. An analysis of prediction skill of monthlymean climate variability. Clim. Dyn. 37: 1119–1131.

Kumar A, Peng P, Chen M. 2014. Is there a relationship between potential andactual skill? Mon. Weather Rev. 142: 2220–2227.

Langford S, Hendon HH. 2013. Improving reliability of coupled model forecastsof Australian seasonal rainfall. Mon. Weather Rev. 141: 728–741.

Li S, Robertson AW. 2015. Evaluation of submonthly precipitation forecastskill from global ensemble prediction systems. Mon. Weather Rev. 143:2871–2889.

Madden RA, Julian PR. 1972. Description of global-scale circulationcells in the Tropics with a 40–50-day period. J. Atmos. Sci. 29:1109–1123.

Neelin JD, Battisti DS, Hirst AC, Jin F-F, Wakata Y, Yamagata T,Zebiak SE. 1998. ENSO theory. J. Geophys. Res. 103: 14261–14290, doi:10.1029/97JC03424.

Palmer TN. 2001. A nonlinear dynamical perspective on model error: Aproposal for nonlocal stochastic dynamic parameterization in weather andclimate prediction models. Q. J. R. Meteorol. Soc. 127: 279–304.

Rodwell MJ, Doblas-Reyes FJ. 2006. Medium-range, monthly, and seasonalprediction for Europe and the use of forecast information. J. Clim. 19:6025–6046.

Stockdale TN. 1997. Coupled ocean–atmosphere forecasts in the presence ofclimate drift. Mon. Weather Rev. 125: 809–818.

Uppala SM, Kallberg PW, Simmons AJ, Andrae U, Da Costa Bechtold V,Fiorino M, Gibson JK, Haseler J, Hernandez A, Kelly GA, Li X, OnogiK, Saarinen S, Sokka N, Allan RP, Andersson E, Arpe K, Balmaseda MA,

Beljaars ACM, Van De Berg L, Bidlot J, Bormann N, Caires S, ChevallierF, Dethof A, Dragosavac M, Fisher M, Fuentes M, Hagemann S, Holm E,Hoskins BJ, Isaksen L, Janssen PAEM, Jenne R, McNally AP, Mahfouf J-F,Morcrette J-J, Rayner NA, Saunders RW, Simon P, Sterl A, Trenberth KE,Untch A, Vasiljevic D, Viterbo P, Woollen J. 2005. The ERA-40 re-analysis.Q. J. R. Meteorol. Soc. 131: 2961–3012.

Vialard J, Vitart F, Balmaseda MA, Stockdale T, Anderson DLT. 2005.An ensemble generation method for seasonal forecasting with anocean–atmosphere coupled model. Mon. Weather Rev. 133: 441–453.

Vitart F. 2004. Monthly forecasting at ECMWF. Mon. Weather Rev. 132:2761–2779.

Vitart F. 2014. Evolution of ECMWF sub-seasonal forecast skill scores. Q. J. R.Meteorol. Soc. 140: 1889–1899.

Vitart F, Buizza R, Balmaseda MA, Balsamo G, Bidlot J-R, Bonet A, FuentesM, Hofstadler A, Molteni F, Palmer TN. 2008. The new VAREPS-monthlyforecasting system: A first step towards seamless prediction. Q. J. R. Meteorol.Soc. 134: 1789–1799.

Vitart F, Leroy A, Wheeler MC. 2010. A comparison of dynamical and statisticalpredictions of weekly tropical cyclone activity in the Southern Hemisphere.Mon. Weather Rev. 138: 3671–3682.

Webster PJ, Toma VE, Kim H-M. 2011. Were the 2010 Pakistan floodspredictable? Geophys. Res. Lett. 38: L04806, doi: 10.1029/2010GL046346.

Wheeler MC, Hendon HH. 2004. An all-season real-time multivariate MJOindex: Development of an index for monitoring and prediction. Mon.Weather Rev. 132: 1917–1932.

Yin Y, Alves O, Oke PR. 2011. An ensemble ocean data assimilation system forseasonal prediction. Mon. Weather Rev. 139: 786–808.

Zhu H, Wheeler MC, Sobel AH, Hudson D. 2014. Seamless precipitationprediction skill in the Tropics and extratropics from a global model. Mon.Weather Rev. 142: 1556–1569.


Seamless precipitation prediction skill comparison between ...

Documents