7 D-R143 576 CLIMATE PREDICTION PART i CYCLONE FREQ JENCY(I) VIRGINIA 1/2 UNIV CHARLOTTESVILLE DEPT OF ENVIRONMENTAL SCIENNES S P HAYDEN JUL 84 TR-30 N00014-8i-K-0033 UNCLASIFEDG 4/2 N EEEEEEELhEEE EEEEEEE EEE EEEEEEEEEmohEEI EomhEEEohEEEEE EEEEomhohEEEEI EhmmhEEEmhE
136
Embed
UNIV CHARLOTTESVILLE DEPT OF ENVIRONMENTAL SCIENNES ... · done by Ron Broome, Bob Wayland, Richard Wayland and Todd Hawes. Graphics were prepared by William Smith and Richard Wayland.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
7 D-R143 576 CLIMATE PREDICTION
PART i CYCLONE FREQ JENCY(I) VIRGINIA
1/2UNIV CHARLOTTESVILLE DEPT OF ENVIRONMENTAL SCIENNESS P HAYDEN JUL 84 TR-30 N00014-8i-K-0033
OFFICE OF NAVAL RESEARCH -JCOASTAL SCIENCES PROGRAM - -
CONTRACT No. NOOOk4-81-K-0033
TASK No. 389-170
APPROVED FOR PUBLIC RELEASEDISTRIBUTION UNLIMITED
V%
TABLE OF CONTENTS
Page
LIST OF FIGURES....................ii
LIST OF TABLES.....................vi
ACKNOWLEDGEMENTS...................vii
ABSTRACT......................viii
INTRODUCTION......................1
BACKGROUND......................
MODEL DEVELOPMENT...................6
MODEL DATA.......................7
MODEL CONSTRUCTION....................10
THE MODELS.......................13
THE FORECASTS..................................19The Hindcast Period: 1960-1960..........19The Operational Period: 1980-1983..........20Forecast Products.................20Inventory of Forecasts..............27
MEASURES OF FORECAST SKILL...............26Percent Correct Score...............28Heidke Skill Score................30Deviation Skill Score...............31Quadratic Skill Score...............31AAE and RMSE.....................2Local Skill....................33Global Skill...................33
ASSESSMENT OF FORECASTS................34The Mean as the Forecast.............34Magnitude Versus Forecast Sign..........38Local Percent Correct Skill Scores........40Global Percent Correct Score...........54Heidke Skill Score................59
operatioalhe forcati moes.eetil of the moedeio aroundm in
Hayenill aorit Smth(98).Te presaen preptot isppleens. Ta
earler whork.clsuisidct imto bu w ek o
the time domain of weather forecasting. Where longer range
predictions of the "state" of the atmosphere are concerned,
specificity in the time domain must be relinquished. The
prediction objective then becomes the specification of the
average state of the system for some suitable time interval
* (month, season, year, etc.). With this modified prediction
objective in mind, the required techniques become more stochastic
4 and less deterministic. This necessity is augmented by the fact
that suitable theories permitting deterministic forecast models
for months, seasons and years are not available at present.
In the absence of a deterministic basis for climate forecasting,
one is left with the need to identify some mode of persistence in
the atmospheric system such that knowledge about the current and
recent states of the atmosphere permits estimation of future
1conditions. Most efforts to identify such persistences in
1q~ temperature and precipitation data-time series have failed or the
magnitude of the resulting forecast skill is so small, and the
N number of forecast trials so few, that it is impossible to
*distinguish the forecast model from a model based on chance. The
Climate Analysis Center's monthly and seasonal forecasts are
based on persistences in the thickness fields. The perception is
that the general circulation may exhibit persistences that are
not apparent in station temperature and rainfall. The research
group at the Scripps Institution under Jerome Namias' direction
base their predictions on the persistences in sea-surface
K%
temperature fields which, in turn, serve as a "memory" for the
Satmosphere through thermodynamic couplings. Our work at the
University of Virginia is based on identified persittences in the
N.' fields of occurrences of cyclones over eastern North America
(Hayden and Smith, 1982). It is clear that occurrences of
cyclones are not independent of structure or thickness fields so
our work is in some sense like that of the Climate Analysis
Center but the forecasts do not always agree so real differences
I exist.
Over the last several years we have completed an extensive
N. Iforecasting and verification effort. This report summarizes the
results of this effort. We are convinced that sufficient success
has been demonstrated that Lorenz's (1973) criterion of
"$conclusive proof" has been fully met and we can advance the
theory that climate is at least partially predictable. Equally
important, however, is the need to study the causes of the
persistences and the nature of failures in persistence. This
awaits further work.
The approach taken in our work is not new. The concept of
analyses of the general circulation via study of ".centers of
action" had its champion in T. Bergeron. He referred to such
study as dynamic climatology (Bergeron, 1930).
a dynamic climatology should describe thefrequencies and intensities of well-defined systems
.44
that are more or less closed in a thermodynamic sense.
Bergeron's concept of dynamic climatology differed from that of
Hesselberg whose concept is close to the definition now generallyaccepted.
Dynamic climatology must be concerned with thequantitative application of the laws of hydrodynamicsand thermodynamics . . . to investigate the generalcirculation and state of the atmosphere, as well as the
'.4. average state and motion for shorter time intervals
The outcome of the Hesselberg approach is best observed in the
computer general circulation models (GCMs). Although GCMs look
promising in identifying probable future states of the atmosphere
.4.14associated with altered boundary conditions, they seem less
likely to provide useful prediction capabilities for the monthly,
seasonal, and year-to-year levels of the forecast problem. With
the aid of modern computers and statistical techniques, the
-4 systematic spatial and temporal variations in the centers of
action of the general circulation can be identified. The present
work is offered as evidence of the value of this approach. Given
Bergeron's concept of dynamic climatology and C. S. Durst's
definition that climate is the synthesis of the weat her, we
conclude that the fundamental elements of climate are the various
%I extant features of the general circulation rather than the more
commonly assumed fundamental elements of weather (temperature,
pressure, humidity, etc.). The task of climate prediction is
V then to specify future states of the general circulation and its
centers of action in a stochastic sense. Given useful
F. prudiction, statements about associated fields of the fundamental
. e
elements of the weather may be possible on climatological time
A .iscales. Forecast trials employing this concept have proved
- - successful and will be discussed in subsequent reports.
MODEL DEVELOPMENT
Three versions of the UVa Climate Forecast Model have been
constructed. The original model (Hayden and Smith, 1982) used
principal components analysis (PCA) to decompose the records of
seasonal patterns of cyclone frequencies into orthogonal
representations of the original data. The temporal persistences
of these orthogonal representations (principal components) are
used in making the forecasts. In the two later versions of the
model, the constraint of orthogonality was 1) eased and 2)
removed. In the former case (the second version of the model)
the property of orthogonality was retained but the axes
(principal components) were rigidly rotated with the constraint
that variance explained by each of the selected lower order
'.4 components be maximized. This is known as the VARIMAX rigid
rotation. In the third version of the model the constraint of
orthogonality is removed from lower order principal components
and each axis is rotated such that each explains the greatest
portion of residual variance unexplained by the sum of all of the
lower order rotated components. This variation is called the
PROMAX oblique rotation. In this report the unrotated principal
!%.
components version is referred to as MODEL I; the VARIMAX rigid
version is referred to as MODEL II; and the PROMAX oblique
version is referred to as MODEL III.
For details on the properties and relative merits of various
types of rotations of principal components the reader is directed
to Richman (1983a, 1983b). Richman (1981) has also shown that
rotated principal components give more faithful representations
of meteorological data fields. Our studies show modest but
Nconsistent 2-by-2 forecast skill improvements with relaxation of
the orthogonality constraint and the capacity to forecast some
geographic locations with Model II and Model III that were not
possible with Model I.
MODEL DATA
- Monthly cyclone frequencies for the years 1885-1984 were
tabulated from monthly charts of the "Tracks of the Centers of
Cyclones at Sea Level" published by Monthly Heather RevieN and in
recent years by The Mariners Heather Log. Multiple entries of a
ogiven storm in a grid cell were ignored. Grid cells south of
0
27.5 N were not included in this study because early forecast
trials showed no forecast skill in this region. The 87 grid
cells forecasted are indicated by the black dots in Fig. 1. Data
spatial inhomogeneities due to the variable density observation
oA - 7 -
network used to make the original storm track charts were ruled
out as a problem in earlier work (Hayden, 1981b). Frequencies
were not adjusted for latitude variations in grid-cell area
because of distortions involved in such adjustments (Hayden,
19831c). For the purpose of constructing and testing the
prediction model, the data matrix was divided into a dependent
(18385-1960) part from which the principal components were
calculated and the forecast models constructed, and an
independent (1960-1980) part which was reserved and used to
evaluate forecast skill. The post-1980 years were forecast in
real time. Real time forecasts were generaklly completed two to
three weeks following the close of a month. This time was needed
to acquire the charts of cyclone tracks from NOAA, extraction of
data from the charts, and running of the models. Alternative
lead time could be planned and evaluated for changes in forecast
* skill. Lag correlation studies indicate that sufficient variance
is explained out to a lag of one year and that useful forecasts
with longer lead times merit study. Tests of shorter lags, i.e.,
one month lag indicate little or no forecast skill at that time
scale.
-- 4
.M4M
e 4 . .. .....
.. .: ..........
050
90 80
Fig. 1. Chart of the study arma. 2.5 latitude by 5.0
longitude grid call center% are indicated. There are101 rectangular grid colls in tbm study area. Onlythose grid cells north of 27.5 N arm used in thisstudy.
-9
MODEL CONSTRUCTION
Figure 2 shows, schematically, model construction. The first
stage in the construction of the models was data preparation.
The archives of cyclone frequencies were first divided into two
parts. All the data from 1885-1959 were reserved for model
construction (the dependent data). The data for the years
1960-1980 were reserved for forecast trials (independent data --
hindcatts). Data for the post-1980 period were used in real time
to make forecasts (independent data -- operational forecasts).
Monthly cyclone frequency data are composited into six-month
seasons. Twelve six-month seasons are defined. The principal
components of cyclone frequencies for each of the 12 seasons are
then calculated. The first five of these components for each
season are then subjected to VARIMAX and PROMAX rotations. The
case weightings for each vector for each season for each year of
the dependent data record are caiculated and reserved.-9
The vector case weightings are used to derive the one-season lag
regression equations. These regression equations are used to
. estimate the case weightings for one season from the known case
* weighting for the previous season. The regression equations in
Model I differ from those of Model II and Model III. In Model I,
I C .
V,4 .* ;
1885
1969
1980
1885
1885
1959
8~1969 1980J J D J D
PCA PCA PCA
REGRESSION
a' ___ ___ __ ___ ___ __ FORECAST j
CYCLONES
Fig. 2. Schematic of assembly of the cyclone frequencyprediction models. Clear portions of the cubesrepresent dependent data (shaded -independent data).PCA refero to principal components analysis. 3-3January-June; J-D -July-December.
the case weightings for the two seasons are regressed for each
component but no cross component regressions are used because the
orthogonality of the components and their season-to-season
similarity always resulted in near zero correlations between
seasons. In Models II and III within- and cross-correlations are
'S examined and the regression with the highest correlation is
-. selected for use. In all cases (Models I. II and III) if there
are correlations below 0.35 the term is not used in the
equation. Previous trials showed that rarely was there a model
forecast skill when the correlations were below 0.35. This
constituted a pre-screening and thus a reduction in the number of
* models that required development and testing.
Using the regression equations, the case weightings for each
vector for each model version (Models 1, 11 and III) are
estimated and used in the forecast equations. The general form
of these equations is given in Hayden and Smith (1982) as
C -X+ aO0E + aO0E + ... +4aO0E (11s 1 1 2 2 5 5
where C is the matrix of predicted cyclone frequencies for eachs
4, grid cell for the season to be forecasted; X is the matrix of
long-term (18385-1959) mean cyclone frequencies at each grid cell
for the season to be forecasted; 0 is the matrix of standard
* -12-
4%7
deviations of X at each grid cell for the season to be
forecasted; E are the principal components for the season to bei
forecasted (non-rotated or rotated depending on model version
being constructed); a-ed a is the forecasted case weightingi
calculated from the one-season lagged regression equations.
Each term in the equation may be considered an individual model.
As five components are used in construction of these models each
term may be evaluated for forecast skill. The additive
combinations of terms can also be evaluated. A large number of
possible model configurations is thus possible. Only the models
with all terms included are reported on here. Model I has four
terms and Models II and III have five terms.
'I THE MODELS
Earlier we (Hayden and Smith, 1982) published the details of the
V. models to predict cyclone frequencies for the October-March and
April-September six-month seasons. The component parts of each
of the 12 six-month season models constructed for all three
versions of the model (I, II and III) are on file at the
University of Virginia. Each model consists of the data matrixes
listed in Table I.
TABLE I
Summary of the Forecast Model Matrixes*
MATRIX N DEFINITION OF THE MATRIX
X 87 Cyclone frequency means
0 87 Standard deviations of X
E 87 Predictor eigenvector variable loadings
E 87 Predictand eigenvector variable loadingsj+l
F 75 Predictor case weightings
..-. F 75 Predictand case weightings'.x, j+l
R 5 F vs F regression coefficientsj j+j
* = 12 six-month season models for versions I, II and III
N = number of elements in the matrix
Examples of the matrixes for the October-March season in Table I
follow. Figure 3 shows matrix X for 1885-1960 long-term means.
Figure 4 shows the matrix of the standard deviations (0) of X.
Figure 5 shows the matrix E and Figure 6 shows the column matrix
* (F) by year. The archives of the forecast models and forecast
- products are voluminous and do not lend themselves to
reproduction in technical reports. They are available for
inspection and study at the University of Virginia.
-o'
. .. ° ...
017
10 80
'$4Z
414
4.4
.15
~S5
I4
.r .- ....w
4 ..
STANORO DEIATI5
8U4
Fi. 4.. Thu. ma.i (0 of t e a d re i t o s o
.y.. n fr q eni.f..h.O t be - a ch.a.n h
uNTE Re ylnsprgi el
STNDR DVATO
I 90
-16-
-ro
-- 4S
AA
.. 05
A-70
ti-I 80
Fig. 5. The matrix E f or the first principal componentof the winter (October-March) season. The valuesplotted are dimensionless.
M -17-
%'4.
4.4
N12
-- 4-
* 8i -
S188S Igoe leis 19:30 194S 1 .t60
I4
€',- Fig. 6. The column matrix F of case weightings of the'.' first principal component of cyclone frequencies for
Sthe October-March season. The values plotted in the~time series are dimensionless.
* In order to generate a forecast for a six-month interval, the
* case weightings on the principal components of the previous
six-month period must be calculated. This requires that
'a:..principal components used in the forecast include data for the
previous six-month period. In the case of the first forecast of
the independent data period (January through June 1960)). the
calculated principal component case weightings for the July
through December 1959 period were entered into the dependent data
period regression equations, and the predicted case weightings
'a'afor the January to June (1960) period-were derived. For this
first forecast the dependent data period contained all the months
needed to predict the first six-month season of the independent
'a...-,.data period. In subsequent forecasts new principal components
analyses had to be run to generate the case weightings needed as
'a a'input into the regression equations. At no time were data for
the independent data period included in the regression equation
development. All forecasts were made for time beyond that used
to build the models.
% .4 .. ..
-7'-
TheOperational Period: 1980-1983
Charts of the tracks of the centers of cyclones for each month
are prepared by NOAA at the end of each month. They are released
and are publicly available about 15 days after the close of the
month. On receipt of the charts, frequencies per grid cell are
counted and entered into the data base. Principal components are
then found for the six months just concluded and case weightings
for each component calculated. The regression equations derived
for the dependent data period (1885-1960) are used to estimate
case weightings for the upcoming six-month season. The
* forecasted weightings are then used in Equation 1 to estimate
cyclone frequencies in coming seasons. Operational forecasts
were begun in 1960.
%I
Forecast Products
Two forecasts are presented here to illustrate the nature of the
forecast products generated. Both were made on an operational
basis. The forecast for October to March 1960-1981 was selected
because it was extreme in the sense of having largely negative
II departures from the mean forecast almost eiverywhere and the
magnitude of the negative anomaly forecasted was large. The
7 7F77.7
second forecast selected for illustration was for the September
to February 1981-1982 period. This forecast contains both large
positive and large negative anomalies from the mean. Three
products are returned from the forecast. First, the long-term
mean cyclone frequencies are presented in map form. Second, the
predicted anomalies in cyclone frequencies for each grid cell are
displayed in map form (Fig. 7 and 8). The third product is a map
of the predicted anomalies added to the means (Fig. 9 and 10).
The range of forecasted anomalies generally averages from six to
ten cyclones per grid cell. As typical maximum values of the
means for a six-month season are on the order o-f 12 cyclones per
grid cell, the forecasted anomalies are large in relative
magnitude. The contoured anomaly fields (Fig. 7 and 8) are
interesting in that one type of axis of maximum values and two
types of axes of minimum values are evident. The axis of maximum
values along the east coast of the U.S. (Fig. 8) can be directly
interpreted as an axis along which more than the normal number of
cyclones is likely to be observed if the forecast is correct.
The axis of absolute minimum values "negative storm track" e-g.,,
as in the track extending eastward from Colorado (Fig. 7 and 8),
is interpreted as an axis along which fewer than the normal
number of storms are expected. Finally, within an area of
forecasted negative anomaly, there may be axes of local "maxima"
or small negative values, e.g., the trace of small negative
values across the Great Lakes in Figure 7. Thus while storms
f . .i _ I
might be less frequent than normal, those that do occur would
tend to move along this track. The three different types of
tracks are illustrated with different symbols in the
illustrations.
I~' Clearly the charts of forecasted anomalies do not provide all the
information that is needed to interpret the forecast, so we added
the forecasted anomaly to the long-term mean (Fig. 9 and 10). The
* resulting chart has positive values everywhere and so the
interpretation difficulties o-f "negative tracks" are no longer
present. The resulting axes of maximum values can directly be
interpreted as the forecasted preferential location of the storm
tracks for the forecasted season. While forecast skill will be
discussed in a subsequent section it should be noted that both
these forecasts were successful. The sign of the anomaly was
forecasted correctly in 74.2% of the 67 grid cells in the October
to March 1960-1981 forecast and 89.7% of the 67 grid cells were
correctly forecast in the September to February 1961-1982
* forecast.
* To show each of these products for each model version and for
each forecast made would require the display of thousands of
maps. This is beyond the scope of this technical report. All of
* the maps are on file at the University of Virginia in the
author's archives. The subsequent observations and verifications
of each forecast are also saved for study.
1%L%
980
Fig 7. Oprtoa. foea.e .yln frequency..8
nOmaisfrOtbrMrh19-91(oe
Forecst ws isued o 24 ctobr 198..Th.unis.ar
.4
-V~ ~~~~~ ...V .% ..... . * . ...
*190 80 8
SO.8 prtoal oecse yln rqec
da atus frmtelogtr eaRo h
Sepembr-Fbrury 981198 seson(Moel ). oli
anogly sOrtiodahed orwsidte axloe ofreminmu
negative anomaly.
12) 4
... .. .....V. .- .. .. ..-
S.
8 AZ
/.~.%,
SIA
380
Fig. 9. Operationally forecasted cyclone numbers forthe October-March 1980-1981 season (Model D). The unitsare cyclones per grid cell. Arrows indicate axes oflocal maximum frequencies.
-25-1
7I
-700
V'0.
'-14
............
'p 80
Fig. 10. Operationally forecasted cyclone numbers forthe September-February 1981-1982 season (Model D). Theunits are cyclones per grid cell. Arrows indicate theaxes of local maximum frequencies.
'26
I
Inventory of Forecasts
Table II lists the number of forecasts made for the independent
data period and the operational period for MODELS I, II, and III.
We made 21,924 forecasts for each model version for the
independent data period, and 2,958 forecasts were made using each
.. model version during the period of operational forecasting.
Comparisons of these forecasts with observations form the basis
* for assessing the forecast skill of the models constructed.
TABLE II
Inventory of Forecasts
Forecast Period
(1960-1980) (1980-1982)
MODEL I lI II I II IIT
Grid cells (A) 87 87 87 87 87 87Seasons (B) 12 12 12 12 12 12No of years (C) 21 21 21 3* 3* 3*AxBxC (total forecasts)for Models I, II and 111 (21,924 (2,958)
* 1983 June-Nov and July-Dec forecasts were not verified in
time for this report.
.77 ----- ----- - -------
5 .- 5
MEASURES OF FORECAST SKILL
Numerous methods have been advanced to quantify estimates of
forecast skill (Brier and Allen, 1951; Vernon, 1953), and as
noted by Brier and Allen the method selected depends on the
purpose of verification. The purpose here is to establish the
level of reliability of the forecast scheme relative to the
climatological means as forecasts. A battery of tests of
forecast skill is reported here. Two types of forecasts are made
and evaluated: category and magnitude forecasts. In most trials
on climate forecasts magnitude forecast skills are not reported.
Rather, various categorical measures are reported (e.g., 2, 5,'
and 4 category tests). Magnitude measure obviates the need -for
complex categorical measures.
Percent Correct Score
The percent correct score is the simplest measure of forecast
skill. This measure is used to assess the skill of forecasts
where only two types of forecast are used, i.e. , above or below
the mean. This is sometimes referred to as the 2-by-C or sign
rs~ test. Chance alone would dictate a percent score of 50%. In the
present study 21 years are forecast in the independent data
4.%
forecast trials (1960-1980). As these forecasts were made after
1980 the term hindcasts is applied. Table III gives the
probabilities that various 2-by-2 percent correct scores could
occur by chance alone.
TABLE III
CHANCE PROBABILITIES IN 2B-2TRIALS
NO. CORRECT FORECASTS PROBABILITY OF EXCEEDINGIN 211 TRIALS (M. BY CHANCE ALONE
the square of deviation of the forecast from the observations.
Here the penalty for large errors is severe. Ideally one would
like a high percent correct skill score and a high quadratic
score.
AAE and RMSE
The average absolute error (AAE) is the average error
irrespective of the sign of the forecasted anomaly relative to
the mean. This value iscompared to the average absolute error
A of the mean as a forecast. A direct error reduction relative to
the mean as a forecast expressed as a percentage can then be
calculated. In the case of the root mean square error (RMSE) the
deviations of the forecasts from the observations are squared,
summed, and divided by the number of forecasts; then the square
3.; root is taken. A reduction of the root mean square error of the
mean as a forecast is desired for the model forecast. If the
44 sign of the forecast is correctly made all the time then the
minimum root mean square error can be insured with a forecast of
the historical average absolute error of the mean as a forecast.
The average absolute error of the forecast, if forecasts are
normally distributed, can be used to divide the distribution into
quarterlies for 4-by-4 skill tests.
Local Skill
N The term local skill is reserved for geographic or point skill.
It is the average skill at a point over time. In the present
study, forecasts were made for 87 grid cells (Fig. 1). Local
skills are reported for each grid cell. Under ideal
circumstances local skill should pass a 0.05 test of statistical
significance (Table 11). The %. of correct forecasts needed to
pass the 0.05 level at an individual grid cell is dependent on
the number of forecast trials. Twenty-one trials is the standard
used in Table II.
Global Skill
When local skills are aggregated or spatially averaged, a single
skill score "representing" all localities is reported. This
score is referred to as a global skill score. Two types of
global skill scores are defined here. As the forecast models are
constructed for six-month duration seasons and 12 such seasons
are defined, we then have within-model global scores and
between-model global scores. Thus we have a global skill score
S. for the six-month season beginning in April and ending in
September and also a global skill score which averages all
7.
possible six-month season models.
Global skill scores are convenient in that a single number can be
forwarded as a most general measure of model reliability.
* . However, it should be remembered that forecast skill varies from
season to season and from place to place. These variations must
be understood if the models are to be properly evaluated and,
- more importantly, used. Because skill at one site may not be
independent of skill at adjacent locations, great care must be
exercised in specifying statistical significance for global
measures of skill. Global skills reported in the absence of
reported local scores may be misleading. A very conservative
standard and one recommended here is that the average global %.
skill score is as large as required to pass a local test of skill
(see Table ID).
ASSESSMENT OF FORECASTS
'V The Mean as the Forecast
Forecasts are usually expressed relative to the mean as the
alternate and simplest forecast. Where the distribution is
normal the mean tends to be the most frequent occurrence. While
mean might well be a prudent and conservative forecast, the mean
is not always a good forecast. To examine the mean as a forecast
we used the 1885-1960 cyclone frequency means for the various
six-month seasons as forecasts for the six-month seasons between
1960 and 1982. Figures 11 and 12 illustrate the average absolute
and root mean square errors of the means as forecasts. It is
clear from both measures that the mean as a forecast varies with
season and that there is a secular trend toward the mean as a
progressively better forecast. Between 1960 and 1982 the root
4mean square errors have fallen from about 5 cyclones per grid
cell to about 2.5 cyclones per grid cell.
The reasons for the decline in average absolute and root mean
square errors of the means as forecasts are unclear. We conclude
that variability has declined because the departures from the
mean have fallen. Whittaker and Horn (1981) tabulated
cyclogenesis over North America and found a general decline in
* cyclogenesis. The overlap between their data and ours is plotted
in Fig. 12. Apparently the decline in cyclone frequency
variability is associated with fewer cyclones developing and
perhaps the "clipping" of extreme occurrences. Whittaker and
Horn suggest that the decline over North America is compensated
for elsewhere in the Northern Hemisphere but they are not able to
detail the compensation. If the downward trend is real, then it
would follow that the mean has become a more difficult standard
to better. As will be seen in later sections, model
4:.
-- - ANNUAL FREQUENCY OF(600
7 NORTH AMERICAN
LL U CYCLOGENESIS 550
a 6 CWHITTAKER AND HORN, 1981)0Y 9iI00 Illi
W)
0LLz
0 0 0 0 N N N N O 0
'. , o < 5 - -1-50o
ZZ zz ZzLZ zz z z) -z0 zV) < 4
"%'< 3 -
I..,, .
Fig. 11. RMSE for the (1885-1960) means as a forecastby season and year (1960-1982). The line with circlesis the trend in the annual frequency of North Americancyclogenesis (after Whittaker and Horn, 1981).
36 -
LLI~~A'\ L - -
i I
.
4.
6-
LL
hIJ 0,
0
uia
A 0 Ld
~Fig. 12. AAE of the mean as a forecast by season andyear ( 1960-1982).
33
0 -U7
U 1 n C N CO •f N C
forecast skill does not show a secuilar decline. Forecast still
of the models being tested remained high during the period Uif
improvement of the mean as a forecast. We interpret this to
indicate that model forecast skill is not sensitive to magnitude
* . of the departure from the mean represented by the observed
4 conditions.
M~agnitude Versuls theSign of Fogrecasted Anomalies
The quadratic skill score measures how well the forecast model
predicts the size of the departure from the observed conditions
with penalty proportional to the square of the departure from the
mean. The percent skill score measures how well the forecast
model predicts whether the departure will be + (above the mean)
or - (below the mean). Clearly, a model that does a good job of
predicting the magnitude of the anomaly should also do a good job
of predicting the sign of the anomaly. The reverse is not
necessarily true. Accordingly, we have plotted the quadratic
skill scores of Model I for all 12 six-month season forecasts for
the period 1960-1963 against the percent correct skill scores for
the same period (Fig. 13). When percent correct forecast skill
U falls below 60%, quadratic skill is negative. The relationship
is strongly linear; however, care should be exercised when
percent correct skill falls below 60% because skill in
forecasting the magnitude of the anomaly cannot be demonstrated.
.0.
Q 0.8-
4Z'
0.
-0.4
so607 8 0 o
PECN SKL
Fig 13 h1eainhpbtwe-ecn orc n
qudrti skl0.2s(16-93 o MdlI arsal 12sxmnhsao oeassJrs niae hmen 0o h w esrso kl;hrzna ieith0 eoqartcsil lee; vria iei h
0.5sgiicnelvlfo oa es fsil
-0.2
2. 2
limit of percent correct skill that is associated with quadratic
magnitude skill (see Fig. 13). Areas with skill less than 607. are
not contoured. The grid cells indicated with a black circle are
those grid cells where 21correct forecasts were made in trials.
This 100%. correct score occurs in regions of generally high
forecast skill and they are not outliers due to chance.
Four areas of excellent skill in all seasons are f ound: 1) off0
the east coast of the U.S.Z 2) tn areas north of 50 N latitude;
3) across the northern plains; and 4) an area extending
northeastward from the southern plains. These four areas
represent four important storm tracks that are not evident in the
charts of the means of cyclone frequencies (Hayden, 1981a and
* - Hayden and Smith, 1982). The central region of the eastern U.S.
is generally forecast with a skill of at least 707.) but small
regions of lower skill occur in some seasons.
If we use actual local skill scores as a proxy for the attribute
of predictability (see Madden and Shea, 1978) then the geography
of skill presented here is at odds with that reported by others.
Madden finds that predictability is highest in coastal areas and
declines toward the interior of the country. This is not the
case for cyclone frequency prediction. Predictability does not
decrease toward the interior of the continent or in the offshore
direction and skill along the coast is generally lower than in
* ar notconoure. 107. orret soresare indiate% ybakcrls
A-42
.......................... R
90-0
eg0
Fi.1. eray-uylca ecetsrecoklscors (960-980 forModl I Skils ess han60 Z
arenotconourd. 0cretsoe r niae
by blck cicles
-4:-
so.o
890
6041
-44-0
Fig 16 Mac-Ags loa pecn corc skill ,-. ---
01
0 0
-44e
0 13
OhO
01
2S 2.. .. . . .S-
I-I
,300
Fig. 18. May-October local percent correct skill scores(1960-1980) for Model I. Skills less than 60% are notcontoured. 100% correct scores are indicated by black Aci r cles.
-46-
. ... . ..
.10
S~10 q
8U
Fig. 19. June-November local percent correct skillscores (1960-1980) for Model I. Skills less than 60%.are not contoured. 100%. correct scores are indicatedby black circles.
-47-
... .... ."..9 7
* 80
Wcre (1 .01.0).o.Md....kils les.ha.6.ar : no.onord.10. corc scrs.r.idcaeby black.circles
Fig. 22. September-February local percent correct skillscores (1960-1980) for Model 1. Skills less than 60%.are not contoured. 100% correct scores are indicatedby black circles.
-50-
-700
AA
.R.
.4.
Fig.~~~~~~~ 23 Oco.-ac l.a pecn corc .kill
4.A
-47
.*.... ... so
'Kwr I, -x~ *-_K-'- I'.~~ T9. 7$9 7." 12 V. W-. 727 L
soO
.......
%00
80
Fig. 24. November-April local percent correct skillscores (1960-1980) for Model I. Skills less than 60%are not contoured. 1007% correct scores are indicatedby black circles.
.52
LILA- . . ' i S . . * * * . *. 4 . * r
90-7
4.80
X a,
.....
Fi. . Noebr-pi loa pecn cor. skill.. ... :*..,§:
.40
Fig 254 ~4. Noebr-p i loa pecn corc skill... -*4 4***
74~ d
Global Skill
Figures 26, 217 and 26 show the global percent correct skill score
by season and year for Models I, II and III. The three time
series of forecast skill are similar in gross form as well as in
- . most of the details. Some important differences are evident.
* Model II (Fig. 27) had a failure in the mid-1960s that was not
-- t present in Model I or Model 111 (Figs. 26 and 26). Model III had
a failure in 1978 that was not evident in either Models I or II.
The failure in mid-1975 is present in all three models but Model
III was clearly the best forecast of the three that season. In
contrast, peaks in the three curves are congruent. These
differences are important in that by running all three models for
each forecast differences will be revealed and possible forecast
failure may be forewarned.
The most serious kind of forecast failure is the general decline
in forecast skill. Such a depression of skill occurred in the
mid-1970s and lasted about three years. During this three-year
period the numbers of cyclones increased and the variability in
N'. cyclone numbers also increased. Apparently a mode of variation
occurred that the models were not able to predict. In earlier
studies (Hayden, 1981a) we used a jackknife procedure
t. Models 1, II and III were used in operational trials beginning in
January 1961. Three years of trials have now been completed. Two
forecasts were inadvertently not verified as of this writing
(June-November and July-December 1983). A total of 34 forecasts
were made with each model version. Eighty-seven grid cell
locations were forecast. In all 2958 forecasts were made using
Keach model. This is a sufficiently large sample such that the
* global scores from this period can be reasonably compared with
t 4 those of the hindcast period (1960-1980). In earlier sections of
this report data from the operational period were merged with the
hindcast period and so some comparisons have already been made.
In this section a specific assessment of the performance of the
models in real time forecasting is presented.
Local skill scores are usually averaged only over time, however,
in this case only three forecasts were made at each grid cell for
each season. This sample is too small to be meaningful so we
have averaged across all seasons. The sample size in each grid
* -86-
cell is now 34 and a reasonable estimate of local skill in the
operational period can be made.
Figures 46, 47 and 48 show the season averaged local skills for
Models I, II and III. The regions of high skill and regions of
low skill during the operational period are essentially the same
as found for the hindcast period (Figs. 14-25). Perfect forecasts
(34 correct in 34 trials) were made for 10 grid cells in Model I,
6 in Model II and 8 in Model 1II. The locations of these perfect
forecasts were like those that occurred in the hindcast period.
The local skills differed little between Models I, II and III. We
conclude that the models are stable in a spatial sense relative
to the hindcast period and because the skills high we assume also
that the stability extends back into the dependent data period
(1885-1980).
Global Skill
Global skill scores by model and season are reported in Tables
XII, XIII and XIV. Percent correct, Heidke, deviation, and
quadratic skill scores are given as are the average absolute
errors, root mean square errors, and their error reductions over
the errors of the long term means as forecasts.
89
so 80
4'
80
Fig. 46. Model I local skill scores averaged across allseasons for the operational forecast period. The unitsare percent correct in 34 forecasts. Solid blackcircles indicate grid calls where 34 correct forecastswere made in 34 trials.
-90-
k7L
.i7
'lee
40 A
-. 4e
80
-a7
Fig. 47. Model II local skill scores averaged across*all seasons for the operational forecast period. The
uits are percent correct in 34 forecasts. Solid blackcircles indicate grid calls where 34 correct forecastswerea made in 34 trial.
-91 -
S-
V4S
801
wer maean34leas
90 -7I-80
Fig. 48 Moe III4~h local~~j skl scre avrgdars
Global percent correct scores averaged across all seasons for all
three models in operational forecasts (76.5%, 75.8% and 76.8%)
out-performed the models in the hindcast period (73.6%, 74.6% and
75.9%). Heidke skill scores followed suit. Deviation and
quadratic skill scores were slightly lower in the operational
trials compared to those of the hindcast period. AAE and RMSE
were smaller during the operational period than in the hindcast
period but the error reductions were also smaller. This
circumstance results from the fact that there has been a decline
in the size of the observed cyclone frequency departures from the
long term means (see Figs. 11 and 12).
Overall there was no degradation of the models when applied on a
real-time forecasting basis. This is extremely encouraging as it
weighs well regarding reliability of the models tested.
-9-
_ Table XII
I%" 5Po -..
, Model I Skill Scores by Season +or the Operational Period
N AVERAGE 76.8 .536 .188 .316 2.09(19.0) 2.60(20.1)
* only 1981 and 1982 included
I
95
S S - -. ..
FOEATCOPRSN
Inti eto oeatmd drn h eido prtoa
foeatn sn l he fth oesi xmndi eal
Th uy t eebr 18 eao a eetd fr ticoprsn hssao a eece eas * &Ba frcs
In thiveiona a forecast madeguring the perid of operationa
Toeatdaomle o he July-toDecember 1982 season waseselcted r ti
comparison. this seasonewast seece boecaulsewhr iw*an al forecas
thatews asedsuessfutilliyao the same ecsta bus as thelaveage
forecas ad incssu forecast. trilsiTe purpriie ar grtestdyt
simil artes differences between the three odeael versosafo
aneds individa g forecast ies 49,aie w0and 51ve shcowe te
74mnmmvausi h forecasted anomale ffrte JuyDcm e d 192reao priedsictedr
byep a Model II diIicaIteis clea Atati cal trek moes gaive
etsentiallyi the sameit foecst Asw Otledn eswhee whdens al thre
there ae diffestrince btweenthel three oforcsenml
fild. herageoffoecstdanmaie ws evn ycons6e
..•
,
9 8 0
i.. 4. Mi c f anml
-- 7
-- JULY-DECEMBE
•~~~Jl 1: i~ 4, 1982
.. r.90 80
L} Fig. 49. Model I predicted cyclone frequency anomaliesf- or the Jul y-December 1982 season. Sol id arrows
• "- "indicate axes a+ maximum positive anomaly. Dotted.'.'.-. arrows i ndi cate axes a+ max imum negat ive anomaly.• °,'Dashed arrows indicate local maxima in a region of
negative anomal ies.
- 97-
90 -. 70
800
NN
*~~.... .........*4c***
W. A
Fig.~~~~~ .0 Moe .I prd. e .yln frqe. an.omalies.
fs-sr the: JulyDecmbe 1982 sa.Sldarw
arrow in MdateI axeied of con maimmregative anomaly.s
negative anomalies.
-98 -
WO!! WW W AZ~
-2-
80
Fig. 51. Model III predicted cyclone frequencyanomalies for the July-December 1982 season. Solidarrows indicate axes of maximum positive anomaly.Dotted arrows indicate axes of maximum negativeanomaly. Dashed arrows indicate local maxima in a
* region of negative anomalies.
-S%
the Gulf of Mexico. Because most of the prediction errors tend to
occur where the forecasted anomalies are between +1 and -1
cyclones per grid cell, there is value in examining which model
- . version has the smallest area between +1j and --1 cyclones per grid
S' cell. Model III is the best in this regard. This relationship
between forecast skill and forecasted anomaly magnitude can be
verified by examining charts of skill scores for each of the
three models (Fig. 52, 53 and 54).
Figures 55, 56, and 57 show the forecasted cyclone frequencies
for the July-December period, i.e., the frequency anomalies plus
the long term mean frequencies. The arrows indicate the "ridge
-Ilines" of maximum forecasted cyclone frequencies. The major
differences between Models I, II and III regarding the forecasted
tracks are found in the southeastern U.S. Analyses of Model II
forecasted frequencies indicated a double track across the Gulf
states with both tracks further north than the single tracks
C. indicated in Models I and 111. The results of analyses of the
9. actual occurring cyclones in July-December 1982 are shown in
Figure 57. The double track indicated by Model II is evident in
the observations. While the field of observed cyclone
frequencies is more complex than the forecasted fields, most of
the features of the forecasted fields are evident in the
observations. Global percent skill was 77.0. for Model 1, 75.9%
for Model 11 and 75.9% for Model 111. While Model II did well in
predicting the tracks across the south, the overall skill for
10 C
080
SAZ
.... .... ...
.90
80
Fig. 52. Model I local percent correct skill .scores
(1960-1980) for the July-December forecast season.Heavy contours indicate skill scores equal to or lessthan 67% correct. Grid cells with 100%. scores are not
V shown.
T .y'~
90 -704.80
S.,Z
-70
.... ...
JUL-DCEBE
YCOEFRQEC
FO CA'
eN"
l
90-7
980
Fig. 53. Model 11 local percent correct skill scores
(1960-1980) for the July-December forecast season.
Heavy contour% indicate skill scores equal to or less
than 67% correct. Grid calls with 100%. scores are not
shown.
-102-
.lee
'p.e
9040
44
JUY-EEME
CYLN.REUNYFOE S
9 0... ....4 7
'80
Fi.5.Mdl II oa ecn oretsilsoe
(16-90 o h uyDeebr frcs esn
Hev otusIdiae silsoeseult rls
tha. 67%oe LZlcl ecn correct. Grdkil it 10 scoresno
4 shown.
90 ~80 7
.. . . . .. . . . .. . . .......
q.7
- SO
*'10
-380
mIu prdited cuycoe per grdcel
Oe.- . C .....
.....
'AC..
4.'z
'..N
%!. . ..
80
Fig. 57. Model Ill predicted cyclone frequencies forthe July-December 1982 season. Arrows indicate theaxeS of maximum predicted cyclones per grid cell.
-106
800
800
Fig.~~~~~~ .9 .b~re cyln frqe.e f. th..July-Dcembe 1962seaso.-Arro--i-dcate---s-o
mFimum58 Oreqencied.Teunr cyclon qeces for trid
call.
-107 -
Model III was not different from the skills for Models I and III.
In general, we find that global skill rarely differs between
models except when there is a persistence and forecast failure.
There are frequently differences in the details of the forecast
and there are differences in local skill between models. The
three models are rarely contradictory and when they are the
forecast that is fundamentally different is usually the forecast
that fails.
- CONCLUS IONS
Climate Predictability
Over the last two decades the predictability of climate has
become a fundamental topic of research and a topic about which
there exists fundamental differences among scientists. This
circumstance prompted Lorenz (1973) to note that the46,
Cpredictability of climate will be established when someone shows
that it can be done. Much of the recent work on climate
predictability focuses on the partitioning of signal and noise in
historical data. The spatial and temporal variations in the
4 signal-to-noise ratio thus serves as a proxy of the attribute of
predictability. Much of the work to date focuses on temperature,
4.. pressure and precipitation. Based on signal-to-noise ratios for
monthly temperatures a general rule of thumb has emerged: climate
predictability is highest along the coastal margins of the
continents and decreases toward the interior of the continents.
Based on our work we conclude that this rule of thumb does not
apply to the prediction of cyclone frequencies. A different
-pattern of predictability emerges. We would then conclude that
predictability will vary from parameter to parameter and
according to season duration.
Given Lorenz's rather pragmatic approach to the question of
predictability we conclude that such demonstration of
predictability has been realized for a climatic parameter of
fundamental synoptic significance. As such, new avenues are now
open to a new approach to the prediction problem.
Categorical Forecast Skill
Most attempts to forecast climate take a categorical approach.
-a.5 Forecasts of above or below the long-term means are forwarded.
On occasion terciles or quartiles are predicted. Both
categorical and numerical forecasts have been prepared and
evaluated in this study. Based on the results of the categorical
2-by-2 tests of forecast skill we place the level of forecast
-109-
skill for each of the three models developed and tested at about
75%.. This is a global skill that covers an 87-location forecast
- domain and a period of forecast trials on independent data that
spans 25 years. This skill level meets the requliremnents of
statistical significance kp = 0.05) at an individual location let
*alone as the average for 837 locations. The categorical skills
achieved could not have occurred by chance alone. Cyclone
frequencies relative to the long-term means are predictable
quantities.
Forecast skill is high in baroclinic and low in barotropic
areas. Also skill is generally low along the coastal margins and
along the northern shores of the Great Lakes. Both of these areas
are axes of maximum frequencies in the long-term means but are
not axes of maximum standard deviations about the means.
Magnitude modulation of the mean pattern is not predictable by
the methods used in this study.
Categorical forecast skills are uniform from season to season and
show no trends in levels over the period of forecast trials.
When the mean for the period 1685-1960 is used to predict the
conditions in the years that followed it turns out that the mean
has become progressively better, as a forecast. This is due to
the general decline in variability in cyclone frequencies over
the last two decades. A similar decline in forecast skill for
the models is not observed even though the average departure
(mean minus observed) has become smaller. The sign of these
smaller anomalies remains as predictable as at the beginning of
the test period when cyclone numbers were higher.
Numerical Forecast Skill
Numerical forecasts were made and evaluated for skill. The skill
was measured using a penalty proportional to the size of error
(deviation skill score) and also using a squaring of the penalty
(quadratic skill score). Positive skill is found in 95%. of the
forecasts made. Since 286 forecasts were made (12 seasons times
25 years less 2 missing seasons) it is highly unlikely that this
result is due to chance.
Numerical forecast skill was found to be linearly related to
* 2-by--2 categorical forecast skill. It is clear that models
- .p.exhibit both categorical and numerical skill. It is interesting
to note that numerical skill goes to zero as the categorical
skill falls below 60%. This then may be a bottom level of skill
for climate prediction models, i.e., when numerical skill cannot
be demonstrated. In our work we have applied a considerably
4 higher standard..
'- '4"
* . Forecast Failures
Forecast failures, i.e., categorical skill below 50%. or numerical
- skill below 0 occurred only about 5%/ of the time. Poor forecast
skill (60 to 65%) occurred and persisted for a few years in the
a. mid-1970s. We conclude that the variability during this period
was not contained within the statistical base used to construct
the models. Earlier studies using jackknifed trials for the
entire 95 year period revealed no other period with a comparable
persistent period of failures. The type of statistical models
employed cannot predict patterns not included in the training
base. The three years beginning about February 1973 then become
.4 a special case that merits additional study.
The duration of forecast failure is interesting. Here a
"forecast failure event" is defined as a 10%/ skill score fall and
a 107. skill score rise (e.g. see Fig. 26). Of the 48 "events" 25
had a one forecast duration; 12 a two forecast duration; 8 a
three forecast duration; and, 3 a four forecast duration. We
infer this to indicate that when cyclone frequency climate
changes and persistence fails that the model fails but recovers
to correctly forecast the changed climate on the next or
following forecast. While models are not instantaneously
responsive to changes in cyclone tracks and numbers the response
4
4
is less than 1/3 of the duration of the period forecast.
Forecast Models
Three versions of the forecast models were constructed and
tested. They differed in regards to the attribute of rotation of
principal component axes. The three models performed in a global
sense essentially the same. There were slight differences in
skill from place to place and from season to season. In general,
the forecast failures found in one model were not the same as
those found in the other models. Forecast successes were common
among models. We conclude that running all three models is a
positive utility and may provide a means of detecting poor
A. forecasts at the time of issue.
Hindcast vs OP-erational Forecasts
Three years of operational forecasting have been completed. The
results of these operational trials are indistinguishable from
those made on independent data in a hindcast mode. We conclude
that the prediction models are stable.
-113-
- q
V BIBLIOGRAPHY
Bergeron, T., 1930: Richtlinien einer dynamischen Klimatologie. Met. Zeit. 47,
246-262.
Brier, G. and Allen, R., 1951: Verification of Weather Forecasts. In Compendium
of Meteorology, p. 841-848, Thomas F. Malone, Ed., American Meteorological
20. ABSTRACT (Continue en reveree otdo If neeory and Identify by block nenber)
.Climate prediction models based on multivariate analyses of cyclonefrequencies are constructed from historical data (1885-1960) and evaluatedfor forecast skill on independent data (1960-1983). Cyclone frequencies arepredicted for six-month duration seasons at 87 locations over North Americaand the vestern North Atlantic from 27.50 to 550. Three types of principalcomponents are constructed and tested. Model I uses unrotated principalcomponent axes, Model II uses rigid rotation of the component axes, andModel III uses oblique rotations of the component axes.
D FORM 1473 EOITION OF I NOV 63 IS OBSOLETE/M 0102-014- 6dol Unlassified
SECURITY CLASSIFICATION OF THIS PAGE (When Dafe Entered)
p.
O. 26, 1977 JOB 463-101
SYSTEMATIC GEOGRAPHY LISTING0001 1 OFFICE OF NAVAL RESEARCH 2 COPIES0001 2 GEOGRAPHY PROGRAMS0001 3 CODE 4620001 4 ARLINGTON, VIRGINIA 22217
0002 1 DEFENSE DOCUMENTATION CENTER 12 COPIES0002 2 CAMERON STATION0002 3 ALEXANDRIA, VIRGINIA 22314
0003 1 DIRECTOR, NAVAL RESEARCH LAe 6 COPIES0003 2 ATTENTION TECHNICAL INFORMATION OFFICER0003 3 WASHINGTON, D. C. 20375
0 . 0004 1 DIRECTOR0004 2 OFFICE OF NAVAL RESEARCH BRANCH OFFICE0004 3 1030 EAST GREEN STREET0004 4 PASADENA, CALIFORNIA 91101 .
C005 1 DIRECTOR0005 2 OFFICE OF NAVAL RESEARCH BRANCH OFFICE0005 3 536 SOUTH CLARK STREET0005 4 CHICAGO, ILLINOIS 60605
0006 1 DIRECTOR0006 2 OFFICE OF NAVAL RESEARCH BRANCH OFFICE0006 3 495 SUMMER STREET0006 4 CSTON, MASSACHUSETTS 02210
0007 1 COMMANDING OFFICERC007 2 OFFICE OF NAVAL RESEARCH BRANCH OFFICE
* 0007 3 BOX 390007 4 FPO NEW YORK 09510
0008 1 CHIEF OF-NAVAL RESEARCH0008 2 ASST. FOR MARINE CORPS MATTERS0008 3 CODE 1O0M0008 4 OFFICE OF NAVAL RESEARCH0008 5 ARLINGTON, VIRGINIA 22217
. 0009 1 OFFICE OF NAVAL RESEARCH0009 2 CODE 4800009 3 NATIONAL SPACE TECHNOLOGY LABORATORIES0009 4 BAY ST. LOUIS, MISSISSIPPI 39520