Detection of inhomogeneities in Daily climate records to Study Trends in Extreme Weather
Detection of Breaks in Random Data,
in Data Containing True Breaks, and in Real Data
Ralf Lindau
Daily Stew Meeting, Bonn – 14. June 2012
Internal and External Variance
Consider the differences of one station compared to a neighbour or a reference.
Breaks are defined by abrupt changes in the station-reference time series.
Internal variancewithin the subperiods
External variancebetween the means of different
subperiods
Criterion:Maximum external variance attained bya minimum number of breaks
Daily Stew Meeting, Bonn – 14. June 2012
Decomposition of Variance
n total number of yearsN subperiodsni years within a subperiod
The sum of external and internal variance is constant.
Daily Stew Meeting, Bonn – 14. June 2012
Three Questions
How do random data behave?
Needed as a stop criterion for the number of significant breaks.
How do real breaks behave theoretically?
How do real data behave?
Daily Stew Meeting, Bonn – 14. June 2012
Segment averages
with stddev = 1
Segment averages xi scatter randomly
mean : 0
stddev: 1/
Because any deviation from zero can beseen as inaccuracy due to the limited number of members.
in
Daily Stew Meeting, Bonn – 14. June 2012
2-distribution
The external varianceis equal to the mean square sumof a random standard normal distributed variable.
Weighted measure for thevariability of the subperiods‘means
Daily Stew Meeting, Bonn – 14. June 2012
From 2 to distribution
n = 21 yearsk = 7 breaks
data
X ~ 2(a) and Y ~ 2(b)
X / (X+Y) ~ (a/2, b/2)
If we normalize a chi2-distributed variable by the sum of itself and another chi2-distributed variable, the result will be -distributed.
)(
)()(),(
ba
babaB
2
1,2
1)(
12
112
knkB
vvvp
knk
with
Daily Stew Meeting, Bonn – 14. June 2012
Incomplete Beta Function
2
1,2
1)(
12
112
knkB
vvvp
knk
External variance v is -distributedand depends on n (years) and k (breaks):
2
ki
1
0
1)(i
l
lml vvl
mvP
Solvable for even k and odd n:
2
3n
m
The exceeding probability P gives thebest (maximum) solution for v
Incomplete Beta Function
v
pdvvP0
1)(
We are interested in the best solution, with the highest external variance.We need the exceeding probability for high varext
Daily Stew Meeting, Bonn – 14. June 2012
P(v) for different k
Can we give a formula for
in order to derive v(k)?
220
breaksdk
dv
Increasing the break number from k to k+1 has two consequences:
1. The probability function changes.
2. The number combinations increase.
Daily Stew Meeting, Bonn – 14. June 2012
dv/dk sketch
P(v) is a complicated function and hard to invert into v(P).
Thus, dv is concluded from dP / slope.
And the solution is:
k breaks
k+1 breaks
1
0
1)(i
l
lml vvl
mvP
vk
vkn
k
knc
kn
v
dk
dv
1
1ln2
11ln
1
12
Daily Stew Meeting, Bonn – 14. June 2012
Solution
5ln21ln
2
1
1
1*
***
k
kk
dk
dv
v
k
***
*
1
5ln21ln
2
1
1
1dk
kk
kdv
v
*
2
1
*
*
2
1)5ln(2* 1
11k
k
kkv
Daily Stew Meeting, Bonn – 14. June 2012
Constance of Solution
101 ye
ars21 yea
rs
The solution for the exponent is constant for different length oftime series (21 and 101 years).
Daily Stew Meeting, Bonn – 14. June 2012
The extisting algorithm Prodige
Original formulation of Caussinus and Mestre for the penalty term in Prodige
Translation into terms used by us.
Normalisation by k* = k / (n -1)
Derivation to get the minimum
In Prodige it is postulated that the relative gain of external variance is a constant for given n.
minln21ln * nkv
0ln21
1*
ndk
dv
v
ndk
dv
vln2
1
1*
minln1
21ln
n
n
kv
min)ln(
1
2
)(
)(
1ln)(
1
2
1
1
2
nn
lk
YY
YYn
YCn
ii
k
j
jj
k
Daily Stew Meeting, Bonn – 14. June 2012
Our Results vs Prodige
We know the function for the relative gain of external variance.
Its uncertainty as given by isolines of exceeding probabilities for 2-i are characterised by constant distances.
Prodige propose a constant of 2 ln(n) ≈ 9
Exceeding probability1/1281/641/321/161/81/4
Daily Stew Meeting, Bonn – 14. June 2012
Wrong Direction
n = 101 years n = 21 years
True Breaks
Daily Stew Meeting, Bonn – 14. June 2012
Only true for constant lengths
True breaks with fixed distances behave identical to random data.
For realistic random lengths the exponent is slightly increased.
Daily Stew Meeting, Bonn – 14. June 2012
Sub-periods withrandom lengths
Sub-periods withconstant lengths
data
theory
theory
data
Distribution of Lengths
The distribution of the sub-periods’ lengths as obtained by randomly inserted breaks is known.
If necessary, it could be taken into account.
Daily Stew Meeting, Bonn – 14. June 2012
Break vs Scatter Regime
The two governing parameters are:
1) The relative amount of break variance compared to the scatter variance
2) The quotient
The latter defines how much faster the internal variance decreases in the “true break regime” compared to the “scatter regime”
If the relative scatter is low (10%) the transition between the regimes is clearly visible at 15 from 19 breaks.
Daily Stew Meeting, Bonn – 14. June 2012
Time series lengthNumber of true breaks
Real Data
1050 Climate Stations exist in Germany.
For each station the next eastward (to avoid identical pairs) neighbour between 10 km and 30 km is searched.
443 stations pairs remain.
Daily Stew Meeting, Bonn – 14. June 2012
All Stations Neighbouring pairs
Data Focus
This project deals with daily climate data.
Findings about their extremes are in the focus.
At least statements about the
•distribution (moments)
•percentiles
•indices (number of wet days per month)
should be possible.
Daily Stew Meeting, Bonn – 14. June 2012
Parameters
Daily Stew Meeting, Bonn – 14. June 2012
Interesting for break detection:
Problem parameters PP
Expected physical problems
Temperature at high sun shine duration
Temperature at high pressure
Temperature at high diurnal cycle
Temperature during snow cover
Temperature depending on general
weather situation
Temperature during rain
Rain at high wind speed
Expected technical problems
Frequency of rainy days below 1 mm
Tenth of precipitation report
Difference between Tmean and (Tmax-Tmin)
Per se interesting parameters P
Monthly means
Temperature
Precipitation, etc.
Breaks are moresensitive to problem parameters. Breaks in PP may help to find breaks in P
Distribution and extremes
Standard deviationSkewnessKurtosisMaximumMinimum90 percentile
project focus(more sensitive?)
Two Parameter Pairs
1a. Monthly mean temperature
1b. Monthly maximum temperature
2a. Monthly precipitation sum
2b. Frequency of rainy days below 1 mm
Can the sensitive parameter help to find breaks in the mean?
Daily Stew Meeting, Bonn – 14. June 2012
(Project focus)
(Problem parameter)
“Drizzle days” are often excluded from rainy days to calculate the interesting indices:
•Monthly Rain Frequency •Consecutive Dry Days
“Drizzle frequency” is not only a technical problem parameter, but also a per se interesting one.
Monthly Mean Temperature
Daily Stew Meeting, Bonn – 14. June 2012
Temperature differencebetween Ellwangen-RindelbachandCrailsheim-Alexandersreutshows 1 strong and 3 further significant breaks.
The statistical signature confirms it:The first break contains much variance.2, 3 and 4 are only slightly larger than the Mestre penalty.
Break Statistics
Daily Stew Meeting, Bonn – 14. June 2012
Individual pair All pairs
r = 0.937
Monthly Maximum
For the monthly temperature maximum, only the largest breaks are detectable, probably due to the reduced correlation.
Daily Stew Meeting, Bonn – 14. June 2012
r = 0.865
Additional Breaks?In maximum temperature there are less breaks. Are they nevertheless new compared to those in mean temperature?
Enhance the penalty from about 12 (i.e. 2 ln(n)) to 60.)
With n = 600, it means that 10% of the remaining internal variance has to be explained by each additional break. Otherwise the search is stopped.
For such increased requirements 297 breaks are found in the mean and 67 in the maximum.
Nearly all breaks in tmax exist also in tmean.
The “stddev” of temporal distance is 1.75 years.
Daily Stew Meeting, Bonn – 14. June 2012
Answer: No
Nearly no new break is found by the sensitive parameter Monthly Maximum Temperature.
The lower correlation (0.865 vs. 0.937 doubled rms) hamper obviously the break finding capability of the sensitive parameter.
However, the high correlation of break positions may the opposite direction become possible: To find break positions in the maximum temperature by considering the mean temperature.
Daily Stew Meeting, Bonn – 14. June 2012
“Drizzle Days”
Monthly frequency of rainy days below 1mm.
This parameter is highly inhomogeneous.
Even for individual stations the break is evident.
Daily Stew Meeting, Bonn – 14. June 2012
Drizzle vs. Mean Precip.
Daily Stew Meeting, Bonn – 14. June 2012
In the drizzle parameter moresignificant breaks are found (index 43.3 compared to 28.8),although the correlation is low,(0.339 compared to 0.855).
Are the break positions againcorrelated?
Correlation of break positions
Many new breaks are found. Only 12 breaks of the drizzle parameter are found at all somewhere the corresponding time series of mean precipitation, but mostly far away.
In 93 time series pairs one or more breaks are found for drizzle, but even not a single in mean precipitation.
Are these new breaks also included, but hidden in mean precipitation?
Daily Stew Meeting, Bonn – 14. June 2012
remember
Forced Breaks (1)
Daily Stew Meeting, Bonn – 14. June 2012
Forced Breaks (2)
Also in average, the external variance decreases only by about 1%, if “drizzle breaks” are inserted into the time series of mean precipitation.
1% is the mean decrease of a random n=100 time series and it is beta-distributed.
However, here n is equal to 600. Is the result then a bit better than random?
Daily Stew Meeting, Bonn – 14. June 2012
Simulated Data
1. Blind try of 3 breaks in a 21 years random time series
2. Blind try of 3 breaks in a 21 years constant time series with 6 true breaks.
3. Blind try 3 breaks in a 21 years time series with 6 true breaks plus random scatter.
Daily Stew Meeting, Bonn – 14. June 2012
1. Purely random 2. Pure true breaks 3. Realistic mix
Realistic Mixed Data
Real data is expected to be similar to a realistic mix, rather than to random scatter.
As it then includes also real breaks, the Null Hypothesis is not random scatter, but a realistic mix.
Here the blindly found external variance is again -distributed, but generally larger. How much is difficult to quantify in advance . It depends on the signal to noise ratio.
Daily Stew Meeting, Bonn – 14. June 2012
Daily Stew Meeting, Bonn – 14. June 2012
Conclusions• The analysis of random data shows that the external variance is -distributed,
which leads to a new formulation for the penalty term.
• True breaks are also -distributed. Their external variance increases faster by a factor of n/nk compared to random scatter.
• Are sensitive parameters helpful to find additional breaks?Monthly maximum temperature:
Due to the reduced spatial correlation Tmax “finds” less breaks.
Those identified are even better visible in Tmean.Drizzle parameter:
Highly inhomogeneous Many breaks found.But they do not coincide with breaks in mean precipitation.
• Vice versa we expect that Tmean breaks are helpful to find breaks in Tmax. But the prove of significance will be difficult.