TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models

TIME SERIES ANALYSISTime Domain Models: Red Noise; AR

and ARMA models

LECTURE 7

Supplementary Readings:

Wilks, chapters 8

Recall:

Statistical Model[Observed Data] = [Signal] + [Noise]

“noise” has to satisfy certain properties! If not, we must iterate on this process...

We will seek a general method of specifying just such a model…

Statistics of the time series don’t change with time

We conventionally assume Stationarity

•Weak Stationarity (Covariance Stationarity)

statistics are a function of lag k but not absolute time t

•Strict Stationarity

•Cyclostationarity

statistics are a periodic function of lag k

Time Series ModelingAll linear time series models can be written in the form:

1111

11

)(

mt

M

mtkt

K

kkt

mxx

We assume that are Gaussian distributed.

Autoregressive Moving-Average Model(“ARMA”)

Box and Jenkins (1976)

ARMA(K,M) MODEL

111

1 )(

tkt

K

kkt xx

Consider Special case of simple Autoregressive AR(k) model (m=0)

Suppose k=1, and define 1

11 )(

ttt xx

This should look familiar!

Special case of a Markov Process (a process for which the state of system depends on previous states)

Time Series ModelingAll linear time series models can be written in the form:

1111

11

)(

mt

M

mtkt

K

kkt

mxx

Suppose k=1, and define 1

11 )(

ttt xx

Time Series Modeling

Assume process is zero mean, then

11

tttxx

Lag-one correlated process or “AR(1)” process…

111

1 )(

tkt

K

kkt xx

Consider Special case of simple Autoregressive AR(k) model (m=0)

All linear time series models can be written in the form:

1111

11

)(

mt

M

mtkt

K

kkt

mxx

11

kky

ky

ky

kky

ky

ky

12

1

ky

kky

ky

ky

n

k

n

k

n

k 12

11

1

1

1

1

1

21

1

1

1

1

ky

ky

ky

n

k

n

k

For simplicity, we assume zero mean

2

1

y

yy

nlii

l

AR(1) Process

-4 -3 -2 -1 0 1 2 3 40

20

40

60

80

100

120

140

Let us take this series as a random forcing

0 100 200 300 400 500 600 700 800 900 1000-4

-3

-2

-1

0

1

2

3

4

85.02

181120 N

AR(1) Process

0 100 200 300 400 500 600 700 800 900 1000-4

-3

-2

-1

0

1

2

3

4

0 10 20 30 40 50 60 70 80 90 100-3

-2

-1

0

1

2

3

4

AR(1) Process

Blue: =0.4Red: =0.7

Let us take this series as a random forcing

What is the standard deviation of an AR(1) process?

2222 yy

2)1

(21

kk

yk

y

11

kky

ky

222

112

kk

yk

y

22

1

2

y

AR(1) Process

0 10 20 30 40 50 60 70 80 90 100-6

-4

-2

0

2

4

6

8

10

Blue: =0.4Red: =0.7Green: =0.9

22

1

2

y

AR(1) Process

010020030040050060070080090010000

5

10

15

20

25

30

35

40

45

50

Suppose =1

Random Walk (“Brownian Motion”)

Not stationary

How might we try to turn this into a stationary time

series?

AR(1) Process

22

1

2

y

11

kky

ky

Variance is infinite!

11

kky

ky

Let us define the lag-k autocorrelation:

We can approximate:

2)(1

xxvar i

n

i

i

n

i

xx 1

varkn

xxxx

krkii

kn

i

)(

))((1

AR(1) ProcessAutocorrelation Function

i

kn

i

xx

1

i

n

ki

xx

1

2)(1

xxvar

i

kn

i

2)(1

xxvar i

n

ki

varvarkn

xxxx

krkii

kn

i

)(

))((1

Let us assume the series x has been de-meaned…


2

1i

n

ixvar

varkn

xx

krkii

kn

i

)(1



Then:

i

kn

i

xx

1

i

n

ki

xx

1

2)(1

xxvar

i

kn

i

2)(1

xxvar i

n

ki

varvarkn

xxxx

krkii

kn

i

)(

))((1

0 50 100 150 200 250-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Serial correlation function

Autocorrelation Function

1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996336

338

340

342

344

346

348

350

352

354

356

CO2 since 1976

varkn

xx

krkii

kn

i

)(1

11

kky

ky

Recursively, we thus have for an AR(1) process,k

kr


kky

ky

11

]1

[

kkk

y

)1

(1

2

kkky

kkyky '112

)/exp()lnexp( kkk

rk

(Theoretical)

varkn

xx

krkii

kn

i

)(1

0 5 10 15 20 25-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

“tcf.m” scf.m

0 10 20 30 40 50 60 70 80 90 100-2

-1

0

1

2

3

4

=0.5 N=100

“rednoise.m”

)/exp()lnexp( kkk

rk


varkn

xx

krkii

kn

i

)(1

0 5 10 15 20 25-0.2

0

0.2

0.4

0.6

0.8

1

1.2

=0.5 N=100

“rednoise.m”

)/exp()lnexp( kkk

rk


0 50 100 150 200 250 300 350 400 450 500-4

-3

-2

-1

0

1

2

3

4

5

=0.5 N=500

“rednoise.m”

varkn

xx

krkii

kn

i

)(1

0 5 10 15 20 25-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 100 200 300 400 500 600 700 800 900 1000-4

-3

-2

-1

0

1

2

3

4

)/exp()lnexp( kkk

rk


Glacial Varves

=0.23 N=1000

varkn

xx

krkii

kn

i

)(1

0 5 10 15 20 250

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 50 100 150-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

)/exp()lnexp( kkk

rk


Northern Hem Temp

=0.75 N=144

varkn

xx

krkii

kn

i

)(1

0 5 10 15 20 25-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0 50 100 150-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

)/exp()lnexp( kkk

rk


Northern Hem Temp (linearly detrended)

=0.54 N=144

varkn

xx

krkii

kn

i

)(1

1 2 3 4 5 6 7 8 9 10 11-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

)/exp()lnexp( kkk

rk


0 50 100 150-2

-1

0

1

2

3

4

Dec-Mar Nino3

AR(1) Process

The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!

n/)1( 22

11

kky

ky

2/1

)(1

2

xxesbi

22

1

2

y

AR(1) Process

The sampling distribution for is given by the sampling distribution for the slope parameter in linear regression!

n/)1( 22

How do we determine if is significantly non-zero?

nt

/21

0

This is just the t test!

11

kky

ky

When Serial Correlation is Present, the variance of the mean must be adjusted,

nxVx

2)var(

121k k

rV

k

k Recall for AR(1) series,

AR(1) Process

Variance inflation factor

This effects the significance of regression/correlation as we saw previously…

Vnn /'

111

121

1121V

11

211

121

V?V

This effects the significance of

regression/correlation as we saw previously…

111

121

1121V

AR(1) Process

12 2

)/exp()lnexp( kkkk

ln/1

Suppose 1 1ln 1/1

Vnn /'

Multiply this equation by xt-k’ and sum,

111

1 )(

tkt

K

kkt xx

Now consider an AR(K) Process

1'11

'1'

tktkt

K

kkkttkt xxxxx

For simplicity, we assume zero mean

111

1

tkt

K

kktxx

11

'1'

kt

K

kkkttkt xxxx

kKKKK

kk

kk

kk

rrrr

rrrrrrrrrrrr

ˆ...ˆˆˆ...

ˆ...ˆˆˆˆ...ˆˆˆˆ...ˆˆˆ

332211

3312213

2132112

1231211

Yule-Walker Equations

Use

varkn

xx

krkii

kn

i

)(1

11

'1'

kt

K

kkkttkt xxxx

AR(K) Process

kk

x

rK

k

11

22

Several results obtained for the AR(1) model generalize readily to the AR(K) model:

kmk

K

km rr

1 nxVx

2)var(

121k k

rV

AR(K) Process

kKKKK

kk

kk

kk

rrrr

rrrrrrrrrrrr

ˆ...ˆˆˆ...


332211

3312213

2132112

1231211


AR(K) Process

kKKKK

kk

kk

kk

rrrr

rrrrrrrrrrrr

ˆ...ˆˆˆ...


332211

3312213

2132112

1231211


is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters

11211

ttttxxx The AR(2) model

The Yule-Walker equations give:

2112

1211

ˆˆˆˆ

rr

rrkk

x

rK

k

11

22

AR(2) Process

is particularly important because of the range of behavior it can describe w/ a parsimonious number of parameters

11211

ttttxxx The AR(2) model

The Yule-Walker equations give:

2112

1211

ˆˆˆˆ

rr

rr

Which readily gives:

2

1

2

122

2

1

211

1ˆ

1)1(ˆ

rrr

rrr

)1)(1(

2

2

1

2

2

2

rx

)1/(

)1/(

22

122

211

rr

kk

x

rK

k

11

22

kmk

K

km rr

1

2

AR(2) ProcessWhich readily gives:

2

1

2

122

2

1

211

1ˆ

1)1(ˆ

rrr

rrr

)1)(1(

2

2

1

2

2

2

rx

)1/(

)1/(

22

122

211

rr

0 1 2-1-2

+1

-1 1

2

1ˆˆ1ˆˆ

1ˆ

12

21

2

For stationarity, we must have:

kmk

K

km rr

1

2


)1/(

)1/(

22

122

211

rr

0 1 2-1-2

+1

-1 1

2

1ˆˆ1ˆˆ

1ˆ

12

21

2


Note that this model allows for independent lag-1 and lag-2

correlation, so that both positive correlation and negative correlation

are possible...kmk

K

km rr

1

2


)1/(

)1/(

22

122

211

rr

1ˆˆ1ˆˆ

1ˆ

12

21

2


Note that this model allows for independent lag-1 and lag-2

correlation, so that both positive correlation and negative correlation

are possible...

0 2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

4.0ˆ3.0ˆ

2

1

“artwo.m”

kmk

K

km rr

1

2

Selection Rules

AR(K) Process

nmmsmnnnmBIC )ln1()(

1ln)( 2

Bayesian Information Criterion

)1(2)(1

ln)( 2

mmsmnnnmAIC

Akaike Information Criterion

The minima in AIC or BIC represent an ‘optimal’ tradeoff between degrees of freedom and variance explained

ENSO

Multivariate ENSO Index

(“MEI”)

AR(K) Process

varkn

xx

krkii

kn

i

)(1

1 2 3 4 5 6 7 8 9 10 11-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

AR(1) FitAutocorrelation Function

0 50 100 150-2

-1

0

1

2

3

4

Dec-Mar Nino3

)/exp()lnexp( kkk

rk

varkn

xx

krkii

kn

i

)(1

1 2 3 4 5 6 7 8 9 10 11-0.4

-0.2

0

0.2

0.4

0.6

0.8

1


0 50 100 150-2

-1

0

1

2

3

4

Dec-Mar Nino3

)1/()1/(

22

122

211

rr kmk

K

km rr

1

(m>2)

varkn

xx

krkii

kn

i

)(1

1 2 3 4 5 6 7 8 9 10 11-0.4

-0.2

0

0.2

0.4

0.6

0.8

1


0 50 100 150-2

-1

0

1

2

3

4

Dec-Mar Nino3

Theoretical AR(3) Fit

varkn

xx

krkii

kn

i

)(1

1 2 3 4 5 6 7 8 9 10 11-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

AR(K) FitAutocorrelation Function

0 50 100 150-2

-1

0

1

2

3

4

Dec-Mar Nino3

Favors AR(K) Fit for K=?

Minimum in BIC?

Minimum in AIC?

MA model

1111

11

)(

mt

M

mtkt

K

kkt

mxx

Now, consider the case k=0

1111

mt

M

mttm

x

Pure Moving Average (MA) model, represents a running mean of the past M values.

Consider case where M=1…

MA(1) model

1111

11

)(

mt

M

mtkt

K

kkt

mxx

Now, consider the case k=0

1111

mt

M

mttm

x tt

11

)1( 2

122 x

)1/( 2

111 r

Consider case where M=1…

ARMA(1,1) model

tttt xx 1111 )(

2

1

1112

1

)221(2

x

11

2

1

11111 21

)()1(

r

TIME SERIES ANALYSIS Time Domain Models : Red Noise; AR and ARMA models

Documents