Stochastic Processes - lesson 8 · Stochastic Processes - lesson 8 Bo Friis Nielsen Institute of Mathematical Modelling Technical University of Denmark 2800 Kgs. Lyngby { Denmark

Stochastic Processes - lesson 8

Bo Friis Nielsen

Institute of Mathematical Modelling

Technical University of Denmark

2800 Kgs. Lyngby – Denmark

Email: bfn@imm.dtu.dk

Bo Friis Nielsen – 3/10-2000 2C04141

OutlineOutline

• Generating functions

OutlineOutline

� Sums of random variables - products of generating

functions

OutlineOutline

functions

� Moments of random variables - derivatives of generating

functions

OutlineOutline

functions

• Classification of Markov chain states

OutlineOutline

functions

• Classification of irreducible Markov chains

OutlineOutline

functions

• Limiting/stationary distribution

OutlineOutline

functions

• Limiting/stationary distribution

• Reading recommendations

Once again: highligt of generating functionsOnce again: highligt of generating functions

• X with pdf f(x),

• X with pdf f(x), GX(s) =

• X with pdf f(x), GX(s) = E(

x=0 sxf(x)

• X with GX(s) and Y with GY (s) independent,

x=0 sxf(x)

Z = X + Y

x=0 sxf(x)

Z = X + Y has GZ(s)

x=0 sxf(x)

Z = X + Y has GZ(s) = GX(s)GY (s)

x=0 sxf(x)

• X with GX(s)

x=0 sxf(x)

• X with GX(s) has E(X) = lims→1 G′(s) usually G′(1)

x=0 sxf(x)

• X with GX(s) has E(X) = lims→1 G′(s) usually G′(1)

� and V (X) = G′′(1) + G′(1)− (G′(1))2

Classification of Markov chain states - 6.2Classification of Markov chain states - 6.2

• States which cannot be left, once entered - absorbing

states

• States where the return some time in the future is certain -

recurrent or persistent states

� The time to return can be

? finite - postive recurrence/non-null persistent

? infite - null recurrent

• States where the return some time in the future is

uncertain - transient states

• States which can only be visited at certain time epochs -

periodic states

First passage probabilitiesFirst passage probabilities

The first passage probability (p. 201)

fij(n) = P{X1 6= j,X2 6= j, . . . , Xn−1 6= j,Xn = j|X0 = i}

This is the probability of reaching j for the first time at time

n having started in i.

The probability of ever reaching j

fij =∞∑

fij(n) ≤ 1

The probabilities fij(n) constitiute a probability distribution.

The probability of ever reaching j

fij =∞∑

fij(n) ≤ 1

The probabilities fij(n) constitiute a probability distribution.

On the contrary we cannot say anything in general on∑

n=1 pij(n) (the n-step transition probabilities)

First passage and first return timesFirst passage and first return times

The random variable Tij for the first passage time.

P{Tij = n} = fij(n)

For the first return time we use the shorter Ti.

We can define Ti by

Ti = min{n > 0 : Xn = i|X0 = i}

State classification by fii(n)State classification by fii(n)

• A state is recurrent (persistent) if fii = 1

� A state is positive recurrent or non-null persistent if

E(Ti) = µi < ∞.

� A state is null recurrent if

E(Ti) = µi < ∞.

� A state is null recurrent if E(Ti) =

E(Ti) = µi < ∞.

� A state is null recurrent if E(Ti) = µi = ∞

E(Ti) = µi < ∞.

• A state is transient if fii < 1.

In this case we define µi = ∞ for later convenience.

E(Ti) = µi < ∞.

• A peridoic state has non-zero pii(nk) for some k.

E(Ti) = µi < ∞.

• A peridoic state has non-zero pii(nk) for some k.

• A state is ergdoic if it is positive recurrent and aperiodic.

Classification of Markov chains - 6.3Classification of Markov chains - 6.3

• We can identify subclasses of states with the same

properties

• All states which can mutually reach each other will be of

the same type

properties

the same type

• Once again the formal analysis is a little bit heavy,

properties

the same type

• Once again the formal analysis is a little bit heavy, but try

to stick to the fundamentals,

properties

the same type

• Once again the formal analysis is a little bit heavy, but try

to stick to the fundamentals, definitions (concepts) and

results

Communicating statesCommunicating states

• we say that i communicates with j

• we say that i communicates with j if fij > 0

• If i communicates with j and j communicates with i

• If i communicates with j and j communicates with i they

intercommunicate, expressed as i ↔ j.

• The set of states of a Markov chain can be partitioned into

sets of intercommunicating states.

Properties of sets of intercommunicating statesProperties of sets of intercommunicating states

Theorem 1 (2) Page 204 If i ↔ j then

• (a) i and j has the same period

• (b) i is transient if and only if j is transient

• (c) i is null persistent (null recurrent) if and only if j is

null persistent

Definition 2 (3) page 205 A set C of states is called

• (a) Closed if pij = 0 for all i ∈ C, j /∈ C

• (b) Irreducible if i ↔ j for all i, j ∈ C.

Theorem 3

Theorem 3 Decomposition theorem (4) page 205. The

state space S can be partitioned uniquely as

S = T ∪ C1 ∪ C2 ∪ . . .

where T is the set of transient states, and the Ci are

irreducible closed sets of persistent states

S = T ∪ C1 ∪ C2 ∪ . . .

where T is the set of transient states, and the Ci are

irreducible closed sets of persistent states

Lemma 4 If S is finite, then at least one state is

persistent(recurrent) and all persistent states are non-null

(positive recurrent)

An example chain (random walk with

reflecting barriers)

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

With initial probability distribution ~µ(0) = (1, 0, 0, 0, 0, 0, 0, 0)

or X0 = 1.

Properties of that chainProperties of that chain

• We have a finite number of states

• From state 1 we can reach state j

• From state 1 we can reach state j with a probability

f1j ≥ 0.4j−1,

f1j ≥ 0.4j−1, j > 1.

• From state j we can reach state 1 with a probability

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1,

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1, j > 1.

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1, j > 1.

• Thus all states intercommunicate and the chain is

irreducible.

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1, j > 1.

irreducible. Generally we won’t bother with bounds for the

fij’s.

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1, j > 1.

fij’s.

• Since the chain is finite all states are positive recurrent

f1j ≥ 0.4j−1, j > 1.

fj1 ≥ 0.3j−1, j > 1.

fij’s.

• Since the chain is finite all states are positive recurrent

• A look on the behaviour of the chain

Bo Friis Nielsen – 3/10-2000 13IMM

A number of different sample paths Xn’s

0 10 20 30 40 50 60 701

The state probabilities µ(n)j

0 10 20 30 40 50 60 700

Limiting distributionLimiting distribution

Theorem 5 (17) page 214 For an irreducible aperiodic

chain, we have that

pij(n)

chain, we have that

pij(n) →1

as n →∞, for all i and j

chain, we have that

pij(n) →1

Three important remarks (also on page 214)

• If the chain is transient or null-persistent (null-recurrent)

chain, we have that

pij(n) →1

pij(n) → 0

chain, we have that

pij(n) →1

pij(n) → 0

• If the chain is positive recurrent pij(n) →

chain, we have that

pij(n) →1

pij(n) → 0

• If the chain is positive recurrent pij(n) → 1µj

chain, we have that

pij(n) →1

pij(n) → 0

• The limiting probability of Xn = j

chain, we have that

pij(n) →1

pij(n) → 0

• The limiting probability of Xn = j does not depend on the

starting state X0 = i

The stationary distributionThe stationary distribution

• A distribution that does not change with n

• The elements of ~µ(n) are all constant

• The implication of this is

• The implication of this is ~µ(n)

• The implication of this is ~µ(n) = ~µ(n−1)P

• The implication of this is ~µ(n) = ~µ(n−1)P = ~µ(n−1)

• The implication of this is ~µ(n) = ~µ(n−1)P = ~µ(n−1) by our

assumption of ~µ(n) being constant

• Expressed differently

• Expressed differently ~π =

• Expressed differently ~π = ~π

• Expressed differently ~π = ~πP

Stationary distributionStationary distribution

Definition 6 The vector ~π is called a stationary distribution

of the chain if ~π has entries (πj : j ∈ S) such that

• (a) πj ≥ 0 for all j, and∑

j πj = 1

• (b) ~π = ~πP, which is to say that πj =∑

i πipij for all j.

j πj = 1

i πipij for all j.

Theorem 7 (3) page 208 VERY IMPORTANT

j πj = 1

i πipij for all j.

An irreducible chain has a stationary distribution ~π

j πj = 1

i πipij for all j.

An irreducible chain has a stationary distribution ~π if and only

if all the states are non-null persistent

j πj = 1

i πipij for all j.

if all the states are non-null persistent (positive recurrent);

j πj = 1

i πipij for all j.

if all the states are non-null persistent (positive recurrent);in

this case, ~π is the unique stationary distribution

j πj = 1

i πipij for all j.

this case, ~π is the unique stationary distribution and is given

by πi = 1µi

for each i ∈ S,

j πj = 1

i πipij for all j.

this case, ~π is the unique stationary distribution and is given

by πi = 1µi

for each i ∈ S, where µi is the mean recurrence

time of i.

chain, we have that

pij(n) →1

pij(n) → 0

chain, we have that

pij(n) →1

pij(n) → 0

chain, we have that

pij(n) →1

pij(n) → 0

= πj.

chain, we have that

pij(n) →1

pij(n) → 0

= πj.

• The limiting probability of Xn = j does not depend on the

starting state X0 = i

The example chain (random walk with

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

Elementwise the matrix equation is πi =∑

j πjpji

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2·

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2·0.3+

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2·0.3+π3

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2·0.3+π3·0.3

0.6 0.4 0.0 0.0 0.0 0.0 0.0 0.0

0.3 0.3 0.4 0.0 0.0 0.0 0.0 0.0

0.0 0.3 0.3 0.4 0.0 0.0 0.0 0.0

0.0 0.0 0.3 0.3 0.4 0.0 0.0 0.0

0.0 0.0 0.0 0.3 0.3 0.4 0.0 0.0

0.0 0.0 0.0 0.0 0.3 0.3 0.4 0.0

0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.4

0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.7

~π = ~πP

j πjpji

π1 = π1·0.6+π2·0.3 π2 = π1·0.4+π2·0.3+π3·0.3 π3 = π2·0.4+π3·0.3+π4·0.3

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π8 = π7 · 0.4 + π8 · 0.7

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π8 = π7 · 0.4 + π8 · 0.7

π2 =1− 0.6

0.3π1

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π8 = π7 · 0.4 + π8 · 0.7

π2 =1− 0.6

0.3π1

πj+1 =1

0.3((1− 0.3)πj − 0.4πj−1)

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π8 = π7 · 0.4 + π8 · 0.7

π2 =1− 0.6

0.3π1

πj+1 =1

0.3((1− 0.3)πj − 0.4πj−1)

Can be solved recursively

π1 = π1 · 0.6 + π2 · 0.3

πj = πj−1 · 0.4 + πj · 0.3 + πj+1 · 0.3

π8 = π7 · 0.4 + π8 · 0.7

π2 =1− 0.6

0.3π1

πj+1 =1

0.3((1− 0.3)πj − 0.4πj−1)

Can be solved recursively to find:

πj =(

)j−1

The normalising conditionThe normalising condition

• We note that we don’t have to use the last equation

• We need a solution which is a probability distribution

πj = 1,8

)j−1

πj = 1,8

)j−1

π1 = π1

πj = 1,8

)j−1

π1 = π1

πj = 1,8

)j−1

π1 = π1

πj = 1,8

)j−1

π1 = π1

1−aN+1

1−aN < ∞, a 6= 1

πj = 1,8

)j−1

π1 = π1

1−aN+1

1−aN < ∞, a 6= 1

N + 1 N < ∞, a = 1

πj = 1,8

)j−1

π1 = π1

1−aN+1

1−aN < ∞, a 6= 1

N + 1 N < ∞, a = 1

11−a

N = ∞, |a| < 1

πj = 1,8

)j−1

π1 = π1

1−aN+1

1−aN < ∞, a 6= 1

N + 1 N < ∞, a = 1

11−a

N = ∞, |a| < 1

Such that

1 = π1

0.40.3

1− 0.40.3

πj = 1,8

)j−1

π1 = π1

1−aN+1

1−aN < ∞, a 6= 1

N + 1 N < ∞, a = 1

11−a

N = ∞, |a| < 1

Such that

1 = π1

0.40.3

1− 0.40.3

⇔ π1 =1− 0.4

0.40.3

Interpretation of πj’sInterpretation of πj’s

• Limiting probabilities

• Long term averages

• Stationary distribution

Reading recommendationsReading recommendations

• For Tuesday October 3, read 6.4

• For Friday October 6, read 6.4-6.5, exercise10, (solution

exercise 10?).

• For Tuesday October 10, read 6.8

• For Friday October 13, read 6.9

Stochastic Processes - lesson 8 · Stochastic Processes - lesson 8 Bo Friis Nielsen Institute of Mathematical Modelling Technical University of Denmark 2800 Kgs. Lyngby { Denmark

Documents

Stochastic Processes (Master degree in...

Introduction to Stochastic Processes

Discrete simulation of colored noise and stochastic...

Stochastic Processes - Sharif

Discrete Stochastic Processes, Chapter 2: Poisson Processes

Probability and Stochastic Processes

Matrix-valued Stochastic Processes- Eigenvalues Processes...

Random Processes Random or Stochastic processes

Stochastic Processes (Stochastik II) - Uni Ulm Aktuelles...

STOCHASTIC PROCESSES Basic notions - Aalto · J. Virtamo...

Interacting stochastic processes

Discrete Stochastic Processes

STOCHASTIC PROCESSES FOR PHYSICISTS Understanding Noisy...

Stochastic Processes - Aalto

Nonequilibrium stochastic Processes at Single-molecule...

Stochastic EMÜ439 Processes