MARKOV CHAIN chap4

Markov Chain

Stochastic process (discrete time):{X1, X2, ..., }

Markov chain Consider a discrete time stochastic process with

discrete space. Xn {0, 1, 2, ...}. Markovian property

P{Xn+1 = j | Xn = i,Xn1 = in1, ..., X0 = i0}= P{Xn+1 = j | Xn = i} = Pi,j

Pi,j is the transition probability: the probability ofmaking a transition from i to j.

Transition probability matrix

P =

P0,0 P0,1 P0,2 P1,0 P1,1 P1,2

.

.

.

.

.

.

.

.

.

.

.

.

Pi,0 Pi,1 Pi,2 ...

.

.

.

.

.

.

.

.

.

1

Example

Suppose whether it will rain tomorrow depends onpast weather condition ony through whether it rainstoday. Consider the stochastic process {Xn, n = 1, 2, ...}

Xn =

{0 rain on day n1 not rain on day n

P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1 | Xn) State space {0, 1}. Transition matrix:(

P0,0 P0,1P1,0 P1,1

)

P0,0 = P (tomorrow rain|today rain) = . Then P0,1 =1 .

P1,0 = P (tomorrow rain|today not rain) = . ThenP1,1 = 1 .

Transition matrix: ( 1 1

)

2

Transforming into a Markov Chain

Suppose whether it will rain tomorrow depends onwhether it rained today and yesterday.

P (Xn+1|Xn, Xn1, ..., X1) = P (Xn+1|Xn, Xn1). Theprocess is not a first order Markov chain.

Define Yn:

Yn =

0 Xn = 0, Xn1 = 0 RR1 Xn = 0, Xn1 = 1 NR2 Xn = 1, Xn1 = 0 RN3 Xn = 1, Xn1 = 1 NN

P (Yn+1|Yn, Yn1, ...)

= P (Xn+1, Xn|Xn, Xn1, ...)= P (Xn+1, Xn|Xn, Xn1)= P (Yn+1|Yn)

{Yn, n = 1, 2, ...} is a Markov chain.

P0,0 P0,1 P0,2 P0,3P1,0 P1,1 P1,2 P1,3P2,0 P2,1 P2,2 P2,3P3,0 P3,1 P3,2 P3,3

3

P0,1 = P (Yn+1 = 1|Yn = 0) = P (Xn+1 = 0, Xn =1|Xn = 0, Xn1 = 0) = 0.

P0,3 = P (Yn+1 = 3|Yn = 0) = P (Xn+1 = 1, Xn =1|Xn = 0, Xn1 = 0) = 0.

Similarly, P1,1 = P1,3 = 0, P2,0 = P2,2 = 0, P3,0 =P3,2 = 0.

Transition matrix

P0,0 0 P0,2 0P1,0 0 P1,2 00 P2,1 0 P2,30 P3,1 0 P3,3

=

P0,0 0 1 P0,0 0P1,0 0 1 P1,0 00 P2,1 0 1 P2,10 P3,1 0 1 P3,1

The Markov chain is specified by P0,0, P1,0, P2,1, P3,1.1. P0,0 = P (tomorrow will rain|today rain, yesterday rain).2. P1,0 = P (tomorrow will rain|today rain, yesterday not rain).3. P2,1 = P (tomorrow will rain|today not rain, yesterday rain).4. P3,1 = P (tomorrow will rain|today not rain, yesterday not rain).

4

Chapman-Kolmogorov Equations

Transition after nth steps:P ni,j = P (Xn+m = j | Xm = i).

Chapman-Kolmogorov Equations:

P n+mi,j =k=0

P ni,kPmk,j, n,m 0 for all i, j.

Proof (by Total probability formula):P n+mi,j = P (Xn+m = j|X0 = i)

=k=0

P (Xn+m = j,Xn = k|X0 = i)

=k=0

P (Xn = k|X0 = i)

P (Xn+m = j|Xn = k,X0 = i)=

k=0

P ni,kPmk,j

5

n-step transition matrix:

P(n) =

P n0,0 Pn0,1 P

n0,2

P n1,0 Pn1,1 P

n1,2

.

.

.

.

.

.

.

.

.

.

.

.

P ni,0 Pni,1 P

ni,2

.

.

.

.

.

.

.

.

.

.

.

.

Chapman-Kolmogorov Equations:P

(n+m) = P(n) P(m), P(n) = Pn. Weather example:

P =

( 1 1

)=

(0.7 0.30.4 0.6

)Find P (rain on Tuesday | rain on Sunday) andP (rain on Tuesday and rain on Wednesday | rain on Sunday).Solution:

P (rain on Tuesday | rain on Sunday) = P 20,0P

(2) = P P =(

0.7 0.30.4 0.6

)(

0.7 0.30.4 0.6

)

=

(0.61 0.390.52 0.48

)P (rain on Tuesday | rain on Sunday) = 0.61

6

P (rain on Tuesday and rain on Wednesday | rain on Sunday)= P (Xn = 0, Xn+1 = 0 | Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0, Xn2 = 0)= P (Xn = 0|Xn2 = 0)P (Xn+1 = 0|Xn = 0)= P 20,0P0,0= 0.61 0.7 = 0.427

7

Classification of States

Accessible: State j is accessible from state i if P ni,j >0 for some n 0. i j. Equivalent to: P (ever enter j|start in i) > 0.

P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i)

n=0

P (Xn = j | X0 = i)

=n=0

P ni,j

Hence if P ni,j = 0 for all n, P (ever enter j|start in i) =0. On the other hand,

P (ever enter j|start in i)= P (n=0{Xn = j}|X0 = i) P ({Xn = j}|X0 = i) for any n= P ni,j .

If P ni,j > 0 for some n, P (ever enter j|start in i) P ni,j > 0.

Examples

8

Communicate: State i and j communicate if they areaccessible from each other. i j. Properties:

1. P 0i,i = P (X0 = i|X0 = i) = 1. Any state icommunicates with itself.

2. If i j, then j i.3. If i j and j k, then i k.

Proof:

i j = P ni,j > 0 and P n

j,i > 0

j k = Pmj,k > 0 and Pm

k,j > 0

P n+mi,k =l=0

P ni,lPml,k Chapman-Kolmogorov Eq.

> P ni,j Pmj,k> 0

Similarly, we can show P n+mk,i > 0. Hence ik.

9

Class: Two states that communciate are said to bein the same class. A class is a subset of states thatcommunicate with each other. Different classes do NOT overlap. Classes form a partition of states.

Irreducible: A Markov chain is irreducible if there isonly one class. Consider the Markov chain with transition proba-

bility matrix:

P =

12 12 01

214

14

0 1323

The MC is irreducible. MC with transition probability matrix:

P =

12

12

0 012

12 0 0

14

14

14

14

0 0 0 1

Three classes: {0, 1}, {2}, {3}.

10

Recurrent and Transient States

fi: probability that starting in state i, the MC will everreenter state i.

Recurrent: If fi = 1, state i is recurrent. A recurrent states will be visited infinitely many

times by the process starting from i. Transient: If fi < 1, state i is transient.

Starting from i, the MC will be in state i for ex-actly n times (including the starting state) is

fn1i (1 fi) , n = 1, 2, ...This is a geometric distribution with parameter 1fi. The expected number of times spent in state iis 1/(1 fi).

11

A state is recurrent if and only if the expected numberof time periods that the process is in state i, startingfrom state i, is infinite.

Recurrent E(number of visits to i|X0 = i) = Transient E(number of visits to i|X0 = i)

Consider a MC with

P =

0 0 1212

1 0 0 00 1 0 00 1 0 0

The MC is irreducible and finite state, hence all statesare recurrent.

Consider a MC with

P =

12

12 0 0 0

12

12

0 0 00 0 12

12 0

0 0 12

12

014

14 0 0

12

Three classes: {0, 1}, {2, 3}, {4}. State 0, 1, 2, 3 arerecurrent and state 4 is transient.

14

Random Walk

A Markov chain with state space i = 0,1,2, .... Transition probability: Pi,i+1 = p = 1 Pi,i1.

At every step, move either 1 step forward or 1 stepbackward.

Example: a gambler either wins a dollar or loses adollar at every game. Xn is the number of dollars hehas when starting the nth game.

For any i < j, P jii,j = pji > 0, P jij,i = (1 p)ji >0. The MC is irreducible.

Hence, either all the states are transient or all thestates are recurrent.

15

Under which condition are the states transient or re-current? Consider State 0.

n=1

P n0,0 =

{ recurrentfinite transient

Only for even m, Pm0,0 > 0.

P 2n0,0 =

(2nn

)pn(1 p)n = (2n)!

n!n!(p(1 p))n

n = 1, 2, 3, ...

By Stirlings approximation

n! nn+1/2en

2pi

P 2n0,0 (4p(1p))n

pin

.

When p = 1/2, 4p(1 p) = 1.n=0

(4p(1 p))npin

=n=0

1pin

,

The summation diverges. Hence, all the states arerecurrent.

When p 6= 1/2, 4p(1 p) < 1. n=0 (4p(1p))npinconverges. All the states are transient.

16

Limiting Probabilities

Weather example

P =

(0.7 0.30.4 0.6

)

P(4) = P4 =

(0.5749 0.42510.5668 0.4332

)

P(8) = P(4)P(4) =

(0.572 0.4280.570 0.430

)

P(4) and P(8) are close. The rows in P(8) are close. Limiting probabilities?

17

Period d: For state i, if P ni,i = 0 whenever n is notdivisible by d and d is the largest integer with thisproperty, d is the period of state i. Period d is the greatest common divisor of all them such that Pmi,i > 0.

Aperiodic: State i is aperiodic if its period is 1. Positive recurrent: If a state i is recurrent and the ex-

pected time until the process returns to state i is finite.

If i j and i is positive recurrent, then j is posi-tive recurrent.

For a finite-state MC, a recurrent state is also pos-itive recurrent.

A finite-state irreducible MC contains all positiverecurrent states.

Ergodic: A positive recurrent and aperiodic state isan ergodic state.

A Markov chain is ergodic if all its states are ergodic.

18

Theorem 4.1: For an irreducible ergodic Markov chain,limnP ni,j exists and is independent of i. Let pij =limnP ni,j, j 0, then pij is the unique nonnegativesolution of

pij =i=0

piiPi,j j 0j=0

pij = 1 .

The Weather Example:

P =

( 1 1

)The MC is irreducible and ergodic.

pi0 + pi1 = 1

pi0 = pi0P0,0 + pi1P1,0 = pi0 + pi1

pi1 = pi0P0,1 + pi1P1,1 = pi0(1 ) + pi1(1 )Solve the linear equations, pi0 = 1+, pi1 =

11+.

19

Gamblers Ruin Problem

At each play, a gambler either wins a unit with prob-ability p or loses a unite with probability q = 1 p.Suppose the gambler starts with i units, what is theprobability that the gamblers fortune will reach Nbefore reaching 0 (broke)?

Solution: Let Xt be the number of units the gambler has at

time t. {Xt; t = 0, 1, 2, ...} is a Markov chain. Transition probabilities:

P0,0 = PN,N = 1

Pi,i+1 = p, Pi,i1 = 1 p = q . Denote the probability that starting from i units,

the gamblers fortune will reach N before reaching0 by Pi, i = 0, 1, ..., N .

Condition on the result of the first game and applythe total probability formula:

Pi = pPi+1 + qPi1 .

20

Changing forms:(p + q)Pi = pPi+1 + qPi1

q(Pi Pi1) = p(Pi+1 Pi)Pi+1 Pi = q

p(Pi Pi1)

Recursion:P0 = 0

P2 P1 = qp(P1 P0) = q

pP1

P3 P2 = qp(P2 P1) =

(q

p

)2P1

.

.

.

Pi Pi1 = qp(Pi1 Pi2) =

(q

p

)i1P1

Add up the equations:Pi = P1[1 +

q

p+ + (q

p)i1]

=

P1

1(qp)i1qp

if qp 6= 1iP1 if qp = 1

We know PN = 1

PN =

P1

1(qp)N1qp

if qp 6= 1NP1 if qp = 1

21

Hence,

P1 =

{1qp

1(qp)Nif p 6= 1/2

1/N if p = 1/2

In summary,

Pi =

1(qp)i1(qp)N

if p 6= 1/2i/N if p = 1/2

Note, if N ,

Pi =

{1 (qp)i if p > 1/20 if p 1/2

When p > 1/2, there is a positive probability that thegambler will win infinitely many units.

When p 1/2, the gambler will surely go broke(with probability 1) if not stop at a finite fortune (as-suming the opponent is infinitely rich).

22