1 Part III Markov Chains & Queueing Systems 10.Discrete-Time Markov Chains 11.Stationary Distributions & Limiting Probabilities 12.State Classification.

1

Part IIIMarkov Chains & Queueing Systems

10.Discrete-Time Markov Chains

11.Stationary Distributions & Limiting

Probabilities

12.State Classification

13.Limiting Theorems For Markov Chains

14.Continuous-Time Markov Chains

2

10. Discrete-Time Markov Chain Definition of Markov Processes A random process is said to be a Markov Process

if, for any set of n time points t1<t2<…<tn in the index set or time range of the process, the conditional distribution of X(tn), given the values of X(t1), X(t2), …, X(tn-1), depends only on the immediately preceding value; that is, for any real numbers x1, x2, …, xn,

})(|)(Pr{

})(,...,)(,)(|)(Pr{

11

112211

nnnn

nnnn

xtXxtX

xtXxtXxtXxtX

3

Classification of Markov Processes

Type of Parameter State Space Discrete Continuous Discrete-Time Continuous-TimeDiscrete Markov Chain Markov Chain

Continuous Discrete-Time Continuous-Time Markov Process Markov Process

4

Discrete-Time Markov Chain Definition ： A discrete-time Markov chain {Xn| n=0, 1, 2, …}, is

a discrete-time, discrete-value random sequence such that given X0, X1, …, Xn, the next random variable depends only on Xn through the transition probability

ij

nn

nnnn

p

iXjXP

iXiXiXjXP

]|[

],...,,|[

1

00111

5

Theorem 10.1 The transition probabilities pij of a Markov chain

satisfy

Pf：

0

.1 ,0j

ijij pp

6

Example 10.1 The two-state Markov chain can be used to model a

wide variety of systems that alternate between ON and OFF states. After each unit of time in the OFF state, the system turns ON with probability p. After each unit of time in the ON state, the system turns OFF with probability q. Using 0 and 1 to denote the OFF and ON states, what is the Markov chain for the system?

Sol：

7

Example 10.2 A packet voice communications is in talkspurts (state 1)

or silent (state 0) states. The system decides whether the speaker is talking or silent every 10 ms (slot time). If the speaker is silent in a slot, then the speaker will be talking in the next slot with probability p= 1/140. If the speaker is talking in a slot, the speaker will be silent in the next slot with probability q = 1/100. Sketch the Markov chain of this system.

Sol：

8

Example 10.3 A computer disk drive can be in one of 3

possible states: 0 (IDLE), 1 (READ), or 2 (WRITE) . When a unit of time is required to read or write a sector on the disk, sketch the Markov chain.

Sol：

9

Example 10.4 In a discrete random walk, a person’s position is

marked by an integer on the real line. Each unit of time, the person randomly moves one step, either to the right (with probability p) or to the left. Sketch the Markov chain.

Sol：

10

Example 10.5 What is the transition matrix of the two-state

ON-OFF Markov chain of Example 10.1 ? Sol：

11

n-Step Transition Probabilities Definition For a finite Markov chain, the n-step transition

probabilities are given by the matrix P(n) which has i, j-th element

.

)()()(

)()()(

)()()(

)(

],|[)(

10

11110

00100

npnpnp

npnpnp

npnpnp

n

iXjXPnp

KKKK

K

K

mmnij

P

12

Theorem 10.2 : Chapman-Kolmogorov Eq.

For a finite Markov chain with K states, the n-step transition probabilities satisfy

Pf：).()()( ,)()()(

0

mnmnmpnpmnpK

kkjikij PPP

13

Theorem 10.3 For a finite Markov chain with transition matrix

P, the n-step transition matrix is

Pf：.)( nn PP

14

Example 10.6 For the two-state Markov chain described in

Example 10.1, find the n-step transition matrix P(n). Given the system is OFF at time 0, what is the probability the system is OFF at time n = 33 ?

Pf：

15

State Probability Vector Definition A vector p=[p0 p1 … pK] is a state probability

vector if , and each element is nonnegative.

10

K

j jp

16

Theorem 10.4 The state probabilities pj(n) at time n can be defined by

either one iteration with the n-step transition probabilities:

or n iterations with the one-step transition probabilities:

Pf：

.)1()( ,)1()(0

K

iijij nnpnpnp Ppp

,)0()( ,)()0()(0

nK

iijij nnppnp Ppp

17

Example 10.7 For the two-state Markov chain described in

Example 10.1 with initial state probabilities p(0)=[p0 p1], find the state probability vector p(n).

Sol：

18




Probabilities




19

11. Stationary Distributions & Limiting

Probabilities Definition of Limiting State Probabilities For a finite Markov chain with initial state

probability vector p(0) , the limiting state probabilities, when they exist, are defined to be the vector

).(lim nn

pπ

20

Example 11.1For a two-state packet voice system of

Example 10.2, what is the limiting state probability

vector

Sol：

? )(lim] [ 10 nn

p

0 11/140

1/10099/100139/140

21

Theorem 11.1If a finite Markov chain with transition matrix

P and initial state probability p(0) has limiting probability vector , thenPf：

)(lim nn

pπ

.πPπ

22

Stationary Probability VectorDefinitionFor a finite Markov chain with transition

matrix P, a state probability vector is stationary if .πPπ

23

Theorem 11.2If a finite Markov chain Xn with transition

matrix P is initialized with stationary probability

vector p(0) = , then p(n) = for all n and the stochastic process Xn is stationary.

Pf：)|()|()(),,( 1|12|11,,

11211

mmXXXXXmXX xxpxxpxpxxpmnmnnnnmnn

)()( 112 1211 mmxxxxx nnpnnp

mm

)|())()(()(

,)(

1|11

1

111

11

jjXXjjxxjjxx

xX

xxpknknpnnp

xp

kjnkjnjjjj

kn

StationaryXxxpxxp nmXXmXX kmnknmnn ),,,(),,( 1,,1,,

11

24

Example 11.2A queueing system is described by a Markov chain in which that state Xn is the number of customers in the

queue at time n. The Markov chain has a unique stationary

distribution . The following questions are all equivalent. (1) What is the steady-state probability of at least 10

customers in the system?(2) If we inspect the queue in the distant future, what is

the probability of at least 10 customers in the system?

(3) What is the stationary probability of at least 10 customers in the system?

(4) What is the limiting probability of at least 10 customers in the system?

25

Example 11.3Consider the two-state Markov chain of Example

10.1 and Example 10.6. For what values of p and q does (a) exist, independent of the initial state probability vector p(0); (b) exist, but depend on p(0); (c) or not exist?

Sol：

)(lim nn

p

26




Probabilities




27

12. State Classification

Accessibility State j is accessible from state i, written i j,

if pij(n) > 0 for some n > 0.

Communicating States State i and j communicates, written i j, if i

j and j i.Communicating Class A communicating class is a nonempty subset of

states C such that if i C , then j C if and only if i j.

28

State ClassificationPeriodic and Aperiodic State i has period d if d is the largest integer such

that pii(n) = 0 whenever n is not divisible by d (pii(n) > 0 whenever n is divisible by d). If d = 1, then state i is called aperiodic.

Transient and Recurrent States In a finite Markov chain, a state i is transient if there

exists a state j such that i j but j i; otherwise, if no such state j exists, then state i is recurrent.

Irreducible Markov chain A Markov chain is irreducible if there is only one

communicating class.

29

Example 12.1The Markov chain is with each branch pij > 0.

How many communicating classes?Sol：

0

216

5

4

3

30

Example 12.2

Consider the five-state discrete random walk with Markov chain

What is the period of each state i ? Sol：

2 4310p

1p

p

1p

p

1

1

1p

31

Theorem 12.1

Communicating states have the same period. Pf：

jijijdid where, and state of periods the:)( ),(

. and somefor to from

path hop-an and to frompath hop-an is there,

nmji

mijnji

mnjdmnp jj divides )( ,0)(

. divides )( then ,0)(such that any For kidkpk ii

, divides )( ,0)( mknjdmknp jj kjd divides )( . have weAnd hops hops hops jiij mkn

)( )( idjd that showcan we, and of labels thereverse Similarly, ji

).()( jdid ).()( jdid

32

Example 12.3

The Markov chain is with each branch pij > 0.

Find the periodicity of each communicating class.

Sol：

0

216

5

4

3

33

Theorem 12.2

If i is recurrent and i j, then j is recurrent.Sol：

. show toneed we, that stateany for Assume jkkjk

,ji

ki

. via to frompath a exists there , and jkikjji

. recurrent, is iki

.jk

34

Example 12.4

The Markov chain is with each branch pij > 0.

Identify each communicating class and indicate

whether it is transient or recurrent.Sol：

2 4310 5

35

Theorem 12.3If state i is transient, then Ni, the number of

visits to state i over all time, has expected value

E[Ni] < .

Pf： ici

i

EE

iE

ofevent ary complement the:

state togoes eventually system theevent that the:

)(]|[)(]|[][ iiici

ciii EPENEEPENENE

.0]|[ ,0 then occurs,not is, that occurs, If ciiii

ci ENENEE

.occurs given , state toreturnsnever system the

event that theis where],[/1]|[ And

on.distributi geometric is ,given then ,occurs If

i

cii

ciiii

iii

Ei

EEPENE

NEE

)(]|[][ iiii EPENENE

36

Theorem 12.4A finite-state Markov chain always has a recurrent communicating class.Pf：

states, ofnumber the:

timeallover state to visitsofnumber the:

K

iN i

have we, any timeat then exist, states recurrent no Assume n

ion.contradict a isit infinite,approach can But n

,][ ,11

K

ki

K

ki NEnNn

classs. ingcommunicat

recurrent a has alwayschain Markov finite a Hence,

37

Example 12.5The Markov chain is with each branch pij > 0.

If the system starts in state j C1 = {0, 1, 2}, the system never leaves C1.

If the system starts in communicating class C3 = {4, 5}, the system never leaves C3.

If the system starts in the transient state 3, then in the first step there is a random transition to either state 2 or to state 4 and the system then remains forever in the corresponding communicating class.

2 4310 5

38




Probabilities




39

13. Limiting Theorems For Markov Chains

For Markov chains with multiple recurrent classes, the limiting state probabilities depend on the initial state distribution.

For understanding a system with multiple communicating classes, we need to examine each recurrent class separately as an irreducible system consisting of just that class.

In this part, we first focus on irreducible, aperiodic chains and their limiting state probabilities.

40

Theorem 13.1

,lim

10

10

10

K

K

K

n

n

πP 1

For an irreducible, aperiodic, finite Markov chain with states {0, 1, 2, …, K}, the limiting n-step transition matrix is

where is the column vector and is the unique vector satisfying

] [ 10 K π1 T1] 1[

.1 , 1ππPπ

41

Theorem 13.1 (cont’d)Pf：

42

Theorem 13.2

.)(lim πp

nn

For an irreducible, aperiodic, finite Markov chain with transition matrix P and initial state probability vector p(0) , Pf：

nn Pppp )0()( ,1)0( and1

πππp

πpPpp

1)0(

)0(lim )0()(lim

1

1n

nnn

43

Example 13.1For the packet voice communications system of Example 12.8, use Theorem 13.2 to calculate the stationary probabilities . Sol：

] [ 10 0 1

1/140

1/10099/100139/140

44

Example 13.2A digital mobile phone transmits one packet in every 20-ms time slot over a wireless connection. A packet is received error with probability p = 0.1, independent of other packets. Whenever 5 consecutive packets are received in error, the transmitter enters a timeout state. During the timeout state, the mobile terminal performs an independent Bernoulli trial with success probability q = 0.01 in every slot. When a success occurs, the mobile terminal starts transmitting in the next slot as though no packets had been in error. Construct a Markov chain for this system. What are the limiting state probabilities?

45

Example 13.2 (Cont’d)Sol：

2 4310p

1p

p

1p

p

1p

p

1p

p

5

q

1p1q

otherwise.0

,51

)1(

,4,3,2,1,01

)1)(1(

65

5

65

npqpq

pp

npqpq

ppq n

n

46

Theorem 13.3Consider an irreducible, aperiodic, finite Markov chain with transition probabilities {pij} and stationary probabilities {i}. For any partition of the state space into disjoint subsets S and S’,

Pf：

.

Sj Si

jijSi Sj

iji pp

otherwise.0

from occurs transition1)(: at time from stransition

SSmImSS SS

.]1)([)]([

Si Sj

ijiSSSS pmIPmIE

, time toup from crossings of no.:)( nSSnN SS

1

0

1

0

]1)([)]([n

m Si Sjiji

n

mSSSS pmIEnNE

Si Sjiji

n

mi

nSi Sj

ijSS

np

np

n

nNE 1

0

1lim

)]([lim

1)()(1)( nNnNnN SSSSSS

.

Sj Si

jijSi Sj

iji pp

47

Example 13.3 In each time slot, a router can either store an arriving data packet in its buffer or forward a stored packet (and remove that packet from its buffer). In each time slot, a new packet arrives with probability p, independent of arrivals in all other slots. This packets is stored as long as the router is storing fewer than c packets. If c packets are already buffered, then the new packet is discarded by the router. If no new packet arrives and n > 0 packets are buffered by the router, then the router will forward one buffered packet. That packet is then removed from the router. Let Xn denote the number of buffered packets at time n. Sketch the Markov chain for Xn and find the stationary probabilities.

48

Example 13.3 Sol：

i i+110p

1p

p

1p

p

1p…1p c

p

1p…

pS S’

.,,2,1,0 ,)()(1

)(111

1

1 ciip

pc

pp

pp

i

49

Periodic States and Multiple Communicating Classes

Theorem 13.4：For an irreducible, recurrent, periodic, finite Markov chain with transition matrix P, the stationary probability vector is the unique nonnegative solution of

Pf：

π

.1 , 1ππPπ

50

Example 13.4

Find the stationary probabilities for the Markov

chain shown to the right. Sol：

2101 1

1

51

Theorem 13.5 For a Markov chain with recurrent communicating classes C1, … Cm, let denote the limiting state probabilities associated with class k. Given that the system starts in a transient state i, the limiting probability of state j is

where P[Bik] is the conditional probability that the system enters class Ck.

Pf：

],[][)(lim )(1

)1(im

mjijij

nBPBPnp

)(kπ

].[]|[][]|[]|[ 110 imimniinn BPBjXPBPBjXPiXjXP

. if 0 where,]|[lim )()(k

kj

kjikn

nCjBjXP

].[][]|[lim)(lim )(1

)1(0 im

mjijn

nij

nBPBPiXjXPnp

52

Example 13.5

For each possible starting state i{0,1,…,4}, find

the limiting state probabilities for the following

Markov chain. Sol：

2103/4 1/2

1/4

431/2 1/2

1/4

1/4 3/4 1/2 3/4

3

1

6

1 0

8

3

8

1π

53

Countably Infinite Chains

Countably infinite Markov chains has infinite set of states {0, 1, 2, …}.

We will consider only a single communicating class here. (Irreducible Markov chains)

Multiple communicating classes represent distinct system modes that are coupled only through an initial transient phase that results in the system landing in one of the communicating classes.

54

Example 13.6Suppose that the router in Example 13.3 has unlimited buffer space. In each time slot, a router either store an arriving data packet in its buffer or forward a stored packet ( and remove that packet from its buffer). In each time slot, a new packet is stored with probability p, independent of arrivals in all other slots. If no new packet arrives, then one packet will be removed from

the buffer and forwarded. Sketch the Markov chain for Xn,

the number of buffered packets at time n. Sol：

210p

1p

p

1p

p

1p…

1p

55

Theorem 13.6 ： Chapman-Kolmogorov Eqs.

The n-step transition probabilities satisfy

Pf ： Omitted.

).()()(0

mpnpmnp kjk

ikij

56

Theorem 13.7The state probabilities pj(n) at time n can be found by either one iteration with the n-step transition probabilities

or n iterations with the one-step transition probabilities

Pf ： Omitted.

).()0()(0

nppnp iji

ij

.)1()(0

i

ijij pnpnp

57

Visitation, First Return Time, No. of Returns

Definitions ： Given that the system is in state i at

an arbitrary time,(a) Eii is the event that the system eventually

returns to visit state i. (b) Tii is the time (number of transitions) until

the system first returns to state i, (c) Nii is the number of times (in the future)

that the system returns to state i.

Definition ： For a countably infinite Markov chain, state i is recurrent if P[Eii]=1; otherwise state i is transient.

58

Example 13.7A system with states {0, 1, 2, …} has Markov chain

Note that for any state i > 0, pi,0 = 1/(i+1) and pi,i+1= i/(i+1).

Is state 0 transient or recurrent?

Sol：

2 43101/2

1/3

2/3

1/4

3/4

1/5

1

1/2

4/5…

01

1limlim

01,

01,

n

ppEPn

n

iii

nn

nncii

,1)(1 ciiii EPEP recurrent. is 0 state

59

Theorem 13.8The expected number of visits to state i over all time is

Pf：.)(][

1

n

iiii npNE

otherwise.0

, at time state toreturns system the1)( Define

ninI ii

,)(1

n

iiii nIN

.)(]1)([1][11

n

iin

iiii npnIPNE

60

Theorem 13.9State i is recurrent if and only if E[Nii]= .Pf：

. at time return tofirst ofy probabilit theis )( where

,)()()(1

rirf

rnprfnp

ii

n

riiiiii

)()()()()(11 11

r rn

iiiin

n

riiii

nii rnprfrnprfnp

1 11 0

)()0()( )()(r n

iiiiiir n

iiii npprfnprf

1 1

)(1)( r n

iiii nprf )(1

)()(

1

1

1

r ii

r ii

nii

rf

rfnp

.1)( recurrent, is state1

r ii rfi .)(][1

niiii npNE

61

Example 13.8The discrete random walk introduced in Example 10.4 has state space {…, 1, 0, 1, …} and Markov chain

Is state 0 recurrent?Sol：

0 2112p

1p

p

1p

p

1p

p

1p

p

1p

p

1p……

,)1()2(][1

2

10000

nn

n

nn

n

ppCnpNE

,2

!!

!2 22

nπnn

nC

nn

n ,)]1(4[

][ 00 n

ppNE

n

staterecurrent diverges 1

][ ,2

1 If

100

n nNEp

state.transient converges ][ ,2

1 If

100

n

n

nNEp

62

Positive Recurrence and Null Recurrence

Definition：

A recurrent state i is positive recurrent if E[Tii] < ; otherwise, state i is null recurrent.

Example 13.9 ： In Example 13.7, we found that state 0 is recurrent. Is state 0 positive recurrent or null recurrent?Sol：

2 43101/2

1/3

2/3

1/4

3/4

1/5

1

1/2

4/5…

)1(

1][]1[][ 000000

nnnTPnTPnTP

100

100 )1(

1][][

nn nnTPnTE

63

Theorem 13.10For a communicating class of a Markov chain,

one of the following must be true：(a)All states are transient.(b)All states are null recurrent.(c)All states are positive recurrent.

64

Example 13.10In a Markov chain of Example 13.7 and 13.9,

is state 33 positive recurrent, null recurrent, or transient?Sol：

65

Stationary Probabilities of Infinite Chains

Theorem 13.11：

For an irreducible, aperiodic, positive recurrent

Markov chain with states {0, 1, …}, the limiting n-step transition probabilities are where are the unique state probabilities satisfying

jijn

np

)(lim

},2,1,0|{ jj

,2,1,0 , ,100

jpi

ijijj

j

66

Example 13.11For the stationary probabilities of the router buffer described in Example 13.6. Make sure

to identify for what values of p that the

stationary probabilities exist. Sol：

210p

1p

p

1p

p

1p…

1p

67




Probabilities




68

14. Continuous-Time Markov ChainsDefinition：A continuous-time Markov chain {X(t)| t 0} is a continuous-time, discrete-value random process such that for an infinitesimal time step of size ,

Continuous-time Markov chain is closely related to the Poisson process. The time until the next transition is an exponential R.V. with parameter

ijij

ij

qitXitXP

qitXjtXP

1])(|)([

])(|)([

. , iqij

iji state of rate departure the

69

Embedded Discrete-Time Markov ChainDefinition：For a continuous-time Markov chain with transition rates qij and state i departure rates

vi, the embedded discrete-time Markov chain

has transition probabilities pij=qij /i for states i with i > 0 and pii = 1 for states i with i = 0.

Definition：The Communicating Classes of a Continuous-

Time Markov Chain are given by the

communicating classes of its embedded discrete-time Markov chain.

70

Irreducible Continuous-Time Markov ChainDefinition：A continuous-time Markov chain is irreducible

if the embedded discrete-time Markov chain is irreducible. Definition：An irreducible continuous-time Markov chain

is positive recurrent if for all states i, the time Tii to return to state i satisfying E[Tii] < .

71

Example 14.1In a continuous-time ON-OFF process,

alternating OFF and ON (state 0 and 1) periods have independent exponential durations. The

average ON period lasts 1/ seconds, while the average OFF period lasts 1/ seconds. Sketch the continuous-time Markov chain. Sol： 10

72

Example 14.2Air conditioner is in one of 3 states ： OFF(0),

Low (1), or High(2). The transitions from OFF to Low occur after an exponential with mean time 3

mins. The transitions from Low to OFF or High are equally likely and transitions out of the Low

state occur at rate 0.5 per min. When the system is

in High state, it makes a transition to Low with probability 2/3 or to the OFF state with probability 1/3. The time spent in the high

state is an exponential (1/2) R.V. Model this air conditioning system using a continuous-time Markov chain.

73

Example 14.2 (Cont’d)Sol：

2101/4

1/3

1/3

1/4

1/6

74

Theorem 14.1For a continuous-time Markov chain, the state probabilities pj(t) evolve according to the differential equations

where

Pf：

.)()(

, ,,2,1,0 ,)()(

Rpp

tdt

tdjtpr

dt

tdp

iiij

j form vector in or

. if

if

ji

jiqr

i

ijij

i

j itXPitXjtXPjtXPtp ])([])(|)([])([)(

ji

itXPitXjtXPjtXPjtXjtXP ])([])(|)([])([])(|)([

ji

iijjj tpqtp )()()1(

jiiijjj

jj tpqtptptp

)()()()(

i

iij tpr )(

75

Theorem 14.2For an irreducible, positive recurrent

continuous-time Markov chain, the state probabilities satisfying

where the limiting state probabilities are the unique solution to

Pf：

,)(lim , ,)(lim pp

tptpt

jjt

form vector in or

.1 , ,1

, , ,0

1

0

p

pR

form vector in or

form vector in or

jj

iiij

p

pr

,)(lim)(

lim

ii

tij

j

ttpr

dt

tdp 0

iiij pr ij

j

qij

Flow Conservation Law ： ave. rate in = ave. rate out

76

Example 14.3Calculate the limiting state probabilities for the ON-OFF system of Example 14.1. Sol： 10

77

Example 14.4Find the stationary distribution for the Markov chain describing the air conditioning system of Example 14.2. Sol： 210

1/4

1/3

1/3

1/4

1/6

.5

1 ,

5

2 : 210 pppAns

78

Birth-Death ProcessDefinition:A continuous-time Markov chain is a birth-death process if the transition rates satisfy qij = 0 for |ij| > 1.

2101

2

2

3

0

1

34

…3

79

Theorem 14.3For a birth-death queue with arrival rate i and service rate i , the stationary probabilities pi satisfy

Pf：

.1 ,0

11

iiiiii ppp

80

Theorem 14.4For a birth-death queue with arrival rate i and service rate i , let i =i /i+1 . The limiting state probabilities, if they exist, are

Pf：

.1

1

1

0

1

0

k

k

j j

i

j j

ip

81

The M/M/1 QueueThe arrivals are a Poisson process of rate , independent of the service requirements of the customers. The service time of a customer is an

exponential () R.V., independent of the system state.

M/M/1

Markovian arrival(Poisson arrivals)

Markovian service time(Exponential R.V.)

One server

82

Theorem 14.5The M/M/1 queue with arrival rate > 0 and service rate , > , has limiting state probabilities,

where = / . Pf：

,2,1,0 ,)1( np nn

83

Example 14.5Cars arrive at an isolated toll booth as a

Poisson process with arrival rate = 0.6 cars per minute. The service required by a customer is an exponential R.V. with expected value 1/ = 0.3 minutes. What are the limiting state

probabilities for N, the number of cars at the toll booth?

What is the probability that the toll booth has zero

cars some time in the distant future? Sol：

1 Part III Markov Chains & Queueing Systems 10.Discrete-Time Markov Chains 11.Stationary Distributions & Limiting Probabilities 12.State Classification.

Documents

states state i

aperiodic state i

state j c

finitestate markov chain

transient state

irreducible markov

markov process

finite markov chain