Connections Between Birth-Death Processes - Inria

* Department of Mathematics, California State Polytechnic University, Pomona, CA 91768, USA # INRIA, IRISA, Campus Universitaire de Beaulieu, 35042 RENNES Cedex, France

(in Recent Advances in Stochastic Processes and Functional Analysis, Marcel Dekker, 2003)

Connections Between Birth-Death Processes

Alan Krinik*, Carrie Mortensen*, Gerardo Rubino#

For M. M. Rao

Mathematics Subject Classifications: 60K25, 60J27. Keywords: Birth-Death process; Birth-Death chain; Dual process; Transient probability

functions; Busy period distribution; Markov chain, n-step transitional probability, 1. Introduction Birth-death chains and processes, with a finite or countable number of states, play a central role in stochastic modeling for applied probabilists. The following books have complete chapters devoted entirely to different aspects of birth-death models: Andersen (1991), Bhattacharya and Waymire (1990), Gross and Harris (1985), Karlin and Taylor (1975), Kijima (1997) and Medhi (2003). Meanwhile, classical probability texts, over many generations, have popularized Markov chains and captivated students with colorful discussions of birth-death chains in terms of gambler’s ruin probabilities, see, for example, Feller (1950) or Hoel, Port and Stone (1972).

The object of this present article is to analyze some interesting connections between birth-death processes. Some of these connections have been found while doing research on the use of dual processes to determine transient probability functions of various Markov processes, cf. Green et al.(2003), Krinik et al.(2003a) and Krinik et al.(2003b). The important relationship between the transient probability functions of an original birth-death process (chain) and its dual birth-death process (chain) is developed in section 2. In section 3, the problem of determining busy period distributions in a queueing system modeled by a birth-death process is shown to be equivalent to finding the transient probability functions in a related birth-death process. This equivalence of problems is then independently verified for the classical single server queueing system. Next, the two traditional problems of determining ruin probabilities and steady state distributions on a finite birth-death chain are shown to be connected via results on dual chains. The equivalence of these two problems and an illustration for the classical gambler’s ruin problem are presented in section 4. Finally, a sweet characterization of the n-step transition probability within a finite state, diagonalizable Markov chain is derived in section 5. A novel part of this approach is utilizing the well-known Cayley-Hamilton Theorem from linear algebra to determine recurrence relations, see Mortensen (2003) and Krinik et al. (2003c) for more details. As an application of this characterization, a new solution method is


2

0

1µ

0λ

2

3µ

2λ

1

2µ

1λ

3

4µ

3λ

4

5µ

4λ

5

6µ

… 5λ

sketched out for determining the transient probability functions of an arbitrary birth-death process on a finite state space. 2. Dual Processes

Consider a general recurrent birth-death process having transition birth rates λi, for i = 0,1,2, … and transition death rates µi for i = 1,2,3,…as shown in the state rate transition diagram, Figure 1. All of these rates are assumed to be positive numbers. The state space may be finite or countable.

Assume throughout this article that the transition rates are uniformly bounded. The transient probability functions, )(, tP ji , where i, j = 0,1,2,3,…may be found by solving the Kolmogorov backward or forward system of differential equations, see Anderson (1991), Bhattacharya and Waymire (1990) or Gross and Harris (1985). This system of differential equations may be written in matrix form as

QtPtPQtP ⋅=⋅=′ )()()( where

= .

........

....)()()(

....

....

.......)(...)()(...)(...)()(

)(

,1,0,

,11,10,1

,01,00,0

tPtPtP

tPtPtPtPtPtP

tP

nnnn

n

n

is the matrix of transition probability functions and

Figure 1


3

0λ

1

2λ

0

1λ

1µ 2

3λ

3µ 3

4λ

4µ 4

5λ

5µ …

2µ

-1

+−+−

+−−

=

........

........

...)(00

...0)(0

...00)(

...000

3333

2222

1111

00

λµλµλµλµ

λµλµλλ

Q

is the transition rate matrix. For birth-death processes having rates λ i, i=0,1,2,…and µ i i=1,2,3,… uniformly bounded, the solution of the Kolmogorov backward equation may be written, see Bhattacharya and Waymire (1990), as

QtetP =)(

The dual process of the birth-death process of Figure 1 has a state rate transition diagram as shown in Figure 2.

Figure 2

The transient probability functions of this dual process are denoted by )(*, tP ji . The following

relationship holds between a general birth-death system and its dual birth-death process. Theorem 2.1 If )(, tP ji and )(*

, tP ji are the transient probability functions of the birth-death processes corresponding to Figures 1 and 2 respectively, then

[ ]∑∞

=−−=

ikkjkjji tPtPtP )()()( *

,1*,, and [ ]∑

=+−=

i

kkjkjji tPtPtP

0,1,

*, )()()(

for all states i, j = 0,1,2,3… with the convention )(*

,1 tP k− = 0 if k > -1.

The proof of this result appears as Proposition 2.3 on page 269 of Anderson (1991) and depends upon the forward and backward equations and some algebraic simplifications. Consequently, if the transient probability functions in either the original birth-death process or dual birth-death system are known, then the transient probability functions in the other system are known as well.

An analog of Theorem 2.1 holds for birth-death chains. In Theorem 2.2, it is stated on a finite state space although it holds as well for infinite birth-death chains. Consider the finite recurrent birth-death chain having transition probabilities diagramed in Figure 3.


4

1

-1

1

2p

2q

0 …

1ρ

0p

3p

3q

2

1q 0ρ 2ρ

1−Hq

Hq

H H-1

1

1−Hρ

1−Hp

1p

2−Hp 0r

2r

3r

1r

0

1q

2

3q

2p

1 3

4q

3p 1p0p

2q

Hq

1−Hp

H H-1

Hr

1−Hr

… 1−Hq

Figure 3 In addition to the usual assumptions that all letters in this diagram represent fractions between 0 and 1 (inclusively) and that

100 =+ pr 1111 =++ prq 1222 =++ prq

… 1111 =++ −−− HHH prq

1=+ HH rq .

We also assume

110 ≤+ qp 121 ≤+ qp 132 ≤+ qp

… 11 ≤+− HH qp .

This implies that the s'ρ in the following dual birth-death diagram are nonnegative fractions less than 1. Figure 4 where 1100 =++ qp ρ


5

1211 =++ qp ρ 1322 =++ qp ρ

… 111 =++ −− HHH qp ρ .

This absorbing birth-death chain shown in Figure 4 is the dual birth-death chain of the birth-death chain pictured in Figure 3. Theorem 2.2 If )(

,njiP and )*(

,n

jiP are the n-step transient probabilities of the birth-death chains

corresponding to Figures 3 and 4 respectively, then

[ ]∑=

−−=H

rk

nks

nks

nsr PPP )(*

,1)(*

,)(

, and [ ]∑=

+−=r

k

nks

nks

nsr PPP

0

)(,1

)(,

)(*,

for 0≥n and for all states r , s = 0,1,2,3,…,H with the convention )(

,1nkP− = 0 if k > -1.

Proof. Use mathematical induction. For n = 0, [ ]∑=

−−=H

rkkskssr PPP )0(*

,1)0(*

,)0(

, is seen to hold by

substituting initial conditions that )0*(,,

)0(, srsrsr PP == δ for all states r , s = 0,1,2,3,…H where

sr ,δ is the Kronecker delta and using the convention )(,1nkP− = 0 if k > -1. Suppose 12 −≤≤ Hs .

As the induction hypothesis, assume [ ]∑=

−−=H

rk

nks

nks

nsr PPP )(*

,1)(*

,)(

, holds and

show [ ]∑=

+−

++ −=H

rk

nks

nks

nsr PPP )1(*

,1)1(*

,)1(

, . But

∑=

+ =H

jsj

njr

nsr PPP

0

)1(,

)(,

)1(,

by the Chapman-Kolmogorov equations for the birth-death chain of Figure 3. This simplifies to

)1(

,+n

srP = )1(,1

)(1,

)1(,

)(,

)1(,1

)(1, ss

nsrss

nsrss

nsr PPPPPP ++−− ++

since the chain of Figure 3 is a birth-death chain and therefore all 1-step transition probabilities are zero except for possibly the above noted three transitions. More specifically

)1(,

+nsrP = 1

)(1,

)(,1

)(1, ++−− ++ s

nsrs

nsrs

nsr qPrPpP

again, see Figure 3. However


6

)1(,

+nsrP = [ ] [ ] [ ] 1

)(*,

)(*,1

)(*,1

)(*,1

)(*,2

)(*,1 +

=+

=−−

=−− ∑∑∑ −+−+− s

H

rk

nks

nkss

H

rk

nks

nkss

H

rk

nks

nks qPPrPPpPP

by the induction hypothesis. Replacing sr by [ ]ss qp −−1 and rearranging terms produces

)1(,

+nsrP = [ ] [ ][ ] [ ] 1

)(*,

)(*,1

)(*,1

)(*,1

)(*,2

)(*,1 1 +

=+

=−−

=−− ∑∑∑ −+−−−+− s

H

rk

nks

nksss

H

rk

nks

nkss

H

rk

nks

nks qPPqpPPpPP

= [ ]( )∑=

+++− +−−+H

rk

nkss

nksss

nkss PqPqpPp )(*

,11)(*

,1)(*

,1 1 [ ]( )∑=

−−−− +−−+−H

rk

nkss

nksss

nkss PqPqpPp )(*

,)(*

,11)(*,21 1

By the definition of sρ from Figure 4

)1(,

+nsrP = ( )∑

=++− ++

H

rk

nkss

nkss

nkss PqPPp )(*

,11)(*

,)(*

,1 ρ ( )∑=

−−−− ++−H

rk

nkss

nkss

nkss PqPPp )(*

,)(*

,11)(*,21 ρ

Substituting in the transition probabilities of the chain of Figure 4 and writing this last result in Chapman-Kolmogorov equation form gives

)1(,

+nsrP = ( )∑

=++−− ++

H

rk

nksss

nksss

nksss PPPPPP )(*

,1)1(*1,

)(*,

)1(*,

)(*,1

)1(*1, ( )∑

=−−−−−−− ++−

H

rk

nksss

nksss

nksss PPPPPP )(*

,)1(*,1

)(*,1

)1(*1,1

)(*,2

)1(*2,1

= ∑∑∑∑= =

−= =

−H

rk

H

j

nkjjs

H

rk

H

j

nkjjs PPPP

0

)*(,

)1*(,1

0

)*(,

)1*(,

Therefore

)1(,

+nsrP [ ]∑

=

+−

+ −=H

rk

nks

nks PP )1*(

,1)1*(

,

by the Chapman-Kolmogorov equations of the birth-death chain of Figure 4. This completes the induction step and establishes the first equality in Theorem 2.2 whenever 12 −≤≤ Hs . If s = 0,1 or H, the preceding argument may be suitable modified to establish the desired result.

We next show that

[ ]∑=

+−=r

k

nks

nks

nsr PPP

0

)(,1

)(,

)(*, .

Consider [ ]∑=

+−r

k

nks

nks PP

0

)(,1

)(, . By what we have just proved we know


7

0

1µ

2

3µ

2λ

1

2µ

1λ

3

4µ

3λ

4

5µ

4λ

5

6µ

… 5λ

[ ]∑=

−−=H

s

nk

nk

nks PPP

ααα)(*,1

)(*,

)(, and [ ]∑

+=−+ −=

H

s

nk

nk

nks PPP

1

)(*,1

)(*,

)(,1

ααα .

So by substitution,

[ ]∑=

+−r

k

nks

nks PP

0

)(,1

)(, = [ ] [ ]∑ ∑∑

= +=−

=−

−−−

r

k

H

s

nk

nk

H

s

nk

nk PPPP

0 1

)*(,1

)*(,

)*(,1

)*(,

ααα

ααα

However, by canceling terms, this last equation simplifies to the following telescoping series

[ ]∑=

+−r

k

nks

nks PP

0

)(,1

)(, = [ ]∑

=−−

r

k

nsk

nsk PP

0

)*(,1

)*(,

which in turn reduces to )*(

,1)*(

,ns

nsr PP −− which equals )*(

,n

srP since )*(,1nsP− = 0 for Hs ≤≤0 because

state -1 is an absorbing state in the birth-death chain of Figure 4. Thus, we have shown that

[ ]∑=

+−r

k

nks

nks PP

0

)(,1

)(, = )*(

,n

srP

which is the second equality of Theorem 2.2. This completes the proof of Theorem 2.2. 3. Relating Busy Period Distributions and Transient Probability Functions

For a given queueing system, the problems of determining transient probability functions and busy period distributions are usually regarded as two different, but related, questions. In this section, it is shown that these two problems for queueing systems modeled by birth-death processes are more closely linked than previously realized. An elegant connection between these problems is described in terms of dual birth-death processes.

Call the busy period distribution of the birth-death process described in Figure 1, )(0, tP bi .

Note that )(0, tP bi satisfies the system of forward or backward equations associated with Figure

5 below, see Gross and Harris (1985).

Now consider the birth-death process having transition rates as shown below

Figure 5


8

-1

1µ

1

3µ

2λ

0

2µ

1λ

2

4µ

3λ

3

5µ

4λ

4

6µ

… 5λ

0

1λ

1µ

2

3λ

3µ

1

2λ

2µ

3

4λ

4µ

4

5λ

5µ

5

6λ

… 6µ

Figure 6

and let )(~, tP ji represent the transient probability functions of the birth-death process of Figure 6.

Then by section 2, the dual of the birth-death process pictured in Figure 6 is and has transient probabilities )(

~*, tP ji Note this diagram is almost the same as the busy period

distribution transitions diagramed in Figure 5, however, the states are offset by 1. Thus

)()(~

0,*

1,1 tPtP bii =−− for i = 1,2,3,…

Proposition 3.1 The busy period distribution )(0, tP b

i of Figure 5 is related to the transient

probability functions of Figure 6 according to

∑−

=

−=1

0,00, )(~1)(

i

kk

bi tPtP for i =1,2,3,… (1)

Proof. By Theorem 2.1

[ ]∑=

+−=i

kkjkjji tPtPtP

0,1,

*, )(~)(~)(~

for i, j = 0,1,2,… . Sum over j = 0,1,2,…and subtract from 1 to get

−=− 1)(~*1, tPi ∑

=

i

kk tP

0,0 )(~

Replace i by i-1 and substitute )()(

~0,

*1,1 tPtP b

ii =−− to obtain the desired result

Figure 7


9

∑−

=

−=1

0,00, )(~1)(

i

kk

bi tPtP for i = 1,2,3,…

This completes the proof. Note a special case of interest occurs when i = 1, which gives

)(~1)( 0,00,1 tPtPb −= (2)

Example 1. M/M/1 Queueing System

It is instructive to verify equation (2) directly for the classical single server queueing system. From page 143 of Gross and Harris (1985), we know that

t

tIt

tPdtd

eb

)2()(

)(1

0,1

λµµλλµ +−

=

A convenient expression for )(~

0,0 tP is found in Parthasarathy (1987) and is restated below

dyyyIyItPt

e∫

+−−=

0020,0

)()2()2()(~ µλλµλµµ

where 210 ,, III are modified Bessel functions of the first kind. It now follows from the Fundamental Theorem of Calculus and elementary properties of Bessel functions, see page 134 of Gross and Harris (1985), that

=)(0,1 tPdtd b )(~

0,0 tPdtd

−

This is equivalent to (2). This example provides a bridge between sections 2.10 and 2.11 of Gross and Harris (1985).

The problem of determining busy period distributions of a queueing system modeled by a given birth-death process is equivalent, by equation (1), to finding transient probability functions in a related birth-death process. This connection unifies transient analysis for queues that are modeled by birth-death processes and provides practitioners an alternative way to calculate time dependent probabilities.

4. Relating Steady State Distributions and Ruin Probabilities in Birth-Death Chains

Suppose we consider a birth-death chain on a finite state space having transition

probability diagram as in Figure 3. For convenience, we reproduce this diagram as Figure 8.


10

2−Hp 0r

2r

3r

1r

0

1q

2

3q

2p

1 3

4q

3p 1p0p

2q

Hq

1−Hp

H H-1

Hr

1−Hr

… 1−Hq

Figure 8 Assume the birth-death chain in Figure 8 has a steady state distribution, )( jπ , for j = 0,1,2 ,…, H. It follows from a well known formula for the stationary distribution, cf., pages 50-52 Hoel, Port and Stone (1972) that

∑=

= H

kk

jj

0

)(π

ππ for j = 0,1,2,…H (3)

where

≤≤

=

= − Hjifqqqq

ppppjif

j

jj 1 ...

...0 1

321

1210π

This formula comes from solving a system of linear equations. The ruin probability is also derived in Hoel, Port and Stone (1972), pages 29-31 as

∑

∑−

=

−

==< 1

0

1

00 )( H

kk

j

kk

Hj TTPγ

γ for j = 0,1,2,…H (4)

where kT represents the time to first reach state k and where

<≤

==

Hkifpppqqq

kif

k

kk 1 ......

0 1

21

21γ

This formula calculates the probability of reaching state H before we reach state 0 assuming we start at state j. Expression (4) is determined by a one-step backwards analysis and skillful manipulation of recurrence relations.

Comparison of expressions (3) and (4) reveal the formulas are similar but with the p’s and q’s switched. Why is this so? It turns out that the connection between the original birth-


11

death chain and its dual birth-death chain is the key to realizing how these two problems are related.

Suppose the original birth-death chain follows Figure 3 (or Figure 8) and its dual birth-death chain is diagramed in Figure 4. Assume all the conditions on the p’s, q’s, r’s and s'ρ stated following Figure 3 and before Theorem 2.2 are still in effect. This ensures that both the original and dual birth-death chains have nonnegative transition probabilities as pictured. We again assume the chain of Figure 3 has a steady state distribution, )( jπ , for j = 0,1,2 ,…, H. By Theorem 2.2,

[ ]∑=

−−=H

rk

nks

nks

nsr PPP )(*

,1)(*

,)(

,

for 0≥n and for all states r , s = 0,1,2,3,…,H. Substitute r = i and sum the preceding expression over s from j to H, obtaining

[ ] [ ] [ ]∑∑∑∑∑∑=

−= =

−= =

−=

−=−=−=H

ik

nkj

nkH

H

ik

H

js

nks

nks

H

js

H

ik

nks

nks

H

js

nsi PPPPPPP )(*

,1)(*

,)(*

,1)(*

,)(*

,1)(*

,)(

, ∑=

−−=H

ik

nkjP )*(

,11

by simplifying the telescoping series and noting that H is an absorbing state. Now suppose

∞→n . On the left hand side, there is convergence to the steady state probabilities of the original birth-death chain. On the right hand side, all but one term convergences to 0 since all states except k = H are transient states and therefore the n-step probabilities vanish as ∞→n . So

)(1lim1)( 11)*(

,1 −−−∞→=

<−=−=∑ TTPPs Hjn

Hjn

H

js

π

By considering the complement of this equation,

)()( 11

1

0−−

−

=

<=∑ TTPs Hj

j

s

π

This last expression says the probability of hitting state H first before hitting state -1 in

the dual birth-death chain of Figure 4 equals a sum of the steady state probabilities in the original birth-death chain of Figure 3. This explains why the p’s and q’s are reversed. Therefore, a solution of the steady state distribution on Figure 3 provides a way to determine the ruin probabilities and vice versa. Note a similar unification of these two problems also occurs for birth-death processes as well. Limit arguments, such as those in Hoel, Port and Stone (1972) may now be used to extend these results to the countable state space setting. Example 2. Classical Gambler’s Ruin

Consider the birth-death chain in Figure 9. Suppose p + q = 1 where p, q > 0, qp ≠ . We wish to determine )( 11 −− < TTP Hj where j = 1,2,3,…, H. This is the classical gambler’s ruin problem except with an extra state, -1, where the usual roles of p and q have been reversed.


12

1

-1

1

p

q p

0 … p

p

q

2

q p

p

q

H H-1

1

q p

p

p

q

0

q

2

q

p

1 3

q

p p p

q

q

p

H H-1

p

p

… q

Figure 9

By the preceding argument, we have )()( 11

1

0−−

−

=

<=∑ TTPs Hj

j

s

π , where )( jπ is the steady state

distribution of the following birth-death chain.

Figure 10 By (3)

∑=

= H

kk

jj

0

)(π

ππ

where j = 0,1,2,…,H and

≤≤

==

Hjifqp

jif

j

jj 1

0 1 π

By summing the finite geometric sequence,

−

−

=+1

1

1)(

H

j

qp

qp

jπ

π

Again, summing the geometric sequence simplifies ∑−

=

1

0

)(j

s

sπ as


13

=∑−

=

1

0

)(j

s

sπ

−

−

=

−

−

−

−

=

−

−

++

−

=+∑ 11

1

01

1

1

1

1

1

1

1

1

H

jj

H

j

sH

s

qp

qp

qp

qp

qp

qp

qp

qp

π )( 11 −− <= TTP Hj

This produces the desired ruin probabilities which are seen to agree with the previously well known expression (4) once the p’s and q’s are switched and H is replaced by H+1. 5. The n-step Transition Probability of Finite, Diagonalizable Markov Chains

In this section, we characterize the n-step transition probability of any Markov chain on a

finite state space having a 1-step transition probability matrix, P, which is diagonalizable over the real numbers. We refer to these chains as finite, diagonalizable Markov chains. Let S = {1, 2, 3,…, N} be the state space. By the Cayley-Hamilton Theorem, we know f(P) = O where O is the zero matrix, that is, O is the N by N matrix of zeros and f(x) = det (P-xI) and I is the usual N by N identity matrix. Recall a minimal polynomial, m(x) (with regards to P), is a polynomial with real coefficients of lowest degree such that m(P) = O. By Theorem 7.4 on page 114 of Nering (1970), P is diagonalizable if and only if m(x) factors into distinct linear factors with real coefficients. So suppose the distinct real roots of m(x) are k,...,z,zz 21 where ≤1 k ≤ N. By Theorem 4.2 on page 101 of Nering (1970), the kzzz ,...,, 21 are also roots of f(x), that is, they are eigenvalues of P. We assume kzzz <<< ...21 . By page 287, Theorem 7.10 (e) of Noble and Daniel (1988)

∞≤ Phz for Nh ≤≤0

where ∑=

∞=

N

ji 1max ji,PP . For P a stochastic matrix,

∞P = 1. So

1≤hz for Nh ≤≤0

Furthermore, since the vector (1,1,…,1) is an eigenvector of P with eigenvalue of 1, we know 1=kz . Suppose m(x) has the form

m(x) = kkkkk axaxaxaxa +++++ −

−− 11

22

110 ...

with real coefficients and 00 ≠a . But m(P) = O or

O = m(P) = IPPPP kkkkk aaaaa +++++ −

−− 11

22

110 ...


14

So for kn ≥ ,

O = knk

knk

nnn aaaaa −+−−

−− +++++ ... 11

22

110 PPPPP

Therefore

knkknknnn

aa

aa

aa

aa −+−−−− −−−−−= ...

0

1

0

12

0

21

0

1 PPPPP

But the entries of n P are )(

,njiP , the n-step transition probabilities, so

)(

,0

)1(,

0

1)2(,

0

2)1(,

0

1)(, ... kn

jikkn

jikn

jinji

nji P

aa

Pa

aP

aa

Paa

P −+−−−− −−−−−=

for all i, j in S = {1, 2, 3,…, N} and kn ≥ . That is )(

,njiP satisfies the same linear, constant

coefficient recurrence relation for any i, j in S = {1, 2, 3,…, N}. Hence, the characteristic equation of these recurrence relations is m(x) for each i, j in S = {1, 2, 3,…, N}. Thus

)(,njiP = n

kk

jin

jin

ji zAzAzA ,22,1

1, ... +++ (5)

for kn ≥ where the coefficients khAh

ji ≤≤1for , may be determined in terms of

khzh ≤≤1for and 12for )(, −≤≤ knkP nji by solving a system of linear equations. Here we are

assuming that khzh ≤≤1for and 12kfor )(, −≤≤ knP nji are known. We have established the

following characterization of n-step transitional probabilities in finite, diagonalizable Markov chains. Theorem 5.1 Suppose P is the 1-step transition probability matrix of a Markov chain on a finite state space. Also suppose P is diagonalizable over the real numbers. Then

)(,njiP = n

kk

jink

kji

nji

nji zAzAzAzA ,1

1,2

2,1

1, ... ++++ −

− (6) for kn ≥ where 1...1 21 =<<<≤− kzzz and ≤1 k ≤ N and where the

coefficients, khAhji ≤≤1for , do not depend upon n.

Corollary 5.2 Suppose P is the 1-step transition probability matrix of a finite, diagonalizable, recurrent, aperiodic Markov chain, then 11 z<− or 01

, =jiA in equation (6).

Proof.


15

2−Hλ

0

1µ

2

3µ

2λ

1 3

4µ

3λ 1λ 0λ

2µ

H H-1

1−Hλ

… 1−Hµ

Hµ

If 11 −=z , then the limit of the right hand side of (6) as ∞→n exists and is nonzero because the chain has a steady state distribution, see Hoel, Port and Stone (1972) Theorem 7 on page 73, while the limit of the right hand side of (6) does not exist unless 1

, jiA = 0 .

Remarks. 1. Theorem 5.1 tells us that on a finite, diagonalizable Markov chain, the n-step transition

probabilities are all described by the same constant coefficient, linear recurrence relation and therefore the n-step transition probabilities can each be expressed as a linear combination of powers of roots corresponding to a single characteristic equation. These roots take values in the interval [-1,1]. Moreover, one of these roots is always 1. If the steady state distribution exists, the roots are in the interval (-1,1] and the coefficient of root 1 is the steady state probability of the Markov chain.

2. A similar result has been derived by a different method for birth-death chains, see Mohanty (1991).

3. Theorem 5.1 may be generalized by removing the condition that P is diagonalizable.

Here we would replace (6) with an expression having coefficients that would vary with “n” as described by the theory of linear, constant coefficient, recurrence relations having multiple roots.

Application.

The transient probability functions of a finite birth-death process has been solved using a variety of techniques over the years, see, for example, Kijima (1997), Mohanty et al. (1993), Rosenlund (1978) and the references within these articles. We now sketch out a new solution method using Theorem 5.1 and the well known result of randomization (see Theorem 5.3). We are content to convey the broad strokes of this new approach here. A more detailed, complete account may be found in Mortensen (2003) and more general results will appear in Krinik et al. (2003c).

Suppose we consider a recurrent birth-death process on a finite state space having rate transition diagram as follows.

Figure 11

From this finite birth-death process of Figure 11, the associated randomized birth-death

chain of Figure 12 is considered below. .


16

2−Hp 0r

2r

3r

1r

0

1q

2

3q

2p

1 3

4q

3p 1p0p

2q

Hq

1−Hp

H H-1

Hr

1−Hr

… 1−Hq

whereb

p ii

λ= , for 1,...,3,2,1,0 −= Hi and

bq i

i

µ= , for Hi ,...,3,2,1= and

br ii

i

µλ +−= 1 for

1,...,3,2,1 −= Hi and b

r 00 1

λ−= and

br H

H

µ−= 1 with ii

Hib µλ +=

=max

,...,0 where 0, µλH , are

taken to be 0. The following theorem, called randomization (or uniformization), is a well-known result

used primarily for the numerical computation of the transition probability functions )(, tP ji of a Markov process. It applies for the preceding finite birth-death process of Figure 11 and more generally, for any Markov process (with countable state space) having uniformly bounded diagonal transition rates in the Q matrix, cf. Gross and Harris (1985) or Medhi (2003). For our purposes here, it suffices to state the randomization result in terms of the birth-death process of Figure 11 and its associated randomized birth-death chain of Figure 12. Theorem 5.3 Suppose )(, tP ji is the transition probability function of the birth-death process of

Figure 11. Then )(, tP ji may be written as

( ) )(,

0, !

)( nji

n

nbt

ji Pn

btetP ∑

∞

=

−= for i, j=0,1,2,3, …

where )(,njiP is the n-step transition probability of the associated birth-death chain of Figure 12.

Note that )(, tP ji is completely determined once )(

,njiP is “known”. This approach to finding

analytic solutions for )(, tP ji by finding )(,njiP began in the 1990’s see, for example, Leguesdron

et al. (1993), Krinik et al. (1997) and Bohm et al. (1997). So by Theorem 5.2 we have

( ) ( ) ( ) )(,

1

)(,

0

)(,

0, !!!

)( nji

Hn

nbtn

ji

H

n

nbtn

jin

nbt

ji Pn

bteP

nbt

ePn

btetP ∑∑∑

∞

+=

−

=

−∞

=

− +==

The birth-death chain diagramed in Figure 12 is a finite Markov chain. It is also known that the one-step probability matrix, P, of such a birth-death chain is diagonalizable with H + 1 distinct real eigenvalues 1...1 121 =<<<≤− +Hzzz , see Buchanan and Turner (1992), Kijima (1997) or Mortensen (2003). Therefore, Theorem 5.1 applies giving

Figure 12


17

)(, tP ji = ( ) ( ) ( )∑∑

∞

+=

+−

=

− +++++1

1,,2

2,1

1,

)(,

0

...!! Hn

Hji

nH

Hji

nji

nji

nbtn

ji

H

n

nbt AzAzAzA

nbt

ePn

bte

=( ) ( ) ( ) ( ) ( )( )

∑∑∞

+=

+−

=

− +++++

1

1,,2

2,1

1,)(

,0 !

...

! Hn

nHji

n

HH

ji

n

ji

n

jibtnji

H

n

nbt

n

btAbtzAbtzAbtzAeP

nbt

e

So

( ) ( )

( ) ( ) ( )( )∑

∑

=

+−

+−−−

=

−

+++−

+++++=

H

n

nHji

n

ji

n

jibt

Hji

zbtHji

zbtji

zbtji

nji

H

n

nbt

ji

n

btAbtzAbtzAe

AeAeAeAPn

btetP H

0

1,2

2,1

1,

1,

)1(,

)1(2,

)1(1,

)(,

0,

!

...

...!

)( 21

In this expression, b, )(

,njiP , h

jiA , and hz are all considered to be known numerical quantities Remarks. 1. A nice aspect of this approach is that it does not require Laplace transforms.

2. The preceding argument applies more generally, see Krinik et al. (2003c) for details.

Acknowledgements.

The authors wish to thank Michael L. Green and Randall J. Swift for their contributions to the development of the material within section 5.


18

References Anderson, W.J. (1991). Continuous-Time Markov Chains, An Applications-Oriented Approach. Springer-Verlag, New York. Bhattacharya, R.N.and E.C. Waymire (1990). Stochastic Processes With Applications. John Wiley & Sons, New York. Bohm, W., A. Krinik, and S.G. Mohanty (1997). The Combinatorics of Birth-Death Processes and Applications to Queues. Queueing Systems, 26, 255-267. Buchanan, J.L. and P.R. Turner (1992). Numerical Methods and Analysis. McGraw-Hill, New York. Feller, W. (1950). An Introduction to Probability Theory and its Applications. John Wiley and Sons, New York. Green, Michael L., A. Krinik, C. Mortensen, R. Swift and G. Rubino (2003). Transient Probability Functions-A Sample Path Approach. Journal of Discrete Mathematics and Theoretical Computer Science (to appear). Gross, D. and C.M. Harris (1985). Fundamentals of Queueing Theory, Second Edition. John Wiley and Sons, New York.

Hoel, P.G., S.C. Port and C.J. Stone (1972). Introduction to Stochastic Processes. Houghton Mifflin Company, Boston. Karlin, S. and H.M. Taylor (1975). A First Course in Stochastic Processes, Second Edition. Academic Press, New York. Krinik, A., D. Marcus, D. Kalman and T. Cheng (1997). Transient Solution of the M/M/1 Queueing System via Randomization. In: J. Goldstein, N. Gretsky and J.J. Uhl, Eds., Stochastic Processes and Functional Analysis, Volume 186 in the Lecture Notes in Pure and Applied Mathematical Series, Marcel Dekker, New York, 137-145. Krinik, A., G. Rubino, D. Marcus, R.J. Swift, H. Kasfy, H. Lam (2003a). Dual Processes To Solve Single Server Systems. Journal of Statistical Planning and Inference (to appear).


19

Krinik, A., G. Rubino, R.J. Swift, M.L. Green, H. Lam (2003b). Dual Processes to Solve Queueing Systems With Catastrophes (preprint). Krinik, A., C. Mortensen, R. J. Swift, M. L. Green (2003c). Transient Solutions of Finite Markov Processes (preprint). Kijima, M.(1997). Markov Processes For Stochastic Modeling. Chapman and Hall, London P. Leguesdron, J. Pellaumail, G. Rubino, B. Sericola (1993). Transient Solution of the M/M/1 Queue. Adv. Appl. Probability, 25.

Medhi, J. (2003). Stochastic Models in Queueing Theory, Second Edition. Academic Press, Amsterdam. Mohanty, S.G., A. Montazer-Haghighi and R. Trueblood (1993). On the Transient Behavior of a Finite Birth-Death Process With an Application. Computers Ops.Res., 20, pp 239-248. Mohanty, S.G.(1991). On the Transient Behaviour of a Finite Discrete Time Birth-Death Process, ASR, 5, 2, pp 1-7. Mortensen, C (2003). Transient Probability Functions of Finite Birth-Death Processes, Masters Thesis, California State Polytechnic University, Pomona, California. Nering, E.V.(1970). Linear Algebra and Matrix Theory, Second Edition. John Wiley and Sons, New York. Noble B. and J.W. Daniel (1988). Applied Linear Algebra, Third Edition, Prentice-Hall, New Jersey. Parthasarathy, P.R.(1987). A Transient Solution to an M/M/1 Queue: A Simple Approach. Adv. Appl. Prob., 19, pp. 997-998. Rosenlund, S.I. (1978). Transition Probabilities for a Truncated Birth-Death Process. Scand. J. Statist, 5, pp 119-122.

Connections Between Birth-Death Processes - Inria

Documents