RANDOM PROCESSES P… · Random Processes The values assumed by the random variables X(t) are called the states and the ... Wide-sense Stationary Process A random process {X(t)} ...

Unit - III

RANDOM PROCESSES

B. Thilaka

Applied Mathematics

Random Processes

A family of random variables {X(t,s)│tεT, sεS} defined over a given probability space and indexed

by the parameter t, where ‘t’ varies over the index set T, is known as random process/ chance

process/ stochastic process.

X: SxT→R

X(s,t)=x

Notation: X(t)

Random Processes

The values assumed by the random variables X(t) are called the states and the set of all possible values is called the state space of the random process.

Since the random process {X(t)} is a function of both s and t, we have the following

Random Processes

Observations:

1. s and t are fixed : X(t) –real number

2. t is fixed : X(t) –random variable

3. s is fixed : X(t) - function of time called the sample function or realisation of the process

4. s and t are varying : X(t) – random process

Classification

Three types

Type I

State Space, Parameter Space:

Discrete/ Continuous

Classification

S

T

Discrete Continuous

Discrete Discrete

random

sequence

Continuous

random

sequence

Continuous Discrete random

process

Continuous random

process

Classification Predictable/ Deterministic :

Future values can be predicted from past values.

eg: X(t)= A cos (wt+θ), where any non-empty combination of A, w, θ may be random variables

Unpredictable/ Non- deterministic:

Future values cannot be predicted from past values.

eg: Brownian motion

Classification

Stationarity

Probability for a random process: For a fixed time t1, X(t1) is a random variable that describes the state of the process at time t1.

The first order distribution function of a random process is defined as

First order Distribution and Density Function

The first order density function of a random process is defined as

11111 )(; xtXPxFtxF XX

txFx

txf XX ;; 1

1

11

First order Distribution Function

Statistical Average

The statistical average of the random process {X(t)} is defined as

provided the quantity on the RHS exists.

1111 ;)]([)( dxtxfxtXEt X

First order stationary process

The random process {X(t)} is said to be a first order stationary process / stationary to order one if the first order density/ distribution function is invariant with respect to a shift in the time origin

(or)

,t,xt;xft;xf 1111X11X

,,;; 111111 txtxFtxF XX


Result:

The statistical mean of a first order stationary random process is a constant with respect to time.

Proof:

Let the random process {X(t)} be a first order stationary process .

Then its first order density function satisfies the property that


,t,xt;xft;xf 1111X11X

The mean of the random process {X(t)} at time t1 is defined as

11111 ;)]([ dxtxfxtXE X


Consider

Let t2 = t1 +∆. Then

21222 ;)]([ dxtxfxtXE

2122 ; dxtxfx

22222 ;)]([ dxtxfxtXE


Hence E[X(t)] is a constant with respect to time.

)]([)]([ tXEtXE

Second order Distribution function

The second order joint distribution function of a random process {X(t)} is defined as

The second order joint density function of the random process {X(t)} is defined as

22112121 )(,)(,;, xtXxtXPttxxFX

2121

21

2

2121 ,;,,;, ttxxFxx

ttxxf XX

Second order stationary process

A random process {X(t)} is said to be a second order stationary process/ stationary to order two if its second order distribution/ density function is invariant with respect to a shift in the time origin.

In other words,

(and / or)

,,,,,;,,;, 212121212121 ttxxttxxfttxxf XX

,,,,,;,,;, 212121212121 ttxxttxxFttxxF XX

Second order Processes

Auto-correlation function:

The auto-correlation function of a random process {X(t)} is defined as

provided the quantity on the RHS exists.

)t(X)t(XEt,tR 2121XX


Significance of auto-correlation function

1. It provides a measure of similarity between two observations of the random process {X(t)} at different points of time t1 and t2.

2. It also defines how a signal is similar to a time-shifted version of itself


Auto-covariance function:

The auto-covariance function of a random process {X(t)} is defined as

)]([)()]([)(),( 221121 tXEtXtXEtXEttCXX

)]([)]([)()]([)]([)()()( 21212121 tXEtXEtXtXEtXEtXtXtXE

)]([)]([)]([)]([)]([)]([()( 21212121 tXEtXEtXEtXEtXEtXEtXtXE

)]([)]([),(),( 212121 tXEtXEttRttC XXXX

Wide-sense Stationary Process

A random process {X(t)} is said to be WSS/ weakly stationary/ covariance stationary process if it satisfies the following conditions:

1. E[X(t)] is a constant with respect to time.

2. RXX(t1,t2) is a function of the length of the

time difference.

i.e. RXX(t1,t2)= RXX(t1-t2)

3. E[X2(t)]<∞.


Remark:

Since

CXX(t1,t2)= CXX(t1-t2), the auto-covariance function CXX(t1,t2) is a function of the length of the time difference.

Hence, a WSS process is also called a covariance stationary process.

)]([)]([),(),( 212121 tXEtXEttRttC XXXX


Alternately, a random process (second order) is said to be WSS/ weakly stationary process if it satisfies

Remark: A second order stationary process is a WSS process, but the converse need not be true.

XXXX Rt,tR

)]t(X[E

nth order distribution/ density function

The nth order joint distribution function of a random process {X(t)} is defined as

The nth order joint density function of a random process {X(t)} is defined as

)(,..,)(,)(,..,,;,..,, 22112121 nnnX tXxtXxtXPtttxxxF

nnX

n

n

nnX tttxxxFxxx

tttxxxf ,..,,;,..,,...

,..,,;,..,, 2121

21

2121

nth order stationary process

A random process {X(t)} is said to be nth order

stationary/ stationary to order n if the nth order density/ distribution function is invariant with

respect to a shift in the time origin

(i.e.)

(and / or)

,,..,,,,..,,

,,..,;,..,,..,,;,..,

2121

21212121

nn

nnXnnX

tttxxx

tttxxxftttxxxf

,,..,,,,..,,

,,..,;,..,,..,,;,..,

2121

21212121

nn

nnXnnX

tttxxx

tttxxxFtttxxxF

Strictly stationary Process

A random process is said to be strictly stationary (SSS)/ stationary in the strict sense if is stationary to all orders n

Aside: For a SSS- CDF/PDF (nth order) is invariant with respect to a shift in the time origin

Mean-constant

Auto-correlation, Auto-covariance- functions of length of time intervals.

Strictly stationary Process

Remark :

If a random process is stationary to order n, then it is stationary to all orders k≤n.

WHY ?

Evolutionary Process

Random processes which are not stationary to any order are called non-stationary/ evolutionary processes

Further Properties Cross correlation function:

The cross correlation function of two random processes {X(t)} and {Y(t)} is defined as RXY(t1,t2)=E[X(t1)Y(t2)]

Cross covariance function:

The cross correlation function of two random processes {X(t)} and {Y(t)} is defined as

CXY(t1,t2)=E{[X(t1)-E(X(t1))][Y(t2)-E(Y(t2))]}

=RXY(t1,t2)-E[X(t1)]E[Y(t2)]

Further Properties

Two random processes {X(t)} and {Y(t)} are said to be jointly WSS if

(i) both {X(t)} and {Y(t)} are each WSS

(ii) the cross-correlation function RXY(t1,t2)=E[X(t1)Y(t2)] is exclusively a function of the length of the time interval (t2-t1)

Independent Random Process

• A random process {X(t)} is said to be an independent random process if its nth order joint distribution function satisfies the property that

A similar condition holds for joint p.m.f./ p.d.f.

nn

nnXXXnnX

tttxxx

txFtxFtxFtttxxxF

,..,,,,..,,

),;().....;();(,..,,;,..,

2121

22112121

Process with independent increments

A random process {X(t)} is defined to be a process with independent increments if for all 0<t1< t2<… <tn<t, the random variables X(t2)- X(t1), X(t3)- X(t2),.., X(tn)- X(tn-1) are independent.

Time averages of a random process

Prelude:

The time average of a quantity f(t) (t- time) is defined as

Time average of a random process:

The time average of a random process {X(t)} is defined as

T

T

dttfTT

lttfA )(

2

1)]([

Time averages of a random process

Time auto-correlation function :

The time auto-correlation function of a random process {X(t)} is defined as

T

T

dttxTT

lttXA )(

2

1)]([

ttttdttxtxTT

lttXtXA

T

T

2121 ,,)()(2

1)]()([

Ergodic Process

A random process is said to be ergodic if its time averages are all equal to the corresponding statistical averages.

Examples ???

Random Process

Correlation coefficient of a random process :

The correlation coefficient (normalized auto-covariance function) of a random process {X(t)} is defined as the correlation coefficient between its random variables X(t1) and X(t2) for arbitrary t1 and t2 .

In other words,

Note: Var[X(t)] = CXX(t, t)

)]([)]([

),(),(

21

2121

tXVartXVar

ttCtt XX

XX

Markov Processes

A random process {X(t)} is called a Markov process if for all t0<t1< t2<… <tn<t, the conditional distribution of X(t) given the values of X(t0), X(t1), X(t2),.., X(tn) depends only on X(tn).

nnnn110o x)t(Xx)t(XP]x)t(X,...,x)t(X,x)t(Xx)t(X[P

nnnn110o x)t(Xb)t(XaP]x)t(X,...,x)t(X,x)t(Xb)t(Xa[P

Markov Processes

In other words, a random process {X(t)} is called a Markov process if the future values of the process depend only on the present and are independent of the past.

Examples: Binomial process, Poisson process, Random telegraph process

Markov Processes

Note:

If the state space of a Markov process is discrete, then the Markov process is called a Markov chain.

Markov Processes

State Space

Discrete Continuous

Time Discrete Discrete- Time

Markov Chain

Discrete- Time

Markov Process

Continuous Continuous - Time Markov

Chain

Continuous - Time Markov

Process

Markov Processes

Markov Processes

• If in a Markov chain, the conditional distribution is invariant with respect to a shift in the time origin, the Markov chain is said to be time homogeneous.

• In a homogeneous Markov chain, the past history of

the process is completely summarized in the current state.

Hence, the distribution of time that the process

spends in the current state must be memoryless.

Counting Process

A random process {X(t)} is called a counting process if X(t) represents the total number of “events” that have occurred in the interval [0,t).

1. X(t) ≥ 0; X(0)=0- begins at time t=0

2. X(t) is integer valued

3. If s ≤ t, then X(s) ≤ X(t) 4. X(t) - X(s) : number of events in [s,t]

Counting Process

Alternately, the random process {N(t)} is called a counting process if it assumes only integer values and it is an increasing function of time.

Types of Random Processes

Bernoulli process : {Xn:n≥1}, Xn’s are i.i.d. Bernoulli variates with parameter p.

Consider a sequence of independent and identical Bernoulli trials.

Let the random variable Yi denote the outcome of the ith trial so that the event {Yi=1} indicates success with probability p on the ith trial for all ‘i’ and the event {Yi=0} indicates failure with probability (1-p)=q on the ith trial for all ‘i’.

Bernoulli Process

Hence the Yi‘s may be considered as independently and identically distributed random variables.

The random process {Yn} is called a Bernoulli process with P[Yn=1]=p and P[Yn=0]=1-p, for all n.

Discussion: Classify Bernoulli process ( 3 types)

Binomial Process

Binomial process :

Consider a Bernoulli process {Yi} where the Yi‘s are independently and identically distributed Bernoulli random variables with parameter p. Form another stochastic process {Sn} with Sn= X1+X2+..+Xn

The random variable Sn=follows a Binomial distribution with parameters n and p.

Binomial Process

The random process {Sn:n≥1} is called a Binomial process.

Discussion: Preliminary classification of Binomial Process

Binomial Process First order pmf:

The first order p.m.f. of the random process {Sn} is given by

Can you classify Binomial process even further??

nkppnCkSPknk

kn ,...,1,0,)1(

npSE n ][

npqpnpSVar n )1(][

Binomial Process

Consider Sn+1= Y1+Y2+…+Yn+1

= Y1+Y2+…+Yn+Yn+1

= Sn+Xn+1

Hence

P[Sn+1= k│ Sn= k] = P[Yn+1= 0]=1-p

P[Sn+1= k+1│ Sn= k] = P[Yn+1= 1]=p

Can you now classify Binomial process even further?

Binomial Process

The Binomial process is also called the Binomial counting process.

Probability generating function:

Second order joint p.m.f.

nS pzqzGn

)(

nllknmmkppkl

mn

k

mlSkSP

lnl

nm ,..,1,0,,,,..,1,0,)1(,

Binomial Process

Observation:

The total number of trials T from the beginning of the process until and including the first success is a geometric random variable.

The number of trials after (i-1)th success upto and including the ith success will have the same distribution as T

Can you generalise?

Binomial Process

If Tn is the number of trials upto and including the ith success, then Tn is the n-fold convolution of T with itself and follows a negative binomial distribution with E[Tn]=n/p and Var[Tn]=n/1-p.

Binomial Process

• If in the Binomial process {Sn}, n is large and p is small such that np is finite, then the Binomial process approaches a Poisson process with parameter λ=np.

Poisson Process

Let E be any random event and {N(t)} denote the number of occurrences of the event E in an interval of length t.

Let pn(t)= P[N(t)=n].

The counting process {N(t)} is called a Poisson process if it satisfies the following postulates:

Poisson Process

1. Independence : {N(t)} is independent of the number of occurrences of E in an interval prior to (0,t) i.e. future changes in N{t} are independent of the past changes.

2. Homogeneity in time: pn(t) depends only on the length ‘t’ of the time interval and is independent of where the interval is situated i.e. pn(t) gives the probability of the number of occurrences of E in the interval (t0, t0 +t) for all t0

3. Regularity: In an interval of infinitesimal length h, the probability of exactly one event occurring is λh+o(h), the probability of zero events occurring is 1-λh+o(h), the probability of more than one event occurring is o(h).

Poisson Process

Poisson Process

Relax the postulates:

3. Regularity- Compound Poisson Process (multiple occurrences at any instant)

2. Homogeneity in time: λ is a function of time (λ(t))- Non-homogeneous

1. Independence: future depends on present- Markov process

Poisson Process

Result : The Poisson process defined above follows Poisson distribution with mean λt. i.e.

Proof:

Let {N(t)} be a Poisson process with parameter λ. We now consider

,.....2,1,0n,t!n

e]n)t(N[P)t(p

nt

n

Poisson Processes- pmf

(Theorem on total probability)

(homogeneity)

])([)( nttNPttpn

n

k

ktNPktNnttNP0

])([])(/)([

n

k

ktNPkntNttNP0

])([])()([

n

k

ktNPkntttNP0

])([]),([


Assuming ∆t to be of infinitesimal length, we have

])([])([0

ktNPkntNPn

k

)(])([])([

]1)([]1)([]0)([])([

2

0

tokntNPktNP

tNPntNPtNPntNP

n

k

2

0

110 )()()()()()(m

k

knknn tptptptptptp


(Regularity)

Dividing throughout by ∆t

2

0

1 )()()()()(1)()(n

k

knnn totptottptottpttp

)()()()()()()()(

2

0

1 tototptottpttptpttpn

k

knnnn

t

totptp

t

tpttpnn

nn

)()()(

)()(1


Taking limits as ∆t →0 on both sides of the above equation,

--------(1)

At n=0, we have

1,1 ntptpdt

tdpnn

n

]0)([)(0 ttNPttp

]0)([]0)(/0)([ tNPtNttNP


(homogeneity)

(regularity)

Taking limits as ∆t →0 on both sides of the above equation,

]0)([]0)()([ tNPtNttNP

]0)([]0),([ tNPtttNP

]0)([]0)([ tNPtNP

)()(1 0 tptot

t

totp

t

tpttp

)()(

)()(0

00


-----(2)

We now solve the above system of differential difference equations (1) and (2) subject to the initial conditions

-------(3)

Consider equation (2) namely,

tpdt

tdp0

0

10)0(,1)0(0 npp n


subject to

At t=0,

-----(4a)

tpdt

tdp0

0 1)0(0 p

dttp

tdp

0

0

cttp 0log

ctetp

0

tKetp

0

Kp 100

tetp

0


Substituting n=1 in equation (1), we have

subject to

On solving the above equation, we have

tptpdt

tdp01

1

tetp

dt

tdp 11

0)0(1 p

0)0(, 111

petpdt

tdp t

cdteeetpdttdt

1

cdteeetpttt 1


At t=0,

-----(4b)

Substituting n=2 in equation (1), we have

ttcetetp

)(1

)0()0(

1 )0(0 ceep

0c

)(1 tetpt

tptpdt

tdp12

2


.

0)0(),( 222

ptetpdt

tdp t

cdteteetpdttdt

2

cdteteetpttt 2

2

ct

2

22


At t=0,

-----(4c)

Proceeding in this manner, we have

ttce

tetp

2

22

2

)0()0(

2 )0(0 ceep

0c

!2

)()(

2

2

tetp

t


Hence, the number of events {N(t)} in an interval of length t follows a Poisson distribution with mean λt.

Can you classify Poisson process??

,.....2,1,0,!

])([)( ntn

entNPtp

nt

n

Poisson Processes

• Poisson process is an evolutionary process

• E[N(t)]=λt and Var[N(t)]=λt . Further,

Hence λ is called the arrival rate of the process

0t

)t(NVarlt,

t

)t(NElt

tt

Poisson Process

Characterization: If {N(t)} is a Poisson process with mean λt, then the –occurrence/ inter-arrival time follows an exponential distribution with mean 1/ λ.

Proof: Let {N(t)} be a Poisson process with parameter λ. Then

,.....2,1,0n,t!n

e)t(p

nt

n

Poisson Process

If W denotes the time between two successive arrivals/ occurrences of the event, the CDF of W is

The above is the CDF of an exponential variate with

mean 1/λ

]0)w(N[P1

]wW[P1]wW[PwFW

w

W e1wF

Poisson Process

If the inter-arrival times are i.i.d. (not necessarily exponential), then we have a renewal process.

Poisson Process- Properties

• Superposition

• Decomposition

• Markov Process

• Difference of two Poisson processes is not Poisson.


Result:

The superposition of n independent Poisson processes with means λ1t, λ2t, …., λnt, respectively is a Poisson process with mean λ1t+ λ2t+ …. +λnt.

Proof:

Consider n independent Poisson processes N1(t), N2(t),…, σn(t), with respective means λ1t, λ2t, ….. λnt.


The moment generating function(m.g.f.) of each Nk(t) is given by

By property of moment generating functions, the mg.f. of the sum

is given by

)(

)( )(tN

tNk

keEM

)1( etke

n

k

k tNtN1

)()(


which is the moment generating function of a Poisson distribution with mean λ1t+ λ2t+ …. +λnt.

n

k

tNtN kMM

1

)()( )()(

n

k

etke1

)1(

n

k

k et

e 1

)1(


Hence by uniqueness property, the sum of n independent Poisson processes with means λ1t, λ2t, …., λnt, respectively is a Poisson process with mean λt=λ1t+ λ2t+ …. +λnt.


Decomposition of Poisson process:

A Poisson process N(t) with mean arrival rate λ can be decomposed into ‘n’ mutually independent Poisson processes with arrival rates p1 λ, p2 λ,.., pn λ such that

p1 + p2 +… + pn =1

Proof: HW

Poisson Process

Note:

If N(t) is a Poisson process with mean arrival rate λ, then the time between

‘k’ successive arrivals/ occurrences follows a

(k-1) Erlang distribution .

WHY??

Poisson Process

Second order joint p.m.f. :

Given a Poisson process with mean arrival rate λ, the second order joint p.m.f. is obtained as follows:

21211111222211 ,,)()()()(,)( nnttntNPntNntNPntNntNP

2121121211 ,,)()()( nnttnntNtNPntNP

2121122111 ,,),()( nnttnnttNPntNP

Poisson Process

(homogeneity)

2121121211 ,,)()( nnttnnttNPntNP

2121

12

)(

12

)(

1

1 ,,!

)(

!

122111

nnttnn

tte

n

tennttnt

Poisson Process

Hence

elsewhere

nnttnnn

ttte

ntNntNP

nnnnt

,0

,,!!

)(

)(,)( 2121

121

)(

121

2211

12122

Poisson Process

Similarly the third order joint p.m.f. of the Poisson process with mean arrival rate λ is given by

Poisson Process

.

elsewhere

nnntttnnnnn

ttttte

ntNntNntNP

nnnnnnt

,0

,,)!(!!

)()(

)(,)(,)( 321321

23121

)(

23

)(

121

332211

2312133

Poisson Process

Auto-correlation function :

If N(t) is a Poisson process with mean arrival rate λ, then

• E[N(t)] = λt

• Var[N(t)] = λt

• E[N2(t)] = Var[N(t)] + E[N(t)] = λt + λ2t2

Poisson Process

The auto-correlation function of N(t) is now obtained as follows:

(by definition) )]()([),( 2121 tNtNEttRNN

211121 ,)()()()( tttNtNtNtNE

211

2

121 ,)()()()( tttNtNtNtNE

211

2

211 ,)(),()( tttNEttNtNE

Poisson Process

(independence)

211

2

121 ,)()()( tttNEttNtNE

211

2

121 ,)()()( tttNEttNEtNE

21

2

1112121 ,)]([),( tttttttttRNN

21121

2 , ttttt

2121

2

21 ,min),( ttttttRNN

Poisson Process

Auto-covariance function of a Poisson process :

The auto-covariance function of a Poisson process N(t) with mean arrival rate λ is

(by definition)

)]([)]([),(),( 212121 tNEtNEttRttC NNNN

212121

2 ,min tttttt

2121 ,min),( ttttCNN

Poisson Process

The correlation function of a Poisson process N(t) with mean arrival rate λ is given by

)]([)]([

),(,

21

21

21tNVartNVar

ttCtt NN

NN

21

21 ,min

tt

tt

21

21 ,min

tt

tt

Poisson Process

Hence

21

2

121 ,, tt

t

tttNN

Random Processes

Process with stationary increments:

A random process {X(t)} is said to be a process with stationary increments if the distribution of the increments X(t+h)-X(t) depends only on the length ‘h’ of the interval and not on end points.

Random Processes

Wiener process/ Wiener-Einstein Process/ Brownian Motion Process:

A stochastic process {X(t)} is said to be a Wiener Process with drift µ and varaince

2, if

(i) X(t) has independent increments

(ii) every increment X(t)-X(s) is normally distributed with mean µ(t-s) and variance

2(t-s)

Poisson Process

1. For a Poisson process with parameter and for show that

Solution : Consider

(defn)

nkt

s

t

snCntNksNP

knk

k ,...2,1,0,1)()(

ntNksNP )()(

ntNP

ntNksNP

)(

)()(

Poisson Process

(by definition)

ntNP

knstNksNP

)(

)()(

ntNP

knstNPksNP

)(

)()(

!

)(

)!(

))((

!

)( )(

n

te

kn

ste

k

se

nt

knstks

Poisson Process

nnt

knknstkks

te

steese

knk

n

)(

)!(!

!

nn

kn

knkn

kt

t

sts

nC

1

n

kn

knk

kt

t

stts

nC

1

Poisson Process

Hence the proof.

kn

k

k

kt

s

t

snC

1

knk

kt

s

t

snC

1

ntNksNP )()( nkt

s

t

snC

knk

k ,....2,1,0,1

Poisson Process

2. Suppose that customers arrive at a bank according to a Poisson process with a mean rate of 3 per minute. Find the probability that during a time interval of 2 minutes (a) exactly 4 customers arrive

(b) more than 4 customers arrive.

Solution: Let N(t) be a Poisson process with mean arrival rate λ.

Poisson Process

Given that λ=3.

Hence

(a)

ktNP )(

,..,2,1,0,!

)(k

k

tekt

4)2( NP!4

)2.3( 4)2(3

e

133.0!4

646

e

Poisson Process

(b)

= 0.715

4)2( NP 4)2(1 NP

4)2(3)2(2)2(

1)2(0)2(1

XPXPXP

XPXP

!4

)6(

!3

)6(

!2

)6(

!1

)6(

!0

)6(1

4636261606eeeee

Poisson Process

3. If a customer arrives at a counter according to a Poisson process with a mean rate of 2 per minute, find the probability that (i) 5 customers arrive in a 10 minute period (ii) the interval between 2 successive arrivals is (a) more than 1 minute (b) between 1 and 2 minutes (c) 4 minutes or less (iii) the first two customers arrive within 3 minutes (iv) the average number of customers arriving in 1 hour.

Poisson Process

Solution:

Let N(t) denote the number of customers who arrive at the counter in an interval of length ‘t’. We are given that σ(t) follows a Poisson process with mean arrival rate λ=2 per minute.

Therefore we have,

,.....2,1,0,!

])([ ntn

entNP

nt

Poisson Process

(i)

(ii) Since N(t) is a Poisson process with mean arrival rate λ=2, the inter-arrival time T between 2 successive arrivals follows an exponential distribution with p.d.f.

!5

)102(5)10(

5)10(2xe

NP

!5

)20( 520

e

Poisson Process

(a)

0,2)( 2 tetf

t

1

22)1( dteTPt

1

2

22

te

201 e

1353.0

Poisson Process

(b) 2

1

22)21( dteTPt

2

1

2

22

t

e

241 ee

42 ee

0183.01353.0

1170.0

Poisson Process

(c) P(T≤ 4) = 1-e-2(4)=1-e-8=0.9996

(iii) Since N(t) is a Poisson process with mean arrival rate λ, we know that the inter-arrival time follows an exponential distribution with mean 1/λ.

If Ti denotes the inter arrival time between the (i-1)th customer and the ith customer, then the time taken for the first

Poisson Process

2 customers to arrive is given by T1+T2.

Since T1 andT2 are independently and identically distributed exponential variates with parameter λ, T1+T2 follows a second order Erlang distribution with p.d.f.

0,0,)2(

2

tte

t

0,2 22 tte

t

Poisson Process

=0.9826

dtteTTPt

3

0

2

21 4]3[

3

0

23

0

2

424

ttete

4

1

2

034

66ee

17 6 e

Poisson Process

(iv) Since E[N(t)]= λt,

the average number of customers arriving in one hour is E[N(60)]= 2x60 = 120.

4. A radioactive source emits particles at the rate of 5 per minute in accordance with a Poisson process. Each particle has a probability of 0.6 of being recorded. Find the probability that 10 particles are recorded in a 4 minute period.

Poisson Process

Solution:

Given that the emission of particles follows a Poisson process with arrival rate λ=5 per minute. From the decomposition property of Poisson process, the number of emitted particles N1(t) follows a Poisson process with mean arrival rate λ1= λp =5x0.6/ min

i.e. λ1=3 per minute

Poisson Process

Hence

,.....2,1,0,!

])([ 11

1

ntn

entNP

nt

!10

)12(

!10

)43(10)4(

101210)4(3

1

exe

NP

Poisson Process

5. A machine goes out of order whenever a component fails. The failure of this part follows a Poisson process with a mean rate of 1 per week. Find the probability that 2 weeks have elapsed since last failure. If there are 5 spare parts of this component in an inventory and that the next supply is not due in 10 weeks, find the probability that the machine will not be out of order in the next 10 weeks.

Poisson Process

Solution:

Let X(t) denote the number of failures of the component in t units of time.

Then X(t) follows a Poisson process with

mean failure rate= mean number of failures in a week = λ = 1

P[2 weeks have elapsed since last failure] = P[ Zero failures in the 2 weeks since last failure] =P[X(2)=0]

Poisson Process

=e-2(1) =e-2 =0.135

There are only 5 spare parts and the machine should not go out of order in the next 10 weeks.

Hence

P[ the machine will not be out of order in the next 10 weeks] = P[X(10)≤5]

!0

)2(0]P[X(2)

02

e

Poisson Process

!5

10

!4

10

!3

10

!2

10

!1

10

!0

10510410310210110010

eeeeee

P[X(10)≤5]=0.068

120

100000

24

10000

6

1000

2

10010110

e

Stationary Processes

6. Determine the mean and variance of the random process {X(t)} is given by

Verify whether {X(t)} is stationary or not.

01

,..3,2,1,)1(

)(

)(1

1

nat

at

nat

at

ntXPn

n


Solution:

Consider the random process {X(t)} defined by

Since the first order p.m.f. of X(t) is a function of ‘t’, the random process {X(t)}

01

,..3,2,1,)1(

)(

)(1

1

nat

at

nat

at

ntXPn

n


is not a stationary process. It is an evolutionary process.

Now, the mean of the process is given by

(by definition)

0

])([)(n

ntXnPtXE

...)1(

)(3

)1(2

)1(

11

10

4

2

32

at

at

at

at

atat

at

....

)1(

)(3

121

)1(

12

2

2at

at

at

at

at


E[X(t)]=1

We now compute Var[X(t)]

....1

31

21)1(

12

2 at

at

at

at

at

2

2 11

)1(

1

at

at

at

2

2 1

1

)1(

1

at

atat

at

2

2 1

1

)1(

1

atat

2

2)1(

)1(

1at

at


Var [X(t)]=E[X2(t)]-{E[X(t)]}2.

Consider

0

22 ])([)(n

ntXPntXE

0

2 ])([][n

ntXPnnn

00

])([])([)1(nn

ntXnPntXPnn

1....)1(

)(43

)1(32

)1(

121

10

4

2

32

at

atx

at

atx

atx

at

at

1....1

101

61

31)1(

232

2

at

at

at

at

at

at

at


E[X2(t)]=2+2at-1

=3at-1

11

1)1(

23

2

at

at

at

11

1

)1(

23

2

at

atat

at

1)1()1(

2 3

2

at

at


Var[X(t)]=2at+1-1

Var[X(t)]=2at.

7. Show that the random process {X(t)} defined by. where A and ω are constants and θ is a uniform random variable over (0,2π) is wide sense stationary. Further, determine whether {X(t)} is mean ergodic, correlation ergodic.

)cos()( tAtX


Solution :

A random process {X(t)} is said to be WSS if it satisfies the following conditions:


2. The auto-correlation function RXX(t1,t2) is a function of the length of the time difference. i.e. RXX(t1,t2)= RXX(t1-t2)

3. E[X2(t)]<∞.


Consider the random process given by

where A and ω are constants and θ is uniform distributed in (0,2π). Hence, the p.d.f. of θ is given by

Consider

)cos()( tAtX

elsewhere

f

,0

20,2

1

)(

)cos()]([ tAEtXE


E[X(t)]=0, a constant with respect to time.

----------(1)

)cos( tAE

2

02

1)cos( dtA

2

0)sin(

2 t

A

ttA

sin2sin2


We next consider the auto-correlation function RXX(t,t+ )=E[X(t)X(t+ )]

)}{cos()cos( tAtAE

)cos()cos(2

2

ttttEA

)cos()22cos(2

2

tEA

)cos(2

)22cos(2

22

EA

tEA


Consider

------(2)

Substituting the above expression in RXX(t,t+ ), we have

)cos(2

)22cos(2

22

AtE

A

dttE 2

02

1)22cos()22cos(

2

02

)22sin(

2

1

t

0)2sin()22sin(4

1

tt


-----(3)

Consider E[X2(t)]= RXX(t,t)= RXX(0) <∞. Hence, the given random process is WSS.

Now, the random process {X(t)} is said to be mean ergodic if E[X(t)] = A[X(t)],

where

)cos(2

),(2

AttRXX

2

2A

T

T

dttxTT

lttXA )(

2

1)]([


Consider

= 0 (since │sinθ│≤1 for all θ) ----(4)

T

T

dttATT

lttXA )cos(

2

1)]([

T

T

dttTT

ltA )cos(

2

1

T

T

t

TT

ltA

)sin(

2

1

)sin()sin(2

1

TTTT

ltA


From equations (1) and (4), we see that E[X(t)] = A[X(t)].

Hence {X(t)} is mean ergodic.

The random process {X(t)} is said to be correlation ergodic if

E[X(t)X(t+ )] = A [X(t)X(t+ )] where

T

T

dttxtxTT

lttXtXA )()(

2

1)]()([


Consider

T

T

dttAtATT

lttXtXA )}{cos()cos(

2

1)]()([

T

T

dttttt

TT

ltA

2

)cos()cos(

2

12

T

T

T

T

dtTT

ltAdtt

TT

ltA)cos(

2

1

2)22cos(

2

1

2

22

T

T

T

T

dtTT

ltAt

TT

ltA)cos(

2

1

22

)22sin(

2

1

2

22


----(5)

From equations (2) and (5), we see that E[X(t)X(t+ )] = A [X(t)X(t+ )]

)(2

1

2

)cos(

2

)22sin()22sin(

4

2

2

TTTT

ltA

T

TT

T

ltA

,1sin1

2

)cos(0

2

T

ltA

)cos(2

),(2

AttAXX


Hence the given random process {X(t)} is correlation ergodic.

Note: Since the above process is WSS, it is first order stationary.

8. Show that the process

where A and B are uncorrelated random variables is wide sense stationary if

tBtAtX sincos)(


Solution : HW

9. Determine the mean and the variance of the random process where A and ω are constants and θ is uniformly distributed in

Solution : HW

22;0 BEAEBEAE

)cos()( tAtX

2,0


10. Two random processes X(t) and Y(t) are defined by and

. Show that X(t) and Y(t) are jointly wide sense stationary, if A and B are uncorrelated zero mean random variables having the same variance and is a constant.

Solution: HW

tBtAtX 00 sincos)( tAtBtY 00 sincos)(

0


11. If X(t) is a WSS process with autocorrelation function RXX( )=Ae-2│ │

where A is any constant , obtain the second order moment of the random variable X(8)-X(5).

Solution : HW

12. Given a random variable Y with characteristic function φ(ω)=E[eiYω] and a

a random process X(t)=Cos(λt+Y). Show


that X(t) is WSS if φ(1)=φ(2)=0.

Solution :

A random process {X(t)} is said to be WSS if it satisfies the following conditions:


2. The auto-correlation function RXX(t1,t2) is a function of the length of the time difference. i.e. RXX(t1,t2)= RXX(t1-t2)

3. E[X2(t)]<∞.


Since φ(ω) is the characteristic function of the random variable Y, we have

φ(ω)=E[eiYω] =E[cos (Yω) +i sin (Yω)]

Since φ(1)=0, we have

E[cos (Y) +i sin (Y)]=0

E[cos Y] + i E[sin Y]=0 (WHY??)

This implies that

E[cos Y] =0 and E[sin Y] =0 (WHY??)

-------(1)


Also, φ(2)=0 yields

E[cos (2Y) +i sin (2Y)]=0

E[cos 2Y] + i E[sin 2Y]=0

This implies that

E[cos 2Y] =0 and E[sin 2Y] =0

------(2)

Consider E[X(t)]=E[Cos(λt+Y)]

= E[cos λt cos Y – sin λt sin Y]

= cos λt E[cos Y] – sin λt E[sin Y]

= 0 (from (1)), a constant.


Now consider RYY(t, t+ )=E[X(t)X(t+ )]

Therefore,

RXX(t, t+ )= E[Cos(λt+Y) Cos(λ[t+ ]+Y)]

=E[{Cos({λt+Y}+{ λ[t+ ]+Y})

+Cos({λt+Y }-{λ[t+ ]+Y})}/2]

= E[cos(2λt+ λ +2Y)]/2+ E[cos(- λ )]/2 (why ??)

= E[cos (2λt+ λ ) cos 2Y- sin (2λt+ λ ) sin 2Y]/2

+ E[cos(λ )]/2 (why???)

= cos (2λt+ λ )E[cos 2Y]/2

- sin (2λt+ λ )E[sin 2Y]/2+ E[cos(λ )]/2


Hence RXX(t, t+ )= E[cos(λ )]/2 (from (2)), is exclusively a function of the length of the time interval and not the end points.

Further, E[X2(t)]= RXX(t,t)= E[cos(λx0)]/2

= 1/2 )]<∞.

Hence the given random process {X(t)} is WSS

ω θω θ

π π ππ


HW

13. Verify whether the random process X(t)=ACos(ωt+θ) is WSS given that A and ω are constants and θ is uniformly distributed in (i) (-π, π) (ii) (0, π) (iii) (0, π/2).


14. If X(t)= Y cos ωt + Z sin ωt, where Y and Z are two independent normal RVs with E[X]=E[Y]=0,E[X2]=E[Y2]= 2 and ω is a constant, prove that {X(t)} is a stationary process of order 2.

Solution: HW


15. Verify whether the random process X(t)=Ycos ωt where ω is a constant and Y is a uniformly distributed random variable in (0,1) is a strict sense stationary process.

Solution : HW

Sine Wave Processes

Definition: A random process {X(t)} of the form X(t) = A cos (ωt+θ) or X(t) = A sin(ωt+θ) where any non-empty combination of A, ω, θ are random variables is called a sine wave process.

A is called the amplitude

ω is called the frequency

θ is called the phase.

Random Processes

Orthogonal Processes:

Two random processes {X(t)} and {Y(t)} are said to be orthogonal if their cross correlation function RXY(t1,t2)=0.

Uncorrelated Processes:

Two random processes {X(t)} and {Y(t)} are said to be uncorrelated if their cross covariance function CXY(t1,t2)=0.

i.e. RXY(t1,t2)=E[X(t1)]E[Y(t2)]

Random Processes

Note: Two independent random processes are uncorrelated but the converse need not be true. (Can you give a Counter-example???)

Remark: If two random processes {X(t)} and {Y(t)} are statistically independent, then their cross-correlation function is given by RXY(t1,t2)=E[X(t1)]E[Y(t2)]

Random Processes

Remark:

If two random processes {X(t)} and {Y(t)} are at least WSS, then RXY(t1,t2)=XY, a constant with respect to time.

Normal/ Gaussian Process

Definition:

A random process {X(t)} is called a Normal/ Gaussian process if its nth order joint density function is given by

TX XXCXX

n

X

nnX eC

tttxxxf][][

2

1

2121

1

)2(

1,..,,;,..,,


where

CX is the covariance matrix given by

T

nn XX

XX

XX

XX

.

.

22

11

nnnn

n

n

X

CCC

CCC

CCC

C

..

........

..

..

21

22221

11211


where Ci,j=CXX(ti,tj) is the auto-covariance function of X(t). Also, is the transpose of the matrix where

= E[X(ti)].

Remarks:

1. A Gaussian process is completely determined by its mean and auto-covariance functions. (WHY???)

XX

TXX

iX


2. If a Gaussian process is WSS, then it is strictly stationary. (WHY???)

(Hint: If {X(t)} is WSS, then its mean is a constant with respect to time and its auto-covariance function is only a function of the length of the time interval and not on the end points.)


16. If {X(t)} is a Gaussian process with mean µ(t)=0 and the auto-covariance function CXX(t1,t2)=16e -│t

1-t2

│ . Find the probability that X(10)≤8 and │X(10)-X(8)│≤ 4.

Solution: Consider the Gaussian process with mean µ(t)=0 or E[X(t)]=0 and the auto-covariance function

CXX(t1,t2)=16e -│t1-t

2│ .


The random variable where ti

is any fixed time point follows a standard normal distribution.

),(

)]([)(

iiXX

ii

ttC

tXEtX

16

108

10,10

)]10([)10(8)10(

XXC

XEXPXP

]5.0[4

2

ZPZP

3085.06915.01


Consider the random variable X(10)-X(8) .

Then E[X(10)-X(8)]=E[X(10)]-E[X(8)]=0.

Var[X(10)-X(8)]

=Var[X(10)]+Var[X(8)]-2Cov[X(8),X(10)]

=16 + 16 – 2x16e–│8-10│

= 32 – 32 e-2

= 32(0.8646) = 27.6692


= 2P[(Z<0.76)-0.5]

= 2[0.7764-0.5]

= 0.5528

6602.27

044)8()10( ZPXXP

76.0 ZP

]76.00[2 ZP

Random Telegraph Process

Let {X(t)} be a random process satisfying the following conditions:

1. {X(t)} assumes only one of the 2 possible levels +1 or -1 at any time.

2. X(t) switches back and forth between its two levels randomly with time.

3. The number of level transitions in any time interval of length is a Poisson random variable. i.e. the probability of


exactly k transitions with average rate of transitions λ is given by

4. Transitions occurring in any time interval are statistically independent of the transitions in any other interval.

5. The levels at the start of any interval are equally probable.

X(t) is called a semi-random telegraph signal process.

,....2,1,0,!

)(

kk

ek


Alternately, if N(t) represents the number of occurrences of a specified event in (0,t), N(t) is Poisson with parameter λt and

Y(t)= (-1) N(t), then the random process Y(t) is called a semi-random telegraph signal process.

If Y(t) is a semi-random telegraph signal process, A is a random variable which is


independent of Y(t) and assumes values +1 and -1 with equal probability, then the ransom process {X(t)} defined by X(t)=AY(t) is called a random telegraph signal process

We now show that the random telegraph process defined above is a WSS process.

To show that X(t) is WSS, we need to prove the following:



2. The auto-correlation function RXX(t1,t2) is exclusively a function of the length of the time difference. i.e. RXX(t1,t2)= RXX(t1-t2)

3. E[X2(t)]<∞.

Consider E[X(t)] = E[AY(t)]


E[X(t)] = E[A]E[Y(t)] (A and Y(t) are independent).

Consider E[A] = (1) P(A=1) + (-1) P(A=-1)

= (1)(½) + (-1)(½)

= 0 -------------(1)

Now, E[A2] = (1)2 P(A=1) + (-1)2 P(A=-1)

= (1)(½) + (1)(½)

= 1 --------------(2)


By definition of Y(t), Y(t) takes the value +1 whenever the number of level transitions N(t) in the interval (0,t) is even and Y(t) assumes the value -1 whenever the number of level transitions N(t) in the interval (0,t) is odd.

Hence P[Y(t)=1] = P[N(t)= even]

= P[N(t)= 0] + P[N(t)= 2]

+ P[σ(t)= 4] + …..


P[Y(t)=1] = e–λt cosh (λt) -------(3)

....!4

)(

!2

)( 42

ee

e

....

!4

)(

!2

)(1

42 e

eee2

1


Also, P[Y(t)=-1] = P[N(t)= odd]

= P[N(t)= 1] + P[N(t)= 3]

+ P[σ(t)= 5] + …..

P[Y(t)= -1] = e–λt sinh (λt) -------(4)

.....!3

)(

!1

)( 31

ee

....

!3

)(

!1

)( 31 e


E[Y(t)] = (1) e–λt cosh (λt) + (-1) e–λt sinh (λt)

E[Y(t)] = e–2λt ------(5)

Substituting expressions (1) and (5) in E[X(t)], we obtain E[X(t)] = 0, which is a constant with respect to time.

22

ee

eee

e


Hence E[X(t)] is a constant with respect to time.

We next consider the auto-correlation function of X(t), viz,

RXX(t,t+ )= E[X(t)X(t+ )]

= E[AY(t) AY(t+ )]

= E[A2Y(t)Y(t+ )]


RXX(t,t+ )= E[A2]E[Y(t)Y(t+ )]

( A is independent of Y(t))

= (1) RYY(t,t+ )

We now compute RYY(t,t+ )= E[Y(t)Y(t+ )]

If Y(t) = 1, then Y(t+ ) = 1, if the number of level transitions in the interval (t, t+ ) is even.


Hence

P[Y(t, t+ ) = 1 │Y(t) = 1] = P[ number of level transitions in (t, t+ ) is even]

= e–λ cosh (λ )

P[Y(t) = 1 ,Y(t, t+ ) = 1]

= P[Y(t, t+ ) = 1 │Y(t) = 1] P[Y(t) = 1] = e–λ cosh (λ ) e–λt cosh (λt) -----(7a)


Similarly,

P[Y(t) = 1 ,Y(t, t+ ) = -1]

= e–λ sinh (λ ) e–λt cosh (λt) -----(7b)

P[Y(t) = -1 ,Y(t, t+ ) = 1]

= e–λ sinh (λ ) e–λt sinh (λt) -----(7c)

P[Y(t) = -1 ,Y(t, t+ ) = -1]

= e–λ cosh (λ ) e–λt sinh (λt) -----(7d)


Hence,

RYY(t,t+ )= (1)(1) e–λ cosh (λ ) e–λt cosh (λt)

+ (1)(-1) e–λ sinh (λ ) e–λt cosh (λt)

+ (-1)(1) e–λ sinh (λ ) e–λt sinh (λt)

+ (-1)(-1) cosh (λ ) e–λt sinh (λt)

= e–λ e–λt {cosh (λ ) cosh (λt)

- sinh (λ ) cosh (λt) - sinh (λ ) sinh (λt)

+ cosh (λ ) sinh (λt)}


RYY(t,t+ ) = e–λ e–λt {cosh (λ ) [cosh (λt)

+ sinh (λt)] - sinh (λ ) [cosh (λt) + sinh (λt)]}

= e–λ e–λt [cosh (λ ) - sinh (λ )] [cosh (λt) +

sinh (λt)]

= e–λ e–λt e–λ eλt

RYY(t,t+ ) = e–2λ -------(8)

Substituting equation (8) in

RXX(t,t+ ) = RYY(t,t+ ) we have


RXX(t,t+ ) = e–2λ , which is exclusively a function of .

Further E[X2(t)] = RXX(t,t) = 1 < ∞. Hence, the random telegraph process X(t)

is a WSS process.


Remark:

From equations (3), (4), (5) and (8) we see that even though the auto-correlation function of the semi-random telegraph signal {Y(t)} is exclusively a function of , since the first order p.m.f is itself a function of , {Y(t)} is not even a first order stationary process. It is an evolutionary process.

Markov Chains

Discrete-Time Markov Chains:

Without loss of generality, assume that the parameter space T={0,1,2,,….}. Hence the state of the system is observed at the time points 0,1,2,…… These observations are denoted by X0,X1,X2,…

If Xn=j, then the state of the system at time step ‘n’ is said to be ‘j’.

Let pj(n)=P[Xn=j] denote the probability that Xn is in state j.

Markov Chains

In other words, pj(n) denotes the p.m.f. of the random variable Xn.

The conditional p.m.f.

pjk(m,n)= P[Xn=k│Xm=j] is called the transition p.m.f.

Markov Chains

Discrete-Time Markov Chains:

Let {Xn} be a discrete-time integer valued Markov chain (starting at n=0) with initial PMF

The joint PMF for the first n+1 values of the process is

,...2,1,0j],jX[P)0(p 0j

]iX[PiXiXP]...iX/iX[PiX,..,iX,iXP 0000111n1nnn001n1nnn

Discrete-Time Markov Chain

If the one-step state transition probabilities are fixed and do not change with time, i.e.

{Xn} is said to have homogeneous transition probabilities.

A Markov chain is said to be a homogeneous Markov chain if the transition p.m.f. pjk(m,n) depends only on the difference n-m.

np]iX/jX[P ijn1n


The homogeneous state transition probability satisfies the following conditions:

since the states are mutually exclusive and collectively exhaustive

n,..,2,1i,1p

1p0

j

ij

ij

Discrete-Time Markov Chain The transition probability matrix P : The one- step transition probabilities of a discrete

parameter Markov Chain are completely specified in the form of a transition probability matrix (t.p.m.) given by P=[pij]

The row sum is 1 for each row of P – Stochastic matrix

All the entries lie in [0,1]

The stochastic matrix is said to be doubly stochastic iff all the column entries add up to 1 for every column.

A stochastic matrix is said to be regular if all its entries are positive.

.

.ppp

......

...ppp

...ppp

P

2i1i0i

121110

020100


The joint PMF of Xn , Xn-1 ,…, X0 is given by

Thus {Xn} is completely specified by the initial PMF and the matrix of the one-step transition probabilities P

)0(pp...piX,..,iX,iXP0,101n ii,ini,i001n1nnn

• Two aging computers are used for word processing.

• When both are working in morning, there is a 30% chance that one will fail by the evening and a 10% chance that both will fail.

• If only one computer is working at the beginning of the day, there is a 20% chance that it will fail by the close of business.

• If neither is working in the morning, the office sends all work to a typing service.

• Computers that fail during the day are picked up the following morning, repaired, and then returned the next morning.

• The system is observed after the repaired computers have been returned and before any new failures occur.

Discrete-Time MC Computer Repair Example

States for Computer Repair Example

Index State State definitions

0

s = (0)

No computer has failed. The office

starts the day with both computers

functioning properly.

1

s = (1)

One computer has failed. The

office starts the day with one

working computer and the other in

the shop until the next morning.

2

s = (2)

Both computers have failed. All

work must be sent out for the day.

Events and Probabilities for Computer Repair Index

Current state

Events

Probability

Next state

0 s0 = (0)

Neither computer fails. 0.6 s' = (0)

One computer fails. 0.3 s' = (1)

Both computers fail. 0.1

s' = (2)

1 s1 = (1) Remaining computer does not fail and the other is returned.

0.8 s' = (0)

Remaining computer fails and the other is returned.

0.2 s' = (1)

2 s2 = (2) Both computers are returned.

1.0 s' = (2)

State-Transition Matrix and Network

The events associated with a Markov chain can be described by the m m matrix: P = (pij).

For computer repair example, we have:

001

02.08.0

1.03.06.0

P

State-Transition Matrix and Network

State-Transition Network

– Node for each state

– Arc from node i to node j if pij > 0.

For computer repair example:

2

0

1

(1)

(0.6)

(0.3)(0.1)

(0.2)

(0.8)


The n-step transition probabilities :

Let P(n)=[pij(n)] be the matrix of n-step transition probabilities, where pij(n)=P[Xn+k=j/ Xk=i] = P[Xn=j/X0=i] for all n≥0 and k≥0, since the transition probabilities do not depend on time.

Then P(n)=Pn


Consider P(2)

)1(p)1(p]iX/kX[P]kX/jX[P

]iX[P

]iX[P]iX/kX[P]kX/jX[P

]iX[P

]iX,kX,jX[P]iX/kX,jX[P

kjik0112

0

00112

0

012

012

k

kjikij j,i)1(p)1(p)2(p


P(2)=P(1)P(1)=P2

P(n)=P(n-1)P

=P(n-2)PP

=P(n-2)P2

=Pn

Hence the n-step transition probability matrix is the nth power of the one-step transition probability matrix.


Chapman- Kolmogorov Equations:

Interpretation: RHS is the probability of going from i to k in r steps &

then going from k to j in the remaining n r steps, summed over all possible intermediate states k.

k

kjikij )rn(p)r(p)n(p


State Probabilities:

The state probabilities at time n are given by

the row vector pj(n)={pj(n)}. Now,

Therefore p(n)=p(n-1)P.

Similarly, p(n)=p(0)P(n)=p(0)Pn , n=1,2,…

Hence the state PMF at time n is obtained by

multiplying the initial PMF by Pn.

1npp]iX[P]iX/jX[P)n(p i

i

ij1n

i

1nnj


Limiting State/ Steady-state Probabilities:

Let {Xn} be a discrete-time Markov chain with N

states

P[Xn=j] - the probability that the process is in

state j at the end of the first n transitions,

j=1,2,..,N.

Then

)(][)(][

1

0 npiXPnpjXP ij

N

k

jn

Discrete-Time Markov Chain - Steady-state

Probabilities

As n→∞, the n- step transition probability pij(n)

does not depend on i, which means that

P[Xn=j] approaches a constant as n→∞. The limiting- state probabilities are defined as

Since,

NjjXP jnn

,...,2,1,][lim

kj

k

ikij pnpnp 1)(

k

kjkkj

k

ikn

ijn

ppnpnp 1lim)(lim

Discrete-Time Markov Chain - Steady-state

Probabilities

Defining the steady-state/ limiting state

probability vector , we have

The last equation is due to the law of total probability.

The probability πj is interpreted as the long

proportion of time that the MC spends in state j.

N ,...,, 21

j

j

kj

k

kj

P

p

1


Classification of States:

A state j is said to be accessible from state i (j can be reached from i) if, starting from state i, it is possible that the process will ever enter state j.

pij(n)>0 for some n>0.

Two states that are accessible from each other are said to communicate with each other.

Discrete-Time Markov Chain - Classification of

States

Communication induces a partition of states

States that communicate belong to the same class

All members of a class communicate with each other.

If a class is not accessible from any state outside the class, the class is said to be a closed communicating class.


States

A Markov chain in which all the states communicate is called an irreducible Markov Chain.

In an irreducible Markov chain, there is only one class.


States

States that the process enters infinitely often and states that the process enters finitely often.

Process will be found in those states that it enters infinitely often


States

Probability of first passage from state i to state j in n transitions - fij(n)

The conditional probability that given that the process is in state i, the first time the process enters state j occurs in exactly n transitions.


States

Probability of first passage from state i to

state j – fij

Conditional probability that the process will ever enter state j given that it was initially in

state i

1n

ijij nff


States

Clearly

and

fii denotes the probability that a process that

starts at state i will ever return to state I

If fii =1, then state i is called a recurrent state.

If fii <1, then state i is called a transient state.

ijij pf )1(

jl

ljilij nfpnf )1()(


States

State j is called

• transient (non-recurrent) if there is a positive probability that the process will never return to j again if it leaves j

• recurrent (persistent) if with probability 1, the process will eventually return to j after it leaves j

A set of recurrent states forms a single chain if every member of the set communicates

with all the members of the set.


States

Recurrent state j is called a • periodic state if there exists an integer d,

d>1, such that pjj(n) is zero for all values of n other than d, 2d, 3d,…; d is called the period. If d=1, j is called aperiodic.

• positive recurrent state if, starting at state j the expected time until the process returns to state j is finite; otherwise it is called a null-recurrent state.


States

Positive recurrent states are called ergodic states

A chain consisting of ergodic states is called an ergodic chain.

A state j is called an absorbing (trapping) state if pij=1. Thus, once the process enters an absorbing/ trapping state, it never leaves the state.


States

States

Recurrent/ persistent Transient/ Non - recurrent

Positive recurrent Null Recurrent Periodic Aperiodic

Ergodic Ergodic

Discrete-Time Markov Chain - Classification of States

• If a Markov chain is irreducible, all its states are of the same type. They are either all transient or all null persistent or all non-null persistent.

• Further, all the states are either aperiodic or periodic with the same period.

• If a Markov chain is finite and irreducible, then all its states are non-null persistent.

Continuous-Time Markov Chain

A random process {X(t)/t≥0} is a continuous-time Markov chain if, for all s,t≥0 and nonnegative integers i,j,k,

In a continuous –time Markov chain, the conditional

probability of the future state at time t+s, given the present state at s and all past states depends only on the present state and not on the past.

isXjstXPsukuXisXjstXP )()(]0,)(,)()([


If in addition, P[X(t+s)=j/X(s)=i] is independent of s, then the process {X(t),t≥0} is said to be time-homogeneous or have the time-homogeneity property. Time-homogeneous Markov chains have stationary (or homogeneous) transition probabilities.

Let j)t(XP)t(p

i)s(Xj)st(XP)t(p

j

ij


In other words, pij(t) is the probability that a MC presently in state i will be in state j after an additional time t and pj(t) is the probability that a MC is in state j at time t.

The transition probabilities satisfy

Further

1)(

1)(0

j

ij

ij

tp

tp

1)( j

j tp


Chapman-Kolmogorov equation

sptpstp ks

k

ikij

Markov Chains 17. A man either drives a car or catches a

train to go to office each day. He never goes 2 days in a row by train, but if he drives one day , then the next day he is just as likely to drive again as he is to travel by train. Now suppose that on the first day of the week, the man tossed a fair die and drove to work if and only if a 6 appeared, find (i) the probability that he takes a train on the third day and (ii) the probability that he drives to work in the long run.

Markov Chains

The travel pattern forms a Markov chain, with

state space = (train,car)

The TPM of the chain is given by

The initial state probability distribution is given

by , since

P(traveling by car)=P(getting a 6 in the toss of the die)= 1/6.

Also, P(traveling by train)= 5/6

2

1

2

110

P

6

1

6

5)0(p

Markov Chains Now,

Therefore,

P(the man travels by train on the third day) = 11/24

Let π = (π1, π2) be the limiting form of the state probability distribution or stationary

12

11

12

1

2

1

2

110

6

1

6

5)1()2(Ppp

24

13

24

11

2

1

2

110

12

11

12

1)2()3(Ppp

Markov Chains

state distribution of the Markov chain.

By property of π, πP= π

-----(1) and

------(2)

Equations (1) and (2) are the same.

Alongwith the equation π1+ π2 = 1, ---(3)

),(

2

1

2

110

),( 2121

122

1

212

1

Markov Chains

since π is a probability distribution, we obtain

Hence π1=1/3, π2 = 2/3.

Therefore P[man travels by car in the long run]= 2/3

22 12

1

12

32

3

22

3

1

3

211

Markov Chains

18. Three boys A,B and C are throwing a ball to each other. A always throws the ball to B and B always throws the ball to C, but C is just as likely to throw the ball to B as to A. Show that the process is Markovian. Find the transition probability matrix and classify the states.

Solution: Let A,B,C denote the states of the Markov chain.

Markov Chains

The transition probability matrix of {Xn} is given by

Since, the states of Xn depend only on Xn-1 and not on Xn-2, Xn-3, …, the process {Xn} is a Markov chain.

We first observe that the chain is finite.

(Draw the tpm/ network)

02

1

2

1100

010

P

Markov Chains We observe that all the states

communicate with each other

Hence, the MC is irreducible

Since the MC is finite, all the states are positive recurrent.

Further since state A is aperiodic (WHY???) all the states are aperiodic, ergodic

Markov Chains

19. The transition probability matrix of a Markov chain {Xn} with three states

1,2 and 3 is and the

initial distribution is . Find

(i) (ii) .

Solution: Consider

3.04.03.0

2.02.06.0

4.05.01.0

P)1.0,2.0,7.0()0( p

)1.0,2.0,7.0()0( p

32 XP 2,3,3,2 0123 XXXXP

Markov Chains

.

][ 2)2(pP

3.04.03.0

2.02.06.0

4.05.01.0

3.04.03.0

2.02.06.0

4.05.01.0

29.035.036.0

34.042.024.0

26.031.043.0

32 XP iXPiXXPi

0

3

1

02 3

313123113 002002002 XPXXPXPXXPXPXXP

]3[]2[]1[ 0

2

330

2

230

2

13 XPpXPpXPp

Markov Chains P[X2=3]

= 0.182 + 0.068 + 0.029 =0.279

(ii)

(conditional probability)

(Markov property)

= 0.4x0.3x0.2x0.2= 0.0048

1.029.02.034.07.026.0

2,3,3,2 0123 XXXXP

2,3,32,3,32 0120123 XXXPXXXXP

2,32,3332 0101223 XXPXXXPXXP

2.. 0

)1(

23

)1(

33

)1(

32 XPppp

Markov Chains

20. A gambler has Rs.2. He bets Re. 1 at a time and wins Re. 1 with probability 0.5. He stops playing if he loses Rs. 2 or wins Rs. 4. What is the transition probability matrix of the related Markov chain?

Solution : HW

Markov Chains

21. There are 2 white marbles in urn A and 3 red marbles in urn B. At each step of the process, a marble is selected from each urn and the 2 marbles selected are interchanged. Let the state ai of the system be the number of red marbles in A after i changes. What is the probability that there are 2 red marbles in A after 3 steps? In the long run, what is the probability that there are 2 red marbles in urn A?

Solution : HW

Markov Chains

22.Find the nature of the states of the Markov chain with the TPM,

P =

Solution : HW

010

2

10

2

1

010

Thank You

RANDOM PROCESSES P… · Random Processes The values assumed by the random variables X(t) are called the states and the ... Wide-sense Stationary Process A random process {X(t)} ...

Documents