ST3236: Stochastic Process Tutorial 3 TA: Mar Choong Hock Email: [email protected] Exercises: 4.

ST3236: Stochastic ProcessTutorial 3

TA: Mar Choong Hock

Email: [email protected]

Exercises: 4

Question 1A markov chain X0,X1,… on state 0, 1, 2 has the transitionprobability matrix

and initial distributions, p0 = P(X0 = 0) = 0.3, p1 = P(X0 = 1) = 0.4 and p2 = P(X0 = 2) = 0.3.

Determine P(X0 = 0, X1 = 1, X2 = 2) and draw the state-diagram with transition probability.

Question 1

P(X0 = 0,X1 = 1,X2 = 2) = P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X0 = 0,X1 = 1)= P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X1 = 1)= p0 x p01 x p12

= 0.3 x 0.2 x 0= 0.

0

1 2

p00=0.1

p11=0.1 p22=0.1

Note: pij = P(Xn=j|Xn-1=i)

p01=0.2

p02=0.7

p20=0.1

p21=0.8

p10=0.9

Question 2A markov chain X0,X1,… on state 0, 1, 2 has the transition probability matrix

Determine the conditional probabilitiesP(X2 = 1,X3 = 1|X1 = 0) and P(X1 = 1,X2 = 1|X0 = 0).

Question 2P(X2 = 1, X3 = 1 | X1 = 0) = P(X2 = 1 | X1 = 1)P(X3 = 1 | X1 = 0, X2 = 1)= P(X2 = 1 | X1 = 0)P(X3 = 1 | X2 = 1)= p01 x p11

= 0.2 x 0.6= 0.12

Similarly (or by stationarity),P(X1 = 1, X2 = 1 | X0 = 0) = 0.12

In general,P(Xn+1 = 1, Xn+2 = 1 | Xn = 0) = 0.12 for any n.That is, it doesn’t matter when you start.

Question 3A markov chain X0,X1,… on state 0, 1, 2 has the transitionprobability matrix

If we know that the process starts in state X0 = 1, determine probability P(X0 = 1,X1 = 0,X2 = 2).

Question 3

P(X0 = 1,X1 = 0,X2 = 2) = P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X0 = 1,X1 = 0)= P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X1 = 0)= p1 x p10 x p02

= 1 x 0.3 x 0.1= 0.03

Question 4A markov chain X0,X1,… on state 0, 1, 2 has the transitionprobability matrix

Question 4a

Compute the two-step transition matrix P(2).

Note: Observe that the rows must always sum to one for all transition matrices.

Question 4b

What is P(X3 = 1|X1 = 0)?

P(X3 = 1|X1 = 0) = 0.13

In general,P(Xn+2 = 1 | Xn = 0) = 0.13 for any n.

Question 4cWhat is P(X3 = 1|X0 = 0)?Note that:

Thus,P(X3 = 1|X0 = 0) = 0.16

In general,P(Xn+3 = 1 | Xn = 0) = 0.16 for any n.

Question 5A markov chain X0,X1,… on state 0, 1, 2 has the transitionprobability matrix

It is known that the process starts in state X0 = 1, determineprobability P(X2 = 2).

Question 5Note that:

P(X2 = 2) = P(X0 = 0) x P(X2 = 2 | X0 = 0)+P(X0 = 1) x P(X2 = 2 | X0 = 1)+P(X0 = 2) x P(X2 = 2 | X0 = 2)= p0p02 + p1p12 + p2p22

= 1 x p12 = 0.35

Question 6 • Consider a sequence of items from a production process, with each item being graded as good or defective.

• Suppose that a good item is followed by another good item with probability and by a defective item with probability 1-.

• Similarly, a defective item is followed by another defective item with probability and by a good item with probability 1-.

• Specify the transition probability matrix.

• If the first item is good, what is the probability that the first defective item to appear is the fifth item?

Question 6 Let Xn be the grade of the nth product.P(Xn+1 = g | Xn = g) = , P(Xn+1 = d | Xn = g) = 1 -P(Xn+1 = d | Xn = d) = , P(Xn+1 = g | Xn = d) = 1 -

Thus, the transition probability matrix is

Question 6 The probability is,P(X5 = d,X4 = g,X3 = g,X2 = g | X1 = g)= P(X2 = g | X1 = g) x P(X3 = g | X2 = g) x P(X4 = g | X3 = g) x P(X5 = d | X4 = g) (why?)= pgg x pgg x pgg x pgd

= 3(1 -)

g d

pgg= pdd= pdg=1-

pgd=1-

Question 7The random variables 1, 2, ... are independent and with

common probability mass function

Set X0 = 0 and let Xn = max {1, 2, ... }.

• Determine the transition probability matrix for the MC {Xn}.

• Draw the state-diagram associated with transition probability

Question 7Observe:

X0 = 0,

X1 = max {X0, 1},

X2 = max {X1, 2},

…

Xn= max {Xn-1, n}

Hence, Xn recursively compares the previous

maximum and the current input to obtain the new

maximum.

Question 7The state space is S = {0, 1, 2, 3}P(Xn+1 = 0 | Xn = 0) = P(n+1 = 0)= 0.1P(Xn+1 = 1 | Xn = 0) = P(n+1 = 1)= 0.3P(Xn+1 = 2 | Xn = 0) = 0.2P(Xn+1 = 3 | Xn = 0) = 0.4P(Xn+1 = 1 | Xn = 1) = P(n+1 = 0) + P(n+1 = 1) = 0.1 + 0.3 = 0.4P(Xn+1 = 2 | Xn = 1) = 0.2…P(Xn+1 = 2 | Xn = 2) = 0.1 + 0.3 + 0.2 = 0.6…P(Xn+1 = 3 | Xn = 3) = 0.1 + 0.3 + 0.2 + 0.4 = 1…P(Xn+1 = j | Xn = i) = 0 if j < i. (Cannot Happen!)

Question 7The transition probability matrix is

1 2p00=0.1 p11=0.4 p22=0.6

Note: pij = P(Xn=j|Xn-1=i)

p01=0.3

p02=0.2

30p33=1

p03=0.4

p12=0.2

p13=0.4

p23=0.4