Lecture 10: Examples of Discrete Time Markov
Chain
Parimal Parag
1
1.1
Discrete Time Markov Chains Contd.
Example 4.3(c) The Age of Renewal Process:
Consider a discret time slotted system. An item is put into use. When the
item fails, it is replaced at the beginning of the next time period. Pi is the
probability of the failure of the item in P
ith time period. Assume that the life
times are independent. Pi is aperiodic.
iPi < ∞. The age of an item is the
number of periods the item is in use. Let us denote the age of the item at time
n as Xn . Let us denote
Pi
λ(i) , P∞
j=i
Pj
λ(i) is the probability that an i unit old item fails. {Xn } is a Markov chain
with transition probability,
λ(i)
1 − λ(i)
P (Xn = in |Xn−1 = i) =
: in = 1
: in = i + 1,
for i ≥ 1. To find the equilibrium distribution of this process we solve for
π = πP .
X
π1 =
πi λ(i)
i
πi+1 =πi (1 − λ(i)).
We can solve the above set of equations
iteratively to obtain πi+1 = π1 (1 −
P∞
λ(1))(1 − λ(2)) . . . (1 − λ(i)) = π1 j=i+1 Pj = π1 P (Y ≥ i + 1), where Y is the
P
P (Y ≥i)
1
life time of an item. Using j πj = 1, we get π1 = E[Y
] and πi = E[Y ] .
1.2
Example 4.3(D) Time slotted system:
Consider a time slotted system and in each slot every member of a population
dies with probability p i.i.d. In each time slot, the number of new members that
join the population is according to a Poisson (λ) process. Let Xn denote the
number of members of the population at the beginning of time period n. Observe
that {Xn } is a Markov chain. We are interested in computing the stationary
1
distribution of the Markov chain. To that end, let X0 be a Poisson random
variable with mean α. Then X0 individuals will be alive at the beginning of slot
1 with probability (1 − p). Since the number of new members in the system will
be a Poisson random variable with parameter λ, X1 is distributed Poisson with
parameter α(1 − p) + λ. If α = α(1 − p) + λ, the chain would be stationary.
Hence by the uniqueness of the stationary distribution is Poisson with mean λp .
1.3
Example 4.3(E)Gibbs sampler:
Gibbs sampler is used to generate values of a random vector X1 , X2 . . . Xn where
it as such difficult to generate values from the mass function PX1 ,X2 ...Xn (.)
The idea is to form a Markov chain whose stationary distribution is given by
PX1 ,X2 ...Xn (.) The Gibbs sampler works as follows: Let X0 = (x01 , x02 . . . x0n )
for which p(x0! , x02 . . . x0n ) > 0. Now generate x1k for k = 1, 2 . . . n according
, . . . xk−1
to PXk |Xj0 =x0j , j6=k (.). Similarly generate Xk given Xk−1 = (xk−1
n ).
1
Observe the Xj , j ≥ 0 is a Markov chain. It is easy to see that PX1 ,X2 ...Xn is
the stationary probability distribution of the Markov chain.
Consider an irreducible positive recurrent Markov chain with stationary
probability {π(i) : i ∈ N0 }. Let N denote the number of transitions between
successive visits to state 0. Visits to state 0 constitutes renewal instants. Then,
the number of visits by time n to state 0, for large n is approximately normally
distributed. i.e.
n
n
nV ar(N )
1 X
1X =0 → N (
,
).
n m=1 m
E[N ] E[N ]3
Note that n/E[N ] = nπ0 and nV ar(N )/(E[N ])3 = nV ar(N )π03 . Since
1
2
V ar(N ) = E[N 2 ] − π+0
2 , we need to find E[N ]. Let Tn be the number of
Pn
transitions from n until next visit to state 0. Note that, n1 m=0 Tm . If N is
the number of transitions between successive visits to state 0, average reward
N (N +1)
2
P
E[
]
]
of the process per unit time= E[N2 ]
= 1 + 21 E[N
i∈N0 π(i)µi,0 . Here
E[N ] =
P 2
µi,0 = E[Tn |Xn = i]. Thus µi,0 = 1 + j6=0 Pi,j µj,0 .
1.4
Transition Among Classes and Mean Times in Transient States
Proposition 1. Let R be a recurrent class of states. If i ∈ R, j 6= R, then
Pij = 0.
Proof. Suppose Pi j > 0. Then, as i and j do not communicate with each other,
n
Pji
= 0, ∀n. Hence, starting from state i, there is a positive probability of at
least Pij that the process will never return to state i. This contradicts the fact
that i is recurrent. Hence, Pij = 0.
Let T denote the set of all transient states. Let j be recurrent, i ∈ T . fij
denote the probability of eventually hitting j starting from i = Pi (Tj < ∞).
Proposition 2. If j is recurrent, then the set of probabilities {fij : i ∈ T }
satisfies
X
X
fij =
Pik fkj +
Pik ,
k∈T
k∈K
2
where R denotes set of states communicating with j.
Proof.
fij =
X
Pi (Ti < ∞, X1 = k)
k
= Pi (Tj < ∞, X1 ∈ T ) + Pi (Tj < ∞, X1 ∈ R)
X
X
=
Pk (Tj < ∞)pik +
pik
k∈T
=
X
k∈R
fkj pik +
k∈T
1.5
X
pik .
k∈R
Gambler’s Ruin Problem
Consider a gambler who at each play of the game has p = P (win) = 1 −
P (loss) = 1 − q. Assume successive plays of the game are independent. We
are interested in knowing the probability that starting with i units the gambler
hits fortune N before hitting 0. Let Xn be the player’s fortune at time n.
{Xn } is a Markov chain. Observe that P00 = 1. PN N = 1. Also observe that
Pi,i+1 = p = 1−Pi,i−1 , i = 2, 3 . . . N −1. Let P (Xn+1 = j|Xn = i). The Markov
chain has three classes {0}, {N }, {1, 2 . . . N − 1}, the first two are recurrent and
the last one is transient. Let fi ≡ fiN . We have, from the previous proposition,
fi = pfi+1 + (1 − p)fi−1 , f0 = 0, fN = 1. Since p + q = 1, we can observe that
fi+1 − fi = pq (fi − fi−1 ). Since f0 = 0, we can write a recursion and observe
that
(
q i
(1−( p
) )
: p 6= 12
q N
1−(
fi =
p)
i
: p = 12 .
N
As N → ∞
fi =
(1 − ( pq )i )
0
: p > 12
: p ≤ 21 .
Consider a finite state Markov chain with transient states with T = {1, hdotst}.
Let Q be the associated transition matrix. Let mij denote the expected total
number of time periods spent if state j stating form state i.
X
mij = E[
1Xn =j |X0 = i]
n∈N
= 1i = j +
X
Pik mkj .
k∈T
[M ]ij = (I)ij + (QM )ij , M = [I − Q]−i . The matrix inverse exists.
3