1. science
2. mathematics
3. arithmetic

Math 5652: Introduction to Stochastic Processes
Homework 5 Solutions
You are welcome and encouraged to discuss the problems with your classmates; please write
up your own solutions, and indicate collaborators on your write-up.
(1) (10 points) (Durrett 3.13) In the Duke vs. Miami football game, possessions alternate
between Duke who has the ball for an average of 2 min and Miami who has the ball for
an average of 6 min.
(a) In the long run, what fraction of time does Duke have the ball?
(b) Suppose that on each possession Duke scores a touchdown with probability 1/4,
while Miami scores with probability 5/6 (note: not 1 as in the book, I don’t believe
Miami is that good). On average how many touchdowns will each team score per
hour?
Note: you can (but don’t have to) solve this problem using the theory of continuous-time
Markov chains.
Solution: Let’s first solve this by renewal processes: this is naturally modeled as an alternating renewal process, with states “ball passed to Duke” and “ball passed to Miami”.
The long-term fraction of time Duke has the ball is therefore
1
2min
= .
2min + 6min
4
What are the assumptions we made here? We’re assuming that times of possession by
Duke are iid, times of possession by Miami are iid, and that the Duke times are independent of the Miami times. All of these are subject to some suspicion, but fortunately
the law of large numbers result we’re using here can take quite a bit of battering with
dependencies.
If we assumed that the team holds the ball for independent, exponentially distributed
lengths of time with those means, we could also set up a CTMC with transition rate
matrix
D
M
1
1
D − 2min
2min
Q=
1
1
M
− 6min
6min
and solve for the invariant distribution: − 12 πD + 16 πM = 0, πD + πM = 1 so πD = 14 is the
proportion of time Duke has the ball. Note that the rates in the rate matrix are inverses
of the mean times of possession: rate is measured in inverse-time units.
For part (b), the number of times Duke has the ball during the course of an hour is
1
4
· 1hr
= 7.5.
2min
This is the number of cycles the alternating renewal process goes through (so we could
have divided 1 hour by the cycle time of 8 minutes). In particular, Miami also gets the
ball 7.5 times in an hour. Note, the number of times Duke has the ball in an hour is
1
actually a random quantity, but this should be its mean. Since each time the probability
that they score is 1/4 for Duke (and 5/6 for Miami), the mean number of scores is
7.5 ·
5
1
= 1.875 for Duke; 7.5 · = 6.25 for Miami.
4
6
We’re using here the fact that the mean of the sum of a random number of iid random
variables is the product of the means (this appeared in Section 2.3 in Durrett, or on our
Homework 3 (problem 3).
(2) (10 points) (Durrett 4.2) A small computer store has room to display up to three computers for sale. Customers come at times of a Poisson process with rate 2 per week to
buy a computer, and will buy one if at least one is available. When the store has only
one computer left, it places an order for two more computers, but the order takes an
exponentially distributed time with mean 1 week to arrive. While the store is waiting
for delivery, sales may reduce the inventory to 0.
(a) Write down the matrix of transition rates Qij and solve πQ = 0 to find the stationary
distribution.
(b) At what rate (number per week) does the store make sales?
Solution:
(a) This is a rare example of a chain where 2 computers arrive at once, so if we index
states by the number of computers in the store, the state can change by 2.
Recall that to determine the transition rate, you look at the time until a particular
transition would occur if the other transitions weren’t in the system at all. Transitioning down from n to n − 1 computers in the store happens at times of a Poisson
process with rate 2/week, so the time until that transition would happen (if no
other possibilities were available) is exponential with rate 2/week. This 2/week is
the rate q(n, n − 1) for n = 3, 2, 1.
For transitioning up, if we’re in state 1, then we’ve just ordered two more computers.
If no other possibilities were there, the time until the two computers arrive would
be exponential with mean 1 week, hence rate 1/week. This is the rate q(1, 3).
If we’re in state 0, it means that we’ve ordered two more computers but they haven’t
arrived yet. Since the time until their arrival is exponential, the remaining time is
still exponential with mean 1 week, hence rate 1/week; this is the rate q(0, 2).
Here’s the picture:
2/week 2/week 2/week
0
1
1/week
2
3
1/week
2
Consequently, the transition rate matrix is
1
2
3 
 0
0 −1 0
1
0

1 2 −3 0
1 

Q= 
2 0
2 −2 0 
3 0
0
2 −2
where the rates are measured in inverse weeks.
Solving for the stationary distribution means solving the equations
− π(0) + 2π(1) = 0
− 3π(1) + 2π(2) = 0
π(1) − 2π(3) = 0
π(0) + . . . + π(3) = 1.
Note that I can omit the equation that corresponds to one of the columns of the
matrix; I’ve omitted the one that involves three variables. The solution comes out
to
π = (0.4, 0.2, 0.3, 0.1).
(b) The store makes sales whenever a customer walks in and finds the store not in state
0. By the PASTA property, the probability that the customer finds the store not
in state 0 is 1 − π(0) = 0.6. Now, the process of sales will not be a Poisson process
(because this thinning is not independent across customers), but still, the long-term
rate at which sales happen is 2/week · 0.6 = 1.2/week.
Another way to compute this number is to observe that the rate of making sales is
equal to the rate at which the store orders new computers. Orders for new computers
get placed every time you visit state 1. From the stationary distribution (of this
irreducible, finite-state CTMC), in a week you spend 0.2 weeks in state 1. Every
visit to state 1 lasts a time with mean 1/3 week, so the average number of visits to
state 1 in a week is 0.2/(1/3) = 0.6. On each visit, you order 2 computers, so the
average number of computers you order in a week is 0.6 · 2 = 1.2.
(3) (10 points) (Durrett 4.7) Two people who prepare tax forms are working in a store at a
locall mall. Each has a chair next to his desk where customers can sit and be served. In
addition, there is one chair where customers can sit and wait. Customers arrive at rate
λ, but will go away if there is someone already sitting in the waiting chair. Suppose that
tax preparer i requires an exponential amount of time with rate µi , and that when both
of them are free, an arriving customer is equally likely to choose either one.
(a) Formulate a Markov chain model for this system with 5 states, corresponding to
which tax preparers are working and whether the waiting chair is occupied. Carefully
state what your states represent.
(b) Consider the special case of λ = 2, µ1 = 3, µ2 = 5 (note: not µ2 = 3 as in the
book). Find the stationary distribution. What is the proportion of time when both
3
tax preparers are working? (Hint: this should correspond to two states of your
chain.)
Solution:
(a) The 5 states we need are: 0 (system empty); 1A and 1B for one tax preparer working
(it matters which one is working, because the rates at which they fill out forms are
different); 2 (both tax preparers working; and 3 (both working, plus someone is
waiting in the chair).
Here’s the picture of the transitions. I’m assuming tax preparer A works at rate
µ1 , and B works at rate µ2 . Note that the arrival rate (which increases the number
of customers) is λ in all states (it’s split up into two λ/2 transitions from state 0);
the service rate from state3 is µ1 + µ2 and goes into state 2 (the customer who was
waiting goes into service), while from state 2 the total service rate is µ1 + µ2 but
it’s split up into µ2 (into state 1A, B done and A working) and µ1 (into 1B, A done
and B working).
λ
λ/2
1A
µ1
0
µ2
λ/2
λ
µ2
µ1
1B
2
λ
3
µ1 + µ2
(b) The proportion of time when both tax preparers are busy is π(2) + π(3). The
stationary distribution here solves
π(0) · λ = π(1A) · µ1 + π(1B) · µ2
π(1A) · (λ + µ1 ) = π(2) · µ2 + π(0) · λ/2
π(1B) · (λ + µ2 ) = π(2) · µ1 + π(0) · λ/2
π(2) · (λ + µ1 + µ2 ) = π(1A) · λ + π(1B) · λ + π(3) · (µ1 + µ2 )
π(3) · (µ1 + µ2 ) = π(2) · λ
π(0) + π(1A) + π(1B) + π(2) + π(3) = 1.
We didn’t have to write out all the top five equations – only four of them are
independent, so if you were solving the system by hand you would drop one of
them.
We could have tried for detailed balance here, but we’d fail. We know that it’s
unlikely to succeed
P from the fact that we get 5 detailed balance equations, plus the
normalization
π(i) = 1, which gives 6 equations in 5 unknowns. (In detailed
balance, the equations you get aren’t linearly dependent.)
The solution (which I obtain by asking a computer) is
π=(
30 10 6 4 1
,
,
,
,
).
51 51 51 51 51
4
Note that it doesn’t satisfy detailed balance, for example π(0) · λ/2 6= π(1A) · µ1 .
The proportion of time both preparers are busy is
π(2) + π(3) =
5
≈ 0.098,
51
or just under 10% of the time.
(4) (10 points) (Durrett 4.15) A computer lab has three laser printers that are hooked to the
network. A working printer will function for an exponential amount of time with mean
20 days. Upon failure it is immediately sent to the repair facility. There machines are
worked on by two repairmen who can each repair one printer in an exponential amount
of time with mean 2 days. However, it is not possible for two people to be working on
the same printer.
(a) Formulate a Markov chain model for the number of working printers, and find the
stationary distribution.
(b) How often are both repairmen busy?
(c) What is the average number of machines in use?
Solution:
(a) The state of the chain is the number of working printers, 0 through 3. The rate of
transitioning down is x/20days when x printers are working: if x things can break,
the time for the first of them to break is exponential with x times the rate. The
rate of transitioning up is 1/2days if there are 2 working printers (only one broken),
or 1/1day if there are 0 or 1 working printers (at least two broken). Note that you
do not fix two printers simultaneously: you fix one printer at a time, twice as fast.
The stationary distirbution solves the detailed balance equations:
π(0) · 1 = π(1) · 1/20
π(1) · 1 = π(2) · 2/20
π(2) · 1/2 = π(3) · 3/20
π(0) + . . . + π(2) = 1
You know you can solve these because there are 4 equations in 4 unknowns.
The solution is
π=(
3
60
600 2000
,
,
,
).
2663 2663 2663 2663
(b) Both repairmen are busy if we’re in states 0 or 1, so the fraction of time is
63
≈ 0.024.
2663
This corresponds to about two days in every semester. (Note that we are dealing
with an irreducible, finite-state continuous-time Markov chain, so the limit theorems
apply.)
π(0) + π(1) =
5
(c) The average number of machines in use (i.e., working) is
3
X
n · π(n) =
n=0
7260
≈ 2.72.
2663
(5) (10 points) (Durrett 4.19–4.20) Consider a barbershop with one barber who can cut hair
at rate 4 (people per hour), and three waiting chairs. Customers arrive at rate 5 per
hour. Customers who arrive to a fully occupied shop leave without being served. Find
the stationary distribution for the number of customers in the shop, and the average
number of customers served per hour.
Solution: The state of the Markov chain is the number of people in the shop. The chain
transitions at rate 5/hr up from states 0 through 3, and at rate 4/hr down from states
1 through 4. The stationary distribution satisfies detailed balance:
π(0) · 5 = π(1) · 4
π(1) · 5 = π(2) · 4
π(2) · 5 = π(3) · 4
π(3) · 5 = π(4) · 4
π(0) + . . . + π(4) = 1
Note that this is 5 equations in 5 unknowns, hence solvable. The solution is π(n) =
5/4−1
(5/4)n π(0) with π(0) = (5/4)
4 −1 ≈ 0.173. Numerically,
π ≈ (0.173, 0.217, 0.271, 0.339, 0.423).
The rate at which customers are served can be computed either as
λ(1 − π(4))
or as
µ(1 − π(0)).
The former looks at people walking into the shop; the latter looks at people walking out
of the shop. The two are equal because the number of people in the shop isn’t growing
or shrinking over time. Thus, the rate in question is ≈ 2.88 people per hour.
(6) (10 points) (Durrett 4.32) Consider a taxi station at an airport where taxis and (groups
of) customers arrive at times of independent Poisson processes with rates 2 per minute
and 3 per minute respectively. A taxi will wait for a customer no matter how many
other taxis are present, but if an arriving group of people doesn’t find a taxi, they leave
to find alternative transportation.
(a) Set up an infinite-state Markov chain to model this problem. The state should
correspond to the number of waiting taxis.
(b) Find the proportion of arriving customers that get taxis.
(c) Find the average number of taxis waiting.
Solution:
6
(a) From the problem statement, we don’t need to model waiting people, only waiting
taxis, so the states of the Markov chain are 0, 1, 2, ... corresponding to the number
of waiting taxis. The transition rate increasing the number of waiting taxis is 2/min;
this is q(n, n + 1) for n ≥ 0. The transition rate decreasing the number of waiting
taxis is 3/min; this is q(n, n − 1) for n ≥ 1.
(b) We are dealing with an irreducible CTMC, and we’ll shortly see that it does have a
stationary distribution. Namely, writing down the detailed balance equations,
2
2
π(n) · 2 = π(n + 1) · 3 =⇒ π(n + 1) = π(n) = . . . = ( )n+1 π(0),
3
3
1
2
and hence π(0)(1 + 2/3 + (2/3) + . . . ) = 1 so π(0) = 3 .
By the PASTA property, the proportion of customers who get taxis is the proportion
of time taxis are waiting, i.e. 1 − π(0) = 2/3.
(c) The average number of taxis waiting is the mean of the geometric distribution with
parameter 1/3, and the only thing we need to be careful of is whether this is the
geometric that’s “number of failures until the first success” or “index of the first
success”. It’s the former, because it is equal to 0 with a positive probability. The
mean is
1
− 1 = 2.
1/3
(The mean of the “index of the first success” geometric is 1/p, where p is the success
probability, here 1/3. The mean of the “number of failures until the first success”
is clearly one less than that.)
Of course, you could also evaluate the infinite sum
∞
X
1 2
k · ( )k ;
3 3
k=0
I’ve explained how you go about doing that in the solution to Problem 7 of Homework 4.
7