ETH Z¨
urich
Dept. Computer Science
Spring Semester 2015
Information theory
Mid-term exam
24 March 2015
Time limit:
Number of pages:
Total points:
90 minutes
8
100 points
You can use the back of the pages if you run out of space. Collaboration on the exam is
strictly forbidden.
[1 point] Please fill in your name and student ID:
1
[24 points] Short questions
a) Below, three binary codes are shown for the source symbols x1 , . . . x4 .
x1
x2
x3
x4
Code A
00
01
10
11
Code B
0
100
11
110
Code C
1
10
100
1000
(1) [3 points] Which codes fulfil the Kraft inequality?
(2) [3 points] Which codes are prefix-free codes?
(3) [3 points] Which codes are uniquely decodable?
(4) [3 points] Is a code that satisfies this inequality always uniquely decodable?
b) [True or false]
Let X, Y , Z be any discrete random variables. For each of the questions below, decide
whether they are true or false, and briefly justify your answer. You can either provide an
intuitive explanation, or show a counterexample, or mathematically prove your answer.
(1) [3 points] I(X; Y ) ≥ 0.
(2) [3 points] I(X; Y ) = H(X) + H(Y ) − H(X, Y ).
(3) [3 points] I(X; Y ) ≥ I(X, Z; Y ).
(4) [3 points] H(X, Y, Z) = H(X) + H(Y | X) + H(Z | Y, X).
2
[30 points] Wine tasting
There are six bottles of wine. It is known that precisely one bottle has gone bad. From
inspection of the bottles it is determined that the probability pi that the i-th bottle is bad
1 1 1
is given by (p1 , p2 , p3 , p4 , p5 , p6 ) = ( 21 , 14 , 18 , 16
, 32 , 32 ). Tasting will determine the bad wine.
Suppose that each time, you can mix several of wines in a fresh glass, and sample the mixture
for tasting. If the mixture contains the bad wine, it will taste terrible.
You start with one bottle or a mixture of several bottles of wines and proceed, mixing and
tasting, stopping when the bad bottle has been determined. The objective is to design a
strategy to taste the wines with the minimum expected number of tastings.
a) [5 points] Before designing any strategy, give a lower bound for the expected number
of tastings of any strategy.
b) [15 points] Design a strategy to taste the wines with the minimum expected number
of tastings.
c) [10 points] Suppose you are allowed to have only two tastings. Design a strategy to
taste the wines, such that you can identify the bad wine with maximal probability.
What is the probability of identifying the bad wine using your strategy?
3
[45 points] Entropy and Mutual Information
Consider the following events U = {u1 , u2 , u3 }, and V = {v1 , v2 , v3 , v4 }. The random variables
U and V follow the joint probability distribution specified in the table below.
p(U, V )
u1
u2
u3
v1
v2
v3
v4
1
6
1
24
1
24
1
12
1
6
1
24
1
24
1
12
1
6
1
24
1
24
1
12
a) [8 points] Find H(U ), H(V | U ). (Hint: you may use the fact that log2 (3) = 1.585.)
In this exercise, we intend to design a code for the source (U, V ). That is, for each
event (ui , vj ) we want to assign a binary string C(ui , vj ). However, instead of directly
encoding the joint event (ui , vj ), we want to do it in a different way as follows:
• We first design a Huffman code for the source U . Let the resulting codewords be
C(u1 ), C(u2 ), C(u3 ).
• For each specific output of the source U , namely U = ui , we then design a Huffman code for the source V assuming that U = ui . We denote the corresponding
codewords by C(v1 | ui ), C(v2 | ui ), C(v3 | ui ), C(v4 | ui ).
• Finally, we let C(ui , vi ) = (C(ui ), C(vi | ui )). That is, the codeword assigned to
(ui , vi ) is obtained by concatenating the two strings C(ui ) and C(ui | vi ).
b) [6 points] Design a Huffman code for U .
c) [8 points] Given U = u1 , design a Huffman code for V . (Hint: What is the distribution
of the source V if you know U = u1 ? ).
d) [4 points] Perform the same procedure as the previous part assuming that U = u2 . Also,
do it for U = u3 .
e) [2 points] What is your final code for the source (U, V )? That is, what is C(ui , vj )?
Let us now generalise the scheme to any source (U, V ) with a joint probability distribution p(U = u, V = v). The procedure is the same:
• We first design a Huffman code for U . The resulting codewords are denoted by
C(u) for u ∈ U .
• Then, for each U = u we design a Huffman code for V given that U = u. The
resulting codewords are denoted by C(v | u) for each v ∈ V .
• Finally, we let C(u, v) = (C(u), C(v | u)) for each (u, v) ∈ U × V .
f ) [2 points] Let l(u) = length(C(u)), and l(v | u) = length(C(v | u)). Express the
length of the codeword C(u, v) in terms of l(u) and l(v | u).
¯=P
g) [15 points] Let L
(u,v)∈(U,V ) p(U = u, V = v)length(C(u, v)). Prove that
¯ < H(U, V ) + 2.
H(U, V ) ≤ L