University of Illinois
Spring 2015
ECE 561: Problem Set 1: Solutions
Statistical Decision-Making Framework, Binary Bayesian and Minimax
Hypothesis Testing
1. [Statistical Decision Making].
(a) We may compute the conditional risks for the eight possible decision rules as follows.
δ
δ1
δ2
δ3
δ4
δ5
δ6
δ7
δ8
1
0
0
0
0
1
1
1
1
2
0
0
1
1
0
0
1
1
3
0
1
0
1
0
1
0
1
R0
0
0
0.6
0.6
0.4
0.4
1
1
R1
10
5
7.5
2.5
7.5
2.5
5
0
0.5(R0 + R1 )
5
2.5
4.05
1.55
3.95
1.45
3
0.5
max(R0 , R1 )
10
5
7.5
2.5
7.5
2.5
5
1
It is clear that there is no rule that can be better than both δ2 and δ8 , and hence there is no best
rule based on conditional risks alone.
(b) It follows from the table that δ8 (always choose 1) is the best Bayes as well as minimax rule in
D.
˜
(c) Randomization does not improve minimum Bayes risk, hence δ8 is still the best rule in D.
However it is easy to check that the rule
δ2
δ˜m =
δ8
with prob. 1/6
.
with prob. 5/6
has a minimax risk of 5/6, which is better than the minimax risk of the nonrandomized minimax
rule from part (b). Since R0 (δi ) is smaller than R0 (δ8 ) for any i ∈ {3, 4, 5, 6}, one can also
randomize δ8 with δi , i ∈ {3, 4, 5, 6} to achieve a smaller minimax error.
2. [Optimal Rule Based on Conditional Risks]
Let δ0 be the decision rule that always chooses hypothesis 0. It is easy to check that
R0 (δ0 ) = C0,0 .
Similarly, let δ1 be the decision rule that always chooses hypothesis 1:
R1 (δ1 ) = C1,1 .
Suppose there exists a “best” rule δ that divides the observation space Y into Y0 and Y1 . Since δ is a
best rule, we have
R0 (δ) ≤ R0 (δ0 )
⇒ C0,0 P0 (Y0 ) + C1,0 P0 (Y1 ) ≤ C0,0
⇒ C1,0 P0 (Y1 ) ≤ C0,0 P0 (Y1 )
Since C0,0 < C1,0 , the inequality above is possible only if P0 (Y1 ) = 0. Similarly, by exploiting the fact
that R1 (δ) ≤ R1 (δ1 ), we get P1 (Y0 ) = 0. Thus, we proved that pi (y) has non-zero support only on Yi ,
i = 0, 1. Since Y0 and Y1 are disjoint, this implies that p0 (y) and p1 (y) have disjoint supports.
3. [Health Insurance]
Since A = {buy, not} and Y = {0, 1}, there are four deterministic decision rules.
δ
δ1
δ2
δ3
δ4
0
buy
buy
not
not
1
buy
not
buy
not
R0 (δ)
1
11/2
11/2
10
R1 (δ)
1
0
1
0
0.2R0 (δ) + 0.8R1 (δ)
1
1.1
1.9
2
max{R0 (δ), R1 (δ)}
1
11/2
11/2
10
where
R0 (δ)
= E0 [C(δ(Y ), 0)] =
X
p0 (y)C(δ(y), 0) =
y=0,1
R1 (δ)
= E1 [C(δ(Y ), 1)] =
X
1
[C(δ(0), 0) + C(δ(1), 0)]
2
p1 (y)C(δ(y), 1) = C(δ(1), 1)
y=0,1
Listing all four decision rules in above table, we observe that δ1 (y) = buy ∀y, is the minimax rule
among all deterministic decision rules. By observing that R0 (δ1 ) < R0 (δi ), i = 2, 3, 4, it is clear that
˜ < 1. As a result, δ1 is also the minimax rule among all
there is no randomized rule δ˜ such that R0 (δ)
randomized rules.
4. [Binary Communication with Erasures]
(a) The Bayes rule chooses the decision with smallest a posteriori cost given Y = y. Now,
C(d|y) =
1
X
C(d, j)π(j|y).
j=0
Thus C(0|y) = π(1|y), C(1|y) = π(0|y), and C(e|y) = c[π(0|y) + π(1|y)] = c, with
π(0|y) =
p0 (y)π0
p0 (y)π0
1
=
=
p(y)
p0 (y)π0 + p1 (y)π1
1 + exp(2y/σ 2 )
and
π(1|y) = 1 − π(0|y) =
1
.
1 + exp(−2y/σ 2 )
We erase (i.e., choose d = e) if c < min{π(0|y), π(1|y)}. Since min{π(0|y), π(1|y)} is always less
that 0.5, we never erase if c ≥ 0.5.
If c < 0.5, we consider two cases:
(i) if y ≥ 0, then min{π(0|y), π(1|y)} = π(0|y) and hence we erase
1
σ2
1
if c <
i.e., if y <
ln
− 1 =: t
1 + exp(2y/σ 2 )
2
c
(ii) if y < 0, then min{π(0|y), π(1|y)} = π(1|y) and hence we erase
1
σ2
1
if c <
i.e.,
if
y
>
−
ln
−
1
= −t
1 + exp(−2y/σ 2 )
2
c
Thus
Ye = (−t, t).
We choose 0 if π0 (y) ≤ min{c, π(1|y)}. From above it is clear that
Y0 = (−∞, −t].
2
Finally, we choose 1 if π1 (y) ≤ min{c, π(0|y)}. Thus
Y1 = [t, ∞).
To summarize,

 0
e
δB (y) =

1
y ≤ −t
−t < y < t ,
y≥t
σ2
ln
where t =
2
1
−1 ,
c
if c < 0.5,
(b) Recall that we erase if c < min{π(0|y), π(1|y)}. Since min{π(0|y), π(1|y)} is always less that 0.5,
we never erase if c ≥ 0.5.
0 y≤0
δB (y) =
, if c ≥ 0.5.
1 y>0
5. [Minimum Bayes Risk Curve and Minimax Rule]
For binary hypothesis testing, the structure of the Bayes rule is given by
(
(y)
≥η
0 L(y) , pp01 (y)
,
δB (y) =
1 L(y) < η
where
η,
π0 (C10 − C00 )
π0
,
=
C01 − C11
10(1 − π0 )
and η is monotonically increasing in π0 .
It is straightforward to see that

 5/8
5/12
L(y) =

∞
When
π0
10(1−π0 )
≤ 5/12, which is equivalent to π0 ≤
25
31 ,
δB (y) = δ8 = 1
When
5
12
<
π0
10(1−π0 )
≤ 58 , which is equivalent to
25
31
δB (y) = δ6 =
When
5
8
<
π0
10(1−π0 ) ,
which is equivalent to
25
29
y=1
y=2 .
y=3
the Bayes rule in this case is
for any y.
< π0 ≤
1
0
25
29 ,
the Bayes rule in this case is
y = 1, 3
.
y=2
< π0 ≤ 1, the Bayes rule in this case is
δB (y) = δ2 =
1
0
Now consider V (π0 ) = min r(π0 , δ). It follows that for
δ
y=3
.
y = 1, 2
π0
10(1−π0 )
≤ 5/12,
V (π0 ) = r(π0 , δ8 ) = π0 ;
and for
5
12
<
π0
10(1−π0 )
≤ 58 ,
V (π0 ) = r(π0 , δ6 ) = 2.5 − 2.1π0 ;
3
5
4.5
4
3.5
3
2.5
δ6
δ2
2
1.5
minimax rule
1
δ8
0.5
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
25/31
Figure 1: V (π0 )
and for
5
8
<
π0
10(1−π0 ) ,
V (π0 ) = r(π0 , δ2 ) = 5 − 5π0 .
The risk function V (π0 ) as a function of π0 is shown in the Figure 1.
25
. Since V (π0 ) is non-differentiable at π0∗ ,
The maximizer of V (π0 ) can be easily computed to be π0∗ = 31
there does not exist a deterministic minimax rule. The minimax rule is constructed by randomizing
between δ8 and δ6 , which is also an equalizer rule. Let
δ8
w.p. ν
δ˜ =
.
δ6
w.p. 1 − ν
˜ and R1 (δ),
˜ we solve for ν as follows
Equalizing R0 (δ)
νR0 (δ8 ) + (1 − ν)R0 (δ6 ) = νR1 (δ8 ) + (1 − ν)R1 (δ8 )
ν + 0.4(1 − ν) = 2.5(1 − ν)
21
ν=
.
31
So the minimax rule is
δ˜ =

 δ8
w.p.
21
31
δ6
w.p.
10
31

.
6. [Binary Hypothesis Testing]
(a) It follows by definition that
L(y) = 3y 2
for y ∈ [−1, 1],
and η = 1 for uniform costs and equal priors. The Bayes rule is given by
(
1
for |y| ≥ √13
δB (y) =
.
0
for |y| < √13
4
The risk associated with the Bayes rule with equal priors is
Z √1
Z 1
2
3 3y
1
1
1
r(δB ) = · 2
dy + · 2
dy
1
2
2
2
2
√
0
3
1
1
= − √ .
2 3 3
(b) The Bayes rule for a given π0 is
δB (y) =

 1
for |y| ≥ √
 0
for |y| < √
√
π0
3(1−π0 )
√
π0
.
(1)
3(1−π0 )
It is known that if there exists a Bayes rule that equalizes the conditional risks under both
hypotheses, it is also a minimax rule. As shown by (1), the structure of any Bayes rule is given
by
1
for |y| ≥ η
δB (y) =
.
0
for |y| < η
And the corresponding conditional error probabilities can be computed to be
Z 1
1
dy = 1 − η,
R0 (δB ) = 2
2
η
and
Z
R1 (δB ) = 2
0
η
3 3
y dy = η 3 .
2
Equalizing R0 and R1 , we get that η = 0.6824. So the minimax rule is given by
1
for |y| ≥ 0.6824
δm (y) =
.
0
for |y| < 0.6824
And the corresponding risk is
R0 = R1 = 0.3177.
(We can reach the same answer by arguing that because the distribution of L(y) has no point
mass under either distribution, the risk function V (π0 ) is differentiable for all π0 ∈ (0, 1). As a
result, the minimax rule is given by the Bayes rule that equalizes the conditional risks under both
hypotheses.)
7. [Binary Hypothesis Testing with Nonuniform Costs]
(a) The likelihood ratio L(y) is given by
L(y) = 2e−|y| .
With equal priors (π0 = π1 = 0.5), the threshold for Bayes rule is given by
τ=
C1,0 − C0,0
1
= .
C0,1 − C1,1
4
Therefore, the Bayes rule is
δB (y) =
1,
0,
if |y| ≤ ln 8
.
otherwise
(b) The corresponding Bays risk is equal to
1
1
1
C1,0 P0 (Y1 ) + C0,1 P1 (Y0 ) =
2
2
2
5
7
1
15
+
=
.
8
32
32