EE178: Homeworks #9 Solutions
1. Moment generating function.
Let X be a Gaussian random variable with X ∼ N (0, σ 2 ) and let U be a Bernoulli
random variable with U ∼ Bern() independent of X. Define V as
V = XU .
(a) Find the moment generating function of V , MV (s) = E esV .
(b) Find the first and second moment of V using the moment generating function.
Solution:
(a) From the definitions,
E esV = E esXU = EU EX esXU U
= (1 − ) + E esX
= (1 − ) + e
s2 σ 2
2
.
(b) From the definition of expectation,
s2 σ 2
d
2
1 − + e
E[V ] =
ds
s=0
h
2 2i
2 s 2σ
= sσ e
s=0
=0,
2 s2 σ 2
d
2
E[V ] =
1 − + e 2
ds2
s=0
h
2 2
2 2i
2 4 s 2σ
2 s 2σ
= s σ e
+ σ e
s=0
2
=σ .
2. The Chernoff bound.
(a) Show that the inequality
P (X ≥ a) ≤ e−sa M (s)
holds for every a and every s ≥ 0, where M (s) = E[esX ] is the moment generating
function of X.
Hint: P(X ≥ a) = P esX ≥ esa .
Homework 9
Page 1 of 13
(b) Let U1 , U2 , . . . , Un be i.i.d. such that Ui ∼ Bern(p) and Xn =
P (Xn ≥ (1 + )np) ≤ e−
np2
3
Pn
i=1
Ui . Show that
.
Hint:
•
•
•
•
Use the result of part (a) to bound P(Xn > (1 + )np).
s
Bound the moment generating function of U by MU (s) ≤ ep(e −1) .
Find s that gives the most tight bound (i.e., minimize the bound term).
You can use the following inequality.
2
e
≤ e− 3 .
1+
(1 + )
Solution:
(a) We have
P (X ≥ a) =P esX ≥ esa
E[esX ]
esa
−sa
=e E[esX ].
≤
(b) For all s > 0,
P (Xn ≥ (1 + )np) =P esXn ≥ es(1+)np
≤E[esXn ]e−s(1+)np
n
= E[esU ]e−s(1+)p
n
= (1 − p + pes )e−s(1+)p
n
s
≤ ep(e −1) e−s(1+)p
= exp (np(es − 1 − s(1 + ))) .
Note that the minimum of es − 1 − s(1 + ) is achieved at s = log(1 + ). If we
use this value, we have
P (Xn ≥ (1 + )np) ≤ exp (np(es − 1 − s(1 + )))
≤ exp (np( − (1 + ) log(1 + )))
np
e
≤
(1 + )1+
≤e−
Homework 9
np2
3
.
Page 2 of 13
3. Confidence intervals.
A population of 108 voters choose between two candidates A and B. A fraction p = 0.55
of them plan to vote for candidate A and the rest for candidate B. A fair poll with
sample size n is performed, i.e., the n samples are i.i.d. and done with replacement
(same person may be polled more than once). We want to find a value of n that will
guarantee that the majority of those sampled vote for candidate A, with probability at
least 0.99. More precisely and equivalently, let U1 , UP
2 , . . . , Un be i.i.d.∼ Bernoulli(0.55).
1
The fraction of 1s in this sequence is thus Xn = n ni=1 Ui and we want n sufficiently
large to guarantee that P{Xn > 0.5} > 0.99.
(a) Use the Chebyshev inequality to find such an n.
(b) Use the Chernoff inequality to find such an n.
Hint: Similar to part (b) of Problem 2, we can show that
!
n
np2
1X
Ui ≤ (1 − )p ≤ e− 2
P
n i=1
(c) Use the central limit theorem to approximate such an n.
Solution:
(a) By Chebysheff’s inequality,
P {Xn − p < −0.05} ≤P {|Xn − p| > 0.05}
σx2
,
≤
(0.052 )n
2
σx
since Xi ∼ Bern(0.1), σx2 = 0.2475. We need (0.05
2 )n < 0.01, which implies n >
9900.
(b) We have
P (Xn ≤ (1 − )p) ≤ e−
Let =
0.05
0.55
np2
2
.
and p = 0.55. Then,
−
P (Xn ≤ 0.5) ≤ e
We want exp
2
n 0.05
− 0.55
2
2
n 0.05
0.55
2
.
< 0.01, which implies n > 2026.
(c) We have
n
X
Ui − p
p
i=1
Homework 9
np(1 − p)
∼ N (0, 1)
Page 3 of 13
Therefore,
r
n
n
(Xn − p) ≤
(0.5 − 0.55)
P (Xn ≤ 0.5) =P
p(1 − p)
p(1 − p)
!
r
n
X
Ui − p
n
p
≤−
× 0.05
=P
p(1
−
p)
np(1
−
p)
i=1
r
n
≈Q
× 0.05 .
p(1 − p)
q
n
We want Q
× 0.05 ≤ 0.01, which implies
p(1−p)
r
2
n ≥ Q−1 (0.01)/0.05 × p(1 − p) ≈ 535.78.
4. Gambling.
Let Xn be the amount you win on the nth round of a game of chance. Assume that
X1 , X2 , . . . , Xn are i.i.d. with finite mean E(X) and variance σ 2 . Make the realistic
assumption that E[X] < 0. Show that P{(X1 + X2 + . . . + Xn )/n < E[X]/2} → 1.
What is the moral of this result?
Solution: P
We know n1 ni=1 Xi → E[X] by the weak law of large numbers, which means P(|Sn −
E[X]| > ) → 0 as n → ∞. The limiting value of P(Sn < E[X]
) depends on E[X].
2
E[X]
When E[X] < 0, P(Sn < 2 ) approaches 1. This is because P(|Sn − E[X]| > ) → 0
as n → ∞ for all finite . But this means P(|Sn − E[X]| < ) → 1 as n → ∞. Since
) → 1.
this requires E[X] negative and Sn → E[X], then we know P(Sn < E[X]
2
5. Feel for the central limit theorem.
Download the MATLAB code ‘hw9p5.m’ and ‘hw9p5.mat’ from the webpage. This
code extracts the 25000 human height data (inches) from ‘hw9p4.mat’. Complete the
code that does the following.
• Similar to the problem 3 of homework 5, plot the normalized histogram of height
data with ∆ = 0.1.
• Using the weak law of large number, estimate the mean (ˆ
µ) and the variance (ˆ
σ2)
of the data set.
• Plot the pdf of N (ˆ
µ, σ
ˆ 2 ).
Complete the code and run it using MATLAB, submit the plot, and comment on its
form.
Solution:
The normalized histogram and the pdf of N (ˆ
µ, σ
ˆ 2 ) look very similar.
Homework 9
Page 4 of 13
Heights
0.25
normalized histogram
pdf of Gaussian
0.2
0.15
0.1
0.05
0
55
60
65
70
75
80
Figure 1: Problem 5
clear all ; clc ; close all ;
% l o a d data
l o a d ( ’ hw9p5 . mat ’ ) ;
% e x t r a c t h e i g h t data
D a t a h e i g h t = Data ( : , 2 ) ;
N height = length ( Data height ) ;
% Gaussian
mu hat = sum ( D a t a h e i g h t ) / N h e i g h t ;
s i g m a h a t = s q r t ( sum ( D a t a h e i g h t . ˆ 2 ) / N h e i g h t − mu hat ˆ 2 ) ;
% length of interval
delta = . 1 ;
% d e f i n e th e range
r an ge = c e i l ( mu hat−6∗ s i g m a h a t ) : d e l t a : c e i l ( mu hat+6∗ s i g m a h a t ) ;
% histogram
h i s t h e i g h t = h i s t ( D a t a h e i g h t , range ) / d e l t a / N h e i g h t ;
Homework 9
Page 5 of 13
% pdf o f N( mu hat , s i g m a h a t ˆ 2 ) ;
f Z = 1/ s q r t (2∗ p i ∗ s i g m a h a t ˆ2)∗ exp (−( range−mu hat ) . ˆ 2 / 2 / s i g m a h a t ˆ 2 ) ;
% plot
figure (1);
p l o t ( range , h i s t h e i g h t , ’ r ’ , range , f Z , ’ b ’ , . . .
’ Linewidth ’ , 2 , ’ MarkerSize ’ , 1 0 ) ;
l e g e n d ( ’ n o r m a l i z e d histogram ’ , ’ pdf o f Gaussian ’ ) ;
t i t l e ( ’ Heights ’ ) ;
6. Generating random variable from a uniform.
3
Let X be a random variable with the pdf fX (x) = 32 x2 e−|x| .
(a) Compute the cdf of X and inverse of it, i.e., find the FX (x) and FX−1 (u).
(b) Generate N = 107 i.i.d. samples X1 , X2 , . . . , XN according to the distribution, by
generating and appropriately mapping uniformly distributed random variables.
Similar to the problem 3 of homework 5, plot the normalized histogram and
compare with the original pdf fX (x). We recommend to use ∆ = 0.05.
Solution:
(a) We have
Z
x
FX (x) =
(−∞
Rx
=
3 2 −|a|3
ae
da
2
3
3 2 a3
a e da = 12 ex
−∞ R2
3
x
1
+ 0 23 a2 e−a da = 1
2
−
Then, it is not hard to see that
(
1
{log(2u)} 3
−1
FX (u) =
1
− {log(2(1 − u))} 3
1 −x3
e
2
if x < 0
if x ≥ 0
if u <
if u ≥
1
2
1
2
(b) Figure 2 shows the comparison between the normalized histogram and the original
pdf. The code is attached at the end.
7. Limit Theorems.
Similar to the previous problem, let X be a random variable with the pdf fX (x) =
Homework 9
Page 6 of 13
0.7
normalized histogram
pdf
0.6
0.5
0.4
0.3
0.2
0.1
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
Figure 2: Problem 6(b)
3 2 −|x|3
xe
.
2
Again, generate N = 107 samples X1 , X2 , . . . , XN according to the distribu-
tion.
(a) For 1 ≤ i ≤
N
,
k
let
Pik
Xj
X(i−1)k+1 + · · · + Xik
k
k
3
where k = 10 . Plot the histogram of Si ’s and compute the fraction of Si ’s where
|Si | > 0.1.
Si =
j=(i−1)k+1
=
(b) The weak law of large numbers tells us that
Pn
2
i=1 Xi
→ E[X 2 ].
n
Therefore, it is natural to estimate the second moment of X by
PN
2
i=1 Xi
.
N
Based on this number, estimate the variance of X.
(c) Let σ
ˆ 2 be your estimate of the variance of X from part (b). For 1 ≤ i ≤
Pik
X(i−1)k+1 + · · · + Xik
j=(i−1)k+1 Xj
√
√
Zi =
=
2
kˆ
σ
kˆ
σ2
N
,
k
let
where k = 103 . Similar to the problem 3 of homework 5, plot the normalized
histogram of Zi ’s using ∆ = 0.05. What does it look like? Explain.
Homework 9
Page 7 of 13
(d) Repeat part (a), (b) and (c) with the discrete random variable X with the pmf
pX (−1) = pX (1) = 21 . Use ∆ = 0.2 when you plot the normalized histogram.
Solution:
(a) Figure 3 shows the histogram of Si ’s. In this simulation, the fraction of |Si | > 0.1
was 0.0005. The code is attached at the end.
350
300
250
200
150
100
50
0
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
Figure 3: Problem 7(a)
(b) The variance of X is equal to the second moment. We have σ
ˆ 2 = 0.9025. Note
that the true value is σ 2 = 0.90275. The code is attached at the end.
(c) Figure 4 shows the comparison between the normalized histogram and the pdf of
standard Gaussian random variable. The code is attached at the end.
(d) Figure 5 shows the histogram of Si ’s. In this simulation, the fraction of |Si | > 0.1
was 0.002. The code is attached at the end.
The variance of X is equal to the second moment. We have σ
ˆ 2 = 1. Note that
the true value is also σ 2 = 1. The code is attached at the end.
Figure 6 shows the comparison between the normalized histogram and the pdf of
standard Gaussian random variable. The code is attached at the end.
Homework 9
Page 8 of 13
0.5
0.45
normalized histogram
pdf
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
0.15
0.2
0.25
Figure 4: Problem 7(c)
800
700
600
500
400
300
200
100
0
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
Figure 5: Problem 7(d)-1
Homework 9
Page 9 of 13
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
-5
-4
-3
-2
-1
0
1
2
3
4
5
Figure 6: Problem 7(d)2
clear all ; clc ; close all ;
%% problem 6
% number o f samples
N = 10ˆ7;
% length of interval
delta = 0.05;
% d e f i n e th e range
r an ge = −5: d e l t a : 5 ;
% Compute t he p r o b a b i l i t y d e n s i t y f u n c t i o n
f X = 3/2∗( range ) . ˆ 2 . ∗ exp(−abs ( range ) . ˆ 3 ) ;
% g e n e r a t e N uniform random v a r i a b l e s
U = rand (N, 1 ) ;
% g e n e r a t e X based on U
X = z e r o s (N, 1 ) ;
i n d e x 1 = f i n d (U< 0 . 5 ) ;
i n d e x 2 = f i n d (U>=0.5);
Homework 9
Page 10 of 13
X( i n d e x 1 ) = −abs ( l o g (2∗U( i n d e x 1 ) ) ) . ˆ ( 1 / 3 ) ;
X( i n d e x 2 ) = abs ( l o g (2∗(1 −U( i n d e x 2 ) ) ) ) . ˆ ( 1 / 3 ) ;
% count t h e number o f samples i n th e bi n
h i s t X = h i s t (X, range ) /N/ d e l t a ;
% plot
figure (1);
p l o t ( range , h i s t X , ’ rx ’ , range , f X , ’ b ’ , . . .
’ Linewidth ’ , 2 , ’ MarkerSize ’ , 1 0 ) ;
l e g e n d ( ’ n o r m a l i z e d histogram ’ , ’ pdf ’ ) ;
%% problem 7
k = 10ˆ3;
%(a )
% rearrange X i ’ s
X mat = r e s h a p e (X, k ,N/k ) ;
% Generate S i ’ s
S = sum ( X mat ) / k ;
% plot
figure (2);
hi st (S , 1 0 0 ) ;
% Compute t he f r a c t i o n o f | S i | >0.1
S f r a c = l e n g t h ( f i n d ( abs ( S ) > 0 . 1 ) ) / (N/k ) ;
d i s p ( [ ’ f r a c t i o n o f | S i | >0.1 = ’ num2str ( S f r a c ) ] ) ;
%(b )
% Estimate th e second moment
s i g m a h a t s q u a r e = sum (X. ˆ 2 ) /N;
d i s p ( [ ’ sigma hat s q u a r e = ’ num2str ( s i g m a h a t s q u a r e ) ] ) ;
%(c )
% Generate Z i ’ s
Z = sum ( X mat ) / s q r t ( k∗ s i g m a h a t s q u a r e ) ;
% count t h e number o f Z i ’ s i n t he b in
h i s t Z = h i s t (Z , range ) / (N/k ) / d e l t a ;
Homework 9
Page 11 of 13
% pdf o f N( 0 , 1 ) ;
f N = 1/ s q r t (2∗ p i )∗ exp(− range . ˆ 2 / 2 ) ;
% plot
figure (3);
p l o t ( range , h i s t Z , ’ r ’ , range , f N , ’ b ’ ) ;
%(d )
% length of interval
delta = 0.2;
% d e f i n e th e range
r an ge = −5: d e l t a : 5 ;
% g e n e r a t e N uniform random v a r i a b l e s
U = rand (N, 1 ) ;
% g e n e r a t e X based on U
X = (U>0.5)∗2 −1;
% rearrange X i ’ s
X mat = r e s h a p e (X, k ,N/k ) ;
% Generate S i ’ s
S = sum ( X mat ) / k ;
% plot
figure (4);
hi st (S , − . 2 1 : 0 . 0 0 6 : . 2 1 ) ;
% Compute t he f r a c t i o n o f | S i | >0.1
S f r a c = l e n g t h ( f i n d ( abs ( S ) > 0 . 1 ) ) / (N/k ) ;
d i s p ( [ ’ f r a c t i o n o f | S i | >0.1 = ’ num2str ( S f r a c ) ] ) ;
% Estimate th e second moment
s i g m a h a t s q u a r e = sum (X. ˆ 2 ) /N;
d i s p ( [ ’ sigma hat s q u a r e = ’ num2str ( s i g m a h a t s q u a r e ) ] ) ;
% Generate Z i ’ s
Z = sum ( X mat ) / s q r t ( k∗ s i g m a h a t s q u a r e ) ;
Homework 9
Page 12 of 13
% count t h e number o f Z i ’ s i n t he b in
h i s t Z = h i s t (Z , range ) / (N/k ) / d e l t a ;
% pdf o f N( 0 , 1 ) ;
f N = 1/ s q r t (2∗ p i )∗ exp(− range . ˆ 2 / 2 ) ;
% plot
figure (5);
p l o t ( range , h i s t Z , ’ r ’ , range , f N , ’ b ’ ) ;
Homework 9
Page 13 of 13