1. No category

Math 321: Show X and S 2 are independent
(Under the assumption the random sample is normally distributed)
A well known result in statistics is the independence of X and S 2 when X1 , X2 , · · · , Xn ∼ N µ, σ 2 .
This handout presents a proof of the result using a series of results. First, a few lemmas are presented
2
is
which will allow succeeding results to follow more easily. In addition, the distribution of (n−1)S
σ2
derived.
Definition 1. The sample variance is defined as
n
S2 =
1 X
(Xi − X)2
n − 1 i=1
Lemma 1. The sum of the squares of the random variables X1 , X2 , · · · , Xn is
n
X
Xi2 = (n − 1)S 2 + nX
2
i=1
Proof. By Definition 1,
(n − 1)S 2 =
n
X
n
X
(Xi − X)2 =
i=1
Xi2 − 2X
i=1
n
X
Xi +
i=1
n
X
2
X =
i=1
n
X
2
2
Xi2 − 2nX + nX =
n
X
i=1
Xi2 − nX
2
i=1
It follows that
n
X
Xi2 = (n − 1)S 2 + nX
2
i=1
Lemma 2. The sum of squares of the random variables X1 , X2 , · · · Xn centered about the mean, µ, is
n
X
(Xi − µ)2 =
n
X
(Xi − X)2 + n(X − µ)2
i=1
i=1
Proof. The sum of squares can be simplified as
n
X
(Xi − µ)2 =
i=1
n
X
Xi2 − 2µ
i=1
=
n
X
n
X
i=1
Xi +
n
X
µ2
i=1
Xi2 − 2nµX + nµ2
(1)
i=1
By Lemma 1, (1) simplifies to
n
n
X
X
2
(Xi − µ)2 =
Xi2 − 2nµX + nµ2 = (n − 1)S 2 + nX − 2nµX + nµ2
i=1
i=1
= (n − 1)S 2 + n(X − µ)2
=
n
X
(Xi − X)2 + n(X − µ)2
i=1
(2)
X and S 2 are independent, page 2
Lemma 3. If Z ∼ N(0, 1), then Z 2 ∼ χ2 (1).
Proof. The moment generating function for Z 2 is defined as
2 Z ∞ 1
2
2
√ etz e−z /2 dz
2
MZ (t) = E etZ =
2π
−∞
∞
Z
=
−∞
1
1
√ exp − (1 − 2t)z 2 dz
2
2π
#
"
1
1 z2
√ exp −
dz
=
1
2 1−2t
2π
−∞
|
{z
}
1
kernel of a N (0, 1−2t
)
#
"
Z ∞
1
1
1 z2
exp −
dz
=√
√ 1
2 1−2t
1 − 2t −∞ 2π √ 1
1−2t
|
{z
}
∞
Z
=√
integrates to 1
1
1 − 2t
= (1 − 2t)−1/2
Note that this is the moment generating function for a χ2 random variable with one degree of freedom.
Hence,
Z 2 ∼ χ2 (1)
Lemma 4. Suppose X1 , X2 , · · · , Xn are independent and identically distributed χ2 (1) random
variables. It follows that
Y =
n
X
Xi ∼ χ2 (n)
i=1
Proof. The moment generating function of Xi is
MXi (t) = (1 − 2t)−1/2 .
It follows that the moment generating function for Y is
n
n
Y
Y
MY (t) = E etY = E etX1 +tX2 +···+tXn =
MXi (t) =
(1 − 2t)−1/2
i=1
= (1 − 2t)−
Pn
i=1
i=1
1/2
= (1 − 2t)−n/2
It follows that this is the MGF for a χ2 distribution with n degrees of freedom. Hence,
Y =
n
X
i=1
Xi ∼ χ2 (n)
X and S 2 are independent, page 3
Theorem 1. Suppose X1 , X2 , · · · , Xn is a random sample from a normal distribution with mean, µ,
and variance, σ 2 . It follows that the sample mean, X, is independent of Xi − X, i = 1, 2, · · · , n.
Proof. The joint distribution of X1 , X2 , · · · , Xn is
(
2 )
n 1
1 X xi − µ
fX (x1 , x2 , · · · , xn ) =
exp −
n
2 i=1
σ
(2π) 2 σ n
Transform the random variables Xi , i = 1, 2, · · · , n to
Y1 = X
X = Y1
Y2 = X2 − X
X2 = Y2 + Y1
Y3 = X3 − X
X3 = Y3 + Y1
..
. =
..
.
..
. =
Yn = Xn − X
..
.
Xn = Yn + Y1
The Jacobian of the transformation can be shown to not depend on Xi or X and is equal to the
constant n. It follows that
fY1 ,Y2 ,··· ,Yn (y1 , y2 , · · · , yn ) = fX (x1 , x2 , · · · , xn )|J|
= nfX (x1 , y1 + y2 , · · · , y1 + yn )
(
n
1X
= constants · exp −
2 i=1
xi − µ
σ
2 )
(3)
Note that the sum in the exponent of the joint pdf can be simplified using Lemma 2. It follows that
2
n X
xi − µ
i=1
σ
=
n
1 X
(xi − µ)2
σ 2 i=1
" n
#
1 X
= 2
(xi − x
¯)2 + n(¯
x − µ)2
σ i=1
"
#
n
X
1
2
2
2
= 2 (x1 − x
¯) +
(xi − x
¯) + n(¯
x − µ)
σ
i=2
Note that since
n
X
(xi − x
¯) = 0, it follows that
i=1
x1 − x
¯=−
n
X
i=2
(xi − x
¯)
(4)
X and S 2 are independent, page 4
Therefore, equation (4) simplifies to
2
n X
xi − µ
i=1
σ
"
#
n
X
1
¯)2 +
= 2 (x1 − x
(xi − x
¯)2 + n(¯
x − µ)2
σ
i=2


!2
n
n
X
1  X
= 2
(xi − x
¯) +
(xi − x
¯)2 + n(¯
x − µ)2 
σ
i=2
i=2


!2
n
n
X
1  X
= 2
yi
+
yi2 + n(y1 − µ)2 
σ
i=2
i=2
Therefore, the pdf of Y1 , Y2 , · · · , Yn , equation (1), simplifies to
(
2 )
n 1 X xi − µ
fY1 ,Y2 ,··· ,Yn (y1 , y2 , · · · , yn ) = constants · exp −
2 i=1
σ
= constants · exp



1 
−
 2σ 2
n
X
!2
yi
i=2
+
n
X
i=2


yi2 + n(y1 − µ)2 



!2
n
n

n n
o
X
1  X
yi
+
yi2  exp − 2 (y1 − µ)2
= constants · exp − 2
|
 2σ
2σ{z
}
i=2
i=2
|
{z
}
g(y1 )


h(y2 ,y3 ,··· ,yn )
= constants · h(y2 , y3 , · · · , yn ) · g(y1 )
Because fY1 ,Y2 ,··· ,Yn (y1 , y2 , · · · , yn ) can be factored into a product of functions that depend only their
respective set of statistics, it follows that Y1 = X is independent of Yi = Xi − X, i = 2, 3, · · · , n.
Pn
Finally, since X1 − X = − i=2 (Xi − X), it follows that X1 − X is a function of Xi − X,
i = 2, 3, · · · , n. Therefore, X1 − X is independent of Y1 = X.
Theorem 2. Suppose X1 , X2 , · · · , Xn is a random sample from a normal distribution with mean, µ,
and variance, σ 2 . It follows that the sample mean, X, is independent of the sample variance, S 2 .
Proof. The definition of S 2 is given in Definition 1. Because S 2 is a function of Xi − X, i = 1, 2, · · · , n,
it follows that S 2 is independent of X.
Theorem 3. Suppose X1 , X2 , · · · , Xn is a random sample from a normal distribution with mean, µ,
and variance, σ 2 . It follows that the distribution of a mulitiple of the sample variance follows a χ2
distribution with n − 1 degrees of freedom. In particular,
(n − 1)S 2
∼ χ2 (n − 1)
σ2
Proof. Equation (2) states
n
X
(Xi − µ)2 = (n − 1)S 2 + n(X − µ)2 .
i=1
X and S 2 are independent, page 5
It follows that
2
n X
Xi − µ
i=1
σ
2
n X
Xi − µ
i=1
σ
=
2
X −µ
(n − 1)S 2
+
n
σ2
σ
(n − 1)S 2
+
=
σ2
X −µ
√
σ/ n
2
U =W +V
2
Xi − µ
Note that since Xi ∼ N µ, σ 2 , it follows that Zi =
∼ N(0, 1). Similarly, since X ∼ N µ, σn ,
2
σ
2
X −µ
Xi − µ
X −µ
√ ∼ N(0, 1). By Lemma 3, it follows that
√
then
∼ χ2 (1) and V =
∼ χ2 (1).
σ
σ/ n
σ/ n
2
n X
Xi − µ
∼ χ2 (n). Therefore, since W and V are independent,
By Lemma 4, it follows that U =
σ
i=1
then the moment generating function of U is
MU (t) = MW (t)MV (t)
(1 − 2t)−n/2 = MW (t)(1 − 2t)−1/2
=⇒ MW (t) =
(1 − 2t)−n/2
= (1 − 2t)−(n−1)/2
(1 − 2t)−1/2
The moment generating function for W is recognized as coming from a χ2 distribution with n − 1
degrees of freedom. Hence,
W =
(n − 1)S 2
∼ χ2 (n − 1)
σ2