Properties of the Sample Mean Consider X1, . . . , Xn independent and identically distributed (iid) with mean µ and variance σ 2. n 1P ¯ X= Xi n i=1 (sample mean) Then n 1P ¯ (X) = µ=µ n i=1 n σ2 1 P 2 ¯ σ = var(X) = 2 n i=1 n Remarks: ◦ The sample mean is an unbiased estimate of the true mean. ◦ The variance of the sample mean decreases as the sample size increases. ◦ Law of Large Numbers: It can be shown that for n → ∞ n 1P ¯ X= Xi → µ. n i=1 Question: ◦ How close to µ is the sample mean for finite n? ◦ Can we answer this without knowing the distribution of X? Central Limit Theorem, Feb 4, 2003 -1- Properties of the Sample Mean Chebyshev’s inequality Let X be a random variable with mean µ and variance σ 2. Then for any ε > 0 ¡ ¢ σ2 |X − µ| > ε ≤ 2 . ε Proof: Let 1{|xi − µ| > ε} = ½ 1 0 if |xi − µ| > ε otherwise Then o n n (x − µ)2 P i > 1 p(xi ) 1{|xi − µ| > ε} p(xi ) = 1 ε2 i=1 i=1 n (x − µ)2 P σ2 i ≤ p(x ) = i ε2 ε2 i=1 n P Application to the sample mean: ³ ´ 3σ 3σ ¯ ≤ µ + √ ≥ 1 − 1 ≈ 0.889 µ− √ ≤X n n 9 However: Known to be not very precise iid Example: Xi ∼ N (0, 1) n 1P ¯ X= Xi ∼ N (0, n1 ) n i=1 Therefore ³ ´ 3 3 ¯ − √ ≤ X ≤ √ = 0.997 n Central Limit Theorem, Feb 4, 2003 n -2- Central Limit Theorem Let X1, X2, . . . be a sequence of random variables ◦ independent and identically distributed ◦ with mean µ and variance σ 2. For n ∈ define n √ X¯ − µ Xi − µ 1 P =√ . Zn = n σ n i=1 σ Zn has mean 0 and variance 1. Central Limit Theorem For large n, the distribution of Zn can be approximated by the standard normal distribution N (0, 1). More precisely, ³ ´ √ X¯ − µ lim a≤ n ≤ b = Φ(b) − Φ(a), σ n→∞ where Φ(x) is the standard normal probability Φ(z) = Z z f (x) dx, −∞ that is, the area under the standard normal curve to left of z. Example: ◦ U1, . . . , U12 uniformly distributed on [ 0, 12). ◦ What is the probability that the sample mean exceeds 9? ³√ ¯ ´ U −6 ¯ (U > 9) = 12 √ > 3 ≈ 1 − Φ(3) = 0.0013 12 Central Limit Theorem, Feb 4, 2003 -3- Central Limit Theorem 0.4 1.0 U[0,1],n=1 0.8 density f(x) 0.3 density f(x) Exp(1),n=1 0.2 0.1 0.6 0.4 0.2 0.0 0.0 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 U[0,1],n=2 0.4 2 3 Exp(1),n=2 0.5 0.4 density f(x) density f(x) 0.3 0.2 0.3 0.2 0.1 0.1 0.0 0.0 −3 −2 −1 0 1 2 3 −3 −1 0 1 0.5 U[0,1],n=6 0.4 −2 2 3 Exp(1),n=6 0.4 density f(x) density f(x) 0.3 0.2 0.1 0.2 0.1 0.0 0.0 −3 −2 −1 0 1 2 3 −3 U[0,1],n=12 0.4 −2 −1 0 1 2 3 Exp(1),n=12 0.4 0.3 0.3 density f(x) density f(x) 0.3 0.2 0.1 0.2 0.1 0.0 0.0 −3 −2 −1 0 1 2 3 −3 −2 −1 0 1 U[0,1],n=100 0.4 2 3 Exp(1),n=100 0.4 density f(x) density f(x) 0.3 0.2 0.1 0.3 0.2 0.1 0.0 0.0 −3 −2 −1 0 Central Limit Theorem, Feb 4, 2003 1 2 3 −3 −2 −1 0 1 2 3 -4- Central Limit Theorem Example: Shipping packages Suppose a company ships packages that vary in weight: ◦ Packages have mean 15 lb and standard deviation 10 lb. ◦ They come from a arge number of customurs, i.e. packages are independent. Question: What is the probability that 100 packages will have a total weight exceeding 1700 lb? Let Xi be the weight of the ith package and T = 100 P Xi . i=1 Then T − 1500 lb 1700 lb − 1500 lb √ > √ (T > 1700 lb) = 100 · 10 lb 100 · 10 lb µ ¶ T − 1500 lb √ = >2 100 · 10 lb µ ¶ ≈ 1 − Φ(2) = 0.023 Central Limit Theorem, Feb 4, 2003 -5- Central Limit Theorem Remarks • How fast approximation becomes good depends on distribution of Xi’s: ◦ If it is symmetric and has tails that die off rapidly, n can be relatively small. iid Example: If Xi ∼ U [0, 1], the approximation is good for n = 12. ◦ If it is very skewed or if its tails die down very slowly, a larger value of n is needed. Example: Exponential distribution. • Central limit theorems are very important in statistics. • There are many central limit theorems covering many situations, e.g. ◦ for not identically distributed random variables or ◦ for dependent, but not “too” dependent random variables. Central Limit Theorem, Feb 4, 2003 -6- The Normal Approximation to the Binomial Let X be binomially distributed with parameters n and p. Recall that X is the sum of n iid Bernoulli random variables, X= n P Xi , i=1 iid Xi ∼ Bin(1, p). Therefore we can apply the Central Limit Theorem: Normal Approximation to the Binomial Distribution ¡ ¢ For n large enough, X is approximately N np, np(1 − p) distributed: ¢ ¡ ¡ 1 1 a ≤ X ≤ b) ≈ a − 2 ≤ Z ≤ b + 2 where ¡ ¢ Z ∼ N np, np(1 − p) . Rule of thumb for n: np > 5 and n(1 − p) > 5. In terms of the standard normal distribution we get µ ¶ 1 ¡ − np a − 12 − np b + p a ≤ X ≤ b) = ≤ Z0 ≤ p 2 np(1 − p) np(1 − p) µ ¶ µ ¶ b + 21 − np a − 12 − np =Φ p −Φ p np(1 − p) np(1 − p) where Z 0 ∼ N (0, 1). Central Limit Theorem, Feb 4, 2003 -7- The Normal Approximation to the Binomial Bin(1,0.5) 1.0 0.8 0.6 0.6 p(x) 0.8 p(x) Bin(1,0.1) 1.0 0.4 0.4 0.2 0.2 0.0 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 0.0 20 0 1 2 3 4 5 6 7 8 9 x 10 12 14 16 18 20 x Bin(2,0.5) 1.0 0.8 0.6 0.6 p(x) 0.8 p(x) Bin(5,0.1) 1.0 0.4 0.4 0.2 0.2 0.0 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 0.0 20 0 1 2 3 4 5 6 7 8 9 x 10 12 14 16 18 20 x Bin(5,0.5) 0.5 0.4 0.3 0.3 p(x) 0.4 p(x) Bin(10,0.1) 0.5 0.2 0.2 0.1 0.1 0.0 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 0.0 20 0 1 2 3 4 5 6 7 8 9 x 10 12 14 16 18 20 x Bin(10,0.5) 0.3 Bin(20,0.1) 0.3 p(x) 0.2 p(x) 0.2 0.1 0.0 0.1 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 0.0 20 0 1 2 3 4 5 6 7 8 9 x 10 12 14 16 18 20 x Bin(20,0.5) 0.3 Bin(50,0.1) 0.3 p(x) 0.2 p(x) 0.2 0.1 0.0 0.1 0 1 2 3 4 5 6 7 8 9 10 12 x Central Limit Theorem, Feb 4, 2003 14 16 18 20 0.0 0 1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 x -8- The Normal Approximation to the Binomial Example: The random walk of a drunkard Suppose a drunkard executes a “random” walk in the following way: ◦ Each minute he takes a step north or south, with probability 21 each. ◦ His successive step directions are independent. ◦ His step length is 50 cm. How likely is he to have advanced 10 m north after one hour? ◦ Position after one hour: X · 1 m − 30 m ◦ X binomially distributed with parameters n = 60 and p = 1 2 ◦ X is approximately normal with mean 30 and variance 15: (X · 1 m − 30 m > 10 m) = (X > 40) ≈ (Z > 39.5) µ ¶ Z − 30 9.5 √ = >√ 15 15 = 1 − Φ(2.452) = 0.007 Z ∼ N (30, 15) How does the probability change if he has same idea of where he wants to go and steps north with probability p = 23 and south with probability 31 ? Central Limit Theorem, Feb 4, 2003 -9-