1. science
2. mathematics
3. statistics

İnsan TUNALI
Econ 311 – Econometrics I
11 February 2014
Lectures 3-5
Revised Feb. 20th
REVIEW OF PROBABILITY AND STATISTICS
(a) Population, random variable, and distribution
(b) Moments of a distribution (mean, variance, standard deviation,
covariance, correlation)
Stock & Watson, Ch.2 [Goldberger Ch.3-4]
1.
2.
3.
4.
The probability framework for statistical inference
(c) Conditional distributions and conditional means
The probability framework for statistical inference
Estimation
Testing
Confidence Intervals
(d) Distribution of a sample of data drawn randomly from a
YBnB (subject of another handout).
population: YB1,…,
B
From the syllabus: “The prerequisites for ECON 311 include MATH 201
(Statistics), and ECON 201 (Intermediate Microeconomics). Students who got a
grade of C− or below in MATH 201 are strongly advised to work independently to
make up for any deficiency they have during the first two weeks of the semester.”
2/36
(a) Population, random variable, and distribution
Population
How to envision
Discrete Probability Distributions
• The group or collection of all possible entities of interest
Urn model: Population = Balls in an urn
• We will think of populations as being “very big” (∞ is an
approximation to “very big”)
Outcomes... sample space... events... MATH 201
Random variable Y
Each ball has a value (Y) written on it;
• Numerical summary of a random outcome
Population distribution of Y
Y has K distinct values: y1, y2, ..., yi, ..., yK.
Suppose we were to sample from this (univariate) population, with
replacement, infinitely many times…
• Discrete case: The probabilities of different values of Y that
occur in the population
• Continous case: Likelihood of particular ranges of Y
3/36
4/36
(Population) Distribution of Y:
pi = Pr(Y = yi), i = 1, 2, …, K.
Cumulative Distribution Function (c.d.f) of Y:
F(y) = Pr(Y ≤ y) = ∑ f ( yi ) = ∑ pi .
Gives the proportion of times we encounter a ball with value Y = yi,
i = 1, 2, …, K.
Alternate notation: f(y) = Pr(Y = y); “probability function (p.f.) of Y.”
Clearly pi ≥ 0 for all i, and Σi pi = 1.
Convention: Σi ≡
Examples:
> Gender: M (=0)/F (=1). We prefer the numerical representation…
> Standing: freshman (=1), sophomore (=2), junior (=3), senior
(=4).
yi ≤ y
yi ≤ y
That is, to find F(y) we sum the pi’s up to the value p = Pr(Y = y).
Use of the c.d.f.:
Pr(a < Y ≤ b) = F(b) – F(a).
Important features of the distribution of Y:
(Population) Mean of Y:
µY = E(Y) = Σi yi pi. (Here Σi ≡
Also known as “the expected value of Y” or simply “expectation of Y.”
Remark: Expectation is a weighted average of the values of Y where
weights are the probabilities with which distinct values occur.
> Ranges of wages (group them in intervals first).
5/36
The idea of “weighted averaging” can be extended to functions of Y.
Suppose Z = h(Y), any function of Y. Then the expected value of h(Y)
is:
E(Z) = E[h(Y)]
= Σi h(yi) pi.
Thus knowledge of the probability distribution of Y is sufficient for
calculating the expectation of functions of Y as well.
With this choice of h(Y), we get the:
(Population) Variance of Y:
σ Y2 = V(Y) = E[(Y– µY)2]
= Σi (yi – µY )2 pi.
In words, variance equals the expected value of (or the expectation
of)“the squared deviation of Y from its mean.”
Example: Suppose random variable Y can take on one of two values,
y0 = 0 and y1 = 1 with probabilities p0 and p1.
Examples:
(i) Take Z = Y2. Then
Since p0 + p1 =1, we may take
2
2
6/36
E(Z) = E(Y ) = Σi yi pi.
Pr(Y = 1) = p and Pr(Y = 0) = 1 – p, 0 < p < 1.
2
(ii) Take Z = (Y– µY) . Then
E(Z) = E[(Y– µY)2] = Σi (yi – µY )2 pi.
We say Y has a “Bernoulli distribution with parameter p” and
write: Y ~ Bernoulli (p).
7/36
8/36
Useful algebra: Let Y* = Y – µY, deviation of Y from its population
mean.
For Y ~ Bernoulli (p):
µY = E(Y) = Σi yi pi = (0)(1 – p) + (1)(p) = p;
This function is linear in Y, as in (iii), where a = – µY, and b = 1.
E(Y2) = Σi y2i pi = (0)2(1 – p) + (1)2(p) = p;
σ Y2 = V(Y) = Σi (yi – µY )2pi
= (0 – p)2(1 – p) + (1 – p)2(p) = … = p(1 – p).
///
(iii) Linear functions: Take Z = a + bY, where a and b are constants.
E(Z) = E(a + bY)
= Σk (a + byk) pk = a Σk pk + b Σk yk pk
= a + b E(Y).
In words, expectation of a linear function (of Y) equals the linear
function of the expectation (of Y).
E(Y*) = – µY + (1)E(Y) = 0.
In words, expectation of a deviation around the mean is zero.
Next, examine Y*2 = (Y – µY)2 = Y2 + µY2 – 2YµY next; this
function is not linear in Y.
E(Y*2) = E(Y2 + µY2 – 2YµY)
(*)
= E(Y2) + µY2 – 2µY E(Y)
2
2
2
2
= E(Y ) + µY – 2µY = E(Y ) – [E(Y)]2.
In line (*) we exploited the fact that E(.), which involves weighted
averaging, is a “linear” operator; thus expectation of a sum equals
the sum of expectations.
9/36
From (ii), V(Y) = E(Y*2); thus
V(Y) = E(Y2) – [E(Y)]2.
In words, variance of Y equals “expectation of squared Y” minus
“square of expected Y”.
10/36
Exercise: Well-drilling project.
Finally, let Z = a + bY as in (iii), and consider the deviation of Z from
its mean:
Z* = Z – E(Z)
= a + bY – [a + bE(Y)] = bY*.
It follows that the variance of Z is related to the variance of Y via:
V(Z) = E(Z*2)
= E[(bY*)2] = E(b2Y*2) = b2 E(Y*2) = b2V(Y).
In words, variance of a linear function equals slope squared times
the variance
11/36
Based on previous experience, a contractor believes he will find
water within 1-5 days, and attaches a probability to each possible
outcome. Let T denote the (random amount of) time it takes to
complete drilling. The probability distribution (p.f.) of T is:
t = time (days)
Pr(T = t) = fT(t)
1
0.1
2
0.2
3
0.3
4
0.3
5
0.1
(i) Find the cumulative distribution function (c.d.f.) of T and interpret
it.
t = time (days)
FT(t) =
1
2
3
4
5
12/36
(ii) Find the expected duration of the project and interpret the number
you find.
The contractors’s total project cost is made up of two parts: A fixed
cost of TL2,000, plus TL 500 for each day taken to complete the
drilling.
Prediction: Consider the urn model, where population consists of
balls in an urn. A ball is picked at random. Your task is to guess the
value Y written on it. What would your guess be?
Example: Suppose you had to predict how long a particular well
drilling project would take. What would your guess be?
One of the possible values of T?
Some other number?
(iii) Find the expected total project cost.
(iv) Find the variance of the project cost.
13/36
We need more structure. Clearly, prediction is subject to error.
Errors can be costly, and large errors can be more costly.
14/36
Proof: Can use calculus.
E(U2) = E[(Y – c)2]) = Σi (yi – c)2 pi.
What is the cost of a poor prediction?
Differentiation yields:
Let “c” be your guess (a number).
∂ E(U2)/∂c = Σi ∂[(yi – c)2 pi]/∂c = Σi [–2(yi – c)pi].
Define prediction error as U = Y – c.
We would like to make U small. Since Y is a random variable, U is also a
random variable.
Setting the derivative to zero yields the first order condition (F.O.C.)
for a minimum: Σi [–2(yi – c)pi] = 0.
More definitions:
That is,
E(U) = E(Y) – c = bias of your guess (“c”).
E(U2) = E[(Y – c)2]) = mean (expected) squared error of guess c.
Mean Squared Prediction Error criterion: Suppose the objective is to
minimize E(U2). Then the best predictor (guess) is c = µY = E(Y).
15/36
Σi yi pi = c Σi pi.
We know: Σi pi = 1, and Σi yi pi = E(Y), so the solution is c = µY.
(Check the second order condition to verify that we located a minimum.)///
16/36
Non-Calculus proof: For brevity let µ = µY and reexamine the
prediction error:
Remarks:
(i)
If we use the mean squared prediction error, then the
population mean (that is, the expectation of the random
variable) is the best guess (predictor) of a draw from that
population (distribution).
(ii) Variance equals the value of the expected squared prediction
error when the population mean is used as the predictor.
(iii) Other criteria may yield different choices of best predictor.
For example, if the criterion were minimization of the
expected absolute prediction error, namely E(|U|), then the
population median would be the best predictor.
*
U = Y – c = Y – µ – (c – µ) = Y – (c – µ),
where Y* = Y – µ as usual. Square both sides and expand:
U2 = [Y* – (c – µ)]2 = Y*2 + (c – µ)2 – 2Y*(c – µ).
Take expectations, and recall “useful algebra”:
E(U2) = E[Y*2 + (c – µ)2 – 2Y*(c – µ)]
= E(Y*2) + (c – µ)2 – 2(c – µ)E(Y*)
= V(Y) + (c – µ)2.
Since V(Y) > 0 and (c – µ)2 ≥ 0, minimum E(U2) is obtained by
setting c = µ. ///
17/36
18/36
Joint (population) distribution of X and Y:
How to envision
⎧ Joint ⎫
⎪
⎪
⎨ Marginal ⎬
⎪Conditional⎪
⎩
⎭
pjk = Pr(X = xj, Y = yk), j = 1, 2, …, J; k = 1, 2, …, K.
Gives the proportion of times we encounter a ball with paired values
(xj, yk), j = 1, 2, …, J; k = 1, 2, …, K.
Probability Distributions
The joint distribution classifies the balls according to values of both
X and Y. To obtain a “marginal” distribution, we reclassify the balls
in the urn according to the distinct values of one “margin”. We
ignore the distinct values of the second margin.
Urn model: Population = Balls in an urn
Bivariate population: Each ball has a pair of values (X, Y) written on it.
Marginal (population) distribution of X:
pj = Pr(X = xj), j = 1, 2,…, J.
X has J distinct values: x1, x2, ..., xj, ..., xJ.
Here we ignore the values of Y, and examine the proportion of times
we encounter a ball with values xj, j = 1, 2,…, J.
Y has K distinct values: y1, y2, ..., yk, ..., yK.
19/36
20/36
How to obtain a marginal distribution of X from the joint distribution
of X and Y:
pj = Σk pjk, j = 1, 2,…, J.
(Stock & Watson)
(Population) Mean of X:
µX = E(X) = Σj xj pj.
(Population) Variance of X:
σ X2 = V(X) = Σj (xj – µX )2 pj.
The marginal distribution of Y, its mean and variance may be
obtained in analogous fashion (write down the formula!).
Exercise: Consider S&W Table 2.3, Panel A. Verify the derivation
of the marginal distributions of A and M; find their means and
variances.
21/36
To obtain a “conditional” distribution, we first sort the balls
according to one of the two values, and put them in different urns.
We then examine the contents of a specific urn.
22/36
These conditional distributions may be different (hence each
subpopulation may have a different mean and variance). We can
distinguish between them as long as we record the distinct value of X
for that urn.
To obtain the conditional distributions of Y given X we sort on
distinct values xj:
Conditional (population) distribution of Y given X = xj:
POPULATION
pk|j = Pr(Y = yk | X = xj) =
p jk
pj
, k = 1, 2, …, K.
The derivation requires pj > 0.
SUBPOPULATIONS
....
X = x1
X = x2
Conditional (population) mean of Y given X = xj:
µY|j = E(Y | X = xj) = Σk yk pk|j.
....
X = xj
Conditional (population) variance of Y given X = xj:
X = xJ
σ Y2| j = V(Y | X = xj) = Σk (yk – µY|j)2 pk|j.
Each urn has a distribution of values of Y!
23/36
24/36
The conditional distributions of X given Y = yk, and their conditional
means and variances may be obtained in analogous fashion (write
down the formula you would use!).
Exercises: Verify the derivation in S&W Table 2.3, Panel B.
The Law of Iterated Expectations:
We saw that “expectation” is weighted average. As a consequence:
µY = E(Y) = Σj E(Y | X = xj) Pr(X = xj).
We may write: E(Y) = EX[E(Y|X)].
Practical uses of conditional expectations:
• Consider the conditional distributions given in S&W Table 2.3
(p.70). Suppose you have an old computer. How would you
justify buying a new computer?
Observe that:
• The “inner” expectation E(Y|X) is a weighted average of the
different values of y, weighed by conditional probabilities Pr(Y =
yk| X = xj) (here X is “given”, we know the urn balls come from).
Hint: Calculate the benefit (reduction in expected crashes) of
switching from an old computer to a new one.
• Consider the urn model. We obtain a random draw from the joint
distribution of (X, Y). We tell you the value of X. What is your
best guess of the value of Y?
• The “outer” expectation EX[.] is a weighted average of the
different values of E(Y|X = xj), weighed by probabilities Pr(X = xj).
Exercise: Earlier we used the marginal distribution of M to calculate
E(M). Can you think of another way to compute E(M)? (see S&W:72)
25/36
26/36
Functions of jointly distributed random variables:
(Population) covariance:
Let Z = h(X, Y), a function of two random variables, X and Y.
Suppose the joint distribution of X and Y is known.
In a joint distribution, the degree to which two random variables
are related may be measured with the help of covariance:
Cov(X, Y) = σXY = E(X*Y*) = E[(X – µX)( Y – µY)]
Then the expectation of Z can be computed in the usual manner,
as a weighted average:
= Σj Σk (xj – µX )(yk – µY ) Pr(X = xj, Y = yk).
Remark: We took Z = h(X, Y) = (X – µX)(Y – µY) and found E(Z)…
E(Z) = E[h(X, Y)] = Σj Σj h(xi, yj) Pr(X = xj, Y = yj)
= Σj Σj h(xi, yj) pij
(☯)
where the probability weights pij, i = 1, 2,…, I and j = 1, 2,…, J are
obtained from the joint distribution.
Exercise: Use S&W Table 2.3 to compute E(MA).
Useful algebra:
E(X*Y*) = E[(X – µX)( Y – µY)]
=…
= E(XY) – E(X)E(Y). ( )
In words, covariance equals the expected value of the product, minus
the product of the expectations.
27/36
28/36
(Population) covariance cont’d:
(Population) correlation:
The “sign” of covariance is informative about the nature of the
relation:
The magnitude of covariance is affected by the units of measurement
of the variables. For a unit-free measure, we turn to correlation:
,
If above average values of X go together with above average
values of Y (so that below average values of X go together with
below average values of Y) covariance will be positive.
It can be shown that –1 ≤ ρXY ≤ 1.
If above average values of one variable go together with below
average values of the other, covariance will be negative.
Random variables are said to be uncorrelated if ρXY = 0. Clearly
for this to happen, σXY = 0 must hold.
Corr(X, Y) = ρXY =
=
.
Recall that in general E(Y|X) is a function of X; it tells us how the
conditional mean of Y given X = xj changes with xj, j = 1, 2, …, J.
Exercise: Suppose X = weight, Y = height of individuals in a
population. Can you guess the sign of Cov(X, Y) = ?
Remark: Think about the urn model. Think about prediction.
29/36
Suppose E(Y|X) = E(Y) = µY, a constant. To describe this case, we say
Y is mean-independent of X.
Claim 1: If Y is mean-independent of X, then σXY = 0 (
Proof:
ρXY = 0).
E(XY) = E(YX) = EX[E(YX|X)] = EX[E(Y|X)X];
30/36
CAUTION: If σXY = 0, it does not follow that E(Y|X) = constant.
Covariance/correlation capture the linear relation between X and Y.
It could be that the relation is non-linear, so that E(Y|X) varies
with X, but yet σXY = 0.
Example: Modify the joint distribution in Assignment 2 Part II as:
*When we “condition” on X, we set it equal to a particular value.
If E(Y|X) = E(Y), the last expression simplifies:
f(x, y)
y=1
y=2
= EX[E(Y)X] = E(Y)E(X).
We showed:
If Y is mean-independent of X, then E(XY) = E(Y)E(X).
x = –1
0.20
0.10
x=0
0.10
0.30
x =1
0.20
0.10
and (re)calculate Cov(X, Y).
Return to ( ) and note that σXY = 0 iff E(XY) = E(X)E(Y).
Thus σXY = 0… ///
31/36
32/36
Independence: Random variables X and Y are (statistically)
independent, if knowledge of the value of one of the variables
provides no information about the other. Formally:
Claim 2: If X and Y are independently distributed, then
E(X|Y) = E(X) and E(Y|X) = E(Y).
Proof: E(Y | X = xj) = Σk yk pk|j
I.1. X and Y are independently distributed if, for all values of x and y,
Pr(Y = y | X = x) = Pr(Y = y). = Σk yk
p jk
pj
= Σk yk (pjpk/pj) = Σk yk pk = E(Y). ///
SUMMARY:
From the definition of conditional probabilities,
Independence
Pr(Y = y, X = x) = Pr(Y = y | X = x)Pr(X = x).
Thus an equivalent condition for, and implication of independence is:
I.2. Pr(Y = y, X = x) = Pr(Y = y)Pr(X = x), for all values of x and y. Mean-independence
Zero correlation.
However: We cannot go from right to left!
Stronger condition implies the weaker condition; not the other way
around.
33/36
Additional Linear Function Rules: (S&W Appendix 2.1)
34/36
Generalizing to linear functions, if
Suppose Z = X + Y. Then using (☯), it is easy to show
Z = a + bX + cY
E(Z) = E(X) + E(Y).
where a, b and c are constants, then
In words, expectation of a sum equals the sum of expectations.
Continuing, if Z = X + Y, then Z* = X* + Y*, and Z*2 = X*2 + Y*2 + 2X*Y*,
where the asterisk denotes the deviation from the expectation. So
V(Z) = E(Z*2) = E(X*2) + E(Y*2) + 2E(X*Y*)
E(Z) = a + bE(X) + cE(Y),
so the deviation from the expectation is Z* = bX* + cY*, and the
variance of Z is
V(Z) = E(Z*2) = b2V(X) + c2V(Y) + 2bcC(X,Y).
= V(X) + V(Y) + 2C(X,Y).
In words, variance of a sum equals the sum of the variances plus twice
the covariance.
Still more generally, for a pair of random variables
Z1 = a1 + b1X + c1Y,
Z2 = a2 + b2X + c2Y,
where a’s b’s and c’s are constants, the covariance of Z1 and Z2 is
Exercise: Use the same logic to find the variance of a difference.
35/36
C(Z1, Z2) = b1b2V(X) + c1c2V(Y) + (b1c2 + b2c1)C(X,Y).
36/36