1. science
2. mathematics
3. statistics

STAC67H: Regression Analysis
Fall, 2014
Instructor: Jabed Tomal
Department of Computer and Mathematical Sciences
University of Toronto Scarborough
Toronto, ON
Canada
October 9, 2014
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
1 / 32
Regression through Origin:
1
Sometimes the regression function is known to be linear and to go
through the origin at (0, 0).
2
Example 1: X is units of output and Y is variable cost, so Y is
zero by definition when X is zero.
3
Example 2: X is the number of brands of beer stocked in a
supermarket and Y is the volume of beer sales in the
supermarket.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
2 / 32
Regression through Origin:
Model
The normal error model for these cases is the same as regression
model Yi = β0 + β1 Xi + i except that β0 = 0:
Yi = β1 Xi + i
where:
1
β1 is a parameter
2
Xi are known constants
3
i are independent N(0, σ 2 ).
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
3 / 32
Regression through Origin:
The regression function is
E(Yi ) = β1 Xi
which is a straight line through the origin, with slope β1 .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
4 / 32
Regression through Origin:
Inferences
The least squares estimator of β1 is obtained by minimizing:
Q=
n
X
(Yi − β1 Xi )2
i=1
with respect to β1 .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
5 / 32
Regression through Origin:
Inferences
The resulting normal equation is:
n
X
Xi (Yi − β1 Xi ) = 0
i=1
leading to the point estimator:
P
Xi Yi
b1 = P 2
Xi
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
6 / 32
Regression through Origin:
Inferences
The maximum likelihood estimator is:
P
Xi Yi
βˆ1 = P 2
Xi
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
7 / 32
Regression through Origin:
Inferences
ˆi for the ith case is:
The fitted value Y
ˆi = b1 Xi
Y
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
8 / 32
Regression through Origin:
Inferences
The ith residual is defined as the difference between the observed and
fitted values:
ˆi = Yi − b1 Xi
ei = Yi − Y
Is
Pn
i=1 ei
zero?
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
9 / 32
Regression through Origin:
Inferences
An unbiased estimator of the error variance σ 2 is:
P 2
P
ˆi )2
ei
(Yi − Y
SSE
=
=
s = MSE =
n−1
n−1
n−1
2
The reason for the denominator n − 1 is that only one degrees of
freedom is lost in estimating the single parameter in the regression
function.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
10 / 32
Regression through Origin:
Table: Confidence Limits for Regression through Origin.
Estimate of
β1
Estimated Variance
MSE
s2 {b1 } = P
X2
Confidence Limits
b1 ± ts{b1 }
i
Xh2 MSE
P 2
Xi
E{Yh }
ˆh } =
s 2 {Y
Yh(new)
s2 {pred} = MSE 1 +
ˆh ± ts{Y
ˆh }
Y
X2
Ph 2
Xi
ˆh(new) ± ts{pred}
Y
Here, t = t(1 − α/2; n − 1)
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
11 / 32
Cautions for Using Regression through Origin:
P
Here,
ei 6= 0. Thus, in a residual plot the residuals will usually
not be balanced around the zero line.
P
Here,
SSE = ei2 may exceed the total sum of squares
P
¯ )2 . This can occur when the data form a curvilinear
(Yi − Y
pattern or a linear pattern with an intercept away from the origin.
The coefficient of determination R 2 has no clear meaning and
may turn out to be negative.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
12 / 32
Cautions for Using Regression through Origin:
Evaluate the aptness of your regression model; the regression
function may not be linear or the variance of the error terms may
not be constant.
It is generally a safe practice not to use
regression-through-the-origin model (Yi = β1 Xi + i ) and instead
use the intercept regression model (Yi = β0 + β1 Xi + i ).
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
13 / 32
Effects of Measurement Errors:
Measurement Errors in Y
Measurement errors could be present in the response variable Y .
Consider a study of the relation between the time required to
complete a task (Y ) and the complexity of the task (X ).
The time to complete the task may not be measured accurately
due to inaccurate operation of the stopwatch.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
14 / 32
Effects of Measurement Errors:
Measurement Errors in Y
If the random measurement errors on Y are uncorrelated and
unbiased, no new problems are created.
Such random, uncorrelated and unbiased measurement errors on
Y are simply absorbed in the model error term .
The model error term always reflects the composite effects of a
large number of factors not considered in the model, one of which
now would be the random variation due to inaccuracy in the
process of measuring Y .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
15 / 32
Effects of Measurement Errors:
Measurement Errors in X
Measurement errors may be present in the predictor variable X ,
for instance, when the predictor variable is presure in a tank,
temperature in an oven, speed of a production line, or reported
age of a person.
Interested in the relation between employees’ piecework earnings
and their ages.
Let Xi be the true age and Xi∗ be the reported age of the ith
employee. Hence, the measurement error is:
δi = Xi∗ − Xi
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
16 / 32
Effects of Measurement Errors:
Measurement Errors in X
The regression model under study is
Yi = β0 + β1 Xi + i
We replace Xi by the observed Xi∗
Yi = β0 + β1 (Xi∗ − δi ) + i
We rewrite the regression model as:
Yi = β0 + β1 Xi∗ + (i − β1 δi )
One can consider this model as a simple linear regression model
with predictor Xi∗ and error (i − β1 δi ). But, here, Xi∗ and
(i − β1 δi ) are not independent.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
17 / 32
Effects of Measurement Errors:
Measurement Errors in X
Let us assume that Xi∗ is an unbiased estimator of Xi
E(δi ) = E(Xi∗ ) − E(Xi ) = 0
As usual, we assume that the error terms i have expectation 0
E(i ) = 0
Let the measurement error δi and the model error i are
uncorrelated
σ{δi , i } = E{δi , i } − E{δi }E{i } = E{δi , i } = 0
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
18 / 32
Effects of Measurement Errors:
Measurement Errors in X
σ{Xi∗ , i − β1 δi } =
=
=
=
=
∗
E Xi∗ − E(X
1 δi ) − E{i − β1 δi }]
i ) [(i − β
E Xi∗ − Xi (i − β1 δi )
E {δ
i (i − β1 δi)}
E δi i − β1 δi2
−β1 σ 2 {δi }
This covariance is not zero whenever there is a linear regression
relation between X and Y .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
19 / 32
Effects of Measurement Errors:
Measurement Errors in X
If we assume that the response Y and the predictor X ∗ follow a
bivariate normal distribution, then the conditional distribution
Yi |Xi∗ , i = 1, · · · , n, are independent normal with mean
E{Yi |Xi∗ } = β0∗ + β1∗ Xi∗
and variance
σY2 |X .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
20 / 32
Effects of Measurement Errors:
Measurement Errors in X
It can be shown that
h
i
β1∗ = β1 σX2 /(σX2 + σY2 )
where σX2 is the variance of X and σY2 is the variance of Y .
Hence, the least squares slope estimate from fitting Y and X ∗ is
not an estimate of β1 , but is an estimate of β1∗ ≤ β1 .
If σY2 is small relative to σX2 , then the bias would be small;
otherwise the bias may be substantial.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
21 / 32
Effects of Measurement Errors:
Measurement Errors in X
One of the approaches to deal with measurement errors in X is to
use additional variables that are known to be related to the true
value of X but not to the errors of measurement δ. Such variables
are called instrumental variables because they are used as an
instrument in studying the relation between X and Y .
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
22 / 32
Simple Linear Regression Model in Matrix Terms:
Consider, the normal error regression model
Yi = β0 + β1 Xi + i
; i = 1, · · · , n
We will not present any new results, but shall only state in matrix
terms the results obtained earlier.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
23 / 32
Simple Linear Regression Model in Matrix Terms:
We write the limple linear regression model in matrix terms as
following
Y = X
n×1
β + n×2 2×1
n×1
where


Y1
Y2 
 
Y = . 
n×1
 .. 
Yn
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
24 / 32
Simple Linear Regression Model in Matrix Terms:


1 X1
1 X2 


X = . . 
n×2
 .. .. 
1 Xn
β
β = 0
β1
2×1
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
25 / 32
Simple Linear Regression Model in Matrix Terms:
 
1
2 
 
=.
n×1
 .. 
n
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
26 / 32
Simple Linear Regression Model in Matrix Terms:
The vector of expected values of the Yi observations is
E(Y) = Xβ
n×1
n×1
The condition E{i } = 0 in matrix terms is:
E{} = 0
n×1
Jabed Tomal (U of T)
n×1
Regression Analysis
October 9, 2014
27 / 32
Simple Linear Regression Model in Matrix Terms:
The condition that the error terms have constant variance σ 2 and
that all covariances σ{i , j } for i 6= j are zero is expressed in
matrix terms as following
 2

σ
0 0 ··· 0
 0 σ2 0 · · · 0 


2


Var{} =  0 0 σ · · · 0  = σ 2 I
n×n
 ..
..
.. . .
.
n×n
.
. .. 
.
.
0
Jabed Tomal (U of T)
0
0
Regression Analysis
· · · σ2
October 9, 2014
28 / 32
Simple Linear Regression Model in Matrix Terms:
Thus the normal error regression model in matrix terms is
Y = Xβ + where, is a vector of independent normal random variables with
E{} = 0 and Var{} = σ 2 I.
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
29 / 32
Least Squares Estimation of Regression Parameters:
The normal equations in matrix terms are
X0 X b = X0 Y
2×2 2×1
2×1
where, b is the vector of the least squares regression coefficients:
b
b = 0
b1
2×1
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
30 / 32
Least Squares Estimation of Regression Parameters:
Estimated Regression Coefficients
The estimated regression coefficients are
b = X0 X
2×1
Jabed Tomal (U of T)
−1
2×2
Regression Analysis
X0 Y
2×1
October 9, 2014
31 / 32
Least Squares Estimation of Regression Parameters:
Exercise 5.6 Refer to Airfreight breakage Problem 1.21. Using
matrix methods, find (1) Y0 Y, (2) X0 X, (3) X0 Y, and (4) b.
1
Y0 Y = 2194
2
10 10
XX=
10 20
0
3
142
XY=
182
0
4
10.2
b=
4.0
Jabed Tomal (U of T)
Regression Analysis
October 9, 2014
32 / 32