Transcript
Applied Econometrics
Department of EconomicsStern School of Business
Applied Econometrics
3. Linear Least Squares
Vocabulary Some terms to be used in the discussion.
Population characteristics and entities vs. sample quantities and analogs
Residuals and disturbances Population regression line and sample regression
Objective: Learn about the conditional mean function. Estimate and 2
First step: Mechanics of fitting a line (hyperplane) to a set of data
Fitting Criteria The set of points in the sample Fitting criteria - what are they:
LAD Least squares and so on
Why least squares? (We do not call it ‘ordinary’ at this point.)
A fundamental result: Sample moments are “good” estimators of their population counterparts We will spend the next few weeks using this
principle and applying it to least squares computation.
An Analogy Principle In the population E[y | X ] = X so E[y - X |X] = 0 Continuing E[xi i] = 0 Summing, Σi E[xi i] = Σi 0 = 0 Exchange Σi and E E[Σi xi i] = E[ X ] = 0 E[ X (y - X) ] = 0
Choose b, the estimator of to mimic this population result: i.e., mimic the population mean with the sample mean
Find b such that As we will see, the solution is the least squares coefficient
vector.
= 1 1 n nXe 0 X(y- Xb)
Population and Sample Moments We showed that E[i|xi] = 0 and Cov[xi,i]
= 0. If it is, and if E[y|X] = X, then
= (Var[xi])-1 Cov[xi,yi]. This will provide a population analog to
the statistics we compute with the data.
An updated version, 1950 – 2004 used in the problem sets.
Least Squares Example will be, yi = Gi on xi = [a constant, PGi and Yi] = [1,Pgi,Yi] Fitting criterion: Fitted equation will be yi = b1xi1 + b2xi2 + ... + bKxiK. Criterion is based on residuals: ei = yi - b1xi1 + b2xi2 + ... + bKxiK
Make ei as small as possible. Form a criterion and minimize it.
Fitting Criteria Sum of residuals: Sum of squares: Sum of absolute values of residuals: Absolute value of sum of residuals
We focus on now and later
ni ie1
ni ie1
2
ni ie1
ni ie1
21
niie
1
niie
Least Squares Algebra
21
A digression on multivariate calculus. Matrix and vector derivatives. Derivative of a scalar with respect to a vector Derivative of a column vector wrt a row vector
niie
e e = (y - Xb)'(y - Xb)
Other derivatives
Least Squares Normal Equations
2
Note: Derivative of 1x1 wrt Kx1 is a Kx1 vector.
Solution
(y - Xb)'(y - Xb) X'(y - Xb) = 0b
(1x1)/ (kx1) (-2)(nxK)'(nx1) = (-2)(Kxn)(nx1) = Kx1
: X'y = X'Xb
Least Squares Solution
-1
1
1
Assuming it exists: = ( )
Note the analogy: = Var( ) Cov( ,y)
1 1 =
Suggests something desirable about least squaresn n
b X'X X'y
x x
b X'X X'y
Second Order Conditions
2
2
=
column vector = row vector
= 2
(y - Xb)'(y - Xb) X'(y - Xb)b
(y - Xb)'(y - Xb)(y - Xb)'(y - Xb) b
b b b
X'X
Does b Minimize e’e?
21 1 1 1 2 1 1
221 2 1 1 2 1 2
21 1 1 2 1
...
...2
... ... ... ......
If there were a single b, we would require this to be
po
n n ni i i i i i i iK
n n ni i i i i i i iK
n n ni iK i i iK i i iK
x x x x xx x x x x
x x x x x
e'e X'X = 2b b'
21
sitive, which it would be; 2 = 2 0.
The matrix counterpart of a positive number is a positive definite matrix.
niix
x'x
Sample Moments - Algebra2 2
1 1 1 1 2 1 1 1 1 2 12 2
1 2 1 1 2 1 2 2 1 2 21
21 1 1 2 1 1 2
... ...
... ...=
... ... ... ... ... ... ... ......
n n ni i i i i i i iK i i i i iK
n n nni i i i i i i iK i i i i iKi
n n ni iK i i iK i i iK iK i iK i
x x x x x x x x x xx x x x x x x x x x
x x x x x x x x x
X'X =
2
1
21 1 2
1
...
= ......
=
iK
i
ini i i iK
ik
ni i i
x
xx
x x x
x
x x
Positive Definite MatrixMatrix is positive definite if is > 0for any . Generally hard to check. Requires a look at characteristic roots (later in the course). For some matrices, it is easy to verify. i
C a'Caa
X'X
K 2kk=1
= v 0
-1
s one of these.
= ( )( ) = ( )'( ) = Could = ?Conclusion: =( ) does indeed minimize .
a'X'Xa a'X X'a X'a X'a v'vv 0
b X'X X'y e'e
Algebraic Results - 1
1
nii
In the population: E[ ' ] = 1In the sample: en i
X 0
x 0
Residuals vs. Disturbances
i i i
i i i
Disturbances (population) yPartitioning : = E[ | ] +
Residuals (sample) y ePartitio
xy y y X
x
ε = conditional mean + disturbance
ning : = + y y Xb
X'
e = projection + residual
( Note: Projection 'into the column space of )
Algebraic Results - 2 The “residual maker” M = (I - X(X’X)-1X’) e = y - Xb= y - X(X’X)-1X’y = My MX = 0 (This result is fundamental!) How do we interpret this result in terms of
residuals? (Therefore) My = MXb + Me = Me = e (You should be able to prove this. y = Py + My, P = X(X’X)-1X’ = (I - M). PM = MP = 0. (Projection matrix) Py is the projection of y into the column space of
X. (New term?)
The M Matrix M = I- X(X’X)-1X’ is an nxn matrix M is symmetric – M = M’ M is idempotent – M*M = M (just multiply it out) M is singular – M-1 does not exist. (We will prove this later as a side result in
another derivation.)
Results when X Contains a Constant Term
X = [1,x2,…,xK] The first column of X is a column of ones Since X’e = 0, x1’e = 0 – the residuals sum to
zero.
+
nii=1
Define [1,1,...,1]' a column of n ones = y ny
implies (after dividing by n)y (the regression line passes through the means)These do not apply if the model has no
y Xb ei
i'yi'y i'Xb+i'e=i'Xb
x b constant term.
Least Squares Algebra
Least Squares
Residuals
Least Squares Residuals
Least Squares Algebra-3
M is nxn potentially huge
Least Squares Algebra-4
top related