Econometrics Module 3

Applied Econometrics

Department of EconomicsStern School of Business

Applied Econometrics

3. Linear Least Squares

Vocabulary Some terms to be used in the discussion.

Population characteristics and entities vs. sample quantities and analogs

Residuals and disturbances Population regression line and sample regression

Objective: Learn about the conditional mean function. Estimate and 2

First step: Mechanics of fitting a line (hyperplane) to a set of data

Fitting Criteria The set of points in the sample Fitting criteria - what are they:

LAD Least squares and so on

Why least squares? (We do not call it ‘ordinary’ at this point.)

A fundamental result: Sample moments are “good” estimators of their population counterparts We will spend the next few weeks using this

principle and applying it to least squares computation.

An Analogy Principle In the population E[y | X ] = X so E[y - X |X] = 0 Continuing E[xi i] = 0 Summing, Σi E[xi i] = Σi 0 = 0 Exchange Σi and E E[Σi xi i] = E[ X ] = 0 E[ X (y - X) ] = 0

Choose b, the estimator of to mimic this population result: i.e., mimic the population mean with the sample mean

Find b such that As we will see, the solution is the least squares coefficient

vector.

= 1 1 n nXe 0 X(y- Xb)

Population and Sample Moments We showed that E[i|xi] = 0 and Cov[xi,i]

= 0. If it is, and if E[y|X] = X, then

= (Var[xi])-1 Cov[xi,yi]. This will provide a population analog to

the statistics we compute with the data.

An updated version, 1950 – 2004 used in the problem sets.

Least Squares Example will be, yi = Gi on xi = [a constant, PGi and Yi] = [1,Pgi,Yi] Fitting criterion: Fitted equation will be yi = b1xi1 + b2xi2 + ... + bKxiK. Criterion is based on residuals: ei = yi - b1xi1 + b2xi2 + ... + bKxiK

Make ei as small as possible. Form a criterion and minimize it.

Fitting Criteria Sum of residuals: Sum of squares: Sum of absolute values of residuals: Absolute value of sum of residuals

We focus on now and later

ni ie1

ni ie1

2

ni ie1

ni ie1

21

niie

1

niie

Least Squares Algebra

21

A digression on multivariate calculus. Matrix and vector derivatives. Derivative of a scalar with respect to a vector Derivative of a column vector wrt a row vector

niie

e e = (y - Xb)'(y - Xb)

Other derivatives

Least Squares Normal Equations

2

Note: Derivative of 1x1 wrt Kx1 is a Kx1 vector.

Solution

(y - Xb)'(y - Xb) X'(y - Xb) = 0b

(1x1)/ (kx1) (-2)(nxK)'(nx1) = (-2)(Kxn)(nx1) = Kx1

: X'y = X'Xb

Least Squares Solution

-1

1

1

Assuming it exists: = ( )

Note the analogy: = Var( ) Cov( ,y)

1 1 =

Suggests something desirable about least squaresn n

b X'X X'y

x x

b X'X X'y

Second Order Conditions

2

2

=

column vector = row vector

= 2

(y - Xb)'(y - Xb) X'(y - Xb)b

(y - Xb)'(y - Xb)(y - Xb)'(y - Xb) b

b b b

X'X

Does b Minimize e’e?

21 1 1 1 2 1 1

221 2 1 1 2 1 2

21 1 1 2 1

...

...2

... ... ... ......

If there were a single b, we would require this to be

po

n n ni i i i i i i iK

n n ni i i i i i i iK

n n ni iK i i iK i i iK

x x x x xx x x x x

x x x x x

e'e X'X = 2b b'

21

sitive, which it would be; 2 = 2 0.

The matrix counterpart of a positive number is a positive definite matrix.

niix

x'x

Sample Moments - Algebra2 2

1 1 1 1 2 1 1 1 1 2 12 2

1 2 1 1 2 1 2 2 1 2 21

21 1 1 2 1 1 2

... ...

... ...=

... ... ... ... ... ... ... ......

n n ni i i i i i i iK i i i i iK

n n nni i i i i i i iK i i i i iKi

n n ni iK i i iK i i iK iK i iK i

x x x x x x x x x xx x x x x x x x x x

x x x x x x x x x

X'X =

2

1

21 1 2

1

...

= ......

=

iK

i

ini i i iK

ik

ni i i

x

xx

x x x

x

x x

Positive Definite MatrixMatrix is positive definite if is > 0for any . Generally hard to check. Requires a look at characteristic roots (later in the course). For some matrices, it is easy to verify. i

C a'Caa

X'X

K 2kk=1

= v 0

-1

s one of these.

= ( )( ) = ( )'( ) = Could = ?Conclusion: =( ) does indeed minimize .

a'X'Xa a'X X'a X'a X'a v'vv 0

b X'X X'y e'e

Algebraic Results - 1

1

nii

In the population: E[ ' ] = 1In the sample: en i

X 0

x 0

Residuals vs. Disturbances

i i i

i i i

Disturbances (population) yPartitioning : = E[ | ] +

Residuals (sample) y ePartitio

xy y y X

x

ε = conditional mean + disturbance

ning : = + y y Xb

X'

e = projection + residual

( Note: Projection 'into the column space of )

Algebraic Results - 2 The “residual maker” M = (I - X(X’X)-1X’) e = y - Xb= y - X(X’X)-1X’y = My MX = 0 (This result is fundamental!) How do we interpret this result in terms of

residuals? (Therefore) My = MXb + Me = Me = e (You should be able to prove this. y = Py + My, P = X(X’X)-1X’ = (I - M). PM = MP = 0. (Projection matrix) Py is the projection of y into the column space of

X. (New term?)

The M Matrix M = I- X(X’X)-1X’ is an nxn matrix M is symmetric – M = M’ M is idempotent – M*M = M (just multiply it out) M is singular – M-1 does not exist. (We will prove this later as a side result in

another derivation.)

Results when X Contains a Constant Term

X = [1,x2,…,xK] The first column of X is a column of ones Since X’e = 0, x1’e = 0 – the residuals sum to

zero.

+

nii=1

Define [1,1,...,1]' a column of n ones = y ny

implies (after dividing by n)y (the regression line passes through the means)These do not apply if the model has no

y Xb ei

i'yi'y i'Xb+i'e=i'Xb

x b constant term.

Least Squares Algebra

Least Squares

Residuals

Least Squares Residuals

Least Squares Algebra-3

M is nxn potentially huge

Least Squares Algebra-4