Gu Yuxian Wang Weinan Beijing National Day School.

Gu Yuxian Wang WeinanBeijing National Day School

Research Project For Linear Regression

Part 1 The Simple Linear Regression

• Given two variables X and Y.• ， … are measured without an error• , … are measured with error • So we can let • We can use the least squares estimators and

the maximum likelihood estimator to estimate parameter and .

1x nx2x

1y ny2y

ii xY 10

0 1

The Least Squares Estimators• Let • All we need to do is to minimize Δ .

• Let ,• Solve the equation.

n

iii

n

ii yxyy

1

210

1

2 ])[()(

XX

XY

S

Sxy 0̂

XX

XY

S

S1̂

00

01

n

iiiXY yyxxS

1

))((

n

iiYY xxS

1

2)(

n

iiXX yyS

1

2)(

The Maximum Likelihood Estimator

• Assume that

• So

iiεii

iiiiY

xββyFxββyεP

yεxββPyYPyF

1010

10

2,0 ~ N

2

210

210

2

1 σ

xββy

iiεiY

ii

eσπ

xββyfyf

• The likelihood function

• Compute and

• Solve

• We get

n

i

ii xynL

12

210

22

1ln

0

01

00

1

XX

XY

S

Sxy 0̂

XX

XY

S

S1̂

Efficiency Analysis

• They are unbiased.

00

11

1011

21

10

1111

ˆ

ˆ

E

n

xn

S

x

S

xx

n

YE

S

x

S

YExE

n

ii

XX

n

ii

XX

n

ii

n

ii

n

ii

XX

n

ii

XX

n

iii

Part2 Errors-in-Variables (EIV) Regression Model

• When the measurements for X is not accurate.• There are two ways to measure errors.• The orthogonal regression and the geometric

mean regression.

The Orthogonal Regression(OR)• The distances between the regression line and points are

• To minimize Compute and solve• We are supposed to get

2

1

10

1

ii xy

n

i

ii xy

12

1

210

1

00

01

XY

YYXXXYYYXX

S

SSSSS

2

4ˆ22

1

10

ˆˆ xy

The Geometric Mean Regression(GMR)

• The area is• To minimize

Compute and solve

we get

}

2{

1 1

210

n

i

ii xy

1

210

2 ii xy

00

01

XX

YY

S

S1̂ 10

ˆˆ xy

Parametric Method

• Assume

X and Y follow a bivariate normal distribution

• We use moment generating function (mgf) to derive the distribution of X and Y :

10

Y

X

),(~

),0(~

),0(~

2

2

2

N

N

N

2101212

12

2222

12

122

02

22

221

22121

2121

02

0121

)()(2

1)(

2

1

2

1

2

1)(

2

1)(

21121

)()(21,

)(M)(M)(M

)(E),(M

tttttt

ttttttt

t

ttYX

e

eeee

etttt

ett

• Since are independent, we can separate mgf.

• The bivariate normal distribution that

• method of moment estimator(MOME)

222

12

1

21

22

10

,N~

Y

X

，，

2

2

21 1

1

21

11

2

2221

2

11

2

22

101

1

)(1

)()()(),(

)(

)(

)(

)(

n

S

n

yxyx

n

YEXEXYEYXCov

n

S

n

y

n

yYD

n

S

n

x

n

xXD

n

yyYE

n

xxXE

XY

n

i

n

iiin

iii

YY

n

ii

n

ii

XX

n

ii

n

ii

n

ii

Y

n

ii

X

• We get:

xy

S

SSSSS

XY

XYXXYYXXYY

10

22

1

ˆˆ

2

4ˆ

Special Situation for MLE • The Orthogonal Regression(OR)

• The Geometric Mean Regression (GMR)

XY

YYXXXYYYXX

XY

XYXXYYXXYY

S

SSSSS

S

SSSSS

2

4ˆ

2

4ˆ

22

11

22

1

XX

YYS

S

XY

XYXXYYXXYY

S

S

S

SSSSS

XX

YY

1

22

1

ˆ

2

4ˆ

•

–This is when Y has no error.

–This is when X has no error, so we get the same answer as our first discussion.

02

2

XY

YY

S

S1̂

XX

XY

S

S1̂ 2

2

Another Estimator • We want to (1)occupy all la (like MLE)

(2)without distributions(like (OR)&(G))

• Calculate

n

iii

n

iii

yxc

c

xxcyyc

1

2102

1

1

22

1

])())(1[(

00

01

0)1()1( 13

14

1 YYXYXYXX ScSccScS

Let

XY

XYXXYYXXYY

S

SSSSS

2

4)(

22

1

22

21

4

2)(

2

1)(

XYXXYY

XYXXXXYYXX

XY SSS

SSSSS

Sd

d

0

04

2)(

2

22

2

XYYYXX

XYXXYY

XYXXXXYYXX

SSS

SSS

SSSSS

)(1 XY

YY

XX

XY

S

S

S

S )()0( 11

],[),0[ 1-1

XY

YY

XX

XY

S

S

S

S

So is increasing and

We get

Prove 1-1 to 1

Let

So there is at least one root for

YYXYXYXX ScSccScSxf )1()1( 13

14

1

1Prove 1-1 to

0)()()(

)(

2

2

XY

YY

XX

XYYYXXXY

XY

YY

XYYYXXXX

XY

S

Sf

S

SfSSS

S

Sf

SSSS

Sf

0)1()1( 13

14

1 YYXYXYXX ScSccScS

We have

c

So there is ONLY one root for

(when )

XY

YY

XX

XY

S

S

S

S,1And when

0)1()1( 13

14

1 YYXYXYXX ScSccScS

XY

YY

XX

XY

S

S

S

S,1

Then we have

XY

YY

XX

XY

S

S

S

Sc ,]1,0[ 1

1-1

0)( 1 fWe can proof

Another Estimator Again• The angle• Let Compute & solve

• We get

• ***

cossin)sin(

sin

1

22

ddd

21

1

210

2

cossin

])[(

n

iii yx

d

00

01

cot

cot1̂

XXXY

XYYY

SS

SS

10

ˆˆ xy

cot1̂

Part3 Multiple Linear Regression

The Least Squares Estimators

• Similar to simple linear regression:

• Compute

• We will get a group of equations:

)()2(2

)1(10

niniii xxxY

n

i

niniii xxxy

1

2)()2(2

)1(10

00,010

n

0

0

0

0

111

22

1

11

10

1

2

1

2

1

323

1

222

1

121

1

20

1

1

1

1

1

313

1

212

1

121

1

10

111

22

1

110

n

ii

ni

n

i

ni

nin

n

ii

ni

n

ii

ni

n

i

ni

n

iii

n

i

niin

n

iii

n

ii

n

iii

n

ii

n

iii

n

i

niin

n

iii

n

iii

n

ii

n

ii

n

ii

n

i

nin

n

ii

n

ii

yxxxxxxxx

yxxxxxxxxx

yxxxxxxxxx

yxxxn

Assume its coefficient matrix is

The solution is

11 nnija

n

ii

ni

n

iii

n

ii

nnij

n yx

yx

y

a

1

1

1

1

1

11

1

0

Errors-in-Variables (EIV) Regression Model(Two Variables)

• The Orthogonal Regression(OR)

• The Geometric Mean Regression(GMR1)(the volume )

• The Geometric Mean Regression(GMR2)(the sum of area )

iii YXZ 210

n

i

iii yxz

1 21

3210

6

)]([

n

i

iii yxz

12

22

1

2210

1

)]([

2210

21 121

)]()[111

(2

1iii

n

i

yxz

Thanks!!!!!!