Linear, Ridge Regression, and Principal Component Analysis Linear, Ridge Regression, and Principal Component Analysis Jia Li Department of Statistics The Pennsylvania State University Email: [email protected]http://www.stat.psu.edu/∼jiali Jia Li http://www.stat.psu.edu/∼jiali
34
Embed
Linear, Ridge Regression, and Principal Component Analysis
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Linear, Ridge Regression, and Principal Component Analysis
Linear, Ridge Regression, and PrincipalComponent Analysis
Jia Li
Department of StatisticsThe Pennsylvania State University
Linear, Ridge Regression, and Principal Component Analysis
Introduction to Regression
I Input vector: X = (X1,X2, ...,Xp).
I Output Y is real-valued.
I Predict Y from X by f (X ) so that the expected loss function
E (L(Y , f (X )))
is minimized.
I Square loss:L(Y , f (X )) = (Y − f (X ))2 .
I The optimal predictor
f ∗(X ) = argminf (X )E (Y − f (X ))2
= E (Y | X ) .
I The function E (Y | X ) is the regression function.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Example
The number of active physicians in a Standard MetropolitanStatistical Area (SMSA), denoted by Y , is expected to be relatedto total population (X1, measured in thousands), land area (X2,measured in square miles), and total personal income (X3,measured in millions of dollars). Data are collected for 141SMSAs, as shown in the following table.
I The loss function E (Y − f (X ))2 is approximated by theempirical loss RSS(β)/N:
RSS(β) =N∑
i=1
(yi − f (xi ))2 =
N∑i=1
(yi − β0 −p∑
j=1
xijβj)2 .
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Notation
I The input matrix X of dimension N × (p + 1):1 x1,1 x1,2 ... x1,p
1 x2,1 x2,2 ... x2,p
... ... ... ... ...1 xN,1 xN,2 ... xN,p
I Output vector y:
y =
y1
y2
...yN
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
I The estimated β is β.
I The fitted values at the training inputs:
yi = β0 +
p∑j=1
xij βj
and
y =
y1
y2
...yN
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Point Estimate
I The least square estimation of β is�� ��β = (XTX)−1XTy
I The fitted value vector is�� ��y = Xβ = X(XTX)−1XTy
I Hat matrix: �� ��H = X(XTX)−1XT
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Geometric Interpretation
I Each column of X is a vector in an N-dimensional space (NOTthe p-dimensional feature vector space).
X = (x0, x1, ..., xp)
I The fitted output vector y is a linear combination of thecolumn vectors xj , j = 0, 1, ..., p.
I y lies in the subspace spanned by xj , j = 0, 1, ..., p.
I RSS(β) =‖ y − y ‖2.
I y − y is perpendicular to the subspace, i.e., y is the projectionof y on the subspace.
I The geometric interpretation is very helpful for understandingcoefficient shrinkage and subset selection.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Example Results for the SMSA ProblemI Yi = −143.89 + 0.341Xi1 − 0.0193Xi2 + 0.255Xi3.I RSS(β) = 52942336.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
If the Linear Model Is True
I E (Y | X ) = β0 +∑p
j=1 Xjβj
I The least square estimation of β is unbiased,
E (βj) = βj j = 0, 1, ..., p .
I To draw inferences about β, further assume:
Y = E (Y | X ) + ε
where ε ∼ N(0, σ2) and is independent of X .
I Xij are regarded as fixed, Yi are random due to ε.
I Estimation accuracy: Var(β) = (XTX)−1σ2 .
I Under the assumption, β ∼ N(β, (XTX)−1σ2) .
I Confidence intervals can be computed and significant testscan be done.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Gauss-Markov Theorem
I Assume the linear model is true.
I For any linear combination of the parameters β0,...,βp,denoted by θ = aTβ, aT β is an unbiased estimation since β isunbiased.
I The least squares estimate of θ is
θ = aT β
= aT (XTX)−1Xy
, aTy ,
which is linear in y.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
I Suppose cTy is another unbiased linear estimate of θ, i.e.,E (cTy) = θ.
I The least square estimate yields the minimum variance amongall linear unbiased estimate.
Var(aTy) ≤ Var(cTy) .
I βj , j = 0, 1, ..., p are special cases of aTβ, where aT only hasone non-zero element that equals 1.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Subset Selection and Coefficient Shrinkage
I Biased estimation may yield better prediction accuracy.
I Squared loss: E (β − 1)2 = Var(β). For β = βa , a ≥ 1,
E (β − 1)2 = Var(β) + (E (β)− 1)2 = 1a2 + (1
a − 1)2.
I Practical consideration: interpretation. Sometimes, we are notsatisfied with a “black box”.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Assume β ∼ N(1, 1). The squared error loss is reduced by shrinking the
estimation.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Subset Selection
I To choose k predicting variables from the total of p variables,search for the subset yielding minimum RSS(β).
I Forward stepwise selection: start with the intercept, thensequentially adds into the model the predictor that mostimproves the fit.
I Backward stepwise selection: start with the full model, andsequentially deletes predictors.
I How to choose k: stop forward or backward stepwise selectionwhen no predictor produces the F -ratio statistic greater thana threshold.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Ridge Regression
Centered inputs
I Suppose xj , j = 1, ..., p, are mean removed.
I β0 = y =∑N
i=1 yi/N.
I If we remove the mean of yi , we can assume
E (Y | X ) =
p∑j=1
βjXj
I Input matrix X has p (rather than p + 1) columns.
I β = (XTX)−1XTy
I y = X(XTX)−1XTy
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Singular Value Decomposition (SVD)
I If the column vectors of X are orthonormal, i.e., the variablesXj , j = 1, 2, ..., p, are uncorrelated and have unit norm.
I βj are the coordinates of y on the orthonormal basis X.
I In general �� ��X = UDVT .
I U = (u1,u2, ...,up) is an N × p orthogonal matrix. uj ,j = 1, ..., p form an orthonormal basis for the space spanned bythe column vectors of X.
I V = (v1, v2, ..., vp) is an p × p orthogonal matrix. vj ,j = 1, ..., p form an orthonormal basis for the space spanned bythe row vectors of X.
I D = diag(d1, d2, ..., dp), d1 ≥ d2 ≥ ... ≥ dp ≥ 0 are thesingular values of X.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Principal Components
I The sample covariance matrix of X is
S = XTX/N .
I Eigen decomposition of XTX:
XTX = (UDVT )T (UDVT )
= VDUTUDVT
= VD2VT
I The eigenvectors of XTX, vj , are called principal componentdirection of X.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
I It’s easy to see that zj = Xvj = ujdj . Hence uj , is simply theprojection of the row vectors of X, i.e., the input predictorvectors, on the direction vj , scaled by dj . For example
z1 =
X1,1v1,1 + X1,2v1,2 + · · ·+ X1,pv1,p
X2,1v1,1 + X2,2v1,2 + · · ·+ X2,pv1,p...
......
XN,1v1,1 + XN,2v1,2 + · · ·+ XN,pv1,p
I The principal components of X are zj = djuj , j = 1, ..., p.I The first principal component of X, z1, has the largest sample
variance amongst all normalized linear combinations of thecolumns of X.
Var(z1) = d21/N .
I Subsequent principal components zj have maximum varianced2j /N, subject to being orthogonal to the earlier ones.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Ridge Regression
I Minimize a penalized residual sum of squares
βridge = argminβ
N∑
i=1
(yi − β0 −p∑
j=1
xijβj)2 + λ
p∑j=1
β2j
I Equivalently
βridge = argminβ
N∑i=1
(yi − β0 −p∑
j=1
xijβj)2
subject to
p∑j=1
β2j ≤ s .
I λ or s controls the model complexity.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Solution
I With centered inputs,
RSS(λ) = (y − Xβ)T (y − Xβ) + λβTβ ,
andβridge = (XTX + λI)−1XTy
I Solution exists even when XTX is singular, i.e., has zero eigenvalues.
I When XTX is ill-conditioned (nearly singular), the ridgeregression solution is more robust.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Geometric InterpretationI Center inputs.I Consider the fitted response
y = Xβridge
= X(XTX + λI)−1XTy
= UD(D2 + λI)−1DUTy
=
p∑j=1
uj
d2j
d2j + λ
uTj y ,
where uj are the normalized principal components of X.I Ridge regression shrinks the coordinates with respect to the
orthonormal basis formed by the principal components.I Coordinate with respect to the principal component with a
smaller variance is shrunk more.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
I Instead of using X = (X1,X2, ...,Xp) as predicting variables,use the transformed variables
(Xv1,Xv2, ...,Xvp)
as predictors.I The input matrix is X = UD (Note X = UDVT ).I Then for the new inputs
βridgej =
dj
d2j + λ
uTj y , Var(βj) =
σ2
d2j
where σ2 is the variance of the error term ε in the linearmodel.
I The factor of shrinkage given by ridge regression is
d2j
d2j + λ
.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
The Geometric interpretation of principal components and shrinkage by
ridge regression.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
Compare squared loss E (βj − βj)2
I Without shrinkage: σ2/d2j .
I With shrinkage: Bias2 + Variance.
(βj − βj ·d2j
d2j + λ
)2 +σ2
d2j
· (d2j
d2j + λ
)2
=σ2
d2j
·d2j (d2
j + λ2 β2j
σ2 )
(d2j + λ)2
I Consider the ratio between squared loss
d2j (d2
j + λ2 β2j
σ2 )
(d2j + λ)2
.
Jia Li http://www.stat.psu.edu/∼jiali
Linear, Ridge Regression, and Principal Component Analysis
The ratio between the squared loss with and without shrinkage. The
amount of shrinkage is set by λ = 1.0. The four curves correspond to