(Simple) Multiple linear regression and Nonlinear models Multiple regression • One response (dependent) variable: – Y • More than one predictor (independent variable) variable: – X 1 , X 2 , X 3 etc. – number of predictors = p • Number of observations = n
24
Embed
(Simple) Multiple linear regression and Nonlinear models Handouts... · (Simple) Multiple linear regression and Nonlinear models Multiple regression • One response (dependent) variable:
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
(Simple) Multiple linear regression and Nonlinear models
Multiple regression
• One response (dependent) variable:– Y
• More than one predictor (independent variable) variable:– X1, X2, X3 etc.
– number of predictors = p
• Number of observations = n
Multiple regression - graphical interpretation
0 1 2 3 4 5 6 7X1
0
5
10
15
Y
7 8 9 10 11 12X2
0
5
10
15
Y
Multiple regression graphical explanation.syd
Two possible single variable models:1) yi = 0 + 1xi1 + I
2) yi = 0 + 2xi2 + i
Which is a better fit?
Multiple regression - graphical interpretation
Multiple regression graphical explanation.syd
Two possible single variable models:1) yi = 0 + 1xi1 + I
2) yi = 0 + 2xi2 + i
Which is a better fit?
0 1 2 3 4 5 6 7X1
0
5
10
15
Y
7 8 9 10 11 12X2
0
5
10
15
Y
P=0.02r2=0.67
P=0.61r2=0.00
Multiple regression - graphical interpretation
Multiple regression graphical explanation.syd
Perhaps a multiple regression model work fit better:
• Test of whether overall regression equation is significant.
• Use ANOVA F-test:– Variation explained by regression
– Unexplained (residual) variation
Assumptions
• Normality and homogeneity of variance for response variable (previously discussed)
• Independence of observations (previously discussed)
• Linearity (previously discussed)• No collinearity (big deal in multiple
regression)
Collinearity
• Collinearity:– predictors correlated
• Assumption of no collinearity:– predictor variables uncorrelated with (ie.
independent of) each other
• Effect of collinearity:– estimates of is and significance tests
unreliable
Checks for collinearity• Correlation matrix and/or SPLOM between
predictors• Tolerance for each predictor:
– 1-r2 for regression of that predictor on all others– if tolerance is low (near 0.1) then collinearity is a
problem• VIF values
– 1/tolerance – (variance inflator function) – look for large values
(>10)• Condition indices (not in JMP – Pro)
– Greater than 15 – be cautious– Greater than 30 – a serious problem
• Look at all indicators to determine extent of colinearity
Scatterplots• Scatterplot matrix (SPLOM)
– pairwise plots for all variables
• Example: build a multiple regression model to predict total employment using values of six independent variables. See Longley.syd– MODEL total = CONSTANT + deflator + gnp + unemployment +
armforce + population + timeDEFLATOR
DE
FLA
TO
R
GNP UNEMPLOY ARMFORCE POPULATN TIME
DE
FLA
TO
R
GN
P
GN
P
UN
EM
PLO
Y
UN
EM
PLO
Y
AR
MF
OR
CE
AR
MF
OR
CE
PO
PU
LAT
N
PO
PU
LAT
N
DEFLATOR
TIM
E
GNP UNEMPLOY ARMFORCE POPULATN TIME
TIM
E
Look at relationship between predictor variables –immediately you can see colinearity problems
Checks for collinearity• Correlation matrix and/or SPLOM between
predictors• Tolerance for each predictor:
– 1-r2 for regression of that predictor on all others– if tolerance is low (near 0.1) then collinearity is a
problem• VIF values
– 1/tolerance – (variance inflator function) – look for large values
(>10)• Condition indices
– Greater than 15 – be cautious– Greater than 30 – a serious problem
• Look at all indicators to determine extent of colinearity
Condition indices
1 2 3 4 5
1.00000 9.14172 12.25574 25.33661 230.42395
6 7
1048.08030 43275.04738
Dependent Variable ¦ TOTAL N ¦ 16 Multiple R ¦ 0.998 Squared Multiple R ¦ 0.995 Adjusted Squared Multiple R ¦ 0.992 Standard Error of Estimate ¦ 304.854
Effect Coefficient Std Error Std Coef Tolerance t P(2 Tail)
TIME ¦ 1,829.151465 798.787513 2,859.515416 758.980597
Solutions to collinearity
• Simplest - Drop redundant (correlated) predictors
• Principal components regression– potentially useful
Best model?
• Model that best fits the data with fewest predictors
• Criteria for comparing fit of different models:– r2 generally unsuitable– adjusted r2 better– Mallow’s Cp better– AIC Best – lower values indicate better fit
Explained variance
r2
proportion of variation in Y explained by linear relationship with X1, X2 etc.
SS RegressionSS Total
Screening models
• All subsets– recommended– many models if many predictors ( a big problem)
• Automated stepwise selection:– forward, backward, stepwise– NOT recommended unless you get the same
model both ways• Check AIC values• Hierarchical partitioning
– contribution of each predictor to r2
Model comparison (simple version)
• Fit full model:– y = 0+1x1+2x2+3x3+…
• Fit reduced models (e.g.):– y = 0+2x2+3x3+…
• Compare
Multiple regression 1
X1
X1
X2 X3 X4 Y
X1
X2 X
2
X3 X
3
X4
X4
X1
Y
X2 X3 X4 Y
Y
y = 0+1x1+2x2+3x3+ 4x4
Any evidence of Colinearity?
Model Building
Again check for colinearity
Compare Models using AIC
• Model 1:
– AIC 78.67– Corrected AIC 85.67
• Model 2
– AIC 77.06– Corrected AIC 81.67
y = 0+1x1+2x2+3x3+ 4x4
y = 0+1x1+2x2+3x3
Formally: Akaike information criterion (AIC, AICc)
Sometimes the following equation is used: AIC = 2k + n[ln(RSS/n)]
where, k = number of fitted parametersn = number of observations
= residual sum of squares (RSS) / AICc = corrected for small sample sizeLower score means better fit
ln 2 1 2 1
ln 2 1 2 1 2 1AIC:AICc:
Model Selection
All Possible Models
Ordered up to best 4 models up to 4 terms per model.