Factors Influencing Item Nonresponse in the NSSE Survey: Why do Students Skip so many Questions? Presented by Chris Maxwell Purdue University AIR 2009
Jan 13, 2016
Factors Influencing Item Nonresponse in the NSSE Survey: Why do Students Skip so many Questions?
Presented by Chris Maxwell
Purdue University
AIR 2009
Introduction
Purdue NSSE 2004: Max Item Nonresponse: 10%
Purdue NSSE 2007: Max Item Nonresponse: 18%
Purdue NSSE 2007:• Web Only• Targeted Oversample: Underrepresented• 24% response rate• 1,388 total respondents (703 FY and 685 SN)
All Students (FY and SN)
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
N=1,388
Investigation
Question order is an apparent factor, but is anything else at play?
Model the response probability with logistic regression:
• Separate models for students grouped by class, ethnicity, gender, and STEM status (18 total).
• Each model includes the explanatory variables question order, type, length, and first page status.
Model Details
Ln(P/(1-P)) =A+B1(order)+B2(page1)+B3(likert4_scale)
+B4(numerical_scale)+B5(length)
Use SAS proc Logistic (binary modeling, stepwise selection)
Input data set
Model Results
Stepwise selection rejected question types and length as useful predictors in every case
15 models use question order and page one status, with 3 models only using question order
Ln(P/(1-P)) = A + B1(order) + B2(page1) -or- P = exp(A + B1(order) + B2(page1)) 1 + exp(A + B1(order) + B2(page1))
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Per
cen
t R
esp
on
din
g
SN
FY
FY and SN Results
N=685
N=703
FY and SN Results
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Per
cen
t Res
po
nd
ing
SN
FYN=685
N=703
Model Parameters
FY SN
A 2.39 2.70
B1 -0.0141 -0.0130
B2 4.69 4.58
Percent Concordant
62.7 61.6
Thus, for FY students the model predicts a 9% decline in response after the first page, and in the middle of the survey a 1% drop for every 6 questions.
FY: STEM and Non STEM Majors
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
STEM
Non STEMN=301
N=399
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
STEM
Non STEM
SN: STEM and Non STEM Majors
N=459
N=219
FY: Male and Female
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
Female
Male
N=347
N=356
SN: Male and Female
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
Female
MaleN=339
N=346
FY: All Underrepresented
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
Underrepresented
N=99
N=564
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
African American
FY: African American
N=564
N=45
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
Hispanic
FY: Hispanic
N=564
N=44
SN: All Underrepresented
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
UnderrepresentedN=592
N=76
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
African American
SN: African American
N=592
N=36
70%
75%
80%
85%
90%
95%
100%
1 11 21 31 41 51 61 71 81
NSSE 2007 Question Order
Pe
rce
nt
Re
sp
on
din
g
White/Other/Asian
Hispanic
SN: Hispanic
N=592
N=36
All
Mo
del
Par
amet
ers
Model A (Intercept) B1 (order) B2 (page1)
sn_stem 3.36 -0.0197
sn_female 3.02 -0.0137 4.26
sn_white 2.74 -0.0131 4.68
sn_all 2.70 -0.0130 4.58
fy_female 2.67 -0.0153 5.35
sn_hispanic 2.65 -0.0123 3.07
sn_not_stem 2.58 -0.0127 4.30
fy_stem 2.50 -0.0126 4.25
fy_white 2.46 -0.0141 4.62
sn_male 2.46 -0.0128 4.83
fy_hispanic 2.42 -0.0184
fy_all 2.39 -0.0141 4.69
fy_not_stem 2.31 -0.0152 5.13
sn_black 2.28 -0.0183
sn_underrep 2.27 -0.0118 4.19
fy_male 2.16 -0.0134 4.44
fy_black 1.98 -0.0158 3.98
fy_underrep 1.92 -0.0133 4.81
Summary
Study Limitations
Conclusions
Discussion
Available at: www.purdue.edu/OIR/irvba/AIR2009/nsse.ppt