Collegiate Learning Assessment: Implications for Institutional Accountability Session 353 Braden J. Hosch, Ph.D. Director of Institutional Research & Assessment Central Connecticut State University AIR Annual Forum, Chicago, IL June 1, 2010
Mar 29, 2015
Time on Test, Student Motivation, and Performance on the Collegiate Learning Assessment: Implications for Institutional AccountabilitySession 353
Braden J. Hosch, Ph.D.Director of Institutional Research & AssessmentCentral Connecticut State University
AIR Annual Forum, Chicago, ILJune 1, 2010
OverviewVoluntary System of Accountability and
the Collegiate Learning Assessment
Institutional Profile
Methodology and Limitations
Findings
Implications for Accountability
Voluntary System of AccountabilityInitiative among public colleges
led by APLU and AASCU to provide public with comparable information in same format
Pre-emptive response to findings issued by Spellings Commission (2006)
Learning outcomes must be posted by Spring 2011
VSA Learning OutcomesResults from cross-sectional administration
to first-year students and seniors of one of three tests:◦Collegiate Learning Assessment (CLA)◦Collegiate Assessment of Academic Proficiency
(CAAP)◦Measure of Academic Proficiency and Progress
(MAPP)
Template reports scaled assessment scores, SAT/ACT scores of tested students, and an institutional relative-to-expected (RTE) score
Collegiate Learning AssessmentConstructed response test that measures:
◦Critical Thinking◦Problem Solving◦Analytical Reasoning◦Writing
Two tasks:◦Performance task (90 minute time limit)◦Analytic writing task (60 minute time limit)
Scored holistically and converted to scaled score,percentile score, and RTE score.
Central Connecticut State UniversityPublic – part of Connecticut State Univ. SystemCarnegie 2005 Master’s-Larger ProgramsNew Britain, CT (Hartford MSA: ~ 1.2 million pop.)Fall 2009 Enrollment:
◦ 12,461 headcount (9,989 undergraduate, 22% residential);
9,619 full-time equivalent enrollment◦ 52% female; 17% minority◦ Full-time, first-time students: 1,281 (56% residential)◦ Mean SAT score:
1025 (F 2009)
Six-year graduation rates of full-time, first-time students entering in Fall 2003: 49%
940960980
100010201040
974
1025
Me
an
SA
T
CCSU CLA Percentile Scores
FY 2
007
(SAT
= 1
019)
Seni
or 2
008
(SAT
=99
4)
FY 2
008
(SAT
=10
45)
Seni
or 2
009
(SAT
= 1
016)
FY 2
009
(SAT
= 1
019)
0
20
40
60
80
100
51 3767 70
5362 6384 97
Raw Percentile Adjusted Percentile
Perc
en
tile
Am
on
g A
ll C
LA
Takers
Detailed Results by SemesterParticipants
Fall 2007First-Year
Spring 2008
Senior
Fall 2008First-Year
Spring 2009
Senior
Fall 2009First-Year
Participants
105 99 110 134 130
HS Rank 57%ile 64%ile 61%ile 63%ile 62%ile
SAT M+CR
1019 994 1045 1016 1019
CLA Scaled
1057 1133 1127 1248 1098
Performance Task
1066 1113 1110 1214 1087
Analytic Writing Task
1048 1147 1144 1282 1109
CLA %ile 51 37 67 70 53
Cum GPA 2.73 3.13 2.87 3.24 2.88
Minutes -- 45 49 63 44
Test AdministrationTest administration procedures evolved
over time because of difficulty in recruitment.
First-year students recruited through FYE courses◦Incentives varied by instructor
Seniors recruited primarily through email◦Incentive = graduation regalia (~$25 - $40)
First-Year StudentsFall 2007 Fall 2008 Fall 2009
Proctor PT EmployeeOIRA Director
Grad Assistant Grad Assistant
Incentives [Various] [Various] [Various]
FYE Sections 7 7 9
Participants 105 110 130
Pre-Test Survey
N Y Y
SeniorsSpring 2008 Spring 2009 Spring 2010
Proctor PT Employee &
OIRA Director
Grad Assistant Grad Assistant
Incentives Initially none$25 Regalia
Discount
$35 Regalia Voucher
(Total Cost)
$40 Regalia Voucher
(Total Cost)
Senior Capstones
3 0 0
TotalParticipants
99 130 105
Pre-test Survey
N Y Y
Methodological Issues & LimitationsMethodological Issues
◦Cross-sectional design◦Different proctors in 2007-08 than 2008-
09 and 2009-10◦Different incentives for FY students and
seniors Within administrations Across administrations
◦Hand-timingTime on test as an imperfect proxy
for motivation
Minutes Spent on Test by Scaled CLA Score (All Available)
1200+ 1101-1200 1001-1100 Below 10000
10
20
30
40
50
60
70
55.4 50.543.5
33.7
63.456.5 52.1
43.8
First-Year Students Seniors
Scaled CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Scaled CLA Score (2007-08)
1200+ 1101-1200 1001-1100 Below 10000
10
20
30
40
50
60
70
52.246.1 40.6 38.3
First-Year Students Seniors
Scaled CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Scaled CLA Score (2008-09)
1200+ 1101-1200 1001-1100 Below 10000
10
20
30
40
50
60
70
54.4 51.1 47.336.0
65.1 59.6 60.553.5
First-Year Students Seniors
Scaled CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Scaled CLA Score (2009-10)
1200+ 1101-1200 1001-1100 Below 10000
10
20
30
40
50
60
70
56.649.8
40.132.6
First-Year Students Seniors
Scaled CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Relative-to-Expected CLA Score (All Available)
Wel
l Abo
ve a
nd A
bove
Exp
ecte
d
At E
xpec
ted
Wel
l Bel
ow a
nd B
elow
Exp
ecte
d0
20
40
60
51.3 48.5 36.462.8 54.3 49.2
First-Year Students Seniors
Relative-to-Expected CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Relative-to-Expected CLA Score (2007-08)
Wel
l Abo
ve a
nd A
bove
Exp
ecte
d
At E
xpec
ted
Wel
l Bel
ow a
nd B
elow
Exp
ecte
d0
20
40
60
50.8 47.8 34.8
First-Year Students Seniors
Relative-to-Expected CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Relative-to-Expected CLA Score (2008-09)
Wel
l Abo
ve a
nd A
bove
Exp
ecte
d
At E
xpec
ted
Wel
l Bel
ow a
nd B
elow
Exp
ecte
d0
20
40
60
51.8 49.2 41.965.0 58.3 57.5
First-Year Students Seniors
Relative-to-Expected CLA Score
Mean
Min
ute
s S
pen
t on
Test
Minutes Spent on Test by Relative-to-Expected CLA Score (2009-10)
Wel
l Abo
ve a
nd A
bove
Exp
ecte
d
At E
xpec
ted
Wel
l Bel
ow a
nd B
elow
Exp
ecte
d0
20
40
60
51.0 47.9 33.1
First-Year Students Seniors
Relative-to-Expected CLA Score
Mean
Min
ute
s S
pen
t on
Test
Correlations (First-Year Students)
CLA Scaled Score
Minutes Spent on CLA
SATCR
SAT Writing
SAT(Math + CR)
High School Rank
Minutes spent on CLA
.468 -- -- -- -- --
SAT Critical Reading
.333 .178 -- -- -- --
SAT Writing .311 .162 .632 -- -- --
SAT (Math + CR) .326 .186 .807 .610 -- --
High School Rank
.264 .227 .133 .161 .180 --
SAT Math .201 .127 .331 .370 .824 .145
Correlations (Seniors) CLA
Scaled Score
SAT(Math + CR)
SATMath SAT CR
End of term cum GPA
High School Rank
SAT (Math + CR) .505 -- -- -- -- --
SAT Math .479 .886 -- -- -- --
SAT Critical Reading
.409 .890 .576 -- -- --
End of term cum GPA
.400 .271 .285 .263 -- --
High School Rank
.338 .214 .178 .147 .336 --
Minutes spent on CLA
.331 .090 .210 .095 .177 .214
Regressions (First-Year Students)First-Year Student CLA Scaled Score (R2=0.261)
β Std. Err.
t Sig.
(Constant) 652 73.3 8.89 ***
Minutes spent on CLA 3.67 .490 7.49 ***
Combined SAT Score 0.281 .071 3.93 ***
First-Year Student CLA Percentile (R2=0.286)
β Std. Err.
t Sig.
(Constant) -28.9 12.6 -2.28 ***
Minutes spent on CLA .685 .084 8.18 ***
Combined SAT Score .049 .012 3.94 ***
Regressions (Seniors)Senior CLA Scaled Score (R2=0.261)
β Std. Err.
t Sig.
(Constant) 395 93.9 4.20 ***
Minutes spent on CLA 2.50 .535 4.67 ***
Combined SAT Score 0.665 .091 7.34 ***
Senior CLA Percentile (R2=0.286) β Std. Err.
t Sig.
(Constant) -69.0 14.2 -4.85 ***
Minutes spent on CLA .372 .081 4.59 ***
Combined SAT Score .099 .014 7.19 ***
Survey ResultsDifferences in test scores and time
spent on test by self-reported motivation were suggestive but not statistically significant.
Scores and time usage aside, the percentage of students who agreed or strongly agreed they were highly motivated to participate in CLA:◦34% First-Year Students◦70% Seniors
Self-Reported Motivation
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 10 4% 40 1071 1 1% ^ ^
Disagree 19 8% 45 1103 1 1% ^ ^
Neutral 123 53% 46 1109 37 28% 59 1240
Agree 64 28% 48 1125 70 53% 63 1250
Strongly agree 14 6% 51 1169 22 17% 67 1244
I feel highly motivated to participate in this activity today
Differences are suggestive but NOT statistically significant
Test Modality Preference
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 35 15% 48 1089 9 7% 57 1220
Disagree 73 32% 45 1102 24 18% 61 1222
Neutral 74 32% 48 1127 51 39% 62 1292
Agree 34 15% 46 1129 36 27% 66 1207
Strongly agree 15 6% 45 1155 11 8% 65 1265
I perform better on essay tests than on multiple choice tests.
Differences are suggestive but NOT statistically significant
Assessment Modality Preference
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 11 5% 42 1083 9 7% 67 1205
Disagree 30 13% 46 1143 16 12% 62 1248
Neutral 51 22% 45 1122 33 25% 62 1290
Agree 81 36% 48 1103 45 34% 67 1233
Strongly agree 11 5% 42 1083 9 7% 67 1205
I prefer to take a test rather than write a paper.
Test Anxiety
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 21 9% 48 1157 13 10% 60 1286
Disagree 54 24% 50 1126 46 35% 67 1271
Neutral 76 33% 48 1114 39 30% 62 1209
Agree 53 23% 42 1092 31 24% 61 1261
Strongly agree 24 11% 45 1112 2 2% ^ ^
I get so nervous when I take tests that I don't usually perform my best work.
Student Responsibility for Learning
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 0 0% ^ ^ 0 0% ^ ^
Disagree 1 0% ^ ^ 2 2% ^ ^
Neutral 19 8% 37* 1022* 11 8% 54 1187
Agree 106 47% 45* 1124* 61 47% 65 1271
Strongly agree 100 44% 51* 1121* 57 44% 63 1233
Students are responsible for learning material assigned by their professors
* Sig. at p<0.05 ANOVA test.
Institution Responsibility for Learning
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 14 6% 48 1168 5 4% 74 1235
Disagree 66 29% 46 1097 36 27% 60 1233
Neutral 98 43% 48 1114 40 31% 63 1271
Agree 40 18% 47 1122 36 27% 64 1242
Strongly agree 8 4% 43 1118 14 11% 62 1245
Colleges and universities are responsible if students don't learn what they need to be successful after they graduate.
Mandatory College Exit Tests
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 45 20% 45 1109 17 13% 62 1254
Disagree 86 38% 48 1137 63 48% 63 1280
Neutral 61 27% 47 1093 30 23% 61 1226
Agree 31 14% 46 1119 13 10% 69 1176
Strongly agree 5 2% 42 1009 8 6% 61 1197
All college students should be required to pass a standardized exit test in order to graduate
Differences are suggestive but NOT statistically significant
Publications of College Rankings
First-Year
(Fall 2008 and Fall 2009)Seniors
(Spring 2009 only)
N PctAvg.
MinutesAvg. CLA Score N Pct
Avg. Minutes
Avg. CLA Score
Strongly disagree 7 3% 41 1199 4 3% 69 1239
Disagree 37 16% 47 1135 33 25% 61 1270
Neutral 118 52% 46 1114 47 36% 62 1231
Agree 62 27% 48 1096 42 32% 63 1260
Strongly agree 3 1% 45 965 5 4% 70 1181
Students should use published college rankings (like US News and World Report) when deciding which school to attend
Differences are suggestive but NOT statistically significant
Overall FindingsTime spent on test MATTERS
What students say about their motivation may not matter (much)
The relationship between time and test scores is generally missing from discussions about accountability
Implications (1)
1. Acknowledge that test scores may be influenced by motivation/time spent on test; support further research into these effects.
2. Longitudinal testing may help control for some of the effects of motivation / time spent on test.
Implications (2)
3. Multi-year moving averages might improve meaningfulness of test-score information.
4. Statistical adjustments based on time spent on test should be explored but may not be technically or politically feasible.
Implications (3)5. Explore portfolio or other contextual
assessment strategies for accountability, esp. among consortia of institutions (not unlike athletics conferences)
6. Recognize that motivation and time effects are also likely present in elementary and secondary education; consider extent to which performance reflects cognitive vs. behavioral/motivational outcomes.
Questions
Session 353
Dr. Braden J. HoschDirector of Institutional Research & AssessmentCentral Connecticut State [email protected]
Paper, handout, and slides online at:http://www.ccsu.edu/page.cfm?p=1973(see “Research and Presentations”)
Contact Information