8/14/2019 2009 Assessment Institute
1/34
E-Portfolios and Nationally
Standardized Tests:
Validation or Depreciation
2009 Assessment InstituteIndianapolis, IN
October 27
Richard Robles | Julie Burdick | Mark Nicholashttp://tinyurl.com/uceportfolioproject
8/14/2019 2009 Assessment Institute
2/34
Presenters
Richard RoblesAssistant Director for First-Year Experience and
Leadership Programs, University Honors Program
Julie BurdickDirector of Academic Planning and Assessment,
Provost Office
Mark NicholasPhD Candidate, CECH
8/14/2019 2009 Assessment Institute
3/34
8/14/2019 2009 Assessment Institute
4/34
About the University of Cincinnati
Top 20 research-intensive university
Liberal arts plus professional colleges
Tradition and culture of decentralized colleges
Current enrollment around 39,000
General Education program began in 2001, with the first
cohort graduating in June 2005
8/14/2019 2009 Assessment Institute
5/34
External Drivers
Spellings Report (2006)
Institutional accountability
Measurement of student outcomes Value-added assessment model
Institutional comparability
Voluntary System of Accountability CLA, MAPP, CAPP
University System of Ohio (2008) Mandate
8/14/2019 2009 Assessment Institute
6/34
Dual Pilot Study
To consider the applicability of learning portfolios(and especially e-portfolios) alongside the CLA among a
group of first-year honors students
Data collection began fall 2008
Applied the written communication and critical thinking
VALUE Metarubrics through the e-portfolio
Follow cohort beyond the first year
8/14/2019 2009 Assessment Institute
7/34
Primary Research Questions
How do the CLA results compare toresults produced via VALUE Metarubricscoring? In terms of..
Measures of individual student learning
Measures of institutional-level value added
As a tool for guiding program improvement As a response to accountability
8/14/2019 2009 Assessment Institute
8/34
Secondary Research Questions
Is one instrument more valuable for students
regarding their undergraduate program of study?
Do e-portfolios link these experiences /
instruments in ways that are meaningful to
faculty, students, or administrators?
What is the diagnostic utility of the two
instruments?
8/14/2019 2009 Assessment Institute
9/34
Methodology
Sample 111 first-year honors students
first-time baccalaureate degree seeking students
Average ACT 31
advanced standing (AP, IB, PSEOP)
Data Collection CLA and e-portfolio assignments embedded in FYE
honors seminar (fall 2008) CLA administered week 4 of the quarter
Students completed test-motivation survey aftertaking CLA
8/14/2019 2009 Assessment Institute
10/34
Use of the E-Portfolio
Opportunity to map progress on the GenEd baccalaureate competencies
Student activities: reflection on learningstyles, goals, accomplishments
Assessment strategy: AACUs VALUEMetarubrics for assessment of specificartifacts
8/14/2019 2009 Assessment Institute
11/34
Use of the E-Portfolio
Nuventives iWebfolio
Embedded in the
honors FYE seminar
Assignments
submitted via
e-portfolio template
8/14/2019 2009 Assessment Institute
12/34
VALUE Metarubrics
AACU collaboration with faculty and student
affairs professionals (LEAP Initiative, 2005)
Developed rubrics for 15 essential learning
outcomes
Encouraged institutions to revise to
accommodate their individual needs
Development is evolving among institutions UC employed the critical thinking and written
communication metarubrics
8/14/2019 2009 Assessment Institute
13/34
Written Communication Rubric
Modifications
Expanded a four scale to five
(to provide a mid-point)
Simplified terminology
Changed text in each criteria to make eachelement distinct as rubric categories overlapped
8/14/2019 2009 Assessment Institute
14/34
Written Communication Rubric
Assessment
Inter-rater reliability of r = 0.6964
Easier to assess because most people know
or are aware of what is good writing Can be assessed outside the context of thenature of a specific artifact or discipline in alimited sense
Norming sessions
Had an expert on the team that guided theevaluation process
8/14/2019 2009 Assessment Institute
15/34
Assessing Critical Thinking (CT)
Nature of Artifact
Proposal for honors experience within
community engagement, global studies,
leadership, research/creative arts
Equivalent to a proposal for an experiment
Part of assigned homework
8/14/2019 2009 Assessment Institute
16/34
CT Rubric Modifications
Changed the scale from four to five (to provide amid point)
Included the dispositional element of open-mindedness
Modified text in criteria to add clarity and detail
Modified the rubric to suit the nature of theartifact
8/14/2019 2009 Assessment Institute
17/34
CT as a construct
Critical thinking is a habit of mind characterized by the
comprehensive exploration of issues, ideas, artifacts, and
events before accepting or formulating an opinion or
conclusion. www.aacu.org
Subject to multiple interpretations which have
philosophical and disciplinary implications
Differing definitions and a lack of definition
among faculty (Paul, Elder & Bartell, 1997*)
* Paul, R., Elder, L., & Bartell, T. (1997). California teacher preparation forinstruction in critical thinking: Research findings and policy recommendations.
California: Commission on Teacher Credentialing.
8/14/2019 2009 Assessment Institute
18/34
CT Rubric Observations
Measures transdisciplinary CT Difficult to separate disciplinary interpretations
of CT (example) Provides guidelines on the nature of
artifacts Very helpful
Advise students on the nature of the artifactand its content (example)
Inter-rater reliability of r = 0.824
8/14/2019 2009 Assessment Institute
19/34
Side-By-Side Comparison
CLA VALUE Metarubrics
Grading Human and computer Human
Inter-rater Reliability Systematically achieved Achieved high IRR
through norming
Test Method Timed open-ended
response
Open ended, over time
Test Incentive Voluntary Homework
Relevance Real world application Academically connected
8/14/2019 2009 Assessment Institute
20/34
Side-By-Side Comparison
CLA VALUE Metarubrics
Scoring Holistically Trait-by-trait
Test time 90 minutes 10 weeks
Turnaround time 8 months As needed
Motivation Add on Embedded in context
with course
8/14/2019 2009 Assessment Institute
21/34
Findings From Student Survey
Usefulness of the test(waste of time or only mildly useful=50%,useful=40%, veryuseful=10%)
Describe the test experience
(unpleasant=37%, indifferent 48%, pleasant=15%)
Effort exerted (little or none=8%, some=40%, a great deal 52%)
Incentives to take the test seriously(would do it for free=8%, self-improvement=39%, financialincentive required=43%)
Experiences that prepared students to do well on the test
Prep courses for ACT and SAT Extensive extracurricular activities
8/14/2019 2009 Assessment Institute
22/34
Institutional CLA Data
Mean CLA
Score
Unadjusted
% Rank
Deviation
Score
Adjusted
% Rank
Total CLA Score 1352 98 1.4 95
Performance Task 1338 98 1.7 96
Analytic Writing Task 1366 98 0.8 79
Make-an-Argument 1348 98 0.3 63
Critique-an-Argument 1383 98 1.4 92
8/14/2019 2009 Assessment Institute
23/34
8/14/2019 2009 Assessment Institute
24/34
8/14/2019 2009 Assessment Institute
25/34
Metarubric Assignments
N Avg Std Dev Avg team
rater
correlation
Overall
Correlation
WrittenCommunication 109 22.7 3.47 0.717 0.696
Critical Thinking 55 18.98 3.91 0.772 0.824
8/14/2019 2009 Assessment Institute
26/34
Correlation
8/14/2019 2009 Assessment Institute
27/34
Interpretation and Explanation
The lack of statistically significant correlationbetween the CLA task scores and the e-portfolio assignment scores indicates adifference in evaluating student work
Although both processes measure theapplication of intellectual and practical skills,each is evaluating student outcomes atdiffering levels of context
Differences in the administration of the twoinstruments must be considered (context,incentives, timing, the ability to draft)
8/14/2019 2009 Assessment Institute
28/34
Findings
Use of the e-portfolio
The platform is an important feature in the use
of metarubrics for feedback and scoring
The platform is best used across the
curriculum
Use of e-portfolio technology presents ease in
evaluation
8/14/2019 2009 Assessment Institute
29/34
Conclusions
While the CLA is useful forINSTITUTIONAL assessment, care mustbe taken in assigning meaning to the
scores The CLA does not appear to provide data
useful for STUDENT diagnostic feedback
While the CLA may meet the requirementsof VSA, it is of limited value for individualstudent assessment
8/14/2019 2009 Assessment Institute
30/34
Conclusions
Use of the VALUE Metarubrics provides agood framework to develop studentdiagnostic feedback
Metarubrics provide a framework for yieldinginter-rater reliability for faculty feedback
VALUE Metarubrics provide more informationthan the CLA for continuous programimprovement
8/14/2019 2009 Assessment Institute
31/34
Conclusions
Institutions should employ a complex andvaried assessment strategy to understandachievement of student outcomes
Demonstrates value of the educationalexperience
Assures students are well-prepared for lifelonglearning and intellectual inquiry
Provides a framework for program feedback andimprovement
8/14/2019 2009 Assessment Institute
32/34
Recent Analyses
Trudy Banta, The Search for a Perfect Test Continues, in Assessment
Update (November-December 2007)
Joan Hawthorne, Accountability & Comparability: Whats Wrong with the
VSA Approach? in Liberal Education (Spring 2008)
Paul Basken, Electronic Portfolios May Answer Calls for More
Accountability, in The Chronicle of Higher Education (April 17, 2008)
Trudy W. Banta, Assessment for Accountability: A Cautionary Tale,
presented at The Ohio State University on June 19, 2008
Klein, Liu, Sconing, 2009. Test Validity Study (TVS) Report.
http://www.voluntarysystem.org/docs/reports/TVSReport_Final.pdf
Schulenberger, D., Keller, C. (2009) Interpretation of Findings from the TVSfor the VSA.
http://www.voluntarysystem.org/docs/reports/VSAabstract_TVS.pdf
8/14/2019 2009 Assessment Institute
33/34
Reflection
If you conducted a similar study at your
institution, what would be your next steps?
8/14/2019 2009 Assessment Institute
34/34
UCDual Pilot Study
http://tinyurl.com/uceportfolioproject