Learning Outcomes Assessment: A National · PDF fileLearning Outcomes Assessment: A National Perspective ... The New Majority and Demographic ... Measuring Quality in Higher Education

Learning Outcomes Learning Outcomes Assessment: Assessment:

A National PerspectiveA National Perspective

George D. Kuh

Council of Graduate Schools

Washington, DCNovember 22, 2010

A National PerspectiveA National Perspective

Advance OrganizersAdvance Organizers� What kind of information about student

learning is compelling and useful for:(a) guiding improvement efforts?(b) responding to accountability demands?

� What can be done to prepare the next generation of faculty and motivate the current generation to collect and useuseassessment results to enhance student learning?

� And what about assessing learning in graduate school?!? ☺☺☺☺

ContextContext

�� Global Competitiveness in Degree Global Competitiveness in Degree AttainmentAttainment

�� The New Majority and Demographic The New Majority and Demographic GapsGaps

�� Questionable Levels of Student Questionable Levels of Student �� Questionable Levels of Student Questionable Levels of Student PerformancePerformance

NOLOA

“Colleges… do so little to measure what students learn between freshman and senior years. So doubt lurks: how much does a college education – the actual teaching and learning that happens on campus – really matter?”

David Leonhardt, NYTimes, Sept 27, 2009

ContextContext

� Global Competitiveness in Degree Attainment

� The New Majority and Demographic Gaps

� Questionable Levels of Student � Questionable Levels of Student Performance

�� In a Most Challenging Fiscal In a Most Challenging Fiscal Environment …Environment …

�������� We Need Higher Levels of Student We Need Higher Levels of Student AchievementAchievement

Assessment 2010Assessment 2010

�Greater emphasis on student learning outcomes and evidence that student performance measures up

“It’s the Learning, “It’s the Learning, Stupid”Stupid”

Working Definition Working Definition

Assess: (v.): to examine carefully

Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development (Palomba & Banta, 1999, p. 4)

Assessment PurposesAssessment Purposes

� Improvement

� Accountability� Accountability

IndicatorsIndicators�learning outcomes

�educational attainment (persistence, graduation)

�course retention

�transfer student success �transfer student success

�success in subsequent courses

�degree/certificate completion

�graduate school

�employment/employer evaluations

�capacity for lifelong learning

Assessment ToolsAssessment Tools

�� Direct Direct (outcomes) measuresmeasures---- Evidence of what Evidence of what

students have learned or students have learned or can do can do can do can do

�� Indirect Indirect (process) measuresmeasures---- Evidence of effective Evidence of effective

educational activity by educational activity by students and institutions students and institutions

Occasional Paper #1

Assessment, Accountability, and ImprovementPeter T. EwellPeter T. EwellAssessments of what students learn during college are typically used for either improvement or accountability, and occasionally both. Yet, since the early days of the “assessment movement” in the US, these two purposes of outcomes assessment have not rested comfortably together.

www.learningoutcomeassessment.org/OccasionalPapers.htm

Continuous Continuous ImprovementImprovement

AccountabilityAccountability

Strategic dimensions

Purpose Formative (improvement) Summative (judgment)

Orientation Internal External

Motivation Engagement Compliance

Implementation

Two Paradigms of AssessmentTwo Paradigms of Assessment

Implementation

Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative, established goal

Comparative or fixed standard

Communication of results

Multiple internal channels Public communication, media

Use of results Multiple feedback loops ReportingEwell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting

for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

Assessment 2010Assessment 2010

�Greater emphasis on student learning outcomes and evidence that student performance measures up

�Demands for comparative measures�Increased calls for transparency ---

public disclosure of student and institutional performance

TemplatesTemplates

� APLU/AASCU Voluntary System of Accountability

�� NAICU’s UNAICU’s U--CANCAN�� College Navigator College Navigator (NCES)�� Transparency by Design/College Transparency by Design/College �� Transparency by Design/College Transparency by Design/College

Choices for Adults Choices for Adults (WCET)�� AACC AACC (yet to be named)�� Degree Qualifications InventoryDegree Qualifications Inventory�� Alliance GuidelinesAlliance Guidelines�� NILOA Transparency FrameworkNILOA Transparency Framework

Assessment 2010Assessment 2010

�Greater emphasis on student learning outcomes and evidence that student performance measures up

�Demands for comparative measures�Increased calls for transparency ---public �Increased calls for transparency ---public

disclosure of student and institutional performance

�Assessment “technology” has improved markedly, but still is insufficient to document learning outcomes most institutions claim

Sample Data SourcesSample Data Sources• Locally-developed measures • National instruments

– National Survey of Student Engagement (NSSE)– Beginning College Survey of Student

Engagement (BCSSE)– Faculty Survey of Student Engagement (FSSE)– Cooperative Institutional Research Program

(CIRP)– Your First College Year (YFCY)– College Student Experiences Questionnaire – College Student Experiences Questionnaire

(CSEQ)– Noel Levitz Student Satisfaction Inventory– ETS MAPP and Major Field Tests– ACT Collegiate Assessment of Academic

Proficiency – Collegiate Learning Assessment (CLA)

• Institutional data -- GPA, financial aid, transcripts, retention, certification tests, alumni surveys, satisfaction surveys…

• Electronic portfolios

Valid Assessment of Valid Assessment of Learning in Undergraduate Learning in Undergraduate Education (VALUE) RubricsEducation (VALUE) Rubrics

� Inquiry and analysis � Critical thinking � Creative thinking � Written communication � Oral communication� Reading � Reading � Quantitative literacy � Information literacy � Teamwork � Problem solving � Civic knowledge and engagement � Intercultural knowledge and competence � Ethical reasoning and action � Foundations and skills for lifelong learning � Integrative learning

AAC&U VALUE AAC&U VALUE Project Project –– 15 15 RubricsRubrics

Measuring Quality in Higher Education Measuring Quality in Higher Education (Vic Borden & Brandi Kernel, 2010)

Web-based inventory hosted by AIR of assessment resources. Key words can be used to search the four categories:� instruments (examinations, surveys, questionnaires,

etc.); �software tools and platforms; �benchmarking systems and data resources; �projects, initiatives and services. http://applications.airweb.org/surveys/Default.aspx

Do we measure what we value?Do we measure what we value?or

Do we value what we measure?Do we value what we measure?

Wise decisions are Wise decisions are needed about needed about whatwhatneeded about needed about whatwhatto measure in the to measure in the context of campus context of campus mission, values, and mission, values, and desired outcomes.desired outcomes.

Summary Summary

��PerhapsPerhaps more assessment underway than some acknowledge or wish to believe

�More attention needed to using and �More attention needed to using and reporting assessment results

�Involving faculty is a major challenge�More investment likely needed to

move from data to improvement

According to Provosts, what is the driving force for assessment?

a. Institutional Commitment to a. Institutional Commitment to Improvement

b. Accreditationc. Faculty & Staff Interestd. Governing Board Mandate

“high importance” 85% Regional

80% Specialized

Summary Summary

�Perhaps more assessment underway than some acknowledge or wish to believe

�More attention needed to using and reporting assessment results

�Involving faculty is a major challenge�Involving faculty is a major challenge�More investment likely needed to move

from data to improvement�Accreditation is a major force shaping

assessment

Regional accreditors cite deficiencies in student learning outcomes assessment with greater frequency

�Middle States - 2/3 of institutions have follow-up; �Middle States - 2/3 of institutions have follow-up; number one reason being assessment

�NEASC - 80% of institutions asked for follow-up on student learning outcomes assessment

�HLC - 7 out of 10 institutions are being monitored; the vast majority for student learning outcomes assessment.

Looking Back: What’s Been Looking Back: What’s Been Accomplished?Accomplished?�Assessment Seen as Legitimate�Goals for Learning Established�Goals for Learning Established�A “Semi-Profession” for

Assessment�Much Better Instruments and

Methods

Looking Back: What Remains to be Looking Back: What Remains to be Done?Done?�Authentic Faculty Ownership�Assessment Still an “Add -On”�Assessment Still an “Add -On”�Use of Information for

Improvement is Underdeveloped�Sincere Institutional Engagement

with Accreditors in Assessment

Advance OrganizersAdvance Organizers� What kind of information about student

learning is compelling and useful for:(a) guiding improvement efforts?(b) responding to accountability demands?

� What can be done to prepare the next generation of faculty and motivate the current generation to collect and useuseassessment results to enhance student learning?

� Do we care about assessing learning in graduate school?!? ☺☺☺☺

Questions Questions & &

DiscussionDiscussion