M ODELS & S TRATEGIES FOR A SSESSING I NSTITUTIONAL LEARNING OUTCOMES Ashley Finley, Ph.D Senior Director of Assessment & Research, AAC&U National Evaluator,

MODELS & STRATEGIES FOR ASSESSING INSTITUTIONAL

LEARNING OUTCOMES

Ashley Finley, Ph.D

Senior Director of Assessment & Research, AAC&U

National Evaluator, Bringing Theory to Practice

Kutztown University

January 19, 2012

LEARNING-CENTERED ASSESSMENT:

WHY ARE WE HERE?

What are you doing?

Why are you doing it?

How do you know you’re doing it well?

What does it mean to be learning-centered at KU?

What practices promote and advance student learning?

How pervasive are these? What is the change you

want to see in your students when they leave the institution?

What are the foundational skills and competencies students should acquire throughout their learning experiences?

What are the effective approaches for implementing a cycle of inquiry and improvement?

THE NATIONAL WORK ON LEARNING-CENTERED ASSESSMENT, LIBERAL EDUCATION OUTCOMES, & GENERAL EDUCATION

Liberal Education: A philosophy of learning that empowers & prepares individuals to deal with complexity, diversity, & change. = Broad knowledge combined w/ in-depth study To help students develop a sense of social

responsibility, strong & transferable intellectual and practical skills & a demonstrated ability to apply knowledge.

Vast majority of institutions have articulated learning outcomes

ANY institution can achieve the goals of a liberal education – 3 Major National Initiatives Compass Project (State University Systems – CA, OR, WI) Quality Collaboratives VALUE Rubric Project

WHAT EMPLOYERS SAY ABOUT LIBERAL EDUCATION SKILLS:

88%

88%

90%

91%

% of Employers who agree with each statement

• Our company is asking employees to take on more responsibilities and to use a broader set of skills than in the past

• Employees are expected to work harder to coordinate with other departments than in the past

• The challenges employees face within our company are more complex today than they were in the past

• To succeed in our company, employees need higher levels of learning and knowledge today than they did in the past

Source: Raising the Bar (AAC&U, 2010)

70%

70%

71%

75%

75%

79%

81%

89%

% of Employers who agree that two- and four-year colleges should place MORE emphasis on helping students develop

the following:

Effective oral/written communication

Critical thinking/ analytical reasoning

Knowledge/skills applied to real world

settings Analyze/solve complex

problemsConnect choices and

actions to ethical decisions

Teamwork skills/ ability to collaborate

Ability to innovate and be creative

Concepts/developments in science/technology

Source: “Raising the Bar: Employers’  Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)

52%

57%

57%

63%

65%

67%

68%AND IN THESE AREAS…

Locate/organize/evaluate information

Understand global context of situations/ decisions

Global issues’ implications for future

Understand & work with numbers/statistics

Understand role of U.S. in the world

Knowledge of cultural diversity in US/world

Civic knowledge, community engagement

“Raising the Bar: Employers’  Views on College Learning in the Wake of the Economic Downturn” (AAC&U and Hart Research Assoc. 2010)

0

30,000

60,000

q1(low) q2 q3 q4 q5(high)

earn

ings

quintiles

Mean Earnings: Use of Speaking Skills

Earni…

Source: Georgetown University, Center for Education and the Workforce (Anthony Carnavale)

HOW MUCH DOES A LIBERAL EDUCATION PAY?: “IMAGINATION IS MORE IMPORTANT THAN KNOWLEDGE”

-ALBERT EINSTEIN

0

30,000

60,000

q1(low) q2 q3 q4 q5(high)

earn

ings

quintiles

Mean Earnings: Use of Originality

Earnings

Source: Georgetown University, Center for Education and the Workforce

0

35,000

70,000

q1(low) q2 q3 q4 q5(high)

earn

ings

quintiles

Mean Earnings: Use of Judgement & Decision-Making Skills

Earnings

Source: Georgetown University, Center for Education and the Workforce

0

30,000

60,000

q1(low) q2 q3 q4 q5(high)

earn

ings

quintiles

Mean Earnings: Use of Social Skills

Earnings

Source: Georgetown University, Center for Education and the Workforce

Content Outcomes

Learning Outcomes

•Theoretical traditions•Methodology•Topical areas

How do we assess these?• Course Exams•Course Papers•Individual/ Group course projects•In-Class work/ participation

•????•????•????

Articulating Student Learning and Development

Kutztown Mission: “provide a high quality education at the undergraduate and graduate levels in order to prepare students to meet lifelong intellectual, ethical, social, and career challenges.”How do we assess these?

• Knowledge of Human Cultures & the Physical &

Natural World• Content areas

• Intellectual & Practical Skills

• Inquiry & Analysis• Critical & Creative Thinking

• Written & Oral Comm.• Reading

• Quantitative Literacy• Information Literacy

•Teamwork & Problem-solving• Personal & Social Resp.

• Civic Knowledge• Intercultural Knowledge &

Competence• Ethical Reasoning• Lifelong Learning

• Integrative & Applied Learning

Gen.Ed./ Major

Institutional Outcomes

•Goal 1: Cultivate intell. & pract. skills• Inquiry & Analysis• Decision making• Written & Oral Communication• Quantitative Literacy• Information Literacy• Teamwork & problem solving• Wellness• Practiced throughout curr, w/ progressively

more challenging probs, projects & standards

•Goal 2: Develop understanding of human cultures & physical and natural world• Sciences, mathematics, social sciences,

humanities, histories, languages, arts • Engage w/ big questions, contemp. &

enduring•Goal 3: Sense of personal & social resp.• Local/Global civic knowledge • Intercultural knowledge & competence• Ethical reasoning & action• Personal qualities & attitudes (passion,

curiosity, self-confidence, imagination, cooperation, commitment, support)

• Active inv. w/diverse comm. & real world chall.

Learning Outcome

s

Assess Learning Outcome

s

What is the story you want to tell about student learning?

Outcomes

AssessmentCurriculum

Learning Outcome

s

Assess Learning Outcome

s

What is the story you want to tell about student learning?

Basic Logic Model

INPUTSResources are needed

for activities –

What is needed to

start or keep

something going?• Human• Financial•Technological• Curricular/

Program

ACTIVITIES

Actions or processes

necessary to produce outputs –

What activities need to occur to produce

evidence?

OUTPUTS

Products used to assess

outcomes - What can

be counted as evidence of change?

OUTCOMES

Expected Changes and

Benefits –What are the

goals you want to reach?

• Short-term

• Intermediate• Long-term

Assumptions

OUTCOMES• Goals &

Domains of Learning

• Understanding of human cultures…

• Intellectual and practice skills• Critical

thinking• Personal &

Social Responsibility• Civic

engagement

Sample Logic Model INPUTS•GE•Depts.•Advising•Writing Center•Stud. Aff.•Health & Wellness Center•Alumni Engagement•Multicult. Services•Athletics•Distance Ed•Career Services•Inst. Res.

ACTIVITIES•Experiential learning•Civic learning•Interaction

OUTPUTS•Signature assignments•Reflection papers•Group projects•Community-based projects

•Engagement in life’s big questions•Practiced extensively & progressively•Active involve. In diverse comm. & real-world challenges

VALUE PROJECT (WWW.AACU.ORG/VALUE)

Project Goals 1) Create dialogue and develop shared understanding of

common learning outcomes 2) Create template for direct assessment of student work

(in text and non-text formats) 3) Create student-friendly format to engage students in

self-evaluation Rubric Development

Teams of faculty/scholars nationwide Reviewed existing rubrics to identify commonalities,

clarify language, and develop broad agreement on outcomes criteria (openedpractices.org)

To date accessed by over 3000 institutions/organizations, 11,000 individuals Domestic & international, K-12, state systems

Reliability study

LIST OF VALUE RUBRICS Knowledge of Human

Cultures & the Physical & Natural Worlds Content Areas No Rubrics

Intellectual and Practical Skills Inquiry & Analysis Critical Thinking Creative Thinking Written Communication Oral Communication Reading Quantitative Literacy Information Literacy Teamwork Problem-solving

Personal & Social Responsibility Civic Knowledge &

Engagement Intercultural

Knowledge & Competence

Ethical Reasoning Foundations & Skills

for Lifelong Learning Integrative &

Applied Learning Integrative &

Applied Learning

Located at: http://www.aacu.org/value/index.cf

m

Located at: http://www.aacu.org/value/index.cf

m

Criteria

The Anatomy of a VALUE RubricThe Anatomy of a VALUE Rubric

Levels

Performance Descriptors

THE CALIBRATION TRAINING PROCESS

Scoring Steps: Review rubric to familiarize yourself with structure,

language, performance levels Ask questions about the rubric for clarification or to

get input from others regarding interpretation Read student work sample Connect specific points of evidence in work sample

with each criterion at the appropriate performance level (if applicable)

Calibration Steps: Review scores Determine common score(s) Hear from outliers Discuss Determine final score

THE GROUND RULES

This is not grading. We are not changing the rubric (today). Our work is time sensitive. Go with your

instinct. Think globally about student work and about

the learning skill. Think beyond specific disciplinary lenses or content.

Start with 4 and work backwards. Pick one performance benchmark per criterion. Avoid “.5”.

Zero does exist. Assign “0” if work does not meet benchmark (cell one) performance level. N/A exists. Assign “not applicable” if the student work is not intended to meet a particular criterion.

SIGNATURE ASSIGNMENTS

Assignment should enable attainment of criteria Break down criteria to determine key components for

assignment

What should students do with content to meet criteria? E.g. What are the pieces to be analyzed, compared,

integrated?

Will the assignment be used for more than one outcome?

What are the types of assignments that will be most helpful for allowing students to demonstrate competency?