2011Learning assessment systems that work a follow up report

LEARNING ASSESSMENT SYSTEMS THAT WORK!A FOLLOW-UP REPORT

23nd Annual Conference,Denver, COOctober 28, 2011

Jeff GrannGeorge LucasThayer Reed

Jennifer StephensModerator: Patricia Case

© 2008 Capella University - Confidential - Do not distribute 2

• Founded in 1993• Charter AQIP participant in 2000• Courses 100% online• For-profit university• Enrollment: 39,447 (average age 39)

• Doctoral: 31%• Master’s: 47%• Baccalaureate: 21%• Certificate: 1%

• Enrollment status: 94% part-time• Faculty: 1,398 (82% doctoral degree)

Capella University

© 2008 Capella University - Confidential - Do not distribute 3

Value Creation Model

© 2008 Capella University - Confidential - Do not distribute 4

Academic Program Definition

© 2008 Capella University - Confidential - Do not distribute 5

© 2008 Capella University - Confidential - Do not distribute 6

© 2008 Capella University - Confidential - Do not distribute 7

Online Course Design and Development

© 2008 Capella University - Confidential - Do not distribute 8

CRITERION

LINK

© 2008 Capella University - Confidential - Do not distribute 11

Fully-Embedded Assessment Model

• Extends outcome assessment throughout program to develop actionable leading indicators.

• Documents alignment of all scoring guide criteria to course competencies and program learning outcomes.

• Projected university completion by Q4 2012.

© 2008 Capella University - Confidential - Do not distribute 12

FEAM Worked Example

Principles of Organizational Communication

Course Competencies

1. Analyze business communication situations.

2. Analyze the interrelationships of communication within organizational systems.

3. Communicate effectively.

Pre FEAM1. Provides Henry advice.2. Identifies Henry's educational/training needs in regard to communication

competency.3. Compares the relationship between communication and motivation.4. Applies the theories of motivation.5. Applies the 10 related evaluations.

Post-FEAM1. Tailors coaching based on analysis of a direct report’s motivational needs. (3)2. Validates approach relative to the theories of motivation. (1)3. Predicts stakeholder consequences of communication. (2)4. Identifies educational/training needs based on limited work sample. (1)5. Analyzes peer evaluations relative to communication competencies. (2)

u03a1 Scoring Guide Criteria

© 2008 Capella University - Confidential - Do not distribute 13

© 2008 Capella University - Confidential - Do not distribute 14

Make learning matter broadly

• Implementing a fully-embedded assessment model (FEAM) has proven effective in:• Promoting a coherent learning experience.• Faculty collaboration.

• Students have a shared basis to view their learning experience:• As a long-term effort to demonstrate professionally-relevant

learning outcomes, not just a series of assignments and courses.• Faculty have a shared basis to establish a community of practice

around their program’s learning outcomes:• By sharing professional experiences, teaching practices,

instructional strategies, performance expectations, technical innovations, and illustrative case studies.

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

WHO WE ARE

American Public University System has been dedicated to educating those who serve since our doors first opened in 1991. The American Public University System (APUS) is a fully online system, encompassing both American Military University and American Public University. We are regionally accredited by the Higher Learning Commission (HLC) of the North Central Association.

“Educating Those Who Serve”

• Founded as American Military University (AMU) in

1991 by James P. Etter, a Marine Corps Officer

• In 2002, AMU expanded into the American Public University System (APUS), adding American Public University (APU)

• Nationally Accredited (DETC) in 1995

• Regionally Accredited (NCA) in 2006

• Serving over 96,000 students worldwide

Mission: To provide quality higher education with emphasis on educating the nation’s military and public service communities by offering respected, relevant, accessible and affordable, student-focused online programs, which prepare them for service and leadership in a diverse, global society.

STRATEGY:

• Focus on quality and affordability

• Graduate successful alumni who make important contributions to society and their professions

• Enhance America’s competitiveness by meaningfully increasing the population withcollege degrees

• Collaborate with other universities to create solutions to U.S. higher education challenges

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

2011: Celebrating 20 Years of Service and Leadership

Data Driven Culture

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“Quality begins on the inside . . . and then works its way out.” -Bob Moawad

• Internal and external benchmarking

• Data tools for stakeholders - dashboards, program and course level fact books, data warehouse, student learning assessment reports.

NATIONALLY BENCHMARKED INSTRUMENTS

18

National Survey of Student

Engagement

Proficiency Profile Tests

Major Field Tests

Community of Inquiry

Survey

APUS utilizes a number of nationally benchmarked tests and validated instruments to measure student engagement, student satisfaction, and the achievement of student learning outcomes.

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Direct and Indirect Measures of Assessment

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“Experts often possess more data than judgment.” – Colin Powell

Regular Dissemination of Data to Stakeholders: Dashboards

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“In the spirit of science, there really is no such thing as a "failed experiment." Any test that yields valid data is a valid test.” - Adam Savage

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Regular Dissemination of Data to Stakeholders: Course and Program Level Fact Books

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Alignment of Student Learning Outcomes

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Culminating Experience

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Key Course Embedded Assessments

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Assessment Data

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Checklist

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Student Learning Assessment Report: Changes Based on Data

PROGRAM REVIEW PROCESS

External Review

Analysis

Program Review

Meeting

Three Year Plan

Follow-Up

Data Collection

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

“Quality is never an accident; it is always the result of high intention, sincere effort, intelligent direction and skillful execution; it represents the wise choice of many”William A. Foster

“One of the great mistakes is to judge policies and programs by their intentions rather than their results.” Milton Friedman

Data Aggregation & Analysis

• Course books

• Electronic resources

• Learning strategies

Library and Learning Resources

• Student learning outcomes

• Instructional strategies

• Evaluation procedures

• Academic rigor

Curriculum Assessment

• Expert reviewer report

• Industry Advisory Council report

External Reviewer Feedback

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

• Analysis of faculty credentials and expertise to ensure breadth and diversity

Faculty

• Student demographic information

• Enrollment History

• Growth trends

Students

• Curricular Mapping

• Assessment measures

• Fact books

• Student learning assessment reports

• reports

Learning Outcomes Assessment

Data Aggregation & Analysis

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

• Program benchmarking with similar programs and institutions

Program Benchmarking

• Evaluation of findings

• Program recommendations

• Three year proposed strategic plan

Program Directory Summary

• Dean’s observations

• Meeting minutes

Review and Findings

Data Aggregation & Analysis

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

CONTACT INFORMATION

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Jennifer Stephens Helm, Ph.D.jstephens@apus.edu

Vice-President, Institutional Research and Assessment

Kaplan University

• Offering online programs since 2001• 58,000+ students online; 7,000+ students at campuses• For-profit• Certificates, Associates, Bachelors & Masters• Incoming students have average of four NCES-identified

risk factors• 1,000+ courses• 4 tracks; 14+ term starts per year; overlapping terms• Outcomes-based institution

So We do All this Assessment …

Measuring activities is usefulExample: How many curriculum changes did we make this year?

Assessing what is really going on is essentialExample: How many of those changes improved retention?

Requirement:Sustainable infrastructure to support assessment approach

Result:Meaningful analysis of data to generate actionable information

• Data collection on outcomes• Real-time assessment data

reporting to faculty• Consistency of outcome and rubric• Assignments mapped to outcomes

and rubrics

• Curriculum development• Structured content

development and management

• Centralized framework for consistency across courses

• Content available in a variety of formats

Assessment-Related Infrastructure

Outside VendorUniversity

Course Authoring

Reporting and Analytics Database

Outcome Repository

Outcomes Package

• Single source of truth for course outcomes and rubrics

• Course to program mapping of outcomes

• Course to course mapping• Versions of all outcomes

and rubrics

Course Delivery

• All course content

How do we know if we are

improving learning?

Sustainable Systems

Information at the Course Level

Information at the Institutional Level

Example: Improvement as defined by improvement in outcomes achievement levels and student retention

Source:

Thayer E. Reed, Jason Levin & Geri H. Malandra (2011): Closing the Assessment Loop by Design, Change: The Magazine of Higher Learning, 43:5, 44-52

Assessment and Academic Quality

at the Rockies

George LucasDirector of Academic Quality

University of the Rockies

Private, independent graduate school of the social and behavioral sciences.

Master's and Doctorate degree programs both on campus and online

Two schools including the School of Professional Psychology, and the School of Organizational Leadership

Clinical Specialization at the campus

Academic Quality and Assessmentat the Rockies are grounded in

Learning Outcomes.

Institutional

Programmatic

Course

Using established Learning Outcomes as a Foundation, University of the Rockies has:• Acquired application tools and resources,• Incorporated and leveraged them in a coordinated manner,• Aligned our Learning Outcomes, and (this quarter)• Developed and Launched a Comprehensive Assessment Plan

Institutional

Programmatic

Course

Our Mission

Two Schools w/Master and Doctoral Programs

Our Curriculum and Syllabi

What are the steps we’ve taken to reach this point?

What will we do in the coming months and in 2012?

2009

2010

2011

2012Gates Model

Doctor of Psychology “Gates” Model for Assessment

Admission Process

Courses

(early, mid, late)

Comprehensive

Internship

Dissertation Process

Post Doc and Post

Graduation Licensure

Practicum 1 Thru 5

Multiple Assessment Methods align with the student experience and learning context including but not limited to Holistic Assessment, Evaluations, Grades, Annual Review, Licensure Exams, etc.

2009

2010

2011

2012Gates Model

Curriculum Mapping

Curriculum Mapping and IRMA tables in

Learning Outcomes are identified in all Curriculum with our Development Template

A successful and comprehensive Curriculum Development Process

2009

2010

2011

2012Gates Model

“Convergence”Curriculum Mapping

Usage Statistics

Integrated Rubrics, Holistic and Analytic Assessment

Faculty Development/Accreditation

Mapping, Outcomes Alignment

Quality Assurance, Curriculum Support

Student Lifecycle

2009

2010

2011

2012Gates Model

“Convergence”

Continuous Improvement

Curriculum Mapping

Cycle of Continuous Improvement

www.ala.org

Assessment that is “done right” never ends!

2009

2010

2011

2012Gates Model

“Convergence”

Continuous Improvement

Curriculum Mapping

Fitting it all together…

Our ongoing goal is to assess and continually improve Institutional Quality

and improve the student experience.

Quality and Assessment belong together.

To continually improve, you must continually:

MeasureAssess

InnovateEvolve

Baseline Concepts of our Assessment Strategy:

University of the Rockies Assessment Plan:

Three dimensions:• Academic Achievement• Student Engagement• Faculty Engagement

Seven (w/potentially more) categoriesserving as data sources:

• Surveys• Campus Vue• WayPoint Outcomes• eCollege• Library Writing Center• Curriculum Development• Quality Matters ™

Assessment and

Quality “Coding Table”

Usage Statistics

Integrated Rubrics, Holistic and Analytic Assessment

Faculty Development/Accreditation

Mapping, Outcomes Alignment

Quality Assurance, Curriculum Support

Student Lifecycle

LEARNING ASSESSMENT SYSTEMS THAT WORK!A FOLLOW-UP REPORT

23nd Annual Conference,Denver, COOctober 28, 2011

Jeff GrannGeorge LucasThayer Reed

Jennifer StephensModerator: Patricia Case