Architecting Evaluation and Assessment Using a Design ... · Architecting Evaluation and Assessment—Using a Design-based Approach to Develop a Complete and Systematic Approach Presenters:

Architecting Evaluation and Assessment—Using a Design-based

Approach to Develop a Complete and Systematic Approach

Presenters:

Dr. Douglas Gilbert – University of the Rockies

Dr. Peter Pintus – University of the Rockies

Dr. Olin Oedekoven – Peregrine Academic Services

@ACBSPAccredited #ACBSP2015

Agenda

The Challenge of Assessment

ACBSP Assessment Requirements

Using a Design Approach to Architect a Model

Multi-Tiered Assessment Instrumentation

Examples

Conclusions

2 @ACBSPAccredited #ACBSP2015





Examples

Conclusions


The Assessment Challenge: Four Key Questions

1. Do we know what graduates need to know and be able to do?

2. Does the curriculum provide what graduates need to know and be able to do?

3. Are students on a learning path for relevant knowledge, skills, and abilities?

4. Do graduates have the competencies demanded by employers?






Examples

Conclusions


Key ACBSP Assessment Requirements

• Criterion 4.1 – Must have a “learning outcomes assessment program”

a. Program learning objectives (PLOs) b. An assessment process c. Internal assessment approach d. External assessment approach e. Formative and summative assessment approach

• Criterion 4.2 – Trends

• Criterion 4.3 – Time series benchmarked comparisons

and also…


Setting strategic direction

•Standard 2 – “Process … to better

address key student and program

performance requirements”






Examples

Conclusions


What is a “complete and systematic approach”?

• Complete — “[H]aving all necessary parts : not lacking anything”

(Merriam-Webster.com. 2015. http://www.merriam-webster.com)

• Systematic — “The term "systematic" refers to approaches that are repeatable

and use data and information so learning is possible.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 71).

— “Well-ordered, repeatable, and exhibiting the use of data and information so that learning is possible.” (Baldrige Excellence Framework 2015-16, p. 53).

• Approach — “[T]he methods used by an organization to address the criteria

requirements. Approach includes the appropriateness of the methods to the requirements and the effectiveness of their use.” (ACBSP Bac/Grad Standards and Criteria—Rev J., December 2014, p. 62).


Is this your assessment design?


A Problematic Assessment System

11

Data: Student surveys and

classroom work, accumulated by

faculty in classroom

Uses: Individual Faculty

Efforts

Data: Departmental studies, e.g.

accounting, OB, etc.

Uses: Departmental Improvement

Data: Institutional res--

• Surveys (NSSE, EOCS, etc.)

• Std-ized tests • IPEDS • Alumni

Uses: Reporting

Uses: Program reviews

Data: Programmatic

assessments, e.g. portfolios.


Data: Academic support

and co-curricular, e.g. library, IT, student

affairs, etc.

Uses: Individual dept. improvement


Faculty Governance

Curriculum approval

Provost / VPAA

Assessment Advisory Committee

Dean / Dept. Chair

Program Chairs

Strategic Planning

VPs

Institutional Research

Adapted from Wolvoord (2010) @ACBSPAccredited #ACBSP2015

Common “Disconnects”

• Structure — Academic data islands of classroom, departments, and programs

— Institutional and instructional data not integrated

• Process — Blurred or non-existent assessment roles and responsibilities

— Non-systematic or isolated cycles of improvement

• Technology — “One-off” faculty classroom technology approaches

— Wide range of surveys and assessment automation

• People — Separated, disjointed assessment responsibilities

— Holistic knowledge management competencies lacking

— Little cross-functional / inter-departmental perspective


A Design-Based Assessment Architecture

13

Business Unit Strategy

Mission Vision Values

Structure

Data Definitions

KPIs / SLOs

Control Limits

Technology People

Architecture Enablers

Policies Procedures Instruments

Process

Performance

Measurement

Performance

Analysis / Review

Performance

Improvement

Technology

Standards

Organizational

Learning Focus

Analysis

Technology

Knowledge Mgmt

Competencies

Collection

Technology

Process-based

Organization


The Common Singular Focus

14

Business Unit Strategy

Mission Vision Values

Structure

Data Definitions

KPIs / SLOs

Control Limits

Technology People


Policies Procedures Instruments

Process

Performance

Measurement

Performance

Analysis / Review

Performance

Improvement

Technology

Standards

Organizational

Learning Focus

Analysis

Technology

Knowledge Mgmt

Competencies

Collection

Technology

Process-based

Organization






Examples

Conclusions


• Exams / Standardized tests

• Case study analyses

• Rubric scores for assignments, reports, and research

projects

• Rubric scores for writing, oral presentations, and

performances

• Observations of field work, internship performance,

service learning, consulting projects

• Quizzes

• Class discussion participation

Course evaluations

Grades not based on explicit criteria related to clear

learning goals

Test blueprints (outlines of the concepts and skills

covered on tests)

Percent of class time spent in active learning

Number of student hours spent on service learning

Number of student hours spent on homework

Number of student hours spent at intellectual or cultural

activities related to the course

Cours

e

Academ

ic

Pro

gra

m

Busi

ness

Unit

F

orm

ativ

e

Su

mm

ativ

e

Fo

rma

tive

S

um

ma

tive

Perfo

rmance

Managem

ent

Asse

ssment

Financial Focus

Student

Focus

Customer

Focus

Workforce

Focus

Leadership and Governance Focus

Yesterday

Today

Tomorrow

Operational Environment

Direct Indirect

Capstone projects, senior theses, exhibits, or

performances (internal to school only)

Student publications or conference presentations

Employer and internship supervisor ratings of students'

performance

Externally normed assessments, e.g. Peregrine, ETS,

etc.

Pass rates or scores on licensure, certification, or

subject area tests

Job Placement

Department or program review data

Employer or alumni surveys

Focus group interviews with students, faculty members,

or employees

Student perception surveys

Internal External

Sta

ndard

4.1

a,b

Sta

ndard

2

or

Outcome / Instrumentation Overview

16

Sta

ndard

4.1

c,d

,e

Course Level Elements

17

Program Level Elements

18

Business Unit Elements

19





Examples

Conclusions


Overview of Examples

1. College-level KPI Reports (Standard 2)

2. Signature assignments using standard rubric (Standard 4.1c)

3. NPS – External Measure (Standard 4.1d)

4. Trending (longitudinal) dashboard (Standard 4.2)

5. External test benchmarking approach (Standard 4.3)


College Dashboard - Layer 1 (National Baldrige Recipient Example)

22 Source: Dallas Community College District

ACBSP Standard 2


College Dashboard - Layer 2 (National Baldrige Recipient Example)

23 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015

College Dashboard - Layer 3

(National Baldrige Recipient Example)

24 Source: Dallas Community College District @ACBSPAccredited #ACBSP2015

Signature Assignments Using a Standardized Rubric (Standard 4.1c)

25

Standardized Rubric


NPS as an External Measure (Standard 4.1d)

26

Source: Bain & Co.

http://www.netpromotersystem.com/about/measuring-your-net-promoter-score.aspx


Trending (longitudinal) dashboard (Standard 4.2)


External test benchmarking approach (Standard 4.3)






Examples

Conclusions


Putting it all together Signature Assignment Example

30

Signature assignment aligned to Mission, Vision, & Values

Structure

• Rubric structure

• Relevant student

learning

outcomes

• Targets

Technology

• LMS

• Rubric software

• Analysis

software

People

• Knowledge mgt

competencies

• CQI approach

• SIPOC for fit of

process


Assessment Policies Overall assessment

cycle Rubric types

Process

• Signature

assignment

selection

• Scoring / rubric

process

• Cycles of review /

improvement


Additional Resources

•Presentation available at

http://bitly.com/ACBSP2015

•White paper to be available at same

location




Architecting Evaluation and Assessment Using a Design ... · Architecting Evaluation and Assessment—Using a Design-based Approach to Develop a Complete and Systematic Approach Presenters:

Documents