Show Me an Evidential Approach to Assessment Design Michael Rosenfeld F. Jay Breyer David M. Williamson Barbara Showers.

Show Me an Evidential Approach to Assessment Design

Michael RosenfeldF. Jay Breyer

David M. WilliamsonBarbara Showers

A Brief Introduction to Evidence Centered Design (ECD)

Michael Rosenfeld, Ph.D.Rosenfeld & Associates

What is Evidence Centered Design? A formal framework for assessment

design Emphasizes assessment as an

evidential argument Encourages models and

representations that strengthen validity arguments

Focuses the test development process to better target decisions of interest

The Basic Models of ECDProficiency Model Evidence Models

Statmodel

EvidenceRules

Task Models

Features1.

xxxxx2.

xxxxx

3. xxxxx

Proficiency Model – What you want to measure

Evidence Model – How to recognize & interpret observable evidence of unobservable proficiencies

Task Models – How to elicit valid and reliable evidence

ECD Validity Chain

Evidence

Evidence

Evidence

Proficiency

Tasks

Tasks

Tasks

Tasks

Tasks

Task Model

Task Model

Task Model

Task Model

What is Gained from ECD? Better understanding of the domain, for new

constructs Explicit chain of evidence from assessment goals,

to models of evidence, to assessment tasks Strong construct validity argument Rapid and efficient item generation from models

automatic item generation Opportunities for innovation, task types other than

multiple choice Philosophical shift from content-authoring to

evidential argument

Typical Test DevelopmentContent Specification

Item Writing

Test Blueprint

Analysis

Cut Scores

The Job

Knowledge & Tasks

Major DimensionsJob Analysis

Incumbent Survey

Content Linking

Statistical Modeling

Typical Test Development and Evidence Centered Design

Content Specification

Item Writing

Test Blueprint

Analysis

Cut Scores

Typical Test Development Evidence Centered Design

Proficiency Model

Task Model

Evidence Model

Tasks

Evidence Centered Design

Proficiency Model

Task Model

Evidence Model

Tasks

Claims (Criterion)

Knowledge, Strategies & Skills

Reporting Variables

Content Linking

Evidential Argument

Job Analysis

Cognitive Task Analysis

Statistical Modeling

Task Design

Still to Come

Example of the Proficiency Modeling process (Jay)

Illustration of Evidence Modeling and Task Modeling (David)

Commentary from an applied perspective (Barbara)

Proficiency Models in the ECD Framework

F. Jay Breyer, Ph.D.

Proficiency Model Combination of

Formalization of assessment goals, population and purpose

Formal claims to be made on the basis of the assessment

He/she is competent to operate an automobile

Proficiencies of interest Knowledge of rules, procedures, and

laws Ability to implement driving

regulations in practice Skill in automobile operation

What is a Proficiency? Proficiencies are knowledge, skills

and abilities that provide the basis for making claims about people

Building a Proficiency Model: Assessment Goals

Assessment Goals Protect the public health, safety & welfare

Examinee population All eligible (age, prior training requirements, etc.)

citizens self-selecting into the program Test users/stakeholders

State regulatory bodies State resident drivers Other citizens exposed to driving

Result Usage/Decisions State issuing the license (to issue or deny) Means of official identification

Building a Proficiency Model: Assessment Claims

Formal Claims are developed for assessment goals Claims can be course-grained and general

Can safely operate a motor vehicle These general claims can be be supported by a

variety of more specific sub-claims in a hierarchy Operates a vehicle in conformance with posted

signs Stops at a posted stop sign Obeys posted speed limits

The final hierarchy of claims drives the evidential argument and the score reporting needs of the test

Building a Proficiency Model: Proficiencies Proficiency variables are the basis of

making claims Proficiency variables, like claims, may be

organized hierarchically Values or levels of proficiency variables

may be selected to support claim distinctions

Informed by investigations of the domain and learning within the domain job analyses, cognitive task analyses, learning

models, etc.

Example of a Hypothetical Proficiency Model for Driving

Can drive safely

Understands rulesand regulations

Can operate amotor vehicle

DrivingAbility

KnowRotR

Psycho-motor

Weather& light

Inter-sections

TrafficFlow

Parking

Turning

Speed

Still to Come

Illustration of Evidence Modeling and Task Modeling (David)

Commentary from an applied perspective (Barbara)

Providing the Evidence in Evidence Centered Design

David M. Williamson, Ph.D.

Building Evidence Models Objective: valid inference

Understanding what constitutes evidence and why

Situations that elicit evidential behaviors Task design Statistical representation of evidential value

of observations Integration of scoring with design

ECD models – conceptual scoring

Establishing Evidence in a “Perfect World”

Proficiency Evidence Task

Targets of Inference

Links behaviorsand inference

Behavior Elicitation

Driving Ability•Knowledge of RotR•Psychomotor

Evidence•Knowledge declarations•Performance

Task Requirements•Situation•Outcomes

Establishing Evidence: Driver’s Exam

Proficiency Evidence Task

•Knowledge of RotR•Weather & Light•Traffic Flow•Intersections•Speed

•Psychomotor•Intersections•Speed•Turning•Parking

Driving Ability•Knowledge of RotR•Psychomotor

Intersections•Recognizing signs

•Specifying intersection traffic behaviors

•Following proper procedure

Building Task Models: Why? Elicits behaviors by predictably

manipulating Focus Difficulty Context

Targets what we want to measure Provides a framework for generating

multiple tasks targeting same proficiencies

Building Task Models: What?

To target elicitation of particular evidence, the task model specifies:

Tools provided Directions and stimulus materials Indication of data collected and retained

for scoring Relevance of specified elements to

particular evidence, and particular proficiencies

Evidence as Inference about Proficiency

Proficiency Evidence Task

IdentificationAccumulation

Stage 1: Evidence Identification Identify and summarize response elements

Stage 2: Evidence Accumulation Using these elements to estimate ability

Beneficial Outcomes of ECD

Development direction w evidential focus Integration of expertise/activities (TD, stat,

etc.) in the design process Chain of reasoning

Integrated design – implications of design changes Validity argument

Item modeling AIG Defensibility Directed item writing – future forms

Somewhat more expensive (Initially)

Still to Come

Commentary from an applied perspective (Barbara)

Discussant

Barbara Showers, Ph.D.Director, Office of Education and

ExaminationsWisconsin Department of Regulation

and Licensing