Skills Diagnosis with Latent Variable Models

Skills Diagnosis with Latent Variable Models

Topic 1:

A New Diagnostic Paradigm

• Assessments should aim to improve, and not merely ascertain the status of student learning

• For test scores to facilitate learning, they need to be interpretative, diagnostic, highly informative, and potentially prescriptive

• Most large-scale assessments are based on traditional unidimensional IRT models that only provide single overall scores

• These scores are useful primarily for ordering students along a continuum

Introduction

• Alternative psychometric models that can provide inferences more relevant to instruction and learning currently exist

• These models are called cognitive diagnosis models (CDMs)

• Alternatively, they are referred to as diagnostic classification models (DCMs)

• CDMs are multiple discrete latent variable models

• They are developed specifically for diagnosing the presence or absence of multiple fine-grained attributes (e.g. skills, cognitive processes or problem-solving strategies)

• Fundamental difference between IRT and CDM: A fraction subtraction example

• IRT: performance is based on a unidimensional continuous latent trait

• Students with higher latent traits have higher probability of answering the question correctly

7412 122

0.00

0.20

0.40

0.60

0.80

1.00

-3.5 -2.5 -1.5 -0.5 0.5 1.5 2.5 3.5

( 1| 0.8) 0.3P X

( 1| 1.2) 0.9P X

0.8 1.2

• Fundamental difference between IRT and CDM: A fraction subtraction example

• IRT: performance is based on a unidimensional continuous latent trait

• Students with higher latent traits have higher probability of answering the question correctly

• CDM: performance is based on binary latent attribute vector

• Successful performance on the task requires a series of successful implementations of the attributes specified for the task

7412 122

1( , , )K

7412 122

127

12161

1291

431

• Required attributes:

(1) Borrowing from whole

(2) Basic fraction subtraction (3) Reducing

• Other attributes:

(5) Converting whole to fraction

(4) Separating whole from fraction

• The response vector of examinee i will be denoted by ,

• The response vector contains J items, as in,

• The attribute vector of examinee i will be denoted by

• Each attribute vector or pattern defines a unique latent class

• Thus, K attributes define latent classes

Basic Elements and Notations of CDM

• Example: When , the total number of latent classes is

• Although arbitrary, we can associate the following attribute vectors with the following latent classes:

• Like IRT, CDM requires an binary response matrix as input

• Unlike IRT, CDM in addition requires a binary matrix called the Q-matrix as input

• The rows of the Q-matrix pertain to the items, whereas the columns the attributes

• The 1s in the jth row of the Q-matrix identifies the attributes required for item j

Basic CDM Input

Attribute

Item

(1)

Borrow from the whole

(2)

Basic fraction

subtraction

(3)

Reduce

(4) Separate

whole from

fraction

(5)

Convert whole to fraction

Examples of Attribute Specification

4 71. 2

12 12 1 1 1 0 0

3 42. 7 2

5 5 11 0 1 0

Attribute

Item

(1)

Borrow from the whole

(2)

Basic fraction

subtraction

(3)

Reduce

(4) Separate

whole from

fraction

(5)

Convert whole to fraction

Examples of Attribute Specification

13. 2

3

74. 3 2

8

• The goal of CDM is to make inference about the attribute vector

• The basic CDM output gives the (posterior) probability the examinee has mastered each of the attributes

• That is, we get

• For example, , indicates that we are quite certain that examinee has already mastered attribute 1

Basic CDM Output

• Each examinee gets a vector of posterior probabilities

• For reporting purposes, we may want to convert the probabilities into 0s and 1s

• We can use different rules for this conversion

• If ;

Otherwise,

Example:

• Each examinee gets a vector of posterior probabilities

• For reporting purposes, we may want to convert the probabilities into 0s and 1s

• We can use different rules for this conversion

• If ;

Otherwise,

• If ; or

If ;

Otherwise,

Example:

? – means we do not have sufficient evidence to conclude one way or the other