Reporting item r esponse theory r esults Jeffrey B. Brookings Wittenberg University

Reporting item response theory results

Jeffrey B. BrookingsWittenberg University

Presented at the SAMR/SWPA Symposium:

Handy tips for communicating and

reporting your findings

April 5, 2013

Ph.D. Comics, 2013

Item Response Theory1. Mathematical models that probabilistically describe the

relation between a person’s response to an item and his/her standing on a latent trait.

2. The Rasch model—a “one-parameter” model (difficulty)—locates person ability and item difficulty on the same scale (logits or log odds).

3. “…a person having a greater ability than another person should have the greater probability of solving any item of the type in question, and similarly, one item being more difficult than another means that for any one person the probability of solving the second item is the greater one.” (Rasch, 1960, p. 117)

4. The purpose of Rasch analysis is to produce unidimensional measures that cover a wide range of the latent trait.

Reporting results froma Rasch analysis

1. Item and scale descriptive statistics

2. PCA of standardized residuals following extraction of the Rasch component (test for unidimensionality)

3. Item “difficulty” estimates (in logits)

4. Item fit statistics

5. Item characteristic curves (ICCs)

6. Category response curves (CRRs)

7. Person/item map

8. Person/item separation reliability

The Psychosocial Risk Factor Survey(Eichenauer, Feltz, Wilson, & Brookings, 2010)

• Assesses psychosocial risk factors for cardiac disease

• 70 items, 5-point response scale: 0 - “Strongly Agree” to 4 - “Strongly Disagree”

• Scales: Depression, Anxiety, Hostility, Social Isolation, and Emotional Guardedness.

• Analyses: Responses to the 14 Depression Scale items (340 patients from five cardiac rehabilitation programs in the Midwest)

Rasch Item Statistics

PCA of Standardized Residuals

Total raw variance in observations 29.5 100.0% Raw variance explained by measures 15.5 52.6% Raw variance explained by persons 6.4 21.6% Raw variance explained by items 9.2 31.0% Raw unexplained variance (total) 14.0 47.4% Unexplained variance in 1st contrast 1.9 6.4% Unexplained variance in 2nd contrast 1.6 5.5% Unexplained variance in 3rd contrast 1.3 4.4% Unexplained variance in 4th contrast 1.2 4.2% Unexplained variance in 5th contrast 1.2 4.0%

0.1 1 10-2

-1

0

1

2

3

PRFS2

PRFS7

PRFS12

PRFS17

PRFS22PRFS27

PRFS32

PRFS37

PRFS42

PRFS47

PRFS52

PRFS57

PRFS62PRFS67

Figure 1. Outfit Plot for PRFS Depression Items

Overfit Outfit Mean-square (log-scaled) Underfit

M

easu

res

Item characteristic curve—with 95% CI—for item 27: “My thoughts feel so scattered lately”

Item characteristic curve—with 95% CI—for item 12:“I think more about ending my life lately”

Rasch Category Responses

Person/Item Map

Mean person measure = -0.94

Mean item measure = .00

Reliability

• Person separation reliability – Analogous to Cronbach’s α; degree to which the scale differentiates persons; range 0 – 1– For PRFS Depression: .88

• Item separation reliability – Degree to which item difficulties are differentiated; range 0 – 1– For PRFS Depression: .99

Summary of Rasch Analysis forthe PRFS Depression Scale

• Good evidence for unidimensionality• Mean point-measure r = .626• Acceptable person and item separation

reliabilities (.88 and .99, respectively)• Some misalignment of persons and items• One mis-fitting item: #12 (“I think more

about ending my life lately”)

Recommended Reading

Bond, T.G., & Fox, C.M. (2007). Appling the Rasch model:

Fundamental measurement in the human sciences

(2nd ed.). Mahwah, NJ: Erlbaum.