Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Using DIF to Examine the Validity and Fairness of Assessments for Students With.

Copyright © 2004 Educational Testing Service

Listening. Learning. Leading.

Using DIF to Examine the Validity and Fairness of Assessments for Students With Disabilities

Cara Cahalan-Laitusis & Linda CookEducational Testing Service


Page 2

Examining Assessments for Students with Disabilities

• DARA Research Using Archival Data– State standards based assessment

– English-language Arts

• Purpose of Studies– Determine how a student’s score is effected

by disability or accommodation

– To improve assessments for all students


Page 3

Overview of Presentation

• What is DIF and why are DIF procedures useful

• Some issues related to using DIF procedures for students with disabilities

• How has DIF been applied to evaluate assessments for students with disabilities

• A summary of a DARA DIF study


Page 4

Why DIF Procedures are Useful

• DIF studies first carried out on a frequent basis in 1960s– Evaluate ethnic differences– Identify and remove biased items

• DIF exists if examinees at same ability level perform differently on the item

• Examples– Deaf and HH examinees– Test item draws on knowledge of popular music


Page 5

Some Issues Related to Using DIF Procedures For Students With Disabilities

• There are some differences between DIF for well defined subgroups and DIF for students with disabilities– Definition of groups– Matching criterion– Ability differences between groups


Page 6

Reviews of Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities

• Tindal & Fuchs, 2000; Sireci, Li & Scarpati, 2003; Thompson, Blount & Thurlow, 2002; Pitoniak & Royer, 2001

• Very little research focusing on DIF


Page 7

Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities

• Lewis, Green & Miller, 1999– Read aloud and extended time– 35 different groups with disabilities– More ELA items had DIF than Math

items– DIF balanced out


Page 8

Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.)

• Bielinski, Thurlow, Ysseldyke, Freidebach & Freidebach, 2001– Read aloud accommodations

– Reading and math tests for grades 3 and 4

– 41 multiple choice grade 3 reading items

– 32 multiple choice grade 4 math items

– DIF for 30 reading and 7 math items

– Results indicated read aloud accommodation may affect comparability of items


Page 9


• Bolt,2004– Read aloud accommodation is less appropriate

for reading tests than for non-reading tests

– Accommodations are more appropriate for students with sensory and physical disabilities than for students with cognitive disabilities

– Accommodations fall on a continuum in terms of the degree to which they affect the validity of a test


Page 10


• Barton& Finch, 2004– Second edition of the TerraNova

– Language and math subtests for grades 3,5,8

• DIF for items with high amount of text provided via read aloud conditions to accommodated students– Concluded some level of boost for students

with disabilities with read aloud accommodation


Page 11

DARA DIF Analysis

• Purpose is to provide information that will lead to the development of more accessible tests for students with disabilities

• Series of studies– English language arts– Grades 3,4,7, and 8– Learning disabled student– Accommodations

• Extended time, read aloud


Page 12

Purpose

1. Identify characteristics of test questions that result in Differential Item Functioning (DIF) for students with Learning Disabilities.

2. Make recommendations to test developers regarding the improvement of test questions for students with Learning Disabilities.


Page 13

Reference-Focal ComparisonsReference Group

No Disability

No Disability

No Disability

Focal Group

LD No Accommodation

LD Extra Time

LD Read Aloud and Extra Time


Page 14

Content of Tests Examined

• English Language Arts Grades 3• 71 multiple-choice items

• 86% reading

• 14% writing strategies

• English Language Arts Grade 7• 81 multiple-choice items

• 77% reading

• 23% writing strategies


Page 15

Proficiency Levels for ELA, Grade 3

Subgroup Below Basic

Basic and Above

Total

No LD 33.5% 66.5% 437,398

LD Time 86.8% 13.3% 2,069

LD Read & Time

79.6% 20.4% 2,003


Page 16

Proficiency Levels for ELA, Grade 7

Subgroup Below Basic

Basic and Above

Total

No LD 26.4% 73.6% 437,689

LD Time 82.3% 17.6% 1,787

LD Read 85.1% 14.9% 2,154


Page 17

DIF Categories ELA Grade 3 Extra Time

Easy

Difficult

FavorsSWOD

FavorsLD

.00

.25

.50

.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cent

Cor

rect

A


Page 18

DIF Categories ELA Grade 3Read Aloud and Extra Time

Easy

Difficult

FavorsSWOD

FavorsLD

.00

.25

.50

.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Per

cent

Cor

rect

A

B

C


Page 19

DIF Categories ELA Grade 7 Extra Time

Easy

Difficult

FavorsSWOD

FavorsLD

.00

.25

.50

.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Perc

ent C

orre

ct

A

B


Page 20

DIF Categories ELA Grade 7 Read Aloud and Extra Time

Easy

Difficult

FavorsSWOD

FavorsLD

.00

.25

.50

.75

1.00

-3 -2 -1 0 1 2 3

MH D-DIF

Perc

ent C

orre

ct

A

B

C


Page 21

Conclusions

• DIF used in an exploratory manner may provide useful insights that will help us understand how to develop good assessments for students with disabilities.


Page 22

Areas for Further Research

• Choice of Criteria

• Choice of Method

• DARA is Currently Carrying Out Research to Address Both Questions


Page 23

Next Steps

• Replicate the findings in this study using data from other ELA tests

• Follow up the study with other types of research– Item manipulation– Protocol analysis– Factor analysis

Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Using DIF to Examine the Validity and Fairness of Assessments for Students With.

Documents

dif slide

applications of dif

math items dif

dif procedures useful

biased items dif

dara dif study slide

useful dif studies

students score