Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Using DIF to Examine the Validity and Fairness of Assessments for Students With.
Post on 27-Mar-2015
212 Views
Preview:
Transcript
Copyright © 2004 Educational Testing Service
Listening. Learning. Leading.
Using DIF to Examine the Validity and Fairness of Assessments for Students With Disabilities
Cara Cahalan-Laitusis & Linda CookEducational Testing Service
Copyright © 2004 Educational Testing Service
Page 2
Examining Assessments for Students with Disabilities
• DARA Research Using Archival Data– State standards based assessment
– English-language Arts
• Purpose of Studies– Determine how a student’s score is effected
by disability or accommodation
– To improve assessments for all students
Copyright © 2004 Educational Testing Service
Page 3
Overview of Presentation
• What is DIF and why are DIF procedures useful
• Some issues related to using DIF procedures for students with disabilities
• How has DIF been applied to evaluate assessments for students with disabilities
• A summary of a DARA DIF study
Copyright © 2004 Educational Testing Service
Page 4
Why DIF Procedures are Useful
• DIF studies first carried out on a frequent basis in 1960s– Evaluate ethnic differences– Identify and remove biased items
• DIF exists if examinees at same ability level perform differently on the item
• Examples– Deaf and HH examinees– Test item draws on knowledge of popular music
Copyright © 2004 Educational Testing Service
Page 5
Some Issues Related to Using DIF Procedures For Students With Disabilities
• There are some differences between DIF for well defined subgroups and DIF for students with disabilities– Definition of groups– Matching criterion– Ability differences between groups
Copyright © 2004 Educational Testing Service
Page 6
Reviews of Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities
• Tindal & Fuchs, 2000; Sireci, Li & Scarpati, 2003; Thompson, Blount & Thurlow, 2002; Pitoniak & Royer, 2001
• Very little research focusing on DIF
Copyright © 2004 Educational Testing Service
Page 7
Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities
• Lewis, Green & Miller, 1999– Read aloud and extended time– 35 different groups with disabilities– More ELA items had DIF than Math
items– DIF balanced out
Copyright © 2004 Educational Testing Service
Page 8
Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.)
• Bielinski, Thurlow, Ysseldyke, Freidebach & Freidebach, 2001– Read aloud accommodations
– Reading and math tests for grades 3 and 4
– 41 multiple choice grade 3 reading items
– 32 multiple choice grade 4 math items
– DIF for 30 reading and 7 math items
– Results indicated read aloud accommodation may affect comparability of items
Copyright © 2004 Educational Testing Service
Page 9
Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.)
• Bolt,2004– Read aloud accommodation is less appropriate
for reading tests than for non-reading tests
– Accommodations are more appropriate for students with sensory and physical disabilities than for students with cognitive disabilities
– Accommodations fall on a continuum in terms of the degree to which they affect the validity of a test
Copyright © 2004 Educational Testing Service
Page 10
Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.)
• Barton& Finch, 2004– Second edition of the TerraNova
– Language and math subtests for grades 3,5,8
• DIF for items with high amount of text provided via read aloud conditions to accommodated students– Concluded some level of boost for students
with disabilities with read aloud accommodation
Copyright © 2004 Educational Testing Service
Page 11
DARA DIF Analysis
• Purpose is to provide information that will lead to the development of more accessible tests for students with disabilities
• Series of studies– English language arts– Grades 3,4,7, and 8– Learning disabled student– Accommodations
• Extended time, read aloud
Copyright © 2004 Educational Testing Service
Page 12
Purpose
1. Identify characteristics of test questions that result in Differential Item Functioning (DIF) for students with Learning Disabilities.
2. Make recommendations to test developers regarding the improvement of test questions for students with Learning Disabilities.
Copyright © 2004 Educational Testing Service
Page 13
Reference-Focal ComparisonsReference Group
No Disability
No Disability
No Disability
Focal Group
LD No Accommodation
LD Extra Time
LD Read Aloud and Extra Time
Copyright © 2004 Educational Testing Service
Page 14
Content of Tests Examined
• English Language Arts Grades 3• 71 multiple-choice items
• 86% reading
• 14% writing strategies
• English Language Arts Grade 7• 81 multiple-choice items
• 77% reading
• 23% writing strategies
Copyright © 2004 Educational Testing Service
Page 15
Proficiency Levels for ELA, Grade 3
Subgroup Below Basic
Basic and Above
Total
No LD 33.5% 66.5% 437,398
LD Time 86.8% 13.3% 2,069
LD Read & Time
79.6% 20.4% 2,003
Copyright © 2004 Educational Testing Service
Page 16
Proficiency Levels for ELA, Grade 7
Subgroup Below Basic
Basic and Above
Total
No LD 26.4% 73.6% 437,689
LD Time 82.3% 17.6% 1,787
LD Read 85.1% 14.9% 2,154
Copyright © 2004 Educational Testing Service
Page 17
DIF Categories ELA Grade 3 Extra Time
Easy
Difficult
FavorsSWOD
FavorsLD
.00
.25
.50
.75
1.00
-3 -2 -1 0 1 2 3
MH D-DIF
Per
cent
Cor
rect
A
Copyright © 2004 Educational Testing Service
Page 18
DIF Categories ELA Grade 3Read Aloud and Extra Time
Easy
Difficult
FavorsSWOD
FavorsLD
.00
.25
.50
.75
1.00
-3 -2 -1 0 1 2 3
MH D-DIF
Per
cent
Cor
rect
A
B
C
Copyright © 2004 Educational Testing Service
Page 19
DIF Categories ELA Grade 7 Extra Time
Easy
Difficult
FavorsSWOD
FavorsLD
.00
.25
.50
.75
1.00
-3 -2 -1 0 1 2 3
MH D-DIF
Perc
ent C
orre
ct
A
B
Copyright © 2004 Educational Testing Service
Page 20
DIF Categories ELA Grade 7 Read Aloud and Extra Time
Easy
Difficult
FavorsSWOD
FavorsLD
.00
.25
.50
.75
1.00
-3 -2 -1 0 1 2 3
MH D-DIF
Perc
ent C
orre
ct
A
B
C
Copyright © 2004 Educational Testing Service
Page 21
Conclusions
• DIF used in an exploratory manner may provide useful insights that will help us understand how to develop good assessments for students with disabilities.
Copyright © 2004 Educational Testing Service
Page 22
Areas for Further Research
• Choice of Criteria
• Choice of Method
• DARA is Currently Carrying Out Research to Address Both Questions
Copyright © 2004 Educational Testing Service
Page 23
Next Steps
• Replicate the findings in this study using data from other ELA tests
• Follow up the study with other types of research– Item manipulation– Protocol analysis– Factor analysis
top related