Designing Accessible Reading Assessments Examining Test Items for Differential Distractor Functioning Among Students with Learning Disabilities Kyndra.

Designing Accessible Reading Assessments

Examining Test Items for Differential Distractor Functioning

Among Students with Learning Disabilities

Kyndra Middleton

The University of Iowa

[email protected]

April 10, 2007

Designing Accessible Reading Assessments

Purpose of the Study

• To examine whether different distractor choices functioned differentially for students with learning disabilities who did not receive an accommodation, students with learning disabilities who received a read-aloud accommodation, and students with learning disabilities who received an accommodation other than a read-aloud

• To help determine whether a test can be modified for students with learning disabilities by removing a distractor choice while maintaining adequate test validity and information

Designing Accessible Reading Assessments

Instrument Used

• 4th grade English Language Arts assessment from a criterion-referenced statewide test– Operational test data

– Reading (42 MC items)– Writing (33 MC items)

Designing Accessible Reading Assessments

Sample Used

• 30,000 non-LD students sampled from 298,622 students

• 9,056 LD students who did not receive an accommodation

• 4,727 LD students who received an accommodation based on their IEP/504 plan

• 1,371 LD students who received an accommodation based on their IEP/504 plus a read aloud accommodation

Designing Accessible Reading Assessments

Sample Used cont’d

Subgroup Far Below Basic

Below Basic

Basic Proficient Advanced Total Number of Students

No Disability 4% 11% 31% 30% 25% 30,000

Learning Disability—no

accommodation

32% 34% 25% 7% 2% 9,056

Learning Disability—IEP/5

04

35% 38% 23% 4% 1% 4,727

Learning Disability—IEP/504 & read aloud

26% 36% 30% 6% 1% 1,371

Percentage of Students at Each Proficiency Level

Designing Accessible Reading Assessments

Sample Used cont’dReference-Focal Comparisons

Reference Group Focal Group

No Disability (Group 0) Learning Disability—no accommodation (Group 20)

No Disability (Group 0) Learning Disability—IEP/504 (Group 21)

No Disability (Group 0) Learning Disability—IEP/504 & read aloud (Group 22)

Learning Disability—no accommodation (Group 20)

Learning Disability—IEP/504* (Group 21)

Learning Disability—no accommodation (Group 20)

Learning Disability—IEP/504 & read aloud (Group 22)

Note: IEP = Individualized Education Plan* = comparison did not show DIF so was not included in the DDF analyses

Designing Accessible Reading Assessments

Procedure

• Examine items that previously displayed DIF for DDF– DDF: when two groups that have been matched on

ability have different probabilities of selecting a distractor

• Standardized Distractor Analysis (SDA)

– Distinguishes between distractors

– Identifies uniformly and nonuniformly biased distractors

– An extension of standardized p-difference

Designing Accessible Reading Assessments

Procedure Used cont’d

• Equation used to test for DDF:

STD(i) =

• : negligible DDF

• : moderate DDF

• : large DDF

10.SDA05.

05.SDA

10.SDA

s

rsfss

W(i)P(i)PW

Designing Accessible Reading Assessments

Results

• 70% of the items that displayed DIF also displayed DDF

• 100% of DDF occurred with a comparison between the read aloud and some other group

• 64% of the distractors that displayed DDF were in favor of the read aloud group

Designing Accessible Reading Assessments

Option D Difference*(moderate DIF)

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Total Score

Diffe

ren

ce in

Perc

en t

F-R

Designing Accessible Reading Assessments

Option B*: large DIF

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Total Score

Dif

fere

nc

e in

Pe

rce

nt

F-R

Designing Accessible Reading Assessments

Option B Difference(no DDF)

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Total Score

Diffe

ren

ce in

Perc

ent

F-R

Designing Accessible Reading Assessments

Option B: moderate DDF

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Total Score

Dif

fere

nc

e in

Pe

rce

nt

F-R

Designing Accessible Reading Assessments

Option C Difference(large DDF)

-50

-40

-30

-20

-10

0

10

20

30

40

50

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75

Total Score

Diffe

ren

ce in

Perc

en

t

F-R

Designing Accessible Reading Assessments

Results cont’dComparison Groups

0-20 0-21 0-22 20-21 20-22 Ite

m

A B C D A B C D A B C D A B C D A B C D 3 *R

R

*R

R

10 + *F

F

+ *F

F 13 + + *F

F

- 25 ++ *F 32 *R

R

- 33 *F

F

+ 34 + *F 45 *R

R

56 *R 64 *R

R

- *R

R

- *R

R

Note: +: moderate DDF in favor of the focal group++: large DDF in favor of the focal group-: moderate DDF in favor of the reference group*R: DIF in favor of the reference group*F: DIF in favor of the focal groupShaded box: Items that did not exhibit DIF

Designing Accessible Reading Assessments

Results cont’d

• 17% that assessed reading standards showed DDF

• 9% that assessed writing standards showed DDF

• No observed pattern across content or cognitive area between groups

Designing Accessible Reading Assessments

Results cont’d

• Item that displayed large DDF was the most difficult item that displayed DIF

• One item displayed DDF in each of the distractors (two favoring the read aloud group and one favoring the non-LD group)

• Item that displayed DDF in two of its distractors was a spelling item– Both were homophones– Additional difficulty caused by read aloud

Designing Accessible Reading Assessments

Conclusions/Future Research

• Measurement dissimilarity between read aloud group and other groups

• Exploratory study: More research needed to determine whether read aloud actually alters test’s validity

• Matched on ability to provide more information at extremes