Potential Bias in Predictive Validity of Universal ...rachaelrobinsonedsi.wiki.westga.edu/file/view/Potential+bias+in... · Potential Bias in Predictive Validity of Universal Screening

Potential Bias in Predictive Validity of UniversalScreening Measures Across Disaggregation Subgroups

John L. HospUniversity of Iowa

Michelle A. HospIowa Department of Education

Janice K. DoleUniversity of Utah

Abstract. Universal screening measures are an integral component of any tieredsystem of instructional delivery. Recent studies of screening measures have oftenexcluded examinations of bias in predictive validity. The present study examineda common screening instrument for evidence of bias in predictive validity acrossthe four disaggregation categories of the No Child Left Behind Act. Performanceof 3,805 students in Grades 1–3 on the Nonsense Word Fluency and Oral ReadingFluency measures of the Dynamic Indicators of Basic Early Literacy Skills wereexamined cross-sectionally in relation to a state criterion-referenced test. Bias inpredictive validity was found, but varied by grade and by disaggregation category.Implications are discussed.

Universal screening is a crucial compo-nent of any comprehensive system of assess-ment (Salvia, Ysseldyke, & Bolt, 2009), espe-cially those used within a problem-solving orresponse to intervention framework (Batscheet al., 2005). Efficient and effective delivery ofthe most appropriate interventions to the rightstudents requires a consistent and accurateprocess of identifying which students needwhat help (Hosp & Ardoin, 2008). While pro-viding this help, it is also crucial to be able toaccurately and efficiently judge each student’sresponse (Barnett et al., 2007). Universalscreening involves the assessment of all stu-dents within a classroom, grade, school, or

district on measures that are valid indicators ofimportant academic or social/emotional out-comes (Ikeda, Neessen, & Witt, 2008). Theseassessments should be quick to administer,score, and provide information that leads tovalid inferences about those outcomes (Hosp& Ardoin, 2008). These inferences are thedecisions that need to be made in identifyingeach student’s level of need as well as group-ing students with similar needs.

Given the recent focus on universalscreening that has come from a renewed em-phasis on problem solving in delivering edu-cational services, it is no surprise that therehave been recent advances in the development

Correspondence regarding this article should be addressed to John L. Hosp, College of Education, N264Lindquist Center, Iowa City, IA 52242; e-mail: [email protected]

Copyright 2011 by the National Association of School Psychologists, ISSN 0279-6015

School Psychology Review,2011, Volume 40, No. 1, pp. 108–131

108

of screening measures (Catts, Fey, Zhang, &Tomblin, 2001; Foorman, Francis, Fletcher,Schatschneider, & Mehta, 1998; O’Connor &Jenkins, 1999). However, there has also beenincreased scrutiny to ensure that they result inreliable data that provide accurate classifica-tion of students as needing intervention or not.Ritchey and Speece (2004) explored charac-teristics of screening assessment that shouldbe considered in the early identification ofreading disabilities. Differences in skills mea-sured, performance tasks required, and contentcoverage are all characteristics that can affectthe classification accuracy of a measure. Thetiming of a measure is important in terms ofboth the interval between screening measure-ment and outcome measurement as well aswhen the screening measurement takes placein a developmental sequence. Selection of out-come is also important as prediction of moreproximal outcomes is likely to be more accu-rate than prediction of more distal ones. Jen-kins, Hudson, and Johnson (2007) suggestedadditional factors to consider when developingand using screening measures such as account-ing for the severity of the problem, differentlevels of risk (use of the dichotomous at risk/not at risk or a polytomous system), and theinclusion of cross-validation of screeningmeasures that is a crucial measurement com-ponent to the development of any measure(Haladyna, 2006).

Predictive Validity

A key component in the determinationof the quality of a screening measure is itspredictive validity. Predictive validity is anindication of how well performance on a cri-terion measure is predicted by performance ona screening measure when there is a differencein the time of administration (typically 3–5months) between the two measures (Salvia etal., 2009). The criterion measure is also de-scribed as a meaningful outcome (Ikeda et al.,2008) such as performance on the state high-stakes test.

Researchers have evaluated the predic-tive validity of Nonsense Word Fluency(NWF) and Oral Reading Fluency (ORF) as

compared to norm-referenced tests of reading(e.g., Woodcock Reading Mastery Test;Woodcock, 1998; see Ritchey, 2008) as wellas the mandated high-stakes state tests ofmany states (e.g., Florida—Buck & Torgesen,2002; Washington—Stage & Jacobson, 2001).Predictive validity coefficients for both NWFand ORF typically average between .65 and.75 (cf. Hintze & Silberglitt, 2005; Ritchey,2008; Roehrig, Petscher, Nettles, Hudson, &Torgesen, 2007; Shanahan, 2003).

Catts, Petscher, Schatschneider, Bridges,and Mendoza (2009) recently extended thispredictive validity work by looking for flooreffects that might influence the predictive ac-curacy of screening measures, particularly Ini-tial Sound Fluency, Phoneme SegmentationFluency, NWF, and ORF from the DynamicIndicators of Basic Early Literacy Skills(DIBELS; Good et al., 2004). Floor effectsoccur when the lower end of the performancerange for a scale does not go low enough toadequately describe participants’ performance(Drew, Hardman, & Hosp, 2008). This is dem-onstrated by a large number of individualsreceiving scores near the minimum possiblescore such that the scores for a group are“bunched” near the minimum performance.This can have a negative effect on predictivevalidity because of the restriction of range ofparticipants’ performance. Using nearly 19,000students from Florida, Catts et al. (2009) dem-onstrated that floor effects were present, whichreduced the predictive validity of the data. Theeffect on predictive validity was most pro-nounced in kindergarten and Grade 1, withdecreasing effect in Grade 2 and little to noeffect in Grade 3. Catts et al. conclude thatmore sensitive measures of early literacy areneeded to overcome these floor effects and theeffect they have on predictive validity.

The Importance of Examining Bias inPredictive Validity

The studies mentioned above have con-tributed to our understanding of screeningmeasures and some of the potential issues thataffect the results provided and the inferencesmade from those results. However, they did

Predictive Validity Bias in Screening

109

not examine the potential differential perfor-mance of screening measures across differentsubgroups of students. The achievement gapbetween subgroups of students has been alongstanding and persistent issue in Americaneducation (Rampsey, Dion, & Donahue,2009). This is one reason that disaggregationacross various traditionally underperformingsubgroups was required for states to demon-strate adequate yearly progress under the NoChild Left Behind Act (NCLB; 2002). Byexplicitly disaggregating the performance ofvarious subgroups (i.e., students from eco-nomically disadvantaged backgrounds, stu-dents with limited English proficiency, stu-dents with disabilities [SwD], and studentsfrom various racial/ethnic backgrounds),schools, districts, and states would be able todetermine whether they were meeting theneeds of all groups of students. However, thisanalysis is on state-identified outcome measuresonly and does not address the potential for dif-ferential prediction on screening measures.

Although overall classification accuracyis an important consideration when evaluatingscreening measures, bias in predictive validityis also an important consideration (Cole &Moss, 1993). Bias in predictive validity (alsoreferred to as “differential prediction”) is adifference in the quality of inferences whenmaking a judgment of individuals from onegroup rather than another (Helms, 2006). Thatis, it is a difference between two groups in thepredictive validity of a measure. Instrumentsused for universal screening are often charac-terized by high rates of under- or overidenti-fication. This is part of the effect that test usehas on students that has been shown to differ-entially affect different subgroups of students(Cleary, Humphreys, Kendrick, & Wesman,1975). It is also often implicated in the dispro-portionate representation of minority studentsin special education (Hosp & Reschly, 2003)and differential provision of services, in thatnot only might some individuals, but somegroups, might be overidentified yet under-served (Donovan & Cross, 2002). With theincreased emphasis on assessment and ac-countability as mandated through NCLB aswell as Race to the Top (2010) and the in-

creased alignment of the Individuals with Dis-abilities Education Act (2004) with the pro-posed revisions to the Elementary and Second-ary Education Act (NCLB, 2002), the influenceof assessment on students and the importanceof examining potential bias in predictive va-lidity arguably has never been higher.

Unfortunately, bias in predictive validityis something that is evaluated less frequentlythan it should (Betts et al., 2008). For example,The National Center on Response to Interven-tion conducts technical reviews of screening in-struments for reading and math. To date, thetechnical review committee has reviewed ninereading-related screening measures, and onlytwo (DIBELS ORF and STAR Reading [Renais-sance Learning, 2011]) have provided evidenceof predictive validity across disaggregationgroups (see http://www.rti4success.org).

There have been a few studies to exam-ine differential predictive validity (i.e., with aspan of �3 months between predictor andcriterion) of screening measures across disag-gregation groups. Wiley and Deno (2005)found differences between English learners(EL) and English fluent (ES) students on bothMaze and ORF tasks as compared to a statehigh-stakes test at Grades 3 and 5. Roehrig etal. (2007) found no differences for studentsreceiving free or reduced-price lunch, EL stu-dents, or African American and Hispanicthird-grade students in prediction of the statehigh-stakes test using ORF. Betts et al. (2008)found no difference for EL students, AfricanAmerican students, or Asian-American stu-dents, but some difference between White andLatino students when predicting second-gradereading outcomes from kindergarten screeningassessments. Last, Fien et al. (2008) foundthat 7 of 24 comparisons between EL and ESstudents demonstrated differential prediction,but that 5 of the 7 were from winter kinder-garten assessments, which was consistent withCatts et al.’s (2009) concern about floor ef-fects and suggesting that the measures maydifferentially affect different subgroups of stu-dents. When examined as a group, no clearpattern of differential prediction has been con-sistent across groups, grade levels, or samples.

School Psychology Review, 2011, Volume 40, No. 1

110

Purpose of the Study

Given the importance of screening inresponse to intervention (Hughes & Dexter,2007), the concerns detailed by Catts et al.(2009), and the need to demonstrate account-ability for disaggregated subgroups withinNCLB, the purpose of the current study was toexamine the possibility of bias in predictivevalidity (including differential prediction anddifferential floor effects) across the disaggre-gation categories included in NCLB. As such,the research questions guiding this study wereas follows:

1. How well do benchmark scores on theNWF and ORF measures of the DIBELSpredict grades 1–3 scores on a state cri-terion-referenced test when examinedacross the disaggregation categories ofNCLB?

2. How much does the accuracy of predic-tion of the NWF and ORF measures ofthe DIBELS on a state criterion-refer-enced test vary as a function of level ofperformance when examined across thedisaggregation categories of NCLB?

These research questions served as thebasis for the following hypotheses:

H1. Benchmark scores on the NWF andORF measures of the DIBELS willdifferentially predict scores on a statecriterion-referenced test when exam-ined across the disaggregation catego-ries of NCLB.

H2. Accuracy of prediction of the NWFand ORF measures of the DIBELS ona state criterion-referenced test willvary as a function of level of perfor-mance when examined across the dis-aggregation categories of NCLB.

Method

Participants

Participants were 3,805 students en-rolled in Grades 1–3 of Utah’s Reading Firstschools during the 2006–2007 school year.This sample included all the students in Utah’sReading First schools who had data on both

measures. The entire sample of studentswas 50.8% male, 71.8% eligible for free orreduced-price lunch, 25.3% EL, 9.4% studentswith disabilities, 45.8% White, 38.7% His-panic, 8.7% American Indian, 2.6% PacificIslander, 2.1% African American, and 1.1%Asian. See Table 1 for the demographic char-acteristics broken out by each subgroup usedin the analysis for each grade level. Analysesindicated no differences between the demo-graphic profile of the final sample and overallschool demographics.

Measures

As part of Utah’s Reading First, all chil-dren were required to be administered ascreening instrument at least three times peryear in order to predict which students werelikely to not reach proficiency on the state’scriterion-referenced test, which is used to re-port adequate yearly progress to the U.S. De-partment of Education as a condition ofNCLB. The reading coaches, reading coordi-nators, and administrators from the participat-ing schools chose to use the DIBELS as theirscreening measures, which was then imple-mented in all Reading First schools in Utah.For the purposes of this study, only NWF andORF were included because these are the onlyDIBELS measures administered in Grades1–3, which are grades in which an outcomemeasure is also administered.

NWF. This is a standardized, individu-ally administered measure of a student’s abil-ity to use letter–sound correspondence to de-code short consonant–vowel–consonant (CVC)and vowel–consonant (VC) nonsense words.Given a page of these words, the student mustverbally produce either the individual lettersounds or each nonsense word. The student’sscore is the number of correct letter–soundcorrespondences produced within 1 min. Re-liability for NWF with first-grade students hasbeen reported as .94 for test–retest (Harn,Stoolmiller, & Chard, 2008) and .83 (Mdn �.67 to .88 range) for 1-month alternate form(Good et al., 2004).


111

ORF. This is a standardized, individu-ally administered measure of the accuracy andrate of a student’s ability to orally read con-nected text. Given a grade-level passage ofpreviously unseen material, the student readsaloud for 1 min. The number of words readcorrectly in that minute is recorded. Threeseparate passages are administered with thestudent’s median words read correctly scoreserving as the student’s recorded score. Reli-

ability of ORF has been reported as .95 foralternate form (Good, Kaminski, Smith, &Bratten, 2001) and .96 for test–retest (Catts etal., 2009).

Utah State Criterion-ReferencedTests (UCRTs). The UCRTs are group-ad-ministered tests given to all students in Grades1–8 in the spring of each school year. Thequestions are in multiple-choice format with

Table 1Demographic Characteristics of the Participants in Each Subgroup at Each

Grade Level

Group n Male FRL EL SwD AA AI As W H PI O

Grade 1 (n � 1353)FRL 945 483 — 368 70 19 78 9 311 487 33 8non-FRL 408 213 — 35 30 7 16 2 311 59 10 2EL 403 203 368 — 24 7 30 6 9 337 10 4non-EL 950 494 577 — 77 19 64 5 613 210 33 6SwD 101 73 70 24 — 4 6 0 56 31 2 2non-SwD 1252 624 875 379 — 22 88 11 566 516 41 8AI 94 39 78 30 6 — — — — — — —W 622 324 311 9 56 — — — — — — —H 547 286 487 337 31 — — — — — — —



Note. FRL � students receiving free/reduced-price lunch; non-FRL � students not receiving free/reduced-price lunch;EL � English learners; non-EL � English-proficient students; SwD � students with disabilities; non-SwD � studentswithout IEPs; AA � African American; AI � American Indian; As � Asian; W � White; H � Hispanic; PI � PacificIslander; O � other race/ethnicity.


112

students recording their answers on a comput-erized Scantron sheet. The items are alignedwith the state core curriculum with cut scoresestablished by the Utah State Office of Edu-cation to determine the minimum score a stu-dent must receive to achieve the level of pro-ficiency in the state curriculum. Because NWFand ORF are designed to predict reading out-comes, the English/Language Arts componentwas used as the outcome in this study. Reli-ability of the UCRTs was reported as .92(Kuder-Richardson 20) and .93 (stratified al-pha) for internal consistency; criterion validitywas reported as .65 with the Grade 3 IowaTest of Basic Skills (Hoover et al., 2003; UtahState Office of Education, 2007). Analysesalso suggest that the UCRT meets standards asa nonbiased instrument (Utah State Office ofEducation, 2007).

Procedures

All measures were administered byclassroom teachers or reading coaches. Train-ing for administration and scoring of the DI-BELS measures was conducted by outside ex-pert consultants hired by the Utah ReadingFirst Director to conduct multiple two-daytrainings for educators across the state. Dis-trict-based coaches and coordinators were alsotrained in using the DIBELS administrationintegrity checklists to use while observingpractice administrations of the measures. Al-though data from those checklists are notavailable for analysis in this study, no individ-ual was allowed to administer the measureswithout demonstrating administration andscorning accuracy �95% to a trainer.

The Utah Reading First evaluation teamprovided a schedule for DIBELS administra-tion to all participating schools in order tohave consistency in administration timesacross schools. Two-week windows for ad-ministration were provided for fall (2– 4weeks after the first day of school, typicallyoccurring in early September), winter (2weeks in January equidistant from the begin-ning and end of the school year), and spring(2–4 weeks before the last day of school,typically occurring in early May). NWF was

administered at all three time points in Grade 1whereas ORF was administered for winter andspring in Grade 1 and all three time points inGrades 2 and 3. The UCRTs are administeredover a 2-week period typically occurring inApril or early May. The spring DIBELS win-dow was scheduled so as not to overlap withthe UCRT administration window to reducescheduling burden.

Data Analysis

To make comparisons, four sets of anal-yses were conducted. These aligned with thedisaggregation categories as required throughNCLB: economic disadvantage (operational-ized as receiving free/reduced-price lunch ornot), limited English proficiency (identifica-tion as an English learner or not), disabilitystatus (identification as having a disability ornot), and race/ethnicity (White, Hispanic, andAmerican Indian—these three groups withlarge enough samples for analysis). The sam-ple sizes for each analysis can be found inTable 2.

Two methods of analysis were used toaddress the research questions for this study.First, Receiver Operating Characteristic (ROC)curves were calculated for each group withineach disaggregation category, for each mea-sure, at each grade level. ROC curve analysisis a method of judging the diagnostic effi-ciency of a measure (Swets, 1996). The threeindexes included from the ROC curve analy-ses are sensitivity (SE; i.e., the proportion ofstudents correctly classified as nonproficienton both measures being compared), specificity(SP; i.e., the proportion of students correctlyclassified as proficient on both measures), andarea under the curve (AUC). AUC is a prob-ability ranging from 0.5 to 1.0 that providesthe probability of a predictor correctly classi-fying a pair of students from two differentcategories (e.g., proficient, nonproficient), andcan be used as an effect size statistic (Swets,1988). The SE, SP, and AUC were all com-pared between groups within disaggregationcategories using a two-proportions test(Sprinthall, 2003). Determination for signifi-cance was adjusted for multiple comparisons


113

with p � .001 being used given 33 compari-sons in each family of analyses (Drew et al.,2008).

The second method of analysis was theuse of quantile regression. Quantile regressionis similar to ordinary least-squares regressionin that it attempts to minimize the sum ofsquared residuals, but rather than calculating aline to represent the best-fit data, quantile re-gression can provide best-fit points by asym-metrically weighting the data above and belowthe point of interest (Koenker, 2005). Theseestimates can be plotted in a simple line graphto illustrate the change in correlation betweentwo variables across levels of the predictorvariable. By plotting the quantile regressionlines for multiple groups on a single graph, thedifferential effect of the relation between thetwo variables for different disaggregationgroups can be examined as well as the pres-ence of floor and ceiling effects.

Results

Descriptive Statistics

The means and standards deviations ofthe performance of each disaggregated groupare shown in Table 2. Although on average thetraditionally underrepresented groups (FRL,EL, SwD, Hispanic, and American Indian)appeared to perform below their comparisongroup (non-FRL, non-EL, non-SwD, andWhite, respectively), this was not a hypothesisof the current study and therefore not testedfor statistical significance. Also, all groupsshowed within-grade growth on both NWFand ORF, but again these gains were not com-pared to a norm or criterion standard to test fortheir significance. Within-grade growth cannotbe compared for the UCRTs as it is onlyadministered once per year. Cross-grade com-parisons are not made here because this sam-ple is cross-sectional, rather than longitudinal.

ROC Curve Analyses

To answer the first research question,“How well do benchmark scores on the NWFand ORF measures of the DIBELS predictGrade 1–3 scores on a state criterion-refer-

enced test when examined across the disaggre-gation categories of NCLB?,” a series of ROCcurves were calculated. The AUC index wasevaluated as good if it was �.80 (Metz, 1978);the SE index was evaluated as good if it was�.80 (Carran & Scott, 1992); the SP indexwas also evaluated as good if it was �.80. Inaddition, the AUC, SE, and SP indexes be-tween the groups were compared using a two-proportions test.

Economic disadvantage. Results ofthe ROC curve analyses for the economic dis-advantage disaggregation comparisons areshown in Table 3. Overall, the AUCs fell intothe desired range for screening measures. AllAUCs for both groups, except for the threeadministrations of the NWF measure for theFRL group, were �.80. For SE, ORF inGrades 2 and 3 exceeded the .80 criterion forboth the FRL and non-FRL groups with theexception of spring Grade 3 for the non-FRLgroup (.79). The only measurement for eithergroup to meet the criterion for Grade 1 waswinter ORF for the non-FRL group. The op-posite was true for SP; no measurements inGrades 2 or 3 met the criterion, but all exceptwinter NWF and winter ORF for the FRLgroup did in Grade 1. Using the conservativecriterion of p � .001, two measurements dem-onstrated differences in AUC: two in SE, andthree in SP. Most of these differences were inGrade 1, with two of the SP differences inORF for Grade 2. There were no significantdifferences in AUC or SE for Grade 2, andno significant differences for any index atGrade 3.

Limited English proficiency. Resultsof the ROC curve analysis for the limitedEnglish proficiency disaggregation compari-sons are shown in Table 4. Overall, the AUCsfell into the desired range for screening mea-sures for ORF but not for NWF. For SE, ORFin Grades 2 and 3 exceeded the .80 criterionfor both groups. In Grade 1, only the winterORF measurement for the EL group exceededthe criterion. The opposite was true for SP; nomeasurements in Grades 2 or 3 met the crite-rion, but all except winter NWF and winter


114

Table 2Means and Standard Deviations for Each Group in Each Grade

Measure Grade Group n

Fall Winter Spring

Mean SD Mean SD Mean SD

NWF 1 FRL 945 34.2 22.5 61.3 27.5 78.3 36.0non-FRL 408 40.4 26.2 69.7 32.8 86.1 36.8EL 403 30.4 20.5 55.4 25.6 74.0 34.5non-EL 950 38.2 24.6 67.0 30.1 82.4 35.4SwD 101 21.4 17.5 45.6 24.5 59.2 35.5non-SwD 1252 37.2 23.9 65.2 29.3 82.3 35.9W 622 39.0 25.8 68.1 31.0 82.9 35.7H 547 31.1 19.7 56.8 25.1 73.5 32.3AI 94 43.6 24.0 74.3 28.7 101.0 47.1

ORF 1 FRL 945 — — 27.9 25.9 49.0 30.3non-FRL 408 — — 40.6 34.0 65.1 35.7EL 403 — — 21.6 20.9 41.7 27.4non-EL 950 — — 35.6 31.0 58.8 33.7SwD 101 — — 16.6 18.0 33.5 27.6non-SwD 1252 — — 32.8 29.5 55.4 32.7W 622 — — 39.0 33.3 62.0 35.0H 547 — — 22.0 20.1 44.4 27.7AI 94 — — 38.2 28.0 53.3 29.4

2 FRL 886 43.4 28.9 66.0 34.9 79.4 37.1non-FRL 351 56.8 33.8 82.7 38.5 96.2 38.5EL 311 36.0 28.2 56.9 35.9 70.3 38.6non-EL 930 50.6 31.1 75.5 35.1 88.9 37.4SwD 106 22.4 20.6 39.2 32.1 50.7 34.4non-SwD 1135 49.3 30.7 73.6 35.6 87.2 37.1W 618 52.4 32.8 78.3 37.4 91.0 38.6H 473 40.0 27.4 61.7 34.2 75.7 36.4AI 66 44.8 27.8 62.6 31.7 77.5 33.2

3 FRL 766 69.5 34.2 82.4 37.2 99.1 37.9non-FRL 322 82.3 35.8 95.5 37.4 111.4 36.0EL 247 56.0 32.0 69.0 36.8 86.0 37.3non-EL 841 79.0 34.6 92.0 36.6 108.0 36.5SwD 141 42.6 32.5 51.5 37.7 66.9 40.9non-SwD 947 77.7 33.1 91.0 35.0 107.6 34.4W 502 81.6 35.5 94.4 37.6 110.3 37.2H 450 66.1 32.9 78.8 35.8 96.3 36.5AI 60 65.2 31.1 77.8 34.9 95.6 37.7

UCRTa 1 FRL 945 — — — — 159.7 11.2non-FRL 408 — — — — 167.3 13.6EL 403 — — — — 155.6 9.9non-EL 950 — — — — 164.5 12.5SwD 101 — — — — 154.4 11.6non-SwD 1252 — — — — 162.5 12.3W 622 — — — — 166.9 12.8H 547 — — — — 157.3 10.2AI 94 — — — — 158.6 10.2

(Table 2 continues)


115

ORF for the EL group did in Grade 1. Usingthe conservative criterion of p � .001, nocomparisons between the EL and non-ELgroups demonstrated significant differences.In Grade 1, the winter NWF and winter ORFcomparisons were significant for SP. WinterORF was also significant for SE. In Grade 2,the winter and spring ORF comparisons weresignificant for SP, but not for SE. In Grade 3,all SE and SP comparisons were significant.

Disability status. Results of the ROCcurve analysis for the disability status disag-gregation comparisons are shown in Table 5.Overall, the ORF AUCs fell into the desiredrange for screening measures, but the NWFAUCs did not. The one exception is Grade 1winter ORF AUC for the SwD group (.794).

For SE, ORF met the criterion at all measure-ment points for the SwD group, but only forGrades 2 and 3 (with the exception of winterORF at Grade 2, .794) for the non-SwD group.SP again had nearly the opposite pattern withthe non-SwD group measurements at Grade 1,except for winter ORF (.795) meeting the cri-terion. For the SwD group, only spring NWFmet the criterion. None of the AUC compari-sons met the criterion for significance; how-ever, almost all of the SE and SP comparisonsmet the significance criterion with the excep-tion of winter ORF (SE p � .033) and springNWF (SP p � .010).

Race/ethnicity. Results of the ROCcurve analysis for the race/ethnicity disaggre-gation comparisons are shown in Tables 6 and

Table 2 ContinuedMeans and Standard Deviations for Each Group in Each Grade

Measure Grade Group n

Fall Winter Spring

Mean SD Mean SD Mean SD

2 FRL 886 — — — — 160.1 10.6non-FRL 351 — — — — 166.2 10.6EL 311 — — — — 156.0 10.7non-EL 930 — — — — 163.8 10.5SwD 106 — — — — 154.5 9.4non-SwD 1135 — — — — 162.5 10.8W 618 — — — — 165.2 10.9H 473 — — — — 158.0 10.2AI 66 — — — — 159.3 8.2

3 FRL 766 — — — — 160.9 9.8non-FRL 322 — — — — 164.5 9.9EL 247 — — — — 156.7 9.9non-EL 841 — — — — 163.6 9.6SwD 141 — — — — 155.4 9.4non-SwD 947 — — — — 162.8 9.7W 502 — — — — 164.7 9.4H 450 — — — — 159.4 10.1AI 60 — — — — 158.8 7.2

Note. NWF � Nonsense Word Fluency; ORF � Oral Reading Fluency; UCRT � Utah Criterion-Referenced Test;FRL � students receiving free/reduced-price lunch; non-FRL � students not receiving free/reduced-price lunch; EL �English learners; non-EL � English proficient students; SwD � students with disabilities; non-SwD � students withoutIEPs; W � White; H � Hispanic; AI � American Indian. Mean for NWF is the mean number of correct letter soundsper minute. For ORF, the mean is the mean number of words read correctly in one minute.aThe UCRT score reported here is the scaled score. This scale is equated across grade levels so that a score of 160 alwaysindicates performance at the median for that grade.


116

Tab

le3

RO

CT

wo-

Pro

port

ion

sT

est

Res

ult

sfo

rth

eE

con

omic

Dis

adva

nta

geD

isag

greg

atio

nA

nal

ysis

Gra

deM

easu

re

FRL

non-

FRL

p

BM

SESP

AU

CPP

PN

PP�

SESP

AU

CPP

PN

PP�

SESP

AU

C

1F

NW

F24

.535

.853

.761

.764

.665

.384

.618

.875

.845

.648

.860

.501

.001

.290

.001

WN

WF

50.5

30.7

90.7

33.6

92.6

48.3

19.6

00.8

62.8

17.6

17.8

53.4

66.0

20.0

02.0

01S

NW

F50

.324

.917

.741

.775

.598

.245

.373

.936

.802

.683

.801

.360

.075

.230

.018

WO

RF

20.7

69.7

54.8

49.7

39.7

84.5

23.8

09.8

38.9

01.6

50.9

22.6

01.0

89.0

01.0

11S

OR

F40

.732

.851

.867

.816

.778

.586

.627

.899

.884

.697

.867

.543

.000

.018

.390

2F

OR

F44

.845

.671

.862

.666

.868

.517

.812

.737

.892

.496

.925

.451

.134

.014

.121

WO

RF

68.8

21.7

21.8

74.6

92.8

44.5

34.8

12.8

20.8

88.5

83.9

24.5

43.6

89.0

00.4

53S

OR

F90

.870

.663

.866

.660

.865

.507

.859

.771

.889

.545

.945

.526

.575

.000

.230

3F

OR

F77

.874

.595

.843

.591

.875

.439

.815

.693

.822

.468

.918

.406

.019

.003

.384

WO

RF

92.8

65.5

77.8

41.5

82.8

75.4

26.8

27.6

56.8

40.4

43.9

19.3

73.1

21.0

17.9

68S

OR

F11

0.8

65.6

13.8

46.6

06.8

81.4

64.7

90.6

68.8

27.4

41.9

04.3

61.0

04.0

89.4

30

Not

e.R

OC

�re

ceiv

erop

erat

ing

char

acte

rist

ic;F

RL

�st

uden

tsre

ceiv

ing

free

/red

uced

-pri

celu

nch;

BM

�be

nchm

ark

scor

e;SE

�se

nsiti

vity

;SP

�sp

ecifi

city

;AU

C�

area

unde

rth

ecu

rve;

PPP

�po

sitiv

epr

edic

tive

pow

er;

NPP

�ne

gativ

epr

edic

tive

pow

er;

��

kapp

a;F

�fa

ll;W

�w

inte

r;S

�sp

ring

;N

WF

�N

onse

nse

Wor

dFl

uenc

y;O

RF

�O

ral

Rea

ding

Flue

ncy.


117

Tab

le4

RO

CT

wo-

Pro

port

ion

sT

est

Res

ult

sfo

rth

eL

imit

edE

ngl

ish

Pro

fici

ency

Dis

aggr

egat

ion

An

alys

is

Gra

deM

easu

re

EL

non-

EL

p

BM

SESP

AU

CPP

PN

PP�

SESP

AU

CPP

PN

PP�

SESP

AU

C

1F

NW

F24

.551

.865

.761

.839

.545

.353

.538

.868

.787

.668

.791

.427

.660

.881

.289

WN

WF

50.5

35.7

47.7

11.7

66.5

09.2

60.5

51.8

31.7

62.6

18.7

89.3

93.5

89.0

00.0

46S

NW

F50

.314

.899

.709

.828

.458

.182

.347

.929

.783

.708

.742

.317

.246

.061

.003

WO

RF

20.8

20.6

90.8

48.8

04.7

12.5

13.7

45.8

09.8

65.6

59.8

65.5

37.0

01.0

00.4

06S

OR

F40

.743

.823

.859

.867

.674

.544

.688

.880

.876

.740

.851

.578

.036

.005

.390

2F

OR

F44

.847

.620

.848

.778

.721

.479

.864

.707

.892

.560

.923

.495

.465

.003

.036

WO

RF

68.8

16.6

53.8

53.7

87.6

93.4

74.8

21.7

75.8

94.6

12.9

09.5

43.8

41.0

00.0

49S

OR

F90

.837

.595

.839

.764

.699

.444

.882

.719

.893

.576

.934

.522

.059

.000

.010

3F

OR

F77

.906

.468

.846

.683

.794

.386

.829

.668

.827

.506

.904

.418

.001

.000

.484

WO

RF

92.9

28.4

68.8

65.6

88.8

33.4

10.8

24.6

42.8

33.4

86.8

99.3

86.0

00.0

00.2

34S

OR

F11

0.9

06.5

41.8

59.7

14.8

17.4

60.8

24.6

60.8

34.5

00.9

01.4

07.0

00.0

00.3

52

Not

e.R

OC

�re

ceiv

erop

erat

ing

char

acte

rist

ic;

EL

�st

uden

tsw

ithlim

ited

Eng

lish

profi

cien

cy;

BM

�be

nchm

ark

scor

e;SE

�se

nsiti

vity

;SP

�sp

ecifi

city

;A

UC

�ar

eaun

der

the

curv

e;PP

P�

posi

tive

pred

ictiv

epo

wer

;N

PP�

nega

tive

pred

ictiv

epo

wer

;�

�ka

ppa;

F�

fall;

W�

win

ter;

S�

spri

ng;

NW

F�

Non

sens

eW

ord

Flue

ncy;

OR

F�

Ora

lR

eadi

ngFl

uenc

y.


118

Tab

le5

RO

CT

wo-

Pro

port

ion

sT

est

Res

ult

sfo

rth

eD

isab

ility

Stat

us

Dis

aggr

egat

ion

An

alys

is

Gra

deM

easu

re

SwD

non-

SwD

p

BM

SESP

AU

CPP

PN

PP�

SESP

AU

CPP

PN

PP�

SESP

AU

C

1F

NW

F24

.723

.714

.755

.825

.581

.415

.520

.868

.779

.720

.734

.408

.000

.000

.575

WN

WF

50.7

08.6

86.7

41.8

07.5

58.3

74.5

22.8

22.7

53.6

55.7

25.3

55.0

00.0

00.7

87S

NW

F50

.523

.857

.760

.872

.492

.325

.308

.927

.753

.731

.672

.260

.000

.010

.873

WO

RF

20.8

46.5

71.7

94.7

86.6

67.4

32.7

68.7

95.8

69.7

10.8

41.5

56.0

33.0

00.0

29S

OR

F40

.831

.714

.841

.844

.694

.541

.696

.875

.875

.785

.815

.584

.000

.000

.317

2F

OR

F44

.947

.514

.859

.810

.818

.523

.822

.702

.867

.604

.891

.497

.000

.000

.810

WO

RF

68.9

47.5

71.8

68.8

19.8

26.5

50.7

94.7

65.8

76.6

47.8

74.5

33.0

00.0

00.8

10S

OR

F90

.933

.457

.841

.788

.762

.446

.845

.711

.870

.614

.898

.516

.000

.000

.384

3F

OR

F77

.968

.370

.822

.729

.870

.397

.831

.647

.827

.519

.891

.413

.000

.000

.881

WO

RF

92.9

68.3

52.8

25.7

23.8

64.3

77.8

25.6

22.8

31.5

05.8

92.3

93.0

00.0

00.8

57S

OR

F11

0.9

46.3

52.8

04.7

18.7

92.3

48.8

22.6

53.8

35.5

26.8

94.4

24.0

00.0

00.3

42

Not

e.R

OC

�re

ceiv

erop

erat

ing

char

acte

rist

ic;

SwD

�st

uden

tsw

ithdi

sabi

litie

s;B

M�

benc

hmar

ksc

ore;

SE�

sens

itivi

ty;

SP�

spec

ifici

ty;

AU

C�

area

unde

rth

ecu

rve;

PPP

�po

sitiv

epr

edic

tive

pow

er;

NPP

�ne

gativ

epr

edic

tive

pow

er;

��

kapp

a;F

�fa

ll;W

�w

inte

r;S

�sp

ring

;N

WF

�N

onse

nse

Wor

dFl

uenc

y;O

RF

�O

ral

Rea

ding

Flue

ncy.


119

Tab

le6

RO

CT

wo-

Pro

port

ion

sT

est

Res

ult

sfo

rth

eR

ace

Eth

nic

ity

Dis

aggr

egat

ion

An

alys

is:

His

pan

icto

Wh

ite

Gra

deM

easu

re

His

pani

cW

hite

p

BM

SESP

AU

CPP

PN

PP�

SESP

AU

CPP

PN

PP�

SESP

AU

C

1F

NW

F24

.547

.831

.761

.795

.604

.366

.640

.869

.845

.640

.871

.511

.001

.069

.000

WN

WF

50.5

74.7

99.7

39.7

74.6

09.3

63.6

22.8

17.8

08.5

48.8

57.4

20.0

97.4

35.0

05S

NW

F50

.329

.896

.743

.790

.526

.212

.433

.930

.828

.689

.820

.412

.000

.038

.000

WO

RF

20.8

29.7

03.8

62.7

69.7

73.5

35.8

23.8

15.8

97.6

16.9

28.5

77.7

87.0

00.0

66S

OR

F40

.738

.843

.859

.849

.728

.573

.756

.878

.911

.693

.910

.618

.478

.084

.005

2F

OR

F44

.837

.659

.844

.723

.800

.506

.894

.697

.909

.509

.949

.475

.007

.180

.001

WO

RF

68.8

01.6

85.8

47.7

29.7

67.4

90.8

56.7

80.9

07.5

76.9

39.5

48.0

18.0

00.0

02S

OR

F90

.829

.655

.841

.719

.791

.493

.925

.712

.911

.529

.964

.512

.000

.042

.000

3F

OR

F77

.876

.594

.846

.635

.855

.453

.837

.662

.830

.443

.883

.302

.085

.030

.503

WO

RF

92.9

00.5

54.8

59.6

20.8

73.4

36.8

37.6

49.8

38.4

34.9

25.3

69.0

04.0

03.3

68S

OR

F11

0.8

86.6

02.8

59.6

43.8

67.4

71.8

29.6

73.8

42.4

49.9

24.3

89.0

12.0

23.4

65

Not

e.R

OC

�re

ceiv

erop

erat

ing

char

acte

rist

ic;

BM

�be

nchm

ark

scor

e;SE

�se

nsiti

vity

;SP

�sp

ecifi

city

;A

UC

�ar

eaun

der

the

curv

e;PP

P�

posi

tive

pred

ictiv

epo

wer

;N

PP�

nega

tive

pred

ictiv

epo

wer

;�

�ka

ppa;

F�

fall;

W�

win

ter;

S�

spri

ng;

NW

F�

Non

sens

eW

ord

Flue

ncy;

OR

F�

Ora

lR

eadi

ngFl

uenc

y.


120

7. Three different patterns of AUCs emerged:for White students, the AUCs met the criterionfor all measurements; for Hispanic students,the AUCs met the criterion for all ORF mea-surements, but not NWF; for American Indianstudents, the AUCs met the criterion for ORFin Grades 1 and 2 only, but not for NWF. Forboth White and Hispanic students, SE met thecriterion in Grades 2 and 3 as well as winterORF in Grade 1. For the American Indianstudents, SE only met the criterion for springORF in Grade 2 and fall ORF in Grade 3. SPmet the criterion for all three groups at allmeasurements in Grade 1 except for winterORF for the Hispanic students. No measure-ments in Grades 2 or 3 met the criterion forSP. Because the comparisons can only be con-ducted between two groups, Hispanic andAmerican Indian students were compared toWhite students separately.

In comparing Hispanic students toWhite students, significant differences in AUCwere found for fall and spring NWF inGrade 1 and fall and spring ORF in Grade 2. Asimilar pattern was noted for SE with theexception of fall ORF in Grade 2. SP wassignificantly different between the groups forwinter ORF in Grades 1 and 2. There were nosignificant comparisons in Grade 3.

In comparing American Indian studentsto White students, only fall NWF met thecriterion for significance. For SE, all compar-isons in Grade 1 met the criterion for signifi-cance, as did fall ORF in Grade 2. For SP, allthree measurements in Grade 3 met the crite-rion for significance.

Summary of ROC results. Across thefour disaggregation groups, there were fewstatistically significant differences in AUC be-tween the groups being compared. All of thesedifferences were with the NWF measure withthe exception of ORF at grade 2 comparingHispanic and White students. When examin-ing the SE and SP indexes, there tended to bemore differences in SE in Grade 1, particularlywith the NWF measure; and more differencesin SP in Grades 2 and 3 (most clearly illus-trated in the American Indian to White com-parisons). A final pattern was that for some

groups, at certain grade levels (EL Grade 3,SwD Grades 2 and 3) there were significantdifferences in SE and SP, but not in AUC.

Quantile Regression Analyses

To answer the second research question,“How much does the accuracy of prediction ofthe NWF and ORF measures of the DIBELSon a state criterion-referenced test vary as afunction of level of performance when exam-ined across the disaggregation categories ofNCLB?,” a series of quantile regression mod-els were developed and the graphs of the re-sultant correlations plotted and compared. In-terpretation of the graphs was conducted byvisual inspection comparing the regressionplots between the groups. Floor or ceiling ef-fects are demonstrated when a line is not hor-izontal (horizontal lines indicating the regres-sion coefficients are similar across all points inthe performance range). When two groupshave similar plots, this indicates that the flooror ceiling effect (or lack thereof) affects bothgroups similarly. When the plots are different,one group is affected by the floor/ceiling effectmore than the other.

Economic disadvantage. As can beseen in Figure 1, there are differences in theplots for the fall and winter NWF and ORF atGrades 1 and 2. The spring measurements inGrades 1 and 2 as well as all three measure-ments in Grade 3 have similar plots for bothgroups.

Limited English proficiency. As canbe seen in Figure 2, there are differences in theplots for spring NWF and winter ORF inGrade 1 as well as fall and winter ORF ingrade 2. All three measurements in Grade 3have similar plots for both groups.

Disability status. As can be seen inFigure 3, there are differences in the plots fornearly every measurement point except springNWF in Grade 1 and fall ORF in Grade 3.

Race/Ethnicity. As can be seen in Fig-ure 4, there are differences among the plots forfall and winter NWF and winter ORF inGrade 1 as well as fall ORF in Grade 2. There


121

Tab

le7

RO

CT

wo-

Pro

port

ion

sT

est

Res

ult

sfo

rth

eR

ace

Eth

nic

ity

Dis

aggr

egat

ion

An

alys

is:

Am

eric

anIn

dian

toW

hit

e

Gra

deM

easu

re

Am

eric

anIn

dian

Whi

tep

BM

SESP

AU

CPP

PN

PP�

SESP

AU

CPP

PN

PP�

SESP

AU

C

1F

NW

F24

.283

.894

.723

.800

.432

.137

.640

.869

.845

.640

.871

.511

.000

.478

.001

WN

WF

50.2

33.9

15.7

35.8

13.4

23.1

15.6

22.8

17.8

08.5

48.8

57.4

20.0

00.0

17.0

73S

NW

F50

.133

.979

.715

.875

.407

.074

.433

.930

.828

.689

.820

.412

.000

.059

.004

WO

RF

20.4

50.8

94.8

22.8

97.5

08.3

17.8

23.8

15.8

97.6

16.9

28.5

77.0

00.0

55.0

20S

OR

F40

.567

.915

.837

.917

.569

.435

.756

.878

.911

.693

.910

.618

.001

.280

.016

2F

OR

F44

.714

.795

.862

.829

.774

.604

.894

.697

.909

.509

.949

.475

.000

.049

.124

WO

RF

68.7

94.7

73.8

73.8

29.7

74.6

04.8

56.7

80.9

07.5

76.9

39.5

48.1

47.8

73.2

67S

OR

F90

.825

.727

.859

.789

.786

.570

.925

.712

.911

.529

.964

.512

.013

.757

.087

3F

OR

F77

.855

.492

.765

.676

.696

.357

.837

.662

.830

.443

.883

.302

.617

.000

.093

WO

RF

92.7

64.4

75.7

43.6

86.6

80.3

60.8

37.6

49.8

38.4

34.9

25.3

69.0

95.0

00.0

12S

OR

F11

0.7

45.4

92.7

33.6

76.6

54.3

27.8

29.6

73.8

42.4

49.9

24.3

89.0

63.0

00.0

04

Not

e.R

OC

�re

ceiv

erop

erat

ing

char

acte

rist

ic;

BM

�be

nchm

ark

scor

e;SE

�se

nsiti

vity

;SP

�sp

ecifi

city

;A

UC

�ar

eaun

der

the

curv

e;PP

P�

posi

tive

pred

ictiv

epo

wer

;N

PP�

nega

tive

pred

ictiv

epo

wer

;�

�ka

ppa;

F�

fall;

W�

win

ter;

S�

spri

ng;

NW

F�

Non

sens

eW

ord

Flue

ncy;

OR

F�

Ora

lR

eadi

ngFl

uenc

y.


122

are also differences for the Hispanic group inspring ORF for Grade 2 (on the higher end ofthe distribution) and the American Indiangroup in fall ORF for Grade 3 (again, partic-ularly on the high end). All other plots aresimilar.

Summary of quantile regression re-sults. In general, there was less bias in predic-tive validity in Grade 3 than in Grades 1 and 2 as

well as less of an influence of a floor effect (i.e.,slope in the regression line). The patterns inGrade 1 appeared similar for NWF and ORFexcept for the EL comparisons. However, itshould be noted that although there are manydifferences in performance of different groupsacross measures, there are also many similari-ties—indicating that patterns of potential bias arenot extreme or consistent.

Figu

re1.

Qu

anti

lere

gres

sion

plot

sfo

rgr

ades

1–3

DIB

EL

Sm

easu

res

for

the

econ

omic

disa

dvan

-ta

gedi

sagg

rega

tion

anal

ysis

.


123

Discussion

In our nation’s push to improve educa-tional outcomes for all students, examinationof bias in predictive validity of educationalmeasures is vital. Consistency in our decisionmaking is important in order to ensure consis-tency in service delivery and outcomes (Bar-nett et al., 2007) and to prevent over- or un-

deridentification of a subgroup of students.Unfortunately, studies of bias in predictivevalidity of screening measures are relativelyuncommon (Betts et al., 2008). In this study,we examined universal screening data for biasin predictive validity across the disaggregationcategories mandated by NCLB. We found thatmeasures with good overall predictive validity

Figu

re2.

Qu

anti

lere

gres

sion

plot

sfo

rG

rade

1–3

DIB

EL

Sm

easu

res

for

the

limit

edE

ngl

ish

profi

cien

cydi

sagg

rega

tion

anal

ysis

.


124

(NWF, ORF) may not demonstrate consistentlevels of predictive validity when focusing ondifferent subgroups. Our results also suggestthat this differential prediction varies acrossthe subgroup analyses. Findings support priorresearch in which the patterns of predictivevalidity (or bias) have varied across studies.

There are many potential explanationsfor this variation in pattern across studies. In

addition to the typical differences across re-search studies (different settings, participants,and so on), studies vary in use of criterionmeasures, inclusion and exclusion of vari-ables, and instruction and intervention. Be-cause studies of prediction bias involve relat-ing a predictor measure to an outcome mea-sure and use of a cut score, each of thesecomponents may contribute to differential pre-

Figu

re3.

Qu

anti

lere

gres

sion

plot

sfo

rG

rade

1–3

DIB

EL

Sm

easu

res

for

the

disa

bilit

yst

atu

sdi

sagg

rega

tion

anal

ysis

.


125

diction patterns. The bias may be conceived asresiding with the predictor measure (i.e., itbeing the dominant factor influencing the dif-ferential prediction because of variation in per-formance or functioning), the criterion mea-sure, or the cut scores for one measure or theother (Flaugher, 1978). Results of predictivevalidity bias studies can indicate the presenceof differential prediction, but generally not the

location of that bias. A second source of vari-ation is differential inclusion of variables. If avariable is correlated with both the predictorand the criterion measures, the coefficients(and therefore the decisions) may be biasedbecause of the omission of the variable ratherthan the performance of the measures (John-son, Carter, Davison, & Oliver, 2001). Thiscould be the influencing factor involved in the

Figu

re4.

Qu

anti

lere

gres

sion

plot

sfo

rG

rade

1–3

DIB

EL

Sm

easu

res

for

the

race

/eth

nic

ity

disa

ggre

gati

onan

alys

is.


126

participants in different studies receiving dif-ferent instruction and intervention. Becausethe predictive validity studies involve a 3- to6- month lag between administration of thepredictor and criterion measures, participantsreceived instruction and most likely differen-tial intervention based on individual needs.This instruction may vary across studies andacross classrooms or schools within studiesadding another source of variance. These con-siderations highlight the importance of exam-ining a phenomenon across studies to examinethe pattern in greater detail.

In relation to the findings of Catts et al.(2009), we found a similar pattern of greaterfloor effects in Grade 1 (for both NWF andORF) than in Grade 2, with little to no flooreffect in Grade 3. As students progress ingrade, their performance distribution is highenough to not have the restriction of rangeindicative of a floor effect. As far as differen-tial floor effects across the disaggregation cat-egories, there was not a clear pattern. Allgroups demonstrated floor effects at the falland winter screenings for NWF and the winterORF screening in Grade 1. All but the race/ethnicity comparisons also demonstrated flooreffects in fall of Grade 2. No subgroup dem-onstrated floor effects in the spring of any year(i.e., potential concurrent predictive bias). Nogroup demonstrated floor effects at all mea-surement points across all grades. The patterndisplayed in this study appears to be that thefirst measurement period within a year(Grades 1 and 2) is the most likely to exhibitdifferential prediction. This makes sense inthat group performance at the beginning of theyear is likely to be lower than at other points,making the impact of floor effects more likely.If two groups perform differently, one wouldbe more susceptible to floor effects than theother (i.e., the group with the lower overallperformance). In addition, if the groups cameinto their education with different prior knowl-edge and experience, or were receiving differ-ential curriculum or instruction (or differen-tially effective curriculum or instruction), dif-ferent levels or patterns of performance couldbe expected (Donovan & Cross, 2002). Thepattern of differential prediction in this study

potentially caused by floor effects in the lowergrades was not duplicated in the ROCanalyses.

From the ROC analyses, although AUCis a valid effect size statistic for use in com-parisons (Swets, 1988), the present resultssuggest that it may not be best to use it inisolation to judge bias in predictive validity.Differences in both SE and SP between twogroups, with decisions for one group havinghigher SE and decisions for the other havinghigher SP, can actually offset each other in thedetermination of AUC. The clearest examplesof this phenomenon are in Grade 3 for thelimited English proficiency comparisons (Ta-ble 4) and all grades for the disability statuscomparison (Table 5). For these comparisons,AUC was similar between the groups, butthere were significant differences between SEand SP. An implication of this pattern is thatthere are different mistakes in terms of deci-sion making being made for different groupsof students. From our results for the limitedEnglish proficiency comparison, ORF atGrade 3 demonstrated greater sensitivity forEL (i.e., the measures were better at identify-ing which individuals in the EL group wouldnot meet proficiency on the outcome than forthe non-EL group). Conversely, both measuresdemonstrated better specificity for non- EL(i.e., the measures were better at identifyingwhich individuals in the non-EL group wouldmeet or exceed the criterion for proficiency onthe outcome measure than for the EL group).If using ORF at Grade 3 to screen studentsusing a direct route approach, wherein thosepredicted to not meet the proficiency criterionon the outcome measure are automaticallyplaced in supplemental instruction, or Tier 2(Jenkins et al., 2007), one would make morefalse-positive errors for the EL group. Thismeans that more EL students would be placedinto Tier 2 intervention programs that they donot necessarily require than their non-ELpeers. The reverse would also occur: morenon-EL students would not receive Tier 2 ser-vices that they needed than their EL peers (i.e.,a higher false-negative rate for non-EL).

One way to examine the presence ofcounteracting SE and SP is to use multiple


127

indexes of classification accuracy. If one indexindicates a difference (e.g., SE), yet anotherdoes not (e.g., SP), there is a different patternof predictive validity than if both indexesdemonstrate differences. In the case of a non-significant AUC with significant SE and SP,this would also provide a check of whetherthere are two phenomena counteracting eachother (indicating that the phenomenon may bea result of differential base rates on the crite-rion measure). Another strategy for examiningand combating differential predictive validityis to identify different cut scores for differentgroups and/or different outcome measures(Roehrig et al., 2007). By systematically iden-tifying different cut scores that maintain thesame levels of sensitivity and specificity, sim-ilarity in proportions of false positives andfalse negatives across different groups couldbe ensured. However, one potential cautionwould be that generalizability of performancecan be lost for the decisions made for individ-ual students. If each school or district usesdifferent criteria to determine the direct routeto supplemental or intensive services, a stu-dent could move from one building where hewas predicted to be proficient on the outcomemeasure (and therefore not receiving supple-mental services) to another where he was notpredicted to be proficient and therefore in needof additional instruction. Although both deci-sions may be correct in determining the stu-dent’s needs, it provides an additional level ofcoordination and judgment for the school towhich the student moves.

In addition to the need to examine mul-tiple indexes in a determination of differentialpredictive validity, there are other implica-tions from our results. First is that individualstudents are included in multiple groups (e.g.,a Latino student receiving free lunch, or astudent with a disability with limited English).As such, if different cut scores are developedfor use with different groups, which onewould be used for making a decision aboutservice delivery for an individual student? Be-cause the disaggregation categories are mutu-ally exclusive (e.g., a student cannot be botheconomically disadvantaged and not economi-cally disadvantaged) and comprehensive (e.g.,

all students are either economically disadvan-taged or not), every student can be classifiedalong the dimensions of every disaggregationcategory. This would mean up to four separatecut scores (five if you add sex) for every studentas identifying which one was the most accuratewould be a difficult web to untangle.

A second implication is that the absenceof prediction bias does not automatically equalfairness. As stated previously, predictive va-lidity has to do with prediction of outcomes;that bias in predictive validity occurs when atest differentially predicts that outcome forone group as compared to another. By con-trast, fairness can be conceived as differencesin the mean test scores that an individual isbeing compared to (or used to develop thecriterion) that are not directly related to thefocus of the measure (i.e., construct-irrelevantvariance). Although approaches to quantita-tively address lack of fairness in assessmentdo exist (see Helms, 2006), they are notwidely adopted or used.

Limitations

The above findings should be inter-preted in light of some potential limitations.First, the sizes of groups being compared werenot equivalent, which could lead to differencesin the consistency of scores and error for thegroups (Tabachnick & Fidell, 2007). Second,some of the group sizes were relatively small(n � 100 students). Although each group in-cluded was large enough to run the analyses,larger samples would provide more stable es-timates—particularly for the quantile regres-sion analyses in which more stable estimateswould provide smoother plot lines (Koenker,2005). A third limitation is the lack of anAfrican American subgroup in the race/ethnic-ity analyses. This is a potential limitation be-cause African American students are, andhave traditionally been, one of the larger ra-cial/ethnic groups in the United States. Fourth,the data for this study came exclusively fromReading First schools. As such, the minority,English learner, and economically disadvan-taged proportions of students are higher thanthose of the state as a whole. Similar to the


128

results from the Catts et al. (2009) study, theextent of this effect in unclear. Last, as withmost any study examining prediction withscreening measures, there is an interveningagent present as the results from the screeningmeasures were intended to be used to makedecisions about placement and interventionprovision. This can affect the classificationaccuracy estimates despite the fact that it isprecisely the purpose for which the measuresare designed (Hosp, Dole, & Hosp, 2006).

Implications for the Practice of SchoolPsychology

Despite the above-mentioned limita-tions, there are messages that school psychol-ogists can take from the current findings. First,use of a single measure is not prudent forscreening decisions. Given these preliminaryfindings of bias in predictive validity, usingother pieces of data to validate the decisionfrom a screening measure should reduce thepotential for false positives or negatives. Tri-angulation of data either from other screeningmeasures that address the same skill in a dif-ferent way or inclusion of progress monitoringdata after the screening provides additionalpieces of information with which to make adecision. Second, using a team to make deci-sions can be useful for screening decisions aswell as eligibility decisions as mandated byIndividuals with Disabilities Education Act(2004). Similar to use of multiple pieces ofdata or a structured decision-making process,it introduces an added layer of accountabilityto make sure that there is agreement in thedecisions.

References

Barnett, D. W., Hawkins, R., Prasse, D., Graden, J., Nan-tais, M., & Pan, W. (2007). Decision-making validityin response to intervention. In S. R. Jimerson, M.Burns, & A. VanDerHeyden (Eds.), Handbook of re-sponse to intervention: The science and practice ofassessment and intervention (pp. 106–116). NewYork: Springer.

Batsche, G., Elliott, J., Graden, J. L., Grimes, J.,Kovaleski, J. F., Prasse, D., et al. (2005). Response tointervention: Policy considerations and implementa-tion. Alexandria, VA: National Association of StateDirectors of Special Education.

Betts, J., Reschly, A., Pickart, M., Heistad, D., Sheran, C.,& Marston, D. (2008). An examination of predictive

bias for second grade reading outcomes from measuresof early literacy skills in kindergarten with respect toEnglish-Language learners and ethnic subgroups.School Psychology Quarterly, 23, 553–570.

Buck, J., & Torgesen, J. (2002). The relationship betweenperformance on a measure of oral reading fluency andperformance on the Florida Comprehensive Assess-ment Test (FCRR Technical Report No. 1). Tallahas-see: Florida Center for Reading Research.

Carran, D. T., & Scott, K. G. (1992). Risk assessment inpreschool children: Research implications for the earlydetection of educational handicaps. Topics in EarlyChildhood Special Education, 12, 196–211.

Catts, H. W., Fey, M. E., Zhang, X., & Tomblin, J. B.(2001). Estimating the risk of future reading difficul-ties in kindergarten children: A research-based modeland its clinical implementation. Language, Speech,and Hearing Services in Schools, 32, 38–50.

Catts, H. W., Petscher, Y., Schatschneider, C., Bridges,M. S., & Mendoza, K. (2009). Floor effects associatedwith universal screening and their impact on the earlyidentification of reading difficulties. Journal of Learn-ing Disabilities, 42, 162–176.

Cleary, T., Humphreys, L. G., Kendrick, S. A., & Wes-man, A. (1975). Educational uses of tests with disad-vantaged students. American Psychologist, 30, 15–41.

Cole, N., & Moss, P. (1993). Bias in test use. In R. L. Linn(Ed.), Educational measurement (3rd ed.; pp. 201–220). Phoenix, AZ: The Oryx Press.

Donovan, M. S., & Cross, C. T. (Eds.). (2002). Minoritystudents in special and gifted education. WashingtonDC: National Academy Press.

Drew, C. J., Hardman, M. L., & Hosp, J. L. (2008).Designing and conducting research in education. NewYork: Sage.

Fien, H., Baker, S. K., Smolkowski, K., Mercier-Smith,J. L., Kame’enui, E. J., & Beck, C. T. (2008). Usingnonsense word fluency to predict reading proficiencyin kindergarten through second grade for Englishlearners and native English speakers. School Psychol-ogy Review, 37, 391–408.

Flaugher, R. L. (1978). The many definitions of test bias.American Psychologist, 33, 671–679.

Foorman, B. F., Francis, D. J., Fletcher, J. M., Schat-schneider, C., & Mehta, P. (1998). The role of instruc-tion in learning to read: Preventing reading failure inat-risk children. Journal of Educational Psychology,90, 37–55.

Good, R. H., Kaminski, R. A., Shinn, M., Bratten, J.,Shinn, M., Laimon, D., et al. (2004). Technical ade-quacy of DIBELS: Results of the Early ChildhoodResearch Institute on measuring growth and develop-ment (Technical Report No. 7). Eugene: University ofOregon.

Good, R. H., Kaminski, R. A., Smith, M. R., & Bratten, J.(2001). Technical adequacy and second grade DIBELSOral Reading Fluency (DORF) passages (TechnicalReport No. 8). Eugene: University of Oregon.

Haladyna, T. M. (2006). Roles and importance of validitystudies in test development. In S. M. Downing & T. M.Haladyna (eds.), Handbook of test development (pp.739–758). Hillsdale, NJ: Lawrence Erlbaum Associ-ates.

Harn, B. A., Stoolmiller, M., & Chard, D. J. (2008).Measuring the dimensions of alphabetic principle onthe reading development of first graders: The role of


129

automaticity and unitization. Journal of Learning Dis-abilities, 41, 143–157.

Helms, J. E. (2006). Fairness is not validity or culturalbias in racial-group assessment: A quantitative per-spective. American Psychologist, 61(8), 859–870.

Hintze, J. M., & Silberglitt, B. (2005). A longitudinalexamination of the diagnostic accuracy and predictivevalidity of R-CBM and high-stakes testing. SchoolPsychology Review, 34, 372–386.

Hoover, H. D., Dunbar, D. A., Frisbie, D. A., Oberley,K. R., Bray, G. B., Naylor, R. J. (2003). The Iowa Testsof Basic Skills. Rolling Meadows, IL: The RiversidePublishing Company.

Hosp, J. L., & Ardoin, S. (2008). Assessment for instruc-tional planning. Assessment for Effective Intervention,33, 69–77.

Hosp, J. L., Dole, J. A., & Hosp, M. K. (2006, July).DIBELS as a predictor of proficiency on high stakesoutcome assessments for at-risk readers. Paper pre-sented at the annual meeting of the Society for theScientific Study of Reading, Vancouver, BC.

Hosp, J. L., & Reschly, D. J. (2003). Referral rates forintervention or assessment: A meta-analysis of racialdifferences. The Journal of Special Education, 37,67–80.

Hughes, C., & Dexter, D. (2007). Universal screeningwithin a response-to-intervention model (report brieffor RTI Action Network). New York: National Centerfor Learning Disabilities.

Ikeda, M. J., Neessen, E., & Witt, J. C. (2008). Bestpractices in universal screening. In A. Thomas & J.Grimes (Eds.), Best practices in school psychology(5th ed., Vol. 2, pp. 103–114). Bethesda, MD: NationalAssociation of School Psychologists.

Individuals with Disabilities Education Improvement Act,20 U.S.C. § 1400 (2004).

Jenkins, J. R., Hudson, R. F., & Johnson, E. S. (2007).Screening for at-risk readers in a response to interven-tion framework. School Psychology Review, 36, 582–600.

Johnson, J. W., Carter, G. W., Davison, H. K., & Oliver,D. H. (2001). A synthetic validity approach to testingdifferential prediction hypotheses. Journal of AppliedPsychology, 86, 774–780.

Koenker, R. (2005). Quantile regression. New York:Cambridge University Press.

Metz, C. E. (1978). Basic principles of ROC analysis.Seminars in Nuclear Medicine, 8, 283–298.

No Child Left Behind Act, 20 U.S.C. § 6301 (2002).O’Connor, R. E., & Jenkins, J. R. (1999). Prediction of

reading disabilities in kindergarten and first grade.Scientific Studies of Reading, 3, 159–197.

Race to the Top, 26 U.S.C. § 1 (2009).Rampsey, B. D., Dion, G. S., & Donahue, P. L. (2009).

NAEP 2008 trends in academic progress (NCES

2009–479). Washington, DC: National Center for Ed-ucation Statistics, Institute for Education Sciences,U.S. Department of Education.

Renaissance Learning. (2011). STAR Reading. Retrievedfrom http://www.renlearn.com/sr/

Ritchey, K. D. (2008). Assessing letter sound knowledge:A comparison of letter sound fluency and nonsenseword fluency. Exceptional Children, 74, 487–506.

Ritchey, K. D., & Speece, D. L. (2004). Early identi-fication of reading disabilities: Current status andnew directions. Assessment for Effective Interven-tion, 29(4), 13–24.

Roehrig, A. D., Petscher, Y., Nettles, S. M., Hudson,R. F., & Torgesen, J. K. (2007). Accuracy of theDIBELS Oral Reading Fluency measure for predictingthird grade reading comprehension outcomes. Journalof School Psychology, 46, 343–366.

Salvia, J., Ysseldyke, J. E., & Bolt, S. (2009). Assessment:In special and inclusive education (11th ed.). NewYork: Wadsworth.

Shanahan, T. (2003). Review of the DIBELS: DynamicIndicators of Basic Early Literacy Skills (6th ed.). InB. S. Plake, J. C. Impara, & R. A. Spires (eds.), Thesixteenth mental measurements yearbook (pp. 310–313). Lincoln, NE: Buros Institute of Mental Measure-ments.

Sprinthall, R. C. (2003). Basic statistical analysis (7thed.). New York: Pearson.

Stage, S. A., & Jacobson, M. D. (2001). Predicting studentsuccess on a state-mandated performance-based as-sessment using oral reading fluency. School Psychol-ogy Review, 30, 407–419.

Swets, J. A. (1988). Measuring the diagnostic accuracy ofdiagnostic systems. Science, 240, 1285–1293.

Swets, J. A. (1996). Signal detection theory and ROCanalysis in psychology and diagnostics. Hillsdale, NJ:Lawrence Erlbaum Associates.

Tabachnick, B. G., & Fidell, L. S. (2007). Using multi-variate statistics (5th ed.). Needham Heights, MA:Allyn & Bacon.

Utah State Office of Education. (2007). Utah ELA CRTtechnical manual. Salt Lake City, UT: Author. Avail-able at www.usoe.k12.ut.us

Wiley, H. I., & Deno, S. L. (2005). Oral reading and mazemeasures as predictors of success for English learnerson a state standards assessment. Remedial and SpecialEducation, 26, 207–214.

Woodcock, R. (1998). Woodcock Reading Mastery Test—Revised/Normative Update. Circle Pines, MN: Ameri-can Guidance Service.

Date Received: November 11, 2010Date Accepted: January 15, 2011Action Editor: Sandy Chafouleas �


130

John L. Hosp, PhD, is an associate professor of teaching and learning at the University ofIowa and codirector of the Center for Disability Research and Education. His currentresearch interests include aligning assessment and instruction through curriculum-basedmeasurement and curriculum-based evaluation, particularly in the elementary grades, aswell as the disproportionate representation of students of color in special education.

Michelle K. Hosp, PhD, is a consultant with the Iowa Department of Education. Herinterests are curriculum-based measurement and curriculum-based evaluation for readingand literacy with elementary students. She has extensive experience writing about readingand assessments as well as presenting at local, state, and national conferences. She is alsocurrently a trainer for the National Center for Response to Intervention.

Janice A. Dole, PhD, is a professor of education at the University of Utah, where sheteaches graduate courses in reading. Her research interests include school reform inreading, professional development, and summer reading loss in high-poverty schools.


131

Copyright of School Psychology Review is the property of National Association of School Psychologists and its

content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's

express written permission. However, users may print, download, or email articles for individual use.

Potential Bias in Predictive Validity of Universal ...rachaelrobinsonedsi.wiki.westga.edu/file/view/Potential+bias+in... · Potential Bias in Predictive Validity of Universal Screening

Documents