Graphing, Calculating, and Graphing, Calculating, and Interpreting Rate of Interpreting Rate of
ImprovementImprovement
Caitlin S. Flinn, Ed.S., N.C.S.P.Caitlin S. Flinn, Ed.S., N.C.S.P.
Andrew E. McCrea, M.S., N.C.S.P.Andrew E. McCrea, M.S., N.C.S.P.
NASP ConventionNASP Convention
March 3, 2010March 3, 2010
ObjectivesObjectives
There needs to be a standardized There needs to be a standardized procedure for calculating RoI procedure for calculating RoI
We’re proposing a method using Simple We’re proposing a method using Simple Linear RegressionLinear Regression
OverviewOverview
Importance of RoIImportance of RoI RoI ResearchRoI Research A Need for A Need for
ConsistencyConsistency Calculating RoICalculating RoI
Individual Student Individual Student GraphsGraphs
Programming ExcelProgramming Excel
Decision MakingDecision Making Grounding the DataGrounding the Data Interpreting GrowthInterpreting Growth
Individual StudentIndividual Student Student GroupsStudent Groups
ConsiderationsConsiderations ResourcesResources
Importance of GraphsImportance of Graphs
Vogel, Dickson, & Lehman, 1990Vogel, Dickson, & Lehman, 1990Speeches that included visuals, especially in Speeches that included visuals, especially in
color, improved:color, improved:Immediate recall by 8.5%Immediate recall by 8.5%Delayed recall (3 days) by 10.1%Delayed recall (3 days) by 10.1%
Importance of GraphsImportance of Graphs
““Seeing is believing.”Seeing is believing.”Useful for communicating large amounts Useful for communicating large amounts
of information quicklyof information quickly ““A picture is worth a thousand words.”A picture is worth a thousand words.”Transcends language barriers (Karwowski, Transcends language barriers (Karwowski,
2006)2006)Responsibility for accurate graphical Responsibility for accurate graphical
representations of datarepresentations of data
Skills Typically GraphedSkills Typically Graphed ReadingReading
Oral Reading Fluency (ORF)Oral Reading Fluency (ORF) Word Use Fluency (WUF)Word Use Fluency (WUF) Reading ComprehensionReading Comprehension
MAZEMAZE Retell FluencyRetell Fluency
Early Literacy SkillsEarly Literacy Skills Initial Sound Fluency (ISF)Initial Sound Fluency (ISF) Letter Naming Fluency (LNF)Letter Naming Fluency (LNF) Letter Sound Fluency (LSF)Letter Sound Fluency (LSF) Phoneme Segmentation Fluency Phoneme Segmentation Fluency
(PSF)(PSF) Nonsense Word Fluency (NWF)Nonsense Word Fluency (NWF)
SpellingSpelling Written Expression Written Expression BehaviorBehavior
MathMath Math ComputationMath Computation Math FactsMath Facts Early NumeracyEarly Numeracy
Oral CountingOral Counting Missing NumberMissing Number Number IdentificationNumber Identification Quantity Quantity
DiscriminationDiscrimination
Importance of RoIImportance of RoI
Multi-tiered modelMulti-tiered modelProgress monitoringProgress monitoringData for decision-makingData for decision-makingGoal setting (Shapiro, 2008)Goal setting (Shapiro, 2008)
Importance of RoIImportance of RoI
Visual inspection of slopeVisual inspection of slopeMultiple interpretationsMultiple interpretations Instructional servicesInstructional servicesNeed for explicit guidelinesNeed for explicit guidelines
RoI ResearchRoI Research
Deno, 1985Deno, 1985Curriculum-based measurementCurriculum-based measurement
General outcome measuresGeneral outcome measuresShortShortStandardizedStandardizedRepeatableRepeatableSensitive to changeSensitive to change
RoI ResearchRoI Research
Fuchs & Fuchs, 1998Fuchs & Fuchs, 1998Hallmark components of Response to Hallmark components of Response to
InterventionInterventionOngoing formative assessmentOngoing formative assessmentIdentifying non-responding studentsIdentifying non-responding studentsTreatment fidelity of instructionTreatment fidelity of instruction
Dual discrepancy modelDual discrepancy modelOne standard deviation from typically performing One standard deviation from typically performing
peers in level and peers in level and raterate
RoI ResearchRoI Research
Ardoin & Christ, 2008Ardoin & Christ, 2008Slope for benchmarks (3x per year)Slope for benchmarks (3x per year)More growth from fall to winter than winter to More growth from fall to winter than winter to
springspringMight be helpful to use RoI for fall to winterMight be helpful to use RoI for fall to winterAnd a separate RoI for winter to springAnd a separate RoI for winter to spring
RoI ResearchRoI Research
Fuchs, Fuchs, Walz, & Germann, 1993Fuchs, Fuchs, Walz, & Germann, 1993Typical weekly growth ratesTypical weekly growth ratesNeeded growthNeeded growth
1.5 to 2.0 times typical slope to close gap1.5 to 2.0 times typical slope to close gap
ExampleExampleBob is below benchmark on ORFBob is below benchmark on ORFTypical slope is 1 wcpm per week growthTypical slope is 1 wcpm per week growthBob would need slope of 1.5 to 2 to close gap Bob would need slope of 1.5 to 2 to close gap
in a reasonable amount of timein a reasonable amount of time
RoI ResearchRoI Research
Deno, Fuchs, Marston, & Shin, 2001Deno, Fuchs, Marston, & Shin, 2001Slope of frequently non-responsive children Slope of frequently non-responsive children
approximated slope of children already approximated slope of children already identified as having a specific learning identified as having a specific learning disabilitydisability
RoI ResearchRoI Research
Algebraic term: Slope of a lineAlgebraic term: Slope of a lineVertical change over the horizontal changeVertical change over the horizontal changeRise over runRise over runm = (ym = (y2 2 - y- y11) / (x) / (x2 2 - x- x11))Describes the steepness of a line (Gall & Gall, Describes the steepness of a line (Gall & Gall,
2007)2007)
RoI ResearchRoI Research
Finding a student’s RoI = finding the slope Finding a student’s RoI = finding the slope of a lineof a lineUsing two data points on that lineUsing two data points on that line
Finding the line itselfFinding the line itselfLinear regressionLinear regressionOrdinary Least SquaresOrdinary Least Squares
RoI ResearchRoI Research
Gall & Gall, 2007Gall & Gall, 200710 data points are a minimum requirement for 10 data points are a minimum requirement for
a reliable trendlinea reliable trendlineHow does that affect the frequency of How does that affect the frequency of
administering progress monitoring probes?administering progress monitoring probes?
RoI ResearchRoI Research
Using RoI for instructional decisions is not Using RoI for instructional decisions is not a perfect processa perfect process
Research is currently looking to address Research is currently looking to address sources of error:sources of error:Christ, 2006 – standard error of measurement Christ, 2006 – standard error of measurement
for slopefor slopeArdoin & Christ, 2009 – passage difficulty and Ardoin & Christ, 2009 – passage difficulty and
variabilityvariabilityJenkin, Graff, & Miglioretti, 2009 – frequency Jenkin, Graff, & Miglioretti, 2009 – frequency
of progress monitoringof progress monitoring
RoI ResearchRoI Research
Questions yet to be empirically answeredQuestions yet to be empirically answeredWhat parameters of RoI indicate a lack of RtI?What parameters of RoI indicate a lack of RtI?How does standard error of measurement How does standard error of measurement
play into using RoI for instructional decision play into using RoI for instructional decision making?making?
How does RoI vary between standard protocol How does RoI vary between standard protocol interventions?interventions?
How does this apply to non-English speaking How does this apply to non-English speaking populations?populations?
How is RoI Calculated? How is RoI Calculated? Which way is best?Which way is best?
Multiple Methods for Multiple Methods for Calculating GrowthCalculating Growth
““Eye ball” ApproachEye ball” Approach Last point minus First point ApproachLast point minus First point Approach Split Middle ApproachSplit Middle Approach Linear Regression ApproachLinear Regression Approach
Eye Ball
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
Last minus First
8
10
7
17
14
11
19
14
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
14 - 8 = 6; 6/ 8 weeks = 0.75 words per week
Split Middle
10.5
14.5
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6 7 8 9 10
14.5-10.5 = 4; 4/8 weeks = 0.5 words per week
Linear Regression
8
10
7
17
14
11
19
14
y = 1.1429x + 7.3571
0
2
4
6
8
10
12
14
16
18
20
1 2 3 4 5 6 7 8
1.1 Words Per Week
RoI Consistency?RoI Consistency?
Eye BallEye Ball ??????
Last minus FirstLast minus First 0.750.75
Split Middle*Split Middle* 0.500.50
Linear Linear RegressionRegression
1.101.10
RoI Consistency?RoI Consistency?
If we are not all using the same model to If we are not all using the same model to compute RoI, we continue to have the same compute RoI, we continue to have the same problems as past models, where under one problems as past models, where under one approach a student meets SLD criteria, but approach a student meets SLD criteria, but under a different approach, the student does under a different approach, the student does not. not.
Hypothetically, if the RoI cut-off was 0.65 or Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to 0.95, different approaches would come to different conclusions on the same student. different conclusions on the same student.
Technical AdequacyTechnical Adequacy
Without a consensus on how to compute Without a consensus on how to compute RoI, we risk falling short of having RoI, we risk falling short of having technical adequacy within our model. technical adequacy within our model.
So, Which RoI Method is Best?So, Which RoI Method is Best?
Literature shows that Linear Literature shows that Linear Regression is Best PracticeRegression is Best Practice
Student’s daily test scores…were entered into a Student’s daily test scores…were entered into a computer program…The data analysis program computer program…The data analysis program generated generated slopes of improvementslopes of improvement for each level for each level using and using and Ordinary-Least SquaresOrdinary-Least Squares procedure procedure (Hayes, 1973) and the line of best fit. (Hayes, 1973) and the line of best fit.
This This procedure has been demonstrated to procedure has been demonstrated to represent CBM achievement data validly within represent CBM achievement data validly within individual treatment phasesindividual treatment phases (Marston, 1988; (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987).Shinn, Good, & Stein, in press; Stein, 1987).
Shinn, Gleason, & Tindal, 1989Shinn, Gleason, & Tindal, 1989
Growth (RoI) Research Growth (RoI) Research using Linear Regressionusing Linear Regression
Christ, T. J. (2006). Short-term estimates of growth using Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence Estimating standard error of the slope to construct confidence intervals. intervals. School Psychology ReviewSchool Psychology Review, 35, 128-133., 35, 128-133.
Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards curriculum based measurement to establish growth standards for students with learning disabilities. for students with learning disabilities. School Psychology School Psychology ReviewReview, 30, 507-524., 30, 507-524.
Good, R. H. (1990). Forecasting accuracy of slope estimates for Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. reading curriculum based measurement: Empirical evidence. Behavioral AssessmentBehavioral Assessment, 12, 179-193., 12, 179-193.
Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much (1993). Formative evaluation of academic progress: How much growth can we expect? growth can we expect? School Psychology ReviewSchool Psychology Review, 22, 27-48., 22, 27-48.
Growth (RoI) ResearchGrowth (RoI) Researchusing Linear Regressionusing Linear Regression
Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM Estimating reading growth using intermittent CBM progress monitoring. progress monitoring. Exceptional ChildrenExceptional Children, 75, 151-163., 75, 151-163.
Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for Varying the difficulty of testing materials: Implications for curriculum-based measurement. curriculum-based measurement. The Journal of Special The Journal of Special EducationEducation, 23, 223-233., 23, 223-233.
Shinn, M. R., Good, R. H., & Stein, S. (1989). Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A Summarizing trend in student achievement: A comparison of methods. comparison of methods. School Psychology ReviewSchool Psychology Review, 18, , 18, 356-370.356-370.
So, Why Are There So Many So, Why Are There So Many Other RoI Models?Other RoI Models?
Ease of application Ease of application How many of us want to calculate OLS How many of us want to calculate OLS
Linear Regression formulas (or even Linear Regression formulas (or even remember how)?remember how)?
An Easy and An Easy and Applicable SolutionApplicable Solution
Get Out Your Laptops!Get Out Your Laptops!
Or Kindly Look Over Your Or Kindly Look Over Your Neighbor’s Shoulder!Neighbor’s Shoulder!
I loveROI
Open Microsoft ExcelOpen Microsoft Excel
Microsoft Office 2003 for PCsMicrosoft Office 2003 for PCsMicrosoft Office 2007 for PCsMicrosoft Office 2007 for PCsMicrosoft Office for MacsMicrosoft Office for Macs
Graphing RoIGraphing RoI
For Individual StudentsFor Individual Students
Setting Up Your SpreadsheetSetting Up Your Spreadsheet
In cell B2, type In cell B2, type School WeekSchool Week In cell C2, type In cell C2, type BenchmarkBenchmark In cell D2, type WPM (or Student Scores)In cell D2, type WPM (or Student Scores)
Labeling School WeeksLabeling School Weeks
In cell B3, type In cell B3, type 11 Continue entering numbersContinue entering numbers
through through 3636 in column B in column BWeek Week 3636 will be in cell B38 will be in cell B38
Entering BenchmarksEntering Benchmarks
In cell C3, type the fall benchmark In cell C3, type the fall benchmark 7777 In cell C20, type the winter benchmark In cell C20, type the winter benchmark 9292 In cell C38, type the spring benchmark In cell C38, type the spring benchmark
110110
Entering Student Data PointsEntering Student Data Points
Student data points are entered between Student data points are entered between cells D3 and D38.cells D3 and D38.
Type the student’s score next to the Type the student’s score next to the corresponding week that it was corresponding week that it was administered.administered.
Entering Student Data PointsEntering Student Data Points
Week 1 – 41Week 1 – 41Week 8 – 62Week 8 – 62Week 9 – 63Week 9 – 63Week 10 – 75Week 10 – 75Week 11 – 64Week 11 – 64Week 12 – 80Week 12 – 80Week 13 – 83Week 13 – 83Week 14 - 83Week 14 - 83
Entering Student Data PointsEntering Student Data Points
Week 15 – 56Week 15 – 56Week 17 – 104Week 17 – 104Week 18 – 74Week 18 – 74Week 20 – 85Week 20 – 85Week 21 – 89Week 21 – 89Week 22 – 69Week 22 – 69Week 23 – 85Week 23 – 85
Entering Student Data PointsEntering Student Data Points
Week 24 – 96Week 24 – 96Week 25 – 90Week 25 – 90Week 26 – 84Week 26 – 84Week 27 – 106Week 27 – 106Week 28 – 94Week 28 – 94Week 32 – 100Week 32 – 100
*CAUTION**CAUTION*
If a student was not assessed during a If a student was not assessed during a certain week, leave that cell blankcertain week, leave that cell blank
Do not enter a score of Zero (0) it will be Do not enter a score of Zero (0) it will be calculated into the trendline and calculated into the trendline and interpreted as the student having read interpreted as the student having read zero words correct per minute during that zero words correct per minute during that week. week.
Creating a GraphCreating a Graph
Highlight the Highlight the data in Columnsdata in ColumnsC and DC and D Include cells Include cells C2 and D2 through C2 and D2 through cells C38 and D38cells C38 and D38 Include any Include any blank cellsblank cells
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Click Click InsertInsert Click Click ChartChart
Excel 2007Excel 2007 Click Click InsertInsert Find the icon for Find the icon for LineLine Click the arrow below Click the arrow below
LineLine
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs A Chart Wizard A Chart Wizard
window will appearwindow will appear
Excel 2007Excel 2007 6 graphics appear6 graphics appear
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Choose Choose LineLine Choose Choose Line with Line with
markersmarkers
Excel 2007Excel 2007 Choose Choose Line with Line with
markersmarkers
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Data RangeData Range tab tab ColumnsColumns
Excel 2007Excel 2007 Your graph appearsYour graph appears
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Chart TitleChart Title School WeekSchool Week (X Axis) (X Axis) WPMWPM (Y Axis) (Y Axis)
Excel 2007Excel 2007 Change your labels by Change your labels by
clicking on the graphclicking on the graph
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Choose where you Choose where you
want your graphwant your graph
Excel 2007Excel 2007 Your graph was Your graph was
automatically put into automatically put into your data spreadsheetyour data spreadsheet
Creating a GraphCreating a Graph
Excel 2003/MacsExcel 2003/Macs Excel 2007Excel 2007
Adding a TrendlineAdding a Trendline
Excel 2003/MacsExcel 2003/MacsRight click on any of the student data pointsRight click on any of the student data points
Excel 2007Excel 2007
Adding a TrendlineAdding a Trendline
Excel 2003/MacsExcel 2003/MacsChoose Choose LinearLinear
Excel 2007Excel 2007
Adding a TrendlineAdding a Trendline
Excel 2003/MacsExcel 2003/MacsChoose Choose CustomCustom and check box next to and check box next to
Display equation on chartDisplay equation on chart
Excel 2007Excel 2007
Adding a TrendlineAdding a Trendline
Clicking on the equation highlights a box Clicking on the equation highlights a box around itaround it
Clicking on the box allows you to move it Clicking on the box allows you to move it to a place where you can see it betterto a place where you can see it better
Adding a TrendlineAdding a Trendline
You can repeat the same procedure to You can repeat the same procedure to have a trendline for the benchmark data have a trendline for the benchmark data pointspoints
Suggestion: label the trendline Suggestion: label the trendline Expected Expected ROIROI
Move this equation under the firstMove this equation under the first
Individual Student GraphIndividual Student Graph
Diego's Rate of Improvement
y = 1.6317x + 50.928
y = 0.9434x + 75.704
0
20
40
60
80
100
120
1 4 7 10 13 16 19 22 25 28 31 34
School Week
Wo
rds
Per
Min
ute
Benchmark (3rd)
Diego's Scores (3rd)
Diego's ROI
Goal Slope
Individual Student GraphIndividual Student Graph
The equation indicates the slope, or rate of The equation indicates the slope, or rate of improvement. improvement.
The number, or coefficient,The number, or coefficient, beforebefore "x" is "x" is the average improvement, which in this the average improvement, which in this case is the average number of words per case is the average number of words per minute per week gained by the student. minute per week gained by the student.
Individual Student GraphIndividual Student Graph
The rate of improvement, or trendline, is The rate of improvement, or trendline, is calculated using a linear regression, a calculated using a linear regression, a simple equation of least squares. simple equation of least squares.
To add additional progress To add additional progress monitoring/benchmark scores once you’ve monitoring/benchmark scores once you’ve already created a graph, enter additional already created a graph, enter additional scores in Column D in the corresponding scores in Column D in the corresponding school week.school week.
Individual Student GraphIndividual Student Graph
Remember to leave cells blank for the Remember to leave cells blank for the weeks that no score was obtained. weeks that no score was obtained.
The graph will incorporate that score into The graph will incorporate that score into the set of data points and into the the set of data points and into the trendline.trendline.
Individual Student GraphIndividual Student Graph
The slope can change depending on The slope can change depending on which week (where) you put the which week (where) you put the benchmark scores on your chart. benchmark scores on your chart.
Enter benchmark scores based on when Enter benchmark scores based on when your school administers their benchmark your school administers their benchmark assessments for the most accurate assessments for the most accurate depiction of expected student progress.depiction of expected student progress.
Options for the GraphOptions for the Graph
ResizingResizingColoringColoringData LabelsData Labels
Programming ExcelProgramming Excel
To Calculate RoITo Calculate RoI
A FormulaA Formula
RoI FormulaRoI Formula
Type Type RoIRoI in cell B39 below the last week in cell B39 below the last week of schoolof school
Calculate Expected SlopeCalculate Expected Slope
Click on cell C39Click on cell C39Put your cursor at the top next to the Put your cursor at the top next to the fxfxType =SLOPE(C3:C38,B3:B38)Type =SLOPE(C3:C38,B3:B38)Hit Enter/ReturnHit Enter/Return
Calculate Actual SlopeCalculate Actual Slope
Click on cell D39Click on cell D39Put your cursor at the top next to the Put your cursor at the top next to the fxfxType =SLOPE(D3:D38,B3:B38)Type =SLOPE(D3:D38,B3:B38)Hit Enter/ReturnHit Enter/Return
ROI as a Decision ToolROI as a Decision Tool
within a Problem-Solving Modelwithin a Problem-Solving Model
StepsSteps
1.1. Gather the dataGather the data
2.2. Ground the dataGround the data
3.3. Interpret the dataInterpret the data
4.4. Figure out how to fit Best Practice into Figure out how to fit Best Practice into Public EducationPublic Education
Step 1: Gather DataStep 1: Gather Data
Universal Screening Universal Screening
Progress MonitoringProgress Monitoring
Common Screenings in PACommon Screenings in PA
DIBELSDIBELSAIMSwebAIMSwebMBSPMBSP4Sight4SightPSSAPSSA
Validated Progress Validated Progress Monitoring ToolsMonitoring Tools
DIBELSDIBELSAIMSwebAIMSwebMBSPMBSPwww.studentprogress.orgwww.studentprogress.org
Step 2: Ground the DataStep 2: Ground the Data
To what will we compare our To what will we compare our student growth data?student growth data?
Multiple Ways to Multiple Ways to Look at GrowthLook at Growth
Needed Growth Needed Growth Expected Growth & Percent of Expected Expected Growth & Percent of Expected
GrowthGrowthFuchs et. al. (1993) Table of Realistic and Fuchs et. al. (1993) Table of Realistic and
Ambitious GrowthAmbitious GrowthGrowth Toward Individual Goal*Growth Toward Individual Goal*
*Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)*Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)
Needed GrowthNeeded Growth
Difference between student’s BOY (or Difference between student’s BOY (or MOY) score and benchmark score at MOY MOY) score and benchmark score at MOY (or EOY).(or EOY).
Example: MOY ORF = 10, EOY Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.wcpm per week to make EOY benchmark.
Expected GrowthExpected Growth
Difference between two benchmarks.Difference between two benchmarks.Example: MOY benchmark is 20, EOY Example: MOY benchmark is 20, EOY
benchmark is 40, expected growth (40-benchmark is 40, expected growth (40-20)/18 weeks of instruction = 1.11 wcpm 20)/18 weeks of instruction = 1.11 wcpm per week.per week.
Tigard-Tualatin School District (www.ttsd.k12.or.us)Tigard-Tualatin School District (www.ttsd.k12.or.us)
Looking at Percent of Looking at Percent of Expected GrowthExpected Growth
Tier ITier I Tier IITier II Tier IIITier III
Greater than Greater than 150%150%
Between Between 110% & 150%110% & 150%
Possible LDPossible LD
Between 95% Between 95% & 110%& 110%
Likely LDLikely LD
Between 80% Between 80% & 95%& 95%
May Need May Need MoreMore
May Need May Need MoreMore
Likely LDLikely LD
Below 80%Below 80% Needs MoreNeeds More Needs MoreNeeds More Likely LDLikely LD
Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)
Oral Reading Fluency Adequate Oral Reading Fluency Adequate Response TableResponse Table
Realistic Realistic GrowthGrowth
Ambitious Ambitious GrowthGrowth
11stst 2.02.0 3.03.0
22ndnd 1.51.5 2.02.0
33rdrd 1.01.0 1.51.5
44thth 0.90.9 1.11.1
55thth 0.50.5 0.80.8
Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)
Digit Fluency Adequate Digit Fluency Adequate Response TableResponse Table
Realistic Realistic GrowthGrowth
Ambitious Ambitious GrowthGrowth
11stst 0.30.3 0.50.5
22ndnd 0.30.3 0.50.5
33rdrd 0.30.3 0.50.5
44thth 0.750.75 1.21.2
55thth 0.750.75 1.21.2
Making Decisions: Best Practice Making Decisions: Best Practice
Research has yet to establish a blue print Research has yet to establish a blue print for ‘grounding’ student RoI data. for ‘grounding’ student RoI data.
At this point, teams should consider At this point, teams should consider multiple comparisons when planning and multiple comparisons when planning and making decisions. making decisions.
Making Decisions: Lessons Making Decisions: Lessons From the FieldFrom the Field
When tracking on grade level, consider an When tracking on grade level, consider an RoI that is 100% of RoI that is 100% of expectedexpected growth as a growth as a minimum requirement, consider an RoI minimum requirement, consider an RoI that is at or above the that is at or above the neededneeded as optimal. as optimal.
So, 100% of expected and on par with So, 100% of expected and on par with needed become the limits of the range needed become the limits of the range within a student should be achieving.within a student should be achieving.
Oral Reading Fluency01/15/09 01/22/09 01/29/09 02/05/09 02/12/09 02/19/09 02/26/09 03/05/09 03/12/09 03/19/09 03/26/09 04/02/09 04/09/09 04/16/09 04/23/09 04/30/09 05/07/09 05/14/09
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark 68 90 1.29
Aiden 61 40 52 60 71 95 1.61 2.17 167%
Ava 49 43 49 77 57 54 87 92 2.28 2.76 213%
Noah 49 48 45 69 61 54 84 2.28 2.01 156%
Olivia 65 49 57 70 79 83 1.39 1.50 116%
Liam 55 53 36 54 70 83 1.94 1.58 122%
Hannah 59 54 64 69 52 60 82 1.72 1.20 93%
Gavin 64 40 67 68 84 79 1.44 1.66 129%
Grace 53 48 46 60 74 79 2.06 1.76 136%
Oliver 50 44 46 68 51 51 57 78 2.22 1.45 112%
Peyton 63 50 47 58 75 77 1.50 1.12 87%
Josh 49 38 49 55 48 36 67 77 2.28 1.62 125%
Riley 42 49 54 69 67 50 76 2.67 1.76 136%
Mason 53 53 50 64 60 74 2.06 1.17 91%
Zoe 34 38 42 68 55 51 58 3.11 1.44 111%
Ian 41 31 45 49 47 30 46 2.72 0.24 19%
Faith 29 36 35 36 36 29 45 44 3.39 0.75 58%
David 30 23 44 52 43 19 63 38 3.33 0.79 61%
Alexa 18 19 25 33 33 23 28 37 4.00 0.94 73%
Hunter 23 23 24 48 38 32 34 3.72 0.75 58%
Caroline 28 20 28 40 37 19 25 30 3.44 0.02 2%
** Actual RoI based on linear regression of all data points
Benchmarks based on DIBELS Goals
Expected RoI at Benchmark Level
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
% of Expected RoI
Needed RoI* Actual RoI**
Realistic Grow thAmbitious Grow th
Oral Reading Fluency Adequate Response Table
1st Grade
2nd Grade
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
0.5
3rd Grade
4th Grade
5th Grade
0.9
0.8
2.0
1.5
1.0
3.0
2.0
1.5
1.1
Step 3: Interpreting GrowthStep 3: Interpreting Growth
What do we do when we do not What do we do when we do not get the growth we want?get the growth we want?
When to make a change in instruction and When to make a change in instruction and intervention?intervention?
When to consider SLD?When to consider SLD?
When to make a change in When to make a change in instruction and intervention?instruction and intervention?
Enough data points (6 to 10)?Enough data points (6 to 10)?Less than 100% of expected growth.Less than 100% of expected growth.Not on track to make benchmark (needed Not on track to make benchmark (needed
growth).growth).Not on track to reach individual goal.Not on track to reach individual goal.
When to consider SLD?When to consider SLD?
Continued inadequate response despite: Continued inadequate response despite: Fidelity with Tier I instruction and Tier Fidelity with Tier I instruction and Tier
II/III intervention.II/III intervention. Multiple attempts at intervention.Multiple attempts at intervention. Individualized Problem-Solving Individualized Problem-Solving
approach. approach.
Three Levels of ExamplesThree Levels of Examples
Whole ClassWhole ClassSmall GroupSmall Group Individual Student Individual Student
- Academic Data- Academic Data
- Behavior Data- Behavior Data
Whole Class ExampleWhole Class ExampleComputation
01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
50th P ercentile 25 31 0.35
25th P ercentile 19 23 0.24
Student 6.5 9 8 8.5 5.5 11 13 1.72 0.61 173%
Student 6 7.5 8.5 5 11 11.5 1.72 0.57 161%
Student 4.5 5.5 6.5 9.5 10.5 1.72 1.06 300%
Student 13 8 9.3 8 5.6 9.6 9.6 1.72 -0.23 -66%
Student 8 10.5 10.5 5.6 9.3 9 1.72 -0.03 -7%
Student 8.5 5.5 9 8 4 8 9 1.72 0.07 21%
Student 6.5 5.5 6 10.5 9 1.72 0.43 122%
Student 6.5 9 4.5 6 8 1.72 0.07 20%
Student 8 10.5 4.5 6.5 4 7 1.72 -0.25 -71%
Student 9 10 5.6 6.6 5 4.6 6.6 1.72 -0.42 -119%
Student 8 8 8.5 4 8 6.6 1.72 -0.18 -51%
Student 9 4.5 4.5 4 3.5 3.5 6.5 1.72 -0.24 -67%
Student 6.5 5 6.5 9 7.5 6.5 1.72 0.09 26%
Student 5.5 3 8 4 6.5 6.3 1.72 0.19 55%
Student 7.5 10 6.6 3.3 3 6.3 1.72 -0.46 -130%
Student 5 5.5 6.5 6 5 6 1.72 0.04 11%
Student 5 4 8 8.5 10 8 6 1.72 0.25 71%
Student 4.5 6 5 2.5 3.5 5.5 1.72 -0.03 -8%
Student 5.5 4.5 10.5 11 5 5.3 1.72 -0.14 -40%
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
** Actual RoI based on linear regression of all data points
Percentiles based on AIMSw eb Grow th Tables
Expected RoI at 50th PercentileExpected RoI at 25th Percentile
Needed RoI* Actual RoI** % of Expected RoI
Digit Fluency Adequate Response Table
Realistic Grow thAmbitious Grow th
1st Grade 0.3 0.5
2nd Grade 0.3 0.5
3rd Grade 0.3 0.5
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
4th Grade 0.75 1.2
5th Grade 0.75 1.2
33rdrd Grade Math Whole Class Grade Math Whole Class
Who’s responding? Who’s responding? Effective math Effective math
instruction? instruction? Who needs more?Who needs more?
N=19N=194 > 100% growth 4 > 100% growth 15 < 100% growth15 < 100% growth9 w/ negative 9 w/ negative
growthgrowth
Small Group ExampleSmall Group ExampleOral Reading Fluency
09/11/09 09/18/09 09/25/09 10/02/09 10/09/09 10/16/09 10/23/09 10/30/09 11/06/09 11/13/09 11/20/09 11/27/09 12/04/09 12/11/09 12/18/09 01/01/10 01/08/10 01/15/10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Benchmark 44 68 1.41
Student 35 39 41 45 42 45 52 57 62 1.83 1.49 106%
Student 28 38 42 40 50 55 64 72 74 2.22 2.77 196%
Student 26 28 32 31 27 29 35 34 38 2.33 0.57 41%
Student 31 35 39 45 42 47 53 58 65 2.06 1.90 135%
Student 40 44 38 48 52 64 72 74 78 1.56 2.62 186%
** Actual RoI based on linear regression of all data points
Benchmarks based on DIBELS Goals
Expected RoI at Benchmark Level
(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)
5th Grade
* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks
% of Expected RoI
2nd Grade
0.9 1.1
3rd Grade
4th Grade
0.5 0.8
1.5 2.0
1.0 1.5
Needed RoI* Actual RoI**
2.0 3.0
Oral Reading Fluency Adequte Response Table
1st Grade
Realistic Grow thAmbitious Grow th
Intervention GroupIntervention Group
Intervention working for how many?Intervention working for how many?Can we assume fidelity of intervention Can we assume fidelity of intervention
based on results?based on results?Who needs more?Who needs more?
Individual Kid ExampleIndividual Kid Example 2nd Grade Reading Progress
44
68
90
31
56
45
53 53
4846
60
74
79
y = 1.5333x + 42.8
y = 0.9903x + 36.873
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
09/12/08 09/19/0809/26/0810/03/08 10/10/08 10/17/08 10/24/08 10/31/08 11/07/08 11/14/08 11/21/08 11/28/08 12/05/08 12/12/08 12/19/08 01/16/09 01/23/0901/30/0902/06/0902/13/0902/20/0902/27/0903/06/0903/13/0903/20/0903/27/0904/03/0904/10/0904/17/0904/24/0905/01/09
Wo
rds
Re
ad
Co
rre
ct
Pe
r M
inu
te
Benchmark Linear (Benchmark) Linear
Individual KidIndividual Kid
Making growth?Making growth?How much (65% of expected growth).How much (65% of expected growth).Atypical growth across the year (last 3 Atypical growth across the year (last 3
data points). data points). Continue? Make a change? Need more Continue? Make a change? Need more
data?data?
RoI and Behavior?RoI and Behavior?
Percent of Time Engaged in Appropriate Behavior
y = 2x + 22
y = 3.9x + 19.8
y = 7.2143x - 1.5
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Per
cen
t
Baseline Condition 1 Condition 2 Linear (Baseline) Linear (Condition 1) Linear (Condition 2) Linear (Condition 2)
Step 4: Figure out how to fit Step 4: Figure out how to fit Best Practice into Public Best Practice into Public
EducationEducation
Things to ConsiderThings to Consider
Who is At-Risk and needs progress Who is At-Risk and needs progress monitoring?monitoring?
Who will collect, score, enter the data?Who will collect, score, enter the data?Who will monitor student growth, when, Who will monitor student growth, when,
and how often?and how often?What changes should be made to What changes should be made to
instruction & intervention? instruction & intervention? What about monitoring off of grade level?What about monitoring off of grade level?
Who is At-Risk and needs Who is At-Risk and needs progress monitoring?progress monitoring?
Below level on universal screeningBelow level on universal screening
Entering 4Entering 4thth Grade Example Grade Example
DORF DORF (110)(110)
ISIP ISIP TRWM TRWM
(55)(55)
4Sight 4Sight (1235)(1235)
PSSA PSSA (1235)(1235)
Student AStudent A 115115 5858 12551255 12321232
Student BStudent B 8585 4848 12161216 11261126
Student CStudent C 7272 3535 10561056 10481048
Who will collect, score, and Who will collect, score, and enter the data?enter the data?
Using MBSP for math, teachers can Using MBSP for math, teachers can administer probes to whole class.administer probes to whole class.
DORF probes must be administered one-DORF probes must be administered one-on-one, and creativity pays off (train and on-one, and creativity pays off (train and use art, music, library, etc. specialists).use art, music, library, etc. specialists).
Schedule for progress monitoring math Schedule for progress monitoring math and reading every-other week.and reading every-other week.
Week 1Week 1 Week 2Week 2
ReadingReading MathMath ReadingReading MathMath
11stst XX XX
22ndnd XX XX
33rdrd XX XX
44thth XX XX
55thth XX XX
Who will monitor student Who will monitor student growth, when, and how often?growth, when, and how often?
Best Practices in Data-Analysis Teaming Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008)(Kovaleski & Pedersen, 2008)
Chambersburg Area School District Elementary Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. Response to Intervention Manual (McCrea et. al., 2008)al., 2008)
Derry Township School District Response to Derry Township School District Response to Intervention Model Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-(http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)
What changes should be made What changes should be made to instruction & intervention?to instruction & intervention?
Ensure treatment fidelity!!!!!!!!Ensure treatment fidelity!!!!!!!! Increase instructional time (active and Increase instructional time (active and
engaged)engaged)Decrease group sizeDecrease group sizeGather additional, diagnostic, informationGather additional, diagnostic, informationChange the intervention Change the intervention
When Instructional Level is Not When Instructional Level is Not the Same as Grade Levelthe Same as Grade Level
Understand needed and expected RoI Understand needed and expected RoI within broader context: within broader context:
Needed growthNeeded growth will only get student will only get student to nextto next level by next benchmark (as opposed to level by next benchmark (as opposed to on on levellevel).).
100% of expected growth100% of expected growth may not be an may not be an acceptable minimum (not enough growth b/c acceptable minimum (not enough growth b/c level is so low).level is so low).
Grounding RoI When Monitoring Grounding RoI When Monitoring Off of Grade Level: Two OptionsOff of Grade Level: Two OptionsBest Practices in Setting Progress Best Practices in Setting Progress
Monitoring Goals for Academic Skill Monitoring Goals for Academic Skill Improvement (Shapiro, 2008).Improvement (Shapiro, 2008).
Tigard-Tualatin SD Chart: 150% instead of Tigard-Tualatin SD Chart: 150% instead of 100% as minimum RoI requirement???100% as minimum RoI requirement???
Questions? & Comments!Questions? & Comments!
ResourcesResources
www.interventioncentral.comwww.interventioncentral.com
www.aimsweb.comwww.aimsweb.com
http://dibels.uoregon.eduhttp://dibels.uoregon.edu
www.nasponline.orgwww.nasponline.org
ResourcesResources
www.fcrr.orgwww.fcrr.org
Florida Center for Reading ResearchFlorida Center for Reading Researchhttp://ies.ed.gov/ncee/wwc//http://ies.ed.gov/ncee/wwc//
What Works ClearinghouseWhat Works Clearinghousehttp://www.rti4success.orghttp://www.rti4success.org
National Center on RtINational Center on RtI
Flinn & McCrea’s RoI Web SiteFlinn & McCrea’s RoI Web Site
http://sites.google.com/site/http://sites.google.com/site/rateofimprovement/rateofimprovement/Download powerpoints, handouts, Excel Download powerpoints, handouts, Excel
graphs, charts, articles, etc.graphs, charts, articles, etc.Caitlin FlinnCaitlin Flinn
[email protected] [email protected] Andrew McCreaAndrew McCrea
ReferencesReferences
Ardoin, S. P., & Christ, T. J. (2009). Curriculum-Ardoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard based measurement of oral reading: Standard errors associated with progress monitoring errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an outcomes from DIBELS, AIMSweb, and an experimental passage set. experimental passage set. School Psychology School Psychology Review, 38(2),Review, 38(2), 266-283. 266-283.
Ardoin, S. P. & Christ, T. J. (2008). Evaluating Ardoin, S. P. & Christ, T. J. (2008). Evaluating curriculum-based measurement slope estimates curriculum-based measurement slope estimates using triannual universal screenings. using triannual universal screenings. School School Psychology Review, 37Psychology Review, 37(1), 109-125. (1), 109-125.
ReferencesReferences
Christ, T. J. (2006). Short-term estimates of Christ, T. J. (2006). Short-term estimates of growth using curriculum-based measurement growth using curriculum-based measurement of oral reading fluency: Estimating standard of oral reading fluency: Estimating standard error of the slope to construct confidence error of the slope to construct confidence intervals. intervals. School Psychology Review, 35School Psychology Review, 35(1), (1), 128-133.128-133.
Deno, S. L. (1985). Curriculum-based Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. measurement: The emerging alternative. Exceptional Children, 52,Exceptional Children, 52, 219-232. 219-232.
ReferencesReferences
Deno, S. L., Fuchs, L.S., Marston, D., & Deno, S. L., Fuchs, L.S., Marston, D., & Shin, J. (2001). Using curriculum-based Shin, J. (2001). Using curriculum-based measurement to establish growth measurement to establish growth standards for students with learning standards for students with learning disabilities. disabilities. School Psychology Review, School Psychology Review, 3030, 507-524., 507-524.
Flinn, C. S. (2008). Graphing rate of Flinn, C. S. (2008). Graphing rate of improvement for individual students. improvement for individual students. InSight, 28(3InSight, 28(3), 10-12.), 10-12.
ReferencesReferences
Fuchs, L. S., & Fuchs, D. (1998). Treatment Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for reconceptualizing validity: A unifying concept for reconceptualizing the identification of learning disabilities. the identification of learning disabilities. Learning Learning Disabilities Research and Practice, 13Disabilities Research and Practice, 13, 204-219., 204-219.
Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of Germann, G. (1993). Formative evaluation of academic progress: How much growth can we academic progress: How much growth can we expect? expect? School Psychology Review, 22School Psychology Review, 22, 27-48., 27-48.
ReferencesReferences
Gall, M.D., & Gall, J.P. (2007). Gall, M.D., & Gall, J.P. (2007). Educational Educational research: An introductionresearch: An introduction (8th ed.). New (8th ed.). New York: Pearson.York: Pearson.
Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). (2009). Estimating reading growth using Estimating reading growth using intermittent CBM progress monitoring. intermittent CBM progress monitoring. Exceptional Children, 75Exceptional Children, 75, 151-163., 151-163.
ReferencesReferences
Karwowski, W. (2006). Karwowski, W. (2006). International International encyclopedia of ergonomics and human encyclopedia of ergonomics and human factorsfactors. Boca Raton, FL: Taylor & Francis . Boca Raton, FL: Taylor & Francis Group, LLC.Group, LLC.
Shapiro, E. S. (2008). Best practices in setting Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes improvement. In A. Thomas and J. Grimes (Eds.), (Eds.), Best practices in school psychology V Best practices in school psychology V (Vol. 2, pp. 141-157). Bethesda, MD: National (Vol. 2, pp. 141-157). Bethesda, MD: National Association of School Psychologists.Association of School Psychologists.
ReferencesReferences
Vogel, D. R., Dickson, G. W., & Lehman, J. Vogel, D. R., Dickson, G. W., & Lehman, J. A. (1990). Persuasion and the role of A. (1990). Persuasion and the role of visual presentation support. The UM/3M visual presentation support. The UM/3M study. In M. Antonoff (Ed.), study. In M. Antonoff (Ed.), Presentations Presentations that persuade. Personal Computing, 14that persuade. Personal Computing, 14..