Top Banner
Graphing, Graphing, Calculating, and Calculating, and Interpreting Rate of Interpreting Rate of Improvement Improvement Caitlin S. Flinn, Ed.S., Caitlin S. Flinn, Ed.S., N.C.S.P. N.C.S.P. Andrew E. McCrea, M.S., Andrew E. McCrea, M.S., N.C.S.P. N.C.S.P. NASP Convention NASP Convention March 3, 2010 March 3, 2010
115

Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Mar 26, 2015

Download

Documents

Riley Russell
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Graphing, Calculating, and Graphing, Calculating, and Interpreting Rate of Interpreting Rate of

ImprovementImprovement

Caitlin S. Flinn, Ed.S., N.C.S.P.Caitlin S. Flinn, Ed.S., N.C.S.P.

Andrew E. McCrea, M.S., N.C.S.P.Andrew E. McCrea, M.S., N.C.S.P.

NASP ConventionNASP Convention

March 3, 2010March 3, 2010

Page 2: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ObjectivesObjectives

There needs to be a standardized There needs to be a standardized procedure for calculating RoI procedure for calculating RoI

We’re proposing a method using Simple We’re proposing a method using Simple Linear RegressionLinear Regression

Page 3: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

OverviewOverview

Importance of RoIImportance of RoI RoI ResearchRoI Research A Need for A Need for

ConsistencyConsistency Calculating RoICalculating RoI

Individual Student Individual Student GraphsGraphs

Programming ExcelProgramming Excel

Decision MakingDecision Making Grounding the DataGrounding the Data Interpreting GrowthInterpreting Growth

Individual StudentIndividual Student Student GroupsStudent Groups

ConsiderationsConsiderations ResourcesResources

Page 4: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Importance of GraphsImportance of Graphs

Vogel, Dickson, & Lehman, 1990Vogel, Dickson, & Lehman, 1990Speeches that included visuals, especially in Speeches that included visuals, especially in

color, improved:color, improved:Immediate recall by 8.5%Immediate recall by 8.5%Delayed recall (3 days) by 10.1%Delayed recall (3 days) by 10.1%

Page 5: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Importance of GraphsImportance of Graphs

““Seeing is believing.”Seeing is believing.”Useful for communicating large amounts Useful for communicating large amounts

of information quicklyof information quickly ““A picture is worth a thousand words.”A picture is worth a thousand words.”Transcends language barriers (Karwowski, Transcends language barriers (Karwowski,

2006)2006)Responsibility for accurate graphical Responsibility for accurate graphical

representations of datarepresentations of data

Page 6: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Skills Typically GraphedSkills Typically Graphed ReadingReading

Oral Reading Fluency (ORF)Oral Reading Fluency (ORF) Word Use Fluency (WUF)Word Use Fluency (WUF) Reading ComprehensionReading Comprehension

MAZEMAZE Retell FluencyRetell Fluency

Early Literacy SkillsEarly Literacy Skills Initial Sound Fluency (ISF)Initial Sound Fluency (ISF) Letter Naming Fluency (LNF)Letter Naming Fluency (LNF) Letter Sound Fluency (LSF)Letter Sound Fluency (LSF) Phoneme Segmentation Fluency Phoneme Segmentation Fluency

(PSF)(PSF) Nonsense Word Fluency (NWF)Nonsense Word Fluency (NWF)

SpellingSpelling Written Expression Written Expression BehaviorBehavior

MathMath Math ComputationMath Computation Math FactsMath Facts Early NumeracyEarly Numeracy

Oral CountingOral Counting Missing NumberMissing Number Number IdentificationNumber Identification Quantity Quantity

DiscriminationDiscrimination

Page 7: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Importance of RoIImportance of RoI

Multi-tiered modelMulti-tiered modelProgress monitoringProgress monitoringData for decision-makingData for decision-makingGoal setting (Shapiro, 2008)Goal setting (Shapiro, 2008)

Page 8: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Importance of RoIImportance of RoI

Visual inspection of slopeVisual inspection of slopeMultiple interpretationsMultiple interpretations Instructional servicesInstructional servicesNeed for explicit guidelinesNeed for explicit guidelines

Page 9: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Deno, 1985Deno, 1985Curriculum-based measurementCurriculum-based measurement

General outcome measuresGeneral outcome measuresShortShortStandardizedStandardizedRepeatableRepeatableSensitive to changeSensitive to change

Page 10: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Fuchs & Fuchs, 1998Fuchs & Fuchs, 1998Hallmark components of Response to Hallmark components of Response to

InterventionInterventionOngoing formative assessmentOngoing formative assessmentIdentifying non-responding studentsIdentifying non-responding studentsTreatment fidelity of instructionTreatment fidelity of instruction

Dual discrepancy modelDual discrepancy modelOne standard deviation from typically performing One standard deviation from typically performing

peers in level and peers in level and raterate

Page 11: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Ardoin & Christ, 2008Ardoin & Christ, 2008Slope for benchmarks (3x per year)Slope for benchmarks (3x per year)More growth from fall to winter than winter to More growth from fall to winter than winter to

springspringMight be helpful to use RoI for fall to winterMight be helpful to use RoI for fall to winterAnd a separate RoI for winter to springAnd a separate RoI for winter to spring

Page 12: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Fuchs, Fuchs, Walz, & Germann, 1993Fuchs, Fuchs, Walz, & Germann, 1993Typical weekly growth ratesTypical weekly growth ratesNeeded growthNeeded growth

1.5 to 2.0 times typical slope to close gap1.5 to 2.0 times typical slope to close gap

ExampleExampleBob is below benchmark on ORFBob is below benchmark on ORFTypical slope is 1 wcpm per week growthTypical slope is 1 wcpm per week growthBob would need slope of 1.5 to 2 to close gap Bob would need slope of 1.5 to 2 to close gap

in a reasonable amount of timein a reasonable amount of time

Page 13: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Deno, Fuchs, Marston, & Shin, 2001Deno, Fuchs, Marston, & Shin, 2001Slope of frequently non-responsive children Slope of frequently non-responsive children

approximated slope of children already approximated slope of children already identified as having a specific learning identified as having a specific learning disabilitydisability

Page 14: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Algebraic term: Slope of a lineAlgebraic term: Slope of a lineVertical change over the horizontal changeVertical change over the horizontal changeRise over runRise over runm = (ym = (y2 2 - y- y11) / (x) / (x2 2 - x- x11))Describes the steepness of a line (Gall & Gall, Describes the steepness of a line (Gall & Gall,

2007)2007)

Page 15: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Finding a student’s RoI = finding the slope Finding a student’s RoI = finding the slope of a lineof a lineUsing two data points on that lineUsing two data points on that line

Finding the line itselfFinding the line itselfLinear regressionLinear regressionOrdinary Least SquaresOrdinary Least Squares

Page 16: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Gall & Gall, 2007Gall & Gall, 200710 data points are a minimum requirement for 10 data points are a minimum requirement for

a reliable trendlinea reliable trendlineHow does that affect the frequency of How does that affect the frequency of

administering progress monitoring probes?administering progress monitoring probes?

Page 17: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Using RoI for instructional decisions is not Using RoI for instructional decisions is not a perfect processa perfect process

Research is currently looking to address Research is currently looking to address sources of error:sources of error:Christ, 2006 – standard error of measurement Christ, 2006 – standard error of measurement

for slopefor slopeArdoin & Christ, 2009 – passage difficulty and Ardoin & Christ, 2009 – passage difficulty and

variabilityvariabilityJenkin, Graff, & Miglioretti, 2009 – frequency Jenkin, Graff, & Miglioretti, 2009 – frequency

of progress monitoringof progress monitoring

Page 18: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI ResearchRoI Research

Questions yet to be empirically answeredQuestions yet to be empirically answeredWhat parameters of RoI indicate a lack of RtI?What parameters of RoI indicate a lack of RtI?How does standard error of measurement How does standard error of measurement

play into using RoI for instructional decision play into using RoI for instructional decision making?making?

How does RoI vary between standard protocol How does RoI vary between standard protocol interventions?interventions?

How does this apply to non-English speaking How does this apply to non-English speaking populations?populations?

Page 19: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

How is RoI Calculated? How is RoI Calculated? Which way is best?Which way is best?

Page 20: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Multiple Methods for Multiple Methods for Calculating GrowthCalculating Growth

““Eye ball” ApproachEye ball” Approach Last point minus First point ApproachLast point minus First point Approach Split Middle ApproachSplit Middle Approach Linear Regression ApproachLinear Regression Approach

Page 21: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Eye Ball

8

10

7

17

14

11

19

14

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8

Page 22: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Last minus First

8

10

7

17

14

11

19

14

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8

14 - 8 = 6; 6/ 8 weeks = 0.75 words per week

Page 23: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Split Middle

10.5

14.5

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10

14.5-10.5 = 4; 4/8 weeks = 0.5 words per week

Page 24: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Linear Regression

8

10

7

17

14

11

19

14

y = 1.1429x + 7.3571

0

2

4

6

8

10

12

14

16

18

20

1 2 3 4 5 6 7 8

1.1 Words Per Week

Page 25: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI Consistency?RoI Consistency?

Eye BallEye Ball ??????

Last minus FirstLast minus First 0.750.75

Split Middle*Split Middle* 0.500.50

Linear Linear RegressionRegression

1.101.10

Page 26: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI Consistency?RoI Consistency?

If we are not all using the same model to If we are not all using the same model to compute RoI, we continue to have the same compute RoI, we continue to have the same problems as past models, where under one problems as past models, where under one approach a student meets SLD criteria, but approach a student meets SLD criteria, but under a different approach, the student does under a different approach, the student does not. not.

Hypothetically, if the RoI cut-off was 0.65 or Hypothetically, if the RoI cut-off was 0.65 or 0.95, different approaches would come to 0.95, different approaches would come to different conclusions on the same student. different conclusions on the same student.

Page 27: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Technical AdequacyTechnical Adequacy

Without a consensus on how to compute Without a consensus on how to compute RoI, we risk falling short of having RoI, we risk falling short of having technical adequacy within our model. technical adequacy within our model.

Page 28: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

So, Which RoI Method is Best?So, Which RoI Method is Best?

Page 29: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Literature shows that Linear Literature shows that Linear Regression is Best PracticeRegression is Best Practice

Student’s daily test scores…were entered into a Student’s daily test scores…were entered into a computer program…The data analysis program computer program…The data analysis program generated generated slopes of improvementslopes of improvement for each level for each level using and using and Ordinary-Least SquaresOrdinary-Least Squares procedure procedure (Hayes, 1973) and the line of best fit. (Hayes, 1973) and the line of best fit.

This This procedure has been demonstrated to procedure has been demonstrated to represent CBM achievement data validly within represent CBM achievement data validly within individual treatment phasesindividual treatment phases (Marston, 1988; (Marston, 1988; Shinn, Good, & Stein, in press; Stein, 1987).Shinn, Good, & Stein, in press; Stein, 1987).

Shinn, Gleason, & Tindal, 1989Shinn, Gleason, & Tindal, 1989

Page 30: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Growth (RoI) Research Growth (RoI) Research using Linear Regressionusing Linear Regression

Christ, T. J. (2006). Short-term estimates of growth using Christ, T. J. (2006). Short-term estimates of growth using curriculum based measurement of oral reading fluency: curriculum based measurement of oral reading fluency: Estimating standard error of the slope to construct confidence Estimating standard error of the slope to construct confidence intervals. intervals. School Psychology ReviewSchool Psychology Review, 35, 128-133., 35, 128-133.

Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using Deno, S. L., Fuchs, L. S., Marston, D., & Shin, J. (2001). Using curriculum based measurement to establish growth standards curriculum based measurement to establish growth standards for students with learning disabilities. for students with learning disabilities. School Psychology School Psychology ReviewReview, 30, 507-524., 30, 507-524.

Good, R. H. (1990). Forecasting accuracy of slope estimates for Good, R. H. (1990). Forecasting accuracy of slope estimates for reading curriculum based measurement: Empirical evidence. reading curriculum based measurement: Empirical evidence. Behavioral AssessmentBehavioral Assessment, 12, 179-193., 12, 179-193.

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L. & Germann, G. (1993). Formative evaluation of academic progress: How much (1993). Formative evaluation of academic progress: How much growth can we expect? growth can we expect? School Psychology ReviewSchool Psychology Review, 22, 27-48., 22, 27-48.

Page 31: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Growth (RoI) ResearchGrowth (RoI) Researchusing Linear Regressionusing Linear Regression

Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). Estimating reading growth using intermittent CBM Estimating reading growth using intermittent CBM progress monitoring. progress monitoring. Exceptional ChildrenExceptional Children, 75, 151-163., 75, 151-163.

Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Shinn, M. R., Gleason, M. M., & Tindal, G. (1989). Varying the difficulty of testing materials: Implications for Varying the difficulty of testing materials: Implications for curriculum-based measurement. curriculum-based measurement. The Journal of Special The Journal of Special EducationEducation, 23, 223-233., 23, 223-233.

Shinn, M. R., Good, R. H., & Stein, S. (1989). Shinn, M. R., Good, R. H., & Stein, S. (1989). Summarizing trend in student achievement: A Summarizing trend in student achievement: A comparison of methods. comparison of methods. School Psychology ReviewSchool Psychology Review, 18, , 18, 356-370.356-370.

Page 32: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

So, Why Are There So Many So, Why Are There So Many Other RoI Models?Other RoI Models?

Ease of application Ease of application How many of us want to calculate OLS How many of us want to calculate OLS

Linear Regression formulas (or even Linear Regression formulas (or even remember how)?remember how)?

Page 33: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

An Easy and An Easy and Applicable SolutionApplicable Solution

Page 34: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Get Out Your Laptops!Get Out Your Laptops!

Or Kindly Look Over Your Or Kindly Look Over Your Neighbor’s Shoulder!Neighbor’s Shoulder!

I loveROI

Page 35: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Open Microsoft ExcelOpen Microsoft Excel

Microsoft Office 2003 for PCsMicrosoft Office 2003 for PCsMicrosoft Office 2007 for PCsMicrosoft Office 2007 for PCsMicrosoft Office for MacsMicrosoft Office for Macs

Page 36: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Graphing RoIGraphing RoI

For Individual StudentsFor Individual Students

Page 37: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Setting Up Your SpreadsheetSetting Up Your Spreadsheet

In cell B2, type In cell B2, type School WeekSchool Week In cell C2, type In cell C2, type BenchmarkBenchmark In cell D2, type WPM (or Student Scores)In cell D2, type WPM (or Student Scores)

Page 38: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Labeling School WeeksLabeling School Weeks

In cell B3, type In cell B3, type 11 Continue entering numbersContinue entering numbers

through through 3636 in column B in column BWeek Week 3636 will be in cell B38 will be in cell B38

Page 39: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Entering BenchmarksEntering Benchmarks

In cell C3, type the fall benchmark In cell C3, type the fall benchmark 7777 In cell C20, type the winter benchmark In cell C20, type the winter benchmark 9292 In cell C38, type the spring benchmark In cell C38, type the spring benchmark

110110

Page 40: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Entering Student Data PointsEntering Student Data Points

Student data points are entered between Student data points are entered between cells D3 and D38.cells D3 and D38.

Type the student’s score next to the Type the student’s score next to the corresponding week that it was corresponding week that it was administered.administered.

Page 41: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Entering Student Data PointsEntering Student Data Points

Week 1 – 41Week 1 – 41Week 8 – 62Week 8 – 62Week 9 – 63Week 9 – 63Week 10 – 75Week 10 – 75Week 11 – 64Week 11 – 64Week 12 – 80Week 12 – 80Week 13 – 83Week 13 – 83Week 14 - 83Week 14 - 83

Page 42: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Entering Student Data PointsEntering Student Data Points

Week 15 – 56Week 15 – 56Week 17 – 104Week 17 – 104Week 18 – 74Week 18 – 74Week 20 – 85Week 20 – 85Week 21 – 89Week 21 – 89Week 22 – 69Week 22 – 69Week 23 – 85Week 23 – 85

Page 43: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Entering Student Data PointsEntering Student Data Points

Week 24 – 96Week 24 – 96Week 25 – 90Week 25 – 90Week 26 – 84Week 26 – 84Week 27 – 106Week 27 – 106Week 28 – 94Week 28 – 94Week 32 – 100Week 32 – 100

Page 44: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

*CAUTION**CAUTION*

If a student was not assessed during a If a student was not assessed during a certain week, leave that cell blankcertain week, leave that cell blank

Do not enter a score of Zero (0) it will be Do not enter a score of Zero (0) it will be calculated into the trendline and calculated into the trendline and interpreted as the student having read interpreted as the student having read zero words correct per minute during that zero words correct per minute during that week. week.

Page 45: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Highlight the Highlight the data in Columnsdata in ColumnsC and DC and D Include cells Include cells C2 and D2 through C2 and D2 through cells C38 and D38cells C38 and D38 Include any Include any blank cellsblank cells

Page 46: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Click Click InsertInsert Click Click ChartChart

Excel 2007Excel 2007 Click Click InsertInsert Find the icon for Find the icon for LineLine Click the arrow below Click the arrow below

LineLine

Page 47: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs A Chart Wizard A Chart Wizard

window will appearwindow will appear

Excel 2007Excel 2007 6 graphics appear6 graphics appear

Page 48: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Choose Choose LineLine Choose Choose Line with Line with

markersmarkers

Excel 2007Excel 2007 Choose Choose Line with Line with

markersmarkers

Page 49: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Data RangeData Range tab tab ColumnsColumns

Excel 2007Excel 2007 Your graph appearsYour graph appears

Page 50: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Chart TitleChart Title School WeekSchool Week (X Axis) (X Axis) WPMWPM (Y Axis) (Y Axis)

Excel 2007Excel 2007 Change your labels by Change your labels by

clicking on the graphclicking on the graph

Page 51: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Choose where you Choose where you

want your graphwant your graph

Excel 2007Excel 2007 Your graph was Your graph was

automatically put into automatically put into your data spreadsheetyour data spreadsheet

Page 52: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Creating a GraphCreating a Graph

Excel 2003/MacsExcel 2003/Macs Excel 2007Excel 2007

Page 53: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Adding a TrendlineAdding a Trendline

Excel 2003/MacsExcel 2003/MacsRight click on any of the student data pointsRight click on any of the student data points

Excel 2007Excel 2007

Page 54: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Adding a TrendlineAdding a Trendline

Excel 2003/MacsExcel 2003/MacsChoose Choose LinearLinear

Excel 2007Excel 2007

Page 55: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Adding a TrendlineAdding a Trendline

Excel 2003/MacsExcel 2003/MacsChoose Choose CustomCustom and check box next to and check box next to

Display equation on chartDisplay equation on chart

Excel 2007Excel 2007

Page 56: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Adding a TrendlineAdding a Trendline

Clicking on the equation highlights a box Clicking on the equation highlights a box around itaround it

Clicking on the box allows you to move it Clicking on the box allows you to move it to a place where you can see it betterto a place where you can see it better

Page 57: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Adding a TrendlineAdding a Trendline

You can repeat the same procedure to You can repeat the same procedure to have a trendline for the benchmark data have a trendline for the benchmark data pointspoints

Suggestion: label the trendline Suggestion: label the trendline Expected Expected ROIROI

Move this equation under the firstMove this equation under the first

Page 58: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Student GraphIndividual Student Graph

Diego's Rate of Improvement

y = 1.6317x + 50.928

y = 0.9434x + 75.704

0

20

40

60

80

100

120

1 4 7 10 13 16 19 22 25 28 31 34

School Week

Wo

rds

Per

Min

ute

Benchmark (3rd)

Diego's Scores (3rd)

Diego's ROI

Goal Slope

Page 59: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Student GraphIndividual Student Graph

The equation indicates the slope, or rate of The equation indicates the slope, or rate of improvement. improvement.

The number, or coefficient,The number, or coefficient, beforebefore "x" is "x" is the average improvement, which in this the average improvement, which in this case is the average number of words per case is the average number of words per minute per week gained by the student. minute per week gained by the student.

Page 60: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Student GraphIndividual Student Graph

The rate of improvement, or trendline, is The rate of improvement, or trendline, is calculated using a linear regression, a calculated using a linear regression, a simple equation of least squares. simple equation of least squares.

To add additional progress To add additional progress monitoring/benchmark scores once you’ve monitoring/benchmark scores once you’ve already created a graph, enter additional already created a graph, enter additional scores in Column D in the corresponding scores in Column D in the corresponding school week.school week.

Page 61: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Student GraphIndividual Student Graph

Remember to leave cells blank for the Remember to leave cells blank for the weeks that no score was obtained. weeks that no score was obtained.

The graph will incorporate that score into The graph will incorporate that score into the set of data points and into the the set of data points and into the trendline.trendline.

Page 62: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Student GraphIndividual Student Graph

The slope can change depending on The slope can change depending on which week (where) you put the which week (where) you put the benchmark scores on your chart. benchmark scores on your chart.

Enter benchmark scores based on when Enter benchmark scores based on when your school administers their benchmark your school administers their benchmark assessments for the most accurate assessments for the most accurate depiction of expected student progress.depiction of expected student progress.

Page 63: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Options for the GraphOptions for the Graph

ResizingResizingColoringColoringData LabelsData Labels

Page 64: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Programming ExcelProgramming Excel

To Calculate RoITo Calculate RoI

A FormulaA Formula

Page 65: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI FormulaRoI Formula

Type Type RoIRoI in cell B39 below the last week in cell B39 below the last week of schoolof school

Page 66: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Calculate Expected SlopeCalculate Expected Slope

Click on cell C39Click on cell C39Put your cursor at the top next to the Put your cursor at the top next to the fxfxType =SLOPE(C3:C38,B3:B38)Type =SLOPE(C3:C38,B3:B38)Hit Enter/ReturnHit Enter/Return

Page 67: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Calculate Actual SlopeCalculate Actual Slope

Click on cell D39Click on cell D39Put your cursor at the top next to the Put your cursor at the top next to the fxfxType =SLOPE(D3:D38,B3:B38)Type =SLOPE(D3:D38,B3:B38)Hit Enter/ReturnHit Enter/Return

Page 68: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ROI as a Decision ToolROI as a Decision Tool

within a Problem-Solving Modelwithin a Problem-Solving Model

Page 69: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

StepsSteps

1.1. Gather the dataGather the data

2.2. Ground the dataGround the data

3.3. Interpret the dataInterpret the data

4.4. Figure out how to fit Best Practice into Figure out how to fit Best Practice into Public EducationPublic Education

Page 70: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Step 1: Gather DataStep 1: Gather Data

Universal Screening Universal Screening

Progress MonitoringProgress Monitoring

Page 71: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Common Screenings in PACommon Screenings in PA

DIBELSDIBELSAIMSwebAIMSwebMBSPMBSP4Sight4SightPSSAPSSA

Page 72: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Validated Progress Validated Progress Monitoring ToolsMonitoring Tools

DIBELSDIBELSAIMSwebAIMSwebMBSPMBSPwww.studentprogress.orgwww.studentprogress.org

Page 73: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Step 2: Ground the DataStep 2: Ground the Data

To what will we compare our To what will we compare our student growth data?student growth data?

Page 74: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Multiple Ways to Multiple Ways to Look at GrowthLook at Growth

Needed Growth Needed Growth Expected Growth & Percent of Expected Expected Growth & Percent of Expected

GrowthGrowthFuchs et. al. (1993) Table of Realistic and Fuchs et. al. (1993) Table of Realistic and

Ambitious GrowthAmbitious GrowthGrowth Toward Individual Goal*Growth Toward Individual Goal*

*Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)*Best Practices in Setting Progress Monitoring Goals for Academic Skill Improvement (Shapiro, 2008)

Page 75: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Needed GrowthNeeded Growth

Difference between student’s BOY (or Difference between student’s BOY (or MOY) score and benchmark score at MOY MOY) score and benchmark score at MOY (or EOY).(or EOY).

Example: MOY ORF = 10, EOY Example: MOY ORF = 10, EOY benchmark is 40, 18 weeks of instruction benchmark is 40, 18 weeks of instruction (40-10/18=1.67). Student must gain 1.67 (40-10/18=1.67). Student must gain 1.67 wcpm per week to make EOY benchmark.wcpm per week to make EOY benchmark.

Page 76: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Expected GrowthExpected Growth

Difference between two benchmarks.Difference between two benchmarks.Example: MOY benchmark is 20, EOY Example: MOY benchmark is 20, EOY

benchmark is 40, expected growth (40-benchmark is 40, expected growth (40-20)/18 weeks of instruction = 1.11 wcpm 20)/18 weeks of instruction = 1.11 wcpm per week.per week.

Page 77: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Tigard-Tualatin School District (www.ttsd.k12.or.us)Tigard-Tualatin School District (www.ttsd.k12.or.us)

Looking at Percent of Looking at Percent of Expected GrowthExpected Growth

Tier ITier I Tier IITier II Tier IIITier III

Greater than Greater than 150%150%

Between Between 110% & 150%110% & 150%

Possible LDPossible LD

Between 95% Between 95% & 110%& 110%

Likely LDLikely LD

Between 80% Between 80% & 95%& 95%

May Need May Need MoreMore

May Need May Need MoreMore

Likely LDLikely LD

Below 80%Below 80% Needs MoreNeeds More Needs MoreNeeds More Likely LDLikely LD

Page 78: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)

Oral Reading Fluency Adequate Oral Reading Fluency Adequate Response TableResponse Table

Realistic Realistic GrowthGrowth

Ambitious Ambitious GrowthGrowth

11stst 2.02.0 3.03.0

22ndnd 1.51.5 2.02.0

33rdrd 1.01.0 1.51.5

44thth 0.90.9 1.11.1

55thth 0.50.5 0.80.8

Page 79: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)Fuchs, Fuchs, Hamlett, Walz, & Germann (1993)

Digit Fluency Adequate Digit Fluency Adequate Response TableResponse Table

Realistic Realistic GrowthGrowth

Ambitious Ambitious GrowthGrowth

11stst 0.30.3 0.50.5

22ndnd 0.30.3 0.50.5

33rdrd 0.30.3 0.50.5

44thth 0.750.75 1.21.2

55thth 0.750.75 1.21.2

Page 80: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Making Decisions: Best Practice Making Decisions: Best Practice

Research has yet to establish a blue print Research has yet to establish a blue print for ‘grounding’ student RoI data. for ‘grounding’ student RoI data.

At this point, teams should consider At this point, teams should consider multiple comparisons when planning and multiple comparisons when planning and making decisions. making decisions.

Page 81: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Making Decisions: Lessons Making Decisions: Lessons From the FieldFrom the Field

When tracking on grade level, consider an When tracking on grade level, consider an RoI that is 100% of RoI that is 100% of expectedexpected growth as a growth as a minimum requirement, consider an RoI minimum requirement, consider an RoI that is at or above the that is at or above the neededneeded as optimal. as optimal.

So, 100% of expected and on par with So, 100% of expected and on par with needed become the limits of the range needed become the limits of the range within a student should be achieving.within a student should be achieving.

Page 82: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Oral Reading Fluency01/15/09 01/22/09 01/29/09 02/05/09 02/12/09 02/19/09 02/26/09 03/05/09 03/12/09 03/19/09 03/26/09 04/02/09 04/09/09 04/16/09 04/23/09 04/30/09 05/07/09 05/14/09

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Benchmark 68 90 1.29

Aiden 61 40 52 60 71 95 1.61 2.17 167%

Ava 49 43 49 77 57 54 87 92 2.28 2.76 213%

Noah 49 48 45 69 61 54 84 2.28 2.01 156%

Olivia 65 49 57 70 79 83 1.39 1.50 116%

Liam 55 53 36 54 70 83 1.94 1.58 122%

Hannah 59 54 64 69 52 60 82 1.72 1.20 93%

Gavin 64 40 67 68 84 79 1.44 1.66 129%

Grace 53 48 46 60 74 79 2.06 1.76 136%

Oliver 50 44 46 68 51 51 57 78 2.22 1.45 112%

Peyton 63 50 47 58 75 77 1.50 1.12 87%

Josh 49 38 49 55 48 36 67 77 2.28 1.62 125%

Riley 42 49 54 69 67 50 76 2.67 1.76 136%

Mason 53 53 50 64 60 74 2.06 1.17 91%

Zoe 34 38 42 68 55 51 58 3.11 1.44 111%

Ian 41 31 45 49 47 30 46 2.72 0.24 19%

Faith 29 36 35 36 36 29 45 44 3.39 0.75 58%

David 30 23 44 52 43 19 63 38 3.33 0.79 61%

Alexa 18 19 25 33 33 23 28 37 4.00 0.94 73%

Hunter 23 23 24 48 38 32 34 3.72 0.75 58%

Caroline 28 20 28 40 37 19 25 30 3.44 0.02 2%

** Actual RoI based on linear regression of all data points

Benchmarks based on DIBELS Goals

Expected RoI at Benchmark Level

(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)

% of Expected RoI

Needed RoI* Actual RoI**

Realistic Grow thAmbitious Grow th

Oral Reading Fluency Adequate Response Table

1st Grade

2nd Grade

* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks

0.5

3rd Grade

4th Grade

5th Grade

0.9

0.8

2.0

1.5

1.0

3.0

2.0

1.5

1.1

Page 83: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Step 3: Interpreting GrowthStep 3: Interpreting Growth

Page 84: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

What do we do when we do not What do we do when we do not get the growth we want?get the growth we want?

When to make a change in instruction and When to make a change in instruction and intervention?intervention?

When to consider SLD?When to consider SLD?

Page 85: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

When to make a change in When to make a change in instruction and intervention?instruction and intervention?

Enough data points (6 to 10)?Enough data points (6 to 10)?Less than 100% of expected growth.Less than 100% of expected growth.Not on track to make benchmark (needed Not on track to make benchmark (needed

growth).growth).Not on track to reach individual goal.Not on track to reach individual goal.

Page 86: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

When to consider SLD?When to consider SLD?

Continued inadequate response despite: Continued inadequate response despite: Fidelity with Tier I instruction and Tier Fidelity with Tier I instruction and Tier

II/III intervention.II/III intervention. Multiple attempts at intervention.Multiple attempts at intervention. Individualized Problem-Solving Individualized Problem-Solving

approach. approach.

Page 87: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Three Levels of ExamplesThree Levels of Examples

Whole ClassWhole ClassSmall GroupSmall Group Individual Student Individual Student

- Academic Data- Academic Data

- Behavior Data- Behavior Data

Page 88: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Whole Class ExampleWhole Class ExampleComputation

01/15/10 01/22/10 01/29/10 02/05/10 02/12/10 02/19/10 02/26/10 03/05/10 03/12/10 03/19/10 03/26/10 04/02/10 04/09/10 04/16/10 04/23/10 04/30/10 05/07/10 05/14/10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

50th P ercentile 25 31 0.35

25th P ercentile 19 23 0.24

Student 6.5 9 8 8.5 5.5 11 13 1.72 0.61 173%

Student 6 7.5 8.5 5 11 11.5 1.72 0.57 161%

Student 4.5 5.5 6.5 9.5 10.5 1.72 1.06 300%

Student 13 8 9.3 8 5.6 9.6 9.6 1.72 -0.23 -66%

Student 8 10.5 10.5 5.6 9.3 9 1.72 -0.03 -7%

Student 8.5 5.5 9 8 4 8 9 1.72 0.07 21%

Student 6.5 5.5 6 10.5 9 1.72 0.43 122%

Student 6.5 9 4.5 6 8 1.72 0.07 20%

Student 8 10.5 4.5 6.5 4 7 1.72 -0.25 -71%

Student 9 10 5.6 6.6 5 4.6 6.6 1.72 -0.42 -119%

Student 8 8 8.5 4 8 6.6 1.72 -0.18 -51%

Student 9 4.5 4.5 4 3.5 3.5 6.5 1.72 -0.24 -67%

Student 6.5 5 6.5 9 7.5 6.5 1.72 0.09 26%

Student 5.5 3 8 4 6.5 6.3 1.72 0.19 55%

Student 7.5 10 6.6 3.3 3 6.3 1.72 -0.46 -130%

Student 5 5.5 6.5 6 5 6 1.72 0.04 11%

Student 5 4 8 8.5 10 8 6 1.72 0.25 71%

Student 4.5 6 5 2.5 3.5 5.5 1.72 -0.03 -8%

Student 5.5 4.5 10.5 11 5 5.3 1.72 -0.14 -40%

* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks

** Actual RoI based on linear regression of all data points

Percentiles based on AIMSw eb Grow th Tables

Expected RoI at 50th PercentileExpected RoI at 25th Percentile

Needed RoI* Actual RoI** % of Expected RoI

Digit Fluency Adequate Response Table

Realistic Grow thAmbitious Grow th

1st Grade 0.3 0.5

2nd Grade 0.3 0.5

3rd Grade 0.3 0.5

(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)

4th Grade 0.75 1.2

5th Grade 0.75 1.2

Page 89: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

33rdrd Grade Math Whole Class Grade Math Whole Class

Who’s responding? Who’s responding? Effective math Effective math

instruction? instruction? Who needs more?Who needs more?

N=19N=194 > 100% growth 4 > 100% growth 15 < 100% growth15 < 100% growth9 w/ negative 9 w/ negative

growthgrowth

Page 90: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Small Group ExampleSmall Group ExampleOral Reading Fluency

09/11/09 09/18/09 09/25/09 10/02/09 10/09/09 10/16/09 10/23/09 10/30/09 11/06/09 11/13/09 11/20/09 11/27/09 12/04/09 12/11/09 12/18/09 01/01/10 01/08/10 01/15/10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Benchmark 44 68 1.41

Student 35 39 41 45 42 45 52 57 62 1.83 1.49 106%

Student 28 38 42 40 50 55 64 72 74 2.22 2.77 196%

Student 26 28 32 31 27 29 35 34 38 2.33 0.57 41%

Student 31 35 39 45 42 47 53 58 65 2.06 1.90 135%

Student 40 44 38 48 52 64 72 74 78 1.56 2.62 186%

** Actual RoI based on linear regression of all data points

Benchmarks based on DIBELS Goals

Expected RoI at Benchmark Level

(Fuchs, Fuchs, Hamlett, Walz, & Germann 1993)

5th Grade

* Needed RoI based on difference betw een w eek 1 score and Benchmark score for w eek 18 divided by 18 w eeks

% of Expected RoI

2nd Grade

0.9 1.1

3rd Grade

4th Grade

0.5 0.8

1.5 2.0

1.0 1.5

Needed RoI* Actual RoI**

2.0 3.0

Oral Reading Fluency Adequte Response Table

1st Grade

Realistic Grow thAmbitious Grow th

Page 91: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Intervention GroupIntervention Group

Intervention working for how many?Intervention working for how many?Can we assume fidelity of intervention Can we assume fidelity of intervention

based on results?based on results?Who needs more?Who needs more?

Page 92: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual Kid ExampleIndividual Kid Example 2nd Grade Reading Progress

44

68

90

31

56

45

53 53

4846

60

74

79

y = 1.5333x + 42.8

y = 0.9903x + 36.873

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

09/12/08 09/19/0809/26/0810/03/08 10/10/08 10/17/08 10/24/08 10/31/08 11/07/08 11/14/08 11/21/08 11/28/08 12/05/08 12/12/08 12/19/08 01/16/09 01/23/0901/30/0902/06/0902/13/0902/20/0902/27/0903/06/0903/13/0903/20/0903/27/0904/03/0904/10/0904/17/0904/24/0905/01/09

Wo

rds

Re

ad

Co

rre

ct

Pe

r M

inu

te

Benchmark Linear (Benchmark) Linear

Page 93: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Individual KidIndividual Kid

Making growth?Making growth?How much (65% of expected growth).How much (65% of expected growth).Atypical growth across the year (last 3 Atypical growth across the year (last 3

data points). data points). Continue? Make a change? Need more Continue? Make a change? Need more

data?data?

Page 94: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

RoI and Behavior?RoI and Behavior?

             

                                                

Page 95: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Percent of Time Engaged in Appropriate Behavior

y = 2x + 22

y = 3.9x + 19.8

y = 7.2143x - 1.5

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Per

cen

t

Baseline Condition 1 Condition 2 Linear (Baseline) Linear (Condition 1) Linear (Condition 2) Linear (Condition 2)

Page 96: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Step 4: Figure out how to fit Step 4: Figure out how to fit Best Practice into Public Best Practice into Public

EducationEducation

Page 97: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Things to ConsiderThings to Consider

Who is At-Risk and needs progress Who is At-Risk and needs progress monitoring?monitoring?

Who will collect, score, enter the data?Who will collect, score, enter the data?Who will monitor student growth, when, Who will monitor student growth, when,

and how often?and how often?What changes should be made to What changes should be made to

instruction & intervention? instruction & intervention? What about monitoring off of grade level?What about monitoring off of grade level?

Page 98: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Who is At-Risk and needs Who is At-Risk and needs progress monitoring?progress monitoring?

Below level on universal screeningBelow level on universal screening

Entering 4Entering 4thth Grade Example Grade Example

DORF DORF (110)(110)

ISIP ISIP TRWM TRWM

(55)(55)

4Sight 4Sight (1235)(1235)

PSSA PSSA (1235)(1235)

Student AStudent A 115115 5858 12551255 12321232

Student BStudent B 8585 4848 12161216 11261126

Student CStudent C 7272 3535 10561056 10481048

Page 99: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Who will collect, score, and Who will collect, score, and enter the data?enter the data?

Using MBSP for math, teachers can Using MBSP for math, teachers can administer probes to whole class.administer probes to whole class.

DORF probes must be administered one-DORF probes must be administered one-on-one, and creativity pays off (train and on-one, and creativity pays off (train and use art, music, library, etc. specialists).use art, music, library, etc. specialists).

Schedule for progress monitoring math Schedule for progress monitoring math and reading every-other week.and reading every-other week.

Page 100: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Week 1Week 1 Week 2Week 2

ReadingReading MathMath ReadingReading MathMath

11stst XX XX

22ndnd XX XX

33rdrd XX XX

44thth XX XX

55thth XX XX

Page 101: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Who will monitor student Who will monitor student growth, when, and how often?growth, when, and how often?

Best Practices in Data-Analysis Teaming Best Practices in Data-Analysis Teaming (Kovaleski & Pedersen, 2008)(Kovaleski & Pedersen, 2008)

Chambersburg Area School District Elementary Chambersburg Area School District Elementary Response to Intervention Manual (McCrea et. Response to Intervention Manual (McCrea et. al., 2008)al., 2008)

Derry Township School District Response to Derry Township School District Response to Intervention Model Intervention Model (http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-(http://www.hershey.k12.pa.us/56039310111408/lib/56039310111408/_files/Microsoft_Word_-_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)_Response_to_Intervention_Overview_of_Hershey_Elementary_Model.pdf)

Page 102: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

What changes should be made What changes should be made to instruction & intervention?to instruction & intervention?

Ensure treatment fidelity!!!!!!!!Ensure treatment fidelity!!!!!!!! Increase instructional time (active and Increase instructional time (active and

engaged)engaged)Decrease group sizeDecrease group sizeGather additional, diagnostic, informationGather additional, diagnostic, informationChange the intervention Change the intervention

Page 103: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

When Instructional Level is Not When Instructional Level is Not the Same as Grade Levelthe Same as Grade Level

Understand needed and expected RoI Understand needed and expected RoI within broader context: within broader context:

Needed growthNeeded growth will only get student will only get student to nextto next level by next benchmark (as opposed to level by next benchmark (as opposed to on on levellevel).).

100% of expected growth100% of expected growth may not be an may not be an acceptable minimum (not enough growth b/c acceptable minimum (not enough growth b/c level is so low).level is so low).

Page 104: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Grounding RoI When Monitoring Grounding RoI When Monitoring Off of Grade Level: Two OptionsOff of Grade Level: Two OptionsBest Practices in Setting Progress Best Practices in Setting Progress

Monitoring Goals for Academic Skill Monitoring Goals for Academic Skill Improvement (Shapiro, 2008).Improvement (Shapiro, 2008).

Tigard-Tualatin SD Chart: 150% instead of Tigard-Tualatin SD Chart: 150% instead of 100% as minimum RoI requirement???100% as minimum RoI requirement???

Page 105: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Questions? & Comments!Questions? & Comments!

Page 106: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ResourcesResources

www.interventioncentral.comwww.interventioncentral.com

www.aimsweb.comwww.aimsweb.com

http://dibels.uoregon.eduhttp://dibels.uoregon.edu

www.nasponline.orgwww.nasponline.org

Page 107: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ResourcesResources

www.fcrr.orgwww.fcrr.org

Florida Center for Reading ResearchFlorida Center for Reading Researchhttp://ies.ed.gov/ncee/wwc//http://ies.ed.gov/ncee/wwc//

What Works ClearinghouseWhat Works Clearinghousehttp://www.rti4success.orghttp://www.rti4success.org

National Center on RtINational Center on RtI

Page 108: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

Flinn & McCrea’s RoI Web SiteFlinn & McCrea’s RoI Web Site

http://sites.google.com/site/http://sites.google.com/site/rateofimprovement/rateofimprovement/Download powerpoints, handouts, Excel Download powerpoints, handouts, Excel

graphs, charts, articles, etc.graphs, charts, articles, etc.Caitlin FlinnCaitlin Flinn

[email protected] [email protected] Andrew McCreaAndrew McCrea

[email protected] [email protected]

Page 109: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Ardoin, S. P., & Christ, T. J. (2009). Curriculum-Ardoin, S. P., & Christ, T. J. (2009). Curriculum-based measurement of oral reading: Standard based measurement of oral reading: Standard errors associated with progress monitoring errors associated with progress monitoring outcomes from DIBELS, AIMSweb, and an outcomes from DIBELS, AIMSweb, and an experimental passage set. experimental passage set. School Psychology School Psychology Review, 38(2),Review, 38(2), 266-283. 266-283.

Ardoin, S. P. & Christ, T. J. (2008). Evaluating Ardoin, S. P. & Christ, T. J. (2008). Evaluating curriculum-based measurement slope estimates curriculum-based measurement slope estimates using triannual universal screenings. using triannual universal screenings. School School Psychology Review, 37Psychology Review, 37(1), 109-125. (1), 109-125.

Page 110: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Christ, T. J. (2006). Short-term estimates of Christ, T. J. (2006). Short-term estimates of growth using curriculum-based measurement growth using curriculum-based measurement of oral reading fluency: Estimating standard of oral reading fluency: Estimating standard error of the slope to construct confidence error of the slope to construct confidence intervals. intervals. School Psychology Review, 35School Psychology Review, 35(1), (1), 128-133.128-133.

Deno, S. L. (1985). Curriculum-based Deno, S. L. (1985). Curriculum-based measurement: The emerging alternative. measurement: The emerging alternative. Exceptional Children, 52,Exceptional Children, 52, 219-232. 219-232.

Page 111: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Deno, S. L., Fuchs, L.S., Marston, D., & Deno, S. L., Fuchs, L.S., Marston, D., & Shin, J. (2001). Using curriculum-based Shin, J. (2001). Using curriculum-based measurement to establish growth measurement to establish growth standards for students with learning standards for students with learning disabilities. disabilities. School Psychology Review, School Psychology Review, 3030, 507-524., 507-524.

Flinn, C. S. (2008). Graphing rate of Flinn, C. S. (2008). Graphing rate of improvement for individual students. improvement for individual students. InSight, 28(3InSight, 28(3), 10-12.), 10-12.

Page 112: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Fuchs, L. S., & Fuchs, D. (1998). Treatment Fuchs, L. S., & Fuchs, D. (1998). Treatment validity: A unifying concept for reconceptualizing validity: A unifying concept for reconceptualizing the identification of learning disabilities. the identification of learning disabilities. Learning Learning Disabilities Research and Practice, 13Disabilities Research and Practice, 13, 204-219., 204-219.

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Fuchs, L. S., Fuchs, D., Hamlett, C. L., Walz, L., & Germann, G. (1993). Formative evaluation of Germann, G. (1993). Formative evaluation of academic progress: How much growth can we academic progress: How much growth can we expect? expect? School Psychology Review, 22School Psychology Review, 22, 27-48., 27-48.

Page 113: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Gall, M.D., & Gall, J.P. (2007). Gall, M.D., & Gall, J.P. (2007). Educational Educational research: An introductionresearch: An introduction (8th ed.). New (8th ed.). New York: Pearson.York: Pearson.

Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. Jenkins, J. R., Graff, J. J., & Miglioretti, D.L. (2009). (2009). Estimating reading growth using Estimating reading growth using intermittent CBM progress monitoring. intermittent CBM progress monitoring. Exceptional Children, 75Exceptional Children, 75, 151-163., 151-163.

Page 114: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Karwowski, W. (2006). Karwowski, W. (2006). International International encyclopedia of ergonomics and human encyclopedia of ergonomics and human factorsfactors. Boca Raton, FL: Taylor & Francis . Boca Raton, FL: Taylor & Francis Group, LLC.Group, LLC.

Shapiro, E. S. (2008). Best practices in setting Shapiro, E. S. (2008). Best practices in setting progress monitoring goals for academic skill progress monitoring goals for academic skill improvement. In A. Thomas and J. Grimes improvement. In A. Thomas and J. Grimes (Eds.), (Eds.), Best practices in school psychology V Best practices in school psychology V (Vol. 2, pp. 141-157). Bethesda, MD: National (Vol. 2, pp. 141-157). Bethesda, MD: National Association of School Psychologists.Association of School Psychologists.

Page 115: Graphing, Calculating, and Interpreting Rate of Improvement Caitlin S. Flinn, Ed.S., N.C.S.P. Andrew E. McCrea, M.S., N.C.S.P. NASP Convention March 3,

ReferencesReferences

Vogel, D. R., Dickson, G. W., & Lehman, J. Vogel, D. R., Dickson, G. W., & Lehman, J. A. (1990). Persuasion and the role of A. (1990). Persuasion and the role of visual presentation support. The UM/3M visual presentation support. The UM/3M study. In M. Antonoff (Ed.), study. In M. Antonoff (Ed.), Presentations Presentations that persuade. Personal Computing, 14that persuade. Personal Computing, 14..