Reading First Evaluation in Georgia: A Multidimensional Approach Ken Proctor Reading First Director Georgia Department of Education Michael C. McKenna.

Reading First Reading First Evaluation in Evaluation in Georgia: A Georgia: A Multidimensional Multidimensional ApproachApproach

Ken ProctorKen Proctor

Reading First DirectorReading First Director

Georgia Department of EducationGeorgia Department of Education

Michael C. McKennaMichael C. McKenna

University of VirginiaUniversity of Virginia

Overview of External Overview of External EvaluationEvaluation University of Georgia, College of University of Georgia, College of

EducationEducation Reading Education DepartmentReading Education Department Educational Research Lab Test Scoring Educational Research Lab Test Scoring

and Reporting Servicesand Reporting Services Occupational Research GroupOccupational Research Group

Implementation, Impact and Implementation, Impact and ProgressProgress

April 2004 – June 2007 April 2004 – June 2007 (3 full years (3 full years of program implementation)of program implementation)

Key UGA Personnel for Key UGA Personnel for External EvaluationExternal Evaluation Reading Education DepartmentReading Education Department

Michelle Commeyras, Professor, Reading Michelle Commeyras, Professor, Reading EducationEducation

Donna Alvermann, Distinguished Research Donna Alvermann, Distinguished Research Professor, Reading EducationProfessor, Reading Education

Doctoral level reading ed students/field Doctoral level reading ed students/field researchersresearchers

TSARS-Education Research Lab (number TSARS-Education Research Lab (number crunching)crunching)

Steve Cramer & Allan Cohen – Professors, Steve Cramer & Allan Cohen – Professors, Educational PsychologyEducational Psychology

Occupational Research Group Occupational Research Group Dorothy Harnish, Assoc. Research Scientist, Dorothy Harnish, Assoc. Research Scientist,

Project DirectorProject Director

Annual State Annual State ReportingReporting Has the state made progress in Has the state made progress in

increasing the percentage of increasing the percentage of students reading at grade level or students reading at grade level or above?above?

Which RF schools have made the Which RF schools have made the largest gains in student reading largest gains in student reading achievement?achievement?

Annual State Annual State ReportingReporting Disaggregated data for students reading at Disaggregated data for students reading at

or above grade level:or above grade level: Economically disadvantaged studentsEconomically disadvantaged students Students from major racial and ethnic groupsStudents from major racial and ethnic groups Students with disabilitiesStudents with disabilities Students with limited English proficiencyStudents with limited English proficiency

Student performance data for percentage of Student performance data for percentage of students in RF schools in grades 1, 2, and 3 students in RF schools in grades 1, 2, and 3 reading at or above grade levelreading at or above grade level

Evaluation QuestionsEvaluation Questions

ImpactImpact ImplementationImplementationProgressProgress

Evaluation Questions: Evaluation Questions: ImpactImpact What is the impact of Reading What is the impact of Reading

First (RF) on student achievement First (RF) on student achievement in reading as measured by in reading as measured by standardized test scores? Is standardized test scores? Is reading achievement in RF reading achievement in RF schools higher than in non-RF schools higher than in non-RF schools?schools?

Evaluation Questions: Evaluation Questions: ImplementationImplementation Is the RF program being implemented Is the RF program being implemented

by schools as intended in the Georgia by schools as intended in the Georgia RF plan? How does the level of RF plan? How does the level of implementation of RF relate to results implementation of RF relate to results being achieved in RF schools? Is the being achieved in RF schools? Is the level of RF implementation positively level of RF implementation positively correlated with higher reading correlated with higher reading achievement? achievement?

Evaluation Questions: Evaluation Questions: ProgressProgress What progress is being made by RF What progress is being made by RF

schools in improving student reading schools in improving student reading achievement? Where progress is not achievement? Where progress is not apparent, what are the reasons for apparent, what are the reasons for this? What interventions are required?this? What interventions are required?

Are RF teachers more knowledgeable Are RF teachers more knowledgeable of scientifically based reading research of scientifically based reading research after three years of professional after three years of professional learning experiences?learning experiences?

Impact of RF on Impact of RF on Student Reading Student Reading AchievementAchievement Comparison of RF and non-RF schools Comparison of RF and non-RF schools

on ITBS reading comprehension on ITBS reading comprehension scores in grade threescores in grade three

Compare RF-funded schools with sample of Compare RF-funded schools with sample of non-RF schools matched by key non-RF schools matched by key demographic variablesdemographic variables

Are there significant differences in reading Are there significant differences in reading test scores that can be explained by the RF test scores that can be explained by the RF program when other variables are program when other variables are controlled?controlled?

Impact of RF on Impact of RF on Student Reading Student Reading AchievementAchievement

Year-to-year changes in RF schoolsYear-to-year changes in RF schools Grades 1, 2, and 3 ITBS scores in readingGrades 1, 2, and 3 ITBS scores in reading Beginning in year 2, compare each grade Beginning in year 2, compare each grade

to previous year mean scores and to previous year mean scores and percent reading at grade level percent reading at grade level

Identify significance of changes for each Identify significance of changes for each grade compared to scores for previous grade compared to scores for previous year for that same gradeyear for that same grade

Impact of RF on Impact of RF on Student Reading Student Reading AchievementAchievement

Cohort analysis of RF students Cohort analysis of RF students First grade cohort each year of RF First grade cohort each year of RF

programprogram Track cohort in subsequent grades each Track cohort in subsequent grades each

year to identify changes in percent year to identify changes in percent reading at grade level using ITBS scoresreading at grade level using ITBS scores

Identify significance of changes for Identify significance of changes for cohorts after one or two years of RF cohorts after one or two years of RF instructioninstruction

Impact of RF on Impact of RF on Student Reading Student Reading AchievementAchievement

Comparison of CRCT (Georgia Test) Comparison of CRCT (Georgia Test) resultsresults

Means of confirmatory evidence of RF Means of confirmatory evidence of RF impactimpact

CRCT passing rates in reading for grades CRCT passing rates in reading for grades 1, 2, and 31, 2, and 3

Identify year-to-year changes in pass Identify year-to-year changes in pass rates for RF schoolsrates for RF schools

Compare RF student pass rates to those Compare RF student pass rates to those of entire state of GA each yearof entire state of GA each year

Fidelity of Implementing Fidelity of Implementing Reading First Instructional Reading First Instructional StrategiesStrategies

Observations of Reading Observations of Reading First TeachersFirst Teachers

Interviews with Literacy Interviews with Literacy CoachesCoaches

Survey QuestionnairesSurvey Questionnaires

Observations of Reading Observations of Reading First TeachersFirst Teachers Instructional Content Emphasis (I.C.E.)Instructional Content Emphasis (I.C.E.) Half of the schools are observed in the Half of the schools are observed in the

spring and the other half in the fall. spring and the other half in the fall. Year to year (spring to spring/fall to fall) Year to year (spring to spring/fall to fall) comparisons are made from observation comparisons are made from observation data. data.

Observers are UGA Professors and Observers are UGA Professors and doctoral-level graduate studentsdoctoral-level graduate students

Use of Observation Use of Observation DataData Assess what instructional strategies are Assess what instructional strategies are

being used in RF classroomsbeing used in RF classrooms Identify percent of teachers and schools Identify percent of teachers and schools

using key RF strategies (status of using key RF strategies (status of implementation) and changes in use from implementation) and changes in use from year to yearyear to year

Report on statewide trends for grade levels Report on statewide trends for grade levels and for all RF schoolsand for all RF schools

Literacy Coach Literacy Coach Surveys- MonthlySurveys- Monthly On-line, web-based ongoing reporting On-line, web-based ongoing reporting

system system Assess implementation processes and Assess implementation processes and

concerns of key participantsconcerns of key participants UGA Reading Education team develop UGA Reading Education team develop State department RF staff review/ approve State department RF staff review/ approve

surveys prior to administrationsurveys prior to administration Data is used in a formative manner to Data is used in a formative manner to

inform decisions made by the Professional inform decisions made by the Professional Development Architects and the GADOE Development Architects and the GADOE regarding upcoming professional regarding upcoming professional development sessions.development sessions.

Interviews with Interviews with Literacy CoachesLiteracy Coaches Telephone interviews once a year Telephone interviews once a year

(end of year report)(end of year report)

Use of LCs Time Use of LCs Time Related to DIBELS Related to DIBELS ResultsResults DIBELS data compared to literacy DIBELS data compared to literacy

coaches’ self-reported time on coaches’ self-reported time on various tasks.various tasks.

Yearly SurveysYearly Surveys Reading First teachersReading First teachers Reading First school Reading First school

administratorsadministrators Parents of Reading First Parents of Reading First

studentsstudents Regional Reading First Regional Reading First

consultantsconsultants

Progress in Reading Progress in Reading AchievementAchievement Within-school growth in reading Within-school growth in reading

achievement at RF schoolsachievement at RF schools DIBELS subtest scores in grades K-3DIBELS subtest scores in grades K-3 Pre-post analysis each year (beginning and end-Pre-post analysis each year (beginning and end-

of-year test results)of-year test results)

Identify areas of progress across RF Identify areas of progress across RF schoolsschools

Teacher knowledge survey of SBRR Teacher knowledge survey of SBRR (pre-post analysis)(pre-post analysis)

Why evaluating the bottom line isn’t

good enough

“Conventional Wisdom”

Old CW: RF evaluation should focus on the bottom line – achievement

“Conventional Wisdom”

Old CW: RF evaluation should focus on the bottom line – achievement

New CW: RF evaluation must focus on contributory factors as well

“Conventional Wisdom”

Why fine-grained evaluation is needed

Degree of implementation is important. RF is not monolithic across states, districts,

schools, or classrooms. Causal relationships cannot be identified

by looking only at the effect and

assuming a uniform, consistent cause.

So, did Reading First work?

So, did Reading First work?

That’s not the right question.

Reading First has (perhaps unintentionally) created a variety of circumstances nationwide that …

can be viewed as quasi-experiments these are long-term and replicable are numerous enough to address

important questions across sites are also rich in qualitative data

Predictions

Many studies of RF will appear over the next decade, in addition to the comprehensive national report.

These studies will be conducted by independent researchers and will appear in a variety of forums.

The “treatment” in these studies will not be RF in a uniform sense, but a set of factors that RF has made possible.

Cardiac Method of

Evaluation

Hypotheses

Findings will converge across studies, substantiating the impact of certain factors.

These factors will include: instructional practice aligned with SBRI emergence of strong teacher support

for the initiative as a result of PD presence of a knowledgeable, tactful, and assertive coach

So, what will the right question be?

So, what will the right question be?

What has Reading First taught us?

http://curry.edschool.virginia.edu/reading/projects/garf/