Evaluation of South Carolina’s Reading First (SCRF) Initiative

Evaluation of South Carolina’s Reading First

(SCRF) Initiative

Achievement and Survey Results from the First Three Years

Presented at the American Educational Research Association’s Annual Meeting

March 27, 2008

Tammiee S. Dickenson, PhD

Robert L. Johnson, PhD

Heather L. Bennett, MSW

Katie A. Sesso-Dahlke, MSW

Brett Ermer

Mugdha Galande

Joanna Gilmore

Jessalyn Smith

Diane M. Monrad, PhD Sarah J. Gareau, MEdJennifer GayDiana MindrilaAnita Rawls

Pam WillsPatricia BranhamBecca DriggersDiane Dunham, EdS

Overview

SCRF evaluation

Data sources

Achievement data

Participant group survey results

Conclusions

SCDE

SCEPCOPE

Planning

Data Collection

Data Analysis

Reporting

SCRF Evaluation

Participants

2004-2005: served 52 schools in 24 districts (approximately 12,000 students)

2005-2006: served 51 schools in 23 districts (approximately 11,000 students)

2006-2007: served 49 schools in 23 districts (approximately 10,500 students)

Data Sources

Stanford Reading First assessment

Palmetto Achievement Challenge Test (PACT)

Participant group surveys

School Leadership Team (SLT) surveys

Professional development workshop surveys

Annual SLT Presentation

Results presented from previous year

Stanford Reading First assessment results

Cohort analysis of all students in previous year

Matched analysis of students in all years

Individual school reports provided

Participant group survey results

Formative feedback for program improvement

Stanford Reading First Assessment

Measures achievement in grades 1 through 3

Sections:

Multiple choice (components – phonemic awareness; phonics; vocabulary development; reading fluency; and reading comprehension strategies)

Oral fluency (components – speaking vocabulary and oral reading fluency)

Results: At Grade Level (AGL) Grade 1

Fall Spring Annual Gain

Year 1 17.0% 48.2% 31.2%

Year 2 18.8% 51.2% 32.4%

Year 3 21.1% 54.3% 33.2%

Percentage of SCRF Grade 1 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains

Results: At Grade Level (AGL) Grade 2

Fall Spring Annual Gain

Year 1 30.8% 35.5% 4.7%

Year 2 35.7% 40.4% 4.7%

Year 3 38.0% 44.6% 6.6%

Percentage of SCRF Grade 2 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains

Results: At Grade Level (AGL) Grade 3

Fall Spring Annual Gain

Year 1 21.0% 41.5% 20.5%

Year 2 25.2% 46.2% 21.0%

Year 3 27.6% 49.0% 21.4%

Percentage of SCRF Grade 3 Students Scoring At Grade Level (AGL) on Stanford Reading First and Annual Gains

Average NCE Scores on Stanford Reading First for SCRF Students with AGL Reference Line

0

10

20

30

40

50

60

Fall Gr1 Spr Gr1 Fall Gr2 Spr Gr2 Fall Gr3 Spr Gr3

SESAT2 Primary 1 Primary 2 Primary 3

NC

E S

co

re

At Grade Level SCRF Students

Average Scale Scores on Stanford Reading First for SCRF Students and the Norm Group

400

450

500

550

600

650

700

Fall Gr1 Spr Gr1 Fall Gr2 Spr Gr2 Fall Gr3 Spr Gr3

SESAT2 Primary 1 Primary 2 Primary 3

Scal

e sco

re

Norm Group SCRF Students

Participant Group Surveys

Respondent groups: interventionists, literacy coaches, principals, and teachers

Sections: implementation, level of support, roles and responsibilities, professional development, and program effectiveness

Comparison across years (2005-2006 compared to 2006-2007)

Results: Participant Group Surveys Better understanding of program goals Reported a high rate of involvement in professional

development activities and found those activities helpful

Decrease in the need for professional development about the program and use of assessments

Increased need for professional development on effective instructional strategies to use for students below grade level

Implementation Survey Items

0% 20% 40% 60% 80% 100%

I understand my rolesand responsibilities

within the SCRF Initiative.

I understand the goals ofthe SCRF Initiative.

I support the SCRFInitiative.

% Agree or Strongly Agree

Principal Literacy Coach Interventionist Teacher

Implementation Survey Items (Cont.)

0% 20% 40% 60% 80% 100%

Teachers, interventionists,administrators, and the

school-based SCRF literacycoach(es) are working

together to implement theSCRF Initiative.

SLT members regularlyshare important information

with our faculty about theSCRF Initiative.

% Agree or Strongly Agree

Principal Literacy Coach Interventionist Teacher

Describe One Benefit of the SCRF Initiative

Commonly cited benefits across groupsFocus on assessment

Increased awareness of students’ needs/strengths/weaknesses

Progress monitoring; Dominie Improved use of assessments; use of assessment data to

guide instruction

Resources Additional books and materials Extra funding/money

Professional development; study groups

Increased collaboration

Learned new strategies

Results: Participant Recommendations

Recommendations

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Changes to study groups Changes related toassessments

Increase writing emphasis

Per

cent

age

Teacher Interventionist Literacy Coach Principal

Results: Effectiveness Ratings

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

Not Effective Somewhat Effective Effective Very Effective

Effectiveness Rating

Per

cent

age

Teacher Interventionist Literacy coach Principal

83.1% of teachers rated SCRF as effective or very effective. 88.1% of interventionists rated SCRF as effective or very effective. 81.8% of literacy coaches rated SCRF as effective or very effective. 90.9% of principals rated SCRF as effective or very effective.

Conclusions: Achievement

Students performed better on the Stanford Reading First assessment in the 2006-2007 school year as compared with the 2005-2006 school year.

Students’ scores on the Stanford Reading First assessment improved from fall to spring in all three years for all grade levels.

The largest gains were made by students in grade 1 and the lowest gains occurred in grade 2 in 2006-2007, which is also consistent with the last two years.

Conclusions: Participant Groups Program participants have a better

understanding of program goals.

Participant groups find professional development useful.

Over 80% of each participant group rated the initiative as either effective or very effective in Years 2 and 3.

Recommendations

More time for sustained reading for students in grade 2

Additional strategies for working with students below grade level

Participant groups’ suggestions regarding study groups

Contact InformationSouth Carolina Educational Policy CenterUniversity of South Carolina, College of EducationWardlaw Suite 010, Columbia, SC 29208

P: 803-777-8244

F: 803-777-0220 E: [email protected] (Diane Monrad)

Office of Program EvaluationUniversity of South Carolina, College of EducationWardlaw Suite 023, Columbia, SC 29208

P: 803-777-3402F: 803-777-8838E: [email protected] (Tammiee Dickenson)