Coaching in Literacy Collaborative and Its Effects on Teachers and Students Gina Biancarosa, University of Oregon Anthony S. Bryk, Carnegie Foundation.

Coaching in Literacy Collaborative and Its Effects on Teachers and Students

Gina Biancarosa, University of OregonAnthony S. Bryk, Carnegie Foundation for the

Advancement of TeachingAllison Atteberry, Stanford UniversityHeather Hough, Stanford University

Annual Meeting of the Society for Research on Educational Effectiveness

March 2010

Key Features of Literacy Collaborative

• Comprehensive school reform program designed to improve elementary children’s reading, writing, and language skills primarily through school-based coaching

• Used in over 700 elementary schools in 200 districts across 26 states

• Intensive professional development– Coaches trained over one year (Lesley University and the Ohio State

University)– Ongoing support from local and national network

• Coaches– In-school professional development courses– One-on-one coaching: viewed as the high leverage activity

• Anatomy of a coaching session– Pre-briefing– Observation– Modeling– Debriefing

• Elements of literacy instruction– Interactive read aloud– Shared reading– Guided reading– Interactive writing– Writing workshop– Word study

Key Features of Literacy Collaborative

Main Research Questions

• Does Literacy Collaborative improve the value-added to student literacy learning?

• Can any effects of Literacy Collaborative be indirectly attributed to coaching via teachers’ changing expertise implementing the instructional practices?

• Can any effects of Literacy Collaborative be directly attributed to coaching? – Does overall coaching activity in a school predict value-

added to student literacy learning?– Does individual teacher participation in coaching predict

value-added to student literacy learning?

Student Data• Value-added analyses focused on grades exposed to LC

professional development (K-2)• Sample: 8576 children, 341 teachers, and 17 coaches in 17

public schools across 8 states in the Eastern U.S. • Children tested in fall and spring for 4 years to measure

change over time in students’ literacy learning using:– Dynamic Indicators of Basic Early Literacy Skills (DIBELS)– Terra Nova in spring

Low Income 46.0%

Race/Ethnicity African-American

LatinoOtherWhite

15.5%5.8%7.2%70.6%

Limited English Proficiency 4.0%

Accelerated Longitudinal Cohort Design6 cohorts studied over 4 years

Year of Study

First Year Second Year Third Year Fourth Year

Fall Spring Fall Spring Fall Spring Fall Spring

K C C D D E E F F

1 B B C C D D E E

2 A A B B C C D DGra

de

Training yearYear 1 of

implementationYear 2 of

implementationYear 3 of

implementation

Our early literacy scale

• Equal differences on scale imply equal differences on the trait measured at any level

• Reported in logits (which describe the probability of a student with a given ability level getting a particular item right or wrong)

• But what do they mean given the particular assessments used?

1

2

3

4

Mean at K entry Names about 30 letters in a minuteVery low phonemic awareness (PA)

Mean at K end & 1st grade entryAccurate and fast letter recognitionGood initial sound PALittle evidence of decoding

Mean at 1st grade end & 2nd grade entryAccurate (not fast) PAReads 50-60 wpmAnswers 1/3 of 1st grade comprehension questions correctly

Mean at 2nd grade endMastery of component skillsReads 90 wpmAnswers 2/3 of 1st grade comprehension questions correctly, 1/3 of 2nd grade questions correctly

Additional Measures

Year of implementationParticipant Construct Instrument 1 2 3Teachers Background characteristics Survey ● ●

Coaching participation Coach logs ● ● ●Frequency of implementation

Coach report

● ●

Expertise of implementation

Coach report

● ● ●

Coaches Background characteristics Survey ● ●Schools Contextual characteristics Survey ● ●

Social-professional network

Survey ● ●

Prior Findings:Coaching sessions per month (n=249)

Coefficient (se) Standardized effect size

Teacher-level moderatorsRole conception 0.048

(0.021)* 0.026

School commitment 0.092 (0.0242)

*** 0.049

10+ years prior teaching experience

-0.099 (0.046)

* -0.053

School-level moderatorsK-2 Staff Size -0.579

(0.110)*** -0.308

Perceived support for LC 0.152 (0.060)

* 0.081

Coach’s prior training 0.148 (0.057)

* 0.079

Teacher influence in school decisions

0.101 (0.064)

* 0.054

* p < .05; ** p < .01; *** p < .001

Prior Findings:Frequency and expertise of implementation (n=249)

ExpertiseCoefficient (se) Standardized effect

sizeInitial status, Teacher-level moderators

School commitment 0.142(0.073)

** 0.252

Prior professional development 0.134(0.078)

* 0.238

≤ 3 years prior teaching experience

-0.534(0.199)

*** -0.951

Growth, Teacher-level moderators

Coaching participation 0.130(0.049)

*** 0.231

* p < .05; ** p < .01; *** p < .001

Value-added Hierarchical Cross-classified Effects Modeling

• Four Levels – time : (students x teachers) : school

– Repeated measures on students (level 1) – Students (level 2) who cross Teachers (level 3) over time – All nested within Schools (level 4)

• The analysis model can be conceptualized as a joining of 2 separate multi-level models

– One two-level model for individual growth in achievement over time, and

– A second two-level model which represents the value-added that each teacher in a school contributes to student learning in that school in a particular year.

Hierarchical Crossed Value-added Effects Model

)_3()_3_()_3(

)_2(

)_2()_2()_2(

)_1(

)_1()_1()_1(

)__(

)__2/1_()__2/1_(

)__1/_()__1/_(

)__()__()__(

)6___(

)5___(

)4___(

)7_()6_()5_()3_()2_(

)6_()5_()4_()3_()2_(

1201201200

1100

1001001000

900

8080800

70

60600

50500

4040400

300

200

100

0100090080070060

050040030020010

0000000

ValueAddedYeardValueAddedYearcValueAddedYear

ValueAddedSummer


ValueAddedSummer


ValueAddedBaseTeacherc

EffectSummerBasedEffectSummerBase

EffectSummerKBasedEffectSummerKBase

EffectAcademicYrBasedEffectAcademicYrBasebEffectAcademicYrBase

AdjustmentMetrictalDevelopmen



CohortCohortCohortCohortCohort

LocationLocationLocationLocationLocation

edbY

Individual growth parameters

overall value-added effects

teacher-level school-level value-added effects

Value-added effects by year (prior to adding coaching as predictor)

Year 1 Year 2 Year 3

Average value-added (overall)

.164 .280 .327

Performance improvement

16% 28% 32%

Effect size .22 .37 .43

Ave. student learning growth is 1.02 per academic year

School 95% plausible value-added range

±.23

Variability in school value-added, year 1Average student gain per academic year

No effect

Year 1 mean effect (.16)

High value-added schools

Low value-added schools


±.23 ±.28


No effect




±.23 ±.28 ±.37


No effect


Year 2 mean effect (.28)Year 3 mean effect (.33)

Teacher 95% plausible value-added range

±.51

Variability in teacher value-added within schools, year 1

Average student gain per academic year

No effect


±.51 ±.71



No effect


±.51 ±.71 ±.91



No effect

Explaining variability in value-added effects

• Tested models with cumulative number of coaching sessions per year (derived from coach logs)– Per teacher– Averaged across teachers at school-level

• Also tested a variety of controls thought to influence teachers’ openness to, participation in, and selection for coaching– Prior use of reform literacy practices– Role conception– School commitment– New to school

Hierarchical Crossed Value-added Effects Model

)_3()_3_()_3(

)_2(

)_2()_2()_2(

)_1(

)_1()_1()_1(

)__(

)__2/1_()__2/1_(

)__1/_()__1/_(

)__()__()__(

)6___(

)5___(

)4___(

)7_()6_()5_()3_()2_(

)6_()5_()4_()3_()2_(

1201201200

1100

1001001000

900

8080800

70

60600

50500

4040400

300

200

100

0100090080070060

050040030020010

0000000


ValueAddedSummer


ValueAddedSummer


ValueAddedBaseTeacherc

EffectSummerBasedEffectSummerBase

EffectSummerKBasedEffectSummerKBase

EffectAcademicYrBasedEffectAcademicYrBasebEffectAcademicYrBase




CohortCohortCohortCohortCohort

LocationLocationLocationLocationLocation

edbY

Individual growth parameters

overall value-added effects

teacher-level school-level value-added effects

Predictors added to baseline and LC value-added effects

Summary of findings

• Only one teacher characteristic significant• Teacher expertise of implementation not

significant• Coaching at the school level not significant• Coaching at the teacher level significant

Teachers’ role conception

• High scorers: Teachers who take an active stance in their professional role in terms of initiating contact and offering help to colleagues

• Higher value-added to student literacy learning in their schools in baseline and Y2

Baseline Year 1 Year 2 Year 3

.049** -.011ns .042* .009ns

Average Value-added of Coaching by year

Year 1 Year 2 Year 3

Average value-added for teacher receiving NO coaching

0.26*** 0.17* 0.14ns

Role conception -.01ns .04* .01ns

Teacher expertise 0.02ns -0.03ns 0.03ns

Value-added per coaching session (cumulative)

-.026* .012* .012*

Average Value-added of Coaching by yearYear 1 Year 2 Year 3

Value-added per coaching session (cumulative)

-.026* .012* .012*

Mean cumulative coaching sessions

2.60 8.96 15.70

Mean coaching value-added

-0.07 0.09 0.19

Unconditional average value-added (overall)

.164 .280 .327

Proportion accounted for by coaching

NA 0.32 0.57

Cumulative coaching sessions min-max

0-12 0-33 0-43

0

10

20

30

40

Cu

mu

lativ

e N

umb

er

of

Se

ssio

ns

pe

r T

ea

che

r

JanY2FebY2

MarY2AprY2

MaJuY2AuSeY3

OctY3NovY3

DecY3JanY3

FebY3MarY3

AprY3MaJuY3

AuSeY4OctY4

NovY4DecY4

JanY4FebY4

MarY4AprY4

MaJuY4

Month of the Study

*Note: Each line represents 1 of the 18 schools in the study

Across the Eighteen Schools, Over TimeAverage Number of Coaching Sessions Accumulated Per Teacher

Across Seventeen Schools, Over Time

17

Minimum coach-ing

25th percentile coaching



Maximum coaching

-0.4

-0.2

0

0.2

0.4

0.6

Year 1Year 2Year 3

Valu

e-ad

ded

0 0 1 5 12

Value-added by coaching, year 1

No coaching effect

Minimum coach-ing




Maximum coaching

-0.4

-0.2

0

0.2

0.4

0.6

Year 1Year 2Year 3

Valu

e-ad

ded

0 4 8 12 33


No coaching effect

Minimum coach-ing




Maximum coaching

-0.4

-0.2

0

0.2

0.4

0.6

Year 1Year 2Year 3

Valu

e-ad

ded

0 8 14 24 43


No coaching effect

Summary of findings

• Evidence that the mechanism for improved value-added shifts from over time– Year 1: Coaching has no value-added– Year 2: Coaching begins to add to value-added for

student learning– Year 3: Coaching becomes the primary mechanism for

value-added to student learning• Cumulative coaching explains differences in

teacher value-added effects, but not school effects

Implications

• Coaching explains differences in teachers’ value-added to student learning

• Shift in coaching effects from negative in Year 1 to positive in Years 2 and 3 raises interesting hypotheses but offer no answers– A selection effect (on the part of coach or teacher)– A dosage effect– A change in coaching expertise effect– Unexplored school/coach effects

• Direct positive effects of coaching on students appear to take time to emerge

Limitations

• Limited sample, especially at school level, limits ability to explore contextual mechanisms

• Coaching was embedded in a school-wide reform model

• Professional development for coaches is more intense than in most other models

Future Steps

• Continued analyses of current data– Length of coaching session– Focus of coaching session– Observation vs. modeling

• Development and piloting of the Performance-based Assessment of Literacy Coaching (PALC)

Thank you!

[email protected]

Coaching in Literacy Collaborative and Its Effects on Teachers and Students Gina Biancarosa, University of Oregon Anthony S. Bryk, Carnegie Foundation.

Documents

student literacy learning

students literacy learning

coaching activity

grade entryaccurate

student datavalue

university of oregonanthony

schoolbased coachingused

lc professional development