1 Barbara Bruns, Lead Education Economist, Latin America and The Caribbean Region Claudio Ferraz Professor of Economics Pontifical Catholic University (PUC) Rio de Janeiro, Brasil Jessica Rodriguez,World Bank Tassia Cruz, Vitor Pereira (consultants) Paying Teachers to Perform: The Effects of Bonus Pay in Pernambuco, Brazil Washington, DC December 7, 2011 Global evidence on bonus pay: what do we know? What do we need to know? Brazilian education context Education context in Pernambuco and the bonus pay program PE Evaluation design: Impact of first year (discontinuous) targets on school performance - RD Impact of first round bonus attainment/not on school performance – RD Impact of introducing bonus pay on system-wide performance – Dif-in-dif Bonus pay and teachers’ classroom practice - panel of classroom observations Bonus pay and schools’ “social capital” - survey of school directors Results to date Next Steps Overview
31
Embed
Paying Teachers to Perform: The Effects of Bonus Pay in ...siteresources.worldbank.org/.../ppt_paying_teachers_Dec.7.2011.pdf · Paying Teachers to Perform: The Effects of Bonus Pay
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Barbara Bruns,
Lead Education Economist,
Latin America and The Caribbean
Region
Claudio Ferraz
Professor of Economics
Pontifical Catholic University (PUC)
Rio de Janeiro,
Brasil
Jessica Rodriguez,World Bank
Tassia Cruz, Vitor Pereira (consultants)
Paying Teachers to
Perform:
The Effects of Bonus Pay
in Pernambuco, Brazil
Washington, DC
December 7, 2011
Global evidence on bonus pay: what do we know? What do we need to know?
Brazilian education context
Education context in Pernambuco and the bonus pay program
PE Evaluation design:
Impact of first year (discontinuous) targets on school performance - RD
Impact of first round bonus attainment/not on school performance – RD
Impact of introducing bonus pay on system-wide performance – Dif-in-dif
Bonus pay and teachers’ classroom practice - panel of classroom
observations
Bonus pay and schools’ “social capital” - survey of school directors
Results to date
Next Steps
Overview
2
Pay for performance: global experience
Increasingly common in OECD and MICs
Annual bonus pay:Rewards for individual teachersRewards for schools
Performance incentives strongest if individual teachers are rewarded for their students’ learning progress, but hard to doRequires testing all grades and subjects twice a year
Relatively expensive (20% of wage bill) compared to information and SBM (but not compared to across-the-board wage increases)
Most systems adopt school-based (group) bonuses
What is the evidence on Pay for Performance?
Country Approach
India:Andhra Pradesh:
Annual bonus based on student test scores – both school-level and individual (RCT)
Rajasthan: Individual bonus for teacher attendance (RCT)
Israel: Individual bonus based on student test results (RD)
School bonus based on multiple student performance measures (tests, graduation rates, credits taken) (RD)
Kenya: School bonus based on student test results (RCT)
(preschools): individual bonus for teacher attendance (RCT)
Brazil:Pernambuco, Rio, SP, MG
School bonus based on student test results and grade progression (RD, DD)
Chile (SNED): School bonus based on multiple measures (tests and other school factors) (DD)
3
What is the evidence on Pay for Performance?
Country Predicted Strength, Core Design
Average Bonus Value (% MW)
Observed Effect Size (Max.)
India:Andhra Pradesh:
4.7(indiv)4.3 (group)
3636
.27
.16
Rajasthan: 4.7 30 .17
Israel: 4.3(indiv) 300 14% higher pass rates and 10% higher test scores
2.7(group) 40 .13
Kenya: 4 43 .14
Brazil:Pernambuco, 3 180 .
Chile: 3 40 .12
What is the evidence on Pay for Performance?
4
What is the evidence on Pay for Performance?
• Bonus pay raised student learning outcomes 0.13-0.27 SD (highest for individual teacher incentives in India)
• Latest US evidence is different: even large bonuses have no impact on student results (Nashville, NYC)
• In some cases, bonus lowered teacher absence, but not always (puzzling)
• Most consistent “pathway” was extra teacher effort out of school
• Research frontier now is to explain these results: Design features
Controllability (noise in performance measure + locality) Size of bonus Coverage/predictability of bonus
• Deepen understanding of how incentives change teacher behavior • inside the classroom:•
Decentralized system with high coverage and low quality – 27 state systems and 5,500 municipal school systems- 40 million students, 1.9 million teachers, 200,000 schools
National concern about education quality for 15 years and consistent reform efforts with some success (PISA improvement 2000-2009)
Teacher quality widely viewed as key issuenon-selective entry into teacher trainingaverage salaries relatively low ((1 time GDP/cap)teacher vacancies in secondary education, esp. math and science
Highly regarded national “results measurement” system – IDEB (Index of Basic Education Quality)
Prova Brasil/SAEB - census-based test of math and portuguese at 3rd, 5th and 9th grade levels every two years and sample based 12th grade test
National school census data on student flows Politically visible, understandable composite IDEB score for all 27 state
and 5,500 municipal school systems
Ready-made framework for states to build on for annual school results measurement (SAEPE, SARESP, SAERJ)
Brazilian Educational Context
Pernambuco Educational Context
1,000 State schools(mainly grades 5-12)
Approx. 1 million students
Approx. 45% of total state enrollments
Large schools (1,000 students and 30-40 teachers/per school)
Multiple shifts (morning, afternoon and secondary school at night)
Many teachers work two jobs (one state and one municipal or private school)
6
Initial School Sample (2009) Escolas observadas: 228Total de municípios: 122
Pernambuco Educational Context
• Weak salary incentives for teachers (unified salary scale – 90% of salary explained by age, years of experience and formal qualifications)
• Weak sanctions and supervision (no dismissal for poor performance or high absence)
• Limited results focus:– School library 6 months into
Pass rate lower grades: Pernambuco vs. Northeast vs. State/Mun
23
Both Portuguese and Math test scores and pass rates
improved significantly in the 9th grade
Smaller effects in the 5th grade, in both test performance
and pass rate
Why? Secretariat believes reason is less state school
influence on 5th grade student performance
5th grade is year most students transfer from
municipal (lower primary) to state (upper primary)
schools
State policies such as bonus likely to have
cumulative results
Overall Bonus Impact
Research Question 3: HOW did the bonus “work”?
24
Bonus Pay and Teacher Classroom practice in
Pernambuco
What does it measure?1. Teacher’s use of instructional time2. Use of materials, including ICT3. Core pedagogical practices4. Ability to keep students engaged
Stallings “classroom snapshot” instrument
How does it work?10 observations of each class taken at regular intervals and coded using a standardized grid
25
OBSERVATION CODING GRID
MATERIAL
ACTIVITY
NO
MATERIALTEXTBOOK NOTEBOOK
BLACK
BOARDLEARNING AIDES ICT COOPERATIVE
1. READING ALOUDT 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
CHECK IF CHORAL READING
2.DEMONSTRATION/ LECTURE
T 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
3. DISCUSSION/QUESTIONS
AND ANSWERS
T 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
4. PRACTICE
& DRILLT 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
5. ASSIGNMENT/ CLASS
WORKT 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
6. COPYINGT 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
7. VERBAL
INSTRUCTIONT 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
8. SOCIAL
INTERACTIONT 1 S L E
I 1 S L
9. STUDENT(S)UNINVOLVED
I 1 S L
10. DISCIPLINET 1 S L E
11. CLASSROOM
MANAGEMENTT 1 S L E
I 1 S L
12. CLASSROOM MANAGEMENT ALONE T
13. TEACHER SOCIAL INTERACTION OR TEACHER UNINVOLVED T
14. TEACHER OUT OF THE ROOM T
School No.:Classroom No. Exact time of observation:
Coding grid
FIGURE 1: EXCERPT FROM CLASSROOM SNAPSHOT (APPENDIX 2)
CLASSROOM OBSERVATION SNAPSHOT
MATERIAL
ACTIVITY
NO MATERIAL
TEXTBOOK NOTEBOOK BLACK BOARD
LEARNING
AIDES ICT COOPERATIVE
1. READING ALOUD T 1 S L E 1 S L E 1 S L E
1 S L E 1 S L E 1 S L E S L E
I 1 S L 1 S L 1 S L 1 S L 1 S L 1 S L S L
CHECK IF CHORAL READING
T line: Indicates activities that
involve the teacher
I line: Indicates activities
that involve the student & not the teacher
1, S, L, E: Indicate one
individual, a small, large group and entire class respectively
26
School and Classroom
Demographic information
#1 Brazil is far from the OECD benchmarks
27
#2 Time spent on instruction is correlated with learning outcomes (Rio de Janeiro – 5th grade)
* Statistically Significant at 10%** at 5%
5th grade
#3 Time spent on instruction predicted bonus pay (Pernambuco)
** Statistically significant at 5%*** at 1%
28
Analysis of
Panel of 200
Schools in
Pernambuco –
2009 and 2010 (after introduction of
school bonus program)
Slight increase in instructional time and decline in class management
…but no change in teacher absence from the class
29
Increase in teacher effectiveness keeping ALL students engaged(share of time entire class engaged with teacher)
Small Group: 2 to 5 students
Large Group: More than 5
students
What is the evidence on Pay for Performance?
PE PE
PE
30
Bonus pay can stimulate improvement in key school results within
1-2 years
Size of incentive and clarity/integrity of results measure important
Early evidence that it is possible to observe differences in
classroom dynamics between improving and stagnant schools, and
system-wide over time
Important design questions in setting up bonus pay programs
Linear or nonlinear? (everyone gets it or only those that pass a
threshold for improvement)
Reward level of performance, value-added, or both? Should
schools that remain at the top be rewarded (PE vs SP)?
Should targets be multi-year? Renegotiated annually?
Summary: what are we learning from Pernambuco?
What strategies DO schools most commonly employ in response to
bonus incentives? (2012 qualitative survey and study)
How does school size (free-rider potential) and school “social capital”
affect bonus attainment? (2012 analysis of existing survey data)
Does classroom-level improvement continue over time? Does it correlate
with bonus? (panel of observations continuing in 2012 and 2013)
How to design effective bonus pay programs? (Comparative study of 3 yr.
results in 3 different Brazilian programs – MG, PE, RJ – 2012/13)
Size of bonus? Linear or nonlinear targets? Reward level of
performance, value-added, or both? Multi-year targets or
renegotiated annually?
Do perverse impacts arise over time (as agents become more familiar