povertyactionlab.org Poverty Action Lab: Translating Research into Action MIT
povertyactionlab.org
Poverty Action Lab: Translating Research into Action
MIT
povertyactionlab.org
The what, why and how of evaluations
Abhijit Vinayak Banerjee MIT and Poverty Action Lab
povertyactionlab.org
What is Evaluation
Start with a Real Example
Program • What was the program?
– Flipcharts donated to village primary schools • Where were the primary schools?
– Western Province of Kenya • When did this happen?
– 1997 • Who donated the flipcharts?
– International Child Support Africa (ICS Africa) (a local chapter of a Dutch NGO)
• Who, specifically, were they meant for? – For Students in 6th - 8th Grade
Why flipcharts?
Three sets of questions
Needs Assessment
Impact Evaluation
Process Evaluation
Needs Evaluation • Why are we doing this project: Laying out the theory • What’s the nature of the problem being solved?
– To compensate for a shortage of textbooks – To compensate for a lack of teaching and learning materials in
general – To help those who cannot read English very well – To help those who find it easier to learn from pictures – How will flipcharts solve it? – Why do we need to answer this question?
• Who is the targeted population? – All children? – ? – Why do we need to answer this question?
Needs Evaluation
• Why do we think it will work in this context? – Do teachers feel comfortable using them? – Does it fit with the curriculum?
• What else could we do?
– Adopt a different strategy for addressing the same need
– Do something entirely different with the money – Why do we need to answer this question?
Needs Evaluation Should Provide
• Clear sense of target population – Weaker students – Students with poor English – ?
• Clear sense of program design – What are teachers lacking in terms using these charts? – How to deliver? How much? What are potential barriers?
• Clear articulation of program benefits – Is a wrong being righted? Is a right being expanded?
• Clear sense of alternatives – Is this the most effective, efficient, cost-effective method of
meeting teacher/student needs?
Process Evaluation • Is the program actually doing what it says? • Are the services being delivered?
– Are flipcharts are reaching the classroom, are teachers are being trained to use them
• Are there ways of improving cost effectiveness? – Are the flipcharts being procured/delivered in the best
possible way? • Are the services reaching the right population?
– Are they being targeted to the classes with lower-performing students?
• Are the clients satisfied with service? – Teachers’, students’ response to new teaching method
Impact Evaluation
• What did the program achieve? • This question is only well-defined if we
have determined: • What measures do we expect impacts on? • What populations do we expect impact for? • When should we have answered
these questions?
Impact Evaluation
• Key question: Was there an impact? • Auxiliary questions:
– What was the nature of the benefit? • For example: Did flipcharts lead to uniform
improvements in test scores (for all students)? – Did some types of people benefit more
than others? • Students who were doing worse, poorer students,
etc.
Cost Benefit Analysis
Cost-Benefit Analysis
Process Evaluation
Impact Evaluation
Needs Assessment
Cost Benefit Analysis
• Why are all three elements useful for a cost benefit analysis?
• Impact evaluation gives benefits • Process evaluation allows careful
quantification of costs – May help to lower costs or increase efficiency
• Needs analysis frames opportunity cost of money
povertyactionlab.org
This Week • Focus on impact evaluation: Why? • It is what we know how to do well. • Surprisingly little is known about what works:
– Empowerment and Poverty Reduction: a Source-book – World Bank publication listing what works – Reports that the Gyandoot program that put computers in
villages in India was hard hit by lack of power. The computer kiosks lost money and many had shut.
– Then says, without any apparent irony, “Following the success of the program..”
povertyactionlab.org
Why impact evaluation?
• Central issues in policy design – Do we know that anything is working? – How do we identify what works? – Pick what really works
povertyactionlab.org
How much is the right answer worth?
povertyactionlab.org
Why Impact Evaluation?
• The devil, unfortunately, seems to be in the details.
• A great advantage of combining design with evaluation is that we can focus on the details
• This not only allows the possibility of refining them but also of redesigning them.
How to assess impact
What is beneficiary’s test score with program compared to without program?
Compare same individual with & without programs at same point in time
Formally, program impact is: α = (Y | P=1) - (Y | P=0)
e.g. How much does an education program improve test scores (learning)?
Solving the evaluation problem
Estimated impact is difference between treated observation and counterfactual.
Need to estimate counterfactual.
Never observe same individual with and without program at same point in time.
Counterfactual is key to impact evaluation.
We need a counterfactual
Counterfactual Criteria Treated & Counterfactual (1) Have identical characteristics (2) Except for benefitting from the intervention No other reason for differences in outcomes of treated and counterfactual
Only reason for the difference in outcomes is due to the intervention.
2 Counterfeit Counterfactuals
Those not enrolled
o Those who choose not to enroll
in the program o Those who were not offered
the program
Before and after
1. Before and After: Examples
School scholarship program on enrollment
Agricultural assistance program o Financial assistance to purchase inputs. o Compare rice yields before and after. o Before is normal rainfall, but after is drought. o Find fall in rice yield. o Did the program fail? o Could not separate (identify) effect of financial
assistance program from effect of rainfall.
Those not enrolled: Example of job training
Compare employment & earning of those who sign up to those who did not
Who signs up? Those who are most likely to benefit -i.e. those with more ability- would have higher earnings than non-participants without job training
Poor estimate of counterfactual
Those not enrolled: Example of those offered health insurance
With no insurance: Those that did not buy, have lower medical costs than that did
Compare health care utilization of those who bought insurance to those who did not o Who buys insurance?: those that expect large medical
expenditures o Who does not?: those who are healthy
Poor estimate of counterfactual
Program placement: example
Compare fertility in villages offered program to fertility in other villages
Government offers a family planning program to villages with high fertility
Program targeted based on fertility, so (1) Treatments have high fertility and (2) counterfactuals have low fertility.
Estimated program impact confounded with targeting criteria
What's wrong? Selection bias: People choose to participate for specific reasons
1
2
3
o Job Training: ability and earning o Health Insurance: health status and medical
expenditures
Many times reasons are related to the outcome of interest
Cannot separately identify impact of the program from these other factors/reasons
Need to know… All the reasons why someone gets the
program and others not.
All the reasons why individuals are in the treatment versus control group.
If reasons correlated w/ outcome cannot identify/separate program impact from other
explanations of differences in outcomes.
Possible Solutions Need to guarantee comparability of treatment and control groups.
ONLY remaining difference is intervention.
In this seminar we will consider: o Experimental design/randomization o Quasi-experiments (Regression Discontinuity,
Double differences) o Instrumental Variables.
These solutions all involve… Knowing how the data are generated. Randomization – Give all equal chance of being in control or
treatment groups – Guarantees that all factors/characteristics will
be on average equal btw groups – Only difference is the intervention If not, need transparent & observable criteria for who is offered program.
THANK YOU