There and Back Again: An Impact Evaluator’s Tale Paul Gertler University of California, Berkeley July, 2015.

There and Back Again:An Impact Evaluator’s Tale

Paul GertlerUniversity of California, Berkeley

July, 2015

Answer these questions

What is impact evaluation?

What makes a good impact evaluation?

Why is impact evaluation valuable?

1

2

3

Answer these questions

What is impact evaluation?

What makes a good impact evaluation?

Why is impact evaluation valuable?

1

2

3

“Impact Evaluation

An assessment of the causal effect of a project , program or policy on beneficiary outcomes.

Estimates the change in outcomes attributable to the intervention.

Impact Evaluation AnswersWhat is effect of information on hand

washing, hygiene and child health?

Does paying primary health care workers for performance improve access & quality?Do early childhood education programs improve subsequent learning?Does expansion of urban sewares reduce open defecation & improve health?

Impact Evaluation AnswersWhat was the effect of the program

on outcomes?

How much better off are the beneficiaries because of the program/policy?

How would outcomes change if changed program design?

Is the program cost-effective?Traditional

M&E cannot

answer these.

Answer these questions

What is impact evaluation?

What makes a good impact evaluation?

Why is impact evaluation valuable?

1

2

3

How are Impact Evaluations Useful?

To inform program design

As an input to funding decisions

As a means of influencing ideas

How are Impact Evaluations Useful?

As an input to funding decisions

PROGRESA / Oportunidades (Mexico)

Inform Program Design Influence IdeasInput to Funding

Decisions

Evaluation shows significant impacts on education and health → scaled up & adopted by new presidential

administration.

• Households paid to send children to school and regular health checkups

• First evaluation of large-scale CCT program

Families enrolled in Oportunidades

Roof Rain Water Cisterns -- Brazil

Inform Program Design Influence IdeasInput to Funding

Decisions

• Northeastern states very dry

• Collect rain from roofs during rainy season

• Store in cistern

Evaluation: • No significant impact on health

• Families value cisterns• Value of house & Depression

Low Cost Pre-Schools (Mozambique)

Inform Program Design Influence IdeasInput to Funding

Decisions

• Pre-schools constructed in 30 villages (Save the Children)

• Volunteer community members trained to staff schools

Evaluation shows significant impacts on health and education indicators → scaled up to cover 600 rural

communities.7.38

3.15

26.2117.82

5.375.55 2.86

14.01 11.483.60

Preschool Control

How are Impact Evaluations Useful?

To inform program design

Improving Access to Essential Medicines (Zambia)

Influence IdeasInput to Funding

DecisionsInform Program Design

SP

ACT Adu

lt

ACT Pe

diat

ric0%

25%

50%

75%58%

43%34%

46%

22%30%

BaselineEndline

Clinics receive monthly supplies from district

stores

SP

ACT Adu

lt

ACT Pe

diat

ric0%

25%

50%

75%

52% 48% 43%

16%6% 12%

BaselineEndline

Clinics receive supplies direct from central

stores

Stock-outs are reduced under both distribution systems:

Direct distribution more cost-effective → replicated across Zambia (World Bank, 2010)

$4.20 / day of averted stock-out

$14.50 / day of averted stock-out

Targeting the Poor (Indonesia)

Influence IdeasInput to Funding

DecisionsInform Program Design

• Randomized field experiments evaluating accuracy of 3 methods for targeting the poor:

• Community-based

• Proxy means test (PMT)

• Self-targeting

PMT & community-based found to be most accurate → findings used by Indonesian government to build registry of

poorest 40% (World Bank, 2012)

How are Impact Evaluations Useful?

As a means of influencing ideas

Conditional Cash Transfer (CCT) ProgramsInput to Funding

DecisionsInform Program Design Influence Ideas

Countries implementing CCT programs in

1997

Conditional Cash Transfer (CCT) ProgramsInput to Funding

DecisionsInform Program Design Influence Ideas

Countries implementing CCT programs in

2011

Results-Based Financing (Rwanda)

Input to Funding Decisions

Inform Program Design Influence Ideas

• Health clinics paid to immunize children and encourage women to give birth in a clinic

• Treatment group received bonuses according to performance;

• Control group received grant regardless of performance

Evaluation shows RBF has significant impact on prenatal care → results inspire RBF designs in other

countries, including Nigeria (Basingra et al, 2010)

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

Treatment (PBF) Facili-ties

Control Facilities

Sta

nd

ard

ized

Pre

nata

lEff

ort

Sco

re

Baseline (2006) Follow-up (2008)

15% Standard deviation increase

due to RBF

Treatment (RBF) Facilities

Control Facilities

Results-Based Financing (Argentina)Input to Funding

DecisionsInform Program Design Influence Ideas

• Plan Nacer used to provide additional health care coverage to women and young children

• Federal resources allocated to provinces based on enrollment and health results achieved by each province

Evaluation shows RBF has significant impact on prenatal care, infant/ maternal mortality → results inspire

RBF program in Dominican Republic & Peru

Answer these questions

Why is evaluation valuable?

What makes a good impact evaluation?

1

2

How to assess impact

What is beneficiary’s diarrhea incidence of diarrhea in last 3-days with program compared to without program?

Compare same individual with & without programs at same point in time

Formally, program impact is:

α = (Y | P=1) - (Y | P=0)

e.g. How much does an safe water intervention reduce diarrhea?

Solving the evaluation problem

Estimated impact is difference between treated observation and counterfactual.

Counterfactual: what would have happened without the program.

Use Control group to estimate counterfactual.

Never observe same individual with and without program at same point in time.

Counterfactual is key to impact evaluation.

Possible SolutionsNeed to guarantee comparability of treatment and control groups.

ONLY remaining difference is intervention.

Consider:o Experimental design/randomization o Quasi-experiments

Counterfactual Criteria

Treated & Counterfactual(1) Have identical characteristics,(2) Except for benefiting from the intervention.

No other reason for differences in outcomes of treated and counterfactual.

Only reason for the difference in outcomes is due to the intervention.

2 Counterfeit CounterfactualsBefore and After

Those not enrolledo Those who choose not to

enroll in the programo Those who were not

offered the program

Same individual before the treatment

Problem: Cannot

completely know

why the treated

are treated and

the others not.

What's wrong?Selection bias: People choose to participate for specific reasons

1

2

3

Many times reasons are related to the outcome of interest

Cannot separately identify impact of the program from these other factors/reasons

Need to know…

All the reasons why someone gets the program and others not.

All the reasons why individuals are in the treatment versus control

group.If reasons correlated w/ outcome cannot identify/separate program impact from

other explanations of differences in outcomes.

Possible SolutionsNeed to guarantee comparability of treatment and control groups.

ONLY remaining difference is intervention.

In this seminar we will consider:o Experimental design/randomization o Quasi-experiments (Regression

Discontinuity, Double differences)o Instrumental Variables.

These solutions all involve…Knowing how the data are

generated.Randomizationo Give all equal chance of being in control

or treatment groups o Guarantees that all factors/characteristics

will be on average equal btw groupso Only difference is the intervention

If not, need transparent & observable criteria for who is offered program.

Working Smarter in IE

• Be strategic in selecting programs to evaluate

• Don’t assume costly data collection is always necessary

• Use IE to maximize program efficiency, not just impactImpact evaluation as an operational research tool

Use administrative data when possible

Use IE’s to fill knowledge gaps & assess alternatives for key programs

Ensuring the Impact of Impact Evaluations

Engage early. Engage Often.

Work locally. Think globally.

• Tale of 2 Evaluations

• Involve stakeholders at every stage

• Prospective – IE part of design

• Foster relationships on the ground with decision-makers

• Results inform decisions beyond the borders of country studied

Water Privatization -- Argentina

o Municipal water & sanitation

o Public versus Private management

o Evaluation:o Increase in water

qualityo Increase in access

by pooro Large reductions in

child mortality

Figure 4: Evolution of Mortality Rates for Municipalities with Privatized vs. Non-Privatized Water Services

3.5

4

4.5

5

5.5

6

6.5

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

Year

Mo

rta

lity

Ra

tes

Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities

No Influence on policy: Academic study without stakeholders involvement

Improving Housing in Urban Slums(Piso Firme)

Inform Program Design Influence IdeasInput to Funding

Decisions

Replacing Dirt Floors with Cement Floors (Piso Firme)

• Evaluation shows significant impacts on reducing diarrhea, parasitic infections & improving cognitive development

• Scaled up to 3 million householdso Moving families to new housing

developments (Tu Casa)o No Impact – canceled program

Ensuring the Impact of Impact Evaluations

Engage early. Engage Often.

Work locally. Think globally.

• Tale of 2 Evaluations

• Involve stakeholders at every stage

• Prospective – IE part of design

• Foster relationships on the ground with decision-makers

• Results inform decisions beyond the borders of country studied

Messageso IE Useful for policy

o Resource allocationo Benefit design

o Influence global ideas

o What to evaluate

o High cost programs with large #’s beneficiaries

o Little existing knowledge

o Start early – prospective

o Build control groups into rollout

o Work locally think globally

Figure 4: Evolution of Mortality Rates for Municipalities with Privatized vs. Non-Privatized Water Services

3.5

4

4.5

5

5.5

6

6.5

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

Year

Mo

rta

lity

Ra

tes

Mortality Rates, Non-privatized MunicipalitiesMortality Rates, Privatized Municipalities