What is Impact Evaluation and When and How Should We Do It?

What is Impact Evaluation and When and How Should We Do It?

Linxiu ZhangCenter for Chinese Agricultural Policy

Chinese Academy of Sciences

Nov.14, 2011, Shanghai

Special note

• This presentation is heavily based on the materials provide by Howard White, the Executive Director of 3ie and Scott Rozelle, A Stanford Professor, REAP co-director.

• So, no specific acknowledgements will be given in presentation.

The Overall Agenda

• When Will We Ever Learn: general introduction to Impact Evaluation

• When Random Assignment is Possible?– Implementing and Evaluating RCTs

• When Random Assignment is Not Possible?– Quasi-experimental methods propensity scores,

matching, IV, Regression Discontinuity and DinD.

Amazing Ideas• Sleeping Bag Incubator

• Treadle Pump Irrigation

• Agricultural Price Services through Mobile Phones

• Computer Assisted Learning for Remedial Tutoring

New programs in China (huge fiscal investment)

• Rural Health Insurance (NCMS or hezuo yiliao in China)

• New Subsidy Program (liangshi butie)

• New Education Programs (e.g., raising teacher salaries … or … eliminating tuition for high school)

• Financial Crisis Stimulus Package (investments by central gov’t; investments by localities)

• How many of the innovations/programs that we heard about on the news …

… how many of the new technologies/programs that we have become excited about …

… how many have been rigorously evaluated?

• Do we have empirical evidence, based on a carefully constructed counterfactuals, that these breakthroughs/programs work, can positively affect the lives of the poor and do so in a cost effective way?

Unfortunately, the answer is almost certainly some, but, not many …

• UN Millennium Development Villages

• USAID Bilateral Investment Program

• World Bank / ADB’s Loan Program

Huge Global Initiatives

Statement of Facts:“Accelerating social progress in low- and middle-

income countries requires knowledge about what kinds of social programs are effective. Yet all too often, such basic knowledge is lacking because governments, development agencies, and foundations/NGOs have few incentives to start and sustain the impact evaluations that generate this important information.” (International Evaluation Gap Working Group)

“When it comes to attribution, there is shockingly little concrete evidence about what works and what does not” (Author of report: When will we ever learn)

I was at a conference about 2 to 3 years ago, where one young researcher claimed (in front of scores of older, experienced development economists after 40+ years of development economics and we had not learned anything … until his work (of course)

The Excuses

• We don’t have time

• It costs too much to do rigorous impact evaluation

• It is unethical

• Project implementation is site and context specific

The Excuses

• We don’t have time

• It costs too much to do rigorous impact evaluation

• It is unethical

• Project implementation is completely site and context specific

• We already know!

Example:

• J. Sachs we already know ITN’s work (insecticide treated nets) for the prevention of malaria

• In fact, this time rigorous public health trials support it:– 90 villages: give residents ITN’s– 90 villages: give residents “0”In treatment villages, reductions of malaria, anemia

and other benefits even positive spillovers: villages/hamlet around the

treatment villages (within 300 meters) also benefitted through reduction of malaria (although no ITN’s) … miracle?

ITN’s(insecticide

treated nets)

Policy implications

• People were not buying them …– Despite people being “very afraid” of malaria

…

• Why?– Stanford University team’s hypothesis:

one-time cost too high– Leads to a new RCT

Micro credit or free

• Through an NGO that had “cells of members” [10 to 20 to 30 households per village) in 100s of villages in India, did RCT with two treatment arms:– Treatment village 1: give away for free to all NGO

members– Treatment village 2: sell ITN’s to households as part

of a Micro credit (peer monitoring) project– Control villages: “0”

• What is the outcome?

Impact: ZERO[none: for malaria / none: for anemia …

NONEnone in treatment village 1 / none in treatment

village 2 / none in control villages]

• Explanation:– Have no definitive proof (though now we may know

why villagers do not buy them … they don’t seem to work …

• Theory:– Revisit the original trial … and Revisit and live in own

project villages• People do not always use ITN’s … trouble / hard to hang /

uncomfortable / too many people, not enough nets

An explanation

• How is it that if people do not use them (even in the original public health trial treatment villages) that they have an impact in the villages AND on surrounding villages?

• Only real difference between original trial (100% of households in trial) and Stanford’s trial (10 to 20% of households in trial) maybe it is that all mosquitos are killed and populations collapse when all households have ITN’s … this would account for efficacy in trial and the spillover …

• However, in the partial roll out villages, the ITN’s not effective!

ITN’s do work … but, with a caveat

• Current most plausible explanation:– In the large public health trial, when all of the villagers

received the ITN’s … and were encouraged to use them (and did, at first) … ALL of the mosquitos died …

this reduces malaria in the treatment villagesand

the surrounding hamlets

• Jeffrey’s response? – Of course, that is why we give them to all of the

families … of course, he had no idea …

– Maybe he did “know” … but, surely he does not understand … [but, then none of us do now]

Rest of the session plan

• Introduce the concept of IE– Definitions and examples of what is right and what is

not right– RCT’s … when possible, sexy gold standards!– When you can’t randomize (still a lot of excitement)

• IE is not enough: Supplementary tools

• Issues in choosing an IE strategy– Selecting a control group– RCTs or Quasi-experimental approaches?”

What is impact?

• Impact = the (outcome) Indicator with the intervention compared to what it would have been in the absence of the intervention

• Unpacking the definition– Can include unintended outcomes– Can include others not just intended beneficiaries– No reference to time-frame, which is context-specific– At the heart of it is the idea of a attribution – and

attribution implies a counterfactual (either implicit or explicit)

Defined in this way – UNFORTUNATELY (as discussed above) – we (the international development community) have little evidence on impact of development programs

[in other words: we don’t (systematically) know the results of many of the uncountable number of programs that development agencies, gov’ts and other organizations have been implementing in recent years]

The attribution problem:factual and counterfactual

Impact varies over time

Impacts also are defined over time … Little attention has been given to the dynamics over time … though people think about this …

With project

With project

Project impact

Change in the CAL program effect on the standardized math test scores over time

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)

Time

Sta

nd

ard

ized

math

tes

t sc

ore

The CAL program effect occurred by the midterm evaluation, less than two months after the start of the program.

0

0.120.11

Change in the CAL program effect on the standardized math test scores over time

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

Baseline (Sept. 1) Midterm (early Nov) Final (late Dec)

Time

Sta

nd

ard

ized

math

tes

t sc

ore

There is no improvement between month two and month three..

0

0.120.11

Impact of nutrition at infancy in Guatemala

• After 2 years greater BMI

• After 10 years higher grades in school

• After 15 years higher school attainment

• After 40 years higher wages / income

What has been the impact of the French revolution?

“It is too early to say”

Zhou Enlai

And, even longer run …

Lets examine a less grandiose intervention

• The venue: Poor areas of South West China … a remote mountainous region … populated by groups of Dai and Dong minorities …

• In 1980s and 1990s only small share of girls attended school … most were involved with farming, tending livestock and raising siblings …

• An NGO began giving scholarships in the early 1990s … objective: increase attendance of girls … they claim in their very polished promotion material and in the many workshops that they attend that they have been effective in their mission …

What do we need to measure impact?

Girl’s primary school enrollment

Before After

Project (treatment) 92

Control

The majority of evaluations have just this information … which means we can say absolutely nothing about impact

NOTE: if you measure this well, what is it? Outcome monitoring

What does 92 percent mean?

• Is it high?

• Is it low?

• What does a single number mean?

• What do we compare this to?

Even if done well … output monitoring in its simplest form TELLS US NOTHING about impact

“Before versus after” single-difference comparisonsBefore versus after = 92 – 40 = 52

This ‘before versus after’ approach is more careful outcome monitoring, which has become popular recently.

Outcome monitoring has its place, but:

outcome monitoring ≠ impact evaluation

Before After

Project (treatment) 40 92

Control“scholarships have led to rising schooling of young girls in the project villages”

The changing macro environment … and rising employment opportunities and wages

0

50

100

1990 1995 2000 2005

0

500

1000

1990 1995 2000 2005

Yuan / monthPercent of cohort

Employment in the off farm labor market – 16 to 25 year olds

Off farm wage rate

Rates of completion of elementary male and female students in all rural China’s poor areas

0

20

40

60

80

100Share of rural children

1993 1993 20082008

Outcome monitoring does not tell us about effectiveness

Results… cannot as a rule be attributed specifically, either wholly or in part, to the intervention

An (important) asideCollecting data in order to measure outcomes “before an intervention”

• Can we collect data about outcomes before interventions, after the intervention (that is: is recollection data valid?)

• No (or be careful): work by economists have shown clearly that there are lots of biases introduced to IE by relying on recollection data (most of them psychological)– If individuals have been given a treatment, they often will selectively

remember … they will exaggerate the benefits as a way of showing their gratefulness …

– Those in the control groups will often want to show they are less fortunate and understate their condition (or improvement)

– Empirically, recollection data have lots of biases … hard to determine the direction …

– Best practice (only practice?): collect baseline before the project begins

Post-treatment control comparisonsSingle-difference = 92 – 84 = 8

Before After

Project (treatment) 92

Control 84

Another common approach (lets compare to another set of villages):

But we don’t know if treatment and control groups were similar before…

• How often are intervention villages / schools / clinics / etcetera / chosen in a way that make them systematically different than control villages?

[either for convenience / political necessity / feasibility / cost considerations / or from leaving it to the local partner who uses who-knows-what-type of selection method]

• In the SW China villages, the NGO went to a poor county, but, the local bureau of education chose the villages … and chose them along the road …

• Is attendance in elementary school lower in the control villages because the NGO did not pass out scholarships, or because villagers in control villages had less use for education (or the cost of going to school higher)

Post-treatment control comparisonsSingle difference = 92 – 84 = 7

Main point: Post treatment control comparisons are only valid if treatments and controls were identical at the time the intervention began …

Before After

Project (treatment) 92

Control 84

Another common approach (lets compare to another set of villages):

Double difference =(92-40)-(84-26) = 52-58 = -6

Conclusion: Longitudinal (panel) data, with a control group, allow for the strongest impact evaluation design (BUT: still need matching … if they are different at the start of the project … is there something different in the village which would affect the village’s response to the intervention?)

Therefore: lets collect data for all of the cells?

Before After

Project (treatment) 40 92

Control 26 84

Main points so far• Analysis of impact implies a counterfactual

comparison• Outcome monitoring is a factual analysis, and so

cannot tell us about impact• The counterfactual is most commonly determined

by using a rigorously/carefully chosen control group

If you are going to do impact evaluation you need a credible counterfactual using a control group (not necessarily RCT / but, still need control)

“Gold Standard” Randomized Control Trials

Zero

Medical

What is the counterfactual?

“Gold Standard” Randomized Control Trials

Zero

Crop field trials

What is the counterfactual?

Can also do randomized control trials in schools to test the effectiveness of new school program …

Social Experimentation …

Step 1: choose 50 schools … randomly divide into 2 groups

25 elementary schools in Gansu 25 elementary schools in Gansu

Does one egg per day, improve test scores / attendance?

One Egg Per Day None

25 elementary schools in Gansu 25 elementary schools in Gansu

0

What is counterfactual?

0

Randomized Control Trial[like in agriculture or medicine]

our question:

Will one egg per day lead to higher test scores?

1. Baseline survey2. POLICY EXPERIMENT

RCT’s

3. Evaluation survey

treated

control

Three Stages

Results: One Egg vs. “0”

-3

-2

-1

0

1

2

3

4

5

control

Difference statistically significant at 95% level of confidence

Change in test scores between baseline and evaluation surveys

One Egg / Day Schools

Control Schools (0)

What is causing the difference between Treatment Schools and Control Schools? What is the counterfactual?

Before, we talk more about pros/cons and

keys/pitfalls of running large RCTs studies:Broaden our set of definitions about IE

• Discussion above was for ‘large n’ interventions– There are a large number of units of intervention, e.g. children, households,

firms, schools.– Examples of “small n” are most (but not all) policy reform and many (but not all)

capacity building projects. – E.g.: some reforms (e.g. health insurance) can be given large n designs

• ‘Small n’ interventions require:– Modelling (computable general equilibrium, CGE, models), e.g. trade and fiscal

policy … or role of agriculture in development?– A theory-based analysis (this is what is modeled in a small-n study …) … it is

the logic through which the reform / new capacity will drive economic change …

In fact, many things can’t be randomized?

• Effect of a road on access to off farm employment

• An agricultural subsidy program (that already has been rolled out?

• Impact of the decisions of migrant families to: leave kids behind (get educated in village’s rural public schools while living with Grandma … or go with Mom and Dad and get educated in the city in a private, unregulated migrant school).

How well do students that attend migrant schools perform in standardized tests?

79.7 80.3

71.9

68.6

65

70

75

80

85

Urban students-BJpublic schools

Migrant students-BJ public schools

Migrant students-BJ Migrant

schools

Rural students-SXRural schools

Children in migrant schools actually are a bit above those in poor rural schoolsStandardized math

score

Control for observable characteristics of students and parents (in both rural schools and migrant schools) … and for length of time that migrant children have been in migrant schools … and using

quasi experimental methods (e.g., matching in this case)

79.7 80.3

68.270.5

65

70

75

80

85

Urbanstudents-BJ

public schools

Migrantstudents-BJ

public schools

Migrantstudents-BJ

Migrantschools

Left BehindChildren (rural

schools)

Standardized math score

The argument is that parents bring their children that are better students into urban areas with them … so after controlling for these factors … the difference goes away …

AND:

If you then compare students in migrant schools that have been in Beijing for > 3 years

In fact, many things can’t be randomized?

• Effect of a road on access to off farm employment

• An agricultural subsidy program (that already has been rolled out?

• Impact of the decisions of migrant families to: leave kids behind (get educated in village’s rural public schools while living with Grandma … or go with Mom and Dad and get educated in the city in a private, unregulated migrant school).

Should we not work on these questions?

If we want to study impact of migrant education

Regardless of being large-n or small-n, our focus is on learning why things work, not just what:

(measurement is not evaluation)

This is where we need:

• Qualitative supplementary work (with quantitative IE)

and/or• Theory-based impact evaluation

Why? To allow us to interpret the IE measurements … and identify why a project is working … or not … or how it can be improved … etc …

This helps address one more criticism of the current wave of IE studies …They only tell us: what works … and

not much else!• This has two disadvantages:

– Why would anyone want to be told that their project does not work

• World Bank employee?• Government official?• NGO?

– If you only know it does not work, what is the implication? Eliminate the program … or fix it? But, how?

International Initiative for Impact Evaluation (3ie) is an international

organization trying to put the “how” [or wow] in rigorous IE with “theory-based evaluation” or “causal chain analysis”

Example: a nutrition project in Bangladesh

• Source: Howard White and Edoardo Masset (2007) ‘The Bangladesh Integrated Nutrition Program: findings from an impact evaluation’ Journal of International Development 19: 627-652

• Bangladesh Integrated Nutrition Project (BINP) … a World Bank Project

• Problem: lots of malnutrition … difficult to solve in traditional institutional structures

• Growth monitoring, nutritional

counselling and supplementary feeding (based on a program in Tamil Nadu, which was successful)

• According to the design of the project, implemented by NGOs at field level, used Community Nutrition Practitioners (CNPs)

Program design (theory of change)

Target group participate in program (mothers of young children)

Target group for nutritional counselling is the relevant one

Exposure to nutritional counselling results in knowledge acquisition and behaviour change

Behaviour change sufficient to change child nutrition

Improved nutritional outcomes

Children are

correctly identified to be enrolled in the program

Food is delivered to those enrolled

Supplementary feeding is supplemental, i.e. no leakage or substitution

Food is of sufficient

quantity and quality

AB1

B2E

C1

C2

D1

D2

D3children

mothers

targeting participants

counselling

supplemental feeding

change behavior in child nutrition

sufficient qnty / qlty

no leakage / substituteimproved nutrition outcome

The evaluation story• Looked like it was working – all bits in place and

outcome monitoring data showed fall in severe malnutrition

• Bank agreed to scale up (this is an expensive program … funded at expense of other projects)

• But Save the Children UK critical, though Bank’s M&E team was positive

• Bank’s evaluation department (IEG) did a more rigorous evaluation – found little or no impact

• Theory-based approach explains why

Measuring outcomes and impacts (M&E)

-0.1

-0.05

0

0.05

0.1

0.15

Project M&E: Post treatment control differences

OED: propensity score matching (PSM)

Hei

ght

for

Age

Sco

res

sing

le d

iffer

ence

s (b

etw

een

trea

tmen

t an

d co

ntro

ls)

Mid term

Mid term

After the project

After the project

The evaluation story• Looked like it was working – all bits in place and

outcome monitoring data showed fall in severe malnutrition

• Bank agreed to scale up (this is an expensive program … funded at expense of other projects)

• But Save the Children UK critical, though Bank’s M&E team was positive

• Bank’s evaluation department (IEG) did a more rigorous evaluation – found little or no impact

• Theory-based approach explains why

Measuring outcomes and impacts

-0.1

-0.05

0

0.05

0.1

0.15

Project M&E: Post treatment control differences

OED: propensity score matching (PSM)

Hei

ght

for

Age

Sco

res

sing

le d

iffer

ence

s (b

etw

een

trea

tmen

t an

d co

ntro

ls)

Two points:

1.) need for control group that is similar to treatment group (this is what PSM does)

2.) demands an explanation of the problems with the program

Mid term

Mid term

After the project

After the project

The evaluation story• Looked like it was working – all bits in place and

outcome monitoring data showed fall in severe malnutrition

• Bank agreed to scale up (this is an expensive program … funded at expense of other projects)

• But Save the Children UK critical, though Bank’s M&E team was positive

• Bank’s evaluation department (IEG) did a more rigorous evaluation – found little or no impact

• Theory-based approach explains why

Assumption FindingsProvide nutritional counseling to care givers

Mothers are not decision makers, especially if they live with their mother-in-law

Women know about sessions and attend

90% participation, lower in more conservative areas

Malnourished and growth faltering children correctly identified

No – community nutrition practitioners (CNPs) cannot interpret growth charts

Women acquire knowledge Those attending training do so

And knowledge is turned into practice No there is a substantial knowledge-practice gap

Supplementary feeding is additional food for intended beneficiary

No, considerable evidence of substitution and leakage

Adopted changes are sufficient to improve intended outcomes

Only sometimes

Implementing theory-based analysis

Program design (theory of change)

Target group participate in program (mothers of young children)

Target group for nutritional counselling is the relevant one

Exposure to nutritional counselling results in knowledge acquisition and behaviour change

Behaviour change sufficient to change child nutrition

Improved nutritional outcomes

Children are

correctly identified to be enrolled in the program

Food is delivered to those enrolled

Supplementary feeding is supplemental, i.e. no leakage or substitution

Food is of sufficient

quantity and quality

AB1

B2E

C1

C2

D1

D2

D3children

mothers

targeting participants

counselling

supplemental feeding

change behavior in child nutrition

sufficient qnty / qlty

no leakage / substituteimproved nutrition outcome

Impacts when mother participated

-0.1

-0.05

0

0.05

0.1

0.15

Project M&E: Post treatment control differences

OED: propensity score matching (PSM)

Hei

ght

for

Age

Sco

res

sing

le d

iffer

ence

s (b

etw

een

trea

tmen

t an

d co

ntro

ls)

When examining just mothers that did not live with their mother in laws … and babies were supposed to be in the project …

Mid term

Mid term

After the project

After the project

Illustrating the principles of rigorous IE (good measurement and astute analysis)• Rigorous evaluation of impact using an appropriate

counterfactual: PSM versus simple control

• Map out the causal chain (programme theory)

• Understand context: Bangladesh is not Tamil Nadu (where CPNs were well trained; they were NOT well trained in Bangladesh)

• Anticipate heterogeneity: more malnourished children are in one subgroup of population than others

• Use mixed methods: informed by anthropology, focus groups, own field visits

Final comments

Things to remember

Why has impact become important?

• The results agenda– E.g. Millennium Development Goals and report cards– Focus on outcomes an improvement over impact

monitoring

• But outcome monitoring is not impact– Experience of USAID

Why has impact evaluation become important?

Other places it is taken more seriously

• Examples of results-orientation

– Academic funding in developed countries

– Mexico: Progresa/Opportunidades and Coneval• Progresa/Opportunidades…

– India: outcome budgeting and IEO

Key steps for impact evaluation design

• Well defined objectives and measurable outcomes

• A credible counterfactual, usually using a comparison group based on either random assignment or matching (before versus after and naïve comparisons don’t work)

Key steps for impact evaluation design (2)

• Baseline data considerably strengthens the design [start early!!]

• A theory-based approach (using causal chain) allowing analysis of causal pathways

The appeal of random assignment

• In a randomized control trial the treatment is randomly allocated

• This is more possible than you may think, with examples from governance, legal reform, and environment as well as health and education

• Most arguments against it are readily countered

Alternatives to randomization

• Randomization not always possible

• Then use statistical matching / other quasi-experimental methods …

• Don’t give up on observational data (either your own …or others Can often examine program impact using existing data (e.g. IADB vocational training studies)

“Dean Karlan fights poverty one RCT at a time …” … this is the type of statements that get Esther duFlo and others who practice rigorous IE in trouble … [on cover of his book: “More Than Good Intentions”

When to do an impact evaluation

• Pilot programs – only a meaningful pilot if you evaluate it (and outcome monitoring tells you nothing about impact)

• Representative or important programs

• We have learned a lot about development in the past 40 years … we could have learned a lot more!

The final word

• Impact evaluation matters

• Matters to spending money well

• Matters to know how to make programs work better

• Matters to ending poverty

Rest of the day

• Breakout into small groups

• Map out theory of change underlying your project – case study you have

• What are the related evaluation questions?

• Possible IE methods