What evidence should policy makers use and how can we make ... · click through Variations in home page layout. Impact evaluation takes one hour Mergers and Acquistions 80% of M&A

The Campbell Collaboration www.campbellcollaboration.org

Howard White

CEO, Campbell Collaboration

What evidence should policy makers use

and how can we make it happen?


Things we probably all agree on

• Large n impact evaluation cannot answer all the questions we have

about how to ‘do development’

• … or even about programme effectiveness

• Rigorous impact evaluation should NOT be carried out on every project

• Rigorous impact evaluate can rarely be used to evaluate a whole

project

• This is partly as large n designs are not always applicable

• And many RCTs to date focus on whacky gimmicks, not things of

interest to policymakers (‘researcher capture’)

… But despite all that we are not doing enough

‘development RCTs’


So what can impact evaluations be used for?

Using IE to measure results:

• Alignment, attribution and

aggregation

• Representative sample

• Agency incentive

structures

Accountability Lesson

learning

• Second generation questions

• Build in learning opportunities

• Causal chain analysis

• Accumulate bodies of evidence:

the role of systematic reviews

Implies strategic selection of IE cases

for lesson learning


Second generation questions don’t ask ‘does it

work’ but design questions about ‘how can it work

better?’

E.g. conditional cash transfer second generation questions:

– Conditions or not?

– What sort of conditions?

– Who to give money to?

– How to give the money?

– When and how often to give money?


The largest growth in RCTs has been in the private sector: small, rapid studies to improve performance

DVD price experiment

Up to US$15 difference for same product

Reimbursed those who paid more, and now all pay lowest price even if order at higher price

Maximizing click through

Variations in home page layout.

Impact evaluation takes one hour

Mergers and Acquistions

80% of M&A bad for bottom line

Cisco tripled profits through 60 M&A

Exploited heterogeneity

Had data on 9,000 cases

5

Researchers working in

partnership with private sector:

• Product placement in super

markets in low-income areas for

healthy choices

• Improving efficiency of water use

in Atlanta

• Front of pack nutrition labelling

and healthy eating


And this really matters… as most things don’t work

80% of businesses fail in first five years

– do we really think development

projects any better?

But there is usually no bottom line for

development projects

Impact evaluation is the bottom line.

And here is what the bottom line says in

developed countries:

• Education: 90 interventions evaluated in

RCTs by IES - 90% had weak or no positive

effects.

• Employment/training: Department of Labor-

commissioned RCTs 75% weak or no

positive effects

• Business: Over 13,000 RCTs of new

products/strategies conducted by Google

and Microsoft, 80- 90% no significant

effects.

Without impact evaluation you can move the

bottom line!


Rapid impact evaluation as formative research: 3ie’s HIV programme (VMMC)

7

• Soccer-related incentives: Food voucher

• Transportation reimbursement and framing

messages

• Pregnant women delivering information to

male partners

• Smartphone raffle

• SMS messages

• Conditional cash transfer for peer referral


=𝑇𝑜𝑡𝑎𝑙 𝑒𝑓𝑓𝑒𝑐𝑡

𝑁𝑜. 𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔

𝑁𝑜.𝑝𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑛𝑔

𝑁𝑜.𝑜𝑓 𝑖𝑛𝑡𝑒𝑛𝑑𝑒𝑑 𝑏𝑒𝑛𝑒𝑓𝑖𝑐𝑎𝑟𝑖𝑒𝑠

𝐴𝑇𝐸 (𝐼𝑇𝑇) =𝑇𝑜𝑡𝑎𝑙 𝑒𝑓𝑓𝑒𝑐𝑡

𝑁𝑜. 𝑜𝑓 𝑖𝑛𝑡𝑒𝑛𝑑𝑒𝑑 𝑏𝑒𝑛𝑒𝑓𝑖𝑐𝑎𝑟𝑖𝑒𝑠

= 𝑇𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡 𝑜𝑓 𝑡𝑟𝑒𝑎𝑡𝑒𝑑 𝑒𝑓𝑓𝑒𝑐𝑡 𝑇𝑜𝑇 𝑥 𝑃𝑎𝑟𝑡𝑖𝑐𝑖𝑝𝑎𝑡𝑖𝑜𝑛 𝑟𝑎𝑡𝑒 (𝑃𝑅)

So the participation rate is a key

determinant of effectivenessAnd participation is often

far less than we expect

The importance of causal chain analysis


The funnel of attrition Only these people

may experience

improved outcomes


Does the community participate?

0

20

40

60

80

100

Heard of project Aware of meeting Attended meeting Spoke at meeting Aware of cost of project

BRA-KDP MASAF Tuungane GoBifo ZAMSIF

JSIF FISE PAMR FUMAC Average

Community participation declines by degree of engagement:

the funnel of attrition


Many interventions fall at the first hurdle

• Free male circumcision: 25% if

free down to just 10% with

partial subsidy

• Pre-school in Mexico, fewer

than 10% of parents who

registered actually took part

• Insurance schemes typically

less than 10% take up


And participation declines over time

• ½ households stopped using

improved cookstoves by 8

month follow up survey

• Water treatment: fewer than

1/3 households using filters in

Cambodia and pasteurising in

Kenya after 3-4 years.. And

only 5% disinfecting in

Guatemala after just one year


The role of systematic reviews: can send messages on what

does and doesn’t work

In fragile and conflict-affected states (FCS), the approach has also

helped rebuild social capital and trust within communities, and between

communities and governments

World Bank CDD Website

NOTE: Weights are from random effects analysis

Overall (I-squared = 48.9%, p = 0.014)

PAC

Armenia SIF

FHIS 2

FISE

JSIF

GoBifo

PAMR

FUMAC

KALAHI-CIDSS

MASAF

project

KDP

CDR

ZAMSIF

BRA-KDP

FUMAC-P

Tuungane

Brazil

Armenia

Honduras

Nicaragua

Jamaica

Sierra Leone

Benin

Brazil

Philipines

Malawi

country

Indonesia

Liberia

Zambia

Indonesia

Brazil

DRC

0.06 (-0.01, 0.14)

-0.12 (-0.58, 0.34)

0.09 (-0.02, 0.20)

0.48 (0.24, 0.71)

-0.04 (-0.85, 0.76)

0.50 (-0.01, 1.01)

0.14 (-0.19, 0.48)

0.17 (-0.05, 0.39)

-0.03 (-0.32, 0.26)

0.02 (-0.14, 0.18)

-0.27 (-0.62, 0.09)

d (95% CI)

0.22 (-0.10, 0.53)

0.06 (-0.08, 0.20)

-0.02 (-0.33, 0.30)

-0.03 (-0.43, 0.38)

-0.28 (-0.58, 0.03)

0.01 (-0.05, 0.07)

100.00

2.48

13.04

6.87

0.90

2.07

4.18

7.34

5.05

10.17

3.79

%

Weight

4.58

11.31

4.60

3.05

4.82

15.74

0.06 (-0.01, 0.14)

-0.12 (-0.58, 0.34)

0.09 (-0.02, 0.20)

0.48 (0.24, 0.71)

-0.04 (-0.85, 0.76)

0.50 (-0.01, 1.01)

0.14 (-0.19, 0.48)

0.17 (-0.05, 0.39)

-0.03 (-0.32, 0.26)

0.02 (-0.14, 0.18)

-0.27 (-0.62, 0.09)

d (95% CI)

0.22 (-0.10, 0.53)

0.06 (-0.08, 0.20)

-0.02 (-0.33, 0.30)

-0.03 (-0.43, 0.38)

-0.28 (-0.58, 0.03)

0.01 (-0.05, 0.07)

100.00

2.48

13.04

6.87

0.90

2.07

4.18

7.34

5.05

10.17

3.79

%

Weight

4.58

11.31

4.60

3.05

4.82

15.74

CDD reduces cohesion CDD increases cohesion

0-.5 0 .5

Social cohesion


Poor assumptions: payment for environmental services

Tiny effect

0.3 % reduction in

deforestation

i.e. after 10 years 97% of

land for which

payments received

would still have been

forested in absence of

the programme

Effects of PES on forest cover change rate due to deforestation

ra - rc

−0.010 −0.005 0.000 0.005 0.010

Costa Rica PSA

Costa Rica PSA

Costa Rica PSA

Mexico PSAH

Mexico MBCF

1998−2005

1997−2000

2000−2005

2004−2006

2000−2009

Arriagada et al. 2011

Pfaff et al. 2008

Robalino et al. 2008

Alix−Garcia et al. 2012

Honey−Roses et al. 2011

Random effects mean

Predictive interval

I2 = 67.9%, t

2 = 0 (0), t = 0


But meta-analysis allows us to get at design

features, for example …

CCTs have a larger effect on enrolment rates

• Secondary than primary

• The larger the transfer

• The less frequent the transfer

• If conditions include achievement not just attendance

And…


Conditionality works

Children 60% more likely to be in school with conditionality

which is monitored and enforced compared to no

conditions

But we need a lot of primary studies

to exploit heterogeneity


The impact of water quality interventions is not

sustained

NOTE: Weights are from random effects analysis

Water supply (12 months or more)

Subtotal

Water quality (under 12 months)

Subtotal

Water quality (12 months or more)

Subtotal

Sanitation (12 months or more)

Subtotal

Hygiene (under 12 months)

Subtotal

Hygiene (12 months or more)

Subtotal

Multiple (under 12 months)

Subtotal

Multiple (12 months or more)

Subtotal

ID

Study

0.82 (0.71, 0.96)

0.56 (0.47, 0.66)

0.81 (0.67, 0.97)

0.64 (0.37, 1.10)

0.72 (0.60, 0.86)

0.67 (0.49, 0.91)

0.41 (0.23, 0.74)

0.77 (0.70, 0.85)

ES (95% CI)

0.82 (0.71, 0.96)

0.56 (0.47, 0.66)

0.81 (0.67, 0.97)

0.64 (0.37, 1.10)

0.72 (0.60, 0.86)

0.67 (0.49, 0.91)

0.41 (0.23, 0.74)

0.77 (0.70, 0.85)

ES (95% CI)

Ratio favours intervention

1.1 .5 .75 1 2


Medical progress

There are irreversible

improvements in

medical practice


In the 1920s teachers were paid according to their

pupils’ exam results. This system resulted in

classes focused on teaching material of relevance

to the exam to the neglect of other aspects of the

child’s education or overall development.

Fortunately we know better now and such policies

have long been abandoned’


i.e. what is our theory of change as to use of

evidence, and how does it differ?

So why is social policy subject to

faddism in a way that medical practice

is not?

Answering that question is

another talk! But…..


3ie 50% ‘hit rate’ with primary studies

• Scaling up

• Closing down

• Informing design

• Affecting other programmes

• Institutionalizing lesson learning

Much less to show for systematic

reviews

Why?

• Production process

• Diffuse stakeholders

• More ambitious goals

Solutions:

• Better reviews

• Demand-driven reviews

• More user engagement

• Derivative products


Thank you

Visit our website

www.campbellcollaboration.org

Follow us on Twitter @C2update & Facebook

What evidence should policy makers use and how can we make ... · click through Variations in home page layout. Impact evaluation takes one hour Mergers and Acquistions 80% of M&A

Documents