OCTOBER 13-17, 2014 | LISBON, PORTUGAL DIME WORKSHOP 1 tinyurl.com/dime-lisbon #IEKnowEE http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
1
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
Operational Lessons Learned from
Implementing IEs Chair: Guadalupe Bedoya, DIME Speakers: Jed Friedman, World Bank Caroline Cogueto, Sao Paulo State Environment Secretariat Susumu Yoshida, DIME Maria Jones, DIME
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
3
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
Ensuring Buy-in: Jed Friedman
World Bank
Take-up: Carolina Cogueto
Sao Paulo State Environment
Secretariat
Susumu Yoshida
DIME
Data Quality: Maria Jones
DIME
Operational Lessons from
Implementing IEs
Why do we care about Implementation?
More often than not you will face challenges in the implementation of your IE
More importantly… if implementation is different
than planned you may be not answering your IE question(s)
Case 1: Take-up
• Intended IE Question: What is the impact of connecting households to the electrical grid on income of rural households without electricity?
• …But only 30% of your treatment group actually connected to the electrical grid
Impact Estimate= 5%
(ITT)
Income in the Treatment
Group
Income in the Control Group
• Answered IE Question? What is the impact of offering access to the electrical grid on income of rural households without electricity?
70% w/o treatment
Case 2: Quality of treatment
• Intended IE Question: What is the impact of a quality financial literacy (FL) program on FL knowledge?
• …But teachers deliver only 50% of the curriculum (which may not really have an effect…)
Impact Estimate
= 0
FL knowledge in the
Treatment Group
FL knowledge in the
Control Group
• Answered IE Question? What is the impact of a low-quality financial literacy program on FL knowledge ?
So, changes in the implementation and bad measures of the outcomes
may lead to answering questions that…
– Are not the intended questions – May be not policy relevant or even interesting – May be misleading if not interpreted correctly – May be incorrect (bad data)
Activities
Sessions with
children and/or
parents on
early education
(# of sessions,
# of activities)
Meals and
nutrition
supplements are
distributed to
children
(# meals per day
and supplements
distributed,
nutritional value)
Outputs
Children
/parents
attending
sessions
(# of days/hours
attended, # of
activities
performed)
Children receive
and take meals
and supplements
(# meals /
supplements
received, taken)
Short-term and
Intermediate outcomes
Parents behavior
and attitudes
toward early
childhood
development
change
(measures of
behavior, awareness,
attitudes…)
Children are healthy and
developmentally ready
for school (cognitive,
physical and
psychosocial tests)
Children perform
better in life
(level of
education, test
scores, rates of
criminality,
wages, etc…)
Long term
outcomes/Goal Components
Early
Education
1
Nutrition
2
Results Chain (Early Childhood Intervention)
Does this happen as planned? • Buy-in • Implementation monitoring
Does your data “says” what it’s supposed to?
Implementation/Process Evaluation
• Monitors program’s delivery and quality over time. • Usually includes measures of outputs and some qualified
outcomes. 1. Helps understand and interpret impact results 2. Helps improve program performance during program
operations.
• Who is served? Is the program serving the intended group?
• What is the program delivering? Is this what the program intended?
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
12
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
Operational Lessons Learned: Ensuring Buy-In
Jed Friedman, Development Research Group
The World Bank
October 15, 2014
Why do we evaluate?
• The first objective: – “What is the causal effect of intervention X on priority outcomes Y”
• Who wouldn’t want to know this? – Helps allocate resources in an efficient manner
• But…there can be many obstacles to stakeholder buy-in to evaluation – consequent hesitancy to evaluate even though many people may ultimately
benefit
• A great deal of experience in room on how to improve buy-in, but I’ll discuss three factors that I have seen help – At times IE designs need to be modified to accommodate systemic or
political realities
– Maximize external validity – a good practice anyway, also helps buy-in
– Continual need to stress that IE is a learning exercise, i.e. a tool used by prudent system managers to benefit the principal stakeholders
IE design and buy-in
• The RCT is considered a gold-standard approach in evaluative research, and for good reasons – Minimal (and largely verifiable) assumptions necessary for causal inference
– Research design (and results) easily communicated to policy makers and public
• BUT in some contexts: – Disquiet among implementers to differentially treat units under their purview
– Geographic proximity of treated and control units can create unanticipated spillovers: disgruntled or transferred service providers, innovation imitation, population movement, etc.
– Spotlight effects: Perhaps system leadership provides experimental facilities with extra attention and resources
• So: where treatment units are not subject to spill-overs or spotlight effects then by all means advocate for RCT as first-best option
• But if we suspect confounders, we need another strategy…
The challenge of clustered studies when the number of clusters is low
• Typically the alternative strategy to an RCT will involve implementation at a more aggregate level, i.e. a district
• Outcome assessed at clinic or household level while randomization at district level still constitutes a viable IE design
• The problem is that there may be relatively few districts in the study, so risk of underpowered design
• Solutions
1. Leverage additional assumptions of quasi-experimental estimator such as district matching to increase power
2. Consider Fisher-exact standard errors with one-sided hypothesis testing to increase power
External validity I
• The first objective:
– “What is the causal effect of intervention X on priority outcomes Y”
• But in reality, any single impact evaluation answers the
modified question:
– “What is the causal effect of intervention X on priority outcomes Y in
study setting Z”
• We want to abstract away from specific setting Z to inform
broader policy “at scale”
Often program evaluation lessons are difficult to generalize
• Evaluation of contract teachers in Kenya
– Bold et al (2012) find positive effect on test scores in schools randomly assigned to NGO implementation, but not in schools assigned to government
• Evaluation of anti-malaria mobilization programs in India
– Das, Friedman, and Kandpal (2014) find gains in malaria net usage and prompt care-seeking in regions assigned to some NGOs but not others
• Evaluation of energy experiments in US
– Conducted by one company in 14 cities find that impacts vary by location and characteristics of local partner (Alcott and Mullainathan, 2012)
External validity II
• Policy makers and practitioners have an implicit understanding
of the challenges of external validity
– “How can I believe the study results apply to the country as a whole?”
• Need to design the IE to maximize generalizability
– And then reassure stakeholders that this is the case
Broad geographic coverage facilitates external validity
• In Zimbabwe, MoH with strong desire to pilot new health program in every
province. For policy learning, selected two districts “representative” for
that province.
The challenge of differential access to treatment
• Resistance to the inclusion of a leave-out group
– We can all think of examples where practitioners are hesitant to
introduce a program in only part of a population
• This can be a tricky issue – we need to make sure there is a
real need for learning to justify a leave-out group
• But, if there is a real need, then cannot stress enough the
importance of learning for stakeholder benefit
– For-profit companies spend substantial resources to assess their
operations and improve efficiency
– Responsible systems stewardship also involves assessment of
effectiveness
The frame of learning: The case of health care quality in country W
• Government of W wanted to incentivize quality measures in primary health clinics
– Substantial money allocated for program ($20 million in first year) …
– … but many questions over what measures to incentivize, and how much to pay
• At the same time, much resistance to an experimental approach to evaluation
– “We do not experiment on our people”
– Recognized need to learn but aversion to the notion of evaluation
• Solution: there would be no experimentation, or even IE, but there would be selected “learning” clinics to receive the first phase of program
– There would also be no “control” facilities, but we can collect information in non-learning facilities to supplement the learning
Increasing buy-in though transparency: Public randomization
• Kyrgyzstan is implementing a hospital reform program on a pilot basis
– There are 60 maternal hospitals in the country, but pilot implementation funds sufficient for only 20 hospitals
• Every hospital wants to participate (participants receive extra resources)
– In principle, hospital directors accept randomization process as fair
– But don’t trust ministry officials with the process
• Solution: the investigators conducted a public randomization ceremony where (a) process is explained and (b) hospitals selected
– Witnessed by hospital representatives and media
– Results accepted by all
Five “guiding principles” to enhance buy-in
• Generalizable to scale – typically need to replicate successful interventions on a wider scale than study context
• Effectiveness studies with operational orientation – evaluations of programs implemented within existing system capabilities oriented towards learning
• Design regular feed-back to implementers – process monitoring validates the program model and identifies areas that need improvement
• Include a cost focus – to inform policy choices and trade-offs
• Local ownership – core involvement of government and local
investigators in key decisions critical both for effective implementation as well as policy adoption
Supplementing design with process evaluation aids buy-in
• What is process evaluation?
– Documents and describes how the program operates, the services it
delivers, and assesses whether it functions as implemented
– High frequency data gives rapid feedback to serve as input for mid-
course correction and program learning
– Will help explain why we observe success or failure
– Can be either quantitative methods, qualitative, or mixed methods
• Process evaluation contributes to buy-in because it offers
implementers useful operational knowledge
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
26
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
Lições operacionais: Adesão ao projeto
Caroline Cogueto, Secretaria de Meio Ambiente de São Paulo
October 15, 2014
DIME Workshop Lisbon, Portugal
PSA - Mina d'Água
• Projeto de Pagamentos por Serviços Ambientais "Mina d'Água"
• Pagamento em dinheiro a proprietários rurais de bacias prioritárias que se comprometam a preservar ou restaurar nascentes de água
• Pagamento varia de acordo com a importância das nascentes no manancial e seu estado de conservação
Interesse
Porque achamos que haveria bastante interesse:
• Eram áreas que já deveriam ser mantidas sem atividade agropecuária por lei
• O custo de oportunidade da terra é pequeno
– Por exemplo, pasto rende R$300 ano/ha
Avaliação de Impacto 1.0
• O PSA faz com que as nascentes sejam conservadas ou recuperadas?
– Imagens de satélite/fotos aéreas ou vistorias, comparação entre a linha de base (2010) e cinco anos depois (2015)
• O PSA muda a percepção ambiental e o comportamento do proprietário rural?
– Questionários aplicados antes e depois do projeto.
Avaliação de Impacto • Quatro municípios
– No máximo 150 nascentes por município
• Plano A >150 interessados: Sorteio para escolher participantes (aleatorização)
• Plano B <150 interessados:
Regression Discontinuity
Adesão
• Edital lançado: início 2012
• Linha de base: início 2013
• Primeiros contratos assinados: Meio 2014
MUNICÍPIO
NÚMERO DE
PRODUTORES
PRODUTORES
INTERESSADOS
CONTRATOS
ASSINADOS
Guapiara 125 53 12
Ibiuna 130 76 8
7 municípios COM EDITAL LANÇADO 14
12 municípios SEM EDITAL LANÇADO
Gargalos • Arranjo institucional muito caro e trabalhoso
• Desconhecimento e mudança da Lei florestal
Gargalos
• Pré-requisitos para participação: – Possuir matrícula (documento de comprovação da
propriedade);
– Não inadimplentes com o Estado;
– Comprometer-se a realizar a adequação ambiental de toda a propriedade (sem saber muito bem o que isso significava…);
• Desconfiança em relação à ação do governo
Lições aprendidas
• Simplificar arranjo institucional
• Simplificar pré-requisitos de participação
• Melhorar informação sobre as responsabilidades assumidas em aderir ao projeto
Here we are again – IE 2.0: testar como aumentar adesão no projeto para entender se são essas realmente as barreiras de participação
More on Friday…
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
38
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
Karaoke intervention
• PDO: Have fun with workshop participants
• Component: karaoke session
• Assumption: Everybody likes “Karaoke”
• But…. it failed. Why?
did not have enough participants. Low Take Up
What could I do?
Incentives
1. Non-monetary: Autograph + photo
2. Recognition: Name/Photo on DIME website
3. Monetary: $100? $200?
Best singer DIME Workshop Lisbon, 2014
Water and Sanitation Project in Kenya
Project: $427 million Water/Sanitation infra. upgrading
Target: Kayole Soweto, Nairobi: 85,000 pp (2200 compounds)
No Hygiene Campaign
Hygiene Campaign
Low Subsidy (Standard subsidy)
C (600)
---
Medium Subsidy T1 (400)
T3 (400)
High Subsidy T2 (400)
T4 (400)
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
44
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
IE Success
• Success of IE depends on buy-in and take-up
• But even an IE with a rigorous design, full support from government, and 100% compliance in the field, IE can be worthless
• Need sufficient amount of high-quality data to accurately and precisely determine impact
– Sufficient amount: tomorrow
– High quality: today
IE Design
Buy- in
Base line
survey
Mid line
survey
End line
survey
IE analysis & report
Policy decision
IE Timeline
Project implementation
True or false?
“We signed a contract with a firm for a baseline survey. Finally we’ll have some time for other
work, no need to think about the baseline again until we receive the firm’s deliverable: the final
baseline report.”
What could go wrong?
• Wrong HHs interviewed
• Survey instrument missing key indicators
• Massive attrition in panel surveys
• No identification information collected, making follow-up surveys impossible
• High rates of non-response on key questions
• Inconsistent or out-of-range responses
What to do?
• Plan sufficient time for survey preparation
• Develop careful, detailed, TOR for firm
• Monitor field work
• Independent audit
• Collect data electronically
Plan sufficiently
Doing a HH survey well takes a lot of time!
Design Questionnaire (1 - 2 mo)
Procure survey firm (3 – 5 mo)
Develop electronic quest (2 mo)
Pilot survey & confirm data (1 mo)
Collect data (2 – 4 months)
Enter data (2 – 4 months)
Clean data (2 – 3 mo) Analyze data (2 – 3 mo)
Develop careful TORs
• Rigorous testing of survey instrument (especially if electronic survey) and submission of pilot data
• Regular submission of raw data (daily or weekly); test for data quality
• Regular submission of logbooks; set minimum response rate with penalties / rewards
• Submission of complete raw dataset at the end of the survey
• Sufficient information (geo-data) at baseline to track HH in future survey rounds
• Beware: ‘too cheap to be true’
Monitor, monitor, monitor!
• IE Field Coordinator works closely with survey firm every step of the process – Participates in enumerator training
– Observes interviewers
– Monitors fieldwork (unplanned visits)
– Checks data quality & provides feedback
• Best practice: independent audits – small sub-sample
– Confirm interview took place
– Check key data points (especially triggering skips)
Collect data electronically
• Potential to increase data quality a lot – Range & logical checks
– Pre-loading of respondent information
– Skip patterns and interview flow enforced
• Speeds up process – data available as soon as
survey is completed
How much is this going to cost?
• Varies a lot
• Depends on:
– Country (wages, fuel, cost of living)
– Quality and competitiveness of survey firms
– Sampling unit & sample size
– Geographic dispersion of sample
• Per-interview cost for multi-modal agricultural HH survey range from $20 - $150
Integrate IE and M&E data
• Data collection time-consuming and costly
• Do not duplicate efforts!
• Think ahead of time how to use IE surveys to meet M&E needs
• Insure questionnaire covers key results framework indicators
• Develop MIS to link all monitoring, survey, and administrative data
Take-aways
• Do not let bad data be the downfall of your IE!
• Collecting high-quality data is not easy
• But research team will help
– Questionnaire design, data cleaning and analysis
– Field coordinator supports in country, closely involved in survey supervision, builds capacity in your team as needed
OCTOBER 13-17, 2014 | LISBON, PORTUGAL
DIME WORKSHOP
60
tinyurl.com/dime-lisbon
#IEKnowEE
http://www.worldbank.org/en/events/2014/10/01/dime-workshop-energy-and-environment
2 questions, followed by a 15 minute discussion
1. Which presentation is most directly relevant to your project?
2. Which, if any, of the presentations
introduced a new idea that you think is worth exploring in the context of your project?
Which presentation is most directly relevant to your project?
1. Ensuring buy-in
2. Take up
3. Data quality
Which, if any, of the presentations introduced a new idea that you think is worth exploring in the context of your project?
1. Ensuring buy-in
2. Take up
3. Data quality