A Guide to Water and Sanitation Sector Impact Evaluationssiteresources.worldbank.org/.../poulos_water_ie.pdf · Guidelines for Impact Evaluations in the Water and Sanitation Sector

Guidelines for Impact Evaluations in the Water and Sanitation Sector

Christine Poulos, Ph.D.Sr. Economist, RTI International

Subhrendu Pattanayak, Ph.D.Sr. Economist and Research Fellow, RTI International

Associate Professor, NC State University

RTI International

June 12, 2007

Motivation

Numerous texts on impact evaluation (IE), in general

Few rigorous IEs in the WSS sector

The Guide:

focuses on specific considerations in evaluating WSH programs and projects; and

supplements existing IE guidelines.See Baker [2000], Ravallion[2001,2005] and others for a comprehensive overview of IE, evaluation designs, or statistical methods

Roadmap

1. IE and IE in WSSInterpret/apply concepts and practices in context of WSS

2. Illustrative Examples of IEs in WSS

3. WSS IEs in practice

What is Impact Evaluation?

IE measures impacts on beneficiaries that are caused by the intervention (programs and policies)

Need to construct counterfactual What would have happened to the beneficiaries in the absence of the intervention?

Identify control or comparison group similar to beneficiaries but for the intervention

Compare comparison group to beneficiaries to measure impact

IE Steps

1. Decide to do an IE

2. Describe the interventionneed/motivation, context, stage of implementation, inputs, and results

3. Design and implement evaluationThe inner workings of a project project-specific issues (Bamberger 2006)

Why decide to do an IE?

Use findings to:

Add to global evidence base

Identify favorable conditions

Identify effective components

Build support

Ensure accountability

Why decide to do an IE?

Select candidates that provide opportunities for feasible, informative studies:

IE has political and financial support, and/or intervention is well-defined

Results generalizable –intervention is scalable, replicable

Interventions that are innovative, controversial, resource intensive

WSS interventions areMulti-dimensional and multi-sectoralRural Development, Private Sector Development, Urban and Local Government, and Environment

Goals guiding most WSS projects:Based on World Bank Water and Sanitation Sector Board guidelines 1. efficient access to safe drinking water and/or basic sanitation

services;2. sustainable access to safe drinking water and/or basic sanitation

services; and3. equitable access to safe drinking water and/or basic sanitation

services.

Describe the Intervention: Objectives of Intervention

WSS

Rural

Env.

BusinessUrban

WSS Interventions

Three types of reform measures:

1. improving operator performance,

2. service provision by the private sector or small-scale independent providers, and

3. decentralized delivery, typically relying on community demand, participation and management.

Outputs of sector reform initiatives:

Hardware: new or improved WSH infrastructure and services

Software: training, education, better provider performance

Describe the Intervention: Features and Linkages

Inputs:

Resources: financial, institutional, legal and regulatory

Activities: what the intervention does

Describe the Intervention: Features and Linkages

Inputs:

Resources: financial, institutional, legal and regulatory

Activities: interventions

Results:

Outputs: direct product of program activities

Outcomes: short-term changes in beneficiaries’ behaviors & knowledge

Impacts: long-term changes in beneficiaries’ wellbeing

Indicators: direct measure of progress toward goals

Also need to understand external influences

Describe the Intervention: Rehabilitating UWSS

Resources Activities Outputs Outcomes Impacts

Funding

Staff

Technical Assistance

Laboratory services

Replacement of pipes, pumps, meters

Trainings for staff

Water quality testing

No. of rehab WSS systems

No. of connections to rehab WSS systems

Operating cost of systems

Water rehab (hrs)

WQ tests

% of pop. with access to WSS

% of pop. served by rehab WSS

Coping costs

Lpcdconsumed

Use of rehab WSS

Individual and household incomes

School enrollment and attendance

Prev. of diarrhealdisease

Under 5 mortality

Design and Implement Evaluation, I

Question: is the intervention effective in increasing efficient, sustainable, and equitable access to improved WSS?

Robust IE design:

Time Pre-intervention

(baseline)Intervention Post-

interventionBeneficiaries (treatment group)

T1 X T2

Control/comparison group

C1 C2

Question: is the intervention effective in increasing efficient,sustainable, and equitable access to improved WSS IEs use baselines, controls/comparisons, and covariates to ensure causal effects can be identified

No baseline, no control – how measure change? what to compare with (i.e., counterfactual)?

Baseline, no control – what to compare with (i.e., counterfactual that captures trends and history)? sufficiently account for selection bias?

No baseline, control – how to sweep out pre-existing differences (behaviors, rates, trends)? sufficiently account for selection bias?

All the above, but no covariates – are you sure nothing else matters? No other factors effect program selection and or modify or mediate treatment?

Design and Implement Evaluation, II

Design and Implement Evaluation, III

Design determines where controls/comparisons come from

Randomized trials

Quasi-experiments – longitudinal or cross-sectional

natural experiments matching (propensity score, covariate, pipeline)

Analysis

Brief Randomized Trial Example

Design – treatment assigned randomly (not purposive, strategic or selective) so that confounders (alternative causes) are balanced across treatment and control group

Example: Do information treatments change hygiene behavior in Delhi, India? [Jalan & Somanathan, 2004]

Tell 500 households (out of 1000) about quality of their drinking water, and check after 1 year if they change hygiene behaviorSimilar in education, health literacy and hygiene behaviorsInformed household 11% more likely to purify water

Challenges: difficult to control, ethical concerns, political issues, and limited external validity

Brief Quasi-Experimental Example

Design: match treatments to controls based on observable factors

Example: Does Jalswarajya – a public, community-driven rural WSH program in Maharashtra, India – improve access to improved WSH and improve children’s health outcomes?

~250 villages (2 propensity score matched controls for each treatment)Baseline and post-intervention data collection from ~10,000 householdsHousehold and community surveys

Challenges: need lots of data, assumes unobservables are uncorrelated with exposure to intervention

OR IS S A

GA N J AM

KO R A PU T

PH U LB A N I

AN U G U L

KE O N JH A R

M A YU R B H A NJ

RA Y G AD A

PU R IKA L AH A N D I

SU N D AR G A R H

BO L AN G I R

SA M B A LP U R

BO U D

BA R A GA R H

M A L K A N G IR I

GA J AP A TI

NU A P AD ACU T TA C K

J A J P URDH E N K AN A L

BA LA S O RE

NA Y A G AR H

NA B A RA N G A PU R

DE O G AR H

KH U R D H A

SO N P UR BH A D RA K

KE N D RA P A DA

JH A RS U G U DA

JA G AT SI N G H PU R

#Y

#Y

#Y#Y

#Y#Y

#Y#Y#Y

#Y

#Y#Y

#Y #Y#Y

#Y

#Y#Y

#Y

#Y

#Y

#Y#Y#Y #Y

#Y#Y #Y

#Y#Y#Y#Y

#Y

#Y

#Y

#Y

#Y#Y#Y

#Y

#Y

#Y #Y

#Y #Y#Y

#Y#Y

#Y

#Y#Y

#Y

#Y

#Y #Y

#Y

#Y

#Y#Y

#Y

#Y

#Y#Y#Y

#Y#Y

#Y

#Y#Y

#Y

#Y#Y

#Y #Y

#Y

#Y#Y

#Y#Y

#Y#Y

#Y

#Y

#Y

#Y

%[

%[%[ %[

%[

%[

%[

%[

%[

%[

%[

%[

%[%[

%[%[

%[%[

%[

%[

%[

%[%[

%[

%[

%[

%[

%[

%[ %[

%[%[%[

%[%[

%[

%[

%[

%[

%[%[

%[

%[%[

%[

%[

%[%[

%[

%[

%[

%[

%[

%[

%[

%[

%[

B O N TH

B H A D R A K

B H A N D A R IP O K H A R I

D H A M N A GA R

TIH I D I

B A S U D E V P U R

C H A N D A B A L I

O R IS S A

B H A D R A K

M a p S h o w in g B h a d ra k D is tr ic tN

EW

S1 0 0 1 0 K m .

B L O C K B O U N D A R YRiv er

RO A DNA TI O N A L H IG H W A YM A JO R R O A DO T H E R R O A D

Dr a in a g e

L EG E N D :

Detailed Example: Randomized Trial of TSC’s IEC in Orissa, India

Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, I

Government of India’s Total Sanitation Campaign

Goal: increase use of IHLsBackdrop: inadequate services, high child mortality, and MDGs

“Community-Led Total Sanitation”

Intensive IECChange knowledge Change attitudesCommunity demand and plan

Small subsidies to the poor; know-how and material to allImplement via local NGOs

Randomized Trial of TSC’s IEC in Orissa, India: Describe Intervention, II

Resources Activities Outputs Outcomes ImpactsFunding: Bank,

GoI, GoO, community contributions

Staff: GoO, NGOs, community monitors

Supply Chain

Technical assistance

Enabling programming -TSC

IEC: walk of shame, fecal calculation, defecation mapping

Distribute subsidies

Training in IHL construction

Establishment and/or stocking of rural sanitation marts

No. of focus groups and completed IECs

No. of community agreements

Total subsidies

No. of households trained in construction

No. of rural sanitation marts

% of households owing IHL

% of pop. using IHL, by gender and age

Coping costs-time

Awareness of environment-health link

Prevalence of diarrhealdisease, by age and gender

Norms regarding OD

Individual and household incomes

School enrollment and attendance

Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, I

Research Questions: Does the TSC’s IEC cause

increases access to and use of IHL? decreases in child morbidity due to diarrheal disease?

Features:Controls: Leverage phasing to randomly assign IEC to 20 out of 40 communities in 1st phaseBaseline: pre-intervention surveys Covariates: household and village surveys Indicators: selection based on literature, pretestsSample: ~1000 households

Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, II

0

10

20

30

40

%

Treatment Control

IHL Ownership and Use

20052006

05

101520253035

%

Treatment Control

U5 Diarrha Rate

20052006

Randomized Trial of TSC’s IEC in Orissa, India: Design and Implement IE, III

0

100

200

300

400

Rs.

Treatment Control

Medical Costs (Rs)

20052006

0

10

20

30

Min.

Treatment Control

Time Spent Walking (min)

20052006

Detailed Example: Quasi-Experimental Study

Galiani, Gertler, and Schargrodsky(2005): Describe Intervention, I

Intervention: Privatization of local water companies in Argentina

In the 1990s, about 30% of the country’s municipalities covering almost 60% of the country’s population were privatized. The remaining municipalities continued receiving water services from either public companies or nonprofit cooperatives.

Research question:

While efficiency gains have been demonstrated, does privatization of water supply improve health outcomes and alleviate poverty?

Galiani, Gertler, and Schargrodsky(2005): Describe Intervention, II

Resources Activities Outputs Outcomes ImpactsEnabling

regulation

Private sector resources

Access to credit

New incentives

PSP process and contracts

New systems for billing, accounting, maintenance

Installation or replacement of pipes, pumps, meters

Training

Efficiency in billing and collection

Improved service

System performance (e.g., hours of service)

Water and sewerage connections , by municipality-level poverty rate†

Under five mortality rate, by municipality-level poverty rate

Galiani, Gertler, and Schargrodsky(2005): Design IE, I

Research Questions:Does privatization of water supply improve health outcomes?

Features:Comparisons: Propensity score matching of municipalitiesBaselines: secondary data Covariates: secondary data on socioeconomic and political characteristics of municipalitiesIndicators: U5 mortality rate, water and sewerage connection rate, private ownership of water company serving majority of municipalities' populationSample: ~4000 municipalities

Galiani, Gertler, and Schargrodsky(2005): Key Findings

Analysis:Difference-in-difference estimation with PSM

Findings:Child mortality fell by 8%

Effect was largest in poorest areas (26%)

Connections to the water network increased by 4.2%

Challenges

Short project cycles1. Measuring sustainability within the project cycle

“no less than 3-5 years are required for an intervention to show an impact.”(Habicht et al. 1999)Use program theory (White 2005)

2. Turnover, impatience, fatigue3. Devarajan and Kanbur (2004) “In short, we probably need to scale up

something that is in short supply, namely, patience.”

Constraints – Bamberger (2006)BudgetTimeDataBamberger discusses the acceptability of compromises

Strengthen Overall Quality

Allow sufficient time to meet with clients and key stakeholdersBuild political and institutional support early and often – counteract turnover, fatigueConvince them to stay the course

Get off to a quick start

If possible, change program design to allow for less expensive IE methods

If possible, make mid-course corrections

Develop program theory/logic modelConsensus on how it work, identify where it breaks down, determine what to measure

Alternative data collection methods

Conclusions

Call for increase in use of rigorous IE methods Address project-specific issues in selection of study design and features

Baseline, controls/comparison, and covariates are key

Endline is equally important! Engage and encourage stakeholders to stay the course

The End

Contact information:

[email protected]

[email protected]