Impact evaluation in 7 or 8 steps
Jun 20, 2015
Impact evaluation in 7 or 8 steps
Step 1
Engage with the stakeholders
June mission in Mali
• Government interested in a field experiment (RCT) using project and control villages
• Government interested in three main issues:o Good governance: how to make sure that children
are fed?o Education: what is the impact on attendance
rates?o Local economy: how the project benefits small
farmers?
Step 2
Define relevant evaluation questions
Impact on education and nutrition
• Impact on enrolment and achievements, and what is the role played by school quality?
• Does the programme improve attention and cognition?
• Does the programme improve nutritional status? Is there catching-up growth?
• What is the extent of substitution effects within the household?
• What is the impact of the programme on the diet of the poor?
Impact on agriculture
• What is the impact on small farmers in the short term (incomes) and in the long term (farm investments)?
• What is the impact of the intervention on prices and therefore on consumers?
• What is the impact on the wider economy at the village and regional levels?
Step 3
Build a theory of change
Overall programme theory
Agriculture pathways
Education pathways
Nutrition pathways
Step 4
Define the indicators
Welfare outcomes
• Welfare outcomes are the MDGs metrics
Intermediate welfare outcomes
• Sometimes welfare outcomes cannot be observed because:• Occur in the very long term (example increase in
employment)• Are not observable (example maternal mortality)
• Intermediate outcomes are used in these cases: proxy indicators of the final outcomes along the causal chain
Education: outcome indicators
• Enrolment, attendance rates and drop-outs• Achievement tests (test scores on maths and
language)• Attention and cognition
Nutrition: outcome indicators
• Anthropometric measurement.• Measures of diet composition
Food security: outcome indicators
• Full income questionnaire for farmers will provide data on:
marketed surplus Farm profits Technology and capitalisation Input use
Step 5
The evaluation design
The Mali evaluation design
• The government is expanding the intervention to 60 of the most vulnerable communes
• A commune is an administrative unit comprising 5 to 15 villages and a similar number of schools
• The groups considered by the study are: Control group (no intervention) Standard school feeding Home grown school feeding
Level 1 comparison: school feeding-control group
• MOE needs to know the impact of the intervention on educational indicators
• First comparison is between any school feeding and a control group with no intervention
• Outcomes of interests are: Enrolment rates Learning achievements Attention and cognition Nutritional outcomes
Level 2 comparison: school feeding /home-grown school feeding
• The second comparison is between the conventional government programme and the ‘home grown’ programme
• Outcomes of interest: Small farmers’ incomes Overall programme performance
Selection of schools and communes
• In each of the 60 communes Mayors will select two school for the intervention of which one will be randomly assigned to the programme (pair-matching design or stratification by commune). A protocol is designed to avoid contamination.
• Of the 60 communes assigned to the programme, 30 will be randomly assigned to the home grown component
Step 6
Set the sample size
Sample size
• We collected data from 30 households in each village: 20 households with children aged 5 to12 10 farmer households (with or without children)
• Sample size is: 1,200 farmer households 3,600 farmer and non-farmer households 6,000 to 7,000 children of primary school age
Calculate the sample size
• The size needs to be sufficiently large to detect the expected effect of the programme
• Detecting sample size is guesswork and the goal is to produce lower and upper bounds rather than exact samples
• There is statistical software which is designed to do this
Power
• You need a powerful sample to detect impact• The power of your sample will be a function of• The expected programme impact (increasing)• The variance of the outcome of interest
(decreasing)• The homogeneity within clusters (decreasing)• The desired level of Type I error (decreasing)
Step 7
Run a household survey
main issues
• Choose the unit of observation
• Find existing datasets and surveys
• Establish the timing of data collection
• Establish whether collecting cross-section or panel data
• Establish the number of surveys
• Design the questionnaires• Administer the survey
Choose the unit of observation
• The preferred level of observation is the ‘individual’ or the ‘household’ • Note that individuals are difficult to interview (ex:
consumption data or panel)• Household is the most frequent unit of observation
• Observations can also be made at cluster level (village, school, clinic or other)• Note that sample size will be small• Data is difficult to collect (who is interviewed?)
Data scoping
• Before starting collecting any data you should first investigate what data and surveys are available:• Census data can be used to frame the sample or to extract
control variables• Existing household surveys (LSMS or DHS) can be used to
form control groups in matching techniques• Project monitoring data can be used to observe trends• Survey maybe underway in the same areas. This is rarely
the case, but piggybacking is theoretically possible
Timing of data collection
• 3 main issues to consider:• How many surveys will be run?• Baseline, midterm and end-line
• When is the survey starting and for how long?• Seasonality issues need to be considered
• What is the recall period adopted in the questionnaire?• Short recall is more reliable but loses information
Cross-section or panel data?
• A cross-section survey collects data from a population at a point in time
• A panel survey collects data from a population repeated times
• Panel data are preferable because they simplify the analysis• But panel data are not always feasible• But attrition can be large and there can be
differential attrition
Questionnaire design
• Identify the modules that are needed. For example: a household roster, an education module, a consumption module etc.
• Look at existing questionnaire designed by other researchers in similar context
• Examples can be taken from:• LSMS surveys• DHS surveys• Other resources
Running a survey in practice
• Hire a firm with the desired capacity• Ensure enumerators are properly trained and
manuals are available• Test the questionnaires many times • Ensure supervision of enumerators in the field• Ensure households collaborate • Obtain ethical approval
Step 8
Analyse the data