Session 4 Randomized evaluation design DFID, Malawi 1
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Session 4
Randomized evaluation design
DFID, Malawi
1
• Session 1: Introduction to Impact Evaluations
• Session 2: Theory of Change and Measurement
• Session 3: Group work: Theory of change
• Session 4: Randomized evaluation design
Workshop Schedule – Day 1
2
• Threats to Validity – Partial compliance
– Externalities
– Attrition
• Unit of Randomization
• Treatment Variations
• Method of Randomization
• Sample Size
Outline
Threats to Validity
• Remember, randomization ensures that
– On average, those in the treatment group are the same as those in the control group
– This means that the impact will be the same if either one of the groups were treated
• From Session 1, we saw what random assignment might look like…
How to Randomize, Part I - 5
Random assignment
2006
Average income per person, per day, kwachas
1000
500
0 Treat Compare
170 168
A successful randomized evaluation requires:
1. Those in the treatment group (red) receive the treatment, while those in the control group (blue) do not
1. Those in the control group are not affected by the treatment in any way
2. Outcomes for members of both treatment and control are measured at the end of the programme
Threats to Validity
When these requirements aren’t fully met: 1) Those in the treatment group (red) receive the treatment, while those
in the control group (blue) do not
2) Those in the control groups are not affected by the treatment in any way
3) Outcomes for members of both treatment and control are measured at the end of the programme
Threats to Validity
→ PARTIAL COMPLIANCE Individuals assigned to the treatment group may not receive the programme or those in the control group do receive it
→ EXTERNALITIES Individuals or communities which did not receive the treatment may nonetheless be affected by it, either positively or negatively
→ ATTRITION Members of the original sample drop out between the beginning of the programme and the endline survey. The subsample may differ systematically from the remaining sample
Partial Compliance
• Definition:
– Some in treatment group DON’T receive intervention
– Some in control group DO receive intervention
• Consequence: – May not detect an effect even if treatment is effective
• Example: – Primary education management in Madagascar: in
the bottom up approach, teaching tools were sent to schools. What if some were delivered to control schools?
8
Spillovers
• Definition:
– Those not in the treatment group are affected (positively or negatively) by the intervention
• Consequence:
– Effect of treatment is measured to be smaller or larger than it actually is
– May negatively affect policy if evidence is used to make decisions
• Example: – Primary education management in Madagascar:
Teachers in neighbouring schools hear about the monitoring program and change their behaviour
9
Attrition (I)
• Definition:
– Failure to collect outcome data from some individuals who were part of the original sample
• Reasons for attrition: – Move house/area
– Refuse to answer endline survey
– Sickness and death
• Examples:
– Primary education management in Madagascar: What would happen to the measured effect if students with low scores in control schools drop out during the intervention?
10
When is attrition particularly problematic? • Random attrition is not a major problem • However, non-random attrition will cause the treatment effect to be calculated incorrectly. Why?
→ People who drop out might be different in some important way to those who don’t drop out.
→ If they are different, then treatment may affect them differently
Attrition (II)
• Threats to Validity
• Unit of Randomization – Individual versus cluster
• Treatment Variations
• Method of Randomization
• Sample Size
Outline
• Unit of randomization:
– The unit for which we ‘flip the coin’
• We can randomly select individuals to take part in an intervention, or we can select whole groups, also known as clusters:
Unit of Randomization
Unit of Randomization: Individual?
Unit of Randomization: Individual?
“Groups of individuals”: Cluster Randomized Trial
Unit of Randomization: Household?
Unit of Randomization: Household?
Unit of Randomization: School?
Unit of Randomization: School?
• Nature of the treatment – How is the intervention administered? – How many people are likely to be affected by intervention?
• Generally, best to randomize at the level at which the treatment is administered – Madagascar: intervention is at the school level so schools
randomized
• BUT there are practical concerns: E.g. randomly assign schools to receive teaching tools
→ Contamination: can we prevent teachers from sharing resources with other schools? → Fairness: Do school principals / teachers / parents agree to our research design?
How to Choose the Level
• Threats to validity
• Unit of Randomisation
• Method of Randomisation • Lottery
• Phase-in design
• Encouragement design
• Treatment variations
• Sample size
Outline
• Suppose there are 2000 (eligible) applicants for a public service project, but only enough resources for 1000 participants
• Randomization can serve the purpose of selecting in a fair way and help us to evaluate
• Randomization mechanisms: – Pull out of a hat/bucket
– Use a computer programme
(e.g. Stata) to generate random
numbers
I. Lottery
• Advantages
– Lotteries are simple, common and transparent
– Not as politically problematic as often claimed
– Participants know the “winners” and “losers”
– Useful when there is no good reason to discriminate
– Perceived as fair
I. Lottery
• Over a period of time, extend intervention to entire population
• Natural approach when expanding programme faces resource constraints
Advantages • Everyone gets something eventually • Provides incentives for those in control group to maintain
contact
Concerns • Can make it difficult to measure long-run effects • Do expectations of future receipt change actions today?
II. Phase-in design
II. Phase-in design
Round 1 Treatment: 1/3 Control: 2/3
Round 2 Treatment: 2/3 Control: 1/3
Round 3 Treatment: 3/3 Control: 0 1
1
1 1
1
1
1
1
1
1 1
1
1
1
2
2
2 2
2
2
2 2
2
2
2
2 2
2
2
2
3
3 3 3
3
3
3
3 3
3
3 3
3
3
3 3
3
Round 1 Treatment: 1/3 Control: 2/3
Round 2 Treatment: 2/3 Control: 1/3
Randomized evaluation ends Randomized evaluation ends
• Sometimes it is practically or ethically impossible to randomize programme access
• Instead, randomize encouragement to receive treatment
• Encouragement = something that makes some people more likely to use programme than others
III. Encouragement design
III. Encouragement design
Encourage
Do not encourage
Participated
Did not participate
Complying
Not complying
Compare encouraged to not encouraged
Do not compare participants to non-participants
• Lottery
• Phase-in design
• Encouragement design
The best randomization mechanism depends on the nature of the intervention and the environment of the study
Method of Randomization
Summary
Not mutually exclusive
• Threats to Validity
• Unit of Randomization
• Method of Randomization
• Treatment Variations – Multiple Treatments
– Cross-cutting Treatments
– Varying Intensity of Treatments
• Sample Size
Outline
• Often there are multiple ways of achieving a desired outcome
E.g. Reducing corruption in road building in Indonesia:
1) Audit of project by govt. officials
2) Community monitoring through meetings
• Design: To find out which intervention is more effective, we can have multiple treatment groups
Multiple Treatments
Govt. audit
Community meetings
Control
Multiple Treatments
• Sometimes interventions are most successful when used in combination
Continuing with the previous example, we could add a third group which gets both interventions:
1) Audit of project by govt. officials
2) Community monitoring through meetings
3) Audit + community monitoring
• This can show us the most cost-effective way to reduce corruption
Cross-cutting treatments
Govt. audit
Community monitoring
Audits + community monitoring
Control
Cross-cutting Treatments
• The intensity of the intervention might also be relevant. Why?
Community monitoring of road construction:
- If corruption is low/easily discouraged, might need only 1 meeting
- If corruption is high/persistent, might need multiple, frequent meetings
• Extra cost of meetings might be justified by savings from reduced corruption
Varying Intensity of Treatments
• Threats to Validity
• Unit of Randomization
– Individual versus cluster
• Treatment Variations
• Method of Randomization
• Sample Size
Outline
Sample size
• How large does the sample need to be to “credibly” detect a given treatment effect?
• What does credibly mean?
• Randomization removes bias, but it does not remove noise
• But how large must “large” be?
Sample size
• Answers depend on a number of factors, including:
– Level of randomization
– Expected effect size
– Underlying variability of the population
– Randomization design
Sample Size Examples
Study # of interventions (+ Control)
Total Number of Clusters
Total Sample Size
Women’s Empowerment 2 Rajasthan: 100 West Bengal: 161
1996 respondents 2813 respondents
Pratham Read India 4 280 villages 17,500 children
Pratham Balsakhi 2 Mumbai: 77 schools Vadodara: 122 schools
10,300 children 12,300 children
Kenya Extra Teacher Program
8 210 schools 10,000 children
Deworming 3 75 schools 30,000 children
Bednets 5 20 health centers 545 women
Related Documents
