Maidan Summit 2012 Dr. Jyotsna Puri, 3ie

Using impact evaluations in sport: An art and a science

Dr. Jyotsna PuriHead of Evaluation

Deputy Executive Director, 3iewww.3ieimpact.org

PROGRESA

So what was different with Progesa?

• Why was it different?• Why did policy

makers take note?• ……• It was an IMPACT

EVALUATION!• ….• What is an IMPACT

EVALUATION??

Providing sports programs to disadvantaged youth

Before After

Project (treatment) 92

comparison

What do we need to measure impact?

Before versus after single difference comparisonBefore versus after = 92 – 40 = 52

Before After

Project (treatment) 40 92

comparison

But this could have happened in the absence of the program?!

The project led a larger number of girls and boys to higher levels of health.

Post-treatment comparisonSingle difference = 92 – 84 = 8

But we don’t know if they were similar before…

Before After

Project (treatment) 92

comparison 84

Before After

Project

Comparison

What do we need to measure impact?

Basic Setup of a Randomized Evaluation

Potential Participants

Evaluation Sample

Random Assignment

TreatmentGroup Comparison

Group

Participants No-ShowsBased on Orr (1999)

RANDOMIZATION RANDOMIZATION RANDOMIZATION

RANDOMIZATION RANDOMIZATIONRANDOMIZATION RANDOMIZATIONRANDOMIZATION RANDOMIZATION

RANDOMIZATION RANDOMIZATION RANDOMIZATION RANDOMIZATION RANDOMIZATION

RANDOMIZATION

RANDOMIZATION

Evidence on the impact of sport on development has been scarce.

An impact evaluation examining the impact of sport

Did the program cause an increase in development indicators amongst program participants?

Were the effects on boys and girls the same?

What are the necessary/sufficient conditions for success?

Can this be extrapolated to other circumstances/contexts?

2010 study on sports - Conclusions

Does sport contribute to personal development and well being of disadvantaged youth?•Unclear and non-significant impact on self-esteem and self-efficacy.•Peer leaders (South Africa: yes, Magic Bus: unclear)•HIV/AIDS, gender beliefs?

Evaluating Sports and development : A need for better thinking

• Ill defined program theory.• Small sample sizes• Survey samples powered

for heterogeneous impacts (e.g. girls vs. boys, starting levels of self-esteem)

• Before vs. after but comparisons not chosen well.

Improving Lives through Impact Evaluation

Improving the supply side and the demand side of evidence based developments

3ie

3ie sub-grant windows• Funds ‘IMPACT’

evaluations and provides technical expertise;

• Necessary attributes:- Counterfactual analysis- Policy relevant, LIMCs- Good program theory- Capacity development- High technical quality- Use mixed methods - Policy impact and

potential for scaling up

Policy impact!?!

We take sophisticated techniques

Which make nice academic studies

Which get published in top journals

Bridging the gap: Bringing facts to policy makers and program managers

• School based deworming: cuts absenteeism by one-fourth and costs only 50 cents/child/year.

• Conditional cash transfer programs: reduce poverty and increase level of secondary school enrollment.

• Financial performance incentives: can improve use and quality of health services.

Policy window grants

Preparation grants:

Policy Window

Sri Lanka, Philippines, Fiji, Ecuador, South Africa, NEPAD, Uganda etc.

3ie: State of the art expertise• More than 100 impact

evaluations.• Systematic reviews• Policy impact• More than 50 countries• Other:

– Staff expertise– Experts network– Training– Monitoring– In-house research

Impact Evaluations

– Efficacy and design– Effectiveness of

interventions. – Cost effectiveness of

interventions– Trade-offs and

unintended consequences.

WWW.3IEIMPACT.ORGFACEBOOK/TWITTER

Thank you.jpuri@3ieimpact.org

#jo_puri