1 Measuring Results and Evaluating Impact: What You Need to Know Paul Gertler and Barbara Bruns Human Development Network November 2006.

1

Measuring Results and Evaluating Impact:

What You Need to Know

Paul Gertler and Barbara BrunsHuman Development Network

November 2006November 2006

2

How do we turn this teacher…

3

…into this teacher?

4

What do we need to know??

• How many teachers • How many are performing like teacher A, like

teacher B, or absent altogether• How many kids:

– enrolled in school– attending school– repeating grades– dropping out – completing primary school

• Trends over time: are things moving in right direction?

5

What do we need to know??

• Student learning – which kids are learning and how much? – who is benefiting?

• Costs (per student) • What inputs, policies and incentives have greatest

impact on learning? – remedial teachers, multi-grade teacher training,

deworming pills– higher teacher standards, curriculum focused

on early grade reading– contract teachers, school-based management,

vouchers• Which are most cost-effective?

6

Message 1: Monitoring and Evaluation is our core business

• All TTLs need tools and techniques for– Better monitoring of Bank/IDA operations:

• defining project development objectives clearly• credible results chains • selecting indicators

– Building client capacity for M&E – Impact evaluation

• Both are important, and some complementarities– Standardized student learning data and good

administrative data (health facility surveys, school census data) lower costs and improve quality of impact evaluations

7

Message 2: We’re not doing a very good job at present

• All Networks struggling with need for better monitoring and more impact evaluation– INF study: over 200 different indicators in water supply

projects, measuring similar things, most w/only 1 obs.

– OPCS study: 78 education projects exiting in FY04-05, • Only 3 reported MDG (primary completion rate)• Only 5 had any data on student learning – and none

comparable• Less than 50% tracked enrollments (mix of GER,

NER, raw numbers, gender)– In S Asia HNP projects, avg. operation had 26 indicators,

only 33% with clear definition, baseline and collection schedule

8

Little Progress of M&E Over Time(from Ben Loevinsohn and Aaka Pande)

0

10

20

30

40

50

60

70

80

90

Indicatorsmeasurable

Clear target Baseline datacollected

Follow-up datacollected

Data collectionroughly in

keeping withPAD

Before 2000

After 2000

9

Message 3: HD under special pressure with “results agenda”

• Greater accountability for major global goals– 2/3 of MDGs are HD outcomes– HD outcomes important in IDA 14 results

• build country capacity to measure these outcomes

• link PDOs to “higher level” sectoral or MDG outcomes

• Greater role in “selling” aid – OECD electorates focused on HD issues – selling IDA 15 on “results”, esp. HD results

10

Message 4: there is help….• Tools

– Checklist for M&E in HNP operations, also education– Guidance on indicators (AFR Generic Results

Framework for HIV projects)– Common instruments for classroom observation and

student learning assessment • Technical advice

– “on demand” impact evaluation clinics– FPSI “on demand” PDO/monitoring clinics

• Financing– BNPP funding to launch prospective impact evaluations

• Training

11

Plan for next 3 daysTODAY• Understanding the value of impact evaluation• Understanding the results agenda from an institutional perspective• TTL experience in improving M and E• 4 hour module on Improving “M” -- results focus in projects

TUESDAY • TTLs will meet in MC-5-100• Researchers will meet in IFC (F-5-P100)• All day on impact evaluation – methods, operational issues

WEDNESDAY• Everyone will meet in MC-4-100• Teams organized along sector lines will prepare evaluation designs

for projects of their choice & write concept notes• Teams present in evaluation clinics