Quality of Economic Analysis in World Bank Education Projects Harry Anthony Patrinos March 6, 2008.

Quality of Economic Quality of Economic Analysis in World Bank Analysis in World Bank

Education ProjectsEducation Projects

Harry Anthony PatrinosHarry Anthony Patrinos

March 6, 2008

Why Economic Analysis?Why Economic Analysis? Improve quality of portfolio

Reduce unsatisfactory projects under supervision

Research indicates that: 1. Strong relation between quality of cost benefit/cost-effectiveness

analysis & project outcomes indicative of importance of good quality analysis at design

2. Quality of fiscal impact analysis, beneficiary assessment, poverty analysis & overall economic analysis related to performance

3. Poorly prepared projects more likely to perform worse Bank-wide, projects rated poor 4X more likely to be unsatisfactory than

those rated good – consistent across regions

Vawda, A., P. Moock, J.P. Gittinger, H.A. Anthony Patrinos. 2003. “Economic analysis of World Bank education projects and project outcomes.” International Journal of Educational Development 23: 645–660.

“Rate of Return”

Not Applicable

“Sector” Ratesof Return

“Sector” RoR and “Project-Specific” C/B

Project Specific

RoR

Pre-1990s ECONReports

1992

Priorities andStrategies,

1995

2000

First Economicsof Education

Course

Ranking Education Projects

Evolution of the Application of Evolution of the Application of Economics in Education ProjectsEconomics in Education Projects

2008

Overall Ranking of Projects in Overall Ranking of Projects in the Education Sector the Education Sector ImprovedImproved

0

20

40

60

80

100

Good andAcceptable

Marginal andPoor

CY93

FY98

Recent PerformanceRecent Performance

Good & acceptable

Good & acceptable

Good & acceptable

Marginal & poor

Marginal & poor

Marginal & poor

0

20

40

60

80

100

2005 2006 2007

Recent PerformanceRecent Performance

2005 and 2006 no better than early 1990s Fortunately 2007 performance as good as late

1990s However, too many are just acceptable

Too Many Projects are Merely Too Many Projects are Merely AcceptableAcceptable

2005

2005

2005

20062006

20062007

2007

2007

0

20

40

60

80

100

Good Acceptable Marginal

Characteristics of FY07 Characteristics of FY07 Education ProjectsEducation Projects

28 loans approved $2.0 billion

$507 million IBRD; $1.5 billion IDA (1 project for additional funds, no PAD, so not included)

Assessment 7 (26%) good 18 (67%) acceptable 2 (7%) marginal None poor

1. Linkage with ESW

2. Quantitative analysis of alternatives

3. Financial & fiscal impact assessment

4. Cost-benefit and cost-effectiveness analysis

5. Sensitivity analysis6. Institutional analysis

7. Poverty impact analysis

8. Environmental assessment

9. Outline of economic performance indicators/criteria

10. Overall project justification

10 Dimensions of Economic 10 Dimensions of Economic AnalysisAnalysis

Table 1 . Ranking of Characteristics in FY07 Education PADs and Project Documents

Projects (#) % in Category

Characteristic/Ranking 1 2 3 4 1 2 3 4

Overall quality 7 18 2 0 26% 67% 7% 0%

Project coherently set in ESW and CAS 25 2 0 0 93% 7% 0% 0%

Rationale for public sector involvement 7 12 8 0 26% 44% 30% 0%

Evidence of consideration of alternatives 11 10 6 0 41% 37% 22% 0%

Fiscal impact analysis 15 7 5 0 56% 26% 19% 0%

Quality and clarity of economic analysis 9 6 12 0 33% 22% 44% 0%

Clarity of price and quantity assumptions 0 13 14 0 0% 48% 52% 0%

Institutional risk analysis 17 10 0 0 63% 37% 0% 0%

Beneficiary assessment and poverty analysis 15 11 1 0 56% 41% 4% 0%

Testing for learning performance* 15 5 5 0 60% 20% 20% 0%

1 = Good. 2 = Acceptable. 3 = Marginal. 4 = Poor.

*In two projects it was judged that testing for learning performance was not applicable.

Good Practice: Overall QualityGood Practice: Overall Quality Ecuador Inclusion and Quality Education Guatemala Education Quality and Secondary

Education Vietnam Second Higher Education

Good Practice: Consideration of Good Practice: Consideration of AlternativesAlternatives

Argentina Lifelong Learning and Training India Vocational Training Improvement Nigeria Science and Technology Education

Post-Basic

Good Practice: Fiscal Impact Good Practice: Fiscal Impact AnalysisAnalysis

Burundi Education Sector Reconstruction Haiti Education for All Kenya Education Sector Support Nepal Second Higher Education

Good Practice: Quality and Good Practice: Quality and Clarity of Economic AnalysisClarity of Economic Analysis

Argentina Lifelong Learning and Training Ecuador Inclusion and Quality Education Guatemala Education Quality and Secondary

Education Nepal Second Higher Education Vietnam Second Higher Education

Testing for Learning Testing for Learning PerformancePerformance

3/5 of projects include provision for testing & judged good

Another 1/5 judged to acceptable

Testing for Learning Testing for Learning PerformancePerformance

3/5 of projects include provision for testing & judged good

Another 1/5 judged to acceptable An example of a good approach is:

Bangladesh Third Programmatic Education Sector Development Program – support annual student assessments (based on internationally recognized assessments such as TIMSS) piloted in Grades 8 and 12

Testing for Learning Testing for Learning PerformancePerformance

3/5 of projects include provision for testing & judged good

Another 1/5 judged to acceptable An example of a good approach is:

Bangladesh Third Programmatic Education Sector Development Program – support annual student assessments (based on internationally recognized assessments such as TIMSS) piloted in Grades 8 and 12

Several projects support development of national testing programs Georgia Education System Realignment and Strengthening

Project – help to establish a system of student assessment so as to identify difficulties in student learning and develop remedial programs accordingly

Type of Assessment by Fiscal Year

50.0% 50.0%

44.4%

50.0%

0.0%

100.0%

75.0%

88.9%

72.2% 71.4%

0.0%

8.3%

44.4%

16.7%

42.9%

0.0%

41.7%

22.2%

55.6%

71.4%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

2002 2003 2004 2005 2006

Fiscal Year

School-managed assessment

National assessment

Regional assessment

International assessment

Impact EvaluationImpact Evaluation FY06: Only 4 had any discussion of evidence or plan for

evaluation FY07: 2/5 (10 of 27) projects make reference to

proposed impact evaluation In several it was clear that a rigorous impact evaluation

envisioned LAC: Argentina, Guatemala, Haiti SAS: Bangladesh ECA: Bulgaria AFR: Kenya EAP: Vietnam In others, however, unclear just how rigorous an evaluation was

proposed

Towards Better Economic Towards Better Economic AnalysisAnalysis

CBA is still a potentially important tool in the economists’ arsenal

Too many projects lack adequate economic analysis More work needs to be done on analyzing alternatives

for project Important research agenda for future is to measure

effects of interventions on learning outcomes

Today’s AgendaToday’s Agenda Appropriate public sector roles Project versus sector justification Impact evaluation Introduction to incorporating learning

outcomes

Today’s AgendaToday’s Agenda Appropriate Public-Private Roles: Emmanuel Jimenez

Project versus Sector Justification: Price Gittinger

Keynote Speaker: George Psacharopoulos

Impact Evaluation: Felipe Barrera

Impact Evaluations in Difficult Circumstances: Tazeen Fasih

Incorporating Learning Outcomes in Project Objectives Panel: Eduardo Velez, Elizabeth King, Marguerite Clarke, Jee-Peng Tan, Robin Horn