Monitoring and Evaluation of International Development Assistance to the Private Sector

Monitoring and Evaluation of Private Sector Development

Hiroyuki Hatashima

Private Sector Evaluation, Independent Evaluation Group (IEG)

World Bank Group

Canadian Evaluation Society Conference

June 11, 2013

2

IEG is part of a larger array of evaluation and control mechanisms

World Bank Group President / Management

Management’s Internal Accountability Framework

And Systems• Self Evaluation

• Monitoring (QAG, DOTS, AS)• Institutional Integrity

• Internal Audit• Compliance Advisor/ Ombudsman (IFC,

MIGA)

Independent Evaluation

Group

Inspection Panel

Board ofExecutive Directors

IEG seeks to address both accountability and learning

• Accountability for results:– Is the Bank Group doing what it says? – Is it achieving its objectives? – To what extent is it contributing to development outcomes?– With what results and development impact?

• Learning:– What projects and programs achieve results – relevance,

efficacy, and efficiency – and why?– How can M&E be strengthened in the Bank Group and in

client countries?

3

Three main messages

Evaluation Across Boundaries• Private Sector operations have specific data

gathering advantage/disadvantage based on their business model

• M&E systems should adopt to the business practices to be effective and efficient

• M&E can influence learning, quality of work and outcome

Outline of Presentation

1. Context of Private Sector Development Assistance

2. Types of World Bank Group Private Sector Development interventions

3. Technical Assistance projects M&E

4. Private Sector Investment projects M&E

5. Use and Influence of M&E

1.1 Context: Private Sector As Important Player

1. The private sector has taken on an essential role in contributing a wide range of development issues.

2. Potential of private sector to contribute to development solution is still underutilized.

3. M&E essential for track progress, assess outcome & performance and generating knowledge of what works, what does not, and why.

Are the monitoring and evaluation systems of IFC and MIGA equipped to inform the organizations on their performance and results?

Systems functions

• To what extent does the mechanism in place ensure that M&E systems generate credible, timely, and relevant information?

Use and Influence

• To what extent does M&E information support evidence-based decision making and learning?

Effectiveness and Efficiency

• What has been the impact of the M&E outputs and use on project quality and development outcomes?

1.2 Assessing M&E systems – Evaluation Questions

Outline of Presentation

1. Context of Private Sector Development Assistance

2. Types of World Bank Group Private Sector Development interventions

3. Technical Assistance projects M&E

4. Private sector Investment projects M&E

5. Use and Influence of M&E

The World Bank Group consists of five closely related institutions:• IBRD: The International Bank for

Reconstruction and Development• IDA: The International

Development Association• IFC: International Finance

Corporation• MIGA: The Multilateral Investment

Guarantee Agency• ICSID: The International Center for

the Settlement of Investment Disputes

1. Types of WBG Private Sector Development interventions

1. Interventions through the Public Sector to influence the Private Sector World Band Financial and Private Sector

Development projects, IFC’s Advisory Services projects to government

(Investment Climate, PPP)

2. Direct interventions in partnership with the private sector and public entities operating commercially IFC’s investments and MIGA’s guarantees

3. Blend interventions IFC’s Advisory Services to private companies, but

with a strong public good component

Outline of Presentation

1. Context of Private Sector Development Assistance

2. Types of World Bank Group Private Sector Development interventions

3. Technical Assistance projects M&E

4. Private Sector Investment projects M&E

5. Business Models and M&E

12

3.1. Technical Assistances follow logical model results measurement

ImpactOutcomesOutputsActivities Inputs

Resources used for advisory

services projects

Resources used for advisory

services projects

Programs associated with an

advisory project

Example:

– Provide advisory

services to local companies

Programs associated with an

advisory project

Example:

– Provide advisory

services to local companies

Immediate deliverables of

the project

Example:

– Diagnostic report and

recommendations made

– 10 managers were trained

– Legislation drafted

Immediate deliverables of

the project

Example:

– Diagnostic report and

recommendations made

– 10 managers were trained

– Legislation drafted

Changes in knowledge,

behaviors, and attitudes as a

result of intervention with

attribution of changes to the

project Example:

– Legislative or inst reforms

– Changes in agricultural

techniques, or CG practices

Changes in knowledge,

behaviors, and attitudes as a

result of intervention with

attribution of changes to the

project Example:

– Legislative or inst reforms

– Changes in agricultural

techniques, or CG practices

Consequences resulting from an

intervention including

improvements in clients’

performance and effects beyond

direct beneficiaries

Example:

– increased investment, sales, jobs

Consequences resulting from an

intervention including

improvements in clients’

performance and effects beyond

direct beneficiaries

Example:

– increased investment, sales, jobs

Track indicatorsProject Supervision Report

(PSR)

Supervise

Approve Disburse Completion

Commit

M&E system

Logframe,Identify indicators Project

Completion Report

(PCR)

3.2. M&E follows project cycle

Development Effectiveness FrameworkDevelopment Effectiveness Framework

Strategic Relevance

Was it the right project at the

right time?

Strategic Relevance

Was it the right project at the

right time?

Output Achievement

Were “key” deliverables achieved? To

what extent were clients satisfied

with the advisory service?

Output Achievement

Were “key” deliverables achieved? To

what extent were clients satisfied

with the advisory service?

Outcome Achievement

Were intended short & medium-term changes in

behavior achieved?

Outcome Achievement

Were intended short & medium-term changes in

behavior achieved?

Efficiency

Were costs reasonable vis-à-

vis results?

Efficiency

Were costs reasonable vis-à-

vis results?

Impact Achievement

Were intended longer-term effects achieved?

What would have happened without the intervention? (What is the counterfactual?)

Impact Achievement

Were intended longer-term effects achieved?

What would have happened without the intervention? (What is the counterfactual?)

• These combine into a “synthesis” rating

In addition: Assessment of IFC Role & Contribution: – Was IFC particularly catalytic or innovative in its advisory services?

– Was IFC able to play a unique role not easily filled by others?

• These combine into a “synthesis” rating

In addition: Assessment of IFC Role & Contribution: – Was IFC particularly catalytic or innovative in its advisory services?

– Was IFC able to play a unique role not easily filled by others?

3.2. Advisory Services Project Completion Report as Self-Evaluation

3.4 Technical Assistance M&E –ways to adopt to business practices

Progress Integration with project

cycle M&E officer playing important

and formal roles in quality assurance

Quality control cycle established

M&E officers sign-off, Development Impact Dept reviews

M&E quality improved Quality of logical framework,

baseline data, quality of results discussion – all improved

Challenges Standard indicators not

sufficient to track results Inadequate information in

PCR to justify outcome/impact ratings

Post completion follow-up system not systematic

Work quality assessment not covered in PCR

Outline of Presentation

1. Context of Private Sector Development Assistance

2. Types of World Bank Group Private Sector Development interventions

3. IFC Advisory Services (Technical Assistance) projects M&E

4. Private Sector investment projects M&E

5. Use and Influence of M&E

4.1. Characteristics of Private Sector investments supported by IFC and MIGA

Operations subject to market testDirect Beneficiaries pay for serviceSome degree of competitionSponsors motivated by profitability

Third party effects (TPE) with no market feedbackEconomic distortionsPositive and negative externalitiesProvide rationale for public intervention

Development Agency’s involvementConsistent with market principlesFocused on third party effects (TPE)

Neighbors / Environment

Neighbors / Environment

CustomersCustomersGovernment / Taxpayers

Government / Taxpayers

EmployeesEmployees

4.2 Assessing Development Results - Stakeholder Framework

Competitors

Producers ofComplementary

Products

SuppliersSuppliers

New EntrantsProject

Company

(financiers)

Project

Company

(financiers)

4.3. Performance dimensions of IFC investments and MIGA guarantees

Objective Stakeholders Criteria Standards Assessment

Development Outcome

Financiers Business success(Market test)

Market benchmarks for ROIC

Actual financial statements and objectives at approval

Customers, suppliers, employees, government

Economic sustainability(Third party effect)

Social benchmarks for ERR

Actual financial statements, adjusted by economic prices and values

Neighbors, environment

Environment and social effects (Third party effects)

WBG performance standards

Actual impact and compliance with performance standards

Competitors, new entrants, producers of complementary products

Private sector development (Third party effect)

Objectives on competition, ownership structure, demonstration effects

Qualitative assessment of achievements of objectives on competition, ownership structure, demonstration effects

Track indicators and development outcomes

Supervise

DOTS*

*IFC’s Development Outcome Tracking System

Approve Disburse Repaid / Exit

Commit

Identify indicators

ExpandedSupervision

Report (XPSR)

4.4. M&E to adopt investment project cycle

21

Development Outcome Framework

FinancialPerformance

Financiers

Economic Performance

Gov’t, taxpayersemployees, customers,

suppliers

Envi. & SocialPerformance

Neighbors & Environment

Private SectorDevelopment

Demo. effects, linkages, corp. gov, investment climate

& new entrants

+Standard Development Reach Indicators

IFC-wide: Taxes, employment (incl. female)Plus sector-specific: e.g. MSME loans, utility connections

• Development Outcome: Overall effect on country’s development

• Framework: Derived from Good Practice Standards for Private Sector Evaluation (investments)

• Each with specific indicators, benchmarks

• Development Outcome: Overall effect on country’s development

• Framework: Derived from Good Practice Standards for Private Sector Evaluation (investments)

• Each with specific indicators, benchmarks

3.2. Performance Assessment

2.2 Investment project M&E

Challenges

• Difficult to gather info beyond company”

• Gaps in tracking private sector development outcome

– Missing indicators, discrepancies between stated outcome and indicators

• Reliance on client-submitted data

– Some data based largely on assumptions

• Financial Institutions – challenges in getting sub-project level information

Advantages• Financial data- good quality

and frequently updated;• Enforcement via contracts /

covenants• Integration with project

cycle– Use of standard indicators, with

baseline and target• Quality control cycle

established –Dept. review, Development Impact Dept review, External Assurance for Annual Report

Political Risk Guarantee Business model and M&E challenges (MIGA)

Business Model•Core competency as guarantee underwriting, not financing•Arms-length relationship to project enterprises;•No statutory reporting requirement from projects;•Constraints to access –

–No membership on project company boards of directors

ChallengesIn data gatheringIn communicating with project entitiesChallenges in contextual analysis

Industry / sector intelligence

23

Function of M&E in MIGA: significant progress in intro of self-evaluation and indicators.

Achievements

•New monitoring system for Environment & Social•Mainstreaming self-evaluation•Development indicator tracking system introduced

Areas of improvementChallenges in data gatheringInconsistent E&S follow-up on missing documentsDEIS –ex-ante data not covering all indicators.Pool of evaluation not adequate to report portfolio results

24

Outline of Presentation

1. Context of Private Sector Development Assistance

2. Types of WBG Private Sector Development interventions

3. Technical Assistance projects M&E

4. Private Sector Investment projects M&E

5. Use and influence of M&E

3.1 Lessons can have a positive influence on results

IFC Investment• Lack of lessons /

application of lessons at appraisal associated with poor outcome

– 80% of unsuccessful projects – low quality at entry was main factor

– Low DO with low quality at entry = 40% of them without lessons60% of them lessons were not integrated in design.

• Improved performance by learning from past failures (Agribusiness case study)

– Newer successful projects applied lessons from earlier failed projects

IFC Advisory Lessons influence design

and implementation New template asks “how

lessons have been applied to project design”

Wealth of lessons from PCR Segmented categories of

lessons High use of past project

PCR in new projects Low use of “SmartLessons”

M&E officers review at entry

3.2 Issues have been detected during supervision and actions have been taken

IFC Investment• High quality of supervision

work: 83% high (XPSR)• Half of XPSR identified

issues during supervision, 90% of them were detected, 63% of them were fully addressed during supervision.

• Credit and Environment risk systems as the most frequently used monitoring systems

IFC Advisory PSR widely used

95% of staff used PSR to make adjustment.

88% of supervision issues identified in PCR captured in PSR.

Poor supervision critical factor of 25% of projects rated low in DE

3.3 M&E information is increasingly used in IFC Strategies• Increasing use of M&E data• Annual Strategy process

– Development impact Dept. provide data and participate discussion.

– Guidance: asks development issues to cover

• Ratings and reach data used in strategy formulation– Historical DOTS rating comparisons

between sectors – prioritization of subsector– DOTS ratings vs. profitability– Gap analysis using “reach indicators” –

strategy to sharpen focus, areas to reach (i.e. BOP)

– Development return of different options using reach data (reach per $ cost of capital)

Three main messages

• Private Sector operations have specific data gathering advantage/disadvantage based on their business model

• M&E systems should adopt to the business practices to be effective and efficient

• M&E can influence learning, quality of work and outcome

Outstanding issues

• Difficult to estimate benefit of M&E–Need innovative approach, some

assumptions about counterfactuals;

Monitoring and Evaluation of Private Sector Development

Hiroyuki Hatashima

Private Sector Evaluation, Independent Evaluation Group (IEG)

World Bank Group

Canadian Evaluation Society Conference

June 11, 2013