Methodologies of monitoring and evaluating capacity development Makoto Kato Japan [email protected] 1 UNFCCC Expert Workshop on Monitoring and Evaluating.

Methodologies of monitoring and evaluating capacity development

Makoto KatoJapan

[email protected]

1

UNFCCC Expert Workshop on Monitoring and Evaluating Capacity-Building in Developing CountriesRio de Janeiro, Brazil, 6-7 November, 2008

Outline• Defining Capacity, and Capacity Development

• Methodologies and Tools for Monitoring and Evaluating Capacity Development

• Comparison? M&E at national level?

• Uniqueness of Capacity Development in Climate Change area

2

Defining Capacity, and Capacity Development

Capacity: Developing countries’ ability to cope with

challenges by their own organization and actions.

Capacity development: Process in which such capacity is being

strengthened and sustained at individual, institutional, and society levels as a whole.

3Source: JICA 2006

Capacity at different levels(Capacity Development must be comprehensive)

Individual Level

Institutional Level

Society Level

Source: JICA 2006 4

Administrators’ Capacity

Enhancement

Strengthened Management of Waste

Control Authority

Consensus in Burden sharing by different

stakeholders

New Waste Segregation Rules

Policy Goals toward Waste Reduction

Awareness raised about

Waste Reduction

Different Approaches to Capacity Development(Capacity Development should be country-driven and sustainable)

Existing Capacity Existing Capacity

External Input(Incl. Knowledge

Sharing)

ExternalCapital Transfer

Level of Capacity necessary to cope with challenges

Approach A: Transfer of capacity from external player(Filling a Gap with external capital transfer)

Approach B: Developing capacity with external facilitation(building capacity under the ownership of host countries/communities)

Source: JICA 20065

Not Sustainable

More Sustainable

Key Drivers of Capacity Development

• Ownership• Enabling Environment• Incentives• Leadership• Knowledge

6

7

Methodologies and Tools for Monitoring and Evaluating Capacity-Building (1)

Plan

Implementation

EvaluationIdentification

Formulation

Preparation

Implementation / Monitoring

Evaluation

Flow of M&EConceptual Figure of Project Cycle Management(PCM)

8

Where indicators are used:Format of Project Design Matrix （ PDM ）

Narrative Summary Indicators Means of Verification

Important Assumption

Overall Goal  

Project Objective  

Outputs  

Activities 

Inputs

Pre-conditions

Methodologies and Tools for Monitoring and Evaluating Capacity-Building (2)

VerticalLogical

Relationship

- - -

Cause

Effect

How Overalls Goals/project objectives and Indicators are made?Problem Tree

Core Problem

Methodologies and Tools for Monitoring and Evaluating Capacity-Building (3)

Expected Situations

FeasibilityMeans

EndsCore ObjectiveCore Objective

Methodologies and Tools for Monitoring and Evaluating Capacity-Building (3)

How Overalls Goals/project objectives and Indicators are made?Objective Tree

Methodologies and Tools for Monitoring and Evaluating Capacity-Building

• Evaluation is heavily dependent on “Target Setting”(based on capacity assessmentcapacity assessment)

Target in different levels: Overall Goal, Objectives of Efforts (eg. Policy/Programme/Project objectives)

• Indicators are useful only if/when they are elaborated with means of verification, important assumption and pre-condition (used in PDM)

• M&E are properly done by Project Owners (and contracted external evaluators)

11

Can we compare?

Same type Projects in 2 different countries

12

Country A Country B

Project X Project Y

Development Priority

Governance Structure

Existing Capacity

Other attributes(eg, Size)

Development Priority

Governance Structure

Existing Capacity

Other attributes(eg, Size)

Comparison is only useful for drawing lessons from “Good Practices”(Simple replication does not happen)

Can we compare?Same type Projects in the same country

13

Country C

Project Q Project P

Development Priority

Governance Structure Existing Capacity Other attributes

(eg, Size)

(Internal Factors)• Willingness of Stakeholders•Entry point of Efforts etc.

(Internal Factors)• Willingness of Stakeholders•Entry point of Efforts etc.

Easier to compare the two, since the external factors are the same or similar.But still internal factors of projects should be taken into account.

Can we compare? M&E at National LevelAccuracy of Comparison (EvaluationAccuracy of Comparison (Evaluation))• Project level>Programme level>Policy level> Regional/International Network

(At higher level, quantified indicators may omit useful qualified information)

Methods of M&E Methods of M&E • We use the same M&E methods(some variety), but indicators are

more tailor-madePre-condition of Defining IndicatorsPre-condition of Defining Indicators• Capacity Assessment (jointly done by D-ing+D-ed countries) is

crucial• Assessment(BAU) Monitoring(Indicators) Evaluation

(Indicators) must be coherent, and reflect a specific context.

14

Narrative Summary Indicators Means of Verification

Important Assumption

Overall Goal  

Project Objective  

Outputs  

Activities 

Inputs

Pre-conditions

- -

Uniqueness of Capacity Development in Climate Change area

15

• This Area is already decided by 2/CP7.• Fill in one of the 15 scopese.g. (promotion of) CDM

-

Entry points of such efforts are different by host countries. Host countries chose such entry point, jointly with our Agencies.

Conclusion from our practice

• A single methodology (with variation) for M&E is used.• Indicators are used and functions in specific context (within

PDM), and simple aggregation of evaluation results is strictly avoided.

• Indicators are useful to interpret lessons from Good Practices.• Evaluation at larger level employs more qualitative/narrative

way.• Entry points of efforts are different by countries, and it affects

selection of indicators.• For climate change, we approach both from “Overall Goal” and

from the bottom side of PDM( and it still works).

16