10 - 01 - 2012EEEN Forum 2012 - Leuven1 Might international programs from the Climate change negotiations reduce tropical deforestation and address ecological.

10 - 01 - 2012 EEEN Forum 2012 - Leuven 1

Might international programs from the Climate change

negotiations reduce tropical deforestation and address ecological

forest challenges ?

The case of the Forest Carbon Partnership Facility (FCPF)

Karine BELNA [email protected]

http://www.afd.fr/jahia/Jahia/home


Outline of the presentation

1- My evaluation project & the related methodological challenges

2- My evaluation design & its underlying normative positionsContributions of the academic research works in Policy & program evaluation AND in

Environmental regime effectiveness

3- An interesting aspect of my work : « evaluation of evaluations »

Does the significant practice of evaluation influence the FCPF’s overall effectiveness?


1- My project : to evaluate the effectiveness of the Forest Carbon

Partnership Facility (FCPF) from an environmental perspective 1.1- The FCPF in a nutshell (1/3)

The REDD+ mechanism : incentive instrument to finance GHG emissions reduction in the forest sector in tropical countries

“REDD+” ? Reducing GHG Emissions from Deforestation and Forest Degradation AND the conservation

of forests, the sustainable forest management and the enhancement of carbon stocks.

« REDD + mechanism» Climate Change Negotiation for the Post-

Kyoto Regime

2007

Bali Road map

2010 Cancun agreement

2012

Extension of the Kyoto protocol

2017




The REDD+ mechanism and the “REDD+ readiness” phase ….

The REDD+ mechanism : incentive instrument for financing GHG emissions reduction in the forest sector in tropical countries.

“REDD+ readiness” phase : capacity building in countries to reduce deforestation and forest degradation in the view of the REDD+ financial incentive

« REDD + mechanism» Climate Change Negotiation for the Post-

Kyoto Regime

« REDD + readiness»Numerous initiatives, i.e. FCPF

2007

Bali Road map

2010 Cancun agreement

2012

Extension of the Kyoto protocol

2017




The Forest Carbon Partnership Facility : major multilateral readiness initiative

Launched in 2008 Multilateral program trusted by the World Bank 37 REDD+ countries / 18 donors (~450 million $) 2 funds : readiness fund (capacity building) & carbon fund (emissions reductions crediting)

37 “REDD+ countries” participants



Partnership Facility (FCPF) from an environmental perspective 1.1- The FCPF in a nutshell (3/3) Example : Mexico

Deforestation : 155 Mha/year (- 0,24 %) (2005 –2010) (FAO, 2010)

Drivers : agriculture, cattle breeding, urbanization, fires





2008 : Enters the FCPF > 200 000 millions $

REDD+ Readiness process in Mexico & the FCPF







Designs its REDD+ national strategy










technical aspects : reference level &monitoring & reporting system

political aspects : institutional arrangements& measures : i.e. payments for environmental services, Environmental policy integration etc.








2009 : Has its strategy approved by the FCPF > 3,6 millions $

Implements its strategy

REDD+ Readiness process in Mexico & the FCPFOther funding

technical aspects : reference level &monitoring & reporting system

political aspects : institutional arrangements& measures : i.e. payments for environmental services, Environmental policy integration etc.




Partnership Facility (FCPF) from an environmental perspective

1.2- My project & the related methodological difficulties

MY AIM : Answer the question : To what extent does the FCPF contribute to the reduction of gross deforestation and help address the ecological forest challenges ?

INPUTS FCPF EFFECTS (OUTPUTS > OUTCOMES)

State of the forests ?IMPACTS

FCPF



Partnership Facility (FCPF) from an environmental perspective 1.2- Difficulties to evaluate the effectiveness of the FCPF (1/4)

Problem of TIME (Mickwitz 2003)

how to assess environmental effectiveness of an on-

going program?


STANDARD to measure Effectiveness


State of the forests

IMPACTS

Other funding

Macro level of the FCPF

National levels of the FCPF : the 37 REDD+ countries. i.e. Mexico

FCPF








IMPACTS



FCPF

Problem of COMPLEXITY (Mickwitz 2003)

how to predict impacts properly from intermediate effects?

At the national levels : interacts with others programs

Complex drivers of deforestationComplex forest dynamicsOther funding








IMPACTS


National levels of the FCPF : the 37 REDD+ countries

FCPF

Problem of SIZE (Mickwitz 2003)

how to encompass the FCPF in its entirety ?

Other funding





STANDARD to measure

Effectiveness



IMPACTS

Other funding

FCPF

Problem of DATA AND SCIENTIFIC CONTROVERSIES(Mickwitz 2003)

how to shrink the scientific controversies on data and

forest concepts such as Sustainable Forest Management?




2- Designing an environmental evaluation of the FCPF 2.1- (1) Defining the standard - (2) determining the scope – (3) assessing causality

How to design an evaluation framework that enable to overcome those methodological

challenges?

• Defining the standard • Determining the scope • Assessing causality


2- Designing the environmental evaluation of the FCPF 2.1- (1) Defining the standard - (2) determining the scope – (3) assessing causality

(1) Defining the standard

(2) Determining the scope

(3) Assessing causality



a) Goals of the FCPF or other goals?

Problem of the “carbon focus ”: Trees = Carbon stocks??

FCPF’s objectives (FCPF 2008)

(a) To assist Eligible REDD Countries in their efforts to achieve Emission Reductions from deforestation and/or forest

degradation by providing them with financial and technical assistance […]

(b) To pilot a performance-based payment system for Emission Reductions generated from REDD activities, […]

(c) Within the approach to REDD, to test ways to sustain or enhance livelihoods of local communities and to

conserve biodiversity; and

(d) To disseminate broadly the knowledge gained in the development of the Facility […]



OECD (2002). Effectiveness : “The extent to which the development intervention’s

objectives were achieved, or are expected to be achieved …” > traditionnal Goal-

attainment model (Vedung, 2005)

Aims of the program

INPUTS Intervention

State of the environment

EFFECTS (OUTPUTS > OUTCOMES > IMPACTS)

Goal attainment evaluation / OECD Principles

a) Goals of the FCPF or other goals? Problem of the “carbon approach”



Traditionnal Goal-attainment model (Vedung, 2005)

Other approaches for policy & program evaluation : Needs-based evaluation (Scriven 1976),

Aims of the program

INPUTS Intervention



« NEEDS » Need based evaluation

Goal attainment evaluation

Research on Policy & program evaluation




Traditionnal Goal-attainment model (Vedung, 2005)

Other approaches for policy & program evaluation : Needs-based evaluation (Scriven 1976),

side effects evaluation (vedung 2005), Goal-free evaluation (Scriven 1976),

Aims of the program

INPUTS Intervention



« NEEDS » Goal free evaluation

Need based evaluation




Side-effects evaluation



Other approaches for policy & program evaluation : Needs-based evaluation (Scriven

1976), side effects evaluation (vedung 2005), Goal-free evaluation (Scriven 1976), concern-focused

evaluation (Mermet & al 2010) ….

Aims of the program

INPUTS Intervention



« Desirable » state of the environment

Goal free evaluation

Concern-focused evaluation

Need based evaluation




Side-effects evaluation

« NEEDS »



Endorse partially the objectives of the FCPF (Stufflebeam 2001; Vedung 2005) > Question the “carbon

approach” (Crabbé & Leroy 2004)

FCPF



problem resolution (“actual vs aspiration”) Improvement (“actual vs counterfactual”)OR

Start of the

interventionCounterfactual situation

Time


Distance to collective optimum

Advantages & Inconvenients

+ : avoid methodological difficulties to set the baseline- : legitimacy?, controversies on ecological concepts and phenomena

+ : conceptual relevance- : methodological feasibility

Research on Environmental regime effectiveness (Underdal 1992; Mitchell 2008)

b) Nature of the standard?



Counterfactual situation


2008

Start of the FCPF

Time

Desirable state of the Forests

• Endorse partially the objectives of the FCPF > Question the “carbon approach” (Crabbé & Leroy 2004)

• Adopt a “problem resolution approach” & establish principles for a desirable state of the forests

• international agreements (Convention on Biological Diversity & Climate change negotiation etc.)• litterature in forest ecology (CBD Secretariat 2009; Davis & al 2009; Sasaki & Putz 2009; Karousakis 2009; Pistorius & al, 2010; Harvey, Dickson, & Kormos, 2010; Pistorius & al, 2010; CBD Secretariat & GIZ 2011)

FCPF



• Endorse partially the objectives of the FCPF > Question the “carbon approach” (Crabbé & Leroy 2004)

• Adopt a “problem resolution approach” & establish principles for a desirable state of the forests• international agreements (Convention on Biological Diversity & Climate change negotiation etc.)• litterature in forest ecology (CBD Secretariat 2009; Davis & al 2009; Sasaki & Putz 2009; Karousakis 2009; Pistorius & al, 2010; Harvey, Dickson, & Kormos, 2010; Pistorius & al, 2010; CBD Secretariat & GIZ 2011)

FCPF

Desirable state of the Forests

Counterfactual situation


2008

Start of the FCPF

Time

P1. Natural forest area (quantitative objective) P2. Conservation of Biodiversity P3. ecosystem services P4. ecosystem dynamics (connectivity) P5. Indigenous people living condition and sites








Approaches to assess effectiveness on outputs and outcomes exist (Owen 2007; Kautto & Similä 2005)

Work on : Integration of an environmental concern at the strategic and instrument levels (Mickwitz 2003)


What to asses in the FCPF ? At which level?

Help us answer the question : How are environmental concerns integrated at the outputs –

outcomes stage ?

Model to explain the regime effectiveness ( Y = f(X) ) based on outputs & outcomes

Y Dependant variable : regime effectiveness (distance to collective optimum & behavioral change)

X Independent variables (> 50) : type of problem, institutional setting, configuration of interests etc.

Research on Environmental regime effectiveness (Miles & al 2002; Young )



What to asses in the FCPF ? At which level?

Importance of structuring multi-level evaluations when possible (Stame 2004)



INPUTS InterventionAIM of the COUNTRY

National strategy designed

National strategy implemented

INTERNATIONAL LEVEL

NATIONAL LEVEL

OUTCOMESOUTPUTS

IMPACTS

Etat des forêts

INCITATION FINANCIERE REDD+

Aims of the FCPF

INPUTSIntervention

FCPF

OUTPUTSGuidelines : Template, expertise criteria etc.

OUTCOMES1- Reviews of the national strategies (Expertise, peer review, ONGs reviews)

2- national strategies adopted3- national strategies implemented

STANDARD « Desirable » state of the forests

• Assess to what extent FCPF’s outputs and outcomes do integrate environmental concerns • Assess outputs and outcomes contents but also formation and strategic uses (Mickwitz 2003; Mermet & al 2010)

• Carry out a Multi-level assessment (Stame 2004; Rogers & al 2008) : macro level + 2 national case studies

FCPF








Advantage of the Timing of the evaluation > Process evaluation (Young 1994, Vedung 2005)


Research on Environmental regime effectiveness (Underdal 1992; Mitchell 2008)

Qualitative Narratives (Miles & al 2002)


3- Emphasis put on evaluations during the implementation of the FCPF

An impressive feature of the FCPF : the number of evaluations carried out during its

implementation.

To what extent do those evaluations influence the overall effectiveness of the

FCPF?

As part of my evaluation of the FCPF process > Analyze those evaluations :

- their content (against my standard)

- their design (the underlying negotiations)

- their influence (strategic use)


3- Emphasis put on evaluations during the implementation of the FCPF 3.1 - An overview of the evaluations carried out during the implementation of the FCPF

Related to the FCPF process in its entirety : independant mid-term

evaluation commisioned by the FCPC, Norway’s initiative of the FCPF, My evaluation

Assessments of the national stategies : experts, peers, NGOs,

implementation review

Assessments to be carried out as part of the national strategies : SEA,

forest policies ex-post evaluations, monitoring of the stratgy implementation

Numerous evaluations carried out during the implementation of the FCPF





INTERNATIONAL LEVEL

NATIONAL LEVEL

OUTCOMESOUTPUTS

3

12

9

10

11

Scope of the evaluation

IMPACTS

Etat des forêts


3 Evaluation

Aims of the FCPF

INPUTSIntervention

FCPF




45

6

78

My evaluation Mid-Term evaluation of the FCPF

Norway’s evaluation of the FCPF’s contribution to REDD+

National strategies’ assessments By EXPERTS

By PEERSBy NGOs

Env. & So. Strategic Assessments

Ex-post evaluation of national policies

Monitoring of implementation


3- Emphasis put on evaluations during the implementation of the FCPF 3.1 - An overview of the evaluations carried out during the implementation of the FCPF

Evaluation form (Owen 2007) / Evaluation Horizon (Hilden 2009)Related to the FCPF process in its entiretyAssessments of the national stategies to be carried out as part of the national strategies

Proactive Clarificative Interactive monitoring Impact

Past Assessment of past forest-

related policies 11 my evaluation 1 > concern focused evaluation

- independant mid-

term review of the FCPF

2 > responsive evaluation

Monitoring & Reporting of

the R-PP implementation

12 > “Systems for Implementation Review” (Victor & al 1998)

my evaluation 3 > process outcomes evaluation

assessments of the national strategies >

expertise 4, peer review 5

ICFI Norway’s evaluation including

FCPF 3 > advocacy evaluation

Mid-term review of the

national strategies 7 assessments of the implementation of the

strategies > expertise 8,

peer review 9NGOs’ review of the

national strategies 6Future Strategic Environmental &

Social Assessment 10

Present





OUTCOMESOUTPUTS

31 2

9 10 11

IMPACTS

Etat des forêts


Aims of the FCPF

INPUTSIntervention

FCPF


STANDARD « Desirable » state of the forests



5 6

78

Assessments of the national strategies by the Technical Advisory Panel

4

Scope of the evaluation

3 Evaluation

INTERNATIONAL LEVEL

NATIONAL LEVEL


3- Emphasis put on evaluations during the implementation of the FCPF 3.2 - An example : how do the TAP reviews influence the effectiveness of the FCPF?

1- Assessment of the TAP reviews’ content P1 : P2 : P 3 :

2- Analysis of the influence & the use of the TAP reviews

Method : - Desk reviews : TAP assessments & national Strategies

- observation : FCPF meetings

- Interviews : experts & participants



June 2009. Guyana,

Indonesia Panama

March 2010. DRC, Ghana,

Mexico

July 2010. Argentina, Costa-

Rica, Kenya, Nepal, Republic

of Congo

November 2010. Laos, Tanzania

March 2011. Peru, Cambodia,

Vietnam, Ethiopia

June2011. Uganda &

LiberiaOctober 2011.

CAR & Colombia

20122010 2011

7 Rounds of TAP reviews – More than 21 reviews of national strategies

Who is this Technical Advisory Panel ? Specialists in different disciplines, proposed by the

FCPF’s members and then selected by the World Bank. 5 to 8 experts / review



1- Assessment of the TAP reviews’ content

a- Gradually more “political correct” : “……… has produced an impressive R-PP, which was highly appreciated

by its reviewers. All those involved in its preparation should feel well pleased with their achievements so far”.

P1 : P2 : P 3 :






P1 : P2 : P 3 :

June 2009. Guyana,

Indonesia Panama


Mexico

July 2010. Argentina, Costa- Rica, Kenya, Nepal, Republic of

Congo


March 2011. Peru, Cambodia, Vietnam,

Ethiopia

June2011. Uganda &

Liberia

October 2011. CAR &

Colombia

20122010 2011

Pressures on the TAP to mitigate criticisms in the reviews






P1 : P2 : P 3 :

b- Quality decreases (BIC 2011). Overall good – level scoring although serious shortcomings


Scores affected by the TAP on the different components of the national strategies (average on 17 national strategies )

(3: standard met - 0 : standard not met)

0 1 2 3

institutional arrangementsconsultations 1

diagnostigstrategy

implementation frameworkEnv. & Soc impacts ex ante

baselineMRV carbon

NRV of non carbon aspectsevaluation framework

- Heterogenous and too positive statements from the TAP

- Environmental aspects are poorly assessed in some reviews






P1 : P2 : P 3 :

Scores affected by the TAP on the different components of the national strategies (means on 17 national strategies ) (3:

standard met - 0 : standard not met)

0 1 2 3

institutional arrangementsconsultations 1

diagnostigstrategy

implementation frameworkEnv. & Soc impacts ex ante

baselineMRV carbon

NRV of non carbon aspectsevaluation framework

b- Quality decreases (BIC 2011). Overall good – level scoring although serious and acknowledged

shortcomings

c- Ex: Requirement to carry out an

Ex-ante assessment of Environmental & social impacts (9)On average : standard MET

whereas very slight and vague statements





a- Decreasing of the role of support to decision makers (charter of the FCPF) > increase in expert

support to REDD countries when designing their R-PP)

- Time devoted to TAP reviews presentation during decision making meetings > has drastically

reduced. - Reviews are not updated




a- Decreasing of the role of support to decision makers (increase in expert support to REDD countries

when designing their R-PP)- Time devoted to TAP reviews presentation during decision making > drastically reduced. - Review not updated

b- TAP review leadership has been decreasing : TAP reviews are now put at the same level than Peer

reviews and NGOs reviews.

June 2009. Guyana,

Indonesia Panama


Mexico

July 2010. Argentina, Costa- Rica, Kenya, Nepal, Republic of

Congo


March 2011. Peru, Cambodia, Vietnam,

Ethiopia

June2011. Uganda &

Liberia

October 2011. CAR &

Colombia

20122010 2011

Start of Peer reviews and NGOs reviews




a- Decreasing of the role of support to decision makers (increase in expert support to REDD countries

when designing their R-PP)- Time devoted to TAP reviews presentation during decision making body meetings > drastically

reduced. - Review not updated

b- TAP review leadership has been decreasing : TAP reviews are now put at the same level than Peer

reviews and NGOs reviews.

c- And when the standard of the TAP is not met ? No mechanism, no milestones nor guaranty that

TAP recommendations will be taken into consideration during implementation of the strategy



Conclusion :

% Content : not really relevant to stress environmental concernsa- On the overall, Decrease in the TAP reviews relevance ( compared to my standard)

b- Heterogeneity among the reviews

c- In some reviews : principles 2-5 of my standard (Biodiversity, ecosystem dynamics, connectivity and

indigenous people living sites) > poorly assessed.

% utilization and decision making influence : TAP becomes a kind of legitimization tool for

the FCPF > a foil for the FCPF (?)a- Role of the TAP has evolved : from Decision makers advisers (charter) to REDD+ countries

supporters/ Good or bad for the improvement of the quality of the strategies?

b- Leadership has diminished

c- Use of their recommendations is uncertain


CONCLUSION

1- On my evaluation framework : mixed achievements regarding methodological

challenges

- TIME : (+) Incorporation of environmental concern … > (-) effectiveness evaluation?

- COMPLEXITY : (+) problem resolution perspective (not counterfactual) …. (+) process evaluation

- SCALE : (+) multi-level evaluation … (-) « Only » 2 national case-studies

- CONTROVERSIES ON CONCEPTS & DYNAMICS : (+) standard simple enough to avoid them … (-) at the

country level, some debate remain.

2- On the analysis of the FCPF’s evaluations : Go on with other evaluations- TAP assessment : limited advocate of environmental concerns

- NGOs assessments : greater relevance : on MRV of biodiversity and governance issues.

Greater influence of the decision making - Strategic Environmental & Social Assessement (SESA) : critical role

…..


Présentation AFD 14-12-2009 Thèse REDD - FCPF


References

CBD. (2010). COP 10 Decision X/2 X/2. Strategic Plan for Biodiversity 2011-2020.CBD Secretariat. (2009). Connecting Biodiversity and Climate Change: mitigation and adaptation, Report of the Second Ad Hoc Technical Expert Group on Biodiversity and Climate Change. CBD Technical Series, 41, 127.FCPF. (2008). Forest Carbon Partnership Facility.Information Memorandum. World Bank.Guéneau, S. (2011, septembre 16). Vers une évaluation des dispositifs de prise en charge du problème dudéclin des forêts tropicales humides. AgroParisTech, Paris.Haas, P. M., Keohane, R. O., & Levy, M. A. (1993). Institutions for the Earth. Sources of effective international environmental protection (The MIT Press.). Cambridge, Massachusetts.Harvey, C. A., Dickson, B., & Kormos, C. (2010). Opportunities for achieving biodiversity conservation through REDD. Conservation Letters, 3, 53–61.Leroy, M. (2006). Gestion stratégique des écosystèmes du fleuve Sénégal (L’Harmattan.).Mermet, L., & al. (2010). Concern-Focused Evaluation for Ambiguous and Conflicting Policies: An Approach From the Environmental Field. American Journal of Evaluation, 31(2), 180-198.Mickwitz, P. (2003). A framework for evaluating environmental policy instruments: context and key concepts . Evaluation, 9(4), 415-436.Mitchell, R. B. (2008). Evaluating the performance of environmental institutions: what to evaluate and how to evaluate it? Young O.R., King L.A. & Shroeder H.(eds). Institutions and environmental change: principal findings, applications, and research frontier (MIT Press., p. 79-114). Cambridge, MA, USA.OECD. (2002). Glossary of Key Terms in Evaluation and Results Based Management.Owen, J. M. (2007). Program Evaluation. Forms and Approaches (The Guilford Press.). New York London.Pistorius, T., & al. (2010). Greening REDD+. Challenges and opportunities for forest biodiversity conservation. Policy Paper (p. 43). Germany: University of Freiburg.Rogers, P. (2008). Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions, 14(1), 29 – 48.Stufflebeam. (2001). Evaluation Models. New Directions for Evaluation, 89, 7-98.fectiveness of forest certification. Canadian Journal of Forest Research, 38, 1357-1365.UNFCCC. (2010c). FCCC/AWGLCA/2010/14:Ad Hoc Working Group on Long-term Cooperative Action under the Convention. Twelfth session. Tianjin, 4–9 October 2010. Negotiating text.UNFCCC. (2011). Decision 1/CP.16 The Cancun Agreements: Outcome of the work of the Ad Hoc Working Group on Long-term Cooperative Action under the Convention (Vol. FCCC/CP/2010/7/Add.1., p. 31).Underdal, A. (1992). The Concept of Regime `Effectiveness’. Cooperation and Conflict, 27(3), 227-240.Underdal, A. (2002). One Question, Two answers. Environmantal regime effectiveness. Confronting thory with evidence. Edward L. Miles and al (eds) (MIT Press., p. 3-45). Cambridge, Massachusetts.Vedung, E. (2005). Public policy and program evaluation (Transaction publishers.). New Brunswick (USA) and London (UK).Young, O.R. (1999). The effectiveness of international environmental regimes. Causal connections and behavioural mechanisms. Massachusetts institute of technology.

10 - 01 - 2012EEEN Forum 2012 - Leuven1 Might international programs from the Climate change negotiations reduce tropical deforestation and address ecological.

Documents

forest sector

forest degradation

fcpf i

environmental perspective

redd readiness phase

evaluation project

redd readiness process

redd countries participants1