Building an Evidence base: Introduction to Economics of Adaptation and Impact Assessment

Building an Evidence base:Introduction to Economics of Adaptation and Impact Assessment.

Babatunde Abidoye (PHD)

What is Evidence base for climate change adaptation?

• Evidence-based adaptation argues that all stages of an adaptation process should use state of the art research and information to guide decisions at all levels.

– Building an evidence base advocates a more rational, rigorous and systematic approach to decision making.

• The argument is that adaptation made based on this framework produces better outcomes.

Approaches for Evidence Based adaptation

• Applying rigorous techniques to understanding Impacts and vulnerabilities in agriculture – e.g. sectoral and national level studies.

• Applying rigorous techniques to appraise adaptation options –Cost Benefit Analysis (CBA)

• Applying Impact Evaluation techniques to understand the benefit of a project

• Linking lessons learned and results to the global question of climate change adaptation and mitigation.

Where Does CBA fit within the project?

• Will primarily contribute to Outcome 1 - Technical capacity and institution-building on NAPs strengthened – through training on CBA of Agriculture and other related sectors stakeholders such as Ministries of

Water and Ministries of Environment

• It will also contribute to Outcomes 2, 3 and 4– Outcome 2: Integrated roadmaps for NAPs developed - Mainstreaming of climate change requires results

framework

– CBA will help with prioritizing actions for integrating climate change adaptation into development policies and strategies at the national and sub-national level, starting with agriculture as the key sector.

– CBA will help the Ministries of Agriculture with access to finance from national planning.

• Outcome 3: Evidence-based results for NAPs improved.– The activities under Outcome 3 involve the development of an impact assessment framework and conducting

sectoral and programme specific cost-benefit analyses of adaptation options in the agriculture sectors (in collaboration with other ongoing UNDP/FAO initiatives).

• Outcome 4: Advocacy and knowledge sharing– CBA will be featured in interactive good practice compilation of case studies.

The role of CBA in the NAP process• The NAP process typically involves four elements:

(A) Laying the groundwork and addressing gaps (B) Preparatory elements (C) Implementation strategy (D) Reporting, Monitoring and Review.

• CBA is an important tool in helping countries at (B) and (C) stages

• CBA is an important tool for project appraisal because it helps to identify and value the costs and potential benefit of a project over the life of the project.

• The major difference between CBA and Impact assessment is the calculation of benefits – IA does not make assumptions on this – it is based on observed behavior of economic agents.

Economic tools for ranking and prioritization of adaptation options

Source GIZ (2013)

Objective ways to know we are maximizing benefits

• Evaluating distinct but equally plausible adaptation options (or measures) requires a CBA of each alternative option

• This will allow an objective comparison to be made between the options.

• For adaptation options that have implications on entire sectors, a market analysis is required to see how entire economic systems are affected. This can be done at two levels – a sector by sector or an inter-sectoral approach

Objective ways to know we are maximizing benefits

• Example of project analysis under uncertainty:

• Suppose a low dam costs $1 billion, a high dam costs $1.5 billion and with current climate, a low dam has benefits of $1.2 billion and high dam of $1.6 billion

low dam is better with $200 million net gain

• With future wetter climate, suppose benefit of low dam rises to $1.3 billion and benefit of high dam rises to $2 billion

high dam now better option with $500 million net gain

Cost benefit analysis: intro - I

• When making a decision, esp. in the public sector, trade-off analysis is inevitable. Costs and benefits of an action and / or inaction need to be understood for policy decisions.

• Add up costs and add up benefits; if net benefits are positive: do the project!

• Simple, right?

• Not quite! Issue of measurement of costs and benefits.

Cost benefit analysis: intro - II

• Often times, costs are simpler – tend to be one-off and are market transactions e.g. one time cost of constructing an irrigation system and operating and maintenance cost.

• Benefits can be trickier e.g.: How do we calculate the market (e.g. wealth) and non-market (e.g. health) benefits over time?

CBA: 8 suggested steps

1. Define the scope of analysis

2. Identify all potential impacts of the project

3. Quantify the predicted impacts

4. Monetize impacts

5. Discount rate to find present values of costs and benefits

6. Calculate the net present value (NPV)

7. Perform expected value and/or sensitivity analysis

8. Make recommendations

Time preference and discounting – I

• When determining the optimal allocation of resources over time, one must deal with time preferences.

• CBA requires consideration of stream of benefits over time against costs which incur largely today (e.g. install irrigation system that is expected to be useful over x years).

• The problem is that the value of money does not stay constant over time.

• Also, most individuals prefer to receive benefits now as opposed to receiving the same level of benefits in future (i.e. they have positive time preferences).

Time preference and discounting – II

So, we need a system:

• The discount rate (value between 0 and 1) is a product of society’s time value of money (composed of the pure rate of time preference and the goods discount rate) [closer to 0: future is more important; closer to 1: today is more important].

• The discount rate will vary by country and who is funding the project.• National development project• International Climate Finance Agencies:

– World Bank– Green Climate Fund– Regional Development Banks

Discounting: How do we do it?

• Use this formula to convert all future values to present values:

• PV = FV(t) / (1+r)t

Where

• FV is expected future value

• r is the discount rate (value between 0 and 1)

• t is time.

• Example: How much is $100000 in 30 years worth today? Try it using different values for r!

$0

$10,000

$20,000

$30,000

$40,000

$50,000

$60,000

$70,000

$80,000

$90,000

$100,000

0 5 10 15 20 25 30

Valu

e

Year

The impact of a discount rate on present value estimates

15

12

%

9%

6%

3%

Source GIZ (2013)

Potentials within the NAP process• CBA works best when it is possible to estimate both the market and

non-market values of all benefits that accrue. • It is a systematically estimated analysis that provides a key piece of

information to facilitate a decision. It is not meant to be used as the sole decision-criteria.

• If properly executed, CBA enables comparison, in equal terms, the investments that have to be made today with benefits that accrue over time.

• Within the NAP process, results from CBA can be used as one of the ways of evaluating and ranking adaptation options after they have been identified. Political, social and other considerations will also matter along with economic considerations.

Limits of CBA

• One potential limitations of a CBA is that in some cases not all direct costs and benefits can be quantified in monetary terms – in many cases we don’t need to calculate all the costs and benefits.

• The discount-factor can matter in the final analysis.

• Need to also account for uncertainty over time. Can use sensitivity analysis to capture some uncertainty.

Application in this project

• Activities under outcome 1, 2 and 3 can assist to build national capacity for economic appraisal techniques and tools through building capacity of departments of ministries involved in CBA as well as other national institutions.

• Many countries and international agencies require CBA be carried out during the design stage of a project.

• speaks the language of most ministry of finance and national planning that can be useful to unlock funding for climate change adaptation projects.

How can the programme implement these activities?

• Country teams can make use of a guidance note on how to conduct a CBA prepared by the global support team

• Provision of earmarked training activities under outcome 1, outcome 2 and outcome 3 for 2016-2018 to build the capacity of national experts, line ministries and national institutions

• Technical and financial assistance to national experts and/institutions to appraise a few selected adaptation options and select them for a NAP agriculture roadmap (Outcome 2)

• Through documentation of case studies in outcome 3 and 4 of the country work-plans as well as documentation by the global team.

Where Does IA fit within the project?

• Will primarily contribute to Outcome 3: Evidence-based results for NAPs improved.– The activities under Outcome 3 involve the development of an impact

assessment framework and the conducting of sectoral and programme specific cost-benefit analyses of adaptation options in the agriculture sectors.

• It will also contribute to Outcomes 1 and 2– Outcome 1 - Technical capacity and institution-building on NAPs strengthened

• This will include understanding the linkage between IA and Cost Benefit Analysis

– Outcome 2: Integrated roadmaps for NAPs developed - Mainstreaming of climate change requires results framework• IA will help in integrating climate change adaptation into development policies and

strategies at the national and sub-national level, starting with agriculture as the key sector.

• IA will enable Ministries of Agriculture to systematically learn about the effectiveness of adaptation options that they implement

The role of Impact Assessment in the NAP process

The NAP process typically involves four elements:

(A) Laying the groundwork and addressing gaps

(B) Preparatory elements

(C) Implementation strategy

(D) Reporting, Monitoring and Review.

• Impact Evaluation is an important tool in helping countries at (B) (C) and (D) but traditionally categorized into D.

• Impact evaluation particularly will support investment appraisal and prioritization of adaptation options

Why Impact Assessment?

• Development programs and investments are designed to change outcomes of individual agents and economies.– For example – we want to know by how much a change (measured in terms

of yields or other outcomes) brought about through investments in risk management in the agriculture sector can be measured

• However, traditional M&E typically focuses on counting inputs rather than results– These inputs can include (1) Number of farmers trained, (2) number of

beneficiaries of the NAP project intervention, (3) percentage of extension worked trained.

• Impact evaluation proposes a shift in focus from inputs to outcomes and results

What is impact assessment?• Causal impact is at the center of all impact

assessments- it is the difference in outcomes that is caused by the program/investment

• How do beneficiaries of an investment perform compared to how they would have fared if they had not participated in the program?

Each farmer that adopts climate smart agriculture practice will have a potential what-if profit level if they did not adopt it and vice versa. This alternative income/outcome level serves as the counterfactual.

This two states of potential income exists only in theory!

• The central focus of impact evaluation framework is finding a way to infer the counterfactual from what happened to other people or what happened to the participants of the program before the start of the program.

Why Use Impact Assessment?

• IA is useful to generate evidence on questions on the impact of adopting specific adaptation options or investments (policies/structures)

• For example: An adaptation practice involved reintroducing livestock varieties and generating awareness of growing of indigenous crops among farmers in MwingiDistrict of Kenya.

• Typical evaluation will ask:– How many farmers are reintroduced to livestock varieties and aware of growing of indigenous

crops in the Mwingi District of Kenya?

• Impact evaluation however will ask:– Did the project reintroducing livestock varieties and promoting the growing of indigenous crops

increase agricultural yield and profit of farmers in the Mwingi District of Kenya? And by how much?

Different Approaches to Impact Evaluation

• The validity of any impact evaluation framework estimate depends on the validity of the assumptions on the counterfactuals.

• once we intervene through a NAP project, we do not have a parallel universe that we can observe what would have been without the NAP project.– So we can only observe the net revenue of a group of farmer that

participated in the project and a group of farmers that did not.

• How the two groups were selected and the kind of influence the household have on getting the intervention information plays a central role in the methodology.

• This provides a fundamental difference between experimental design approach to evaluation and regression based approaches to evaluation.

Experimental Design

• Experimental design is perfect for a reliable identification of program effects in the face of complex and multiple channels of causality.

• Experiments make it possible to vary one factor of a program at a time and therefore provide “internally” valid estimates of the causal effect.– Will Savings and Fertilizer Initiative (SAFI) program in Kenya increase fertilizer usage?

• This research can produced a number of interesting results.– In the first season, the basic SAFI increased usage by 14 percentage points, on a base of 24

percentage points.– SAFI with ex ante Choice of Timing was also successful, increasing usage by 22 percentage

points.

• On the other hand, one might also get surprisingly results: – the subsidy increased usage by 13 percentage points. – A free delivery visit later in the season had no significant effect on usage.

• This suggests that it was the lack of commitment mechanism that was preventing farmers from purchasing and using fertilizer.

So how does experimental design work?

• Experimental design is premised on the ability to randomize treatment.

• The key feature of a randomized evaluation is that the people who have access to the program or benefit from the policy are selected randomly.– This is called random assignment

– This ensures that there are no systematic differences between those who receive the program and those who serve as the comparison group.

• It is this random assignment that gives randomized evaluation an advantage in measuring program impact.

Quasi-experimental design:

• Quasi-experiments are experimental designs with no randomized treatment and ability to claim causality.

• Two major forms of quasi-experimental designs are the nonequivalent groups design and regression-discontinuity design:

1. Nonequivalent groups design does not control the assignment to groups before the project as is typically done in the randomized control treatments.

2. Regression Discontinuity approach on the other hand is used to evaluate program that have eligible cutoff.

Regression Based Impact Evaluation

• Regression based approaches is one of the major ways economists evaluate projects.

• Regression analysis typically relates a causal variable of interest and an outcome variable of interest - the relationship between them statistically is considered the impact of the project.

• Individuals who received the program are compared with those who did not, and other factors that might explain differences in the outcomes are “controlled” for.

What is required?

Nonequivalent groups design Regression Discontinuity Regression Based

Data on outcomes

“variables for matching” both participants and nonparticipants

Information on exactly how program participation is decided

Data on “explanatory” or “control” variables such as age, income levels, education. (The control variables need either to be collected at baseline or after.)

Application in this project

• Activities under outcome 3 can assist to build national capacity for impact assessment through building capacity of departments of ministries involved in M&E as well as other national institutions.– Can assist in building an evidence base for adoption of adaptation options

• Impact Assessment can help understand new programs/interventions –especially with the application of experimental design

• It can also be used to understand if a project achieved the intended outcomes – applied ex post

• With competing needs for funding – Impact evaluations provides compelling evidence for funding from ministry of finance and national planning for climate change adaptation projects.– One of the major part of mainstreaming is ensuring evidence-based impact and

strategies.

• In the working group, we will discuss the different approaches with application on how to evaluate current projects.

How can the programme implement these activities?

• Develop and refine tool-kit/guidance applying impact assessment to adaptation activities in the agriculture sector.

• Apply the toolkits/guidance through training to build the capacity of government monitoring units and national institutions on impact assessment applying the guidance to an existing agricultural project

• Through consultations, select 1-2 countries, and identify targeted technical and financial support to ongoing efforts related to IA

• Through support and mentoring of national activities by impact assessment experts in the global UNDP-FAO team

• Through documentation of case studies, learning, and knowledge exchange (outcomes 3 and 4) related to IA