IMPACT ASSESSMENT OF AGRICULTURAL RESEARCH FOR …agri.eco.ku.ac.th/CASAF/files/Impact assessment_Dr. Maredia.pdf · developmental goals (public goods), such as: • Reducing poverty

IMPACT ASSESSMENT OF AGRICULTURAL RESEARCH FOR DEVELOPMENT: OVERVIEW OF METHODOLOGIES AND CHALLENGES

Mywish K. Maredia ([email protected]) Professor, Dept. of Agricultural, Food and Resource Economics, Michigan State University

Workshop on Impact Assessment of Agricultural and Food Research, Kasetsart University, Bangkok, ThailandJune 5, 2015

Outline1. The context of agricultural research for

development (R4D)2. Making agricultural research impactful

• Challenges and implications for a research system

3. Methods and tools of impact assessment in the context of agricultural R4D

2

1. THE CONTEXT OF AGRICULTURAL RESEARCH

3

Rationale for Public Sector Investments in Agricultural Research• Public investment in research is one of the important modes of achieving developmental goals (public goods), such as:• Reducing poverty• Increasing food security and enhancing nutrition• Promoting environmental sustainability

• It is a means to an end (development); not an end itself

• Goal of research funded by government and international aid agencies is to generate outputs that have positive impact on people’s lives

4

Examples of impact-oriented research outputs• New varieties that

• Increase yield• Reduce risk due to drought, insects, diseases• Reduce inputs and costs• Have traits preferred by consumers• Reduce growing season

• New storage technologies that reduce crop losses• New value-added products that save labor/improve

nutrition • Information on how credit markets function or not that

helps introduce institutional innovations• Validation of results on efficacy of beans in improving

nutritional status of children

5

Research in a development context

• Two key parameters determine ‘impacts’ of research - Adoption (the use and uptake of research outputs)- Effect size (the benefit per unit of adoption of a research

output in relation to an existing practice / technology)• Larger the values of these two parameters, larger will be the ‘impact’

• If no adoption No impact• If zero or low effect size (benefit) per unit of adoption No adoption No impact

6

Enhancing the ‘Effect Size’• The ‘effect size’ depends on:

• the science embedded in the research output (i.e., the breakthroughs, the inventions and discoveries, etc.)

• the reality at the end-user level (i.e., the environment, socio-economic conditions, etc.)

• existing or alternative practice / technology available at the end-user level (the counterfactual)

7

Necessary conditions for adoption to occur:• Cost-effective—must be profitable for end-users

to adopt (i.e., benefits > costs)• Awareness—farmers/consumers must know that

the technology/product exists• Physically accessible—technology must be

available for farmers to adopt• Economically accessible—farmers must have

access to cash/labor needed to buy/use it• Scaling Up—technologies must be replicable • Sustainability—must be possible to extend the

technology without subsidies

Enhancing ‘Adoption’8

2. MAKING AGRICULTURAL RESEARCH IMPACTFUL

Challenges and implications for a research system

9

How to make agricultural research impactful? Role of researchers• Ensure a positive and large ‘effect size,’

• The output(s) resulting from a research effort must be superior than existing/alternative practice/technology under the conditions found at the end-user level.

• This means the importance of assessing the effect size not only in scientific units (e.g. kg/ha) but also in socio-economic units (e.g. $/ha, consumer acceptability, market accessibility, availability of labor, etc.)

10

How to make agricultural research impactful? Role of researchers (cont’d)

• Ensure ‘Adoption’ of research outputs do take place• Researchers funded under development assistance banner

must compensate for institutional underdevelopment

• Researcher need to devote some efforts on ‘outreach’activities

• Researchers need to have forward-looking vision of the pathway that goes beyond outputs

• They need to think about creative ways for translating outputs into outcomes and impacts; think about the gaps in pathways to impact, and what they need to plan in order to fill those gaps.

11

Making agricultural research more impactful: Implications for the research system

• Need to inculcate an ‘impact culture’ in research community• Need incentive structures to internalize and integrate M&E data

collection and reporting as part of a research implementation plan • So that it is not viewed as an ‘external’ data collection and reporting

requirement

• Need to allocate and budget adequate resources for M&E (for reporting), impact evaluation (for learning), and ex post impact assessment (strategic validation of benefits from research)

12

Making agricultural research more impactful: Implications for the research system (cont’d)• Need investments to build capacity in research systems to

design and implement effective M&E systems, conduct impact evaluation research, and undertake long-term impact assessment studies

• Need different strategies, indicators, methods and tools to meet accountability and learning needs of research investments

13

3. METHODS AND TOOLS OF IMPACT ASSESSMENT

In the context of Agricultural Research for Development (R4D)

14

Impact pathway – On a Time and Scale DimensionScale

Uptake and Adaptation—pilot scale (initial outcome)

Dissemination and large scale adoption (outcome)

Early benefits (economic, social, environmental)

Impacts on poverty, hunger, nutrition

Time

Research outputs generated

Research Starts

15

• Is long and complex• Need diversity of methods

and approaches to assess progress towards reaching the goal

Methods and tools for impact assessment of agricultural R4D• Can be grouped in to two types depending on when they

occur on the time and scale dimension of the impact pathway

16

Uptake and Adaptation at pilot scale

Dissemination and large scale adoption (outcome)

Early benefits (economic, social, environmental)

Impacts on poverty, hunger, nutrition

Research outputs generated

TYPE 1

Impact evaluations undertaken in the EARLY PHASE of technology diffusion (adopter level)

TYPE 2

Impact assessments that take place EX POST of technology adoption and measure longer-term consequences of large scale adoption

Methods and tools for impact assessment of agricultural R4D (cont’d)• And can be further sub-grouped in to two types depending

on the level of analysis (Adopter level or Aggregate level)

17

TYPE 1

Impact evaluations undertaken in the EARLY PHASE of technology diffusion

TYPE 2

Impact assessments that take place EX POST of technology adoption and measure longer-term consequences of large scale adoption

TYPE A: Adopter level

TYPE B: Aggregate level

Type 1: Impact evaluations in the early phase of technology uptake and diffusion• Impact evaluations in this phase mostly focus on testing

the effectiveness and impact of programs (or institutional innovations) that promote the adoption of research outputs to end users.

• These types of assessments consider the project promoting technology adoption as the unit of evaluation and center around addressing two types of questions—• How to design ‘interventions’ to increase the adoption of technology

so as to increase the impact of investments in R4D? And, • What are the impacts of development projects that promote the

adoption of agricultural technology on end users?

18

Type 2: Impact assessments undertaken ex post of technology adoption• As implied by its name, these types of assessments take

place after the evidence of adoption of technology• They involve evaluating the magnitude and scale of

economic and social impacts resulting from the adoption of research outputs (i.e., technologies, practices, institutional innovations)

• These types of assessments consider the technology adoption as the unit of evaluation and center around addressing following types of questions—• What are the costs and benefits (or positive and negative impacts)

resulting from the adoption of research outputs (i.e., technology) adopted by end users?

19

Type A: Adopter level impact assessments• These types of assessments are concerned with

establishing a causal link between realized outcomes (the effect) and an intervention (the ‘cause’) at the ADOPTER level

20

• These types of assessments are concerned with establishing a plausible link between realized longer term impacts (the effect) and adoption of research outputs (the ‘cause’) at the SOCIETAL level

Type B: Aggregate level impact assessments

Few words about ‘ex ante’ impact assessment (which is NOT the focus)• It is a tool for decision makers to:

• Assess potential impacts (forward looking)• Help in priority setting (in the context of scarcity)• Make strategic decisions

• Uses diverse methods• Quantitative (e.g., scoring, models that quantify impacts (e.g.,

DREAM, General/partial equilibrium models)• Qualitative (outcome mapping, logical framework analysis,

foresight studies, trend analysis, participatory)

• These methods range in costs (time and resources) and data requirement

21

Few words about ‘ex ante’ impact assessment (which is NOT the focus)What is ex ante impact assessments based on?

• Trends/projections • Past performance (leap of faith)• Theory• Opinions (biases)• Assumptions

Role of ex ante IA:• Information from ex ante assessments can serve as an input in:

– Influencing resource allocation decisions (i.e., the size of the agricultural R&D pie and its distribution)

– Help in system strategic planning

22

Methodological options for impact assessment of agricultural R4D

23

Type 1 (early phase) Type 2 (Ex post)

Type A(adopter level)

Randomized controlled trialsPipeline comparisonsDifference in Difference (DD)Propensity score matching (PSM)Instrumental variables (IV)Regression discontinuity

Difference in Difference (DD)Propensity score matching (PSM)Instrumental variables (IV)Regression discontinuity

Type B (aggregatelevel)

Social Accounting Matrix (SAM)Cost-benefit analysisEconomic surplusComputational General Equilibrium (CGE)

Methodological challenges: Type 1A and 2A

24

• These types of impact evaluations are focused on adopter level impacts

• They are concerned with establishing a causal link between realized outcomes (the effect) (at the adopter level) and an intervention (the ‘cause’) which could be a program, activity, policy change, designed to promote the adoption of a technology generated by a research program or the technology adoption itself

• The goal of the analysis is to ‘rule out’ other possibilities / explanations for the observed effects (internal validity)

Generic methodological challenges• Impact evaluations of Type 1A and 2A in general are

concerned with measuring the ‘effect size’ of an intervention (ESYT), defined as the difference in the outcome with intervention (Yi

T ) vs. without the intervention (Yi

C) for the same unit of analysis i (i.e., thecounterfactual)

(1) ESYi = YiT – Yi

C

• Problem: The same unit of analysis cannot simultaneously be observed ‘with’ and ‘without’ intervention

• This is the classic challenge of estimating the counterfactual – the problem of missing data

25

The Challenge of Missing Data (cont’d)

• Solution to the problem of missing data: Estimate the expected average effect in a population as:

E[YT –YC]

• The credibility with which the counterfactual YC is identified to estimate the expected average effect is at the heart of ‘evaluation designs’

26

The Challenge of Internal Validity• The following methods for estimating the counterfactuals can

give biased estimates:1. Same group (i.e. participants) before the intervention2. Other groups not included in the intervention (non-targeted, non-

participants)3. Other groups who rejected the intervention (targeted, non-

participants)• Why?

1. Many things change between ‘before’ and ‘after’ and one is not sure what was caused by the intervention and what by these other things

2. Development programs are often targeted based on certain criteria—Thus the non-targeted group may be different in characteristics that influence the outcome

3. Self-selection of people in program participation – Thus participants will differ from non-participants in observable and unobservable ways

27

Generic Issues (Challenges) in Impact Evaluations of Type 1A and 2A

• How do we deal adequately with program placement bias and self-selection bias?

• Are there spillover effects on non-participants?• What are the data requirements? (the issue of sample

size)

28

Methodological innovations to address the challenges…(Type 1A and 2A)• Use and promotion of rigorous quantitative methods for estimating counterfactuals• Experimental designs or RCTs (based on the principle

of random assignment)• Pipeline comparisons• Other methods/approaches (using econometric

techniques to form credible comparison groups)• Propensity score matching (PSM)• Instrumental variables (IV)• Regression discontinuity (RD)• Difference in difference (DD)

29

Type B: Aggregate level impact assessments

• They mostly occur ex post of technology adoption• They are concerned with establishing a plausible link

between realized (longer-term) impacts (the effect) and technology adoption (the ‘cause’)

• The goal of the analysis is to assess the contribution of technologies generated by agricultural research in realizing the observed effects at the aggregate or societal level (contribution)

30

Type B: Aggregate level impact assessments (cont’d)

• Involves using an implicit step-wise process by tracing the results along the impact pathway from:• Research to outputs, • Outputs to outcomes (adoption)• Outcomes to impacts

• The step from research outputs to outcomes involves estimating two key parameters—the size of the adoption of a research output and the average effect size, which measures the effect of a research output per unit of adoption compared with a counterfactual

31

Type B: Aggregate level impact assessments (cont’d)

• The step from outcomes to impacts involves using models (such as economic surplus, social accounting matrices or computable general equilibrium) that take into account the equilibrium effects of scaling up the estimated effects per unit of adoption over time and space.

32

Type B: Aggregate level impact assessments (cont’d)

• The aggregate societal ‘benefits’ (over time and space) can be expressed in monetary and non-monetary (physical) terms

• ‘Benefits’ are often compared with ‘costs’ to justify research investments (where feasible)

• This types of assessments are often concerned about who benefits from the technology / output? For e.g.,

Gender—men vs. women (crop, cropping activity) Growing environment--hills vs valleys, rainfed vs. irrigated farms Income group—wealthy vs. limited resource farmers Ethnic group—indigenous vs. non-indigenous people

33

In Summary…• There are multiplicity of methods and approaches that can

be used to assess impacts of agricultural R4D

• There is no ‘one size fits all’ method/approach• This is because the impact pathway from research

investments to generating outputs, and realizing outcomes and scalable impacts is long and complex

• The diversity of methods and approaches of impact assessment reflect the attempt by impact assessment practitioners to apply recent methodological innovations, yet building on a long and rich tradition of impact assessment in agricultural R4D

34

In Summary…(cont’d)• This impact culture includes evaluations and impact

assessment research that occur throughout the continuum extending from • pre-research ex-ante impact assessments, to • causal attribution evaluations linking research outputs to outcomes,

and• plausible contribution modeling analysis linking research outputs to

long-term impacts.

35

In Summary…(cont’d)• The methods and approaches for assessing the impacts

vary across these types of studies depending on:• The time frame along the impact pathway they take place• The size/scale of intervention, • The type of research and evaluation questions they address, and • The unit of evaluation

• But common across this multitude of approaches is the push for measuring and evaluating impacts using rigorous methods that provide meaningful measurements of the worth of agricultural R4D

36

Examples and resources for further exploring….

• Food Policy issue 44 (2014) Special Section – has examples of several types of impact assessments discussed here

• de Janvry, A. Dustan, and E. Sadoulet (2011) Recent Advances in Impact Analysis Methods for Ex-post Impact Assessments of Agricultural Technology: Options for the CGIAR

• Web site: impact.cgiar.org• Walker et al. (2008) Strategic guidance for Ex Post Impact

Assessment of Agricultural Research

37

THANKS

WELCOME QUESTIONS AND DISCUSSION