Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012

Africa RISINGM&E Expert Meeting

Addis Ababa, 5-7 September 2012

Outline

• Timeline

• Objectives of M&E

• Targeting vs Research

• Monitoring and Evaluation

• Theory of Change

• Evaluation designs

• Open issues

3-9 Months

9-12 Months

Timeline

✔1-3

Months

Site selection/Characterization

ComponentInventory

Activity -> Indicator List

Survey Design

Baseline Survey

Component DB

✔

Evaluation Design

Objectives of M&E

• Keep track of the AR’s outputs (M)

• Assess the impact of AR (E)

• Create knowledge of what works (M&E):

-assess effectiveness

-rank policy/project alternatives

-appraise scale-up and external validity

-identify problems in the project

-inform project management

What is AR M&E after?

• M could be an easy(ier) task but:

-we need constant and timely information

-we need close collaboration among CG, implementers, and IFPRI

->Solution: Outcome mapping? Cost-benefit analysis? Surveys?

• E must be well thought in advance…

Targeting vs Research

1. Do we need to select the poorest/hungriest farmers to benefit from AR?

2. Or, being AR a research project, is it possible to use a stratified sample (based on different criteria)?

2a. common E approach?

2b. project-specific E approach? It depends on the question(s)!

Evaluation

• Determine the causal effect of AR on outcomes (not only on outputs):– Farmers’ wellbeing?– Land productivity?– Input supply, labor productivity, environment, women’s

conditions, health and nutrition,…?– …all of the above plus-> for whom? For which development

domain? For which type of households? For which livelihood?

• What would be the impact with a different technology package?

Monitoring Evaluation

‘Traditional’ M&E: monitoring to track implementation efficiency

(input - output)

Impact Evaluation: estimate causal effectiveness on outcomes (output

- outcome)

INPUTS OUTCOMES/IMPACTSOUTPUTS

MONITOR EFFICIENCY

EVALUATE EFFECTIVENESS

$$$

BEHAVIOUR

Note: Diagram from WorldBank training material produced by Arianna Legovini, Lead Economist - AIEI

Theory of Change

• Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of AR

• How would you think that AR can affect the life of (beneficiary) farmers?

Livelihood Strategies /Coping Strategies

/Vulnerability to Shocks

AR implementation

Productivity increase

Income and Expenditure Saving/

Investment

General Household Expenditure

School Enrolment and

Attendance

Expenditure on Health and Education

for children

School ProgressionHealth Status

Food IntakeDietary

DiversityFood Security

Psychological well being *

Other Expenditures for children: Food, Clothing,

Recreation

Targeting ACTIVITIES

OUTPUTS

FIRST ORDER OUTCOMES

SECOND ORDER OUTCOMES

Asset Building

THIRD ORDER OUTCOMES

Labour Participation

Child Labour Remittances

Access to Credit

Time Allocation of

Children

Time Allocation/ Caring arrangements/

Migration of Adults and Caregivers *

Intra-household decision making *

Utilization of Health Services

Time and risk preferences

Impact Evaluation

• How would you go about measuring the causal impact of AR on …

-productivity?

Y

t

Impact?

Impact Evaluation - Method

AfterBefore

Bens

Y

t

Impact

Impact Evaluation - Method

AfterBefore

Bens

Non Bens

RCTs

Y

t

Pre-existing Difference

Impact

Impact Evaluation - Method

AfterBefore

Bens

Non Bens

Diff in Diff

Y

t

Propensity Score Matching

Impact

Impact Evaluation - Method

AfterBefore

Bens

Non Bens

Diff in Diff

with

Impact Evaluation

• How would you go about measuring the causal impact of AR on …

-Gross margin per unit of land? [assume we are not interested in farmers with <.3 ha]

RDD0

500

01

0000

150

002

0000

250

00G

ross

ag

r. r

eve

nue

per

hec

tare

in P

PP

0 .2 .4 .6 .8 1(sum) landcult

Impact Evaluation - Method

• Causal effect: change that is due to AR and not to other actors or factors (confounders)– … taking into account any other factors also

changing during the program period– … taking into account any systematic

differences between beneficiaries and non-beneficiaries of AR intervention

It is very important that the “control group” is comparable to the “treatment group”

Evaluation DesignTreatment Sites Control Sites

• How can we ensure that treatment and control sites are comparable?

Evaluation Design

• How can we ensure that treatment and control farmers are comparable?

Evaluation Design

Random Treatment Assignment

Evaluation Design

Random Treatment Assignment

Evaluation DesignTreatment sites Control Sites

BENEFICIARY

FARMERS

WOULD BE BENEFICIARY

FARMERS

A B

Y

t

Pre-existing Difference

Impact

Where do we stand?

AfterBefore

BASELINE

A

B

BENEFICIARY

FARMERS

Evaluation DesignTreatment sites Control sites

WOULD BE BENEFICIARY

FARMERS

A BWOULD BE NON

BENEFICIARY FARMERS

NON BENEFICIARY

FARMERS

C D

Open issues/1• What questions we would like AR to

answer?

• What R4D lessons can we learn from jumpstart projects? What would you carry over to longer-term AR activities?

• Do we need to target AR to specific farmers?

• Role of FtF indicators -which data to collect-?

Open issues/2• Choice of outcome indicators/variables

• Ethics for control (same as placebo effect in medicine)

• Sampling frame for randomization

• Sample design

• Statistical power for causal impact

Site 1 Site 2