Using multiple method to join the dots

1

Using multiple methods to join the dots:

Assessing Rural Transformations

Fiona Remnant, IDS

Oct 2015

2

The ChallengeHow can we really tell what impact a project

has had on beneficiaries?

Trade-offs between credibility and cost-effectiveness in impact evaluation… particularly in complex environments.

– Large surveys – often inflexible, expensive and slow– RCTs – limited applicability; honeymoon over; – Qualitative approaches – but what is credible whilst

still cost-effective?

3

GoalsAction research to design and pilot a ‘good enough’ credible, rapid and cost-effective qualitative impact assessment protocol (QUIP) which in the context of complex rural transformations can: • confirm project theories of change, and • explore unanticipated drivers of change• complement other M&E activities by

addressing the impact attribution gap

4

Development Agencies

(X)

Diverse,

complex

risky

contexts

(Z)

Improved livelihoods &

wellbeing (Y)

QUIP:Respondents’

accounts

Development Agencies

(X)

Improved livelihoods &

wellbeing (Y)

Livelihoods & wellbeing?

The attribution problem

5

Case study projects (X), indicators (Y) and confounding factors (Z)

X Y Z

Project 1. Groundnut value chain (Central Malawi)Project 2. Climate change resilient livelihoods (Northern Malawi)Project 3. Malt barley value chain (Southern Ethiopia)Project 4. Climate change resilient livelihoods (Northern Ethiopia)

• Food production • Cash income • Food consumption• Cash spending• Quality of

relationships • Net asset

accumulation• Overall wellbeing

• Weather • Climate change• Crop pests and

diseases• Livestock mortality• Activities of other

organisations • Market conditions• Demographic

changes• Health shocks

6

Ten features of the QUIP1. Data collection by independent field researchers

without knowledge of the project (blinding).2. Purposive then random household sampling based

on quantitative project monitoring.3. Semi-structured household interviews plus focus

groups.4. Data collection instruments structured by wellbeing

domain, with alternating open then closed question sections.

5. Data entry using pre-formatted Excel sheets to facilitate coding and analysis.

7

… continued6. Systematic coding of impact evidence as explicit,

implicit or incidental to project actions.7. Additional flexible coding of drivers of change

(positive and negative). 8. Rapid semi-automated report generation to speed

analysis. 9. Easy to drill down from summary evidence to raw

data for auditing and learning purposes.10. Summary report, starting point for project level

debriefing between project staff and researchers.

8

Central Malawi improved groundnut seed: responses to QUIP2 closed questions

Code Gender Beneficiary?* 1. Food Production 2. Cash income

3. Purchasing power

4. Food consumption 5. Assets

6. Overall Wellbeing

PK3 Male ++ + + + + + + PK6 Female + + - = - = = = PK7 Female + + - = - - + - KM3 Female + + + + + + + +

KM10 Male + + + + + + + + KM11 Male + + + + + + + + KM12 Male + + - + + = + + PK8 Female + - - - - + - PK5 Female + - - - - = =

PK10 Female + - - - = - - KM1 Female + = - - - + - KM2 Male + + + + + + +

KM15 Female + - - - = = = PK1 Female / - - - - = - PK2 Female / - + = = + + PK4 Female / - = - - - + PK9 Female / - - - = - - KM4 Female / = - = = - = KM5 Female / = - = = + = KM6 Female / = - = - = - KM7 Female / - + - - + + KM8 Male / - - - - + - KM9 Male / - - - - + -

KM13 Female / - - - - + - KM14 Female / + = - = + =

KM16 Female / - - - - = -

* + Beneficiary to some degree; ++ Beneficiary to a larger degree; / Non-beneficiary

9

Positive change statements by domain

Bold figures indicate beneficiary households

1 Project explicit 3 Project implicit 5 Other

Food production KM2, KM3, KM10, KM11, KM12,

PK3 PK1, PK2, PK9, KM13,

FGKM2, FGPK1,

Cash income KM2, KM3, KM5, KM10, KM11, KM12,

PK3, KM12, PK1, PK2 PK5, KM7

FGKM2, FGPK1,

Purchasing power KM2, KM3, KM10, KM11, PK3 PK3, PK5,

FGKM2, FGPK1,

Food consumption KM2, KM3, KM10, KM11, PK3

FGKM2, FGPK1,

Intra-hh & villagerelationships

KM10, KM11, PK4,

FGKM2, FGKM2, FGPK1 FGPK1, FGPK2

Asset accumulation KM2, KM10, PK3,

FGKM2, FGPK1,

Links with external orgs

PK4, KM2, KM3, KM5, KM10, KM11, KM12, KM15,

KM16 PK3, KM7, KM13, KM14, KM15,

FGKM2, FGPK1, FGKM1 FGKM1, FGPK1

Overall wellbeing KM10, KM11, KM12 PK3, PK4 PK2, KM7, PK3

FGKM2, FGPK1

10

Drilling into positive statements Section C1: Food Production

1. Explicit project and linked causes (positive) Attribution

[KM10 C1] The respondent noted that the ability to produce their own food had changed for the better. He attributed this to use of livestock manure, compost and Bocash manure in the fields as taught by Self Help Africa. He also said that the adoption of hybrid seeds as recommended by Self Help Africa has contributed to this because the hybrids are high yielding. He also said that the ability to own livestock had increased too. He said he received two pigs from Self Help Africa some time back which at some point sold some and then bought cattle. On taking up new activity he cited the manure making and application as an activity he had just embarked on recently which has helped ensure that own produced food never runs out in his household. He said he had stopped total reliance on inorganic fertilizer. He said that he was not doing anything differently to the majority of the community members because they too have taken up manure application and use of hybrid seeds in their crop production.

SHA Training

SHA seed credit

11

Negative change statements by domain

Bold figures indicate beneficiary households

2 Project explicit

4 Project implicit 6 Other

Food production KM15, PK7 PK1, PK2, PK4, PK5, PK6, PK7, PK8, PK9, PK10, KM6, KM7, KM8, KM9, KM12, KM13, KM14, KM15,

FGKM1, FGPK1, FGPK2Cash income PK1, PK2, PK4, PK8, PK9, PK10, KM1, KM7, KM8, KM9, KM12, KM15,

FGKM1, FGPK2Purchasing power PK1, PK2, PK4, PK8, PK9, PK10, KM5, KM7, KM8, KM9, KM12, KM13,

FGKM1, FGPK1, FGPK2Food consumption

PK1, PK4, PK5, PK10, KM6, KM8, KM9, KM13,

FGKM1, FGPK2Intra-hh & villagerelationships

KM15, PK5, PK7, KM5,

FGPK2, FGKM1, FGKM2, FGPK2Asset accumulation

PK10, KM7, KM8

FGKM1, FGKM2, FGPK1, FGPK2Links with external orgs

PK6, PK7, KM7,FGKM1, FGPK2

KM5 FGKM1, FGKM2

Overall wellbeing PK7, KM15 PK1, PK5, PK8, PK9, PK10, KM1, KM5, KM6, KM8, KM9, KM13,KM16 FGKM1, FGKM2

12

Positive Drivers

Health Food production Income Purchasing

powerFood

consumption

Relationships (intra hh and

village)Assets Overall

wellbeing

SHA Training

KM10, PK3

KM2, KM10, KM11, PK3

FGKM2, FGPK1

KM2, KM10

FGKM2, FGPK1

KM10, KM11

FGKM2, FGPK1

KM2, KM10, KM11

FGKM2, FGPK1

KM10 PK3, KM10

FGKM2

KM10

FGKM2, FGPK1

SHA Seed Credit KM10, KM3, KM12

KM3, KM5, KM11, KM12

KM2, KM3 KM3 KM2 KM11, KM12, PK4

SHA Gender training KM10, KM11

FGKM2

Religion/ God PK5, KM11 FGPK2

Investment in livestock PK1, PK2

Investment in soy production PK2, PK9

Village savings and loans group KM13 PK3 PK3, KM7

Increased farming of wetland

PK3 PK3 PK3 FGPK1, FGKM2 FGPK1 PK2, PK3

Other income generation KM7, PK1,

PK2, PK5 PK5

13

Negative DriversHealth Food production Income Purchasing power Food consum-

ptionRelationships (intra

hh and village) Assets Overall wellbeing

Fertiliser cost (and subsequent lower yields)

PK1, PK5, PK7, PK8, PK10, KM6, KM7, KM9, KM13, KM15

KM7 PK1, PK2, KM7, KM13

PK1, PK5, KM13 KM7 KM5, PK1,

PK5, KM13

FGKM1, FGPK1, FGPK2 FGKM1, FGPK2 FGKM1,

FGPK2FGKM2 FGPK2, FGKM1 FGPK2 FGKM1

Illnesss/ deathPK8, KM1, KM4, KM12, KM15, KM16

PK7, PK9, PK10, KM8, KM12, KM13, KM16

PK8, PK10, KM1, KM8, KM9, KM12

PK8, PK9, PK10, KM8, KM12

KM6, KM8, PK10 PK10, KM8

PK8, PK10, KM1, KM6, KM8, KM16

Loss of livestock (disease, theft) PK1, KM12, KM14 KM9 KM9 KM9

Erratic rainfall PK2, PK6FGPK1 PK2

Unprofitable business (crop sales and other)

KM15FGPK1, FGPK2

FGPK1 KM9

Village savings group debt repayments

KM5 FGPK1 FGKM2FGKM1FGKM2 KM5, KM9

FGKM2

SHA seed credit debt (failure to germinate)

PK7, KM15 PK7

Domestic dis-agreements KM14 PK9 PK9

Absent husband PK4 PK4 KM5, PK4 PK4 KM5, PK5 Low tobacco prices (stopped growing as a crop)

PK1, PK2

14

Interventions & reported drivers of changeDATE Intervention KM1 KM2 KM3 KM10 KM11 KM12 KM15 PK3 PK5 PK6 PK7 PK8 PK10

Q4 2012

Training in crop husbandry practices ü ü ü

Groundnut seed distributed ü ü ü ü

Q1 2013 Training in value addition

Q4 2013

Training of farmers in seed multiplication principles and general crop husbandry practices

ü ü ü ü

Groundnut seed distributed (basic)

ü

Groundnut seed distributed (certified)

ü ü

Soya seed distributed (basic) ü

Q3 2014

4 groundnut shellers given to the cooperative

Q4 2014

Groundnut seed distributed (basic)

ü

15

Methodological discussionIssue Limitations Ways forward

Scope for relying on self-reported attribution.

Ok for evidence of contribution, but weak on magnitude of impact.

Combine with quantitative monitoring and micro-simulation.

Addressing confirmation bias through blinding of researchers.

Possibly a one-shot opp. for field researchers & difficult for NGOs to replicate.Analysts can’t be blinded as they must also know the theory of change.

No substitute for professional integrity of field researchers. Analysis can be transparent and replicable.

Improve generalizability of findings through opportunistic (re)sampling given that the power of each additional interview is additive (cf. “problem driven iterative adaptation”)

Lack of sufficiently systematic monitoring data to inform selection.Opportunistic sampling may suffice for contribution analysis, but not quantitative attribution.

Improve monitoring of impact indicators (walk before you can run).Need for more systematic sampling if it is to inform micro-simulation.