Can systematic reviews help identify what works and why?

Can systematic reviews help identify what works and why? The case of microfinance in

sub-Saharan Africa

Presentation to 3rd Biennial SAMEA conference8 September 2011

ByCarina van Rooyen, Dr Ruth Stewart & Prof Thea de Wet

How do you want it – the crystal mumbo-jumbo or statistical probability?

• Large development funders wants to know ‘what works’ in development

• Looking for evidence of effectiveness – evidence-informed development policy

Need to demonstrate impact

“UK government support for aid organisations will be targeted at those agencies which

demonstrate they can deliver best value for money while they improve the health,

education and welfare of millions of people in the poorest countries…. We expect these

charities to work hard to prove to UK taxpayers that they will and can make a real difference to the lives of the poorest and deliver real value

for money.” ~ DFID 2010

Impact evaluations (IE)

IE about showing causality

• Causation:– A change in X is related to a change in Y– Not the same as correlation

Counterfactual crucial

Randomistas provide the answers?

‘Gold standard’ study design advocated by ‘randomistas’ – led by influential academics at the Abdul Latif Jameel

Poverty Action Lab (JPAL)

Population

“Creating a culture in which randomised evaluations are promoted, encouraged and financed has the potential to revolutionise social policy during the 21st century, just as randomised trials revolutionised medicine during the 20th.” ~ Esther Duflo quoted in Lancet Editorial, “The World Bank is finally embracing science” (2004)

RCTs in development

But are RCTs sufficient?Methodological debates about RCTs raise number of concerns within the development community, including

Dismissal of other evaluation techniques: hierarchies and ‘gold

standard’

Lack of consideration of contextual information: over-simplification with

generalisable information stripping out contextual details

Narrow focus on linear causal relationships: experimental designs over-simplify complex issues

RCTs questioned in development

Narro

w a

pp

roach

to

evid

en

ce

Trials are costly, have ethical dilemmas & are often lacking

Solutions are urgently required

Heterogeneity raises serious concerns about external validity of such trials

Systematic reviews (SRs) to the rescue?

• Can these concerns about RCTs be overcome through the use of SRs?– Led by the Cochrane Collaboration, SRs

routinely used in health care to combine results of RCTs

– Integrated into health policy internationally

– In development promoted by funders

SRs in the development field• About 100 SRs in

international development commissioned so far ~ Howard White (chair of IDCG)

• First SRs in development published: water and environmental sanitation (Waddington & Snilstveit 2009), HIV behaviour change (Noar et al 2009), microfinance (Stewart et al 2010)

SRs in the development field (cont.)

• Four registered SRs with IDCG (Campbell Collaboration)– cash transfers for health & nutritional outcomes in

poor families– deworming for improving school attendance in

school-aged children– impact of farmer field schools– effectiveness & sustainability of water, sanitation

hygiene interventions in combating child diarrhoea

• IDCG expects to register more titles later in 2011 in CCTs in education, governance and anti-corruption, urban development, social protection & microfinance

What is a systematic review?

• Is about the evidence of effectiveness

• Thorough & systematic collection of all relevant evidence & its quality appraisal and synthesis– Typically combine evidence from RCTs

• Designed to minimise biases & errors inherent to traditional, narrative reviews

Elements of a SR• Formulate the review question & write a

protocol which is peer reviewed• Search for and include primary studies

• Assess study quality• Extract data• Analyse data

• Interpret results & write a report, which is peer reviewed

Comprehensive strategy to search for relevant studies (unpublished & published)Explicit & justified criteria for inclusion or exclusion of any study

Statistical synthesis of data (meta-analysis) if appropriate and possible, or qualitative synthesis

Rigidity of SRs: Hierarchy of evidence?

Do you ever think sometimes, you might be overdoing the whole moisturiser thing?

Risks with methodological rigid SRs?

• Narrow menu of methodological options could mean reduction of development to simple interventions, in order to facilitate its measurement (Guijt et al 2011:4)

• “Those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable.” (Natsios, ex USAID quoted in Guijt et al 2011:3)

Similar critiques against methodological rigid SRs in

development that against RCTs in development

Our SR on the impact of microfinance on the poor in SSA

Our pragmatic approach

Followed pragmatic approach for our SR in five important ways:1. Focused on REGIONAL

rather than worldwide evidence

Our pragmatic approach (cont.)

2. Multi-disciplinary nature of our team3. Using range of sources: not only

electronic data bases (publication bias)

Our pragmatic approach (cont.)

4.Methodological: – Drew on well-conducted evaluations with

comparative research design, including RCTs, but also non-randomised trials, quasi-experimental designs, and simple with-and-without studies• For purists this ‘weakened’ confidence in

evidence of impact– ‘rigour’ narrowly defined in terms of statistically

significant indication of difference with and without an intervention – internal validity (Guijt et al 2011:7)

Our pragmatic approach (cont.)

–We argue for ‘good enough quality’: rigour include aspects such as utility, external validity, method mastery, sense-making & substantiated methodological choice (Guijt et al 2011:7)

– In practice we also broadened the scope of our study • Able to look at additional types of interventions &

outcomes which haven’t yet been evaluated by RCTs

• Draw on evidence from additional countries

Details of 15 included studies

• 4 RCTs 2 quasi-experimental studies 9 with/without studies• 11 = microcredit, 2 = savings, 2

= combined credit & savings• Ethiopia, Ghana, Kenya, Malawi,

Madagascar, Rwanda, South Africa, Tanzania (Zanzibar), Uganda & Zimbabwe

• Rural & urban initiatives

Is ‘good enough quality’ good enough?

 Outcomes Source of

evidence

Savings Credit Combined credit

& savings

Savings All evidence YES + (2) YES + (1) YES + (1)

Trials only YES + (2) No evidence YES + (1)

RCTs only YES + (2) No evidence No evidence

Expenditure &

Assets

All evidence MAYBE mixed (1) YES + (4) MAYBE mixed (2)

Trials only MAYBE mixed (1) YES + (1) MAYBE mixed (1)

RCTs only MAYBE mixed (1) No evidence No evidence

Incomes All evidence YES + (2) YES + (1) YES + (1)

Trials only NO no effect (1) YES + (2) No evidence

RCTs only NO no effect (1) YES +(1) No evidence

“The quality of evidence about effectiveness should be judged not by

whether it has used a particular methodology, but whether it has

systematically checked internal and external validity, including paying attention to differential effects.”

(Rogers 2010:195)

If methodological purist (exclude any study with indication of bias) possible conclusion that evidence not good, e.g. Duvendack et al’s SR on impact of microcredit worldwide

• Clemens & Demombynes (2010:1) refer to luxury versus necessity

• White (2011a) refers to choice between technical quality & policy influence– Risk of purist is that have nothing to say to

policy makers as want definitive free-from-bias answer

– Risk of pragmatist is that, while providing policy makers with ‘better’ information than what otherwise would have, might have bias

Our pragmatic approach (cont.)

5. Development of causal pathway in which we explored how microfinance works to be able to draw conclusions about why microfinance does or does not work & for whom– What achieved (outcome) & how (process)– Conventional SRs limited to evidence of

effectiveness, but this more enhanced approach allowed informed conclusions to be drawn

– Evaluative ‘proving’ & improving

41

Causal pathway analysis

What we now think is happening

1. Invest in

immediate future:

a. Business

b. Productive assets

c. Adult education

d. Workers’ health & nutrition

Micro-credit

Given to individuals or groups

Scope for increased income via business or

employment

Able to repay loan and avoid

increase in debt

Able to save

Spend money differently

Social cohesion

Women’s empowermen

t

Long-term benefits

Micro-savings

2. Consumpti

ve spending

with scope for

productivity:

a. Add on housing

b. Assets which retain

value

Improved capabilities

Better able to deal with shocks

3. Invest in

long- term future:

a. Children’s education

b. Children’s health and nutrition

4. Consumpti

ve spending

(non-productive)

:

Assets which do not retain

value

Actual increased income

Actual decreased income

Default on loan, lose collateral and/or forced

to borrow more

Use other MFI

FOR CREDIT CLIENTS ONLYInability to repay loan

Determined by external factors:

Entrepreneurial ability

Appropriateness of business in context

Competition from other MFI clients

Gender and power relations

Use same MFI

Some of our recommendations

More and better impact evaluations of microfinance (especially savings)

On-going discussion of how to deliver pragmatic systematic reviews for international development

Next steps

• SR methodology to be further enhanced to serve the needs of development– Incorporating studies of poor people’s

experiences, priorities & views (constructivist view): something similar has been done in health promotion, e.g. EPPI healthy eating review

– Combining reviews of published evidence with primary research, e.g. Thuthuka project

– Systemic approach to M&E and impact evaluations

Three challenges for M&EConsider findings of SRs to enhance individual programme evaluations, establishing what best available evidence shows & placing evaluation of individual projects within context of this broader evidence base

Consider RCT designs as one part of solution to impact evaluation, and explore where evaluations which you are able to conduct can fit within broader evidence base to shed light on key issues in development

Conduct pragmatic SRs to inform decision-making in development – flexibility

The latest research shows that we really should do something with all this

research

Conclusion

• About what works for whom under what circumstances and how

• SRs help to think about strategic issues, rather than specific project intervention

• There are limitations with SRs & they are very reliant on existence & clear reporting of individual evaluations– SR is only as good as the included

studies (garbage in, garbage out)

Conclusion (cont.)

• But – They are bigger than individual studies– They take into account relevance, rigour

& vigour

• With causal pathway analysis (theory of change), they go some way to translating research evidence into meaningful policy & practice insights

• So, can SRs help identify what works and why?

• Based on our SR on the impact of microfinance on the poor in sub-Saharan Africa, yes

• But have to be pragmatic / flexible in approach to SR in the field of development

Source

of ca

rtoon: G

uijt e

t al 2

01

1:i

Source

: http

://ww

w.h

ow

-matte

rs.org

/2011

/05/2

4/rcts-b

an

d-a

id-o

n-d

eeper-issu

e/

Not everything that can be counted counts, and not everything that counts can be counted ~ Albert Einstein

Thank you for [email protected] online at

http://www.slideshare.net/carinavr

References / Acknowledgements• Blattman C 2008 Impact evaluation 2.0. Presentation to the Department for International

Development (DFID) London on 14 February 2008. Available at http://www.chrisblattman.com/documents/policy/2008.ImpactEvaluation2.DFID_talk.pdf

• Blattman C 2011 Impact evaluation 3.0? 5 lessons and reflections after a couple of more years of failure and success. Presentation to DFID on 1 September 2011. Available at http://www.chrisblattman.com/documents/policy/2011.ImpactEvaluation3.DFID_talk.pdf

• Cummings S 2010 Evaluation revisited 1: Rigorous versus vigorous. Blog posting at http://thegiraffe.wordpress.com/2010/06/17/evaluation-revisited-i-rigorous-vs-vigorous/ on 17 June 2010

• Deaton A 2010 Instruments, randomisation and learning about development. Journal of Economic Literature 48: 424–455

• Gertler PJ, Martinez S, Premand P, Rawlings LB & Vermeersch CMJ 2010 Impact evaluation in practice: Ancillary material. World Bank: Washington DC (www.worldbank.org/ieinpractice)

• Guijt I, Brouwers J, Kusters C, Prins E & Zeynaloba B 2011 Evaluation revisited: Improving the quality of evaluative practice by embracing complexity (conference report)

• Hughes K & Hurchings C 2011 Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental Global Performance Framework (3ie Working Paper 13). New Delhi: 3ie

• Rogers P 2010 Learning from the evidence about evidence-based policy. In Banks G (eds) Strengthening evidence-based policy in the Australian Federation. Melbourne VIC: Productivity Commission: 195-214

• Photos and cartoons not acknowleged on slide were found via Google Images