IDS Impact, Innovation and Learning Workshop March 2013: Day 2, Paper Session 3 Laura Camfield

Ensuring rigour and reliability

Laura Camfield, University of East Anglia

[email protected]

Qualitative methods in modern impact evaluation Who’s doing it?

How are they doing it?

How is its quality being assured? And are existing checks sufficient?

Ethical codes, peer review

Archiving

Restudies

2

Endorsement

from DFID (funded

reports by Garbarino and

Holland, Stern)

And from

NONIE, 3ie,

World Bank IEG,

Ford

Foundation, etc.

Growing interest in approaches to

causation that incorporate

qualitative methods

Small n/ case study

Theory Based

Realist

Process tracing

Range of research designs limited

Qualitative research typically post-hoc

rather than ex-ante

Integrated analyses are rare

Hard to access full methodological accounts

– including how the analysis was done - or

raw data

Growing numbers of mixed methods designs, many ongoing 3ie – 700+ studies, 50 qual or mixed

methods (education, health, social protection, rural livelihoods)

But qualitative component often small or superficial, ◦ e.g. sampling through participatory wealth

ranking or using criteria from PPA, ‘field visits’, ‘qualitative surveys’

‘Empowering parents to improve education: evidence from rural Mexico’

◦ Reports data from 30 focus groups with parents from treatment and control schools, but no analysis of differences between groups or how the programme promoted parental participation. For further information we are referred to an internal report

‘Changing Households’ Investments and Aspirations through Social Interactions’

◦ Mentions ‘focus groups and semi-structured interviews with a wide set of beneficiaries and other local actors’ which ‘suggests that aspirations and perspectives towards the future may have been a key part of program impact’ (p9). The methodology and data underpinning these claims are in a report that has not been translated and isn’t available online

‘Assessing the Impacts of Farmer Field Schools on Excessive Fertilizer Use in China’ Detailed description of RCT methodology and sampling -

‘qualitative data will also be collected to see what worked and how it worked'

‘Evaluating the Impact of Technology Development Funds in Emerging Economies: Evidence from Latin America’ Qualitative methods not included in the methodology, but

emerges like a rabbit from a hat ... ‘finally, based on qualitative evidence from this study, financial support should be accompanied by the infrastructure and technological services of research centres and universities’

◦ Data production

‘Intellectual biographies’ of researchers and fieldworkers

Quality of note taking – capturing embodied knowledge?

◦ Data management

Transcription

Translation

◦ Analysis

‘Tertiary’ analysis

◦ Representation

Moving beyond quotes and text boxes

Evidencing claims - ethnographic authority?

9

1. Record of publications, presentations etc which explain and /or draw on the archived data.

2. Research design

a) Brief context and logic of research design (in each data collection period if appropriate).

b) Is research exploratory or question driven? What are the questions?

3. What were the sampling decisions and how do they relate to the research questions?

a) Was the desired sample achieved? How does the sample relate to wider empirical evidence across the population and/or theoretical issues?

b) Are there implicit as well as planned ways in which the sample is structured?

4. An overview of what data is provided as part of the project.

5. A descriptive profile of each participant specifying units of data (interview; diary; by wave).

10

Rigour, Credibility, Relevance ◦ 1. Was there a clear statement of the aims of the research?

◦ 2. Is a qualitative methodology appropriate?

◦ 3. Was the research design appropriate to address the aims of the research?

◦ 4. Was the recruitment strategy appropriate to the aims of the research?

◦ 5. Were the data collected in a way that addressed the research issue?

◦ 6. Has the relationship between researcher and participants been adequately considered?

◦ 7. Have ethical issues been taken into consideration?

◦ 8. Was the data analysis sufficiently rigorous?

◦ 9. Is there a clear statement of findings?

◦ 10. How valuable is the research?

Validity in Quantitative and qualitative research

12

Quantitative research

Qualitative research

Bryman Guba & Lincoln “Naturalistic inquiry”

“trustworthiness and authenticity”

Internal validity Legitimate causality claims Do the observations correspond well to the

theoretical constructs.

Can you support claims that you observed

what you claim?

Credibility Multiple accounts of the same phenomenon

researcher validation

respondent validation

triangulation

External validity Generalisability

(representative samples)

Can the qualitative findings be transferred

across social contexts.

Transferability Research on small groups;

Depth vs. breadth

“Thick” description (Geertz)

Reliability Repeatability, replicablity Could the qualitative research be replicated

(access to qualitative inquiry “texts”)?

Researchers adopt similar social roles

Do researchers agree about what they see

and here? – inter-observer credibility

Dependability Trustworthiness by providing “audit trail” –

extensive records of all aspects of the research,

including raw data

Re-studies

Objectivity Objectivity, methodological

rigour

Objectivity in social science is spurious

Values more embedded in qualitative

research

Confirmability Researcher worked in good faith

Possible solutions (Camfield and Palmer-Jones, 2013)

Ethical codes, peer review

Data archiving

Restudies

13

Social science ethical codes typically focused on care of the subject, but see GoS’s (2007:1) emphasis on ‘respect, rigour and responsibility’ ‘act[ing] with skill and care in all scientific work’

‘communicating results and intentions honestly and accurately, and understanding that your work or its outputs will have an impact on society in its broadest sense’

Not all countries/ institutions/ disciplines have ethical codes or committees

Ethics committees assess research designs, but do not monitor data production or presentation

Peer reviewers rarely see the data – no qualitative data deposit with journals – so much is taken on trust/ reputation

Grey literature and working papers may not receive the same level of scrutiny, but are still influential

‘Audit trail’ obscure in earlier examples

What does it offer? ◦ Methodological insights

◦ Increased use of data for research and teaching

◦ Respect for respondents’ accounts and time

◦ Judging validity of claims

◦ Historical perspective

Secondary analysis is difficult to do well, but within impact evaluation, most qualitative analysis is secondary, or even tertiary

‘Scientific replication’ ◦ Goldthorpe and Lockwood (1962-3) tested the ‘affluent worker’

hypothesis by taking highly-paid car assembly workers at the Vauxhall factory in Luton as a ‘critical case’ to investigate whether everyone was becoming middle-class. They concluded that this wasn’t the case, albeit on the basis of a small quantitative sample and (Savage argues) inadequate engagement with the data. Savage (2005:39) suggests that the conclusion arose because they had fitted their data into a particular typology which closed off alternative interpretations.

Methodological insights ◦ Gillies and Edwards (2011:23) describe how in Townsend and

Marsden’s (1965) study of single mothers, their “physical attractiveness (or lack of it) is commented on [...and] perceived intelligence and character was also subject to evaluation”.

Some examples from developing countries, e.g. ESRC funded restudy of Indian villages studied in the 1950s by Bailey, Mayer and Pocock; van Schendel, 1981 (Bangladesh); Breman et al., 1997 (S. & S.E. Asia); Breman, 2007 (Gujarat, India),

And some controversies, often due to ‘interpretive overreach’ (e.g. Freeman vs. Mead, Lewis vs. Redfield, Tierney vs. Chagnon)

Why restudy? (Dis)confirming original findings

Epistemological/ methodological insights

Longitudinal perspective

Running out of field sites...

Constructivist revisits (types 1 and 2) assume the site being studied at two points in time does not change so differences are due to the different relation of the ethnographer to the site (type I) or theory that the ethnographer brings to the site (type 2) e.g. Weiner's (1976) feminist reconstruction of Malinowski’s

(1922) Argonauts of the Western Pacific highlighted the importance of mortuary ceremonies in cementing women’s control over ancestral identity alongside Malinowski’s celebration of the ‘Kula ring’

Positionality and assumptions also important in evaluation

Realist revisits (types 3 and 4) study historical change. Type 3 revisits focus on internal processes in accounting for differences between authors’, while type 4 emphasise external forces Hutchinson’s (1996) revisit to Nuerland (now Southern

Sudan), which was studied by Evans-Pritchard (1940), explicitly to explore the impact of decolonization, war, Christianity, and transnational capital on the lives of the Nuer

Hard to reliably attribute and understand effects of particular interventions in complex, changing environments

Would Bemba society (N. Zambia) break down when men migrated to southern African mines because the slash and burn agricultural system (citimene) needed men to cut down the trees? ◦ Richards – yes (conclusions enthusiastically adopted by colonial

administrators/ chiefs for their own reasons)

◦ Moore and Vaughan (50 years later) – no (real threat was Zambian government’s agrarian reforms which promoted the labour-intensive cash crop maize)

But Moore and Vaughan’s analysis may also have been ‘one-sided, governed by specific feminist and Foucauldian assumptions’ (Burawoy, 2003:667) and effects will be seen in future revisits…

What lessons can be learned for evaluation? ◦ Watch out for interest groups!

◦ Don’t forget about the women

◦ Impacts of interventions are often unintended, multi-dimensional, and affect every member of the household, not just the target maize income and women’s labour controlled by men,

women diverted from subsistence farming, family becomes food insecure and women not entitled to take a share of the income to buy food, children’s health suffers...

Quality of research is an ethical issue

Increasing role for qualitative research in impact

evaluation requires increased reflection on current

standards

Epistemological and methodological problems

cannot easily be detected or influenced through

conventional processes (standards, committees....)

But qualitative researchers have own strategies for

ensuring rigour and reliability

Data archiving and restudies contribute to these