(1) Systematic Reviews: The Potential of Different Methods Steve Higgins Henry Potts James Thomas Geoff Wong ESRC Research Methods Festival Oxford, 5 July.

(1)

Systematic Reviews: The Potential of

Different Methods

Steve HigginsHenry Potts

James ThomasGeoff Wong

ESRC Research Methods FestivalOxford, 5 July 2012

(2)

Introduction

• Aims– To outline major approaches to systematic

reviewing and research synthesis– To demystify what on first glance looks like a

bewildering set of overlapping approaches– To examine some of the assumptions that

underpin these approaches– To discuss how these approaches might help in

my (your!) situation

(3)

Introduction

• Session structure:– 20 Minute presentation + 10 minutes discussion

/ clarification of four approaches to synthesis– Coffee break after the 3rd / 4th presentation– Discussion in pairs– Questions + panel discussion

(4)

What is a systematic review?

• Systematic: ‘done or acting according to a fixed plan or system; methodical’

• Review: ‘a critical appraisal of a book, play, or other work’

(OED)

(5)

What is systematic research synthesis?

• Synthesis: ‘The process or result of building up separate elements, especially ideas, into a connected whole, especially a theory or system’ (OED)

• Not just a report of the findings of the individual studies in a review

• Involves a transformation of the data from primary studies in some way

• Synthesis of the findings of all the included studies in order to answer the review question

(6)

Aims of synthesis reviews

• Answer questions: what do we want to know and how can we know it?

• Bring together and ‘pool’ the findings of primary research (i.e. clarify what we know)

• Any question so potentially any type of evidence

• Driven by review users to answer relevant questions in relevant ways

(7)

Introduction

• We will look at four main approaches to reviewing:– Meta-analysis– The synthesis of ‘qualitative’ research– Realist review– Meta narrative review

• Each have different disciplinary backgrounds and underlying assumptions

• They all bring together research using two main logics -

(8)

Aggregation in reviews

Aggregation refers to ‘adding up’ (aggregating) findings from primary studies to answer a review question…

… to indicate the direction or size of effect

… and our degree of confidence in that finding

(9)

Configuration in reviews

Configuration involves the arrangement (configuration) of the findings of primary studies to answer the review question….

… to offer a meaningful picture of what research is telling us

... across a potentially wide area of research

•

(10)

Most reviews contain elements of both, but some patterns are common

Aggregation Configuration

Approach to theory Test Generate

Research question Closed Open

Concepts Secure prior to review Emergent in review

Timing of methods decisions

A priori Iterative / emergent

Search for studies Representative / unbiased / exhaustive

Sufficient / purposive

Quality appraisal Avoiding bias Richness of data

Explore

(11)

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London, UK1

Methods for the synthesis of qualitative

research

James Thomas1

ESRC Research Methods FestivalOxford, 5 July 2012

(12)

Synthesis of qualitative research: outline

• What can qualitative research tell us?

• Current developments/approaches in qualitative review methods

 • Examples of syntheses of qualitative

research

(13)

What can qualitative research tell us?

(14)

Qualitative research…

• aims to provide an in-depth understanding of people’s experiences, perspectives and histories in the context of their personal circumstances or settings…*

• with the use of unstructured methods which are sensitive to the social context of the study*

• the capture of data which are detailed, rich and complex…*

• is generalisable on the basis of theory not statistical probability **

* Spencer et al (2003); ** Popay (2006)

(15)

Methods used include*…

• exploratory interviews• focus groups• observation• conversation• discourse and narrative analysis• documentary and video analysis

*Spencer et al (2003)

(16)

Is it simply the analysis of textual data?

“Increasingly, the terms ‘quantitative research’ and ‘qualitative research’ came to signify much more than ways of gathering data; they came to denote divergent assumptions about the nature and purposes of research in the social sciences”

Bryman (1988) Quantity and Quality in Social Research. London: Unwin Hyman. p.3.

(17)

Qualitative research can*

• help to define policy questions• look in detail at how a programme or trial

was actually implemented• help to determine appropriate outcome

measures by looking at ‘subjective’ outcomes

• help to clarify what counts as effective or successful* Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36

(18)

…continued

• identify and explore unintended consequences• contribute to service delivery and policy

development by describing processes and contexts

• inform and illuminate quantitative studies, e.g. by contributing to the design of structured instruments, assessing the fairness of comparisons in experimental studies, or unpacking variation within aggregated data

Davies (2000) ‘Contributions from qualitative research’ in Davies et al (eds) What works? Evidence-based policy and practice in public services. Bristol: Policy Press. Cited in *Spencer et al (2003). p.36

(19)

Current developments in synthesising qualitative research

• Growing recognition of the value of qualitative research to inform policy & practice

• Methods developing to facilitate this

(20) Try: http://www.wordle.net/

(21)

Methodological study

• We searched for methodological papers concerning the synthesis of concepts or theories– Purposive, rather than systematic search– Reference ‘chasing’– Google scholar search– Handsearching key journals

• 203 papers retrieved• 9 distinct methods for synthesis

(22)

Methods for the synthesis of qualitative research identified

• Meta narrative synthesis• critical interpretive synthesis• meta-study• meta-ethnography• grounded theory• thematic synthesis• framework synthesis• ecological triangulation

(23)

Many methods: similar or different?

• Examined the methods across different dimensions:– Epistemology– Approach to quality assessment– Attitudes towards problematizing the literature– Use of review question– How similar / different the included studies were– Characteristics of the synthetic product

• Found they fell into two broad camps: ‘idealist’ and ‘realist’

(24)

‘Realist’ approaches

Purpose• To answer a policy relevant questionMethods• Qualitative/ quantitative data analysed with

qualitative/ quantitative methods• Searching linear or iterative• Quality assessment of study methodsProduct • Directly applicable to policy and practice

decisions

(25)

‘Idealist’ approaches

Purpose• To explore and construct concepts from the data• For generating theoryMethods• Qualitative data analysed with qualitative methods• Searching iterative• Qualitative assessment of study content > methodProduct • Complex, requiring further interpretation before being

used for policy or practice

(26)

Conclusions

• Operationally, many methods are very similar

• Underlying principles differ• Product differs in terms of the amount of

additional interpretation required. This may reflect…

• (Barnett-Page and Thomas (2009))

(27)

Perspectives on how evidence informs the policy process

Idealist Realist

Knowledge-driven model; Problem-solving (policy driven) model*

(either researcher disseminates research or policy-maker commissions / requests specific research)

Less suited More suited

Interactive model; Enlightenment model*

(ongoing interactions between researcher and policymaker; ‘gradual sedimentation of insights…’)

More suited Less suited

*Weiss (1979)

(28)

Two examples

• All this is a bit abstract… what do systematic reviews of qualitative research look like?

• Two examples:– The ‘fine grain’ of a synthesis– ‘Mixed methods’ synthesis

(29)

Example 1: Informing policy about the ‘obesity epidemic’

What are children’s views about:

• the meanings of obesity or body size, shape or weight (including their perceptions of their own body size)?

• influences on body size?• changes that may help

them to achieve or maintain a healthy weight?

Views = attitudes, opinions, beliefs, understanding or experiences

As distinct from health/weight status, behaviour, factual knowledge

Rees et al 2009

(30)

‘Thematic synthesis’

• Similar to other methods of synthesising qualitative research (e.g. ‘meta-ethnography’)

• Source data = text (documents)• Source material = conceptual• Key method = translation• Final product = interpretation

• (Thomas and Harden 2008)

(31)

Stages of thematic synthesis

• Stages one and two: coding text and developing descriptive themes– Identifying the ‘findings’– Line-by-line coding– Developing descriptive themes

• Stage three: generating analytical themes– In the light of the review question

(32)

Screenshot – line-by-line coding

(33)

Screenshot – descriptive codes diagram

(34)

Example 2. adding insights to other existing reviews

• How might a systematic review of qualitative research support or add insight to a meta-analysis?

• Product of qualitative synthesis can form the conceptual framework within which heterogeneity can be explored in a meta-analysis

Searching, screening mapping and user involvement

e.g. Statistical meta-analysis of trials a) Data extraction

b) Quality assessmentc) Effect sizes pooledd) Narrative synthesis

Addresses sub RQ e.g. ‘which interventions are effective?’

e.g. Thematic synthesis of qualitative studies a) Data extraction

b) Quality assessmentc) Descriptive themesd) Analytical themes

Addresses sub RQ e.g. ‘what are people’s perspectives and experiences ?

Synthesis 1 Synthesis 2

Synthesis 3Driven by overall review question

Integration of separate syntheses e.g. a) Matches, mis-matches and gaps

b) Hypotheses generated in synthesis 2 tested amongst trials in synthesis 1

Review Questione.g. ‘What is known about the barriers to, and facilitators of, outcome X (e.g. physical activity)

amongst population A (e.g. young people

(36)

Cross study synthesis: Findings

Children’s views Outcome evaluations

Recommendation for interventions

Good quality Other

Do not promote fruit and vegetables in the same way

No soundly evaluated

interventions

No other interventions

identified

Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one

5 soundly evaluated

interventions identified

5 other interventions

Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health

5 soundly evaluated

interventions identified

6 other interventions identified

(37)

Cross study synthesis: an example of sub-group analysis

Increase (standardised portions per day) in vegetable intake across trials

0

0.1

0.2

0.3

0.4

0.5

0.6

War

dle

Liqu

ori

Henry

Ander

son

Reyno

lds

Auld

Auld

(b)

Baran

owsk

i

Perry

Study

Po

rtio

ns

Little or no emphasis on health messages

(38)

This method of synthesis across study types:

• preserves the integrity of the findings of the different types of studies

• allows us to integrate ‘quantitative’ estimates of benefit and harm with ‘qualitative’ understanding from people’s lives

• allows the exploration of heterogeneity in ways in which it would be difficult to imagine in advance– protects against ‘data dredging’

(39)

Assumptions / theoretical models

• Mixed methods synthesis can draw on thinking relating to mixed methods in primary research

• In terms of a frequently used taxonomy of paradigm stances (Creswell 2011) e.g.– Incommensurability (cannot be mixed)– A-paradigmatic (can be mixed and matched in different ways)– Complementary strengths (not incompatible, but are different

and should be kept separate)– Dialectic (paradigms are important in different ways leading to

useful tensions & insights)– Alternative paradigm (‘mixed methods’ paradigm; foundation

in e.g. pragmatism)

(40)

Conclusions

• Qualitative research offers valuable insights that can inform policymaking

• Methods for synthesising qualitative research are still being developed

• Selection of a particular method depends on the type of answer required (and the means by which the review will inform policy)

(41)

References

• Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical review. BMC Medical Research Methodology, 9:59. doi:10.1186/1471-2288-9-59. (http://www.biomedcentral.com/1471-2288/9/59)

• Creswell J (2011) Mapping the Developing Landscape of Mixed Methods Research in Teddlie C, Tashakkori A: SAGE Handbook of Mixed Methods in Social & Behavioral Research. New York: Sage.

• Gough D, Oliver S, Thomas J (2012) An Introduction to Systematic Reviews. London: Sage

• Thomas J, Harden A (2008) Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Medical Research Methodology, 8:45 doi:10.1186/1471-2288-8-45 (http://www.biomedcentral.com/1471-2288/8/45)

• Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J (2004) Integrating qualitative research with trials in systematic reviews: an example from public health. British Medical Journal 328: 1010-1012. (http://www.bmj.com/cgi/content/full/328/7446/1010)

(42)

Systematic Reviews: The Potential of

Different Methods

Steve HigginsHenry Potts

James ThomasGeoff Wong

ESRC Research Methods FestivalOxford, 5 July 2012

(43)

Summary

Heterogeneity Aggregation / configuration

Meta-analysis A problem in fixed effectAllowance made for in random effectsFor ‘exploration’

Fixed & random effects models are pure aggregationSub-group & meta-regression both aggregate & configure

Synthesis of qualitative research

Expected. Key challenge is judging when contexts / concepts are too dissimilar to be compared.

Similar concepts are ‘aggregated’; most synthesis involves configuration

Realist review Necessary to obtain different contexts etc

Aggregates within CMO configurations; configures between them

Meta narrative review Necessary to obtain diversity in paradigms

Aggregation within paradigms; configuration between them

(44)

Your turn

Research question

Heterogen-eity?

Meta-analysis

Qualitative synthesis

Realist review

Meta narrative review

Research question 1

Research question 2