Extensions to traditional statistical meta-analysis James Thomas

(1)

Extensions to traditional statistical meta-analysis

James Thomas

Systematic Reviews for Complicated and Complex Questions, ESRC Methods Festival, St

Catherine’s College, Oxford, 10th July 2014

EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London

(2)

Outline

• What is meta-analysis, and why are ‘extensions’ needed?

• What is complexity? And why is it a challenge in systematic reviews?

• Extensions to traditional meta-analysis when complexity is encountered

• Acknowledgement: presentation draws on: O’Mara-Eves A, Thomas J (2013) Methods for configurational synthesis: extensions to traditional meta-analysis for addressing intervention complexity and contextual variation in reviews. 21st Cochrane Colloquium: Quebec 19-23 September

(3)

Form review team (involve ‘users’)

Formulate review question, conceptual framework and inclusion criteria (develop ‘protocol’)

Search for and identify relevant studiesDescribe studies

Assess study quality (and relevance)

Synthesise findings

Communicate and engage

Map

Synthesis

The common stages of a systematic review; focus here on synthesis

(4)

• Typically are used to address three key research questions:– What is the overall estimate of the size of the

effect and its precision?– Is there heterogeneity across the study effects?– What (if any) variables explain differences

across the study effects (if heterogeneity is present)?

Traditional meta-analytic models

(5)

Aggregative approaches in research

Aggregative reviews predominately add up (aggregate) findings of primary studies to answer a review question…

… to indicate the direction or size of effect

(6)

Newman M, Bird K, Tripney J, Kalra N, Kwan I, Bangpan M, Vigurs C (2010) Understanding the impact of engagement in culture and sport: A systematic review of the learning impacts for young people. London: Department for Culture, Media and Sport. http://culture.gov.uk/images/research/CASE-systematic-review-July10.pdf

RCT forest plot: Does children’s participation in structured arts activities improve their cognitive learning outcomes?

(7)

• Current popular meta-analytic methods are limited to:– Questions of differences between

two groups or correlations or variables

– One causal / relational proposition at a time

– One outcome at a time

Are we limiting ourselves?

(8)

• Policymakers and practitioners usually do not ask a single narrow aggregative question

• They begin with a particular problem and ask “what is the best way to achieve outcome X?”

• They also ask “does it vary according to…?” and “What does X mean to Y?”

• And they ask these questions relating to complex problems -

Whose questions are we addressing?

(9)

A complex intervention

• Defined in MRC guidance as: “interventions with several interacting components… Many of the extra problems relate to the difficulty of standardising the design and delivery of the interventions, their sensitivity to features of the local context, the organisational and logistical difficulty of applying experimental methods to service or policy change, and the length and complexity of the causal chains linking intervention with outcome.”

• Craig P et al (2008): Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 337

Some would say the above is merely complicated…

(10)

Complicated and complex

• Truly complex interventions are best conceptualised as dynamic processes– Virtuous circles– Feedback loops– Non-linear step changes in responses /

outcomes– Multiple ‘routes’ to effectiveness

• Rogers PJ. Using Programme Theory to Evaluate Complicated and Complex Aspects of Interventions. Evaluation. 2008;14(1):29-48

(11)

Complex reviews of social research:

• Start from a given ‘problem’– often a known outcome and population

• Aim to identify a range of possible ‘solutions’• Rarely aim to come to a single answer• Acknowledge that there are rarely replications

of interventions in social research– Use heterogeneity to better understand factors that

influence the impact of interventions– Contain detailed and complex conceptual frameworks

(programme theories etc)• So to configure findings (as well as aggregate)

(12)

Configurative approaches in research

• Configurative reviews predominately arrange (configure) the findings of primary studies to answer the review question….

• … to offer a meaningful picture of what research is telling us

(13)

• Sub-group analysis• Meta-regression• Network meta-analysis• Multivariate meta-

analysis• Path analysis• Factor analysis• Qualitative

comparative analysis• …?

Other tools in our toolbox

(14)

Subgroup analysis

DiCenso et al (2002) BMJ;324:1426

(15)

Meta-regression Catalá-López et al. BMC Psychiatry 2012, 12:168

(16)

Network meta-analysis

• Facilitates an estimate of the relative effectiveness of interventions – even when they have not been directly compared with one another in a trial

Thorlund and Mills Systematic Reviews 2012, 1:41

(17)

Problems with ‘traditional’ approaches

• The above approaches work well in some situations, BUT

• There are rarely replications of complex interventions (is it possible to have a genuine replication?); leads to a lack of data

• Even when an analysis has many studies, interventions, contexts etc. all differ– Lots of unexplained heterogeneity

• Symmetrical nature of correlational analysis

(18)

Complexity and correlational analysis

• Correlation is symmetric– When testing for a connection between cause

and effect, also tests equally for absence of cause and absence of effect

• Correlation therefore cannot detect multiple causal pathways*– E.g. Asserting that ‘interventions which are

delivered by peers tend to be effective’ should not require that those not delivered by peers are not effective

* There are usually too few studies to use interaction variables

(19)

Another approach: qualitative comparative analysis (QCA)

• Originally developed by Charles Ragin in political science and historical sociology

• Not a correlational approach• Useful for small numbers of

studies• Focus can be on

combinations of intervention components

• More inductive mode of analysis than above

Thomas J, O’Mara-Eves A, Brunton G (2014) Using Qualitative Comparative Analysis (QCA) in systematic reviews of complex interventions: a worked example. Systematic Reviews. 3: 67

Suggests that intensity & qualityare sufficient to gain effective outcome

(20)

Summary

• Meta-analysis can address a single focused question, requiring a straightforwardly aggregative answer

• “Real world” questions tend to require configuration AND aggregation

• Extensions to meta-analysis are able to configure and aggregate study findings successfully

• But few current methods can cope with genuinely complex situations

(21)

WebsitesEPPI-Centre Website http://eppi.ioe.ac.uk

Twitter @James_M_Thomas @EPPICentre

Emailj.Thomas@ioe.ac.uk

Thank you for your attention

EPPI-CentreSocial Science Research UnitInstitute of EducationUniversity of London18 Woburn SquareLondon WC1H 0NR

Tel +44 (0)20 7612 6397Fax +44 (0)20 7612 6400Email eppi@ioe.ac.ukWeb eppi.ioe.ac.uk/

The EPPI-Centre is part of the Social Science Research Unit at the Institute of Education, University of London