Using meta-analyses in your literature review BERA Doctoral Workshop 3rd September 2008 Professor Steven Higgins Durham University [email protected].

Using meta-analyses in your literature review

BERA Doctoral Workshop

3rd September 2008Professor Steven Higgins

Durham [email protected]

Acknowledgements

• This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011.

• The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University.

• The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).

• The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

• You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.

Aims

• To support understanding of meta-analysis of intervention research findings in education;

• To extend understanding of reviewing quantitative research literature;

• To describe the techniques and principles of meta-analysis involved to support understanding of its benefits and limitations;

• To provide references and examples to support further work.

ESRC Researcher Development Initiative

• Quantitative synthesis of intervention research findings in education– Collaboration between

• Durham University• York University• Institute of Education, London

Why review?

• Ask the person next to you what the purpose of the literature review is in their thesis

• See how many different purposes you can think of

• Join another pair and identify which are the 3 you think are the most important

Why review?

• Summarise existing knowledge• What we know, and how we know it• For what purpose?

– Expectation– Scenery– State of the art (summary)– Positioning (conceptual)– Progressing knowledge (logic)

The PhD literature review

• Narrative summary of the area

• Grand tour of the concepts and terminology

• Synthesis of empirical findings

• Background to the study

A systematic review

• is usually more comprehensive;

• is normally less biased, being the work of more than one reviewer;

• is transparent and replicable(Andrews, 2005)

Examples of systematic reviews

• EPPI Centre– UK based - wide range of educational topics

• The Campbell Collaboration – 5 education reviews

• Best Evidence Encyclopedia– John’s Hopkins’ - aimed at practice

Systematic reviewing

• Key question

• Search protocol

• Inclusion/exclusion criteria

• Coding and Mapping

• In-depth review (sub-question)

• Techniques for systematic synthesis

Systematic reviews

• Research and policy

• Specific reviews to answer particular questions– What works? - impact and effectiveness

research with a tendency to focus on quantitative and experimental designs

Literature reviewing - conceptual relations

Systematic review

Meta-analysis

Narrative review

Meta-analysis

• Synthesis of quantitative data– Cumulative– Comparative– Correlational

• “Surveys” educational research (Lipsey and Wilson, 2001)

Origins1952: Hans J. Eysenck concluded that there were no

favorable effects of psychotherapy, starting a raging debate which 25 years of evaluation research and hundreds of studies failed to resolve

1978: To proved Eysenck wrong, Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studiesGlass (and colleague Smith) concluded that psychotherapy

did indeed work - “the typical therapy trial raised the treatment group to a level about two-thirds of a standard deviation on average above untreated controls; the average person received therapy finished the experiment in a position that exceeded the 75th percentile in the control group on whatever outcome measure happened to be taken” (Glass, 2000). Glass called the method “meta-analysis”

( adapted from Lipsey & Wilson, 2001)

Historical background• Underpinning ideas can be identified earlier:

– K. Pearson (1904)Averaged correlations for typhoid mortality after inoculation across 5 samples

– R. A. Fisher (1944)“When a number of quite independent tests of significance have been

made … although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99).

Source of the idea of cumulating probability values

– W. G. Cochran (1953)Discusses a method of averaging means across independent studiesSet out much of the statistical foundation for meta-analysis (e.g., Inverse

variance weighting and homogeneity testing)( adapted from Lipsey & Wilson, 2001)

Significance versus effect size

• Traditional test is of statistical ‘significance’

• The difference is unlikely to have occurred by chance– However it may not be:

• Large• Important, or even• Educationally ‘significant’

The rationale for using effect sizes

• Traditional reviews focus on statistical significance testing– Highly dependent on sample size– Null finding does not carry the same “weight” as a

significant finding

• Meta-analysis focuses on the direction and magnitude of the effects across studies– From “Is there a difference?” to “How big is the

difference?”– Direction and magnitude represented by “effect

size”

• Comparison of impact

• Same AND different measures

• Significance vs effect size– Does it work? vs How well does it work?

Effect size

• Standardised way of looking at gain scores

• Different methods for calculation

• Experimental group mean - Control mean/ Standard deviation

‘Effect size’

What is “effect size”?

• Standardised way of looking at difference– Different methods for calculation

• Odds Ratio• Correlational (Pearson’s r)• Standardised mean difference

– Difference between control and intervention group as proportion of the dispersion of scores

Calculating effect size

• Control group gain minus experimental group gain divided by the standard deviation of the groups

Effect size and impact

From: Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).

•Relative effects - average is about 0.37 - 0.4 (Sipe and Curlette, 1997; Hattie, Biggs and Purdie, 1996)

•Doing something different makes a difference

•Visualising the difference

Interpreting effect sizes

0.1 = percentile gain of 6 pointsie a class ranked 50th in a league table of 100 schools would move from 50th to about 44th place

0.5 = percentile gain of 20 pointsie move from 50th to 30th place

1.0 = percentile gain of 34 pointsie move from 50th to 16th place

How much is the impact?

0.2 “small” = difference in height between 15-16 year olds

0.5 “medium” = difference in height between 14 and 18 year olds

0.8 “large” = difference in height between 13 and 18 year olds

Other interpretations

Cohen 1969

Meta-analysis

• Key question• Search protocol• Inclusion/exclusion criteria• Coding• Statistical exploration of findings

– Mean– Distribution– Sources of variance

Some findings from meta-analysis

Pearson et al. 2005• 20 research articles, 89 effects ‘related to digital tools and

learning environments to enhance literacy acquisition’. Weighted effect size of 0.489 indicating technology can have a positive impact on reading comprehension

Bernard et al. 2004• Distance education and classroom instruction - 232 studies, 688

effects - wide range of effects (‘heterogeneity’); asynchronous DE more effective than synchronous

More findings

Hattie and Timperley, 2007• ‘The Power of Feedback’, synthesis of other meta-analyses on

feedback to provide a conceptual review 196 studies, 6972 effects - average effect of feedback on learning 0.79

Formative assessment

CASE (Cognitive Acceleration Through Science Education)

Individualised instruction

ICT

Homework

Direct instruction

Rank (or guess) some effect sizes…

1. 04 CASE (Cognitive Acceleration Through Science Education) (Boys science GCSE - Adey & Shayer, 1991)

0.6 Direct instruction (Sipe & Curlette, 1997) 0.43 Homework (Hattie, 1999)

0.32 Formative assessment (KMOFAP)

0.31 ICT (Hattie, 1999)

0.1 Individualised instruction (Hattie, 1999)

Rank order of effect sizes

‘Super-syntheses’

• Syntheses of meta-analyses

• Relative effects of different interventions

• Assumes variation evens out across studies with a large enough dataset (Marzano/Hattie) or attempts to control for the variation statistically (Sipe & Curlette)

Synthesis of study skills interventionsMeta-analysis of 51 studies of study skills interventions. Categorised the inverventions using the SOLO model (Biggs & Collis, 1982), classified studies into four hierarchical levels of structural complexity and as either ‘near’ or ‘far’ transfer. The results support situated cognition, and that training for other than simple mnemonic tasks should be in context, use tasks within the same domain as the target content, and promote a high degree of learner activity and metacognitive awareness.(average effect 0.4)

Hattie Biggs and Purdie, 1996

Sipe and Curlette, 1997

• “A metasynthesis of factors relating to educational achievement” - testing Walberg’s ‘educational productivity’ model - synthesis of 103 meta-analyses

‘Theory driven’

Self system - metacognition - cognition/ knowledgeSelf - 0.74

Metacogntive 0.72

Cognitive 0.55

Marzano, 1998

Discussion

• Work with a colleague to put the statements in order of how comparable you think the research findings are

• Join another pair (or pairs) and decide how comfortable would you be with comparing the findings

Issues and challenges in meta-analysis

• Conceptual– Reductionist - the answer is 42– Comparability - apples and oranges– Atheoretical - ‘flat-earth’

• Technical – Heterogeneity– Publication bias– Methodological quality

Reductionist or ‘flat earth’ critique

The “flat earth” criticism is based on Lee Cronbach’s assertion that a meta-analysis looks at the “big picture” and provides only a crude average. According to Cronbach,

“… some of our colleagues are beginning to sound like a Flat Earth Society. They tell us that the world is essentially simple: most social phenomena are adequately described by linear relations; one-parameter scaling can discover coherent variables independent of culture and population; and inconsistencies among studies of the same kind will vanish if we but amalgamate a sufficient number of studies…The Flat Earth folk seek to bury any complex hypothesis with an empirical bulldozer…” (Cronbach, 1982, in Glass, 2000).

Comparability• Apples and oranges

– Same test– Different measures of the same construct– Different measures of different constructs– What question are you trying to answer?– How strong is the evidence for this?

“Of course it mixes apples and oranges; in the study of fruit, nothing else is sensible; comparing apples and oranges is the only endeavor worthy of true scientists; comparing apples to apples is trivial” (Glass, 2000).

Empirical not theoretical?

• What is your starting point?

• Conceptual/ theoretical critique– Marzano– Hattie– Sipe and Curlette

Technical issues

• Interventions

• Publication bias

• Methodological quality

• Sample size

• Homogeneity/ heterogeneity

Interventions

• “Super-realisation bias” (Cronbach & al. 1980)

– Small-scale interventions tend to get larger effects

– Enthusiasm, attention to detail, quality of personal relationships

Publication bias

• Statistically significant (positive) findings

• Smaller studies need larger effect size to reach significance

• Larger effects– ‘Funnel plot’ sometimes used to explore

this Scatterplot of the effects from individual studies (horizontal axis) against a

study size (vertical axis)

Methodological quality

• Traditional reviews privilege methodological rigour– Low quality studies higher effect sizes (Hattie

Biggs & Purdie, 1996)

– No difference (Marzano, 1998)

– High quality studies, higher effect sizes (Lipsey & Wilson, 1993)

• Depends on your definition of quality

Sample size

“Median effect sizes for studies with sample sizes less than 250 were two to three times as large as those of larger studies.” (Slavin & Smith, 2008)

Heterogeneity

• Variation in effect sizes

• Investigate to find clusters (moderator variables)

• Assumption that the effect will be consistent

Questions and reactions

• With a colleague see if you can identify a question arising from the presentation so far

• What is your reaction to the technique• How useful is it

– Generally– To your own work?

Strengths of Meta-Analysis

Uses explicit rules to synthesise research findings

Can find relationships across studies which may not emerge in qualitative reviews

Does not (usually) exclude studies for methodological quality to the same degree as traditional methods

Statistical data used to determine whether relationships between constructs need clarifying

Can cope with large numbers of studies which would overwhelm traditional methods of review

Summary

• “Replicable and defensible” method for synthesizing findings across studies (Lipsey & Wilson, 2001)

• Identifies gaps in the literature, providing a sound basis for further research

• Indicates the need for replication in education• Facilitates identification of patterns in the

accumulating results of individual evaluations• Provides a frame for theoretical critique

Other approaches to synthesis

• Narrative• Quantitative (meta-analysis)• Best-evidence synthesis (Slavin)• Realist (Pawson)• Meta-ethnography (Noblitt & Hare) • Thematic synthesis (Thomas & Harden)• Grounded theory

Suggestions

• Be explicit about your rationale

• Be systematic (or at least methodical)

• Be transparent

• Describe

• Analyse (content and methodology)

• Synthesise

A (narrative) metaphor…

• Literature review as rhetoric

• An act of persuasion

• Introduce your study…

Some useful websites

EPPI, Institute of Education, Londonhttp://eppi.ioe.ac.uk/

The Campbell Collaborationhttp://www.campbellcollaboration.org/

Best Evidence Encyclopedia, Johns Hopkinshttp://www.bestevidence.org/

Best Evidence Synthesis (BES), NZhttp://www.educationcounts.govt.nz/themes/BES

Institute for Effective Education (York)http://www.york.ac.uk/iee/research/#reviews

Further training

• ESRC RDI in quantitative synthesis– One day training sessions

• Introduction (for interpretation)• Methods Training (for application)• Issues Seminars (methodological issues)

– Durham, London, Edinburgh, Bristol, Belfast, York

[email protected]

ReferencesBernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M.,& Huang, B. (2004) How Does

Distance Education Compare with Classroom Instruction? A Meta-Analysis of the Empirical Literature Review of Educational Research, 74. 3, (Autumn, 2004), pp. 379-439.

Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). Journal of Educational Research, 98, pp 35-44.

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements . San Francisco, Ca.: Jossey-Bass.

Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)Hattie, J. A. (1992). Measuring the effects of schooling. Journal of Education, 36, pp 5-13Hattie, J., Biggs, J. and Purdie, N. (1996) Effects of Learning Skills Interventions on Student Learning: A Meta-analysis Review of

Educational Research 66.2 pp 99-136.Hattie, J.A. (1987) Identifying the salient facets of a model of student learning: a synthesis of meta-analyses International Journal of

Educational Research, 11 pp 187- 212.Hattie, J. & Timperley, H. (2007) The Power of Feedback Review of Educational Research 77. 1, pp. 81–112.Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks,

CA: SAGE Publications.Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational

Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).Pearson, D.P., Ferdig, R.E., Blomeyer, R.L. & Moran, J. (2005) The Effects of Technology on Reading Performance in the Middle-School

Grades: A Meta-Analysis With Recommendations for Policy Naperville, Il: University of Illinois/North Central Regional Educational Laboratory .

Sipe, T. & Curlette, W.L. (1997) A Meta-Synthesis Of Factors Related To Educational Achievement: A Methodological Approach To Summarizing And Synthesizing Meta-Analyses International Journal of Educational Research 25. 7. pp. 583-698.

Slavin, R.E. and Smith, D. (2008) Effects of Sample Size on Effect Size in Systematic Reviews in Education Paper presented at the annual meetings of the Society for Research on Effective Education, Crystal City, Virginia, March 3-4, 2008.

Acknowledgements

• This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011.

• The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University.

• The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).

• The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

• You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.