5 Meta-analysis.ppt

Gavin StewartCentre for Evidence-Based ConservationUniversity of Birmingham, UK

Meta-analysis

Talk overview

A word of warning Meta-analysis Data extraction for meta-analysis

Warning

Do you want or need a meta-analysis?

heterogeneity Sample sizes standardisation

Meta-analysis

Steps, usefulness and limitations of meta-analysis

Heterogeneity and methods for its investigation

Weighting and bias

Systematicreview Meta-

analysis

quantitativepooling of results of individual studies

Systematic review in relation to meta-analysis

What does Meta-analysis do?

Meta-analysis aims to quantitatively combine results of different studies

Pooled estimate should be a weighted average of all studies included in a meta-analysis

increase statistical power improves generalisability

Weight studies based on amount of information

The larger studies are given more weight than smaller studies

Sample size Given the same sample size, more weight

will be given to studies with smaller variance (Inverse of variance)

Scale and Pseudoreplication

Weight based on the level of validity

Quality scales Quality components e.g., randomisation method,

baseline, sampling methods SCALE and PSEUDOREPLICATION (again) Different weightings for different elements, so very

controversial and often for the purpose of sensitivity analysis

limited by sample size so maybe necessary to sum

Heterogeneity in meta-analysis

Variation in results across studies Distinguish between statistically significant

heterogeneity and ecologically important heterogeneity

Causes/sources of heterogeneity

Chance Variations in populations Variations in interventions Different methodological quality Different outcome measures

Why investigate heterogeneity?

to decide whether the results of individual studies could be combined (FE models)

to identify effect modifiers (Study-level variables that are associated with the results of studies) e.g. time, method, ex situ/in situ

Methods for investigating heterogeneity

Graphical methods Statistical testing Excluding outliers Subgroup analysis Meta-regression

Statistical testing for heterogeneity

Are the differences in result across studies greater than could be expected by chance?

Q statistic

Excluding outliers

Outliers are excluded one by one until the statistical test of heterogeneity is no longer significant

Should be used very cautiously or not at all

Subgroup analysis in meta-analysis

To separate studies according to certain study-level variables

Then, to conduct quantitative pooling separately for each subgroup of studies

Meta-regression

The estimate of study results is the dependent variable and one or more study-level variables are the independent variables (predictors)

Biases and errors in meta-analysis

Meta-analysis is basically retrospective Results of meta-analysis may be misleading Biases may be introduced if the

identification, inclusion and assessment of primary studies are not systematic

because of publication related biases

Publication bias

studies with significant, positive, results are easier to find than those with non-significant or 'negative' results. The subsequent over-representation of positive studies in systematic reviews may mean that our reviews are biased toward a positive result.

Also have time lag bias, language bias and citation bias

Funnel plot (se against es) asymmetry

Bad or inappropriate meta-analysis

Not systematic in study identification and assessment Inappropriate pooling of heterogeneous results, (FE models) No investigation of heterogeneity Lack of details of included studies Inappropriate weighting individual studies Failed to consider publication and related biases Lack of sensitivity analysis Inappropriate interpretation of the results of subgroup analysis Inappropriate interpretation of the pooled average

-

-

-

-

Data extraction

Extract data with synthesis in mind e.g. Mean, n, sd for treatment and control

Extract data on effect modifiers Use standardised piloted method and check repeatability Consider scale and pseudoreplication Contact authors for missing data where possible Do not get side tracked into extracting more than you need

References

Cooper, H. and Hedges, L.V. (1994) (eds.) The Handbook of Research Synthesis. Russell Sage Foundation, New York.

Deeks, J.J., Altman, D.G. and Bradburn, M.J. (2001) Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. Systematic Reviews in Health Care. Meta-analysis in Context. (eds M. Egger, G.D. Smith and D.G. Altman), pp 285-312. British Medical Journal Publishing Group, London.

DerSimonian, R. and Laird, N. (1986) Meta-analysis in clinical trials. Controlled. Clinical. Trials, 7, 177-188. Egger, M., Davey-Smith, G., Schneider, M. and Minder, C. (1997) Bias in meta-analysis detected by a simple

graphical test British Medical Journal, 315, 629-34. Gates, S. (2002). Review of methodology of quantitative reviews using meta-analysis in ecology. Journal of

Animal Ecology, 71, 547–557. Gurevitch, J. and Hedges, L.V. (1999) Statistical issues in ecological meta-analyses. Ecology, 80, 1142–

1149. Gurevitch, J. and Hedges, L.V. (2001) Meta-analysis. Combining results of independent experiments. Design

and Analysis of Ecological Experiments (eds S.M. Scheiner and J. Gurevitch), pp. 347–369. Oxford University Press, Oxford.

Hedges, L.V., and Olkin, I. (1985). Statistical Methods for Meta-analysis. San Diego: Academic Press, San Diego.

Hurlbert, S.H, (1984) Pseudoreplication and the design of ecological field experiments. Ecological. Monographs, 54, 187-211.

Osenberg, C.W., Sarnelle, O., Cooper, S.D. and Holt, R.D. (1999) Resolving ecological questions through meta-analysis: goals, metrics and models. Ecology, 80, 1105–1117.

Thompson, S.G. and Sharp, S.J. (1999) Explaining heterogeneity in meta-analysis: a comparison of methods. Statistics in Medicine, 18, 2693-708.