gbiondizoccai@gmail .com www.metcardio.org Failures of meta- analyses: the evidence against the evidences Giuseppe Biondi-Zoccai, MD Division of Cardiology, Department of Internal Medicine, University of Turin, Turin, Italy Meta-analysis and Evidence-based medicine Training in Cardiology (METCARDIO), Turin, Italy
52
Embed
[email protected] Failures of meta-analyses: the evidence against the evidences Giuseppe Biondi-Zoccai, MD Division of Cardiology,
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Why meta-analyses are important: exponential increase in pertinent PubMed citations on…
PubMed search strategy: ("2001"[PDAT] : "2005"[PDAT]) AND (("systematic"[title/abstract] AND "review"[title/abstract]) OR ("systematic"[title/abstract] AND "overview"[title/abstract]) OR ("meta-analysis"[title/abstract] OR "meta-analyses"[title/abstract]))
14 4/2009 - PubMed query: (thrombosis OR hemostasis OR haemostasis OR coagulation OR clot* OR tranexamic OR aprotinin* OR rfviia OR desmopressin*) AND (meta-analy* OR metanalys* OR metaanalys* OR (systematic AND (review OR overview)))
Baby steps of meta-analysis• 1904 - Karl Pearson (UK): correlation between inoculation of
vaccine for typhoid fever and mortality across apparently conflicting studies
• 1931 – Leonard Tippet (UK): comparison of differences between and within farming techniques on agricultural yield adjusting for sample size across several studies
• 1937 – William Cochran (UK): combination of effect sizes across different studies of medical treatments
• 1970s – Robert Rosenthal and Gene Glass (USA), Archie Cochrane (UK): combination of effect sizes across different studies of, respectively, educational and psychological treatments
• 1980s – exponential development/use of meta-analytic methods
The Cochrane Collaboration is an world-wide organization that aims to help people make well informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions
The Cochrane Collaboration• Over 6000 contributors & over 5000
reviews• 50 Collaborative Review Groups (CRGs)• 12 centers throughout the world• 9 fields• 11 Methods Groups• 1 Consumer Network• The Campbell Collaboration (focusing on
The Cochrane Collaboration• Over 6000 contributors & over 5000
reviews• 50 Collaborative Review Groups (CRGs)• 12 centers throughout the world• 9 fields• 11 Methods Groups• 1 Consumer Network• The Campbell Collaboration (focusing on
education/social sciences)
Thus currently the best reviews available (but still
Review: Late percutaneous coronary intervention for infarct-related artery occlusionComparison: 01 Late percutaneous coronary intervention vs best medical therapy for infarct-related artery occlusion Outcome: 01 Death
• Many scales for the quality of included studies have been reported, but none is reliable or robust
• The recommended approach is to individually appraise the potential risk of the 4 biases (eg A-low, B-moderate, C-high, D-unclear from reported data):
– Selection bias (one group is different than the other)
– Performance bias (treatment is systematically different)
– Adjudication bias (outcome adjudication is selectively
different)
– Attrition bias (follow-up duration or completeness is
Appraisal tools: Oxman and Guyatt’sEvaluates the internal validity of a review on 9 separate questions for which 3 distinct answers are eligible (“yes”, “partially/can’t tell”, “no”):
Oxman et al, J Clin Epidemiol 1991
1. Where the search methods used to find evidence stated?
2. Was the search for evidence reasonably comprehensive?
3. Were the criteria for deciding which studies to include in the overview reported?
4. Was bias in the selection of studies avoided?
5. Were the criteria used for assessing the validity of the included studies reported?
6. Was the validity of all studies referred to in the text assessed using appropriate criteria?
Appraisal tools: Oxman and Guyatt’s7. Were the methods used to combine the findings of the
relevant studies reported?
8. Were the findings of the relevant studies combined appropriately relative to the primary question the overview addresses?
9. Were the conclusions made by the author(s) supported by the data and/or analysis reported in the overview?
10. This summarizes the previous ones and, specifically, asks to rate the scientific quality of the review from 1 (being extensively flawed) to 3 (carrying major flaws) to 5 (carrying minor flaws) to 7 (minimally flawed).
If the “partially/can’t tell” answer is used one or more times in questions 2, 4, 6, or 8, a review is likely to have minor flaws at best and is difficult to rule out major flaws (ie a score≤4). If the “no” option is used on question 2, 4, 6 or 8, the review is likely to have major flaws (ie a score≤3). Oxman et al, J Clin Epidemiol 1991
A few references• Biondi-Zoccai GGL et al. Parallel hierarchy of scientific studies in cardiovascular medicine. Ital Heart J 2003; 4: 819-20• Biondi-Zoccai GGL et al. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of
acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006;332:202-209• Biondi-Zoccai GGL et al. A practical algorithm for systematic reviews in cardiovascular medicine. Ital Heart J 2004;5:486 -7• Bucher HC et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J
Clin Epidemiol 1997;50:683– 9• Cappelleri JC et al. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332-8• Clarke M et al, eds. Cochrane reviewers’ handbook 4.2.0. (www.cochrane.org/resources/handbook/handbook.pdf)• Cooper H et al, eds. The handbook of research synthesis. New York, NY: Russell Sage Foundation, 1994• Cucherat M et al. EasyMA: a program for the meta-analysis of clinical trials. Comput Methods Programs Biomed
1997;53:187- 90• Egger M et al, eds. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group,
2001• Glass G. Primary, secondary and meta-analysis of research. Educ Res 1976;5:3-8• Glasziou P et al. Systematic reviews in health care. A practical guide. Cambridge: Cambridge University Press, 2001• Guyatt G et al, eds. Users’ guides to the medical literature. A manual for evidence-based clinical practice. Chicago, IL: AMA
Press, 2002• Higgins JPT et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557 – 60• Lau J et al. Summing up evidence: one answer is not always enough. Lancet 1998;351:123 -7• Moher D et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement.
Lancet 1999; 354: 1896-900• Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine.
New York, NY: Oxford University Press, 2000• Song F et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analysis. BMJ 2003;326:472• Thompson SG et al. How should meta-regression analyses undertaken and interpreted? Stat Med 2002;21:1559-73