gbiondizoccai@gmail .com www.metcardio.org Value and Limitations of Meta-Analysis in the Era of Evidence- Based Medicine Giuseppe Biondi-Zoccai, MD Division of Cardiology, Department of Internal Medicine, University of Turin, Turin, Italy Meta-analysis and Evidence-based medicine Training in Cardiology (METCARDIO), Turin, Italy
70
Embed
[email protected] Value and Limitations of Meta-Analysis in the Era of Evidence-Based Medicine Giuseppe Biondi-Zoccai, MD Division.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Why are meta-analysis important: exponential increase in worldwide PubMed citations
PubMed search strategy: ("2001"[PDAT] : "2005"[PDAT]) AND (("systematic"[title/abstract] AND "review"[title/abstract]) OR ("systematic"[title/abstract] AND "overview"[title/abstract]) OR ("meta-analysis"[title/abstract] OR "meta-analyses"[title/abstract]))
Baby steps of meta-analysis• 1904 - Karl Pearson (UK): correlation between inoculation of
vaccine for typhoid fever and mortality across apparently conflicting studies
• 1931 – Leonard Tippet (UK): comparison of differences between and within farming techniques on agricultural yield adjusting for sample size across several studies
• 1937 – William Cochran (UK): combination of effect sizes across different studies of medical treatments
• 1970s – Robert Rosenthal and Gene Glass (USA), Archie Cochrane (UK): combination of effect sizes across different studies of, respectively, educational and psychological treatments
• 1980s – exponential development/use of meta-analytic methods
The Cochrane Collaboration is an world-wide organization that aims to help people make well informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions
• Over 6000 contributors• 50 Collaborative Review Groups (CRGs)• 12 centers throughout the world• 9 fields• 11 Methods Groups• 1 Consumer Network• The Campbell Collaboration (focusing on
Review: Late percutaneous coronary intervention for infarct-related artery occlusionComparison: 01 Late percutaneous coronary intervention vs best medical therapy for infarct-related artery occlusion Outcome: 01 Death
Appraisal tools: Oxman and Guyatt’sEvaluates the internal validity of a review on 9 separate questions for
which 3 distinct anwers are eligible (“yes”, “partially/can’t tell”, “no”):
1. Where the search methods used to find evidence stated?2. Was the search for evidence reasonably comprehensive?3. Were the criteria for deciding which studies to include in the overview reported4. Was bias in the selection of studies avoided5. Were the criteria used for assessing the validity of the included studies reported?6. Was the validity of all studies referred to in the text assessed using appropriate
criteria7. Were the methods used to combine the findings of the relevant studies reported?8. Were the findings of the relevant studies combined appropriately relative to the
primary question the overview addresses?9. Were the conclusions made by the author(s) supported by the data and/or
analysis reported in the overview?Question 10 summarizes the previous ones and, specifically, asks to rate the
scientific quality of the review from 1 (being extensively flawed) to 3 (carrying major flaws) to 5 (carrying minor flaws) to 7 (minimally flawed). The developers of the index specify that if the “partially/can’t tell” answer is used one or more times in questions 2, 4, 6, or 8, a review is likely to have minor flaws at best and is difficult to rule out major flaws (ie a score≤4). If the “no” option is used on question 2, 4, 6 or 8, the review is likely to have major flaws (ie a score≤3).
Biondi-Zoccai et al, Int J Epidemiol 2005 Biondi-Zoccai et al, Am Heart J 2008
Biondi-Zoccai et al, Am Heart J 2005
A simple PubMed strategy for clinical studies on percutaneous coronary intervention for left main coronary artery disease: left AND main AND coronary AND stent* NOT case reports [pt] NOT review [pt] NOT editorial [pt]
A complex PubMed strategy for randomized clinical trials on invasive vs conservative strategies in acute coronary syndromes: (randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized controlled trials[mh] OR random allocation[mh] OR double-blind method[mh] OR single-blind method[mh] OR clinical trial[pt] OR clinical trials[mh] OR (clinical trial[tw] OR ((singl*[tw] OR doubl*[tw] OR trebl*[tw] OR tripl*[tw]) AND (mask*[tw] OR blind[tw])) OR (latin square[tw]) OR placebos[mh] OR placebo*[tw] OR random*[tw] OR research design[mh:noexp] OR comparative study[mh] OR evaluation studies[mh] OR follow-up studies[mh] OR prospective studies[mh] OR cross-over studies[mh] OR control*[tw] OR prospectiv*[tw] OR volunteer*[tw]) NOT (animal[mh] NOT human[mh]) NOT (comment[pt] OR editorial[pt] OR meta-analysis[pt] OR practice-guideline[pt] OR review[pt])) AND ((invasive OR conservative AND (coronary OR unstable angina OR acute coronary syndrome* OR unstable coronary syndrome* OR myocardial infarction)))
• Many scales for the quality of included studies have been reported, but none is reliable or robust
• The recommended approach is to individually appraise the potential risk of the 4 biases (eg A-low, B-moderate, C-high, D-unclear from reported data):
– Selection bias (one group is different than the other)
– Performance bias (treatment is systematically different)
– Adjudication bias (outcome adjudication is selectively
different)
– Attrition bias (follow-up duration or completeness is
• The risk difference (RD), ie absolute risk difference, is the difference between the incidence of events in the experimental vs control groups
• The RD is theoretically the most clinically relevant statistics, but changes too much with disease prevalence
• The number to treat (NNT), defined as 1/RD, identifies the number of patients that we need to treat with the experimental therapy to avoid one event*
• The NNT is the most clinically meaningful parameter to express the impact of a treatment on a dichotomic outcome (eg death), but has the same limits of RD
*Numbers needed to harm (NNH) similarly express the number of patients that we have to treat with the experimental therapy to cause one adverse event
Our advice• Both RR and OR can be your first choice statistics for
uncommon events
• For common events, the OR is clearly less informative than the RR for the busy reader
• Complete your analyses by reporting RD and/or NNT for the sake of clarity
• Fixed effect methods are quite fine for homogeneous/ consistent data
• Random effect methods may be more appropriate for heterogeneous/inconsistent data, but often meta-regression (or even refraining from meta-analysis at all) might be the best option
Small study bias• Publication bias (eg the lower likelihood of
being published for studies with negative findings, or those originating in non-English speaking countries) may bias the results of a meta-analysis
• Other types of small study bias may undermine the validity of a meta-analysis
• A number of tests, analogical (eg the funnel plot) or analytical (eg Egger’s or Peter’s) have been proposed to appraise the likelihood of such small study bias
• Statistical inconsistency (I2) has been recently introduced to overcome the risk of alpha and beta error of standard tests for statistical heterogeneity
• It is computed as [(Q – df)/Q] x 100%, where Q is the chi-squared statistic and df is its degrees of freedom
• I2 values of 25% suggest low inconsistency, 50% moderate inconsistency, and 75% severe inconsistency
Typical Revman outputReview: Late percutaneous coronary intervention for infarct-related artery occlusionComparison: 01 Late percutaneous coronary intervention vs best medical therapy for infarct-related artery occlusion Outcome: 01 Death
Study PCI Medical Rx OR (random) OR (random)or sub-category n/N n/N 95% CI 95% CI O - E Variance
Total (95% CI) 597 592 0.48 [0.28, 0.85]Total events: 20 (PCI), 41 (Medical Rx)Test for heterogeneity: Chi² = 4.25, df = 6 (P = 0.64), I² = 0%Test for overall effect: Z = 2.53 (P = 0.01)
0.1 0.2 0.5 1 2 5 10
Favours PCI Favours medical Rx
Review: Late percutaneous coronary intervention for infarct-related artery occlusionComparison: 01 Late percutaneous coronary intervention vs best medical therapy for infarct-related artery occlusion Outcome: 01 Death
Study PCI Medical Rx OR (fixed) OR (fixed)or sub-category n/N n/N 95% CI 95% CI O - E Variance
A few references• Biondi-Zoccai GGL et al. Parallel hierarchy of scientific studies in cardiovascular medicine. Ital Heart J 2003; 4: 819-20• Biondi-Zoccai GGL et al. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of
acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ 2006;332:202-209• Biondi-Zoccai GGL et al. A practical algorithm for systematic reviews in cardiovascular medicine. Ital Heart J 2004;5:486 -7• Bucher HC et al. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. J
Clin Epidemiol 1997;50:683– 9• Cappelleri JC et al. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332-8• Clarke M et al, eds. Cochrane reviewers’ handbook 4.2.0. (www.cochrane.org/resources/handbook/handbook.pdf)• Cooper H et al, eds. The handbook of research synthesis. New York, NY: Russell Sage Foundation, 1994• Cucherat M et al. EasyMA: a program for the meta-analysis of clinical trials. Comput Methods Programs Biomed
1997;53:187- 90• Egger M et al, eds. Systematic reviews in health care: meta-analysis in context. 2nd ed. London: BMJ Publishing Group,
2001• Glass G. Primary, secondary and meta-analysis of research. Educ Res 1976;5:3-8• Glasziou P et al. Systematic reviews in health care. A practical guide. Cambridge: Cambridge University Press, 2001• Guyatt G et al, eds. Users’ guides to the medical literature. A manual for evidence-based clinical practice. Chicago, IL: AMA
Press, 2002• Higgins JPT et al. Measuring inconsistency in meta-analyses. BMJ 2003;327:557 – 60• Lau J et al. Summing up evidence: one answer is not always enough. Lancet 1998;351:123 -7• Moher D et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement.
Lancet 1999; 354: 1896-900• Petitti DB. Meta-analysis, decision analysis, and cost-effectiveness analysis: methods for quantitative synthesis in medicine.
New York, NY: Oxford University Press, 2000• Song F et al. Validity of indirect comparison for estimating efficacy of competing interventions: empirical evidence from published meta-analysis. BMJ 2003;326:472• Thompson SG et al. How should meta-regression analyses undertaken and interpreted? Stat Med 2002;21:1559-73