Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Cert Endo December 17, 2014.

Clinical Epidemiology Boot Camp:

Systematic Reviews

Selina Liu

MD MSc FRCPC Cert Endo

December 17, 2014

Outline

Introduction – Evidence-Based Medicine (EBM) Levels of evidence

To discuss the definition of a systematic review vs. traditional/narrative reviews

The process of conducting a systematic review Strengths & limitations of systematic reviews

To describe how to critically appraise a systematic review

Example of a systematic review

Evidence-Based Medicine

What is Evidence-Based Medicine?

“…the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients”

“ It’s about integrating individual clinical expertise and the best external evidence”

philosophical origins – date back to mid-19th century Paris (or possibly earlier)

Sackett DL et al. BMJ. 1996;312(7023):71-2


Five Steps of Evidence-Based Medicine

1. Asking Focused Questions Translation of uncertainty to an answerable question

2. Finding the Evidence Systematic retrieval of the best evidence available

3. Critical Appraisal Testing evidence for validity, clinical relevance, and applicability

4. Making a Decision Application of results in practice

5. Evaluating Performance Auditing evidence-based decisions

Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net

http://www.cebm.net/

Evidence-Based Medicine Why Evidence-Based Medicine?

clinical decision making is complex!

Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91


How do we practice Evidence-Based Medicine?

It can be difficult!

“information overload” difficult for clinicians to “keep up” with all of the latest evidence

often there are multiple studies examining the same or similar questions may be of variable quality, generalizability

estimated time required for reading (general medicine): 19 articles per day, 365 days per year

Davidoff F et al. BMJ. 1995;310(6987):1085-6

Evidence-Based Medicine Weighing the evidence - “Levels of Evidence”

OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford Centre for Evidence-Based Medicine.

www.cebm.net/index.aspx?o=5653

http://www.cebm.net/index.aspx?o=5653


Systematic reviews of cohort studies

of RCTs

Systematic Reviews

What is a Systematic Review?

the application of strategies that limit bias in the assembly, critical appraisal, and synthesis of all relevant studies on a specific topic use rigorous, standardized methods for selecting &

assessing articlesOxford Centre for Evidence-Based Medicine www.cebm.net/?o=1116

OR

a report that summarizes all evidence that can be drawn from research (or other sources), that is relevant to a specific clinical question

http://www.cebm.net/?o=1116

Systematic Reviews

Systematic Reviews vs. Traditional Review Articles

traditional review articles written by senior expert in the field, summarizes

evidence and recommendations usually address broad areas/questions (i.e.

“management of T2DM”) often lack structure may include personal experience/anecdotal evidence

Fletcher RH & Fletcher SW 2005. Clinical Epidemiology: The Essentials

Systematic Reviews Systematic Review vs. Traditional/Narrative

Review

Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380

Systematic Reviews

Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)

Systematic Reviews

Process of Conducting a Systematic Review

1. Define the question

2. Conduct literature search3. Apply inclusion and exclusion criteria4. Create data abstraction5. Conduct analysis


Systematic Reviews - Process

1. Define the Question

single, focused question i.e. What is the effectiveness of using a powered (electric)

toothbrush compared with using a manual toothbrush for maintaining oral health?

specify inclusion and exclusion criteria Population, Intervention or Exposure, Outcome, Methodology

For the systematic review to be useful: strong studies of the question should be available, but their results

should not be so much in agreement that the question is already answered!

there should not be so few studies of the question that each individual study could be fully critiqued directly


2. Conduct literature search

need to ensure that all of the appropriate studies are included NOT just a biased sample of studies

decide on information sources i.e. MEDLINE, recent reviews, textbooks, experts in the

field, articles cited by references already found by other approaches, databases of articles, clinical trial registries etc.

identify titles and abstracts


3. Apply inclusion and exclusion criteria

Apply inclusion and exclusion criteria to titles and abstracts obtain full articles for eligible titles and abstracts

Apply inclusion and exclusion criteria to full articles Select final eligible articles

Assess agreement on study selection

Of the initial titles and abstracts retrieved, usually only a small proportion of articles are selected


4. Create data abstraction

Assess methodologic quality of each article Assess agreement on validity assessment Data abstraction

Participants Interventions and Comparison Interventions Study Design Results


5. Conduct analysis

Summarize data If appropriate: meta-analysis – statistical technique to

combine quantitative data usually combine studies vs. combine patients

Describe results – often graphically Forest Plot – shows point estimate and confidence interval (for

RCTs, observational studies) Summary Receiver-Operator Curves (for studies of diagnostic

tests) Explore heterogeneity, conduct subgroup analysis (if

appropriate) Explore possibility of publication bias (and other biases)

(i.e. funnel plot)


How to decide if appropriate to perform a meta-analysis? Two general approaches:

1. statistical test for homogeneity BUT – even if fail to reject H0 (i.e. no evidence of a statistically

significant difference between studies), usually have high risk of false-negative (saying studies are homogeneous when they really are not) Limited power - meta-analyses are usually of few number of studies,

- affected also by number of patients/study, distribution of patients among studies

2. informed judgement

Systematic Reviews - Process Meta-analysis – mathematical models:

Fixed-Effect Model Assumes that studies are of exactly the same question, so

results differ only by chance Confidence intervals calculated may imply more precision (i.e.

are narrower) than in reality (since studies usually differ somewhat)

Random-Effects Model Assumes that the studies address somewhat different

questions, but that they form a closely related family of studies of a similar question

Studies taken to be a random sample of all studies bearing on the question

Produces WIDER confidence intervals (more “realistic”)Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4th Edition)

Systematic Reviews – Forest Plot

Study name Statistics for each study Odds ratio and 95% CI

Odds Lower Upper ratio limit limit

Kelly, 1964 0.590 0.096 3.634Hedrin, 1980 0.464 0.201 1.074Leigh, 1962 0.394 0.076 2.055Novak, 1992 0.490 0.088 2.737Saint, 1998 1.250 0.479 3.261Pilbean, 1936 0.129 0.027 0.605Day, 1960 0.313 0.054 1.805Kelly, 1966 0.429 0.070 2.620Singh, 2000 0.718 0.237 2.179Stewart, 1994 0.143 0.082 0.250

0.328 0.233 0.462

0.01 0.1 1 10 100

Favours Tx Favours Pbo

Impact of Treatment on Mortality

Meta Analysis

Systematic Reviews - Bias

Several types of bias:

publication bias published studies may be systematically different than

unpublished studies (“positive” studies vs. “negative” studies?)

language bias i.e. if only English-language articles are selected

size bias large studies that result in several publications may be more

readily noticed than smaller studies bias related to funding?

industry-sponsored studies

How to detect publication bias? Funnel plots – plot effect vs. study size/precision

symmetrical,peaked distribution(inverted funnel)


How to detect publication bias? Funnel plots

asymmetricaldistribution


Systematic Reviews - Strengths

provide an efficient way to become familiar with the best available research evidence for a focused clinical question

can establish whether results are consistent, generalizable across populations/settings, treatment variations, and whether findings vary by certain subgroups

can extend the available literature (if the review team has obtained unpublished information from the primary authors)

meta-analyses – may provide a more precise estimate of the underlying “true effect” than any individual study

Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60.

Systematic Reviews - Limitations

summarized results are limited by the quality of the primary studies “garbage in garbage out”

results dependent on selection of included articles quality threshold, publication bias, language bias etc.

meta-analyses - may be inappropriate to mathematically combine primary study results if the primary studies differ in design, quality, population, intervention etc. subjectivity involved in deciding whether to pool or not subjectivity in interpretation of summarized results

(“over-interpretation)Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60.

Systematic Reviews – Critical Appraisal

Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71

Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157

Critical Appraisal – Tools





Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157





Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

AMSTAR – 2007 Assessment of Multiple SysTemAtic Reviews

Shea BJ, Grimshaw JM, Wells GA et al.

11 item tool developed via exploratory factor analysis of a 37-item

assessment tool

Critical Appraisal - Tools

Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10

Reporting Systematic Reviews - Tools

The PRISMA Statement – 2009 Preferred Reporting Items for Systematic reviews and

Meta-Analyses

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group

PLoS Medicine Annals of Internal Medicine BMJ Journal of Clinical Epidemiology Open Medicine International Journal of Surgery

The PRISMA Statement - 2009

Aim – to help authors improve the reporting of systematic reviews and meta-analyses **NOT intended to be a quality assessment tool

27 item checklist four-phase flow diagram

update and expansion of prior QUOROM statement QUality Of Reporting Of Meta-analyses - 1996

focused on reporting of meta-analyses of randomized controlled trials

Table 1. Checklist of items to include when reporting a systematic review or meta-analysis.

Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

Figure 1. Flow of information through the different phases of a systematic review.

Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097

Example of a Systematic Review

Cochrane Database of Systematic Reviews 2014 June, Issue 6: CD002281

Powered Toothbrushes for Oral Health

Objective – to compare manual and powered tootbrushes in everyday use, by people of any age, in relation to the removal of plaque, the health of the gingivae, staining and calculcus, dependability, adverse effects and cost

Selection criteria – RCTs of ≥ 4 weeks of unsupervised powered toothbrushing vs. manual toothbrushing for oral health in children/adults included:

cross-over trials if wash-out period length was > 2 weeks (to diminish any carry-over effects)

any type of powered toothbrushes side to side, counter oscillation, rotation oscillation, circular, ultrasonic, ionic

excluded: trials only comparing different kinds of powered brushes or different kinds of

manual brushes “split-mouth” trials – not representative of everyday useYaacob M. et al. Cochrane Database of Syst Rev 2014, Issue 6:CD002281


Primary outcomes – quantified levels of plaque, gingivitis, or both

plaque index – Quigley Hein (Turesky), Silness and Löe, Ainamo Bay, Navy plaque index mod Rustogi, O’Leary index

gingivitis index – Löe Silness, Lobene gingivial index, Bleeding on Probing (BOP), Papillary bleeding index

where possible, values recorded on arrival at the assessment

If necessary, measures of gingivitis taken after participants permitted to brush teeth at the assessment visit were used (assumed that toothbrushing would not affect gingivitis within such a short period)

but, measures of plaque taken after participants brushed teeth at assessment visit were NOT used

Yaacob M. et al. Cochrane Database of Syst Rev 2014, Issue 6:CD002281


Secondary outcomes - levels of calculus and staining, dependability and cost of the brush used, adverse effects (hard/soft tissue injury, damage to orthodontic appliances and prostheses)

Data classification – short term (1-3 months), long term (>3 months)

If ≥ 4 studies in meta-analysis, random-effects model used (otherwise fixed-effects model used)



Review authors’ judgements about each “risk of bias” item for each included study

Risk of Bias Summary


Review authors’ judgements about each “risk of bias” item presented as percentages across all included studies

Risk of Bias Graph


Primary outcome – plaque scores at > 3 months

Results – Powered vs. Manual Brushes

Primary outcome – gingival scores at > 3 months


Primary outcome – gingival scores at > 3 months


Assessment of Publication Bias

Assessment of Publication Bias

Secondary outcomes: Cost – none of the trials reported on relative costs Reliability – 2 trials

mechanical failure in 1/48 and 4/20 powered toothbrushes Calculus – 3 trials

2 trials – no significant difference; 1 trial – powered brush better

Stain – 3 trials no significant difference between brush types

Adverse events – tissue trauma – 40 trials 27 trials – no trauma; 6 trials – no difference between brush

types; 7 trials – differences between brush types

Results


Moderate quality evidence exists that demonstrate that powered toothbrushes provide a statistically significant benefit compared with manual toothbrushes for both reduction of plaque and gingivitis plaque: as per Quigley Hein index - 11% reduction short term,

21% reduction long term gingivitis: as per Löe Silness index – 6% reduction short term,

11% reduction long term

Discussion

However, high levels of heterogeneity that was not explained by the different powered toothbrush type subgroups Greatest body of evidence – for rotation oscillation

brushes (statistically significant reduction in plaque and

gingivitis at both time points)

Powered toothbrushes reduce plaque and gingivitis more than manual toothbrushes in the short and long term clinical importance of findings remain unclear

Greater standardization of design of future studies/meta-analyses would be beneficial

Cost, reliability and side effects were not consistently reported any reported side effects were localized and only

temporary

Conclusions

Systematic Reviews – Critical Appraisal

Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71

http://upload.wikimedia.org/wikipedia/commons/b/bd/Checkmark_green.svg

Useful Resources

Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill available online via Western Libraries

Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net

Cochrane Database of Systematic Reviews www.thecochranelibrary.com


http://www.thecochranelibrary.com/

References Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ.

1996;312(7023):71-2 Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91 Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):1085-6 Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”,

Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653 Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4th Edition).

Baltimore MD, Lippincott Williams & Wilkins Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical

Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill

Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380 Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60. Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA.

1994;272(17):1367-71 Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33 Cochrane Database of Systematic Reviews www.thecochranelibrary.com



http://www.thecochranelibrary.com/

Clinical Epidemiology Boot Camp: Systematic Reviews Selina Liu MD MSc FRCPC Cert Endo December 17, 2014.

Documents

evidence levels

evidencebased medicinehow

latest evidence

evidencebased medicine

best evidence available

general medicine

systematic reviewvs

systematic review example