Systema(c review and metaanalysis of Diagnos(c Test Studies KuanFu Chen CGMH EM SR/MA workshop January 8, 2013
Aug 07, 2015
Systema(c review and meta-‐analysis of Diagnos(c Test Studies
Kuan-‐Fu Chen CGMH EM
SR/MA workshop January 8, 2013
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 2
Difference of Dx SR/MA
• ID studies, assessing bias • Method to combine results • Paired of summary sta(s(cs to pool • Design of diagnos(c studies
1/8/13 KFC CGMH EM Dx SR/MR talk 3
Diagnos(c studies
• Defini(on – Dia + gnosis = apart/separated + knowledge – Dia: Greek ‘through’ – Gnosis: Greek ‘knowledge’ – To reduce uncertainty
• Purposes – Screening, triage, add-‐on or replacement tests, predict outcomes or monitor dz process
– Purpose must be considered during evalua(on
1/8/13 KFC CGMH EM Dx SR/MR talk 4
Phases of Dx test evalua(on
• Prelim evalua(on (case-‐control) – I: whether results different for pts ± dz – II: whether dz more likely to have + results
• Accuracy & probability of condi(on (cross-‐sec(onal/cohort study) – III: how well dis(nguishes btw pts ± dz with suspicion – IV: how informa(ve as add-‐on
• Health outcomes (Randomized or before-‐ader study) – V: whether leads to beeer outcomes – VI: acceptable costs
1/8/13 KFC CGMH EM Dx SR/MR talk 5
Phases of biomarker evalua(on
1/8/13 KFC CGMH EM Dx SR/MR talk 6
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 7
Steps of (diagnos(c) SR/MA
• Formula(ng a focused ques(on • Reviewing guidelines • ID databases/sources of studies • Run searches and save cita(ons • First screen by two reviewers • Second screen by two reviewers • Data extrac(on and quality assessment • Data analyses
1/8/13 KFC CGMH EM Dx SR/MR talk 8
1/8/13 KFC CGMH EM Dx SR/MR talk 9
1/8/13 KFC CGMH EM Dx SR/MR talk 10
1/8/13 KFC CGMH EM Dx SR/MR talk 11
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 12
PICO(S) for of diagnos(c studies • Popula(on
– What are the characteris(cs of the pa(ents? – What is the condi(on that may be present?
• Interven(on (diagnos(c test) – Which diagnos(c test am I considering?
• Comparison – What is the diagnos(c gold standard?
• Outcome – How likely is the test to predict/rule out this condi(on?
• Study design/sekng – What study design would provide the best level of evidence? – What clinical sekng would this study apply to?
1/8/13 KFC CGMH EM Dx SR/MR talk 13
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 14
Guidelines for (SR/MA) studies
• QUA D A S – QUAlity of Diagnos(c Accuracy research Studies in SR
• STA R D – STAndard for Repor(ng of Diagnos(c accuracy
• RE MARK – REpor(ng recommenda(ons for tumor MARKer prognos(c studies
• P R I S M – Preferred Repor(ng Items for SR/MA
1/8/13 KFC CGMH EM Dx SR/MR talk 15
QUADAS (2)
• Four phases to establish tool for each review • Four key domains to review • Three sec(ons in each domain for risk of bias
1/8/13 KFC CGMH EM Dx SR/MR talk 16
QUADAS – phases
• Review ques(ons • Review specific tailoring • Flow diagram • Judgments on bias and applicability
1/8/13 KFC CGMH EM Dx SR/MR talk 17
QUADAS – phases
• Review ques(ons – Four-‐part ques(on:
• (P) Pa(ents, • (I) Index test(s), • (C) Reference standard, and • (O) Target condi(on
– Diagnos(c pathway: • Sekng, inten(on, pa(ent presenta(on, and/or prior tes(ng
1/8/13 KFC CGMH EM Dx SR/MR talk 18
QUADAS – phases
• Review specific tailoring
1/8/13 KFC CGMH EM Dx SR/MR talk 19
QUADAS – phases
• Flow diagram
1/8/13 KFC CGMH EM Dx SR/MR talk 20
QUADAS – phases
• Judgments on bias and applicability – Three sec(ons in each domain for risk of bias
• Informa(on to support • Signaling ques(ons • Judgment of risk of bias
– Applicability • Similar but not including signaling ques(ons • Refer to first phase (review ques(on)
1/8/13 KFC CGMH EM Dx SR/MR talk 21
QUADAS – domains
• Pa(ent selec(on • Index test • Reference standard • Flow and (ming
1/8/13 KFC CGMH EM Dx SR/MR talk 22
QUADAS – domains • Pa(ent selec(on – Signaling ques(ons
1. Was a consecu(ve or random sample of pa(ents enrolled? 2. Was a case-‐control design avoided? 3. Did the study avoid inappropriate exclusions?
– Risk of bias • Spectrum bias (one of selec(on biases) (lab)? • Excluding difficult to diagnose; liele diagnosis uncertainty:
– overop(mis(c • Exclusion of “red flags”: underes(ma(on • Selec(on towards more severe manifesta(ons
– Increase prevalence – Bias in any direc(on
1/8/13 KFC CGMH EM Dx SR/MR talk 23
QUADAS – domains
• Pa(ent selec(on – Applicability concerns
• Target vs. Study popula(on: generalizability • Any discrepancy re: severity, demographics, presence of differen(al diagnosis, comorbidity, sekng and previous tes(ng protocol
– Supplemental: prospec(ve vs. Retrospec(ve? • Retrospec(ve:
– No verifica(on of Dz status – Same reference standard for all? – Same assessors for reference standard for all?
Target'Population
Study&population
Target'Population
Study&population
Target'Population
Study&population
1/8/13 KFC CGMH EM Dx SR/MR talk 24
QUADAS – domains
• Index test – Signaling ques(ons
1. Were the index test results interpreted without knowledge of the results of the reference standard?
2. If a threshold was used, was it pre-‐specified?
– Risk of bias • Ques(on 1: blinding, subjec(vity • Ques(on 2: overop(mis(c
1/8/13 KFC CGMH EM Dx SR/MR talk 25
QUADAS – domains
• Index test – Applicability concerns
• Same index tests?
– Supplemental: (QUADAS 1) • Reproducibility: for all tests, different phases
1/8/13 KFC CGMH EM Dx SR/MR talk 26
QUADAS – domains
• Reference standard – Signaling ques(ons
1. Is the reference standard likely to correctly classify the target condi(on?
2. Were the reference standard results interpreted without knowledge of the results of the index test?
– Risk of bias • Ques(on 1: direc(on? Misclassifica(on bias (textbook?) – Bias confounding and role of chance slide
• Ques(on 2: similar to index test, Incorpora(on bias
1/8/13 KFC CGMH EM Dx SR/MR talk 27
QUADAS – domains
• Reference standard – Applicability concerns:
• target condi(on defined by the reference standard does not match the ques(on? • Example: U/C for UTI with different cutoffs
1/8/13 KFC CGMH EM Dx SR/MR talk 28
QUADAS – domains
• Flow and (ming – Signaling ques(ons
1. Was there an appropriate interval between index test and reference standard?
2. Did all pa(ents receive the same reference standard? 3. Were all pa(ents included in the analysis?
– Risk of bias • Ques(on 1: misclassifica(on bias (if delay or treatment started). Direc(on? Acute vs. Chronic
• Ques(on 2: Verifica(on bias: overes(mate • Ques(on 3: lost to follow-‐up: Direc(on? Could be either direc(on (selec(on bias)
1/8/13 KFC CGMH EM Dx SR/MR talk 29
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 30
Literature searching
• Electronic database – Less produc(ve and more difficult
• Different strategies – Different indexing terms/text words – MeSH: diagnosis – Textwords: specificity, sensi(vity
1/8/13 KFC CGMH EM Dx SR/MR talk 31
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 32
Evalua(on of studies
• Selec(on of sample. – Ideal: consecu(ve/randomly selected
• Reference test • Blinding • Quality repor(ng • Evidence of bias: – table of rela(ve DOR in different study design
• Incorpora(ng QA in SR: Checklist
1/8/13 KFC CGMH EM Dx SR/MR talk 33
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 34
Data extrac(on • Accuracy expression
– Sensi(vity/specificity: Based on cut-‐off – ROC curve – PPV → influence of prevalence
• Choice of parameters: – May consider DOR instead of Sensi(vity/specificity, LR
• DOR (Diagnos(c Odds Ra(o) also = LR(+)/LR(-‐) – DOR 25: LR(+) 5, LR(-‐) 0.2, DOR 100: LR(+) 10, LR(-‐) 0.1
• However, discarded informa(on • Predic(ve value: high heterogeneity→prevalence varia(on
– Post-‐test probability can be used instead • Examples of abstract/ar(cle review form
1/8/13 KFC CGMH EM Dx SR/MR talk 35
Objec(ves
• Differences of diagnos(c SR/MA? • Steps of diagnos(c SR/MA • Formula(ng a focused ques(on • Reviewing guidelines: QUADAS • Literature searching • Evalua(on of studies • Data extrac(on • Data synthesis
1/8/13 KFC CGMH EM Dx SR/MR talk 40
Data synthesis
Meta-regression DOR on
characteristics, investigate source of
heterogeneity
LM methodtest of
asymmetry,SROC or pool ROC curves
Pool sensitivity & specificity
Test of cut-off effect
Test of heterogeneity
heterogenous
homogenous
no
yes sym
Bayesian adaptation
SROC
asym
SROC: summary ROC LM method: Lieenburg – Moses regression method DOR: diagnos(c odds ra(o
1/8/13 KFC CGMH EM Dx SR/MR talk 41
WORKSHOP TIME! Systema(c review and meta-‐analysis of Diagnos(c Test Studies
1/8/13 KFC CGMH EM Dx SR/MR talk 42
Objec(ves
• Understand and workout poten(al bias during diagnos(c SR/MA – Spectrum bias – Misclassifica(on bias – Incorpora(on bias – Lost of follow up – Selec(ng threshold – verifica(on bias
• Use QUADAS-‐2 form to evaluate one study
1/8/13 KFC CGMH EM Dx SR/MR talk 43
Selec(on of pa(ent (spectrum bias)
• Excluding difficult to diagnose – Overop(mis(c or underes(ma(ng, why?
• Excluding ‘red flag’ – Overop(mis(c or underes(ma(ng, why?
• Evaluate and discuss the reasons in examples
1/8/13 KFC CGMH EM Dx SR/MR talk 44
Example 1: Centor’s CPR
1/8/13 KFC CGMH EM Dx SR/MR talk 45
Example 2: BNP
1/8/13 KFC CGMH EM Dx SR/MR talk 46
Example 2: BNP
1/8/13 KFC CGMH EM Dx SR/MR talk 47
Misclassifica(on bias
• Use those tables to discuss the reasons – PCR vs. B/C for sepsis diagnosis – PCT vs. B/C for sepsis diagnosis
• Use blood culture as example – Discuss interval and reference test issue
1/8/13 KFC CGMH EM Dx SR/MR talk 48
Misclassifica(on bias Cases: B/C + Control: B/C -‐ Total
PCR + 100 40 150
PCR -‐ 50 60 100
150 100 250
False Sensi(vity=66% Specificity=60%
Cases: real sepsis Control: real ctrl Total
PCR + 110 40 150
PCR -‐ 40 60 100
150 100 250
True Sensi(vity=73% Specificity=60%
DifferenAal misclassificaAon bias Say Sensi(vity of B/C = 80%, specificity = 90%, and related to PCR results
1/8/13 KFC CGMH EM Dx SR/MR talk 49
Misclassifica(on bias Cases: B/C + Control: B/C -‐ Total
PCT + 100 40 150
PCT -‐ 50 60 100
150 100 250
False Sensi(vity=66% Specificity=60%
Cases: real sepsis Control: real ctrl Total
PCT + 125 25 150
PCT -‐ 63 37 100
188 62 250
True Sensi(vity=66% Specificity=60%
Non-‐DifferenAal misclassificaAon bias Say Sensi(vity of B/C = 80%, specificity = 75%, and not related to PCT results
1/8/13 KFC CGMH EM Dx SR/MR talk 50
Incorpora(on bias
• Example: SIRS vs. WBC for sepsis diagnosis – Overop(mis(c or underes(ma(ng, why?
1/8/13 KFC CGMH EM Dx SR/MR talk 51
Lost of follow up
• Example: transferred to other hospital – Overop(mis(c or underes(ma(ng, why?
1/8/13 KFC CGMH EM Dx SR/MR talk 52
Selec(ng threshold
• Example: – Post-‐hoc determina(on of BUN/Cre > 15 as risk factor in acute stroke
– Overop(mis(c or underes(ma(ng, why?
1/8/13 KFC CGMH EM Dx SR/MR talk 53
Verifica(on bias
• Example: PSA – Verifica(on and incorpora(on biases in studies assessing screening tests: prostate-‐specific an(gen as an example (Gupta & Roehrborn)
– Overop(mis(c or underes(ma(ng, why?
1/8/13 KFC CGMH EM Dx SR/MR talk 54
Verifica(on bias Disease + Disease -‐ Total % Verified
PSA + 231 32 263 61%
PSA -‐ 27 54 8 37%
Total 258 86 344
Sensi(vity=90% Specificity=63%
Corrected Disease + Disease -‐ Total
PSA +
PSA -‐
Total
Sensi(vity=__% Specificity=__%
1/8/13 KFC CGMH EM Dx SR/MR talk 55
Verifica(on bias Disease + Disease -‐ Total % Verified
PSA + 231 32 263 61%
PSA -‐ 27 54 8 37%
Total 258 86 344
Sensi(vity=90% Specificity=63%
Disease + Disease -‐ Total
PSA + 377 52 429
PSA -‐ 74 147 221
Total 451 199 650
Sensi(vity=84% Specificity=74%
1/8/13 KFC CGMH EM Dx SR/MR talk 56
Verifica(on bias
1/8/13 KFC CGMH EM Dx SR/MR talk 57
Verifica(on bias
1/8/13 KFC CGMH EM Dx SR/MR talk 58