Diagnostic Accuracy and Reproducibility of WHO-Endorsed Phenotypic Drug Susceptibility Testing Methods for First-Line and Second-Line Antituberculosis Drugs

Page 1

  Published Ahead of Print 14 November 2012. 10.1128/JCM.02724-12.

2013, 51(2):393. DOI:J. Clin. Microbiol. Steingart and Jessica MinionGrace Lin, Edward Desmond, Laura L. Flores, Karen R. David J. Horne, Lancelot M. Pinto, Matthew Arentz, S.-Y. Antituberculosis DrugsFirst-Line and Second-Line Susceptibility Testing Methods forWHO-Endorsed Phenotypic Drug Diagnostic Accuracy and Reproducibility of

http://jcm.asm.org/content/51/2/393Updated information and services can be found at:

These include:

SUPPLEMENTAL MATERIAL Supplemental material

REFERENCEShttp://jcm.asm.org/content/51/2/393#ref-list-1at:

This article cites 24 articles, 10 of which can be accessed free

CONTENT ALERTS more»articles cite this article),

Receive: RSS Feeds, eTOCs, free email alerts (when new

http://journals.asm.org/site/misc/reprints.xhtmlInformation about commercial reprint orders: http://journals.asm.org/site/subscriptions/To subscribe to to another ASM Journal go to:

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 2

Diagnostic Accuracy and Reproducibility of WHO-EndorsedPhenotypic Drug Susceptibility Testing Methods for First-Line andSecond-Line Antituberculosis Drugs

David J. Horne,a Lancelot M. Pinto,b Matthew Arentz,a S.-Y. Grace Lin,c Edward Desmond,c Laura L. Flores,d Karen R. Steingart,e

Jessica Minionf

Division of Pulmonary and Critical Care Medicine, University of Washington, Seattle, Washington, USAa; Respiratory Epidemiology and Clinical Research Unit, McGillUniversity, Montreal, Quebec, Canadab; California Department of Public Health, Richmond, California, USAc; College of Chemistry, University of California, Berkeley,California, USAd; Department of Health Services, University of Washington School of Public Health, Seattle, Washington, USAe; Department of Laboratory Medicine, ReginaQu’Appelle Health Region, Regina, Saskatchewan, Canadaf

In an effort to update and clarify policies on tuberculosis drug susceptibility testing (DST), the World Health Organization(WHO) commissioned a systematic review evaluating WHO-endorsed diagnostic tests. We report the results of this systematicreview and meta-analysis of the diagnostic accuracy and reproducibility of phenotypic DST for first-line and second-line antitu-berculosis drugs. This review provides support for recommended critical concentrations for isoniazid and rifampin in commer-cial broth-based systems. Further studies are needed to evaluate critical concentrations for ethambutol and streptomycin thataccurately detect susceptibility to these drugs. Evidence is limited on the performance of DST for pyrazinamide and second-linedrugs.

The global epidemic of drug-resistant tuberculosis (DR-TB),particularly multidrug-resistant TB (MDR-TB, defined as re-

sistance to at least isoniazid and rifampin), is one of the mostserious problems facing TB care and control efforts. In their mostrecent worldwide survey, the World Health Organization (WHO)documented the highest rates of MDR-TB ever reported (1). Ef-fective management of drug-resistant TB relies on multiple com-ponents, including detection, treatment, prevention, surveillance,and continuous program evaluation (2). Expanding the capacityto diagnose cases of drug-resistant TB is a priority for global TBcontrol, requiring clear policies on the use of diagnostic tests andstrengthened laboratories in which testing can be safely and effec-tively carried out (3).

Conventional phenotypic drug susceptibility testing (DST) us-ing the proportion method (PM) on solid media has been wellstudied for isoniazid and rifampin, with a general consensusachieved regarding methodology, critical concentrations, and ex-pected performance (4). However, the diagnostic accuracy andreproducibility of DST for other first-line and second-line drugsare inadequate (5). DST for second-line anti-TB drugs has notbeen standardized internationally, which is reflected in the widevariability of practices among supranational reference laborato-ries, underscoring the need for standardization of the methodsand interpretive criteria for second-line DST (4, 6). Further com-plicating the lack of consensus is the increasing number of differ-ent DST methods that are available. In 2008, the WHO Stop TBDepartment published interim laboratory policy guidance forDST of second-line anti-TB drugs (4). Guideline recommenda-tions for a specific DST method should ideally be based on thediagnostic test accuracy, reproducibility, ease of use, cost, andrapidity of result availability.

In an attempt to address gaps in the evidence base, WHO ini-tiated an update to the 2008 interim guidelines with an expandedscope to include DST for all first-line and second-line drugs. Aspart of the update, we conducted a systematic review to determinethe diagnostic accuracy and reproducibility of WHO-endorsed

phenotypic DST methods and commercial genotypic DST meth-ods for first-line and second-line anti-TB drugs. While severalsystematic reviews have previously evaluated specific DST meth-ods (7–14), we assessed a variety of DST methods, focusing onstudies that used WHO-defined criteria for drug resistance in thereference standard. We also collected data on test reproducibility.

(The evidence from this systematic review was presented at aWHO Expert Group meeting in March 2012. The current minire-view presents results for phenotypic DST methods. Upon finaliza-tion, the full Expert Group meeting report, including results forgenotypic DST methods, is to be posted on the WHO website[www.who.int/tb/laboratory/policy_statements/].)

APPROACH

We followed standard guidelines for systematic reviews of diag-nostic test accuracy as recommended by the Cochrane Collabora-tion Diagnostic Test Accuracy Working Group, including the de-velopment of a detailed protocol prior to starting the review(15, 16).

Criteria for considering studies for this review. We consid-ered primary studies, regardless of study design, that evaluated aphenotypic method of DST on Mycobacterium tuberculosis. Weincluded direct and indirect methods of DST, performed on anyspecimen, from all patients confirmed or suspected of having TB,from all settings and countries.

Studies were eligible for inclusion that either (i) evaluated anindex test against a defined reference standard and allowed an

Published ahead of print 14 November 2012

Address correspondence to David J. Horne, [email protected].

Supplemental material for this article may be found at http://dx.doi.org/10.1128/JCM.02724-12.

Copyright © 2013, American Society for Microbiology. All Rights Reserved.

doi:10.1128/JCM.02724-12

MINIREVIEW

February 2013 Volume 51 Number 2 Journal of Clinical Microbiology p. 393–401 jcm.asm.org 393

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 3

estimation of sensitivity, specificity, and agreement between thetests or (ii) studied the reproducibility of DST results with anindex test when testing the same M. tuberculosis isolate two ormore times. Acceptable reference standards, determined in con-sultation with WHO, were the proportion method (PM) per-formed on Löwenstein-Jensen (LJ), Middlebrook 7H10, orMiddlebrook 7H11 agar medium and use of the Bactec 460 system(Becton, Dickinson). We excluded studies from which we wereunable to extract data for true positives (TP), true negatives (TN),false positives (FP), and false negatives (FN). We excluded studiesthat were reported in correspondence or conference abstracts.

Search methods for identification of studies. We searchedMEDLINE, EMBASE, and the Cochrane Library (from the earliestrecord dates to 30 November 2011). We restricted the search tostudies published in English, French, or Spanish. If peer-reviewedsystematic reviews had been performed for an index test, we up-dated the systematic review to include studies published sinceprior reviews were completed. Details of the MEDLINE search areprovided in Appendix SA in the supplemental material.

Selection of studies and data extraction. Three pairs oftrained reviewers screened the titles and abstracts of identifiedarticles for relevance. Each reviewer within a pair worked inde-pendently to identify potentially eligible studies. Any citationidentified as “include” during this stage (screen 1) was selected forfull-text review and assessed for study eligibility using predefinedinclusion and exclusion criteria (screen 2). In screen 2, any dis-crepancies were resolved by a discussion between the reviewers inthe pair or, if they were unable to resolve the discrepancy, by adecision of the lead reviewer (D. J. Horne). We reviewed the ref-erence lists of select original papers and reviews to identify addi-tional relevant studies. A list of excluded studies and their reasonsfor exclusion is available upon request.

Two reviewers (D. J. Horne and K. R. Steingart) developed,piloted, and refined a standardized data extraction form (see Ap-pendix SB in the supplemental material). The same pairs of re-viewers extracted data from each study.

Assessment of methodological quality. Study quality was as-sessed with Quality Assessment of Diagnostic Accuracy Studies(QUADAS), version 2 (17). As recommended, all four domains inthe QUADAS-2 tool (patient selection, index test, reference stan-dard, and flow and timing) were judged as low, high, or unclearrisk of bias, and the first three domains were judged for concernsof applicability. Disagreements were resolved by a third reviewer(K. R. Steingart). Appendix SC in the supplemental material de-scribes the criteria that needed to be met for judgments concern-ing risk of bias or applicability for each of the QUADAS-2 do-mains.

Statistical analysis and data synthesis. We adopted the fol-lowing overall approach to data analysis a priori to account for theconsiderable heterogeneity in results expected across studies ofthe different anti-TB drugs and different index tests included inthe review. First, we classified the following subgroups: (i) com-mercial broth-based tests, (ii) noncommercial solid-medium-based tests, and (iii) novel noncommercial tests. Second, withineach subgroup, we separately synthesized data for each drug ac-cording to the index test and whether the reference standard useda currently recommended critical concentration. Finally, we clas-sified studies by whether the index tests were performed as director indirect tests.

For each study, we determined sensitivity and specificity along

with 95% confidence intervals (CIs) with exact methods usingStata/IC 11.0 (Stata Corp.) (18) and generated forest plots usingMeta-DiSc (version 1.4) (19). When possible, we performedmeta-analyses by drug for each index test to determine pooledperformance estimates. To be included in a meta-analysis sub-group, studies were required to satisfy the following criteria: (i)availability of at least four studies using the same critical concen-tration for the index test and (ii) use of the currently recom-mended critical concentration for the reference standard. We sep-arately pooled direct and indirect testing. We used a bivariaterandom effects regression model to determine summary estimatesfor sensitivity and specificity (16, 20), using the user-written met-andi command in Stata/IC 11.0 (18). If the regression model didnot fit the data and there was little discordance between the indextest and the reference standard, we pooled sensitivity and specific-ity data separately using Meta-DiSc with the reasoning that therewas little or no correlation between these measures across studies(19). Pooled agreement estimates were determined using randomeffects regression modeling in Stata/IC 11.0.

FINDINGSSearch results. The initial search yielded 8,464 citations (Fig. 1).After full-text review of 229 papers identified in electronic search-ing and 25 papers identified in the bibliography review, we iden-tified 108 distinct papers on diagnostic accuracy and/or reproduc-ibility of phenotypic DST methods: 45 papers evaluating DST forfirst-line drugs, 12 papers evaluating DST for second-line drugs,and 51 papers evaluating DST for both first-line and second-linedrugs (see Appendix SD in the supplemental material). Amongthe 96 papers that evaluated DST for first-line drugs, 93 papersevaluated only diagnostic test accuracy; 2 papers evaluated diag-nostic test accuracy and reproducibility; and 1 paper evaluatedonly reproducibility. Of the 63 on DST for second-line drugs, 57papers evaluated only diagnostic test accuracy; 3 papers evaluateddiagnostic test accuracy and reproducibility; and 3 papers evalu-ated only reproducibility. As some papers evaluated two or moreindex tests, drugs, and/or their critical concentrations, we identi-fied these different evaluations (here referred to as studies) withineach paper. In total, there were 405 studies containing 18,666 sam-ples (42,926 total tests) that addressed diagnostic accuracy and 43studies (5,670 total tests) that addressed reproducibility.

Study characteristics. Eleven different index tests are repre-sented in this review (Table 1). These tests include (i) commercialbroth-based systems, such as MGIT 960 (BD Diagnostic Systems,Sparks, MD), BBL Mycobacterial Growth Indicator Tube (“MGITmanual”; BD Diagnostic Systems), and VersaTREK MycobacteriaDetection & Susceptibility (Thermo Fisher Scientific, Sun Prairie,WI); (ii) noncommercial solid-medium methods, such as the re-sistance ratio and absolute concentration techniques on LJ media;and (iii) noncommercial novel tests, such as colorimetric redoxindicator assays using alamarBlue, resazurin, or tetrazolium, theMicroscopic Observation Direct Susceptibility (MODS) assay(now available in both a noncommercial and a commercial ver-sion [Hardy Diagnostics]), and the nitrate reductase assay (NRA)using either solid or liquid media. The majority of studies (354/405; 87.4%) performed indirect testing on culture isolates. Fifty-six percent of first-line drug testing and 38% of second-line drugtesting occurred in low- or middle-income countries. Of the totalincluded studies, 367 (90.6%) chose a critical concentration forthe reference standard (when available) according to current rec-

Minireview

394 jcm.asm.org Journal of Clinical Microbiology

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 4

ommendations. The median number of samples included in thestudies was 77 (interquartile range, 45 to 132).

Methodological quality of included studies. In the studies as-sessing DST for first-line anti-TB drugs, around 80% of studieswere considered to be at high or unclear risk of bias in patientselection because these studies lacked consecutive or random se-lection of patients or samples, used a case-control study design, ordid not report this information (see Fig. S1 in the supplementalmaterial). For the domain concerning the index test, only 38% ofstudies were considered to be at low risk of bias because the indextest result was interpreted blindly. For the domain concerning thereference standard, around 44% of studies were considered to beat low risk of bias because the reference standard result was inter-preted blindly. Almost all studies were considered to be at low riskof bias for flow and timing since we could account for all patientsin the 2-by-2 tables. We considered almost all studies to be at low

concern for the three domains (patient selection, index test, andreference standard) addressed by applicability (data not shown).We found similar results for studies assessing DST for second-lineanti-TB drugs: the majority of studies were considered to be athigh risk of bias for patient selection, index test, and referencestandard and low concern for applicability (data not shown).

Diagnostic accuracy of the index test compared with the ref-erence standard. (i) Isoniazid. We identified 110 studies thatevaluated the diagnostic accuracy of DST for isoniazid (for a com-plete list, see Table S1 in the supplemental material). In general,agreement was high (�90%) for all assays. As the critical concen-tration of isoniazid for the novel noncommercial assays has notbeen established, the definition of resistance was not consistentand may have led to variations in diagnostic accuracy. For exam-ple, alamarBlue was evaluated in 12 studies with eight differentcritical concentrations, resazurin was evaluated in 12 studies with

FIG 1 Flow of studies in the review. *, a total of 45 papers evaluated DST for first-line drugs, 12 for second-line drugs, and 51 for both first- and second-linedrugs.

Minireview

February 2013 Volume 51 Number 2 jcm.asm.org 395

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 5

five different critical concentrations, and tetrazolium was evalu-ated in 11 studies using four different critical concentrations.

(ii) Rifampin. We identified 110 studies that evaluated the di-agnostic accuracy of DST for rifampin. Rifampin showed highagreement for all commercial broth-based tests (range, 93.1% to100%). Agreement was moderate to high when all tests were in-cluded (range, 83.9% to 100%). There were several studies withlevels of agreement of less than 90%, including colorimetric indextests (n � 2) and MODS direct testing (n � 1).

(iii) Ethambutol. Among 66 (22.8%) studies that evaluatedethambutol DST, there was a wide range in agreement (57.1% to100%). When studies were limited to those that used a recom-mended critical concentration for the reference standard (n � 57),

the range remained similar. This wide range in agreement wasshared by all methods that evaluated ethambutol DST and indi-cates relatively high variability in ethambutol DST performance.

(iv) Streptomycin. There were 61 studies evaluating strepto-mycin DST. Individual studies found sensitivities rangingfrom 29.0% to 100% and specificities ranging from 54.5% to100%, indicating high variability in the performance of strep-tomycin DST.

(v) Ofloxacin. A total of 16 studies looked at ofloxacin DST.The majority of these found 100% accuracy; the range of sensitiv-ities was 86.3% to 100% and the range of specificities was 85.7%to 100%.

(vi) Other antituberculosis drugs. Concerning pyrazinamide

TABLE 1 Summary of characteristics of included studies

Category

First-line drugs Second-line drugs All drugs

No. of studies(290) % of total

No. of studies(115) % of total

No. of studies(405) % of total

DrugIsoniazid 110 37.9 110 27.2Rifampin 110 37.9 110 27.2Pyrazinamide 4 1.4 4 1.0Ethambutol 66 22.8 66 16.3Amikacin 3 2.6 3 0.7Capreomycin 7 6.1 7 1.7Cycloserine 1 0.9 1 0.2Ethionamide 7 6.1 7 1.7Gatifloxacin 1 0.9 1 0.2Kanamycin 9 7.8 9 2.2Linezolid 2 1.7 2 0.5Moxifloxacin 3 2.6 3 0.5Ofloxacin 16 13.9 16 4.0p-Aminosalicylic acid (PAS) 4 3.5 4 1.0Prothionamide 1 0.9 1 0.2Streptomycin 61 53.0 61 15.1

Index testNitrate reductase assay, solid media 60 20.7 21 18.3 81 20.0MGIT manual 53 18.3 19 16.5 72 17.8MGIT 960 41 14.1 34 29.6 75 18.5Tetrazolium 33 11.4 6 5.2 39 9.6Resazurin 30 10.3 17 14.8 47 11.6Microscopic observation drug susceptibility 29 10.0 5 4.4 34 8.4alamarBlue 28 9.7 7 6.1 35 8.6VersaTREK 6 2.1 1 0.9 7 1.7Nitrate reductase assay, liquid media 5 1.7 1 0.9 6 1.5Löwenstein-Jensen, resistance ratio method 3 1.0 2 1.7 5 1.2Löwenstein-Jensen, absolute concn method 2 0.7 2 1.7 4 1.0

Type of index testDirect 47 16.2 4 3.5 51 12.6Indirect 243 83.8 111 96.5 354 87.4

Type of specimenSputum 52 17.9 5 4.4 57 14.1Isolate, not otherwise specified 238 82.1 110 95.7 348 85.9

Location of laboratory testing by World Bankcountry designation

Low and middle income 162 55.9 44 38.3 206 50.9High income 121 41.7 64 55.7 185 45.7Both low/middle and high income 7 2.4 7 6.1 14 3.5

Minireview

396 jcm.asm.org Journal of Clinical Microbiology

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 6

and second-line drugs other than streptomycin and ofloxacin, thenumber of included studies was limited. Only one to nine studieswere available for each drug, with no more than three studiesevaluating comparable assays (i.e., the same index test, using thesame critical concentration, evaluated in comparison to a refer-ence standard using a recommended critical concentration) (seeTable S2 in the supplemental material).

Meta-analysis. Using only those studies that met our criteriafor meta-analysis, pooled sensitivity, specificity, and agreementestimates for isoniazid, rifampin, ethambutol, streptomycin, andofloxacin are presented in Tables 2, 3, 4, and 5. There were nosubgroups of any given index test with two or more critical con-centrations that met meta-analysis criteria; thus, no direct com-parison between different critical concentrations could be made.

Isoniazid testing by either MGIT manual or MGIT 960 yieldedhigh pooled estimates of agreement (97.3% [95% CI, 95.7 to 98.8]and 98.7% [95% CI, 97.7 to 99.7], respectively) (Table 2). Two ofthe three colorimetric assays, using resazurin and tetrazolium anda critical concentration of 0.2 or 0.25 �g/ml, had high sensitivityand specificity; the sensitivity of alamarBlue was lower at 92.8%(95% CI, 80.2 to 97.6). The NRA on solid media had accuracyusing both indirect and direct methods. MODS using direct meth-ods had an accuracy of 92.9% (95% CI, 88.9 to 96.8%).

For rifampin susceptibility testing using commercial broth-based and noncommercial novel methods, pooled specificity esti-mates were all �98%, as were most pooled sensitivity estimates(Table 3). Exceptions to this included MGIT manual (sensitivity,95.0% [95% CI, 89.2 to 97.8]), tetrazolium (sensitivity, 92.5%[95% CI, 78.2 to 97.7]), and direct testing of both MODS andNRA-solid media (respective sensitivities, 97.9% [95% CI, 85.3 to99.7] and 96.3% [95% CI, 93.6 to 98.1]).

Ethambutol DST using commercial broth tests at the recom-mended critical concentrations of 3.5 �g/ml for MGIT manualand 5 �g/ml for MGIT 960 showed relatively low sensitivities:83.3% (95% CI, 42.0 to 97.2) and 83.9% (95% CI, 72.7 to 91.1),respectively. In comparison, both pooled sensitivity and specific-ity estimates for NRA performed on solid media (using a criticalconcentration of 2 �g/ml) exceeded those of the commercial liq-uid tests (Table 4).

Streptomycin testing by MGIT 960 (critical concentration, 1�g/ml) yielded a high pooled sensitivity estimate (99.7% [95% CI,74.3 to 100]) but only moderate specificity (94.3% [95% CI, 76.7to 98.8]). In comparison, MGIT manual (critical concentration,0.8 �g/ml) was found to have a lower pooled sensitivity (94.1%[95% CI, 81.9 to 98.3]) with a similar pooled specificity (Table 5).Meta-analyzable subgroups were available for MGIT 960 and res-azurin testing for ofloxacin resistance, both of which showed ex-cellent accuracy (Table 5).

Reproducibility. A limited number of studies were included inour systematic review on reproducibility. For first-line drugs,three papers contributed 24 studies for the data on test reproduc-ibility (see Table S3 in the supplemental material) (21–23). Fivestudies assessed the intralaboratory and three studies the inter-laboratory reproducibility of isoniazid DST using 7H10, LJ, MGIT960, and alamarBlue. The agreement within and between sites washigh (90.0% to 100%). Rifampin reproducibility, evaluated usingthe same tests as isoniazid, was excellent within sites (98.7% to100%, 5 studies) and between sites (95.0% to 99.2%, 3 studies).Eight studies evaluated ethambutol reproducibility. Agreementfor studies that evaluated conventional DST was high (97.1 to

TA

BLE

2Ison

iazidD

STm

eta-analyses:pooled

estimates

ofsen

sitivity,specificity,an

dagreem

ent

bytest

and

typeof

specimen

a

Isoniazid

testm

ethod

Indirect

DST

Direct

DST

No.of

studies

(n)%

sensitivity

(95%C

I)%

specificity

(95%C

I)%

agreemen

t(95%

CI)

No.of

studies

(n)%

sensitivity

(95%C

I)%

specificity

(95%C

I)%

agreemen

t(95%

CI)

Com

mercial

MG

ITm

anu

al(CC

�0.1

�g/m

l)15

(1,339)97.1

(92.7–98.9)97.6

(94.5–99.0)97.3

(95.7–98.8)1

(101)M

GIT

960(C

C�

0.1�

g/ml)

10(811)

98.9(94.4–99.8)

98.2(95.4–99.3)

98.7(97.7–99.7)

Non

comm

ercialM

icroscopicO

bservationD

rug

Susceptibility

(MO

DS)

assayb

(CC

�0.1

�g/m

l)2

(84)4

(691)94.4

(90.1–96.9)91.8

(82.9–96.2)92.9

(88.9–96.8)

NR

A,solid

media

(CC

�0.2

�g/m

l)13

(1,587)96.8

(94.6–98.1)100

(95.5–100)99.2

(98.6–99.8)8

(934)97.2

(94.1–98.7)98.5

(96.2–99.4)98.4

(97.3–99.5)alam

arBlu

e(C

C�

0.2or

0.25�

g/ml)

4(263)

92.8(80.2–97.6)

97.1(85.5–99.5)

95.6(90.4–100)

Resazu

rin(C

C�

0.25�

g/ml)

7(603)

99.8(85.9–100)

98.6(95.8–99.6)

99.5(98.6–100)

Tetrazoliu

m(C

C�

0.2or

0.25�

g/ml)

9(748)

98.2(92.7–99.6)

98.8(96.9–99.5)

99.1(98.1–100)

aT

he

models

used

indeterm

inin

gdiagn

ostictest

accuracy

arecalled

“hierarch

icalmodels”

because

they

involve

statisticaldistribution

sat

two

levels.At

the

first

level,they

modelth

evalu

esin

cellsofth

e2-by-2

tablesextracted

fromeach

study

usin

gbin

omialdistribu

tions

and

logistic(log-odds)

transform

ations

ofproportions.A

tth

esecon

d(h

igher)

level,the

models

assum

eran

domstu

dyeffects

toaccou

nt

forh

eterogeneity

indiagn

ostictest

accuracy

between

studies

beyond

that

accoun

tedfor

bysam

pling

variabilityat

the

lower

level.Incases

wh

ereit

was

not

possibleto

sum

marize

the

dataw

ithth

eh

ierarchicalm

odel,we

pooledsen

sitivityan

dspecifi

cityestim

atesseparately.C

C,critical

concen

tration;N

RA

,nitrate

reductase

assay.b

MO

DS

isn

owavailable

inboth

non

comm

ercialand

comm

ercialversions

(Hardy

Diagn

ostics).

Minireview

February 2013 Volume 51 Number 2 jcm.asm.org 397

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 7

98.8) using the 7H10 PM (critical concentration, 5.0; 2 studies)and LJ resistance ratios (100%, 1 study). In the single paper thatevaluated MGIT 960, agreement was 92.2% for intralaboratoryand 97.1% for interlaboratory evaluations. There was only mod-erate agreement (80.3%, 1 study) for alamarBlue for within-labo-ratory reproducibility.

For second-line drugs, five papers were identified that contrib-uted 19 studies on test reproducibility (see Table S3 in the supple-mental material) (21–25). There were only one or two studies foreach second-line drug, except for streptomycin (8 studies). Forstreptomycin, MGIT 960 (critical concentration, 1.0; 2 studies)and the 7H10 proportion method (critical concentration, 2.0; 2studies) yielded agreement of 92.2% to 96.2% and 91.3% to94.0%, respectively.

CONCLUSIONS

We performed a systematic review and meta-analysis of the diag-nostic accuracy of drug susceptibility testing for first-line and sec-ond-line antituberculosis medications. We also collected data onthe reproducibility of included tests. Our findings agree with theliterature in a number of ways, including support for the excellentperformance of commercial broth tests in DST for isoniazid andrifampin. In addition, we found high performance of MODS andNRAs in DST for isoniazid and rifampin resistance.

We identified the following key points.

1. We found poor and variable agreement when testing forethambutol DST for all methods, including MGIT 960 atthe currently recommended critical concentration of 5 �g/ml. This poor agreement appears to be driven by moderatesensitivity of the tests in detecting ethambutol resistance.Although there may be a number of explanations for thispoor performance (e.g., an issue with the reference standard“over-calling” resistance), a reevaluation of the currentlyrecommended ethambutol critical concentrations for theindex tests studied may be warranted. This recommenda-tion is supported by a recent study published by the U.S.Centers for Disease Control and Prevention (CDC), basedon results of a proficiency testing program in which cultureswith known susceptibility or resistance were sent by theCDC to participating laboratories over a 15-year period(26). This study concluded that the ethambutol testing withMGIT is not equivalent to the critical concentrations usedfor the agar proportion methods or Bactec 460. Angra andcolleagues recommended evaluation of a different test con-centration for ethambutol in MGIT.

2. The number of studies evaluating the reproducibility ofDST for first-line and second-line drugs is limited.

3. The colorimetric test results should be interpreted with cau-tion. Several of the included studies either did not deter-mine index test critical concentrations in advance or evalu-ated a range of concentrations that may have been a sourceof bias.

4. The evaluation of the best-performing critical concentra-tion of a test was not the primary objective of this review;aside from ethambutol, we are unable to make recommen-dations regarding evaluations of the currently recom-mended critical concentrations.

This review had several strengths, including a broad search strat-TA

BLE

3R

ifam

pin

DST

met

a-an

alys

es:p

oole

des

tim

ates

ofse

nsi

tivi

ty,s

peci

fici

ty,a

nd

agre

emen

tby

test

and

type

ofsp

ecim

ena

Rif

ampi

nte

stm

eth

od

Indi

rect

DST

Dir

ect

DST

No.

ofst

udi

es(n

)%

sen

siti

vity

(95%

CI)

%sp

ecifi

city

(95%

CI)

%ag

reem

ent

(95%

CI)

No.

ofst

udi

es(n

)%

sen

siti

vity

(95%

CI)

%sp

ecifi

city

(95%

CI)

%ag

reem

ent

(95%

CI)

Com

mer

cial

MG

ITm

anu

al(C

C�

1�

g/m

l)16

(1,4

47)

95.0

(89.

2–97

.8)

100

(95.

1–10

0)99

.0(9

8.3–

99.7

)1

(101

)M

GIT

960

(CC

�1

�g/

ml)

10(8

00)

98.2

(92.

8–99

.6)

99.6

(98.

5–99

.9)

99.5

(98.

6–10

0)1

(222

)

Non

com

mer

cial

Mic

rosc

opic

Obs

erva

tion

Dru

gSu

scep

tibi

lity

(MO

DS)

assa

yb(C

C�

1�

g/m

l)2

(95)

5(8

23)

97.9

(85.

3–99

.7)

98.8

(90.

8–99

.8)

97.5

(94.

9–10

0)

NR

A,s

olid

med

ia(C

C�

40�

g/m

l)15

(1,6

65)

98.2

(95.

4–99

.3)

99.9

(97.

6–10

0)99

.5(9

9.0–

100)

9(1

,200

)96

.3c

(93.

6–98

.1)

99.5

c(9

8.8–

99.9

)99

.4(9

8.7–

100)

alam

arB

lue

(CC

�0.

25�

g/m

l)4

(166

)98

.0c

(89.

4–99

.9)

98.3

c(9

3.9–

99.8

)98

.5(9

5.7–

100)

Res

azu

rin

(CC

�0.

5�

g/m

l)6

(504

)99

.6(8

2.2–

100)

99.5

(97.

6–99

.9)

99.4

(98.

4–10

0)T

etra

zoliu

m(C

C�

1�

g/m

l)5

(544

)92

.5(7

8.2–

97.7

)99

.9(7

4.8–

100)

98.9

(96.

8–10

0)a

Th

em

odel

su

sed

inde

term

inin

gdi

agn

osti

cte

stac

cura

cyar

eca

lled

“hie

rarc

hic

alm

odel

s”be

cau

seth

eyin

volv

est

atis

tica

ldis

trib

uti

ons

attw

ole

vels

.At

the

firs

tle

vel,

they

mod

elth

eva

lues

ince

llsof

the

2-by

-2ta

bles

extr

acte

dfr

omea

chst

udy

usi

ng

bin

omia

ldis

trib

uti

ons

and

logi

stic

(log

-odd

s)tr

ansf

orm

atio

ns

ofpr

opor

tion

s.A

tth

ese

con

d(h

igh

er)

leve

l,th

em

odel

sas

sum

era

ndo

mst

udy

effe

cts

toac

cou

nt

for

het

erog

enei

tyin

diag

nos

tic

test

accu

racy

betw

een

stu

dies

beyo

nd

that

acco

un

ted

for

bysa

mpl

ing

vari

abili

tyat

the

low

erle

vel.

CC

,cri

tica

lcon

cen

trat

ion

;NR

A,n

itra

tere

duct

ase

assa

y.b

MO

DS

isn

owav

aila

ble

inbo

thn

onco

mm

erci

alan

dco

mm

erci

alve

rsio

ns

(Har

dyD

iagn

osti

cs).

cIn

case

sw

her

eit

was

not

poss

ible

tosu

mm

ariz

eth

eda

taw

ith

the

hie

rarc

hic

alm

odel

,we

pool

edse

nsi

tivi

tyan

dsp

ecifi

city

esti

mat

esse

para

tely

.

Minireview

398 jcm.asm.org Journal of Clinical Microbiology

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 8

egy and inclusion of papers in English, French, and Spanish. Allphases of the review, including screening of citations, full-textreview, and data extraction, were done by at least two trainedreviewers working independently. In addition, we used rigorousstatistical methods. To determine pooled accuracy estimates, weused random effects modeling, which provides more conserva-tive estimates than fixed effects modeling when heterogeneity ispresent.

This review was limited by the lack of consensus regarding the“gold standard” for DST. For example, WHO and CDC recom-mendations regarding the preferred solid media for first-line DSThave varied (4, 27), and many experts in the field consider com-mercial broth systems to be equivalent to, or more reliable than,those using solid media. We did not accept genotypic tests (com-mercial or “in-house”) as the reference standard or confirmatorytests for discrepancies based on the search criteria developed withWHO. Two recent studies have raised concerns about phenotypicDST methods for rifampin using the recommended critical con-centrations (28, 29). Van Deun and colleagues reported that cer-

tain conventional drug susceptibility methods did not detectstrains of M. tuberculosis which have rpoB mutations associatedwith small increases in the drug MIC (28). Furthermore, using theXpert MTB/RIF assay and gene sequencing, Williamson and col-leagues identified four patients (three with clinical informationsuggestive of resistance) whose M. tuberculosis isolates containedmutations to the rpoB gene but appeared to be rifampin suscepti-ble using phenotypic methods (29). In the Williamson study, thepresence of these mutations was associated with diminished treat-ment effectiveness or treatment failure. Our findings should becautiously interpreted in light of these emerging data. Furtherclinical observations are required before the significance of rpoBmutations associated with “low level” resistance can be known.

The review was also limited by the small number of studies invarious subgroups, which precluded determination of pooled ac-curacy estimates. Where data were not reported in papers, we didnot contact authors and considered this information not reported(missing). An additional concern is that analyses were often per-formed on a “per sample” basis, meaning that results could be

TABLE 4 Ethambutol DST meta-analyses: pooled estimates of sensitivity, specificity, and agreement by test and type of specimena

Ethambutol test method

Indirect DST Direct DST

No. ofstudies(total n)

% sensitivity(95% CI)

% specificity(95% CI)

% agreement(95% CI)

No. ofstudies(total n)

% sensitivity(95% CI)

% specificity(95% CI)

% agreement(95% CI)

CommercialMGIT manual (CC � 3.5 �g/ml) 9 (890) 83.3 (42.0–97.2) 96.3 (91.2–98.5) 93.1 (89.8–96.5)MGIT 960 (CC � 5 �g/ml) 7 (647) 83.9 (72.7–91.1) 95.8 (80.9–99.2) 95.3 (92.5–98.0) 1 (222)

NoncommercialNRA, solid media (CC � 2 �g/ml) 10 (1,216) 94.3 (89.0–97.1) 99.0 (95.8–99.8) 97.8 (96.5–100) 1 (47)Tetrazolium (CC � 4 �g/ml) 4 (215) 91.9 (77.3–97.4) 91.6 (75.5–97.5) 89.6 (79.5–99.7)

a The models used in determining diagnostic test accuracy are called “hierarchical models” because they involve statistical distributions at two levels. At the first level, they modelthe values in cells of the 2-by-2 tables extracted from each study using binomial distributions and logistic (log-odds) transformations of proportions. At the second (higher) level,the models assume random study effects to account for heterogeneity in diagnostic test accuracy between studies beyond that accounted for by sampling variability at the lowerlevel. CC, critical concentration; NRA, nitrate reductase assay.

TABLE 5 Second-line DST meta-analyses: pooled estimates of sensitivity, specificity, and agreement by test and type of specimena

Drug and test method

Indirect DST Direct DST

No. ofstudies(total n)

% sensitivity(95% CI)

% specificity(95% CI)

% agreement(95% CI)

No. ofstudies(total n)

% sensitivity(95% CI)

% specificity(95% CI)

% agreement(95% CI)

StreptomycinCommercial

MGIT manual (CC � 0.8 �g/ml) 10 94.1 (81.9–98.3) 94.6 (88.3–97.3) 93.0 (89.5–96.6)MGIT 960 (CC � 1 �g/ml) 7 99.7 (74.3–100) 94.3 (76.7–98.8) 96.4 (94.0–98.8)

NoncommercialNRA, solid media (CC � 4 �g/ml) 10 91.2 (81.7–96.1) 97.5 (92.2–99.3) 94.6 (91.9–97.4)Tetrazolium (CC � 1 �g/ml) 4 90.8b (82.7–95.9) 91.4b (85.1–95.6) 92.9 (87.4–98.5)

OfloxacinCommercial

MGIT 960 (CC � 2 �g/ml) 4 99.2 (76.4–100) 99.9 (95.8–100) 100 (99.8–1.00)Noncommercial

Resazurin (CC � 2 �g/ml) 4 100b (91.6–100) 100b (99.0–100) 100 (99.2–100)a The models used in determining diagnostic test accuracy are called “hierarchical models” because they involve statistical distributions at two levels. At the first level, they modelthe values in cells of the 2-by-2 tables extracted from each study using binomial distributions and logistic (log-odds) transformations of proportions. At the second (higher) level,the models assume random study effects to account for heterogeneity in diagnostic test accuracy between studies beyond that accounted for by sampling variability at the lowerlevel. CC, critical concentration.b Data represent values calculated using univariate random effects logistic regression model due to inability of bivariate model to converge.

Minireview

February 2013 Volume 51 Number 2 jcm.asm.org 399

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 9

affected in cases in which several samples were taken from a pa-tient. In addition, we may have missed papers that were publishedin languages that were not included in the review. Finally, inter-pretation of the data is limited in that the majority of studies werecase control or “not reported” in design. Ideally, diagnostic studiesshould employ consecutive or random sampling of eligible pa-tients with the suspected disease or condition to limit the potentialfor bias, although we acknowledge that this is difficult for theevaluation of drug resistance where there are low resistance rates.Data included in this review did not allow formal assessment ofpublication bias using methods such as funnel plots or regressiontests because such techniques are not recommended for diagnos-tic studies (16, 30).

It should be noted that one of the techniques considered to bea reference method in this study, the Bactec radiometric methodusing 12B medium and the 460TB instrument, is no longer avail-able. This method was well regarded and played a critical role inthe development of currently recommended critical concentra-tions. Due to its wide acceptance as an accurate method for DST,we included studies that used 12B radiometric medium as thereference standard in evaluating the accuracy of newer methods,such as MGIT medium, MODS, and colorimetric redox methods.Future diagnostic evaluations are unlikely to have access to thismethod.

In summary, this systematic review provides support for thecritical concentrations recommended for commercial broth sys-tems for DST for isoniazid and rifampin. Further studies areneeded to evaluate critical concentrations for ethambutol andstreptomycin that would accurately detect significant changes insusceptibility to these drugs. Evidence is limited on the perfor-mance of drug susceptibility testing for pyrazinamide and mostsecond-line drugs.

ACKNOWLEDGMENTS

We thank Sherry Dodson and Yuki Durham (University of Washington)and Vittoria Lutje (Cochrane Infectious Diseases Group and LiverpoolSchool of Medicine) for assistance with literature searching.

This work was supported in part by the World Health OrganizationStop TB Department.

WHO had no role in the data collection, data analysis, data interpre-tation, or writing of the manuscript. We had full access to the data and aresolely responsible for the decision to submit these results for publication.

REFERENCES1. Zignol M, van Gemert W, Falzon D, Sismanidis C, Glaziou P, Floyd K,

Raviglione M. 2012. Surveillance of anti-tuberculosis drug resistance inthe world: an updated analysis, 2007–2010. Bull. World Health Organ.90:111–119D.

2. World Health Organization. 2011. Guidelines for the programmaticmanagement of drug-resistant tuberculosis—2011 update. WorldHealth Organization, Geneva, Switzerland. http://whqlibdoc.who.int/publications/2011/9789241501583_eng.pdf. Accessed 25 February2012.

3. World Health Organization. 2011. Global tuberculosis control: WHOreport 2011. WHO/HTM/TB/2010.7. World Health Organization, Ge-neva, Switzerland. http://www.who.int/tb/publications/global_report/2011/gtbr11_full.pdf. Accessed 25 February 2012.

4. World Health Organization. 2008. World Health Organization Policy guidanceon drug-susceptibility testing (DST) of second-line antituberculosis drugs.WHO/HTM/TB/2008.392. World Health Organization, Geneva, Switzerland.http://whqlibdoc.who.int/hq/2008/WHO_HTM_TB_2008.392_eng.pdf. Ac-cessed 25 February 2012.

5. Kim, SJ. 2005. Drug-susceptibility testing in tuberculosis: methods andreliability of results. Eur. Respir. J. 25:564 –569.

6. Kim SJ, Espinal MA, Abe C, Bai GH, Boulahbal F, Fattorin L, Gilpin C,Hoffner S, Kam KM, Martin-Casabona N, Rigouts L, Vincent V. 2004.Is second-line anti-tuberculosis drug susceptibility testing reliable? Int. J.Tuberc. Lung Dis. 8:1157–1158.

7. Bwanga F, Hoffner S, Haile M, Joloba ML. 2009. Direct susceptibilitytesting for multi drug resistant tuberculosis: a meta-analysis. BMC Infect.Dis. 9:67. doi:10.1186/1471-2334-9-67.

8. Chang KC, Yew WW, Chan RC. 2010. Rapid assays for fluoroquinoloneresistance in Mycobacterium tuberculosis: a systematic review and meta-analysis. J. Antimicrob. Chemother. 65:1551–1561.

9. Chang KC, Yew WW, Zhang Y. 2009. A systematic review of rapid drugsusceptibility tests for multidrug-resistant tuberculosis using rifampin re-sistance as a surrogate. Expert Opin. Med. Diagn. 3:99 –122.

10. Ling DI, Zwerling AA, Pai M. 2008. GenoType MTBDR assays for thediagnosis of multidrug-resistant tuberculosis: a meta-analysis. Eur. Respir. J.32:1165–1174.

11. Martin A, Panaiotov S, Portaels F, Hoffner S, Palomino JC, Angeby K.2008. The nitrate reductase assay for the rapid detection of isoniazid andrifampicin resistance in Mycobacterium tuberculosis: a systematic reviewand meta-analysis. J. Antimicrob. Chemother. 62:56 – 64.

12. Martin A, Portaels F, Palomino JC. 2007. Colorimetric redox-indicatormethods for the rapid detection of multidrug resistance in Mycobacte-rium tuberculosis: a systematic review and meta-analysis. J. Antimicrob.Chemother. 59:175–183.

13. Minion J, Leung E, Menzies D, Pai M. 2010. Microscopic-observationdrug susceptibility and thin layer agar assays for the detection of drugresistant tuberculosis: a systematic review and meta-analysis. Lancet In-fect. Dis. 10:688 – 698.

14. Morgan M, Kalantri S, Flores L, Pai M. 2005. A commercial line probeassay for the rapid detection of rifampicin resistance in Mycobacteriumtuberculosis: a systemic review and meta-analysis. BMC Infect. Dis. 5:62.doi:10.1186/1471-2334-5-62.

15. Leeflang MM, Deeks JJ, Gatsonis C, Bossuyt PM. 2008. Systematicreviews of diagnostic test accuracy. Ann. Intern. Med. 149:889 – 897.

16. Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. 2010.Chapter 10: analysing and presenting results. In Deeks JJ, Bossuyt PM,Gatsonis C (ed), Cochrane handbook for systematic reviews of diagnostictest accuracy, version 090. The Cochrane Collaboration, Birmingham,England. http://srdta.cochrane.org/. Accessed 6 March 2011.

17. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, ReitsmaJB, Leeflang MM, Sterne JA, Bossuyt PM. 2011. QUADAS-2: a revisedtool for the quality assessment of diagnostic accuracy studies. Ann. Intern.Med. 155:529 –536.

18. StataCorp. 2009. Stata Statistical Software: release 11. StataCorp, CollegeStation, TX.

19. Zamora J, Abraira V, Muriel A, Khan KS, Coomarasamy A. 2006.Meta-DiSc: a software for meta-analysis of test accuracy data. BMC Med.Res. Methodol. 6:31. doi:10.1186/1471-2288-6-31.

20. Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinder-man AH. 2005. Bivariate analysis of sensitivity and specificity producesinformative summary measures in diagnostic reviews. J. Clin. Epidemiol.58:982–990.

21. Giampaglia CM, Martins MC, Vieira GB, Vinhas SA, Telles MA, PalaciM, Marsico AG, Hadad DJ, Mello FC, Fonseca Lde S, Kritski A. 2007.Multicentre evaluation of an automated BACTEC 960 system for suscep-tibility testing of Mycobacterium tuberculosis. Int. J. Tuberc. Lung Dis.11:986 –991.

22. Laszlo A, Helbecque DM, Tostowaryk W. 1987. Proficiency testing ofconventional drug susceptibility tests of Mycobacterium tuberculosis.Can. J. Microbiol. 33:1064 –1068.

23. Leonard B, Coronel J, Siedner M, Grandjean L, Caviedes L, Navarro P,Gilman RH, Moore DA. 2008. Inter- and intra-assay reproducibility ofmicroplate Alamar blue assay results for isoniazid, rifampicin, ethambu-tol, streptomycin, ciprofloxacin, and capreomycin drug susceptibility test-ing of Mycobacterium tuberculosis. J. Clin. Microbiol. 46:3526 –3529.

24. Lin SY, Desmond E, Bonato D, Gross W, Siddiqi S. 2009. Multicenterevaluation of Bactec MGIT 960 system for second-line drug susceptibilitytesting of Mycobacterium tuberculosis complex. J. Clin. Microbiol. 47:3630 –3634.

25. Rüsch-Gerdes S, Pfyffer GE, Casal M, Chadwick M, Siddiqi S. 2006.Multicenter laboratory validation of the BACTEC MGIT 960 techniquefor testing susceptibilities of Mycobacterium tuberculosis to classical sec-ond-line drugs and newer antimicrobials. J. Clin. Microbiol. 44:688 – 692.

Minireview

400 jcm.asm.org Journal of Clinical Microbiology

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from

Page 10

26. Angra PK, Taylor TH, Iademarco MF, Metchock B, Astles JR,Ridderhof JC. 2012. Performance of tuberculosis drug susceptibilitytesting in U.S. laboratories from 1994 to 2008. J. Clin. Microbiol. 50:1233–1239.

27. Kent PT, Kubica GP. 1985. Public health mycobacteriology: a guide forthe level III laboratory. U.S. Department of Health and Human Services,Atlanta, GA.

28. Van Deun A, Barrera L, Bastian I, Fattorini L, Hoffmann H, Kam KM,Rigouts L, Rusch-Gerdes S, Wright A. 2009. Mycobacterium tuberculo-

sis strains with highly discordant rifampin susceptibility test results. J.Clin. Microbiol. 47:3501–3506.

29. Williamson DA, Roberts SA, Bower JE, Vaughan R, Newton S, Lowe O,Lewis CA, Freeman JT. 2012. Clinical failures associated with rpoB mu-tations in phenotypically occult multidrug-resistant Mycobacterium tu-berculosis. Int. J. Tuberc. Lung Dis. 16:216 –220.

30. Tatsioni A, Zarin DA, Aronson N, Samson DJ, Flamm CR, Schmid C,Lau J. 2005. Challenges in systematic reviews of diagnostic technologies.Ann. Intern. Med. 142(12 Pt 2):1048 –1055.

Minireview

February 2013 Volume 51 Number 2 jcm.asm.org 401

on June 17, 2013 by Shou-Y

ean Linhttp://jcm

.asm.org/

Dow

nloaded from