Conducting flawless research – the right question, the right design Dr. Dick Menzies July 10 th, 2015.

Conducting flawless research – the right question, the right design

Dr. Dick Menzies

July 10th, 2015

The starting point in Research – a Problem• An unusual clinical observation

– An unusual cluster (Outbreak of XDR in S Africa)– An unusual complication (Acquired Rifamycin resistance)

• An observation from reported data– Temporal trend (Increase in TB in Foreign-born)– Geographic clustering (TB in hospitals)

• Problems in diagnosis– Delayed diagnosis (cultures, DST)– Missed diagnosis (AFB smears in HIV infected)

• Problems in treatment:– Treatment too long (Standard “short-course” therapy)– Treatment too toxic (INH)– Treatment too long AND too toxic (MDR, 9INH)

An unusual clinical observation – exampleExtensively drug-resistant tuberculosis as a cause of death

in patients with TB and HIV in rural South AfricaNeel R Gandhi, Anthony Moll, A Willem Sturm, Robert Pawinski,

Thiloshini Govender, Umesh Lalloo, Kimberly Zeller, Jason Andrews,Gerald Friedland

• Among 475 patients with culture-confirmed TB: – 39% (185 patients) MDR-TB– 6% (30 patients) XDR-TB.

• Of XDR-TB:– 45% had been previously treated for TB ; – 67% had a recent hospital admission. – 100% of tested for HIV were co-infected. – 98% died, with median survival of 16 days

From clinical problem (XDR outbreak) to Research question

• Questions arising:

• How (and why) did the patients acquire XDR?

• Why the association with HIV co-infection?

• Why the association with hospitalization?

• Why was mortality so high?

• Why was mortality so rapid?

• Each of these questions needs a separate study, and unique design - to be answered.

• WHICH question should be answered?

• IS this the right problem?

Choosing the right study question

Step 1: Definition of the problem– Who gets the problem,

– How important is it - does it affect death? Costs? Quality of life?

Step 2: Understanding the biology?• Manifestations - what organs are affected

• Pathogenesis - probable or know

• Transmission/hosts/reservoirs

Step 3: make the question as precise as possible – (PICO or pseudo-PICO format)

Choosing the right study question“That year in the lab saved me a day in the library”

Step 4. Review the literature (again) (and again)– This is often under-utilized - can avoid wasted time– Avoid repeating errors others have made– Do not study a problem that has been well described.

• RCT of FDCs (have been 15), • Comparing positive TST vs IGRA (have been hundreds)

– Know what has been described• Effect of standard treatment• Accuracy of standard diagnostics

Step 5. Get the opinions of others– Speak to experts (or pseudo-experts) – Speak to providers– Speak to patients, community members, others affected

From Research qst to study design

The biggest determinants of study design:

1) Time – investigators’ and patients’

2) Money (“The biggest determinant of sample size is the budget”)

3) Resources – what do you have to work with? (Diagnostic Laboratory, Molecular Epi, Molecular Biology, Genetics)

4) Collaborators – who do you have to work with? What can they do?

These 4 factors determine the kinds of studies you can do – which determines the question you can answer.

Hierarchy of study designs • Descriptive studies

• Case reports, and case series• Reported data, • Prevalence surveys

• Analytic studies– Observational

• Cross-sectional • Ecologic studies• Case control studies• Cohort studies

– Experimental studies• Uncontrolled trials• Randomized controlled trials

– Individual level– Field trials = group level

Incre

as

ing

cos

ts a

nd

c

om

plex

ity

Reported Data

• Useful to:– Define incidence/prevalence

– Identify geographic or temporal differences

– Describe clinical characteristics

– Describe outcomes.

• Implicit comparison with general population – Risk factors can be identified.

– Useful if data is COMPLETE and ACCURATE

0

20

40

60

80

100

120

Years

TB

inci

den

ce (

per

100

,000

ha.

)

0

10

20

30

40

50

60

To

tal r

esid

enti

al c

oal

co

nsu

mp

tio

n

(Mill

ion

sh

ort

to

ns)

Coal use ( ) and TB incidence ( ) .

Temporal trends may indicate clues to causal exposure Coal use and TB in USA: 1953 – 2003

05

101520253035404550556065707580859095

100

1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004

Years

TB

no

tific

atio

n r

ate

(per

100

,000

ha.

)

0

5

10

15

20

25

30

35

To

tal c

oal

co

nsu

mp

tion

(e

xajo

ule

s)

Total coal combustion ( ) Notified cases of TB ( ).

Temporal trends may indicate clues to causal exposure Coal use and TB in China: 1978 - 2004

Advantages• Usually very easy and quick studies

• Take advantage of already gathered data– Exposures

– Diseases

Disadvantages• Relationship may be due to completely unmeasured

factors

• VERY substantial potential for confounding

Ecologic Studies

Directionality in research

• Retrospective: Start with persons with disease and ‘look backward’ in time to ascertain exposures.

– Advantages: Biggest is convenience – do not have to wait a long time, because disease HAS happened. Makes these studies much quicker to complete, and much cheaper.

• Prospective: Start with a population and observe them ‘going forward’ in time. Exposures are measured first, and health events are measured afterward, as they occur.

– Advantages: Biggest is accuracy of exposure, and certainty that exposure precedes disease.

General approach:

Measure disease(s) and exposure(s) all at once in a population.

Estimate effect: Prevalence odds ratio

Advantages• Good for common/chronic diseases • Good for common exposures• Allows one to measure multiple disease or conditions and multiple

exposures

Disadvantages• Measurement of exposure may be difficult

– Recall problems if long latency– May change over time (Alcohol, smoking, blood pressure)

• Can not distinguish cause and effect – (Tobacco Industry defense)

Cross-sectional or Prevalence Studies

Tuberculin (or IGRA) SurveysA special type of cross-sectional survey• Once TST or IGRA convert to positive with TB infection –

they remain positive lifelong (some exceptions)• Cross-sectional survey – detects all with positive tests –

from recent or remote• From prevalence of positive test at a given age – can

calculate average annual risk of TB infection. • Can compare prevalence in different populations• If different ages, or different exposure periods can

estimate trends in infection

University students in Brazil(All BCG vaccinated in infancy)

Silva et.al. IJTLD2000; 4:420-426

02468

1012141618

Early 0-2 Intermediate3-4

Senior 5-6

Years of training

% o

f R

eact

ion

Engineering StudentsMedical Students

Average ARI: Preclinical 0.2% Clinical 2.9%Incidence of TB in Brazil: 75/100 000

Example of Kaplan-Meier analysis: General Hospital Ventilation and time to TST conversion

General Approach:•Measure exposures retrospectively in persons with disease (cases) and without disease (controls)•Estimate odds of exposure for disease (Odds ratio)

Advantages•Relatively cheap and quick•Particularly useful for studying rare conditions

– Or conditions with long latency

Disadvantages•Controls, Controls, Controls

– Very difficult to select proper controls– This is the source of most problems in case control studies– And is why they are generally considered weak evidence.

•Difficulties of retrospective exposure assessment – particularly if long latency

Case Control Studies

Fitness costs of drug-resistance mutations in MDRTB: a household-based case-control study.

Salvatore…. and Cohen T, JID 2015

• Cases: MDR index patients with at least 1 other HH member with same strain

• Controls: MDR but no-one in HH with same strain• Matching:

– Same number of contacts in House-Hold– Same extent of Drug Resistance– Same time to detection

• Not matched: Extent of disease, Cavitation, smear, duration of symptoms, prior episodes, Size of house

• Findings: katG Ser315Thr mutation associated (OR: 2.39; 95% CI: 1.21- 4.70). BUT combination of katG Ser315Thr & rpsL-Lys43Arg mutations was protective.

Nested Case Control study – exampleTB outbreak in Inuit village

Behr, Fox, Khan, Lee, Menzies & many others

• Major ‘outbreak’ of TB in small village in N Quebec

• 695/940 residents investigated as contacts during outbreak

– Newly infected

– 50 Confirmed disease, 19 probable disease

• Case control study of housing, nutrition, lifestyle determinants

– “Nested” within all 695 investigated as contacts

• Case-control1: New Infection

– Cases=Newly infected, Controls = Not infected

• Case-control2: Disease

– Cases=Confirmed or Probable, Controls = Infected but no disease

General approach:

• Start with a population free of disease. Measure exposures, and follow them all.

• Estimate of effect: Risk ratio (Risk disease in Exposed/unexposed)

Advantages:

• Can measure many exposures, and many diseases

• Temporal relationship clearer (cause before disease)

Disadvantages

• Long and expensive (often very $$$)

• Good for common diseases (some cancers, cardiovascular).

• But not for rare diseases or long latency

• Also what if you fail to measure key determinants

– (Solution = freezer)

Cohort Studies

Average Annual Incidence of Tuberculosis Among Navy Recruits Average Annual Incidence of Tuberculosis Among Navy Recruits By History of Household ContactBy History of Household Contact

Comstock, Edwards, and Livesay; Am Rev Respir Dis 1974; 110:576

Country N Test Incidence of active TB in IGRA+ groups

The Gambia [Hill et al. 2008]

2348 ELISPOT (in-house)

9/1000 person-yr

Turkey [Bakir et al. 2008]

908 ELISPOT (T-SPOT.TB)

21/1000 person-yr

S Africa [Mahomed et al. 2009 (abstract)]

5248 QFT 6/1000 person-yr

Colombia [del Corral et al. 2009]

2060 In-house whole-blood CFP-10 assay

11/1000 person-yr

Senegal [Lienhardt et al. PLoS One 2010]

2679 ELISPOT (in-house)

14/1000 person-yr

Advantages• Best method to evaluate an intervention • Best control of bias and confoundingDisadvantages• Not easy or feasible for all interventions• Very Expensive and long• Not for studies of risk factors or natural history• Substantial refusal or drop-out rates can restrict

generalizability • Population selected may not be representative

– Younger adults with only one condition– Often exclude pregnant woman, kids, elderly!

Solution – Biomarker?– 2 month culture conversion instead of fail/relapse

Randomized Trials

Five Phase 2 trials: 2 month culture conversion with FQN(all patients received 4 months HR in continuation)

AuthorYear

Patients(N)

FQN regimen

Control Regimen

2 Month Culture Conversion

FQN Control

Burman 2006

277 HRZM HRZE 71% 71%

Rustomjee 2008

205 HRZG/M HRZE 79%/42% 64%/36%

Conde 2009

146 HRZM HRZE 80% 63%

Dorman 2009

328 MRZE HRZE 60% 55%

Wang 2010

123 HRZEM HRZE 90% 78%

Three Phase 3 trials of 4 month regimens with FQN

Study Pts.(N)

FQN – 4 months Standard - 6 months

Cure Death Fail/ Relapse

Cure Death Fail/ Relapse

OFLUTB 1585 79% 1% 16% 83% 1% 10%

Remox 1900 83% 1% 11% 92% 1% 3%

Rifaquin 592 82% 0 18% 95% 1% 4%

Experimental Community or Field Trials

General Design

• Pick an intervention to be applied at a community level– Fluoride in water, public education, vaccination

• Find several communities or population groups

• Apply intervention to some and not others – Randomly again

• Measure outcomes at population or group level

Cluster randomized Trials – exampleEffect of improved tuberculosis screening and isoniazid

preventive therapy on incidence of tuberculosis and death inpatients with HIV in clinics in Rio de Janeiro, Brazil: a stepped

wedge, cluster-randomised trialBetina Durovni, Valeria Saraceni, Lawrence H Moulton, Antonio G Pacheco,

Solange C Cavalcante, Bonnie S King, Silvia Cohn, Anne Efron,Richard E Chaisson, Jonathan E Golub

• 29 HIV clinics in Rio de Janeiro. • Staff trained in TB screening, TST and INH.• Clinics randomly allocated to date of starting the

intervention. 2 clinics started every 2 months starting from Sept 2005, until Aug 2009

• Outcome: TB incidence +/- death

A randomized trial to compare 4 months Rifampin vs 9 months

INH for the treatment of LTBI

Phase 1: Compliance and completion

Completed in 2003

Phase 2 – Adverse events and costsCompleted in 2007

Phase 3: Efficacy and effectiveness

Enrolment completed in 2014: 840 children, 6020 adults

Follow-up will finish in 2016

Publication in ???

Thanks