How to Analyze Therapy in the Medical Literature (part 1) Akbar Soltani. MD.MSc Tehran University of Medical Sciences (TUMS) Shariati Hospital www.soltaniebm.com
Dec 14, 2015
How to Analyze Therapy in the Medical Literature
(part 1)
Akbar Soltani. MD.MScTehran University of Medical Sciences (TUMS)
Shariati Hospitalwww.soltaniebm.com
Objectives
• What is randomized controlled trial?• How to appraise RCT using standard
checklist• Review different concepts such as
– Randomization, allocation concealment, blinding, loss to follow up, intention to treat analysis
The hierarchy of evidence
• Systematic reviews and meta-analyses • Randomised controlled trials • Cohort studies • Case-control studies • Cross sectional surveys • Case reports• Expert opinion
Three Step Guide in Using an Article to Assess Therapy
1. Are the results of the study valid?
2. What are the results? What measures of precision of effects were reported (CIs, p-values)?
3. How can I apply these results to patient care?
Randomized control trial design
Assess Validity and Applicability to my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?
Benefits of Random Allocation (Randomization)
1.Reduces bias in those selected for treatment– guarantees treatment assignment will
not be based on patients’ prognosis
2.Prevents confounding– known and unknown potential
confounders are evenly distributed
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?
Ensuring Allocation Concealment
BEST – most valid technique
Central computer randomization
DOUBTFUL Envelopes, etc
NOT RANDOMIZED Date of birth, alternate days,
etc
Do Not Confuse Allocation Concealment with Blinding
• Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented
Do Not Confuse Allocation Concealment with Blinding (Cont’d)
• Blinding seeks to prevent information bias, protects sequence after allocation, and cannot always be successfully implemented
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to
treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?
Placebo effect Trial in patients with chronic severe itching
0
10
20
30
40
50
60
I tching scoreCyproheptadine
HCL
Trimeprazine tartrate
No treatment
Treatment vs no treatment for itching
Placebo effect Trial in patients with chronic severe itching
0
10
20
30
40
50
60
I tching scoreCyproheptadine
HCL
Trimeprazine tartrate
Placebo
No treatment
Treatment vs no treatment vs placebo for itching
Placebo effect - attributable to the expectation that the treatment will have an effect
Blinding and Reporting
• Usually reduces differential assessment of outcomes (information bias)
• Authors should explicitly state who was blinded – and how.
• Many investigators and readers consider a randomized trial as high quality simply because it is “double-blind,” as if double-blinding is the sine qua non of an RCT.
– Trials not “double-blinded” should not automatically be deemed inferior trials.
Blinding in randomised trials: hiding who got what
• Double blinding prevents bias but is less important, than adequate allocation concealment.
• open studies are more likely to favour experimental interventions over the controls and that studies that are not double-blinded can exaggerate effect estimates by 17%
• Furthermore, in some trials, blinding cannot be successfully implemented, whereas allocation concealment can always be successfully implemented.
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?
Significance tests for baseline differences
Baseline data
AzathioprineAzathioprine PlaceboPlacebo
Mean ageMean age 54.754.7 54.954.9
Serum bilirubin (Serum bilirubin (mol/L)mol/L) 37.237.2 30.930.9
Stage I disease %Stage I disease % 1414 1212
Stage II disease %Stage II disease % 4444 4343
Stage III disease %Stage III disease % 1515 1515
Stage IV disease %Stage IV disease % 2727 3030
Christensen et al. Gastro 1985
Effect of azathioprine on the survival of patients with primary biliary cirrhosis
Baseline data
AzathioprineAzathioprine PlaceboPlacebo
Mean ageMean age 54.754.7 54.954.9
Serum bilirubin (Serum bilirubin (mol/L)mol/L) 37.237.2 30.930.9
Stage I disease %Stage I disease % 1414 1212
Stage II disease %Stage II disease % 4444 4343
Stage III disease %Stage III disease % 1515 1515
Stage IV disease %Stage IV disease % 2727 3030
Christensen et al. Gastro 1985
Effect of azathioprine on the survival of patients with primary biliary cirrhosis
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?
How complete was the follow up? How many dropouts were there?
• Conventionally, a 20% drop out rate is regarded as acceptable, but this depends on the study question.
• Some regard should be paid to why participants dropped out, as well as how many.
• It should be noted that the drop out rate may be expected to be higher in studies conducted over a long period of time.
• A higher drop out rate will normally lead to downgrading, rather than rejection of a study.
Over-estimation oftreatment effect
Bias: a one-sided inclination of the mind
• Not random 40%
• Not double-blind 17%
• Duplicate information 20%
• Small trials 30%
• Poor reporting quality 25%
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?were randomized (“intention-to-treat” analysis)?
Montorri V, Guyatt G. CMAJ 2001 165(10) p1340
Intention to treat
Montorri V, Guyatt G. CMAJ 2001 165(10) p1340
Intention to treat
Montorri V, Guyatt G. CMAJ 2001 165(10) p1340
Intention to treat
High risk?
Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting
1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and
randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to
treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the
start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they
were randomized (“intention-to-treat” analysis)?