Statistics for the Interventionist Gregory J. Dehmer, MD Professor of Medicine, Texas A&M College of Medicine Director, Cardiology Division Scott & White.

Statistics for the InterventionistStatistics for the InterventionistGregory J. Dehmer, MD

Professor of Medicine, Texas A&M College of Medicine

Director, Cardiology DivisionScott & White Clinic

Gregory J. Dehmer, MDProfessor of Medicine, Texas A&M College of

MedicineDirector, Cardiology Division

Scott & White Clinic

Statistics means never having to say your certain

SCAI Interventional Cardiology Fellows Course2007

Gregory J. Dehmer, MD, FSCAI

I have no relevant financial disclosures to make.

Expectations - HypothesisExpectations - Hypothesis

• You will completely understand statistics at the end of this brief talk

• This is inherently boring materialand I can’t fix that

• Provide some comments about studies and statistics that I hope are helpful

• You will completely understand statistics at the end of this brief talk

• This is inherently boring materialand I can’t fix that

• Provide some comments about studies and statistics that I hope are helpful

p=1.0

p<0.000000001

Some Things Are Just Really Bad IdeasSome Things Are Just Really Bad Ideas

Another Questionable CombinationAnother Questionable Combination

Hazardous equipment – Don’t operate unless you know what you are doing

+

The busy interventional cardiologist Statistics-in-a- Box“SYSTAT has every statistical

procedure you need”

You Will Be ReviewedYou Will Be Reviewed

• Many papers now undergo formal statistical review• Many papers now undergo formal statistical review

StatisticalConsultants

NEJMCirculationCirc Res

JACCetc . . .

Evolution of EvidenceEvolution of Evidence

Primary Evidence

Randomized

controlled trial

Observational studies

Uncontrolled trials

Descriptive studies

Case reports

Synthesized quantitativeData (meta-analyses)

Systematic reviews

Summary reviews

Opinions of respectedauthorities

Secondary Evidence

Evolution of EvidenceEvolution of Evidence

Primary Evidence

Randomized

controlled trial

Observational studies

Uncontrolled trials

Descriptive studies

Case reports

• Issues related to RTCs

– Exclusions

– Missing data

– Power calculations

– Confidence intervals

– Confusing endpoints “Non-inferiority”

• Issues related to RTCs

– Exclusions

– Missing data

– Power calculations

– Confidence intervals

– Confusing endpoints “Non-inferiority”

• Exclusion of cases is a major weakness (during analysis)

– Most common reason is a desire to ensure that all patients are “adequately treated”

– Awkward to retain a patient in the analysis who died during the 1st week of therapy or were unwilling to stick with the therapy

• Exclusion of cases is a major weakness (during analysis)

– Most common reason is a desire to ensure that all patients are “adequately treated”

– Awkward to retain a patient in the analysis who died during the 1st week of therapy or were unwilling to stick with the therapy

RTCs Problem 1: ExclusionsRTCs Problem 1: Exclusions

RTCs Problem 1: ExclusionsRTCs Problem 1: Exclusions

Exclusions more likely in: - the “aggressive treatment” arm or the “high-risk” group

• Likely to have more non-adherent patients

• Non-adherent patients are a higher risk group

• Exclusion of high-risk patients improved the average of the remaining patients

• If exclusions are permitted, the more aggressive arm appears artificially better

• Likely to have more non-adherent patients

• Non-adherent patients are a higher risk group

• Exclusion of high-risk patients improved the average of the remaining patients

• If exclusions are permitted, the more aggressive arm appears artificially better

Standard Therapy

More aggressive than standard therapyR

RTCs – The Problem With ExclusionsRTCs – The Problem With Exclusions

Ideally, data analysis should look forward from randomization

R

Good Events Good Events

Bad Events Side-Effects

These are the data (warts and all) that the clinician needs to know to assess what will happen to their patient if this therapy is selected

RTCs – The Problem With ExclusionsRTCs – The Problem With Exclusions

Data analysis when cases of inadequate treatment are excluded it like looking backward through rose-colored glasses

R

Good Events Good Events

Bad Events Side-Effects

Data Analysis

Excluding bad events and focusing only on the good results of the remaining cases may look impressive, but is not of practical value to clinicians who need to make

prospective therapy decisions for their patients.

RTC Lesson 1 – Avoid ExclusionsRTC Lesson 1 – Avoid Exclusions

Beware of potential bias

Check to see if the size of the analyzed groups are similar

R

n = 4932 n = 4931

n = 4100

Standard therapy New therapy

n = 4932

RTCs Problem 2: Missing DataRTCs Problem 2: Missing Data

45

50

55

60

65

70

Baseline 6 months 12 months 18 months 24 months

45

50

55

60

65

70

Baseline 6 months 12 months 18 months 24 months

LVE

F

Missing random data weakens the study, but is not a serious concern

However when data are missing because of aspects of treatment or disease, major bias can arise. Patients missing outcomes observations are more likely those with poor outcomes

# of patients: 200 120 50

Higher-risk ptsdon’t tolerate the therapy, drop out

leaving the low-riskpts who naturallyhave higher EFs

RTC Problem 2 - Beware of Missing DataRTC Problem 2 - Beware of Missing Data

•Make every effort to have data values at all key time points

– Can use “imputed values”

Carry previous measure forward

Inserting a conservative value

Averaging adjacent values

Computer models which use similar patients with complete information

•Make every effort to have data values at all key time points

– Can use “imputed values”

Carry previous measure forward

Inserting a conservative value

Averaging adjacent values

Computer models which use similar patients with complete information

RTC Problem 2 - Beware of Missing DataRTC Problem 2 - Beware of Missing Data

• Sensitivity analysis: determines the impact of the missing data and the imputation method used.

– If the results are qualitatively similar, one can deduce that the basic study conclusion does not depend on the type of imputation used (or the use of imputation).

• Sensitivity analysis: determines the impact of the missing data and the imputation method used.

– If the results are qualitatively similar, one can deduce that the basic study conclusion does not depend on the type of imputation used (or the use of imputation).

Treatment 1 Treatment 2

Replace missing values with a conservative

imputation

Replace missing values with an anti-conservative

imputation

Analyze and flip-flopimputation strategy

RTC Lesson 2 – Avoid Missing Data

RTC Lesson 2 – Avoid Missing Data

Rule of ThumbIf the proportion of cases excluded or with missing data in the size

of the treatment difference reported,the study is likely unreliable

Consolidated Standards for Reporting Trials

Lancet. 2001;357:1191–1194

RTCs Problem 3: Power CalculationsRTCs Problem 3: Power Calculations

Power calculation

• Determine what is a clinically meaningful difference between the two groups. (10%)

Would anyone care if the difference in restenosis was 2%?

• Amount of variation in the measurement of the endpoint (standard deviation)

• One-tailed (sided) test– Used when previous data,

physical limitations or common sense tells you that the difference, if any, can only go in one direction

– Example: Contrast effect on renal function

• Two-tailed (sided) test– Used when the difference,

if any, can go in either direction.

– Example: Drug effect on serum K+

RTCs Problem 3: Understanding PowerRTCs Problem 3: Understanding Power

Power Calculation:

- Null hypothesis %RS (control) - %RS (treatment)

= zero (0) If you reject the null hypothesis then you are saying there is a difference between the two - = threshold of significance typically ≤ 0.05 (5%) If you reject the null hypothesis when it is actually true Type I error There is a ≤ 5% chance that there no difference, but your analysis concludes there is Probability of a Type I error =

Power Calculation:

- Null hypothesis %RS (control) - %RS (treatment)

= zero (0) If you reject the null hypothesis then you are saying there is a difference between the two - = threshold of significance typically ≤ 0.05 (5%) If you reject the null hypothesis when it is actually true Type I error There is a ≤ 5% chance that there no difference, but your analysis concludes there is Probability of a Type I error =

RTCs Problem 3: Understanding PowerRTCs Problem 3: Understanding Power

Power Calculation:- = threshold of significance typically ≤ 0.05

Saying there is a difference when there is none

- = the level you are willing to accept for the chance of missing an important difference when there really is one (20%) (Type II error)

Accepting the null hypothesis when it is, in fact, false

Power = 1 - 1 – 0.20 = 0.80 or 80%

RTCs Problem 3: Understanding PowerRTCs Problem 3: Understanding Power

RTC Lesson 3 – Know What Power MeansRTC Lesson 3 – Know What Power Means

RTCs Problem 4: Understanding CIsRTCs Problem 4: Understanding CIs

• Standard deviation– Relates to one data set

Fasting cholesterol of everyone in this room

Mean (average) SD is an expression of how

much spread there is around the mean value

• Equation for SD

• Standard deviation– Relates to one data set

Fasting cholesterol of everyone in this room

Mean (average) SD is an expression of how

much spread there is around the mean value

• Equation for SD SD mark the limits of scatter

Approximately 68% are within 1 SDApproximately 95% are within 2 SD

RTCs Problem 4: Understanding CIsRTCs Problem 4: Understanding CIs

• Confidence intervals

– Relate to populations (consider this room a population)

– Measure cholesterol in a sample of the population (n = 10)

– How well does the sample mean represent the population mean?

– 95% CI tells you that the mean of the population has a 95% chance (19 out of 20 times) of being within the range of the sample mean

• Confidence intervals

– Relate to populations (consider this room a population)

– Measure cholesterol in a sample of the population (n = 10)

– How well does the sample mean represent the population mean?

– 95% CI tells you that the mean of the population has a 95% chance (19 out of 20 times) of being within the range of the sample mean

RTCs Problem 4: Understanding CIsRTCs Problem 4: Understanding CIs

RTCs Lesson 4: Know Your CIsRTCs Lesson 4: Know Your CIs

• Confidence intervals– Each sample has a mean

and SD

– SEM = SD/ √n

– The 95% CI is ± 1.96 x SEM

– There is only a 5% chance that this range of values excludes the true population mean value

• Confidence intervals– Each sample has a mean

and SD

– SEM = SD/ √n

– The 95% CI is ± 1.96 x SEM

– There is only a 5% chance that this range of values excludes the true population mean value

0.0 0.3 0.5 0.8 1.0 1.3 1.5 1.8 2.0

#1

#2

#3

#4

Variable

RTCs Lesson 5: OR & RR Are Not the Same

RTCs Lesson 5: OR & RR Are Not the Same

The PURSUIT Trial

The primary endpoint (composite of death or MI at 30 days) was comparedin patients receiving eptifibatide vs. placebo

Eptifibatide group: 672 out of 4722 reached the primary endpointPlacebo group: 745 out of 4739 reached the primary endpoint

Odds Ratio

Odds in E: 672 / 4050 = 0.166

Odds in P: 745 / 3994 = 0.187

Odds ratio: 0.166 / 0.187 = 0.899

Risk Ratio

Odds in E: 672 / 4722 = 0.142

Odds in P: 745 / 4739 = 0.157

Odds ratio: 0.142 / 0.157 = 0.905

The PURSUIT Investigators: NEJM 1998;339:436-443

There is a separate, but similar mechanism for calculating CI for ORs and RRs

RTCs Problem 5: Ratio ConfusionRTCs Problem 5: Ratio Confusion

Relationship between ORs and RRs for studies assessing harm

Each line on the graph relates to a different baseline prevalence, or event rate in the control group

When the prevalence of the event is low, say 1%, the RR is a good approximation of the OR For example, when the OR is 10, the RR is 9, an error of 10%

We can use this graph to get a grasp of how misleading it could be to interpret ORs as if they were RRs.

Rel

ativ

e R

isk

Odds Ratio

RTCs Problem 5: Ratio ConfusionRTCs Problem 5: Ratio ConfusionR

elat

ive

Ris

k

Odds Ratio

Relationship between OR and RR for studies which are assessing benefit

Each line on the graph relates to a different baseline prevalence, or event rate in the control group

When event rates are very low the approximation is close, but breaks down as event rates increase

For example, if the event rate is 50% and there is a 20% reduction in the odds, the relative risk adjustment will be little over 10%

RTCs Problem 6: Confusing Endpoint RTCs Problem 6: Confusing Endpoint

• Superiority trial: Designed to test for a statistically significant and clinically meaningful improvement (or harm) from the use of the experimental treatment over the usual care.

• Superiority trial: Designed to test for a statistically significant and clinically meaningful improvement (or harm) from the use of the experimental treatment over the usual care.

0.0 0.3 0.5 0.8 1.0 1.3 1.5 1.8 2.0

#1

#2

#3

#4

Experimental Treatment Better

Control Treatment Better

Superior

Superior

Superior

Not different

RTCs Problem 6: Confusing Endpoint RTCs Problem 6: Confusing Endpoint

• Equivalence trial: Evaluates whether the difference in outcome for the experimental treatment compared with standard care falls within the boundary of a clinically-defined minimally important difference (MID)

• Equivalence trial: Evaluates whether the difference in outcome for the experimental treatment compared with standard care falls within the boundary of a clinically-defined minimally important difference (MID)

0.0 0.3 0.5 0.8 1.0 1.3 1.5 1.8 2.0

#1

#2

#3

#4

Experimental Treatment Better

Control Treatment Better

MID

Clinically and statistically equivalent

Neither clinically nor statistically equivalent

Statistically equivalent

Statistically equivalent

Had these come from a

superiority trial they would be

clinically equivalent, but

statistically inferior (#2) or superior (#1)

RTCs Problem 6: Confusing Endpoint RTCs Problem 6: Confusing Endpoint

• Noninferiority trial: Results are evaluated assuming that the experimental treatment is not worse than the standard treatment by a clinically-meaningful amount.

• Noninferiority trial: Results are evaluated assuming that the experimental treatment is not worse than the standard treatment by a clinically-meaningful amount.

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

#1

#2

#3

#4

Experimental Treatment Better

Control Treatment Better

Not inferior

Not inferior

CI does not crossthe MID

CI too wide for any conclusions

CI crosses MIDindicating

inferiority of the experimental Rx

MID

ResourcesResources

• http://www.jr2.ox.ac.uk/bandolier/

• http://www.statsoft.com/textbook/stathome.html

• http://www.bettycjung.net/Statsites.htm

• http://www.tufts.edu/~gdallal/bmj.htm– Link to Br Med J series of papers on statistics

• 2006-2007 Circulation series “Statistical Primer for Cardiovascular Research”

• Motulsky H. Intutitive Statistics. Oxford University Press 1995

A statistician is a person who comes to the rescue of figures that cannot lie for themselves

RememberRemember

Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital