MDM Review 2009 12.14.09 Jason Sanders. Outline Measures of frequency Measures of association Study designs INTERMISSION Threats to study validity Defining.

MDM Review 2009

12.14.09Jason Sanders

Outline

• Measures of frequency• Measures of association• Study designs• INTERMISSION• Threats to study validity• Defining test and study utility• Descriptive statistics• Q and A

Measures of disease frequency

• Incidence (risk, cumulative incidence, incidence proportion)

I = # new cases of disease during time period # subjects followed for time period

Important points: only new cases counted in numerator; time period must be specified

Benefits: easy to calculate and interpretDrawback: competing risks make I inaccurate over long time

periods

Measures of disease frequency

• Incidence rate (rate)

R = # new cases of disease during time period total time experienced by followed subjects

Important points: only new cases counted in numerator; person time summed for each individual

Benefits: accounts for competing risksDrawback: not as easy to interpret

Measures of disease frequency

• Prevalence (prevalence proportion)

P = # subjects with disease in the population # of people in the population

Important points: All people with active disease in numerator; can calculate “point” or “period” prevalence

Benefits: illustrates disease burdenDrawback: cross-sectional

Disease frequency example

• You have a group of 100 people. At the start of the study, 10 have active disease. Over the course of 3 years, 18 new cases develop. You accrue 200 person-years of follow-up.

• Prevalence at start: 10/100 = 0.1 = 10%• Risk over 3 years: 18/(100-10) = 0.2 = 20%• Incidence rate: 18/200 = 0.09 cases per py

= 9 cases per 100 py

Measures of disease frequencyProperty Incidence

(risk)Incidence rate (rate)

Prevalence

Smallest value 0 0 0Largest value 1 Infinity 1Dimensionality None 1/time NoneInterpretation Probability Rate; inverse

of waiting timeProportion

Attributable risk = Risk (E+) – Risk (E-) “Excess risk due to exposure”

Attributable risk % = [Risk (E+) – Risk (E-)] / Risk (E+) “% excess risk due to exposure”

Questions on measures of disease frequency?

Measures of association

RR = risk in E+ risk in E-

RR = rate in E+ rate in E-

OR = odds of E+ in cases odds of E+ in controls

Exposed Unexposed

# Cases NE NU

Total # or Total person-time

NTE or PTE NTU or PTU

E+ E-

Case A B

Controls C D

Absolute vs. Relative measures of disease frequency

• Risk, rate, prevalence, AR are absolute measures– Used for describing disease burden, policy, etc.

• Relative risk, relative rate, prevalence proportion, odds ratio, AR% are relative measures– Used to describe etiology, association of disease with

exposure, etc.

RR can mean risk ratio or rate ratio

Illustration of cohort study

Time

Risk E+

Risk E-

RR = Risk E+ Risk E-

“Exposed people are at X-fold greater risk to develop disease.”

Illustration of case-control study

Time

Odds E+

Odds E+

OR = Odds E+ cases Odds E+ controls

“Cases have X-fold greater odds of being exposed.”

• What if we could simultaneously achieve:– Prospective measurement of disease (i.e.

exposure came before disease)– Measurement of lots of confounders (for

adjustment)– Controls coming from same population as cases– Less recall bias– Less selection bias– Efficient, low cost study

Nested case-control or case-cohort study

Time

You easily measure case/control status

But you know:1) E preceded D2) Other

confounders3) Controls

came from same group as cases

Study design: observational studies (that count)Prospective cohort Retrospective

cohortCase-control

Study group E+ and E- groups E+ and E- groups Cases and controlsMeasures Rate ratio, risk

ratio, odds ratioRate ratio, risk ratio, odds ratio

Odds ratio

Temporal relationship

Possible to establish

Possible to establish Difficult to establish (except nested)

Time required Long follow-up Can be efficient Less time than othersCost Expensive Depends Relatively inexpensiveWhen to use? E is rare and/or D is

frequent among E+; investigate result of exposures

E is rare and/or D is frequent among E+; save time vs. prospective cohort

D is rare and E is frequent among D+; investigate causes of disease

Issues Selection of E-; loss to follow-up; change in E over time

Selection of E-; loss to follow-up; change in E over time

Selection of control group (selection bias); accurate E assessment (recall bias)

Study design: experimental study (RCT)• Requirement: equipoise• Design:

– Randomize groups to new treatment or standard• Benefit: Balance frequency of KNOWN and UNKNOWN confounders in

groups (matching)• Drawback: Expensive; inefficient; doesn’t always work; can’t analyze

variables that are matched on

– Follow groups through time and assess endpoints (risk, survival, etc.)• Analysis:

– Intent-to-treat (on-treatment) • Benefit: Preserve randomization• Drawback: Subjects might not have followed treatment

– Efficacy• Benefit: Analyzes subjects who followed treatment for more accurate

assessment of treatment effects• Drawback: Breaks randomization; introduces more confounding

• Issues: loss to follow-up; time; cost; changing standard of care during study

Measures in RCTs

Absolute risk reduction…attributable risk backwards:

ARR = Risk (Placebo) – Risk (Treatment) “Risk reduction attributable to treatment”

NNT = 1 / ARR “Number of patients you need to treat to prevent 1 case”

Relative risk reduction…attributable risk % backwards:

RRR = [Risk (Placebo) – Risk (Treatment)]/Risk (Placebo) “% Risk reduction attributable to treatment”

What if we’re interested in the time to the event, and not just the event?

Survival analysis, log rank, Cox proportional hazards

Bernier et al., NEJM. 2004.

HR=0.70, 95% CI 0.52-0.95

Proportional?

Meta-analysis: steps

1) Formulate purpose2) Identify relevant studies3) Establish inclusion and exclusion criteria4) Abstract data5) Describe effect measure (OR, RR)6) Assess heterogeneity (Forrest plot, Q, I2)7) Perform sensitivity and secondary analyses8) Assess publication bias (Funnel plot)9) Disseminate results

Can you group data: Forrest plot, Cochrane’s Q, I2

• Forrest plot – Illustrates size and precision of effect estimates for multiple studies.

• Cochrane’s Q – A hypothesis test of whether variation in effect estimates across studies is due to chance (H0) or not due to chance (H1).

• I2 – Percent of variation in effect estimates across studies that is due to heterogeneity rather than chance.

Meta-analysis: heterogeneity and dealing with it

Funnel plot: assessing publication bias

• Plot Sample size (y-axis) vs. Effect (x-axis)

Unskewed distribution: bias minimal Skewed distribution: bias present

Questions on measures of association or study design?

Break time?

• Scope and Scalpel 2004: Episode 1

• Scope and Scalpel 2004: Episode 2

• Mr. Pitt Med "Blue Steel" Ad

• MTV Cribs: Pitt Med

• Pitt Med Office: "New PBL Group Day"

Bias, confounding, modification…a wine digression

Apricot wine Likability

Peach, Grape

Apricot wineLikabilityLikability

RoomIced

If differ by >10%, modification present

Confounding – mixing of effects; results in inaccurate estimate of exposure-outcome association; is never “controlled,” rather “adjusted for”

Effect modification – difference of effect depending on the presence or absence of a second factor; interesting phenomenon to investigate; detected with stratification or interaction term in model

Bias – systematic error (due to study) resulting in non-comparability; error that will remain in an infinitely large study; difficult to remove once there

Will a person who enjoys apricot like Bonny Doon if it comes from a bad barrel?

Examining your new test: Sn, Sp, PPV, NPV

Sn = A / (A + C) “Of those with disease, how many did you identify?” Sp = D / (B + D) “Of those without disease, how many did you identify?” PPV = A / (A + B) “Of those you said had disease, how many truly did?” NPV = D / (C + D) “Of those you said did not have disease, how many truly did not?”

Gold standardPositive Negative

New

test

Positive A B

Negative C D

Prevalence alters PPV most

Examining your new test: Likelihood ratios

LR is a ratio of two proportions: proportion of those with a particular result among the diseased compared to the proportion with that result among the non-diseased

LR(+) = A / (A + C) = Sn LR(-) = C / (A + C) = 1-Sn B / (B + D) 1-Sp D / (B + D) Sp “The likelihood of a test outcome (+ or -) if you have the disease is X-fold

higher than if don’t have the disease.”

Gold standardPositive Negative

New

test

Positive A B

Negative C D

Examining various tests: ROC curves

Sn

1-Sp

Picking the best test depends on:1) Optimizing Sn and

Sp (highest AUC)2) Real world

conditions

HIV: We want highest Sp and sacrifice Sn

Parametric biostats: T-test, ANOVA, χ2, Pearson• T-test: if you want to test the difference in means of 2 groups (continuous)

– Assumptions and how to verify them: • Independence (are subjects related?)• Random sampling (assumed)• Normal distribution of variable (histograms, formal test)• Equal variance of variable in each group (F-test)

• ANOVA: if you want to test the difference in means between ≥2 groups (continuous)– Assumptions and how to verify them:

• Same as T-test• χ2: if you want to test the difference in frequencies among ≥2 groups (categorical)

– Assumptions and how to verify them: • Cell sizes in table (>5, formal test Use Fisher’s exact test if unfulfilled)

• Pearson r: if you want to test the degree of linear relationship between two continuous variables; does not imply causal association or a mathematical association other than linear– Assumptions and how to verify them:

• Linear relationship (look at it)• Independence, random sampling (as above)• At least 1 variable must be normally distributed

Nonparametrics: Rank sum, Kruskal-Wallis, Spearman• Mann-Whitney rank sum: if you want to test the difference in means of 2 groups

(continuous)– Assumptions and how to verify them:

• Independence (are subjects related?)• Random sampling (assumed)• Variable follows same distribution in both groups, whatever the distribution may

be• Kruskal-Wallis: if you want to test the difference in means between ≥2 groups (continuous)

– Assumptions and how to verify them: • Same as rank sum

• Spearman r: if you want to test the degree of linear relationship between two continuous variables; does not imply causal association or a mathematical association other than linear– Assumptions and how to verify them:

• Linear relationship (look at it)• Independence, random sampling (as above)

• Nonparametrics do have assumptions!• Great alternative if assumptions met, but can lack power and don’t give a good idea of

how the data are different (rely on significance)

P-values and confidence intervals

• P-value– “Is the data consistent with the null hypothesis? If not,

then there is a “statistically significant” difference.”– Depends upon sample size and magnitude of effect;

doesn’t illustrate real values A POOR MEASURE

• Confidence interval– “What is the range of possible values for the difference

observed?”– Provides information on precision of data and possible

range of values A BETTER MEASURE

Extra slides

Odds and probability

Odds = Chance of something = p Chance of not something 1-p

If p=50%, odds are 0.5/(1-0.5) = 0.5 / 0.5 = 1. Hence, 50% chance means that it is equally likely that “something” and “not something” will happen.

If p=33%, odds are 0.33/(1-0.33) = 0.33/0.67 = ½. Hence, 33% chance means that it is ½ as likely that “something” will happen compared to “not something” happening. Alternatively, it is twice as likely that “not something” will happen compared to “something” happening.

Standard Deviation vs. Standard Error• The SD is a measure of the variability in the measurements you took.

Variability can come from biologic variability, measurement variability, or both. If you believe the tool you use to measure has zero error, then the variability is solely due to biologic variability. If you want to emphasize the biologic variability (i.e. scatter) in your sample, then the SD is the appropriate statistic.

• The SEM is a measure of how well you approximated the true population mean with your sample. Again, error can come from biologic variability, measurement variability, or both. If you assume there is no biologic variability, then the only error comes from the tool you use to measure. With larger sampling sizes from the population, the measurement error becomes less and less because you are more likely to determine the true population mean with sample sizes that become closer to the true population. If you want to emphasize how precisely you determined the true population mean, then the SEM is the appropriate statistic

• The SEM is used to calculate Confidence Intervals.

Extra study design: observational studiesCase report/series Cross-sectional Ecological

Study group Patient(s) Defined pop; measure E and O simultaneously in each person

Select groups (country, county); measure E and O in population

Measures None Prevalence Correlation coefficientTemporal relationship

Unable to establish Unable to establish

Time required Little Very littleCost Intermediate InexpensiveWhen to use? Interesting/new

case(s)E and O are common Aggregate data

available; establish hypotheses

Issues Not population-based

No temporality; prevalence bias

Ecological fallacy; difficult to adjust for confounding