1 Sensitivity Analysis for Residual Confounding Sebastian Schneeweiss MD, ScD Division of Pharmacoepidemiology and Pharmacoeconomics Department of Medicine, Harvard Medical School,
1
Sensitivity Analysis for Residual Confounding
Sebastian Schneeweiss MD, ScD
Division of Pharmacoepidemiology and Pharmacoeconomics
Department of Medicine, Harvard Medical School,
2
Outline
1. Residual Confounding and what we can do about it
2. Simple sensitivity analysis: Array Approach3. Study-specific analysis: Rule Out
Approach4. Using additional information: External
Adjustment
3
Unmeasured (residual) Confounding:
[smoking,healthy lifestyle, etc.]
Drug exposure
Outcome
RREO
OREC RRCO
CU
CM
4
Unmeasured Confounding in Claims Data
Database studies are criticized for their inability to measure clinical and life-style parameters that are potential confounders in many pharmacoepi studies OTC drug use BMI Clinical parameters: Lab values, blood pressure, X-
ray Physical functioning, ADL (activities of daily living) Cognitive status
5
Strategies to Minimize Residual Confounding
Choice of comparison group Alternative drug use that have the same perceived
effectiveness and safety Multiple comparison groups
Crossover designs (CCO, CTCO) Instrumental Variable estimationHigh dimensional proxy adjustment
6
Strategies to Discuss Residual Confounding
Qualitative discussions of potential biasesSensitivity analysis
SA is often seen as the ‘last line of defense’ A) SA to explore the strength of an association as a
function of the strength of the unmeasured confounder B) Answers the question “How strong must a
confounder be to fully explain the observed association”
Several examples in Occupational Epi but also for claims data
Greenland S et al: Int Arch Occup Env Health 1994
Wang PS et al: J Am Geriatr Soc 2001
7
Dealing with confounding
Schneeweiss, PDS 2006
Confounding
Unmeasured Confounders
Measured Confounders
Design
•Restriction
•Matching
Analysis
•Standardization
•Stratification
•Regression
Unmeasured, but measurable in
substudy
•2-stage sampl.
•Ext. adjustment
•Imputation
Unmeasurable
Design Analysis
•Cross-over
•Active comparator (restriction)
•Instrumental variable
•Proxy analysis
•Sensitivity analysis
Propensity scores
•Marginal Structural Models
8
A simple sensitivity analysis
The apparent RR is a function of the adjusted RR times ‘the imbalance of the unobserved confounder’
After solving for RR we can plug in values for the prevalence and strength of the confounder:
1)1(
1)1(
0
1
CDC
CDC
RRP
RRPRRARR
1)1(
1)1(
0
1
CDC
CDC
RRP
RRP
ARRRR
9
A made-up example
Association between TNF-a blocking agents and NH lymphoma in RA patients Let’s assume an observed RR of 2.0 Let’s assume 50% of RA patients have a more
progressive immunologic disease … and that more progressive disease is more likely
to lead to NH lymphoma Let’s now vary the imbalance of the hypothetical
unobserved confounder
10
Bias by residual confounding
4.5
2.5
0.8
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
RRadjusted
RRCD
PC1
Fixed:ARR = 2.0
PC0 = 0.5
11
2. Array approach
fix X Y fix Z2 Z1
ARR RRCD PC1 PC0 RRadjusted % Bias % Bias = [(ARR-RRadj.)/(RRadj.-1)]*100
2.0 4.5 0.0 0.5 5.5 -77.782.0 4.0 0.0 0.5 5.0 -75.002.0 3.5 0.0 0.5 4.5 -71.432.0 3.0 0.0 0.5 4.0 -66.672.0 2.5 0.0 0.5 3.5 -60.002.0 2.0 0.0 0.5 3.0 -50.002.0 1.5 0.0 0.5 2.5 -33.332.0 1.0 0.0 0.5 2.0 0.002.0 0.8 0.0 0.5 1.8 33.332.0 0.5 0.0 0.5 1.5 100.002.0 4.5 0.1 0.5 4.1 -67.472.0 4.0 0.1 0.5 3.8 -64.862.0 3.5 0.1 0.5 3.6 -61.542.0 3.0 0.1 0.5 3.3 -57.142.0 2.5 0.1 0.5 3.0 -51.062.0 2.0 0.1 0.5 2.7 -42.112.0 1.5 0.1 0.5 2.4 -27.592.0 1.0 0.1 0.5 2.0 0.002.0 0.8 0.1 0.5 1.8 25.812.0 0.5 0.1 0.5 1.6 72.732.0 4.5 0.2 0.5 3.2 -55.262.0 4.0 0.2 0.5 3.1 -52.942.0 3.5 0.2 0.5 3.0 -50.002.0 3.0 0.2 0.5 2.9 -46.152.0 2.5 0.2 0.5 2.7 -40.912.0 2.0 0.2 0.5 2.5 -33.332.0 1.5 0.2 0.5 2.3 -21.432.0 1.0 0.2 0.5 2.0 0.002.0 0.8 0.2 0.5 1.8 18.752.0 0.5 0.2 0.5 1.7 50.002.0 4.5 0.3 0.5 2.7 -40.582.0 4.0 0.3 0.5 2.6 -38.712.0 3.5 0.3 0.5 2.6 -36.362.0 3.0 0.3 0.5 2.5 -33.332.0 2.5 0.3 0.5 2.4 -29.272.0 2.0 0.3 0.5 2.3 -23.532.0 1.5 0.3 0.5 2.2 -14.812.0 1.0 0.3 0.5 2.0 0.002.0 0.8 0.3 0.5 1.9 12.122.0 0.5 0.3 0.5 1.8 30.772.0 4.5 0.4 0.5 2.3 -22.582.0 4.0 0.4 0.5 2.3 -21.432.0 3.5 0.4 0.5 2.3 -20.002.0 3.0 0.4 0.5 2.2 -18.182.0 2.5 0.4 0.5 2.2 -15.792.0 2.0 0.4 0.5 2.1 -12.502.0 1.5 0.4 0.5 2.1 -7.692.0 1.0 0.4 0.5 2.0 0.002.0 0.8 0.4 0.5 1.9 5.882.0 0.5 0.4 0.5 1.9 14.292.0 4.5 0.5 0.5 2.0 0.002.0 4.0 0.5 0.5 2.0 0.002.0 3.5 0.5 0.5 2.0 0.002.0 3.0 0.5 0.5 2.0 0.00
4.5
3.5
2.5
1.5
0.8
0.0 0.
2 0.4 0.
6 0.8 1.
0-100
-50
0
50
100
150
200
250
300
350
% Bias
RRCD
PC1
Fixed:ARR = 2.0
PC0 = 0.5
4.5
2.5
0.8
0.0 0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0
RRadjusted
RRCD
PC1
Fixed:ARR = 2.0
PC0 = 0.5
1)1(
1)1(
0
1
.
CDC
CDC
adj
RRP
RRP
ARRRR
drugepi.o
rg
12
Pros and cons of “Array approach”
Very easy to perform using ExcelVery informative to explore confounding with
little prior knowledge Problems: It usually does not really provide an answer
to a specific research question4 parameters can vary -> in a 3-D plot 2
parameter have to be kept constantThe optical impression can be manipulated
by choosing different ranges for the axes
13
Same example, different parameter ranges
3.0
1.7
0.8
0.2
0.3
0.3
0.4
0.4
0.5
0.5
0.6
0.6
0.7
0.7
0.0
0.5
1.0
1.5
2.0
2.5
3.0
RRadjusted
RRCD
PC1
Fixed:ARR = 2.0
PC0 = 0.5
14
Conclusion of “Array Approach”
Great tool but you need to be honest to yourself
For all but one tool that I present today: Assuming conditional independence of CU and CM
given the exposure status If violated than residual bias may be overestimated
Drug exposure
Outcome
RREO
OREC RRCO
CU
CM
Hernan, Robins: Biometrics 1999
?
15
More advanced techniques
Wouldn’t it be more interesting to know How strong and imbalanced does a confounder
have to be in order to fully explain the observed findings?
RRCO OREC
16
Example:
Psaty et al: JAGS 1999;47:749
CCB use and acute MI.
The issue:
Are there any unmeasured factors that may lead to a preferred prescribing of CCB to people at higher risk for AMI?
OREC
RRCO
ARR = 1.57
ARR = 1.30
17
3. Rule Out Residual ConfoundingHow strong does an unmeasured confounder have to be to fully explain the observed findings?
The relationship between OREC and RRCD for a given ARR, RRC, PC, PE.
Data from Psaty et al.: Assessment and control for confounding by indication in observational studies. J Am Geriatri Soc 1999;47:749-54.a(prim)
RRCD PC PE ARR=1.57 OREC ARR=1.3 OREC
1.2 0.2 0.01 1.57 1.3 0.0311771.5 0.2 0.01 1.57 1.3 23.90 0.0143442 0.2 0.01 1.57 28.79 1.3 5.11 0.008733
2.5 0.2 0.01 1.57 9.03 1.3 3.42 0.0068633 0.2 0.01 1.57 5.97 1.3 2.78 0.005928
3.5 0.2 0.01 1.57 4.73 1.3 2.45 0.0053674 0.2 0.01 1.57 4.06 1.3 2.25 0.004993
4.5 0.2 0.01 1.57 3.65 1.3 2.12 0.0047255 0.2 0.01 1.57 3.36 1.3 2.02 0.004525
5.5 0.2 0.01 1.57 3.15 1.3 1.94 0.0043696 0.2 0.01 1.57 2.99 1.3 1.88 0.004244
6.5 0.2 0.01 1.57 2.87 1.3 1.84 0.0041427 0.2 0.01 1.57 2.77 1.3 1.80 0.004057
7.5 0.2 0.01 1.57 2.68 1.3 1.77 0.0039858 0.2 0.01 1.57 2.61 1.3 1.74 0.003924
8.5 0.2 0.01 1.57 2.56 1.3 1.71 0.003879 0.2 0.01 1.57 2.51 1.3 1.69 0.003824
9.5 0.2 0.01 1.57 2.46 1.3 1.68 0.00378210 0.2 0.01 1.57 2.42 1.3 1.66 0.003746
0.00
2.00
4.00
6.00
8.00
10.00
0 2 4 6 8 10
RRCD
OR
EC
ARR=1.57
ARR=1.3
drugepi.o
rg
18
Caution!
Psaty et al. concluded that it is unlikely that an unmeasured confounder of that magnitude exists
However, the randomized trial ALLHAT showed no association between CCB use and AMI
Alternative explanations: Joint residual confounding may be larger than
anticipated from individual unmeasured confounders Not an issue of residual confounding but other biases,
e.g. control selection?
19
Pros and cons of “Rule Out Approach”
Very easy to perform using Excel Meaningful and easy to communicate
interpretationStudy-specific interpretationProblems:Still assuming conditional independence of CU
and CM “Rule Out” lacks any quantitative assessment
of potential confounders that are unmeasured
20
External Adjustment
One step beyond sensitivity analysesUsing additional information not available in
the main studyOften survey information
21
Strategies to Adjust residual con-founding using external information
Survey information in a representative sample can be used to quantify the imbalance of risk factors that are not measured in claims among exposure groups
The association of such risk factors with the outcome can be assess from the medical literature (RCTs, observational studies)
Velentgas et al: PDS, 2007
Schneeweiss et al: Epidemiology, 2004
22
In our example:
Rofecoxib Acute MI
RREO
From Survey data in a
subsampleFrom medical
literature
OREC RRCO
[smoking,aspirin, BMI, etc.]
CU
CM
23
More contrasts
% Bias††
COX-2 (872) vs.
non-selective NSAIDs (1,302)
COX-2 (872) vs.
non-users (6,611)
COX-2 (872) vs.
naproxen (238)
Rofecoxib (244) vs.
naproxen (238) Potential confounder:*
Obesity (BMI30 vs. BMI<30)
-0.11 4.31 2.42 0.01
Aspirin use (use vs. non-use)
0.29 -0.08 -0.34 -1.28
Smoking (current vs. never)
-1.97 -2.41 -2.36 -0.61
Educational Attainment ( high school vs. >high school)
-2.36 -1.13 -3.67 -5.61
Income status ($20,000 vs. >$20,000)
-1.44 -1.08 -1.47 -1.65
Net confounding:
Sum of all negative biases: -5.88 -4.69 -5.08 -9.15
Weighted average: -1.56 -0.54 -1.86 -3.15
Sum of all positive biases: 0.29 4.31 -0.34 0.01
24
Sensitivity of Bias as a Function of a Misspecified RRCD :
-20
-15
-10
-5
0
5
10
15
20
1 1.5 2 2.5 3 3.5 4 4.5RRCD
Bia
s o
f R
RE
D i
n %
COX-2 vs. non-selective NSAIDsCOX-2 vs. non-usersCOX-2 vs. naproxenRofecoxib vs. naproxen
Literature estimate
RRCD = 1.7
Obesity (BMI >=30 vs. BMI<30)
25
Sensitivity towards a misspecified RRCO from the literature:OTC aspirin use (y/n)
-20
-15
-10
-5
0
5
10
15
20
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
RRCD
Bia
s o
f R
RE
D i
n %
COX-2 vs. non-selective NSAIDsCOX-2 vs. non-usersCOX-2 vs. naproxenRofecoxib vs. naproxen
Literature estimate
RRCD = 0.7
26
4. External AdjustmentGiven external information for selected factors on OREC from survey data and RRCD from the literature,
how much confounding is caused by not controling for these factors?
Data from Schneeweiss et al.: Assessment of bias by unmeasured confoundersin pharmacoepidemiologic claims data studies using external information. Epidemiology 2004, in press.
Unmeasured covariate: Aspirin (use vs. non-use) Bias as a function of misspecification of the RRCD from the literature:Data source: Lit MCBS MCBS assumed MCBSParameter: RRCD p(C) OREC true RRED p(E) app RRED CRR % biasSensitivity: varry const const const const calc calc
COX vs. 0.1 0.1 0.9 1 0.4 1.0093282 1.009 0.933
NSAID 0.2 0.1 0.9 1 0.4 1.0081979 1.008 0.820
0.3 0.1 0.9 1 0.4 1.0070929 1.007 0.709
0.4 0.1 0.9 1 0.4 1.0060124 1.006 0.601
0.5 0.1 0.9 1 0.4 1.0049555 1.005 0.496
0.6 0.1 0.9 1 0.4 1.0039215 1.004 0.392
0.7 0.1 0.9 1 0.4 1.0029096 1.003 0.291
0.8 0.1 0.9 1 0.4 1.0019192 1.002 0.192
0.9 0.1 0.9 1 0.4 1.0009495 1.001 0.095
1 0.1 0.9 1 0.4 1 1.000 0.000COX vs. 0.1 0.09 1.03 1 0.12 0.997608 0.998 -0.239
non-user 0.2 0.09 1.03 1 0.12 0.9978943 0.998 -0.211
0.3 0.09 1.03 1 0.12 0.9981752 0.998 -0.182
0.4 0.09 1.03 1 0.12 0.9984507 0.998 -0.155
0.5 0.09 1.03 1 0.12 0.998721 0.999 -0.128
0.6 0.09 1.03 1 0.12 0.9989863 0.999 -0.101
0.7 0.09 1.03 1 0.12 0.9992468 0.999 -0.075
0.8 0.09 1.03 1 0.12 0.9995024 1.000 -0.050
0.9 0.09 1.03 1 0.12 0.9997535 1.000 -0.025
1 0.09 1.03 1 0.12 1 1.000 0.000COX vs. 0.1 0.09 1.15 1 0.79 0.9892571 0.989 -1.074
naproxen 0.2 0.09 1.15 1 0.79 0.9905337 0.991 -0.947
0.3 0.09 1.15 1 0.79 0.9917884 0.992 -0.821
0.4 0.09 1.15 1 0.79 0.9930216 0.993 -0.698
0.5 0.09 1.15 1 0.79 0.9942339 0.994 -0.577
0.6 0.09 1.15 1 0.79 0.9954259 0.995 -0.457
0.7 0.09 1.15 1 0.79 0.996598 0.997 -0.340
0.8 0.09 1.15 1 0.79 0.9977507 0.998 -0.225
0.9 0.09 1.15 1 0.79 0.9988846 0.999 -0.112
1 0.09 1.15 1 0.79 1 1.000 0.000Rofecox vs. 0.1 0.1 1.6 1 0.51 0.9597175 0.960 -4.028
naproxen 0.2 0.1 1.6 1 0.51 0.9644945 0.964 -3.551
0.3 0.1 1.6 1 0.51 0.9691917 0.969 -3.081
0.4 0.1 1.6 1 0.51 0.9738113 0.974 -2.619
0.5 0.1 1.6 1 0.51 0.9783551 0.978 -2.164
0.6 0.1 1.6 1 0.51 0.9828249 0.983 -1.718
0.7 0.1 1.6 1 0.51 0.9872227 0.987 -1.278
0.8 0.1 1.6 1 0.51 0.99155 0.992 -0.845
0.9 0.1 1.6 1 0.51 0.9958085 0.996 -0.419
1 0.1 1.6 1 0.51 1 1.000 0.000
Unmeasured covariate: BMI (obese vs. non-obese)Data source: Lit MCBS MCBS assumed MCBSParameter: RRCD p(C) OREC true RRED p(E) app RRED CRR % bias
-20
-10
0
10
20
1 2 3 4 5 6 7 8
R RCD
-20-15-10-505101520
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
R RCD
-20
-10
0
10
20
1 1.5 2 2.5 3 3.5 4 4.5
R RCD
-20
-10
0
10
20
1 1.5 2 2.5 3 3.5 4 4.5
R RCD
-20
-10
0
10
20
1 1.5 2 2.5 3 3.5 4 4.5
R RCD
-20
-10
0
10
20
0 1 2 3 4 5 6 7 8
R RCD
COX-2 vs. non-selective NSAIDs
COX-2 vs. non-users
COX-2 vs. naproxen
Rofecoxib vs. naproxen
-20
-15
-10
-5
0
5
10
15
20
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
RRCD
Bia
s o
f R
R ED i
n %
COX-2 vs. non-selective NSAIDsCOX-2 vs. non-users
COX-2 vs. naproxenRofecoxib vs. naproxen Literature estimate
RRCD = 0.7
drugepi.o
rg
27
Limitations Example is limited to 5 potential confounders
No lab values, physical activity, blood pressure etc. What about the ‘unknow unknowns’?
To assess the bias we assume an exposure–disease association of 1 (null hypothesis) The more the truth is away from the null the more bias
in our bias estimate However the less relevant unmeasured confounders
become Validity depends on representativenes of sampling
with regard to the unmeasured confounders We could not consider the joint distribution of
confounders Limited to a binary world
28
Solving the Main LimitationsNeed a method
That addresses the joint distribution of several unmeasured confounders
That can handle binary, ordinal or normally distributed unmeasured confounders
Lin et al. (Biometrics 1998): Can handle a single unmeasured covariate of any
distribution But can handle only 1 covariate
Sturmer, Schneeweiss et al (Am J Epidemiol 2004): Propensity score calibration Can handle multiple unmeasured covariates of any
distribution
29
Summary
Sensitivity analyses for residual confounding are underutilized although they are technically easy to perform
Excel program for download (drugepi.org)The real challenge is the interpretation of
your findingsThis is all summarized in Schneeweiss PDS
2007