Covariate Selection and Balance in Propensity Score Methods M Sanni Ali University Medical Center Utrecht, the Netherlands
Covariate Selection and Balance in
Propensity Score Methods
M Sanni Ali
University Medical Center Utrecht, the Netherlands
Outline
Confounding
Propensity Score (PS) Methods
Covariate Selection
Balance Diagnostics and their Applications
Balance Assessment
PS Methods in Time-varying Treatment
Reporting of PS Analysis
Confounding
Non-intermediate common cause of exposure
and outcome
Unblocked backdoor path in causal diagrams
Observed, unobserved, unknown
Propensity Score Methods
Propensity Score
- Summary Score – Variables included?- Balancing Score – Covariate balance?
Often derived using logistic regression
- Selection of variables and forms
Ensemble methods
- Automated – Interactions/polynomials
Matching, stratification, weighting, and covariate adjustment
Rubin & Rosenbaum 1983; Westreich et al 2010; Lee 2010
Advantages of PS
Confidence on the causal inference
Transparency/Easy to communicate
Design a study separate from analysis
Summary Scores – rare outcome
Covariate Selection
Clinical knowledge - Sufficient?
Variable with Instrumental Variable like properties
- Exacerbate the imbalance in unmeasured confounders = Amplify the bias
When could bias be amplified
- Unmeasured Confounding – Strong!!!
- Strong IV !!!- Linear models
Balance diagnostics are powerful tools
- Absolute standardized difference is robust
Bhattacharya & Vogt 2007; Pearl 2010&2011; Myers et al 2011; Austin 2009;
Belitser et al 2011; M S Ali et al 2014
A simulation study using binary covariates, treatment, and outcome.
In different scenarios, confounding variables, risk factors, instrumental variables, their interaction and square terms were considered.
M S Ali et al. 2014
Covariate Selection
A
X9
Y
X7
X8
X6
X3
X1, X2 , X5
X4
Covariate Balance
PS model versus covariate sets (balance)
Four sets of covariates
- Full PS Model
- True PS Model
- Outcome PS model (Prognostic score)
- Confounder PS model
Main terms versus Main + Interaction/squares
Covariate Selection
9
Description of the Different Propensity Score Models
Covariate Selection
Treatment effects estimates (risk ratios) were derived using Poisson models.
PS model selection was made based on the balance achieved on different sets of covariates, their interaction/square terms.
Covariate balance was assessed using the absolute standardized difference.
Covariate sets in balance assessment were compared with respect to bias and precision of the treatment-outcome relation as well as the PS model selected.
Covariate Selection - Balance
11
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
Table 1. Balance of Covariates Measured Using Standardized Difference When Different Sets of Covariates Were Included in the PS Model
Covariate Selection - Balance
12
Table 2. Balance of Covariates Measured Using Standardized Difference When Different Sets of Covariates Were Included in the PS Model
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
Covariate Selection: balance
13
Table 3. Balance of Covariates Measured Using Standardized Difference When Different Sets of Covariates Were Included in the PS Model
Data was unmatched* and matched using Full PS model (PS_2), True PS model (PS_18), Risk Factor PS model (PS_25),
Confounder PS model (PS_11), Confounder PS model omitting confounder, X4 (PS_15), Confounder PS model omitting confounder,
X4, but risk factors were included (PS_29), Confounder PS model omitting confounder, X4, but instrumental variables were included
(PS_22), Confounder PS model omitting confounder, X4, but risk factors and instrumental variables were included (PS_8).
ModelAll Covariates
were Independent
RR (IQR)
X4 and X5 were
Correlated
RR (IQR)
X4 and X7 were
Correlated
RR (IQR)
Crude 0.93 0.15 0.89 0.17 0.91 0.14
Full PS Model 1.73 0.40 1.78 0.68 1.72 0.59
True PS Model 1.71 0.47 1.73 0.63 1.67 0.58
Outcome PS Model 1.71 0.39 1.77 0.48 1.72 0.51
Conf PS Model 1.77 0.40 1.79 0.56 1.71 0.50
Omit.Conf PS Model 1* 1.55 0.34 1.57 0.40 1.46 0.35
Omit.Conf PS Model 2** 1.54 0.32 1.59 0.39 1.45 0.34
Omit.Conf PS Model 3† 1.50 0.36 1.50 0.50 1.51 0.47
Omit.Conf PS Model 4†† 1.50 0.34 1.55 0.52 1.52 0.45
Table 4. Median (Interquartile Range, IQR) of Estimated Treatment Effect Using Different PS Models in Different Scenarios
(True RR=1.75)
Covariate Selection
15
PS matching improved balance of measured covariates included in the PS model
It exacerbated the imbalance in the unmeasured covariate that was unrelated to measured covariates
In choosing covariates for a PS model, the pattern of association among covariates has substantial impact on other covariates’ balance and the bias of the treatment effect estimate.
Table 5. Median (IQR) Treatment Effect When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75)
Calliper All
CovariatesConfounding
Factors
Confounding and Risk Factors
Confounding and Treatment Related Factors
0.05 1.51 0.42 1.77 0.64 1.75 0.62 1.48 0.43
0.10 1.50 0.43 1.75 0.63 1.75 0.58 1.48 0.42
0.15 1.50 0.44 1.73 0.67 1.76 0.65 1.48 0.43
0.20 1.48 0.42 1.75 0.62 1.75 0.60 1.46 0.43
0.25 1.46 0.43 1.75 0.61 1.72 0.58 1.43 0.42
0.30 1.47 0.43 1.74 0.62 1.72 0.60 1.43 0.42
0.35 1.44 0.41 1.69 0.61 1.69 0.57 1.41 0.39
0.40 1.40 0.41 1.68 0.59 1.67 0.61 1.37 0.39
0.50 1.38 0.36 1.68 0.54 1.63 0.50 1.35 0.35
0.60 1.33 0.37 1.58 0.51 1.55 0.52 1.31 0.34
*Balance was assessed on main terms as well as interaction/square terms
Table 6. Median (IQR) Treatment Effect When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75)
Calliper All Covariates
Confounding
Factors
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
0.05 1.73 0.59 1.74 0.47 1.73 0.51 1.77 0.67
0.10 1.76 0.64 1.78 0.52 1.75 0.54 1.75 0.62
0.15 1.72 0.68 1.75 0.50 1.73 0.54 1.75 0.63
0.20 1.74 0.60 1.74 0.50 1.74 0.45 1.74 0.61
0.25 1.70 0.65 1.74 0.52 1.71 0.50 1.72 0.61
0.30 1.71 0.59 1.72 0.53 1.72 0.53 1.71 0.58
0.35 1.67 0.56 1.70 0.48 1.68 0.47 1.68 0.58
0.40 1.67 0.60 1.70 0.49 1.68 0.48 1.66 0.56
0.50 1.60 0.53 1.65 0.49 1.62 0.48 1.61 0.55
0.60 1.54 0.48 1.65 0.45 1.64 0.46 1.53 0.50
*Balance was assessed only main terms not interaction/square terms
Table 7. Median Number of Subjects Included When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75, n=2000)
*Balance was assessed main and interaction/square terms
Calliper All Covariates
Confounding
Factors
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
0.05 1388 1388 1388 1388
0.10 1390 1389 1389 1390
0.15 1391 1391 1391 1391
0.20 1393 1392 1392 1393
0.25 1394 1394 1394 1394
0.30 1397 1396 1396 1397
0.35 1397 1398 1398 1398
0.40 1401 1401 1401 1401
0.50 1408 1408 1406 1408
0.60 1415 1417 1415 1415
Table 8. Median Number of Subject Included When Propensity Score Model was Chosen Based on Balance Calculations on Different Sets of Covariates (True RR =1.75, n=2000)
*Balance was assessed only main terms not interaction/square terms
Calliper All Covariates
Confounding
Factors
Confounding and
Risk Factors
Confounding and
Treatment Related
Factors
0.05 1388 1481 1479 1388
0.10 1390 1481 1479 1389
0.15 1392 1479 1478 1391
0.20 1393 1479 1479 1392
0.25 1393 1479 1478 1393
0.30 1396 1480 1480 1396
0.35 1398 1482 1481 1397
0.40 1401 1484 1484 1401
0.50 1407 1488.5 1489 1407
0.60 1415 1496 1496 1415
PS Model Selection
When PS model was chosen based on balance calculation on confounding variables, the confounder PS is often selected followed by the Risk factor PS models.
When balance calculation involves all covariates or treatment related covariates, the full PS and true PS model are often selected compared to the confounder PS and Risk factor PS models.
PS Model Selection
In selecting PS model based on covariate balance, the choice of covariates, interaction/square terms in balance calculation has substantial impact on bias and precision of the treatment effect.
PS model selection based on the balance achieved on confounding variables, risk factors and important interaction terms among confounders and risk factors is optimal approach.
Balance DiagnosticsQuantitative Falsification of Instrumental Variable Assumptions
Instrumental Variable Methods
three Basic IV assumption
- Associated with Exposure
- Independent of Confounders
- Independent of outcome except through exposure
Association of IV with observed confounders could be
quantified with balance metrics
M S Ali et al 2014
IV Methods and assumptions
PS and time-varying treatment
In the presence of treatment switching or
channeling - “Intension to treat analysis”?
Time-varying Cox model or
Time-varying propensity score or
Inverse probability weighting or others
Time-varying treatment
Example
Associations Between Current SSRI Use and the Risk of Hip Fracture
Study using two EU databases (Mondriaan and BIFAP)
A cohort of patients with a first prescription for Antidepressant (SSRI or tricyclic AD, TCA) period 2001-2009
Data sources (GP databases) :- the Dutch Mondriaan - the Spanish BIFAP
M S Ali et al 2015
Time-varying treatment
HFSSRItSSRIt-1
BenzotBenzot-1
Cox models with time-varying coefficients to control for time-varying nature of covariates
Time-varying treatment
When time-varying confounders are themselves affected by the previous treatment (SSRI use at t-1), conventional time-varying Cox model gives biased estimate
Adjusting-away part of treatment effect?
HFSSRItSSRIt-1
BenzotBenzot-1
Time-varying treatment
In the presence of unmeasured common causes of confounders and outcome (U):
collider-stratification bias?
HFSSRItSSRIt-1
BenzotBenzot-1
U
Time - varying treatment
Adjusted for
Mondriaan BIFAP
HR 95% CI HR 95% CI
Crude 1.75 1.12, 2.72 2.09 1.89, 2.32
Gender 1.73 1.10, 2.69 2.07 1.87, 2.30
Gender +Age 2.36 1.51, 3.68 1.51 1.37, 1.68
Gender +Age + TCAt 2.59 1.63, 4.12 1.56 1.40, 1.73
Gender +Age + TCAt + Benzot 2.60 1.63, 4.16 1.54 1.38, 1.71
All Confounders* 2.62 1.63, 4.19 1.52 1.37, 1.69
Table 9. Associations Between SSRI Use and the Risk of Hip Fracture Using Time-Varying Cox Models
Time-varying treatment
Table 10. Associations Between SSRI Use and the Risk of Hip Fracture Using Propensity Score Based Cox Analyses
Adjusted for
Mondriaan BIFAP
HR 95% CI HR 95% CI
Crude 1.75 1.12, 2.72 2.09 1.89, 1.71
PS* Stratification
Quintiles 2.64 1.63, 4.25 1.54 1.39, 1.71
Deciles 2.72 1.63, 4.54 1.53 1.38, 1.70
PS* Adjustment 2.82 1.63, 4.25 1.61 1.45, 1.78
Time-varying treatmentTable 11. Associations Between SSRI Use and the Risk of Hip Fracture
Using Propensity Score Based Cox Analyses
Adjusted forMondriaan BIFAP
HR 95% CI HR 95% CI
Without Accounting
for Censoring*
Crude 1.69 1.05, 2.67 2.14 1.91, 2.39
Gender 1.68 1.06, 2.67 2.11 1.89, 2.38
Gender +Age 2.46 1.55, 3.99 1.54 1.37, 1.72
Accounting for
Censoring**
Crude 1.73 1.08, 2.77 2.05 1.83, 2.30
Gender 1.71 1.07, 2.74 2.03 1.81, 2.28
Gender +Age 2.47 1.53, 3.98 1.51 1.35, 1.70
* Only inverse probability of treatment weights were used
** Combined inverse probability of treatment and censoring weights were used + Trimming at 1% and 99%
Time-varying treatment
Differences between the various methods to adjust for time-dependent confounding (i.e., time-varying Cox and PS as well as inverse probability weighting of marginal structural models) were small.
The observed differences in treatment effects estimates between the datasets are likely attributable to different confounding information in the datasets.
Adequate information on (time-varying) confounding.
Conduct and Reporting PS Analysis: Literature review
M S Ali et al 2015
92 Articles were excluded:
63 Non clinical
20 Methodological
6 Other language
2 Systematic reviews
1 Editorials/letters
388 articles
PubMed search:
296 Articles available for review
Drug-related intervention: 108 (36.5%)
Clinical* 50 (16.9%)
Surgical intervention: 138 (46.6%)
Figure 1. Flow chart of abstracts or articles extraction for the systematic review
Reporting PS Analysis
PS matching is the commonly used approach.
Balance is often checked with PS matching followed by inverse probability weighting
P-values are the most commonly used tests.
Absolute standardized difference was used only in 25% of the studies in which balance was assessed.
M S Ali et al 2015
Reporting PS Analysis
Transparency in conducting the analysis
What needs to be reported??
Guidelines: STROBE or ENCePP
M S Ali et al 2015
PS Matching
- Matching algorithm, caliper width, and matching ratio used.
- Whether matching was with replacement. and matched nature of the data accounted in the analysis.
- Number of patients at the start and after matching.
- Number and characteristics of excluded patients (versus matched ones)
- The distribution of baseline characteristics between treated and control patients
in the matched and starting population.-Balance of Covariates between matched groups
♠
Variable Selection for PS Model
- Method used for variable selection - Whether empirical knowledge was considered - Whether variable’s association with treatment
and/or outcome
Propensity Score Estimation
- Method used to estimate the PS, e.g. logistic regression
- Variables and/or interaction terms included in the PS
Balance Assessment
- Balance measure used.- Quantifying the balance and whether imbalance on
covariates was detected after the final PS model.
Treatment Effect Estimation- Statistical method used.- Whether additional adjustment was made for
covariates.- Whether sensitivity analysis was performed.
Applying the PS methods
- Type of PS method employed.
Interpretation of Effect estimate
- The interpretation of the effect estimates in relation to the research question, target
- population & type of PS method used (ATE, ATT)
PS Weighting- The range (mean, max, min) of unstabilzed
and stabilized weights.- Variables included in the PS models for
both numerator and denominator of the weights.
- Whether weights were truncated and method used.
- Balance of Covariates in the weighted sample groups.
-
PS Stratification- The quantile used for stratification.- The overlap of PS with quantiles of PS using
plots or balance measures.- Balance of covariates with in quintiles of the
PS.
PS Adjustment
- Overlaps of the PS distribution between
treatment groups. *
- Whether linear relationship was checked
between outcome and the PS.
M Sanni Ali 2015
Limitations of PS
Unmeasured Confounding
Treatment Modification/Interaction
Multilevel Treatment Modeling
Dynamic Prescription Patterns/rare Exposure
Pooling Data on PS? Treatment guidelines
Alternatives: Disease Risk Score?
Acknowledgement
Olaf H. Klungel, PharmD, PhD1,2
Anthonius de Boer, MD, PhD1
Rolf HH Groenwold, MD, PhD1,2
Arno W. Hoes, MD, PhD2
Svetlana V. Belitser, MSc1
Kit C.B. Roes, PhD2
1 Division of Pharmacoepidemiology and Clinical Pharmacology, Utrecht Institute for Pharmaceutical
Sciences, Utrecht University, Utrecht.
2 Julius Centre for Health Science and Primary Care, University Medical Centre Utrecht, Utrecht.
Thank You!