Using Propensity Score Analysis to Assess Effectiveness of Social Marketing Campaigns in Healthcare: An Example from Medicare Open Enrollment Frank Funderburk, Diane Field, & Clarese Astrin Division of Research Office of Communications Centers for Medicare and Medicaid Services The statements expressed here are those of the authors and do not necessarily reflect the views or policies of CMS
17
Embed
Funderburk et al 2011 ICHPS_Using_Propensity_Score_& MI_Analysis_10-6-11
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Using Propensity Score Analysis to Assess Effectiveness of Social Marketing Campaigns in Healthcare: An Example
from Medicare Open EnrollmentFrank Funderburk, Diane Field,
& Clarese AstrinDivision of Research
Office of Communications Centers for Medicare and Medicaid Services
The statements expressed here are those of the authors and do not necessarily reflect the views or policies of CMS
Assessing Medicare Open Enrollment (OE)
• Annual pre-post survey to assess beneficiary awareness of benefits and behavior during OE
• Domains covered –– Awareness– Knowledge– Review/Compare Rate– Satisfaction with plan
Log likelihood = -476.02371 Pseudo R2 = 0.0206 Prob > chi2 = 0.0179 LR chi2(9) = 20.00Logistic regression Number of obs = 731
. logistic review adexpose age_yrs male healthst married ed_gt_hs white hh_incom active, or
Limitations• Treatment is not randomly assigned, so other
variables (other than seeing the Medicare TV ad) may contribute to the beneficiaries decision to review coverage options.
• Self-selection or other nonrandom selection processes can be mistaken for treatment effects.
• Missing data on one or more covariates can also be a source of bias.
Propensity Scores• Propensity score matching aims to “correct” the estimation
of treatment effects in observational studies. Apples to apples comparisons.
• Identify treated and untreated subjects who are as identical as possible on key covariates
• Summarize characteristics of subjects into a single variable to facilitate matching
• Allow one to mimic counterfactual substitutes and make causal inferences (under certain conditions)
• Illustrate approach with data from Medicare OE, with focus on exposure to TV advertising and reviewing coverage options during OE
Missing Data
• 29% missing categorical household income• 20% missing data for audience segmentation• 5% missing satisfaction rating• 2.5% missing education• Multiple imputation can be used to address
this issue and allow these variables to be used in propensity scoring
How It’s Done
• Forget about your outcome variable(s)• Model treatment exposure (saw Medicare TV ad)
with logistic regression – use potential predictors of treatment exposure EXCEPT those that are outcomes of treatment exposure
• Estimate predicted value of exposure from model• Use propensity score in analysis to estimate
treatment effect
Example Using STATA
Total 1,034 1,034
Treated 378 378
Untreated 656 656
assignment On suppor Total
Treatment support
psmatch2: Common
psmatch2:
Note: S.E. does not take into account that the propensity score is estimated.
ATT .677248677 .558201058 .119047619 .044404683 2.68
Note: S.E. does not take into account that the propensity score is estimated.(running psmatch2 on estimation sample)> .01) logit common. bootstrap r(att), reps(500) : psmatch2 adexpose age_yrs white male ed_gt_hs hh_incom active internet healthst , outcome (review) caliper (0
Modeling Other Effects
Propensity Score Analysis Issues
• Model specification• Matching approach• Effect estimates• Control for unobserved variables [absence of]• Missing values for covariates• Trade offs of precision and bias related to
“support”• Others
Some Benefits
• Increases attention on need to evaluate degree of overlap/balance between conditions
• Helps one think about design of observational studies
• Clear diagnostics• Reduces confounding• Complements rather than replaces other
analytic tools
Contact Information
Frank FunderburkDirector, Division of ResearchOffice of CommunicationsCenters for Medicare and Medicaid [email protected](410)786-1820