Can administrative data increase the practicality of clinical trials? An example from the Women’s Health Initiative Garnet Anderson Fred Hutchinson Cancer.

Can administrative data increase the practicality

of clinical trials?An example from the Women’s Health Initiative

Garnet AndersonFred Hutchinson Cancer Research Center

Pragmatic trial (coined by Schwartz and Lellouch in 1967)• A randomized controlled trial designed to inform decisions about

practice• Used to describe a trial designed to test the effectiveness of the

intervention in a broad routine clinical practice setting (as opposed to testing the efficacy in an ideal setting)

Pragmatic methods and motivations

• Broaden eligibility to improve generalizability, increase recruitment yield and perhaps reduce costs

• Test interventions/delivery mechanisms that better emulate clinical practice and thus improve estimates of population level impact and reduce costs

• Limit data collection to minimize participant burden and reduce costs

Was the original Women’s Health Initiative a pragmatic trial?

DM HT

CaD

os

48,835 27,347

36,282

93,676

CT=68,132 Total WHI =161,809

WHI transitions in post-intervention phase• Protocol streamlined to annual mail follow-up• In 2010, documentation and central adjudication of outcomes limited

to African American, Hispanic and former HT participants (Medical Records Cohort, n~22,000)

• Outcomes for remaining participants (Self-Report Cohort, N~71,000) limited to self-report or passive follow-up sources (NDI, Medicare)

A Pragmatic Trial : Physical Activity to Improve CV Health in Women

Marcia L. Stefanick, Ph.D.Charles L. Kooperberg, Ph.D.

Andrea Z. LaCroix, Ph.D.

Women’s Health Initiative Strong & Healthy Trial

1 U01 HL122280-01

Primary Hypothesis

To assess whether aerobic physical activity combined with muscle strengthening, balance and flexibility exercises, and reduced sedentary behavior, will reduce major CV events (MI, stroke, CV death) in older women, compared to “Usual Activity” (Control) over 4-5 years of follow-up

Based on: Report of the Physical Activity Guidelines Advisory Committee, 2008. (Chapter 5: Active Older Adults) http://www.health.gov/paguidelines/guidelines/chapter5.aspx

Implemented through: National Institutes on Aging (NIA) http://go4life.nia.nih.gov/

Eligibility

• WHI participant, alive and in active follow-up• In Medical Records Cohort or in Self Report cohort and enrolled in

Medicare Part A/B• Exclusions

• Inability to walk• Dementia• Residing in nursing home

Eligible based on existing data

Intervention(n ~ 26,000)

Consent

WHISH PA (Go4Life®) Interventiondeliver mail-based [+ website, etc.] ± IVR** (phone) + live advisor, PRN

Follow, per WHI protocolno

yes

yes

Randomize

Follow, per WHI protocol

Follow, per WHI protocol

Control(n ~26,000)

** Interactive Voice Response System (Consent)

Opt Out: no

Study Design: Zelen’s randomized consent design

Zelen, M. The New England Journal of Medicine 1979; 300: 1242–1245.

Sources of outcomes data

• Self-reported health events annually, for all WHI participants• Fully documented, adjudicated outcomes for the Medical Records

Cohort• Outcomes derived from Medicare claims and NDI among women from

the Self-Report Cohort

Considerations in using Medicare claims data for outcomes assessment• Coverage• Types of data available for outcomes assessments• Quality of inference• Logistics

Medicare data routinely available

• Denominator• Inpatient (MedPar)• Outpatient• Home Health• Skilled Nursing• Hospice• Durable Medical Equipment• Carrier• Part D—Prescription Drug

WHI Medicare linkage

• Submitted 151,116 names, all with valid SSN• 140,471 (93%) were age eligible as of 12/31/07 or had already died at age>65 • 142,195 names returned by Center for Medicare/Medicaid Services (CMS)• 132,109 perfect matches on 5 identifiers (94% of submitted, age-eligible)• 3,203 “fuzzy” matches (2% of submitted age eligible)

• Small discrepancies in one identifier

• 90% linkage of submitted participants • 96% linkage of age eligible participants

Note: All WHI participants are now over age 65

Linkage to Medicare does not mean claims data are available• Medicare claims not available for Managed Care (MC) members

• Included in denominator file but because reimbursement is capitated, MC organizations don’t submit claims

• MC penetrance varies over time and geographic region • Individuals may change their coverage type over time

WHI participants’ Medicare enrollment status by calendar year

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20120

20000

40000

60000

80000

100000

120000

140000

Any CMSFFS A+BMC

Calendar Year

WHI

Par

ticip

ants

Individuals can change their type of coverage• ~98% kept the same type of coverage over a one year period

(1/1/2011-1/1/2012)• 64% had Fee-For-Service plans (Part A or A+B)• 34% were in Managed Care

• Of the 2.12% that change types of coverage,• 0.55% went from Managed Care to Fee-For-Service• 1.57% went from FFS to MC

Determining “events” in claims data

• Claims include up to 10 diagnosis codes based on ICD-9-CM• One “primary” diagnosis• Up to 9 secondary diagnosis

• “Primary” designation and number of secondary diagnoses included is determined by the submitting institution

Assessing agreement rates between WHI adjudicated events and Medicare claims-based diagnoses

• Eligibility:• Age ≥ 65 at WHI enrollment• Linked to Medicare and continuously enrolled in fee-for-service Medicare Part

A (in-patient)

• WHI events• Documented, centrally adjudicated (hospitalized) cardiovascular outcomes• Only first post-enrollment event of each type considered

Chronic Disease Warehouse provides a catalog of algorithms for diagnoses• Myocardial infarction: hospital discharge codes (410.x0, 410.x1) as

either primary or one of 9 secondary discharge diagnosis codes. • Coronary revascularization: ICD-9-CM procedures codes for CABG

(36.1x, 36.2) or PCTA (00.66, 36.0, 36.00, 36.01, 36.02, 36.05, 36.97)

• Stroke: ICD-9-CM codes (430.xx, 431.xx, 433.x1, 434.x1, 436.xx, 437.1x, and 437.9x) in any diagnostic position

• Abdominal aortic aneurysm ICD-9-CM diagnosis (441.3-441.5, 441.9) or procedure (38.34,38.44,39.25,39.52,39.71) or CPT codes (35081, 35082, 35102, 35103, 35091, 0001T, 0002T, 35800, 34802-34805, 34830-34832)

Research Data Assistance Center (ResDAC.org) is an excellent resource for working with CMS data

Agreement rates for acute MI using principle diagnosis code

WHIMedicare Yes No TotalYes 914 281 1195No 431 35771 36202 K (95% CI)Total 1345 36052 0.71 (0.69-0.73)

Sens 68%

Spec 99.2%

PPV 76%

NPV 98.8%

Overall accuracy

98.1%

Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

Sensitivity improves but PPV declines using principle or secondary diagnosis codes for aMI

WHIMedicare Yes No TotalYes 1062 439 1501No 283 35613 35896 K (95% CI)Total 1345 36052 0.74 (0.72-0.75)

Sens 79%

Spec 98.8%

PPV 71%

NPV 99%

Overall accuracy

98.0%

Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

Agreement rates for procedures are higher: Coronary bypass graft surgery

WHIMedicare Yes No TotalYes 795 96 891No 53 36549 36506 K (95% CI)Total 848 36549 0.91 (0.90-0.93)

Sens 94%

Spec 99.7%

PPV 89%

NPV 99.9%

Overall accuracy

99.6%

Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

Agreement rates: Stroke events

WHIMedicare Yes No TotalYes 478 318 796No 104 30817 30603 K (95% CI)Total 582 30817 0.91 (0.90-0.93)

Sens 82%

Spec 99.0%

PPV 60.1%

NPV 99.7%

Overall accuracy

98.7%

Lakshminarayan et al., Stroke (2014)

Medicare up-coding or missing WHI events

WHI No/ Medicare Yes

PPV after Adjustment

All 318 60.1%No self-reported, hospitalized event

182 (57%) 77.9%

No adjudication 57 (18%) 85.8%Adjudication did not identify stroke

79 (25%)

Lakshminarayan et al., Stroke (2014)

Sensitivity and PPV improve when looking at agreement rates at the person level: Stroke

WHIMedicare Yes No TotalYes 505 240 745No 77 30817 30603 K (95% CI)Total 582 30817 0.91 (0.90-0.93)

Sens 86.8%

Spec 99.0%

PPV 67.8%

NPV 99.7%

Overall accuracy

98.7%

Lakshminarayan et al., Stroke (2014)

Agreement rates for cancer incidence

Site WHI-Yes CMS-Yes

WHI-No CMS-Yes

WHI-No CMS-Yes

WHI-No CMS-No

Sens (%)

Spec (%)

PPV (%)

NPV (%)

Overall

Breast 3451 1562 56 98302 98.4 98.4 68.8 99.94 98.4

Colorectal 1145 613 39 105947 96.7 99.4 65.1 99.96 99.4

Endometrial 621 146 17 61642 97.3 99.8 81.0 99.97 99.7

Lung 1236 634 63 105468 95.2 99.4 66.1 99.99 99.4

Melanoma 357 785 22 105355 94.2 99.3 31.3 99.98 99.2

Ovarian 353 372 19 86168 94.9 99.6 48.7 99.98 99.6

Medicare derived cancers based on presence of relevant ICD-9 cods in MedPar (in patient data), any position, or the first occurring combination of 2 outpatient or carrier claims containing these codes that are 1-365 days apart.

Analyzing outcomes found in claims but not in WHI

Breast (n=1562)

Colorectal (N=613)

Endometrial (N=146)

Ovarian (N=372)

N(%) N(%) N(%) N(%)Related diagnosis 888 (56.9) 210 (34.3) 62 (42.5) 199 (53.5)Other cancer orhysterectomy reported

292(18.7) 130 (21.2) 47 (32.2) 91 (24.5)

Other non-cancer hospitalization reported

199 (12.7) 148 (24.1) 11 (7.5) 41 (11.0)

Cancer self-reported & denied, or not adjudicated

95 (6.1) 85 (13.9) 19 (13.0) 24 (6.5)

No information 88 (5.6) 40 (6.5) 7 (4.8) 17 (4.6)

Correspondence of event dates

aMI Stroke AAA LE PAD CASExact same date 82% 83% 89% 75% 90%Within ± 1 day 88% 89% 79% 93%Within ± 7 days 95% 89% 82% 96%Within ± 30 days 94% 96% 92% 85% 96%Exactly ± 365 days 0.7% 0.4%

Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)Lakshminarayan et al., Stroke (2014)Mell et al., J of Vascular Surgery (2014)

Summary of comparisons between claims-based and protocol-defined events in sample with continuous FFS Medicare enrollment

• More events counted in Medicare than in WHI across all conditions• Agreement rates ranged from good to excellent for clinical diagnoses• Excellent agreement rates for procedures• Accuracy of dates is acceptable for most failure time analyses• Errors in both sources contribute to the disagreements

Is outcome misclassification in an RCT benign?

Existing methods for mismeasured outcomes focus on discrete proportional hazards• Halloran ME and Longini IM. Using validation sets for outcomes and exposure to

infection in vaccine field studies. Am J of Epimiol 2001• Meier AS, Richardson BA, and Hughes JP. Discrete proportional hazards models

for mismeasured outcomes. Biometrics 2003• Magaret AS. Incorporating validation subsets into discrete proportional hazards

models for mismeasured outcomes. Statist in Med 2008

Comparing RCT results using adjudicated and claims based outcomes• Sample: Women > 65 years of age at randomization in either the HT

trial component• ITT analyses comparing HT to placebo

• WHI adjudicated outcomes, with censoring at last-follow-up date or death from other causes

• Claims-based outcomes, with censoring at end of enrollment in FFS Medicare, death from other causes, or 12/31/2007 (last available claims data)

Effect of misclassification of failure time events on inference in a RCT setting: WHI HT trial

Click icon to add picture

Hlatky et al., Circulation: Cardiovasc Qual Outcomes (2014)

Medicare found fewer clinical diagnoses, more procedures, similar hazard ratios and overall inference

Assumptions:

Independent measurement error: • Errors in outcomes data are comparable across treatment arms• May not be defensible if treatment leads to symptoms the result in greater

physician contact/diagnostic procedures, etc.

Outcomes information from secondary source (claims) does not affect the failure risk, given information on true failure time and treatment assignment

Some impracticalities of using claims data for outcomes surveillance

Events data collection is not defined by protocol• Study outcomes primarily limited to ICD-9 diagnoses or procedures• Diagnoses represent the community standard(s)• Changes in diagnostic procedures/codes occur outside of the

protocol, may be driven by economic factors• Data availability is at the mercy of another agency, its policies and

practices and those of the institutions who submit data to it

Trial monitoring complexities

• Will data be available soon enough for monitoring purposes?• Annual installments, based on calendar year• Timeline for release is not guaranteed

• Changes in data file structure over time• Requires additional processing time• Difficulty in locking the code for definitions

Outcomes collection and analysis plan

• For participant in the Medical Records Cohort, document and adjudicate all key outcomes

• For the remaining participants in the Self Report cohort, use Medicare Part A/B

• Documenting/adjudicating events for those who switch to managed care

• Analyses based on a stratified Cox Proportional Hazards model with source of data as one of the stratification factors

Monitoring plan considerations

• Intervention is currently available to all (including the comparison group)

• Ethical requirements to stop for efficacy are reduced• Need a full-scale evaluation of this public health program

• Safety concerns are dominant• Medicare may be too tardy, insensitive to monitor safety

• Options: Supplement Medicare data with self-report, particularly in the early phases

Conclusions

• Claims data provide a key source of selected outcomes data that are standardized and available across the nation for a large segment of the population

• The correspondence between claims derived outcomes and traditionally documented and adjudicated outcomes are good to excellent but vary by type of outcome

• Randomized trials using claims data for outcomes may derive valid inference if measurement error assumptions are met

• Timeliness of obtaining claims data may not be adequate for trial monitoring

Dale BurwenRoss PrenticeMary PettingerRoberta RayJoseph LarsonCharles KooperbergAndrea LaCroixMarcia Stefanick

Acknowledgement of colleagues and collaborators