1 Department of Biomedical Informatics Department of Biomedical Informatics University of Utah University of Utah Practice-Based Evidence: Practice-Based Evidence: A New Paradigm for Comparative Effectiveness A New Paradigm for Comparative Effectiveness Research Research October 23, 2014 October 23, 2014 by by Susan D. Horn, Ph.D Susan D. Horn, Ph.D Institute for Clinical Outcomes Research Institute for Clinical Outcomes Research 699 East South Temple, Suite 215 699 East South Temple, Suite 215 Salt Lake City, Utah 84102 Salt Lake City, Utah 84102 801-466-5595 (V) 801-718-9149 (C) 801-466-5595 (V) 801-718-9149 (C) [email protected][email protected]www.isisicor.com
52
Embed
1 Department of Biomedical Informatics University of Utah Practice-Based Evidence: A New Paradigm for Comparative Effectiveness Research October 23, 2014.
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
1
Department of Biomedical Informatics Department of Biomedical Informatics University of UtahUniversity of Utah
Practice-Based Evidence:Practice-Based Evidence:A New Paradigm for Comparative Effectiveness ResearchA New Paradigm for Comparative Effectiveness Research
October 23, 2014October 23, 2014
byby
Susan D. Horn, Ph.DSusan D. Horn, Ph.DInstitute for Clinical Outcomes ResearchInstitute for Clinical Outcomes Research
699 East South Temple, Suite 215 699 East South Temple, Suite 215 Salt Lake City, Utah 84102Salt Lake City, Utah 84102
Post-Stroke Rehabilitation StudyPost-Stroke Rehabilitation Study
Trans-Disciplinary Project Clinical TeamTrans-Disciplinary Project Clinical Team
6
Post-Stroke Rehabilitation StudyPost-Stroke Rehabilitation Study
Examples of OUTCOME VARIABLESExamples of OUTCOME VARIABLES
• Change in FIM scoreChange in FIM score
• Length of rehab stayLength of rehab stay
• Discharge disposition Discharge disposition
• ContractureContracture
• DeathDeath
• Deep vein thrombosisDeep vein thrombosis
• Major bleedingMajor bleeding
• Pulmonary embolismPulmonary embolism
• Pressure ulcerPressure ulcer
• PneumoniaPneumonia
7
Post-Stroke Rehabilitation StudyPost-Stroke Rehabilitation Study
Examples of PROCESS VARIABLESExamples of PROCESS VARIABLES
• MedicationsMedications
• Nutritional processNutritional process
• Pain managementPain management
• Time to first rehabTime to first rehab
• Intensity, frequency, Intensity, frequency, and duration of PT and duration of PT interventionsinterventions
• Intensity, frequency, Intensity, frequency, and duration of OT and duration of OT interventionsinterventions
• Intensity, frequency, Intensity, frequency, and duration of SLP and duration of SLP interventionsinterventions
• Other therapy Other therapy interventions and interventions and dosagedosage
8
Post-Stroke Physical Therapy FormPost-Stroke Physical Therapy Form
Patient ID:
S a m p l eTherapist:
Date of Therapy Session:
/ /
I N T E R V E N T I O N C O D E SNeuromuscular Interventions:01. Balance training02. Postural awareness03. Motor learning04. PNF05. NDT06. Gait with body weight support07. Involved upper extremity addressed08. Constrained induced movement therapyMusculoskeletal Interventions:09. Strengthening10. Mobilization11. PROM/Stretching12. Manual Therapy13. Motor ControlCardiopulmonary Intervention:14. Breathing15. Aerobic/Conditioning exercisesCognitive/Perceptual/Sensory Interventions:16. Cognitive training17. Perceptual training18. Visual training19. Sensory trainingEducation Interventions:20. Patient21. Family/Caregiver22. StaffEquipment Interventions:23. Prescription/Selection24. Application25. Fabrication
26. OrderingModality Interventions:27. Electrical Stimulation28. Biofeedback29. UltrasoundPet Therapy:30. Use of dog31. Use of other animalAssistive Device:32. Ankle dorsi flex assist33. Cane - Large base34. Cane - Small base35. Cane - Straight36. Crutches - Axillary37. Crutches - Forearm38. Crutches - Small base forearm39. Dowel40. Grocery cart41. Hemirail42. Ironing board43. KAFO44. Lite gait
45. Mirror46. Parallel bars
Pre-Functional Activity
Duration of Activity:Enter in 5 minute increments.
Interventions:Enter one intervention code per group of boxes.
Group Physical Therapy Time:PT Group/Dovetail: minutes
Patients Therapists Assistants
Enter the number of each that participated in the Group PT:
Aides/Techs Students
Physical Therapy Rehabilitation Activities
Physical Therapy Time:PT Assistant
minutes
PT Aide/Tech
minutes
PT Student
minutes
to functional activity
to functional activity
Time session begins:
:
Physical Therapist
minutes
6649
Traumatic Brain InjuryTraumatic Brain Injury
What treatments are associated with better What treatments are associated with better outcomes at routcomes at rehabilitationehabilitation discharge for discharge for patients with traumatic brain injury?patients with traumatic brain injury?
9
10
Children Hospitalized with RSVChildren Hospitalized with RSV
What treatments are associated with better What treatments are associated with better outcomes for children hospitalized with RSV outcomes for children hospitalized with RSV infections, controlling for child differences? infections, controlling for child differences?
We have Efficacy trials that determine whether an intervention produces a specified result(s) under well controlled conditions in a selected population – includes randomized controlled trials (RCTs).
We need Effectiveness trials that measure outcomes of an intervention under “real world” conditions in an unselected clinical population. Hypotheses and study designs for an effectiveness trial are formulated based on conditions of routine clinical practice and on outcomes essential for clinical decisions.
What We Have and What We Need
11
12
Databases for Effectiveness Databases for Effectiveness TrialsTrials
• RCT databasesRCT databases
• Large claims databases, Large claims databases, e.g., Medicare, Medicaid, CDCe.g., Medicare, Medicaid, CDC
• HMO or VA databases from claims and electronic HMO or VA databases from claims and electronic medical recordsmedical records
• Specific condition registries, such as arthritis Specific condition registries, such as arthritis registryregistry
• Practice-based evidence study registriesPractice-based evidence study registries PBE studies overcome limitations of RCTs (that limit patient types and PBE studies overcome limitations of RCTs (that limit patient types and
treatments)treatments)
More detailed patient, process, and outcome evaluation than is More detailed patient, process, and outcome evaluation than is possible with traditional registries or large claims datasetspossible with traditional registries or large claims datasets
13
RCTRCT Databases Databases for Comparative Effectiveness Researchfor Comparative Effectiveness Research
• RCT databases RCT databases – gold standard for efficacy and value – gold standard for efficacy and value of interventionsof interventions
• Advantages includeAdvantages include::
High internal validityHigh internal validity
Causal inferences can be made since patients with Causal inferences can be made since patients with knownknown confounders are excluded and randomization confounders are excluded and randomization eliminates eliminates unknownunknown confounders confounders
14
RCTRCT Databases Databases for Comparative Effectiveness Research for Comparative Effectiveness Research (cont)(cont)
• Limitations includeLimitations include::
Small sample sizes – too small to detect uncommon risksSmall sample sizes – too small to detect uncommon risks
Follow-up periods too short to assess long-term Follow-up periods too short to assess long-term benefits/risksbenefits/risks
Higher-risk patients are excluded typically; limited Higher-risk patients are excluded typically; limited external validityexternal validity
Level of monitoring is more rigorousLevel of monitoring is more rigorous than done in routine than done in routine practicepractice
High rates of treatment discontinuationHigh rates of treatment discontinuation
15
Non-RCT DatabasesNon-RCT Databases for Comparative Effectiveness Research for Comparative Effectiveness Research
• Advantages includeAdvantages include::
Cover thousands to millions of people including minority and elderlyCover thousands to millions of people including minority and elderly
Ability to provide treatment exposures and adverse events, including Ability to provide treatment exposures and adverse events, including hospitalizations and mortality, over extended periods of timehospitalizations and mortality, over extended periods of time
Provide population and subpopulation-based estimates for various Provide population and subpopulation-based estimates for various outcomesoutcomes
Better suited to evaluate safety as opposed to effectivenessBetter suited to evaluate safety as opposed to effectiveness
Many exist in electronic form so some data elements may be exported Many exist in electronic form so some data elements may be exported for use in comparative effectiveness research analysesfor use in comparative effectiveness research analyses
16
Non-RCT DatabasesNon-RCT Databases for Comparative Effectiveness Research for Comparative Effectiveness Research
• Limitations includeLimitations include::
Restricted ability to capture patients’ severity of illness, Restricted ability to capture patients’ severity of illness, functional and cognitive status, health behaviors (other than functional and cognitive status, health behaviors (other than smoking), pain, etc. These can be important unmeasured smoking), pain, etc. These can be important unmeasured confoundersconfounders
Restricted access for CER unless researcher is part of Restricted access for CER unless researcher is part of organization that owns the dataorganization that owns the data
Often required variables are in text format so are not Often required variables are in text format so are not exportableexportable
• Minimizing Bias in drawing conclusions from existing databases
• Capturing variation in patients
• Capturing variation in interventions/ intervention combinations
• Capturing variation in outcomes
Issues to Address in CER Studies
17
Basic Problem in Basic Problem in Non-Randomized Studies Non-Randomized Studies
• Confounding by IndicationConfounding by Indication« Therapies are administered in non-random fashionTherapies are administered in non-random fashion
« Prognostic characteristics influence therapy usedPrognostic characteristics influence therapy used
« Recipients of therapy are at high risk for outcomesRecipients of therapy are at high risk for outcomes
« Users differ from non-users in key respectsUsers differ from non-users in key respects
18
Propensity Score Theory to Address Propensity Score Theory to Address Confounding by IndicationConfounding by Indication
• PS is a multivariable scoring method that collapses PS is a multivariable scoring method that collapses multiple multiple observedobserved predictors of treatment into a single predictors of treatment into a single value (a probability score)value (a probability score)» PS represents the probability that a subject with PS represents the probability that a subject with
given characteristics receives specified treatmentgiven characteristics receives specified treatment» Used to: match, stratify, or modelUsed to: match, stratify, or model» Assumption is by matching on propensity score, it Assumption is by matching on propensity score, it
removes confounding by components included in the removes confounding by components included in the scorescore
» Sensitive to unobserved predictor variables such as Sensitive to unobserved predictor variables such as missing severity of illness measuresmissing severity of illness measures
19
Overcoming Selection Bias/ Overcoming Selection Bias/ Confounding by IndicationConfounding by Indication
• Statistical adjustments:Statistical adjustments:–MatchingMatching–Propensity score or instrumental variablesPropensity score or instrumental variables–Covariate adjustments (Severity of Illness)Covariate adjustments (Severity of Illness)
• Ongoing debate about the adequacy of Ongoing debate about the adequacy of adjustmentsadjustments
20
What makes this approach What makes this approach different? Why do a PBE study?different? Why do a PBE study?
PBE Methodology
21
22
History of PBEHistory of PBE
• Started with development of Comprehensive Severity Index (CSI) Started with development of Comprehensive Severity Index (CSI) to measure risk-adjusted outcomes at The Johns Hopkins to measure risk-adjusted outcomes at The Johns Hopkins HospitalHospital
• CSI found to explain up to 50% of the variation in outcomes of CSI found to explain up to 50% of the variation in outcomes of cost, LOS, mortalitycost, LOS, mortality
• Found patients with the similar CSI scores for a condition could Found patients with the similar CSI scores for a condition could have very different treatmentshave very different treatments
• This stimulated development of PBE study design to account forThis stimulated development of PBE study design to account for patient heterogeneitypatient heterogeneity treatment heterogeneitytreatment heterogeneity outcome heterogeneityoutcome heterogeneity
• Both patients and providers report data
• Data come from existing EMR with standardized structured data elements about patient characteristics, treatments, processes, patient-reported data, and multiple outcomes
• Data are part of routine documentation, so not an ‘add-on’
• Rapid patient accrual since documentation is standard of care
• Longitudinal and ongoing
CER Issues Addressed Using PBE
23
• Patient comparability is addressed with the Comprehensive Severity Index (CSI): disease-specific, physiologic-based, >2,200 criteria, >5,500 disease-specific criteria sets
• CSI addresses confounding by indication and selection bias
• Database includes all treatments with date/dose/intensity/route. Many details collected in point-of care (POC) documents.
• Can assess drug and non-drug combination therapies
• Findings of PBE-CER are more readily translated into practice
CER Issues Addressed Using PBE (cont)
24
Components of Practice-Based Evidence Designs
Process Factors• Patient Education and Management Strategies• Interventions and surgeries • Medications
Patient Factors• Psychosocial/demographic Factors• Co-occurring Conditions• Severity of Illness and Injury• Genetic informationGenetic information• Measured at Multiple Points in Time
7 Signature Features of 7 Signature Features of PBE StudiesPBE Studies
1.1. Hypotheses can be focused or broadHypotheses can be focused or broad
2.2. All interventions are considered to determine the All interventions are considered to determine the relative contribution of each relative contribution of each
3.3. Broad patient selection criteria maximize Broad patient selection criteria maximize generalizability and external validitygeneralizability and external validity
4.4. Detailed characterization of the patient by robust Detailed characterization of the patient by robust measures of patient severity, genetic information, measures of patient severity, genetic information, and functional statusand functional status
26
7 Signature Features of 7 Signature Features of PBE StudiesPBE Studies
5.5. Patient differences controlled statistically rather than Patient differences controlled statistically rather than through randomization through randomization
6.6. Facility and clinical/patient buy-in through use of Facility and clinical/patient buy-in through use of trans-disciplinary Clinical Practice Teamtrans-disciplinary Clinical Practice Team
7.7. Strength of evidence built through the research processStrength of evidence built through the research process
PBE findings are more PBE findings are more generalizablegeneralizable and and transportabletransportable than RCT findingsthan RCT findings 27
PBE Study HallmarksPBE Study Hallmarks
• Decisions are made by Decisions are made by front-line clinicians front-line clinicians vs. vs. researchersresearchers
• ““Bottom-upBottom-up” vs. “Top-down” approach” vs. “Top-down” approach
• Guidance from researchers (scientific Guidance from researchers (scientific advisory board) and patient experienceadvisory board) and patient experience
28
PBE Study HallmarksPBE Study Hallmarks
–Non-experimentaNon-experimentall: Follows outcomes of : Follows outcomes of treatments actually prescribedtreatments actually prescribed
–PragmaticPragmatic: Uses actual clinical outcomes: Uses actual clinical outcomes
–Lower CostLower Cost than RCTsthan RCTs
–FasterFaster than RCTsthan RCTs29
Practice-Based Evidence Study DesignPractice-Based Evidence Study Design
• High external validityHigh external validity
- Includes essentially all patients with specific Includes essentially all patients with specific condition or in specific setting(s)condition or in specific setting(s)
- Captures confounders that could affect relevant Captures confounders that could affect relevant treatment responses treatment responses
- Reduces accidental associations between Reduces accidental associations between treatments and outcomestreatments and outcomes
30
Components of Practice-Based Evidence Designs
Process Factors•Patient Education and Management Strategies•Interventions and surgeries •Medications
Patient Factors•Psychosocial/demographic Factors•Co-occurring Conditions•Severity of Illness and Injury•Measured at Multiple Points in Time
Primary Outcomes•Change in CSI/discharge CSI•Discharge Disposition•Length of Stay•Post-discharge Outcomes
Standardize documentation for :
Control for:
Measure:
31
Examples of Severity Systems to address Examples of Severity Systems to address Selection Bias/Confounding by IndicationSelection Bias/Confounding by Indication
Diagnostic/Procedure Based Systems• Clinical definition of severity
• Body Systems Count • Charlson Comorbidity Index (1-yr death)
Comprehensive Severity Index (CSIComprehensive Severity Index (CSI®®)) used to account for selection bias or confounding by indicationused to account for selection bias or confounding by indication
• Severity defined as “Severity defined as “physiologic complexity physiologic complexity presented to medical personnel due presented to medical personnel due to the extent and interactions of a patient’s diseasesto the extent and interactions of a patient’s diseases””
• Disease-specificDisease-specific: : 5,500 disease-specific groups; over 2,200 distinct criteria5,500 disease-specific groups; over 2,200 distinct criteria . . ICD-9 codes trigger disease-specific patient signs, symptoms, and physical findings used to ICD-9 codes trigger disease-specific patient signs, symptoms, and physical findings used to score disease-specific and overall severity levelsscore disease-specific and overall severity levels
• No treatments usedNo treatments used as criteriaas criteria
• Clinically credibleClinically credible: computes disease-specific and overall severity levels: computes disease-specific and overall severity levels
• Can measure severity at Can measure severity at multiple time points multiple time points
• Allows statistical comparison Allows statistical comparison of interventions without confounding by severity of of interventions without confounding by severity of
illnessillness33
34
CSI Severity IndicatorsCSI Severity Indicators
Physiological signs and symptoms of a Physiological signs and symptoms of a diseasedisease- Vital signs- Vital signs
- Laboratory values- Laboratory values
- Radiology findings- Radiology findings
- Other physical findings- Other physical findings
Severity indicators are specific to each Severity indicators are specific to each disease based on ICD-9-CM codingdisease based on ICD-9-CM coding
Coordination/Balance Unsteady on Feet/Clumsiness NOS
Dizziness/Mild to Moderate Ataxia
Severe Ataxia N/A
Sensation alteration Sensation Alteration NOS Complete loss of Sensation/Parathesia or Dysesthesia
N/A N/A
Aphasia No Aphasia Mild Aphasia Moderate/Severe Aphasia Global Aphasia Dysarthria No/mild Dysarthria Dysphonia Dysarthria Incomprehensible Speech No SpeechDysphagia Dysphagia NOS Unable to swallow liquids Unable to swallow solids N/A
+ Advanced gait time in 1time in 1stst 3 hrs 3 hrs
OT OT InterventionsInterventions
+ Home management
SLP SLP InterventionsInterventions
Horn et al., Arch Phys Med Rehabil 2005;86(12 Supplement 2):S101-S114
Policy Changes from Stroke PBE Study
Early Rehabilitation Admission – get patient into
rehabilitation as soon as possible after stroke onset;
possibly start in Neuro ICU
Early gait in PT – start gait as soon as possible after rehab admission; put patient in harness on treadmill for safety
Early Feeding – continue or start enteral nutrition at rehab admission if patient is not able to eat full meals
Use Opioids for Pain – continue or start opioids at rehab admission if patient misses therapy due to pain 41
Traumatic Brain InjuryTraumatic Brain Injury
What treatments are associated with better What treatments are associated with better outcomes at routcomes at rehabilitationehabilitation discharge for discharge for patients with traumatic brain injury?patients with traumatic brain injury?
42
TBI Admission FIM CognitiveTBI Admission FIM CognitiveSubgroups (N=2130)Subgroups (N=2130)
43
Adm FIM Cognitive N during Rehab N at 3 months N at 9 months
Score <=6 339 286 262
Score 7-10 374 312 302
Score 11-15 495 411 394
Score 16-20 408 326 311
Score >=21 504 401 373
Total 2120* 1742** 1649***
Ns decrease due to non-consent for follow up, death, or incarceration
*N=10, **N=6, ***N=7 Missing Admission FIM cognitive score43
Rehab Length of Stay (LoS) Discharge FIM Motor (dcM) Discharge FIM Cognitive (dcC) Discharge to Home (dcH) 9-Month FIM Motor (fuM) 9-Month FIM Cognitive (fuC)
PT Formal Assessment Min/Wk LoS fuM LoS fuM LoS dcH fuM LoS
ST Education Min/Wk dcH LoS dcH fuC LoS dcH dcC dcH dcH
ST Basic Motor/Speech Min/Wk dcM LoS dcM dcC fuM
ST Problem Solving, Math, Money, Memory, Orientation Min/Wk
dcM dcC fuM fuC
dcM dcC fuM
dcM fuM
% Stay Atypical Antipsychotics LoS dcM dcC dcM dcC fuC
Rehab Length of Stay (LOS) Discharge FIM Motor (dcM) Discharge FIM Cognitive (dcC) Discharge to Home (dcH)9-Month FIM Motor (fuM) 9-Month FIM Cognitive (fuC) Red = negative coeff, p<.05 Green = positive coeff, p<.05
All Regressions Summary:Regressions Summary: Treatment Significant Covariates
Red = negative coeff, p<.05; Green = positive coeff, p<.05
48
Children Hospitalized with RSVChildren Hospitalized with RSV
What treatments are associated with better What treatments are associated with better outcomes for children hospitalized with RSV outcomes for children hospitalized with RSV infections, controlling for child differences? infections, controlling for child differences?
ICU LOSICU LOS 5.8 days5.8 days 7.7 days7.7 days 4.2 days4.2 days 3.8 days3.8 days 0.0210.021
Hospital LOSHospital LOS 6.8 days6.8 days 8.4 days8.4 days 4.9 days4.9 days 4.1 days4.1 days <0.0001<0.0001
Admitted to ICUAdmitted to ICU 39.3%39.3% 48.4%48.4% 30.0%30.0% 27.9%27.9% 0.1010.101
HX of Hosp. for HX of Hosp. for RSV /BronchiolitisRSV /Bronchiolitis 14.3%14.3% 16.1%16.1% 6.7%6.7% 6.1%6.1% 0.1370.137
Significant Differences by Gestational Age GroupsSignificant Differences by Gestational Age Groups33-35 week GA infants had highest hospital resource use33-35 week GA infants had highest hospital resource use
51
RSV Hospital Outcomes and Policy ChangesRSV Hospital Outcomes and Policy Changes
ConclusionsConclusions• 33-35 week GA infants had highest hospital resource use33-35 week GA infants had highest hospital resource use
• 36 week infants have risk similar to full term infants36 week infants have risk similar to full term infants
• Changed guidelines for immunoprophylaxis Changed guidelines for immunoprophylaxis for 33-35 week for 33-35 week infantsinfants
• Changed guidelines for intubation Changed guidelines for intubation – try ‘stimulating’ first– try ‘stimulating’ first
52
SummarySummary
• PBE methodology provides a structured way to design PBE methodology provides a structured way to design studies of patients in routine care settingsstudies of patients in routine care settings
• PBE studies develop comprehensive databases of PBE studies develop comprehensive databases of patient, treatment, and outcome differencespatient, treatment, and outcome differences
• PBE findings associated with better outcomes are PBE findings associated with better outcomes are easily transferable for use in other sites because all easily transferable for use in other sites because all patients with a condition can be included in a PBE patients with a condition can be included in a PBE studystudy