1 EPI235: Epi Methods in HSR May 3, 2007 L10 Outcomes and Effectiveness Research 4: HMO/Network (Dr. Schneeweiss) Methodologic issues in benchmarking physician.

1

EPI235: Epi Methods in HSR

May 3, 2007 L10

Outcomes and Effectiveness Research 4: HMO/Network (Dr. Schneeweiss)

Methodologic issues in benchmarking physician and hospital performance. Analysis of patient data clustered in physicians and in clinics using hierarchical (multi-level) models. Reporting benchmarking results.

Background reading: Austin PC, Goel V, van Walraven C. An introduction to multilevel regression models. Can J Public Health 2001;92:150-154.Austin PC, Tu JV, Alter DA: Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: Should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003;145:27-35.Carey K: A multilevel modeling approach to analysis of patient costs under managed care. Health Econ 2000;9:435-446 .

2

3

4

Pennsylvania Consumer Report

www.phc4.org

5

6

7

8

9

10

11

Quality improvement by quality measurement in NY

0

1

2

3

4

5

6

7

8

9

1989 1990 1991 1992

Year

Ris

k-ad

just

ed m

ort

alit

y .

<50 OPs

50-100 OPs

100-150 OPs

>150 OPs

Hannan ED et al. JAMA 1995

12

Risk factor reporting in the NY Bypass Reporting System (%)

Risk factor 1989 1990 1991

Renal insufficiency 0.4 0.5 2.8

CHF 1.7 2.9 7.1

COPD 6.9 12.4 17.4

Unstable Angina 14.9 21.1 21.8

Low EF (<40%) 18.9 23.1 22.2

13

Multilevel Modeling in Health Services Epidemiology or Hierarchical Modeling or Random Effects Modeling (mixed effects…) or Variance Components Analysis Major issue in health services research: Data clustered at multiple levels: Patients clustered by physician (provider) Physicians clustered by facility (group practice, hospital) Hospitals clustered by region Classical Epi examples: Patients clustered in studies (meta-analysis) Nutrients, food items, and meals

14

Covariates: patient level (severity, comorbidity), provider level (training, experience), facility level (teaching hospital, practice guidelines), regional level (salary structure)

Cluster sizes may vary substantially Example: Patient costs increase with age Outcome: annual costs per patient Exposure: patient age Covariates: Patient level (1): gender Physician level (2): gender, practice volume -> Two level model What do we want to estimate?

1. Patient-level (covariate) effects adjusted for clustering 2. Cluster specific measures of utilization or outcomes adjusted for patient-

level risk factors (case-mix adjustment) HSR: #2 = benchmarking

15

Aggregated analysis versus individual level analysis Aggregated analysis results in loss of power Individual level analysis ignores the structure of nesting within organizations: -> Ordinary least squares regression (OLS):

individual outcome and covariate data are mixed with higher level covariates.

Such 'intra-class correlation' violates the OLS assumption of random errors, i.e. independent observations with common variance In nested data the error term is composed of an individual-level and an group-level variance component. Disregarding the correlation of observations leads to underestimates of OLS standard errors and thus overestimates of the significance -> false 'positive' results. Multilevel modeling is most efficient and provides unbiased estimates of standard errors.

No Clusters

Clusters but No Clustering

Clusters with much Clustering

Extreme Clustering: Pure Lumps

16

17

CABG Mortality among 55 hospitals

MLwiN®

L e v e l o n e m o d e l :

iii exy 10 i = i n d i v i d u a l s ; j = p r o v i d e r L e v e l t w o : V a r i a t i o n i n i n t e r c e p t i s p r e d i c t e d b y

jj u 000

S u b s t i t u t e i n l e v e l o n e m o d e l :

ijjijjij euxy 010

V a r i a n c e c o m p o n e n t s ! F i x e d p a r a m e t e r

R a n d o m p a r a m e t e r

T h e m o d e l i s m a d e t w o - l e v e l b y a l l o w i n g t h e l i n e s f o r a l l h o s p i t a l s t o v a r y a r o u n d t h e m e a n i n t e r c e p t (

0 ) b y a n a m o u n t u 0 j , w h i c h i s n o r m a l l y d i s t r i b u t e d a r o u n d t h e m e a n .

T h e s e d e p a r t u r e s f r o m t h e a v e r a g e a r e k n o w n a s t h e l e v e l 2 r e s i d u a l s . T h e y a r e i n t e r p r e t e d a s t h e h o s p i t a l e f f e c t .

T h e t r u e l e v e l 2 r e s i d u a l s a r e u n k n o w n , b u t w e c a n e s t i m a t e t h e m w i t h t h e i r 9 5 % c o n f i d e n c e l i m i t s . T h e s e e s t i m a t e s c a n b e p l o t t e d f o r a g r a p h i c a l c o m p a r i s o n o f h o s p i t a l s .

18

19

20

Mixed effect models: Patient level covariates = random effects, provider level covariates = fixed effects SAS for continuous (linear) outcomes: proc mixed data=temp; class doc_id; model cost= pat_age pat_sex doc_sex doc_volume /solution; random intercept /subject* = doc_id; * complete independence is assumed between subjects random intercept age /subject* = doc_id; Non-linear mixed effects models: e.g. binary outcomes, count outcomes SAS V8: proc nlinmixed Expansion to more than two levels. MLwin can handle up to 15 levels.

21

SAS data format:

Patient characteristics Hospital char. Patient Hospital DCG Age gender Teaching MC

1 1 … … M 1 0.56 2 1 M 1 0.56 3 1 F 1 0.56 4 1 M 1 0.56 5 1 F 1 0.56 6 1 M 1 0.56 7 1 M 1 0.56 8 1 M 1 0.56 9 1 M 1 0.56 10 1 M 1 0.56 11 1 M 1 0.56 1 2 M 0 0.78 2 2 F 0 0.78 3 2 M 0 0.78 1 3 M 1 0.60 2 3 M 1 0.60 3 3 M 1 0.60 4 3 M … 0.60 5 3 M 0.60 6 3 F … 7 3 … 8 3 9 3 10 3 11 3 12 3 13 3 14 3

22

23

24

25

26

What is the best comparator for benchmarking?1) Average of all institutions 2) Stratified by institution characteristics

Austin et al. Am Heart J 2004

27

Class Exercise Background: Some previous literature has shown that hospitals that initiate education programs for patients in management of chronic stable angina have shown improved outcomes. However, hospitals that have initiated such programs tend to differ from facilities that do not implement such programs. Objective: To evaluate a group outpatient educational intervention to improve outcomes in stable angina. Specific Aims: (1) To determine whether a group outpatient education program on the management of chronic stable angina improves patient health as measured by the Seattle Angina Questionnaire (SAQ). (2) To determine what patient characteristics and facility characteristics are associated with the success of such an intervention. Resources: 30 hospitals have agreed to participate. The intervention/ non-intervention selection can only be done on a facility level, because if some patients are receiving special group education, all physicians will want their patients to participate. Therefore, we cannot randomize on a patient level! The facilities will not be self-selecting into the intervention/ non-intervention arms. Rather, the researcher (you) must decide how to allocate the educational intervention. Primary data collection on patient health status (SAQ) will be used to measure outcomes to assess the effect of the intervention. Complete during and post-intervention data, both administrative and primary data are available for all facilities. Individual participation in the group education program (i.e. percentage of sessions attended) will be recorded for those individuals in the intervention facilities. Patient characteristics will also be available for some variables, including age, gender, race, and presence of supplemental insurance. Patients’ incomes will not be known, but their zipcode will be. Facility characteristics will be available for a large number of variables including size, teaching hospital status, financial ownership status, average demographics of patients, presence of cardiology programs, staff specialties, competitive environment, expenditures per patient and average distance of patients’ zip codes to the hospital.

28

Dealing with clustering

Generalized estimating equations (GEE) SAS: proc genmod

Random effects modeling (multi-level modeling)

29

Effect of physician specialty on patient outcomes

Attending cardiology specialty

Austin et al. Am Heart J 2003

30

Teaching hospital

31

GEE vs. random effects model

Twisk, Eur J Epi, 2004

Expl.: Risk factors of increased cholesterol levels

32

-6 -4 -2 0 2 4 6

-6-4

-20

24

6

Variance of Z= 1

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

-6-4

-20

24

6

Variance of Z= 2

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

-6-4

-20

24

6

Variance of Z= 3

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

-6-4

-20

24

6

Variance of Z= 4

x

Expecte

d V

alu

e Y

GEE

Rand. intercept

33

-6 -4 -2 0 2 4 6

0.0

0.2

0.4

0.6

0.8

1.0

Variance of Z= 1

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

0.2

0.4

0.6

0.8

Variance of Z= 2

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

0.2

0.4

0.6

0.8

Variance of Z= 3

x

Expecte

d V

alu

e Y

-6 -4 -2 0 2 4 6

0.2

0.4

0.6

0.8

Variance of Z= 4

x

Expecte

d V

alu

e Y

GEE

Rand. intercept

34

Final Note: Parameter Interpretation

Marginal parameters are often thought to be more appropriate for policy questions, “What is the difference in the rate of illness in the treated population versus the untreated?”

Conditional parameter estimates closer to individual effects, “What is the effect of treatment in an individual person?”

Conditional parameter are more clinically relevant?