Hierarchical Generalized Linear Models for Behavioral ...The final metrics were a hospital-wide risk adjusted 30-day readmission rate percent and a hospital-wide risk adjusted 90-day

1

Hierarchical Generalized Linear Models for Behavioral Health Risk-Standardized

30-Day and 90-Day Readmission Rates

Allen Hom PhD, Optum, UnitedHealth Group, San Francisco, California

Abstract

The Achievements in Clinical Excellence (ACE) program encourages excellence across all

behavioral health network facilities by promoting those that provide the highest quality of care.

Two key benchmarks of outcome effectiveness in the ACE program are the risk adjusted 30-day

readmission and risk adjusted 90-day readmission rates. Risk adjustment was performed with

hierarchical general linear models (HGLM) to account for differences across hospitals in patient

demographic and clinical characteristics. One year of administrative admission data (June 30,

2013 to July 1, 2014) from patients for 30-day (N=78,761, N Hospitals=2,233) and 90-day

(N=74,540, N Hospital =2,205) time frames were the data sources. HGLM simultaneously

models two levels 1) Patient level – models log-odds of hospital readmission using age, sex,

selected clinical covariates, and a hospital-specific intercept, and 2) Hospital level – a random

hospital intercept that accounts for within-hospital correlation of the observed.

PROC GLIMMIX was used to implement a HGLM with hospital as a random (hierarchical)

variable separately for substance use disorder (SUD) admissions and mental health (MH)

admissions and pooled to obtain a hospital-wide risk adjusted readmission rate. The HGLM

methodology was derived from Centers for Medicare & Medicaid Services (CMS)

documentation for the 2013 Hospital-Wide All-Cause Risk-Standardized Readmission Measure

SAS® package. This methodology was performed separately on 30-day and 90-day

readmission data. The final metrics were a hospital-wide risk adjusted 30-day readmission rate

percent and a hospital-wide risk adjusted 90-day readmission rate percent. HGLM models were

cross-validated on new production data that overlapped with the development sample. Revised

HGLM models were tested in April, 2015, and the outcome statistics were extremely similar. In

short, the test of the revised model cross-validated the original HGLM models, because the

revised models were based on different samples.

Background of Hospital Readmissions Reduction Program

Section 3025 of the Affordable Care Act added section 1886(q) to the Social Security Act establishing the Hospital Readmissions Reduction Program, which requires Centers for Medicare & Medicaid Services (CMS) to reduce payments to hospitals with excess readmissions, effective for discharges beginning on October 1, 2012. Hospitals with greater than expected readmission rate are subject to financial penalty. Performance was based on 30-day readmission metrics for three conditions that started in 2013 - acute myocardial infarction, heart failure, and pneumonia.

CMS contracted with Yale New Haven Health Services Corporation/Center for Outcomes

Research and Evaluation (YNHHSC/CORE) to develop a claims-based, risk-adjusted hospital-

2

wide readmission (HWR) measure for public reporting that reflects the quality of care for

hospitalized patients in the United States. The hospital-wide risk-standardized readmission rate

(RSRR) is a summary score derived from the weighted geometric mean of five statistical models

built for groups of admissions that are clinically related: medicine, surgery/gynecology,

cardiorespiratory, cardiovascular, and neurology. For each specialty, an index admission is the

hospitalization to which the readmission outcome is attributed and includes admissions criteria

for patients in the CMS study:

• Must meet Peer Review • Enrolled in Medicare fee-for-service (FFS); • Aged 65 or over; • Discharged from non-federal acute care hospitals; • Without an in-hospital death; • Not transferred to another acute care facility; and, • Enrolled in Part A Medicare for the 12 months prior to the date of the index admission. Excluded were admissions without at least 30 days post-discharge enrollment in FFS Medicare, patients who leave against medical advice, admissions for medical cancer treatment, admission for primary psychiatric diagnoses, and admissions for rehabilitation. Despite encouraging results from Medicare quality improvement interventions since 2008, the overall national readmission rate remains high, with a 30-day readmission following nearly 20% of discharges. RISK ADJUSTMENT METHODOLOGY Readmission Outcome Definition Thirty-Day and Ninety-Day Timeframe The two outcomes are unplanned all-cause 30-day readmission and unplanned all-cause 90-day readmission. A 30-day readmission is a subsequent inpatient admission to any acute care facility which occurs within 30 days of the discharge date of an eligible index admission. A 90-day readmission is a subsequent inpatient admission to any acute care facility which occurs within 90 days of the discharge date of an eligible index admission. All-Cause Readmission Admissions for acute illness or for complications of care are not “planned.” Any procedure completed during an admission for an acute illness is not likely to have been planned, even if that procedure is usually planned in other non-acute cases. The CMS methodology for Hospital-Wide Risk-Standardized Readmission Metrics was applied to unplanned behavioral health acute inpatient readmissions within 30 and 90-days of discharge.

3

Definition of Eligible Admissions

Patient must have continuous eligibility with the health plan for 12 months prior to the initial admission. Eligibility at least one day after the discharge date is required to ensure that the absence of a readmission is not simply due to loss of the behavioral health benefit.

• Must meet Peer Review • For 30-day readmission, the readmission had to be greater than or equal to two days and

less than or equal to thirty days. • For 90-day readmission, the readmission had to be greater than or equal to two days and

less than or equal to ninety days. Exclusion Criteria for 30-day readmission:

• In-hospital death

• Patient is without at least 30 days post‐discharge continuous enrollment (because the 30‐day readmission outcome cannot be assessed in this group).

• Patient is without at least 365 days of continuous enrollment prior to admission start date • Transferred to another acute care facility • Discharged against medical advice (AMA) • Same day discharges

For 90-day readmission: • In-hospital death

• Patient is without at least 90 days post‐discharge continuous enrollment (because the 90‐day readmission outcome cannot be assessed in this group).

• Patient is without at least 365 days of continuous enrollment prior to admission start date • Transferred to another acute care facility • Discharged against medical advice (AMA) • Same day discharges Substance Use Disorder Cohort Behavioral health admissions were divided into two categories that represent major divisions of acute inpatient treatment programs within hospitals: Substance Use Disorder (SUD) and Mental Health (MH) therapy. A substance use disorder describes a problematic pattern of using alcohol or another substance that results in impairment in daily life or noticeable distress. A person with this disorder will often continue to use the substance despite negative consequences. The SUD sample was 19% of the 30-day readmission sample (14,786 out of 78,761 patients, tables 1 and 2) and the 90-day readmission sample (13,993 out of 74,540 patients, tables 3 and 4).

Behavioral Health Cohort

The mutually exclusive MH group was composed of index admissions for all other DSM-5 primary diagnosis categories that were not SUD such as: Depressive Disorders, Bipolar and Related Disorders, Schizophrenia Spectrum and Other Psychotic Disorders, Neurocognitive

4

Disorders, Anxiety Disorders, Disruptive, Impulse-Control and Conduct Disorders, Neuro-developmental Disorders, Feeding and Eating Disorders, and Personality Disorders. The MH

sample was 81% of the 30-day readmission sample (63,975 out of 78,761 patients, tables 5 and 6) and the 90-day readmission sample (60,547 out of 74,540 patients, tables 7 and 8). In short, the MH sample was five times larger than the SUD sample.

Data Sources

Primary and secondary DSM-5 mental health diagnoses were extracted for a year prior to the index admissions from a hospital administrative claims data warehouse. If the primary and secondary prior diagnosis were from the same DSM-5 category, they were combined to create a single indicator DSM-5 risk variable for the past year (see Table 1). Modelling Approach Risk adjustment was performed with hierarchical general linear models (HGLM) that accounts for age, gender, current admissions, mental health diagnoses in the past year, product type (Commercial, Medicaid, Medicare), involuntary admission to acute mental health hospitalization, and current electro-convulsive therapy. Logistic regression was used to screen variables and obtain estimated odd ratios for variables. The goal of risk adjustment is to account for differences across hospitals in patient demographic and clinical characteristics that might be related to the outcome. • Hospital-level 30-day and 90-day all-cause Behavioral Health Risk Standardized

Readmission Rates (RSRR) for SUD and MH readmissions were estimated using Hierarchical Generalized Linear Models (HGLM) to adjust for patient clustering (hierarchically correlated) effects within hospitals.

• HGLM simultaneously models two levels (patient and hospital)

- Patient level – models log-odds of hospital readmission using age, sex, selected clinical covariates, and a hospital-specific intercept.

- Hospital level – a random hospital intercept for Hierarchical General Linear Model (HGLM) that accounts for within-hospital correlation of the observed outcomes and models the assumption that underlying differences in quality among the health care facilities being evaluated lead to systematic differences in outcomes.

Steps in Calculating the Behavioral Health Risk-Standardized 30-Day Readmission Rate A logistic regression (PROC LOGISTIC in SAS®) was performed to screen variables prior to HGLM analysis, obtain fit statistics, odd ratio (OR) estimates, OR 95% confidence limits, and residual statistics. PROC GLIMMIX was used to implement a HGLM with hospital as a random (hierarchical) variable separately for SUD admissions and MH admissions. The HGLM methodology and SAS code were derived from CMS documentation for the 2013 Hospital-Wide All-Cause Risk-Standardized Readmission Measure SAS package that was developed by the

5

Yale-New Haven Health Services Corporation/Center for Outcomes Research & Evaluation. For each model, the GLIMMIX procedure implements an HGLM that outputs the predicted number of admissions for index admissions and the number of expected admissions that will be used in the next step to compute the Standardized Readmission Ratio. Standardized Readmission Ratio

The Standardized Readmission Ratio (SSR) is calculated as the ratio of the predicted number of

admissions to the number of expected readmissions for a given hospital. • For each hospital, the numerator of the SRR is the number of readmissions within 30 days

predicted based on the hospital’s performance with its observed case mix and service mix. • The denominator is the number of readmissions expected based on the performance of an

average hospital with similar case mix and service mix. This approach is analogous to a ratio of “observed” to “expected” used in other types of statistical analyses.

SRRCj = predictCj / expectCj

c = admissions in cohort (SUD, MH) j = hospital

• Two SRRs are computed: SRR_SUD and SRR_MH. 1) A national raw readmission rate for the all SUD admissions is computed. 2) A national raw readmission rate for the all MH admissions is computed. 3) A program-wide RSRR is computed separately for each SUD and MH where:

- SUD RSRR = SUD SSR x SUD national raw readmission rate. - MH RSRR = MH SSR x MH national raw readmission rate.

• To report a single SRR for Hospital-Wide Readmission (SRR_HWR), the SUD and MH

SRRs were pooled to compute a composite hospital volume (admissions) weighted logarithmic mean:

SRRj = exp ( ( Σ mcj log(Rcj)) / Smcj )

c = admissions in cohort (SUD, MH) j = hospital

mcj = admissions per cohort for hospital

Rcj = SRR for the condition

A SRR less than one indicates a lower-than-expected readmission rate (better quality), while a SRR greater than one indicates a higher-than-expected readmission rate (worse quality).

The corresponding SAS code takes into account that some hospitals have a SUD or MH program, but not both:

mod_30day_readmit_admit_cnt = volume_sud + volume_mh; IF volume_sud >0 THEN SUD_NUM = volume_sud * LOG(SRR_SUD); ELSE SUD_NUM=0; IF volume_mh >0 THEN MH_NUM = volume_mh * LOG(SRR_MH); ELSE MH_NUM=0; total_num = sum(sud_num, mh_num); SRR _HWR = exp(total_num/mod_30day_readmit_admit_cnt);

SRR_HWR is the Standardized Readmission Ratio (SRR) for Hospital-Wide Readmission (HWR)

6

Risk Adjusted 30-Day and 90-Day Readmission Percent A national unadjusted average readmission rate is obtained from the combined overall SUD and

MH samples. The composite Facility-wide SRR_HWR is multiplied by the national average readmission rate to produce hospital-wide Risk Adjusted 30-day Readmission Percent (RSK_ADJ_30day_readmit_pct). RSK_ADJ_30day_readmit_pct = (SRR _HWR * &HWYBAR);

&HWYBAR is a SAS macro variable that contains the national unadjusted average readmission rate. The Risk Adjusted 30-day Readmission Percent (e.g., RSK_ADJ_30day_reamit_pct) is equivalent to the CMS metric that is known as Risk Standardized Readmission Rate (RSRR) The entire Risk Adjustment Methodology section (page 2-5) is repeated for 90-day readmission to obtain a hospital-wide Risk Adjusted 90-day Readmission Percent (RSK_ADJ_90day_readmit_pct).

RESULTS Variables in Substance Use Disorder HGLM Analysis Updated 30 and 90-day readmit models were created based upon the new mental health DSM-5 variables, after DSM-4 became obsolete. Index admissions for the current development data were from June 30, 2013 to July 1, 2014. The new models were cross-validated on new production data, which spanned a year from January 1, 2014 to December 31, 2014. There is, however, an overlap of half year of 2014 between the two samples. Slightly more than half of the production sample had patients in common with development sample. Tables 1, 2, 3 and 4 show the variables in the validation production data for Substance Use Disorder HGLM analysis sorted by F values. The top four variables in the 30-day SUD readmit HGLM were:

1. Presence of primary or secondary SUD diagnosis in the past year (F=71.03).

2. Presence of primary or secondary bipolar and related disorder diagnosis in the past year (F=58.54).

3. Presence of primary or secondary depressive disorder diagnosis in the past year (F=49.26).

4. Patient is covered by public sector health insurance - Medicaid/Medicare (F=25.76).

The same variables are in top four for the 90-day SUD readmit HGLM, with different ranking:

1. Presence of primary or secondary SUD diagnosis in the past year (F=119.24).

2. Patient is covered by public sector health insurance - Medicaid/Medicare (F=45.78).



The odds ratio indicates that patients with a substance-related and addictive disorder diagnosis in the past year are twice (OR=2.19 and OR=2.25) as likely to have an inpatient readmission. In

7

addition, the importance of bipolar and depressive disorders suggests that patients may be self-medicating to control negative mood swings. There is also the issue of co-occurring mental illness and substance use disorders. In 2014, a national survey on drug use and health in the United States from the Substance Abuse and Mental Health Services Administration (SAMHSA) found that among adults with diagnosis of SUD in the past year, 39.1% also had a co-occurring diagnosis for mental illness in the past year.

N=14,786 % Acute

Admissions Logistic

Regression HGLM HGLM

Variable Description For Variable OR (95% CL) T Value F Value

Substance Abuse Disorder Risk

Presence of primary or secondary substance-related and addictive disorder DX in the past year

72.70% 2.19 (1.86, 2.58) 8.43 71.03

Bipolar Disorder Risk

Presence of primary or secondary bipolar and related disorder DX in the past year

9.89% 1.85 (1.58, 2.18) 7.65 58.54

Depressive Disorder Risk

Presence of primary or secondary depressive disorder DX in the past year

34.34% 1.64 (1.45, 1.86) 7.02 49.26

Public Sector

1= Medicare / Medicaid 0=Commercial

18.61% 1.69 (1.46, 1.95) 5.08 25.76

Gender female =1 male=0 37.47% 0.77 (0.68, 0.87) -3.81 14.55

Anxiety Disorder Risk

Presence of primary or secondary anxiety disorder DX in the past year

15.01% 1.46 (1.26, 1.69) 3.80 14.45

Schizophrenia Disorder Risk

Presence of primary or secondary schizophrenic spectrum and other psychotic disorder DX in the past year

4.60% 1.37 (1.09, 1.72) 2.61 6.80

Age Category

1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65

See Table 2

0.82 (0.73, 0.92) -2.39 5.73

Table 1: Production Facility Readmit 30-Day Model for SUD

Age Category Frequency Percent

Less than or equal to 12 years 3 0.02%

13-17 years old 123 0.83%

18-25 years old 4,463 30.18%

26-64 years old 9,776 66.12%

Greater than or equal to 65 years 421 2.85%

Total 14,786

Table 2: Age Breakdown for Readmit 30-Day Model for SUD

8

N= 13,993

% Acute Admissions

Logistic Regression HGLM HGLM



Presence of primary / secondary substance-related and addictive disorder DX in the past year

72.49% 2.25 (2.00, 2.53) 10.92 119.24

Public Sector

1= Medicare or Medicaid 0=Commercial

18.26% 1.73 (1.55, 1.94) 6.77 45.78


Presence of primary / secondary bipolar and related disorder DX in the past year

9.83% 1.58 (1.38, 1.80) 6.71 45.07


Presence of primary / secondary depressive disorder DX in the past year

34.34% 1.47 (1.33, 1.61) 6.42 41.16

Age Category

1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65

See Table 4

0.73 (0.67, 0.80) -5.14 26.46



14.92% 1.43 (1.27, 1.60) 4.94 24.38


Presence of primary / secondary schizophrenic spectrum and other psychotic disorder DX in past year

4.6% 1.39 (1.15, 1.67) 3.45 11.89

Other Mental Disorder Risk

Presence of primary / secondary other mental disorder DX in the past year

2.17% 1.54 (1.20, 1.99) 2.55 6.51


Neurocognitive Disorder Risk

Presence of primary / secondary neurocognitive disorder DX in the past year

0.81% 1.06 (0.69, 1.65) 0.38 0.14

Table 3: Production Facility Readmit 90-Day Model for SUD


Less than or equal to 12 years 3 0.02%

13-17 years old 118 0.84%

18-25 years old 4,255 30.41%

26-64 years old 9,210 65.82%

Greater than or equal to 65 years 407 2.91%

Total 13,993

Table 4: Age Breakdown for Facility Readmit 90-Day Model for SUD

9

Variables in Mental Health Production HGLM Analysis Tables 5, 6, 7 and 8 show the variables in the validation production data for Mental Health HGLM analysis sorted by F values. The F values are larger (than SUD HGLM) and odd ratio confidence limits are narrower (and therefore more confidence/reliability) because the MH sample is five times larger than the SUD sample. The top five variables in the 30-day MH

readmit HGLM were:


2. Presence of primary or secondary schizophrenic spectrum and other psychotic disorder diagnosis in the past year (F=181.30).


4. Patient is covered by public sector health insurance - Medicaid/Medicare (F=168.84). 5. Presence of primary or secondary substance-related and addictive disorder DX in the past

year (F=106.23).

The same variables are in top five for the 90-day MH readmit HGLM, but with different rankings:


2. Patient is covered by public sector health insurance - Medicaid/Medicare (F=339.63). 3. Presence of primary or secondary schizophrenic spectrum and other psychotic disorder

diagnosis in the past year (F=338.61). 4. Presence of primary or secondary bipolar and related disorder diagnosis in the past year

(F=305.03). 5. Presence of primary or secondary substance-related and addictive disorder DX in the past

year (F=159.68).

For both SUD and MH 90-day readmission models, the second most important variable is public sector (i.e., Medicare/Medicaid) health insurance. For 30-day readmission models public sector insurance is also in one of the top four predictors. The MH models also show that after mental health problems (bipolar, schizophrenia, and depressive disorders) substance abuse is also an important issue in readmission.

10

N= 63,975

% Acute Admissions





25.64% 1.60 (1.51, 1.70) 13.7 187.65


Presence of primary / secondary schizophrenic spectrum and other psychotic disorder DX in the past year

23.6% 1.55 (1.45, 1.64) 13.46 181.30


Presence of primary or secondary depressive disorder DX in the past year

50.24% 1.46 (1.38, 1.54) 13.27 176.18

Public Sector 1= Medicare / Medicaid 0=Commercial

33.53% 1.57 (1.48, 1.66) 12.99 168.84



19.00% 1.44 (1.36, 1.52) 10.31 106.23


Presence of primary or secondary other mental disorder DX in the past year

6.68% 1.48 (1.37, 1.61) 7.76 60.22



19.00% 1.17 (1.10, 1.25) 4.95 24.48

Age Category

1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65

See Table 6

0.92 (0.89, 0.95) -4.74 22.44

Trauma or Stress-Related Disorder Risk

Presence of primary / secondary trauma or stressor-related disorder DX in the past year

16.08% 1.19 (1.11, 1.27) 4.55 20.71

Feeding and Eating Disorder Risk

Presence of primary secondary feeding and eating disorder DX in the past year

2.00% 1.54 (1.30, 1.83) 4.29 18.43


Personality Disorder Risk

Presence of primary or secondary other personality disorder DX in the past year

2.68% 1.30 (1.14, 1.47) 3.47 12.06

Index Schizophrenia Disorder Risk

Presence of primary schizophrenic spectrum and other psychotic disorder DX

15.98% 1.44 (1.15, 1.80) 2.94 8.64

Impulse-Control and Conduct Disorder Risk

Presence of primary or secondary impulse-control and conduct disorder DX in the past year

4.55% 1.16 (1.03, 1.29) 2.49 6.20

Table 5: Production Facility Readmit 30-Day Model for MH

11

N= 63,975 % Acute

Admissions



Index Obsessive Compulsive Disorder Risk

Presence of primary obsessive-compulsive and related disorder DX

0.20% 1.95 (1.15, 3.29) 2.31 5.35

Index Trauma or Stress-Related Disorder Risk

Presence of primary trauma or stressor-related disorder DX

3.19% 0.83 (0.63, 1.09) -1.5 2.25

Index Anxiety Disorder Risk

Presence of primary anxiety disorder DX

1.42% 0.77 (0.56, 1.07) -1.44 2.07

Index Bipolar Disorder Risk

Presence of primary bipolar and related disorder DX

20.78% 1.18 (0.94, 1.47) 1.26 1.58

Index Impulse-Control and Conduct Disorder Risk

Presence of primary impulse-control and conduct disorder DX

0.93% 1.25 (0.90, 1.73) 1.10 1.20

Index Feeding and Eating Disorder Risk

Presence of primary feeding and eating disorder DX

0.55% 0.82 (0.53, 1.28) -1.04 1.08

Index Neurocognitive Disorder Risk

Presence of primary neurocognitive disorder DX

1.44% 0.87 (0.63, 1.19) -0.86 0.74

Index Neurodevelopmental Disorder Risk

Presence of primary neurodevelopmental disorder DX

0.60% 1.16 (0.79, 1.71) 0.69 0.47

Index Depressive Disorder Risk

Presence of primary depressive disorder DX

52.87% 0.98 (0.79, 1.22) -0.42 0.18

Index Personality Disorder Risk

Presence of primary personality disorder DX

0.25% 1.04 (0.63, 1.72) 0.25 0.06

Index Other Mental Disorder Risk

Presence of primary other mental disorder DX

0.21% 0.99 (0.56, 1.74) -0.08 0.01

Table 5 - Production Facility Readmit 30-Day Model for MH, continued


Less than or equal to 12 years 2,318 3.62%

13-17 years old 12,909 20.18%

18-25 years old 12,320 19.26%

26-64 years old 30,373 47.47%

Greater than or equal to 65 years 6,055 9.46%

Total 63,975

Table 6: Breakdown of Age for Facility Readmit 30-Day Model for MH

12

N= 60,547 % Acute

Admissions




Presence of primary / secondary depressive disorder DX in the past year

50.32% 1.58 (1.51, 1.65) 19.06 363.23

Public Sector 1= Medicare or Medicaid 0=Commercial

32.89% 1.68 (1.60, 1.77) 18.44 339.61


Presence of primary / secondary schizophrenic spectrum and other psychotic disorder DX in the past year

23.40% 1.69 (1.61, 1.78) 18.4 338.61



25.62% 1.60 (1.53, 1.69) 17.45 305.03



18.91% 1.50 (1.42, 1.57) 12.63 159.68

Age Category 1=age <=12, 2=13-17, 3=18-25, 4=26-64, 5= >=65

See Table 8

0.88 (0.85, 0.90) -8.67 75.25


Presence of primary / secondary other mental disorder DX in the past year

6.60% 1.43 (1.33, 1.54) 8.21 67.37

Trauma or Stress-Related Disorder Risk

Presence of primary / secondary trauma or stressor-related disorder DX in the past year

16.18% 1.19 (1.13, 1.26) 5.82 33.71


Presence of primary / secondary anxiety disorder DX in past year

19.05% 1.17 (1.11, 1.23) 5.46 29.84

Feeding and Eating Disorder Risk

Presence of primary / secondary feeding and eating disorder DX in the past year

2.03% 1.61 (1.39, 1.86) 5.43 29.59

Personality Disorder Risk

Presence of primary / secondary personality disorder DX in the past year

2.68% 1.35 (1.21, 1.51) 4.79 22.94

Index Schizophrenia Disorder Risk

Presence of primary schizophrenic spectrum and other psychotic disorder DX

15.92% 1.57 (1.29, 1.90) 4.30 18.55

Involuntary admission

Involuntary admission to acute MH hospitalization

28.23% 0.93 (0.89, 0.97) -2.90 8.36

Impulse-Control and Conduct Disorder Risk

Presence of primary / secondary impulse-control and conduct disorder DX in the past year

4.55% 1.15 (1.04, 1.26) 2.61 6.82

Table 7: Production Facility Readmit 90-Day Model for MH

13

N= 60,547 % Acute

Admissions Logistic

Regression HGLM HGLM



Index Trauma or Stress-Related Disorder Risk

Presence of primary trauma or stressor-related disorder DX

3.17% 0.77 (0.61, 0.97) -2.28 5.17

Electro-Convulsive Therapy

Presence of electro-convulsive therapy

1.16% 1.17 (0.98, 1.40) 1.84 3.36

Index Bipolar Disorder Risk

Presence of primary bipolar and related disorder DX

20.78% 1.21 (1.00, 1.47) 1.73 2.97

Index Neurocognitive Disorder Risk

Presence of primary neurocognitive disorder DX

1.31% 0.80 (0.60, 1.06) -1.53 2.35

Index Anxiety Disorder Risk

Presence of primary anxiety disorder DX

1.41% 0.85 (0.65, 1.11) -1.24 1.52

Index Feeding and Eating Disorder Risk

Presence of primary feeding and eating disorder DX

0.56% 0.79 (0.55, 1.13) -1.16 1.35

Index Obsessive Compulsive Disorder Risk

Presence of primary other obsessive-compulsive and related disorder DX

0.20% 1.39 (0.86, 2.25) 1.06 1.13

Neuro-developmental Disorder Risk

Presence of primary or secondary neurodevelopmental disorder DX in the past year

9.01% 1.03 (0.96, 1.11) 0.85 0.74

Index Impulse-Control and Conduct Disorder Risk

Presence of primary impulse-control and conduct disorder DX

0.94% 1.16 (0.87, 1.53) 0.80 0.66

Index Personality Disorder Risk

Presence of primary other personality disorder DX

0.26% 1.10 (0.73, 1.68) 0.52 0.27

Index Depressive Disorder Risk

Presence of primary depressive disorder DX

53.12% 1.00 (0.82, 1.20) -0.27 0.07

Index Other Mental Disorder Risk

Presence of primary other mental disorder DX

0.21% 0.94 (0.58, 1.54) -0.27 0.07

Index Neuro-developmental Disorder Risk

Presence of primary neuro-developmental disorder DX

0.60% 0.99 (0.71, 1.38) -0.10 0.01

Table 7: Production Facility Readmit 90-Day Model for MH, continued

14


Less than or equal to 12 years 2,196 3.63%

13-17 years old 12,416 20.51%

18-25 years old 11,782 19.46%

26-64 years old 28,650 47.32%

Greater than or equal to 65 years 5,503 9.09%

Total 60,547

Table 8: Breakdown of Age for Facility Readmit 90-Day Model for MH

Index admissions for the development data were from June 30, 2013 to July 1, 2014. The new models were cross-validated on new production data, which spanned a year from January 1, 2014 to December 31, 2014. Thus, there is an overlap of half year of 2014 between the two samples. SUD and MH Model Performance (Logistic Regression) Table 9 and 10 shows the performance for the logistic regression models (SUD and MH) and the unadjusted readmission rate versus the risk adjusted readmission rate (from HGLM analysis). Summary statistics were computed to assess model performance: calibration (a measure of over‐fitting), discrimination in terms of predictive ability, discrimination in terms of c statistic (equivalent to area under the receiver operating curve [ROC]), distribution of residuals, and model chi‐square. Over-fitting refers to the phenomenon in which a model describes the relationship between predictive variables and outcome well in the development dataset, but fails to provide valid predictions in new patients. Since the γ0 in the validation sample is close to zero and the γ1 is close to one in each of the models, there is little evidence of over-fitting. Discrimination in predictive ability measures the ability to distinguish high-risk subjects from low-risk subjects. It appears that the 90-day SUD model is better is more discriminating in terms of high risk subjects (larger range of difference) than the 30-day SUD model. The 90-day MH model also shows the same difference versus the 30-day MH model, but to a lesser extent. The c statistic is a measure of how accurately a statistical model is able to distinguish between a patient with and without an outcome. For binary outcomes the c statistic is identical to the ROC. A c statistic of 0.50 indicates random prediction, implying all patient risk factors are useless. A c statistic of 1.0 indicates perfect prediction, implying patients’ outcomes can be predicted completely by their risk factors, and physicians and hospitals play no role in patients’ outcomes. While higher c statistic is desirable, we do not want to maximize it by adjusting for factors that should not be adjusted for. For example, we do not want to include in -hospital complications as a risk factor. The range of c statistic results is 0.654 to 0.683 for all models which is in line with results we have seen for other 30-day and 90-day readmission measures and the CMS models. The Pearson Residuals show that about 90% are within the range of -2 to 0, which is consistent with the CMS model results.

15

Table 11 shows unadjusted readmission rates, the risk adjusted readmission rates, and deciles for combined HGLM models that were pooled over SUD and MH per hospital. Table 12 shows the descriptive statistics for mean and median national observed (unadjusted) readmission rate and risk adjusted rates for hospitals with 25 or more admits broken down by overall 30 and 90-days models, SUD 30 and 90-day models, and MH 30 and 90-day models. Quartile and decile statistics, and the total number of admissions and total number of hospitals are also displayed.

Indices Development SUD 30 Day

Validation Production SUD 30 Day

Development SUD 90 Day

Validation Production SUD 90 Day

Number of hospital stays 15,532 14,786 14,487 13,993

Number of hospitals 1,080 1,068 1,061 1,046

Number of hospitals with 25 or more admits

171 159 165 154

Unadjusted rate for hospitals with GE 25 admits

7.9% 8.3% 18.2% 18.3%

Risk adjusted rate for hospitals with GE 25 admits (HGLM)

8.5% 9.0% 19.2% 19.3%

Calibration (γ0, γ1) (0.096, 1.030) (0.087, 1.029) (0.035, 1.015) (0.057, 1.026)

Discrimination Predictive Ability (lowest decile %, highest decile %) Among Facilities with 25 or More Admits

Unadjusted readmission rate 1.6% - 16.4% 2.0% - 15.9% 6.7% - 32.1% 7.9% - 30.8%

Risk adj readmission rate 5.3% - 12.6% 6.0% - 12.7% 13.5% - 26.5% 13.6% - 27.3%

Discrimination – C statistic 0.658 0.663 0.655 0.654

Breakdown of Distribution of Pearson Residuals

Less than -2 0 0 0 0

-2 to < 0 91.96 91.46 81.76 81.91

0 to < 2 0.57 0.83 8.63 8.60

Greater Than or Equal to 2 7.47 7.71 9.62 9.48

Model Likelihood χ2 (DF) 360.3 (8) 405.6 (8) 605.9 (10) 636.9 (10)

R-Square 0.023 0.027 0.044 0.045

Max Rescaled R-Square 0.055 0.061 0.072 0.073

Table 9: SUD Model Performance (Logistic Regression)

16

Indices Development

MH 30 Day

Validation Production MH 30 Day

Development MH 90 Day

Validation Production MH 90 Day



Number of hospitals with 25 or more admits

699 685 671 635

Unadjusted rate for hospitals with 25 or more admits

11.8% 11.9% 20.8% 20.3%

Risk adjusted rate for hospitals with 25 or more admits (HGLM)

12.1% 12.1% 21.4% 20.5%

Calibration (γ0, γ1) (0, 1) (0, 1) (0, 1) (0, 1)

Discrimination Predictive Ability (lowest decile %, highest decile %)

Among Facilities with 25 or More Admits

Unadjusted readmission rate

5.0% - 18.8% 5.0% - 19.4% 11.5% - 30.4% 11.3% - 30.3%

Risk Adj readmission rate 10.8% - 13.7% 10.5% - 14.0% 19.4% - 23.6% 18.6% - 22.7%

Discrimination – C statistic 0.673 0.670 0.683 0.682

Breakdown of Distribution of Pearson Residuals

Less than -2 0 0 0 0

-2 to < 0 87.96 88 78.81 79.59

0 to < 2 3.38 3.19 13.38 12.45

Greater Than or Equal to 2 8.67 8.82 7.81 7.96

Model Likelihood χ2 (DF) 2804.7 (25) 2543.7 (25) 4321.1 (28) 4222.4 (28)

R-Square 0.042 0.039 0.071 0.067

Max Rescaled R-Square 0.080 0.075 0.111 0.106

Table 10: MH Model Performance (Logistic Regression)

17

Indices Development

Overall 30 Day

Validation Production

Overall 30 Day

Development Overall 90 Day

Validation Production

Overall 90 Day



Number of hospitals with GE 25 admits 821 785 781 753

Unadj rate hospitals with GE 25 admits 11.0% 11.1% 20.2% 19.8%

Risk adj rate hospitals with GE 25 admits 11.4% 11.5% 20.9% 20.2%

Discrimination Predictive Ability (lowest decile %, highest decile %) Among Facilities with 25 or More Admits

Unadjusted readmission rate 3.9% - 18.3% 4.0% - 18.6% 10.6% - 30.1% 10.3% - 30.1%

Risk Adjusted readmission rate 9.9% - 13.1% 9.7% - 13.5% 18.6% - 23.5% 17.9% - 22.8%

Table 11 - Overall (Pooled SUD and MH per hospital) HGLM Model Performance

Mean SD Min 10th Per centile

25th Per centile

50th Per centile

75th Per centile

90th Per centile

100 Per centile

Overall 30-Day Readmissions N Admissions = 78,761 N Hospitals = 2,233

Unadjusted 30-Day 11.1% 6.0% 0.0% 4.0% 7.1% 10.7% 14.3% 18.6% 50.0%

Risk adjusted 30-Day 11.5% 2.0% 5.6% 9.7% 10.4% 11.3% 12.3% 13.5% 35.9%

Overall 90-Day Readmissions N Admissions = 74,540 N Hospitals = 2,205

Unadjusted 90-Day 19.8% 7.8% 0.0% 10.3% 14.4% 19.2% 24.7% 30.1% 52.5%

Risk adjusted 90-Day 20.2% 2.7% 9.9% 17.9% 18.9% 20.0% 21.1% 22.8% 46.5%

SUD 30-Day Readmissions N Admissions = 14,786 N Hospitals = 1,068

SUD Unadj 30 Day 8.3% 6.3% 0.0% 2.0% 3.7% 7.1% 11.6% 15.9% 50.0%

SUD Risk adj 30 Day 9.0% 3.2% 4.2% 6.0% 6.8% 8.1% 10.2% 12.7% 27.0%

SUD 90-Day Readmissions N Admissions = 13,993 N Hospitals = 1,046

SUD Unadj 90-Day 18.3% 9.1% 2.1% 7.9% 11.1% 16.9% 24.6% 30.8% 50.0%

SUD Risk adj 90-Day 19.3% 5.4% 9.0% 13.6% 15.4% 18.0% 22.9% 27.3% 42.1%

MH 30-Day Readmissions N Admissions = 63,975 N Hospitals = 1,895

MH Unadj 30-Day 11.9% 5.8% 0.0% 5.0% 7.8% 11.4% 15.0% 19.4% 40.0%

MH Risk adj 30-Day 12.1% 1.5% 7.7% 10.5% 11.1% 11.9% 13.0% 14.0% 20.1%

MH 90-Day Readmissions N Admissions = 60,547 N Hospitals = 1,873

MH Unadj 90-Day 20.3% 7.5% 0.0% 11.3% 15.4% 19.5% 24.7% 30.3% 52.5%

MH Risk adj 90-Day 20.5% 1.6% 15.9% 18.6% 19.5% 20.4% 21.5% 22.7% 26.3%

Table 12: Distribution of Unadjusted and Risk Adjusted Rates for Hospitals with

25 or More Admits

18

Evaluation of Hospital Risk Adjusted Readmissions

Overall, the effect of risk adjustment methodology was to reduce the range and the variability of the original distribution. For 30-day readmissions, the range dropped from 50% to 34%, while the standard deviation dropped from 6% to 2%. For 90-day readmissions, the ranged dropped from 52.5% to 36.6% (Table 12). To estimate the confidence interval around hospital (a random intercept for HGLM) means, CMS used a bootstrap procedure because of complications with determining the standard error for the random effect in the HGLM model. Creating Confidence Intervals with Bootstrap Procedure • The bootstrap procedure was adapted from a CMS SAS macro for repeated resampling of

hospitals (second level data) with replacement Bootstrap programs were created separately for 30-day hospital RSRR and 90-day hospital RSRR.

• Any hospital with less than five inpatient admissions were excluded and resample size was increased to 6000 hospitals for each replication to minimize problems with the GHLM solution. Hospitals represent the hierarchical random effect and a random multiplier (to SE of random effect) was also computed during each replication.

• 3000 replications each were run for 30-day and 90-day RSRR bootstrap - To eliminate extreme outliers (an artifact of bootstrapping), the 95% percentile for each

overall bootstrap distribution was used as a cutoff value to trim the distributions. After the 5% trim, the bootstrap means were equivalent to the respective RSRR means of the original sample of hospitals with 25 or more admits (Table 13).

RSRR Model Facilities Mean SD Min 25th

Percentile 50th

Percentile 75th

Percentile Max

Current 30-Day 785 11.5% 1.98% 5.6% 10.4% 11.3% 12.2% 35.9%

Bootstrap 30-Day 4,036,081 11.5% 5.61% 0.0% 7.3% 11.1% 15.3% 26.2%

Current 90-Day 753 20.2% 2.68% 9.9% 18.9% 20.0% 21.1% 46.5%

Bootstrap 90-Day 4,048,198 20.0% 7.33% 0.2% 14.8% 20.0% 25.0% 37.5%

Table 13: 30-day and 90-day RSRR Bootstrap Means Compared to Validation Production

Sample of Hospitals with 25 or More Admits

• Hospital Specific CI. Each hospital had more than 2,900 values for RSRR. The 2.5th and 97.5th percentiles were extracted for each hospital distribution to obtain the lower confidence limit (LCL) and upper confidence limit (UCL) for the 95% CI.

• The average regional RSRR was computed from the original data for four United States regions: Central, Northeast, Southeast, and West. A hospital’s confidence limits were evaluated against their average regional RSRR.

• A hospital received a passing score if one of two conditions were met: - 1) If the UCL is lower than the corresponding average regional RSRR, or - 2) If the average regional RSRR is within the confidence interval

19

Table 14 shows that the 95% CI versus the average regional RSRR passed 20% more hospitals than the regional median method for both the 30-day and 90-day RSRR. Due to the time-consuming nature of the bootstrap procedure, the regional median method was retained.

Measure Evaluation Frequency Percent

30-Day Readmission

Current Pass - Regional Median RSRR 323 51.43

95% CI Pass - Regional Average RSRR 452 71.97

95% CI Pass - National Average Readmission 333 53.03

90-Day Readmission

Current Pass - Regional Median RSRR 322 51.27

95% CI Pass - Regional Average RSRR 444 70.7

95% CI Pass - National Average Readmission 288 45.86

Table 14: Evaluations Based on Hospitals Without and With 95% Confidence Interval

Summary and Conclusions

Based upon the CMS methodology for Hospital-Wide Risk-Standardized Readmission Metrics, a robust hierarchical general linear case-mix model for 30-day and 90-day readmission was created based on behavioral health admissions. The case-mix models allow for comparison across hospitals by adjusting for patient demographic and clinical characteristics for actual types of cases that it handles. RSRR metrics are incorporated with other measures (behavioral health cost, 7 and 30-day follow-up rates, peer review rate, and a length of stay metric) to assign a facility to a performance tier level. The highest tier rating has benefits such as increased recognition, more referrals, additional market support, and streamlined clinical review.

REFERENCES

• CMS Hospital-Wide All-Cause Unplanned Readmission Measure: 2013 SAS Pack Software Documentation, Lin, Zhenqiu and Grady, Jacqueline N., 2013.

• 2013 Measure Updates and Specifications Report: Hospital-Wide All-Cause Unplanned Readmission Measure (Version 2.0). Submitted by Yale New Haven Health Services Corporation/Center for Outcomes Research and Evaluation (YNHHSC/CORE). Prepared For: Centers for Medicare & Medicaid Services (CMS), March 2013.

• Hospital-Wide All-Cause Unplanned Readmission Measure Final Technical Report. Submitted by Yale New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (YNHHSC/CORE): Horwitz, Leora et al., Prepared For: Centers for Medicare & Medicaid Services (CMS), July 2012.

• Medicare Hospital Chartbook Performance Report on Outcomes Measures, Centers for Medicare & Medicaid Services (CMS), September 2013.

• Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition. Edited by American Psychiatric Association, 2013.

20

• Behavioral Health Trends in the United States: Results from the 2014 National Survey on

Drug Use and Health Report prepared for the Substance Abuse and Mental Health Services Administration (SAMHSA) by RTI International under Contract No. HHSS283201300001C with SAMHSA, U.S. Department of Health and Human Services (HHS). http://www.samhsa.gov/data/sites/default/files/NSDUH-FRR1-2014/NSDUH-FRR1-2014.pdf

ACKNOWLEDGMENTS Thank you to Nghi Ly, Rachel Lu, Brent Bolstrom, Laura Ten Eyck, Wade Bannister, Charlotte Wu, Ronald Ozminkowski, and everyone who have worked with me on SAS and behavioral health outcome effectiveness.

CONTACT INFORMATION

Allen Hom, PhD Senior Research Analyst Consumer Solutions Group – Healthcare Analytics 425 Market Street, San Francisco 94105

(415) 547-5813 [email protected]

SAS and all other SAS Institute Inc. product or service names are registered trademarks or

trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Other brand and product names are trademarks of their respective companies.

Hierarchical Generalized Linear Models for Behavioral ...The final metrics were a hospital-wide risk adjusted 30-day readmission rate percent and a hospital-wide risk adjusted 90-day

Documents