USE OF NATIONWIDE OUTCOMES MONITORING DATA TO COMPARE CLINICAL OUTCOMES IN SPECIALIZED MENTAL HEALTH PROGRAMS AND GENERAL PSYCHIATRIC CLINICS IN THE VETERANS HEALTH ADMINISTRATION

Psychiatric Quarterly, Vol. 77, No. 2, Summer 2006 ( C© 2006)DOI: 10.1007/s11126-006-9004-0

USE OF NATIONWIDE OUTCOMESMONITORING DATA TO COMPARE

CLINICAL OUTCOMES IN SPECIALIZEDMENTAL HEALTH PROGRAMS AND

GENERAL PSYCHIATRIC CLINICS IN THEVETERANS HEALTH ADMINISTRATION

Greg A. Greenberg, Ph.D. and Robert A. Rosenheck, M.D.

Published online: 9 June 2006

Abstract There has been a growing interest in the implementation of evidence-based specialized mental health programs. However, there has been little studyof the effectiveness of these programs in comparison with standard men-tal health care in real world mental health systems. This study used a na-tional sample of patients from the Veterans Health Administration to comparechanges in mental health status in various specialized mental health outpa-tient programs and in general psychiatric clinics. Hierarchical linear modelswere used to compare the association of both regularity and intensity of carein six specialized mental health programs with GAF change scores in patients

Greg A. Greenberg, Ph.D., Project Director, Northeast Program Evaluation Center,VAMC West Haven, CT; Lecturer, Yale University Department of Psychiatry, New Haven,CT.

Robert A. Rosenheck, M.D., Director, Northeast Program Evaluation Center, VAMCWest Haven, Ct., VA New England Mental Illness Research, Education, and ClinicalCenter; Professor, Department of Psychiatry and Yale University School of Epidemiologyand Public Health, New Haven, CT.

Address correspondence to Greg A. Greenberg, Ph.D., Northeast Program EvaluationCenter, 950 Campbell Ave, West Haven, CT 06516 ; e-mail: [email protected].

151

0033-2720/06/0600-0151/0 C© 2006 Springer Science+Business Media, Inc.

152 GREENBERG AND ROSENHECK

treated in general psychiatric clinics. While improvements were observed inall programs, two specialized programs performed better overall than gen-eral psychiatric care, one was not significantly different, and three had pooreroutcomes than general psychiatric clinics. Programmatic differences in targetpopulations accompanied by imperfect risk adjustment for population differ-ences most likely explain why these results differ from those observed in clin-ical trials. While the analytic strategies demonstrated here may have widerapplicability to comparative performance assessment, this study provides acautionary tale concerning the limits of conclusions that can be drawn fromlarge scale outcomes monitoring efforts.

KEY WORDS: primary psychiatric care; specialized psychiatric care; GAF; intensityand regularity of care; outcomes; Veterans Health Administration.

In recent years, the delivery of mental health services has changed dra-matically with a major shift in emphasis from inpatient to outpatientand community-based care. Parallel to this shift, there has been grow-ing controversy over the role of specialized mental health programs inthe outpatient setting. On the one hand there has been an emphasis onprimary psychiatric care provided by generalists since, as elsewhere inmedicine, such care is viewed as less expensive, and as suggested byseveral studies, of equivalent effectiveness (1, 2). The development ofdetailed models of collaborative care, allowing for the effective deliv-ery of mental health services within primary care clinics (3, 4) is onewell-known example of this generalist thrust.

On the other hand, there has also been growing emphasis on the dis-semination of evidence based specialized mental health treatments.Controlled evaluations have found that several specialized mentalhealth treatment models are more effective, and in some cases lesscostly, than standard care. Among these interventions are: i) supportedand transitional employment (5–7); ii) Assertive Community Treat-ment (8–10); iii) specialized PTSD treatment (11, 12); iv) communityresidential care (CRC) (13–15); v) day hospital treatment (16–18); andvi) specialized substance abuse treatment (19–22). However, since stud-ies showing the benefits of specialized care for the most part have beenconducted under highly controlled conditions, and most have been ledby proponents of the interventions being evaluated, there is little evi-dence of the value of these interventions in comparison with standardmental health care, in real world settings or even exploration of meth-ods to access the their effectiveness under such circumstances.

In 1999 in recognition of both the importance and potentiallyhigh cost of system-wide outcomes monitoring, the Veterans Health

NATIONWIDE OUTCOMES MONITORING DATA TO COMPARE CLINICAL 153

Administration (VHA) issued a policy directive that required mentalhealth clinicians to record a Global Assessment of Functioning (GAF)score at the conclusion of each episode of inpatient care and thatoutpatients be rated with the GAF at least once every 90 days duringactive treatment (23). The collection of this data has allowed theassessment of outcomes associated with real world practices in severalstudies of mental health care delivered nationally in the VHA (24–26).

Many of the evidence-based specialized programs whose value hasbeen demonstrated in experimental studies have been widely dissem-inated throughout the VHA in recent years. Among these programsare: (1) transitional employment (Compensated Work Therapy in theVHA); (2) Assertive Community Treatment (Mental Health IntensiveCase Management in the VHA)1; (3) specialized treatment for military-related Posttraumatic Stress Disorder (the PTSD Clinical Teams Pro-gram); and (4) programs that provide specialized treatment for sub-stance abuse. In addition many (5) day hospitals and (6) communityresidential care programs established in the VHA in prior decades arestill in operation. Fortunately, services delivered by each of these spe-cialized VHA programs can be differentiated from services deliveredin general psychiatry clinics because each of these six specialized pro-grams relies on program-specific “stop codes” to document its workloadin national outpatient workload databases. In this study nationallyavailable GAF data are used to compare changes in mental health sta-tus among participants in these six specialized VHA mental healthoutpatient programs with changes in mental health status among par-ticipants in general psychiatric outpatient clinics.

METHODS

Sample

The sample consists of all VHA mental health outpatients in FY 2003for whom GAF data were available. More specifically, it consists ofthree distinct sub-groups of patients who received outpatient servicesin FY 2003; 1) those with at least one inpatient GAF rating and a lateroutpatient GAF rating; 2) those beginning a new episode of outpatienttreatment; and 3) those patients in continuing outpatient treatment.

1Compared to the typical ACT program MHICM teams are staffed at a lower level andtherefore provide a more limited range of services in the community. National MHICMprogram fidelity to the ACT model has remained steady at 4.0 on the Dartmouth ACTFidelity Scale since 1997.


The later two groups were differentiated by whether or not the veteranreceived any VHA mental health outpatient services in the last quarterof the previous fiscal year (2002). Veterans who had not received ser-vices were considered “new” outpatients while those who had a clinicvisit were considered to be “continuing” outpatients. In addition, to beincluded in the sample, the second outpatient GAF rating in each casehad to have been made between 90 and 180 days after the initial rating(inpatient or outpatient).

GAF data meeting these criteria were available for 160,697 veter-ans who received outpatient mental health services in 2003 and 9,184inpatients. The veterans for whom two GAF ratings were availablerepresent 54% of all veterans who had two outpatient mental healthclinic visits between 90 and 180 days apart, and 31% of the dischargedinpatients who had at least one outpatient clinic visit between 90 and180 days following their inpatient discharge. The average baseline GAFscore was about 41.9 (s.d., 13.1) for discharged inpatients and 53.3 (s.d.,11) for outpatients.

The patients in the sample received general psychiatric servicesand/or care in one of the six types of specialized programs specifiedabove: (1) compensated work therapy (CWT); (2) Mental Health In-tensive Case Management (MHICM); (3) the PTSD clinical teams pro-grams; (4) specialized substance abuse outpatient clinics; (5) day hos-pitals; and (6) CRC programs. Detailed data on current program oper-ations are available for CWT (27), MHICM (28), PTSD clinical teams(29), and specialized substance abuse programs (30).

Source of Data

GAF ratings were obtained from a national administrative file, whichcontains all GAF ratings made by VHA clinicians along with; a) patientidentifiers, b) an indicator of whether the rating was made at the endof an inpatient stay or during an episode of outpatient care, c) the datethe rating was made, and d) a code documenting the specific facility atwhich the rating was made. This study was reviewed by the institu-tional review board of the VA Connecticut Health System and a waiverof informed consent was approved.

Data on veterans’ sociodemographic and diagnostic characteristicswere obtained from other VHA administrative workload files: the pa-tient treatment file, the outpatient encounter file, and the outpatientcare file. The patient treatment file is a discharge abstract file thatcontains basic data on all completed episodes of inpatient care whilethe outpatient care and encounter files document the date, clinic type,


and diagnoses pertaining to each outpatient clinic visit or contact. Datafrom the outpatient care and encounter files document all VHA outpa-tient service delivery were used to construct the regularity and inten-sity of care measures described below.

Measures

GAF Change Measure

The primary outcome of interest was change in the GAF scores com-puted by subtracting the baseline value from the follow-up value oc-curring between 90 and 180 days later. Positive values thus reflectimprovement. The GAF is a single item rating through which a treat-ing clinician evaluates the current global functioning of their patientson a 1–100 scale with brief anchors at 10-point intervals. The VHAselected the GAF because it is inexpensive, practical to administer,and has demonstrated potential to be used reliably (31–33). The GAFis well known since it is an integral part of the standard multiaxialdiagnostic system described in the Diagnostic and Statistical Manual-IV (34). Additionally, Moos and associates (35) recently demonstratedthat GAF scores collected by VHA clinicians are significantly associatedwith current symptoms and functioning as measured with standard-ized instruments, although they do not predict future health status orcosts. Another recent study (24) presented evidence of the discriminantvalidity of the GAF score in the VHA as well as of the usefulness ofGAF derived measures for examining changes over time in outcomeswithin facilities.

Regularity and Intensity of Care

By using program specific outpatient stop codes regularity of care andintensity of care measures specific to each specialized program werecreated. Regularity of care was measured by the number of months inthe six months following the initial assessment in which the veteranhad at least one clinic visit (range 0–6), while intensity of care wasmeasured as the total number of clinic visits between a veteran’s initialGAF and their last GAF within 180 days. Because these measuresdid not have a normal distribution it was necessary to create binaryvariables representing various levels of regularity and intensity. Bothtypes of measures were coded as being either medium or high basedon whether they were above or below the median among clients withnon-zero values. Thus, for each type of specialized program two binary


variables were created reflecting the intensity and regularity of servicesobtained in each program by each patient. The specific cut off pointsused for each measure are presented in Table 1.

Risk Adjustment

A major challenge to fair comparison of programs is that patients dif-fer on various characteristics (such as, age, gender or diagnosis) thatmay affect outcomes (36). This is especially important since patientsreferred to specialized programs are often referred because they aredoing poorly in standard care. Programs also vary in the diagnosticmix of their clients, particularly the two programs that treat clientsfor specific illnesses (i.e., substance abuse and PTSD clinical teams)and possibly with respect to the severity of clients’ illnesses. As a re-sult outcomes must be risk-adjusted, as well as possible, for differencesin sociodemographic and clinical characteristics. To risk adjust GAFchange measures, as many potentially confounding patient character-istics were identified as allowed for by available data. These measuresdocumented sociodemographic characteristics including age, gender,ethnicity, income level, and marital status. Data was also availableon the receipt of VA compensation (10–49% disability, greater than orequal to 50% disability from any injury or illness resulting from mili-tary service, or no VA disability rating). In addition, ICD-9 psychiatricdiagnoses were grouped into nine non-mutually exclusive clusters onthe basis of inpatient and outpatient diagnostic information from theentire current fiscal year (24, 25). Dually diagnosed veterans (i.e., veter-ans with both psychiatric and substance abuse/dependence disorders)were also represented by a dichotomous variable.

We also included the baseline GAF score in each model, and a mea-sure of the number of days between the client’s first and last GAF, inorder to risk adjust for potential regression to the mean and since therewould be greater possibility for improvement in the GAF score amongthose clients who experienced longer periods of treatment.

Analysis

Analysis occurred in several stages. First, the means and distributionsof the regularity and intensity of care measures were examined. Thisinformation was used to create the dichotomous measures describedabove.

Next, paired t-tests were used to determine whether there was sig-nificant improvement among participants in each program. Because


Tab

le1

Ran

ges

for

Bin

ary

Var

iab

leC

reat

ion

Inte

nsi

tyof

Car

eR

egu

lari

tyof

Car

e

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

Med

ium

Su

bsta

nce

Abu

se1–

171–

101–

91–

2(4

5%)a

1–2

(46%

)1–

3

Hig

hS

ubs

tan

ceA

buse

>17

>10

>9

>2

>2

>3

Med

ium

PT

SD

1–4

1–2

1–4

1–2

1–2

(64%

)1–

3(6

4%)

Hig

hP

TS

D>

4>

2>

4>

2>

2>

3M

ediu

mM

HIC

M(A

CT

)1–

111–

171–

31–

4

Hig

hM

HIC

M(A

CT

)>

11>

17>

3>

4M

ediu

mC

WT

1–13

1–9

1–11

1–2

1–2

(59%

)1–

2H

igh

CW

T>

13>

9>

11>

2>

2>

2M

ediu

mD

ayH

ospi

tal

1–9

1–6

1–13

1–2

(62%

)1–

2(6

2%)

1–3

Hig

hD

ayH

ospi

tal

>9

>6

>13

>2

>2

>3

Med

ium

CR

C1–

2(4

4%)

1–2

(44%

)1–

4(5

4%)

1–2

(57%

)1–

21–

3(6

2%)

Hig

hC

RC

>2

>2

>4

>2

>2

>3

aP

aren

thes

esin

dica

tecu

tof

fpo

ints

oth

erth

an50

%.


the analyses were based on a large sample and thus had substan-tial power, some statistically significant findings may not be clinicallymeaningful. There is no standard for determining how large a changein the GAF score is clinically meaningful. However, secondary anal-ysis of data from a clinical trial indicated that small differences inthe GAF (2.2 points) which favored clozapine over haloperidol paral-leled significant differences in other accepted measures such as thePositive and Negative Syndrome Scale that were found in a study com-paring these two medications (37). Given this result we applied thefollowing guidelines: GAF change scores above 2 are considered clini-cally meaningful; GAF change scores between 1 and 2 are intermediatein size and indicate possible clinical improvement; while GAF changescores below 1 are small and do not represent clinically meaningfulchanges.

In the subsequent stage of analysis, six models were used to com-pare the change in mental health status between participants in eachtype of specialized program and general psychiatric clinics. These mod-els examined the association of regularity and intensity of specializedcare and GAF change scores, controlling for potentially confoundingmeasures in each of the three clinical subgroups (i.e., discharged in-patients, new outpatients, and continuing outpatients). Due to theirintercorrelation, measures of regularity of care and treatment inten-sity were examined in separate models. Thus, two models were run foreach of the three patient sub-groups. The regularity and intensity ofcare measures for the MHICM program were not included in the mod-els that used data concerning new outpatients, since these clients werenot eligible for the MHICM program. Receipt of general non-specializedpsychiatric services was the reference condition for all comparisons.

Rather than having separate models for each type of specialized pro-gram, resulting in approximately 36 models in total (i.e., 6 specializedprograms∗3 patient sub-groups ∗ 2 independent measures of interest)we chose to use one model that addressed all six programs. This ana-lytic technique maximizes parsimony and can be clearly interpreted asa comparison of diverse specialized programs with a common referencegroup, non-specialized programs, controlling for factors that may differbetween the populations served by each type of program (i.e. covaryingfor diagnosis and other factors).

In these analyses random effects were modeled for site using an un-structured covariance matrix, thereby adjusting standard errors forthe correlated nature of the data in these models (i.e., for the poten-tial autocorrelation of observations within sites). This technique is of-ten referred to as hierarchical linear modeling (HLM) (38). The PROC


MIXED procedure of the SAS r© software system Version 6.12 (SASInstitute, Cary, NC) was used for these analyses.

RESULTS

Sample Characteristics

Consistent with the population served by the VHA, the sample in-cluded white, predominantly middle aged and elderly males, althoughthe inpatient sample had a large percentage of African Americans(see Table 2). As would be anticipated, individuals in the inpatientsample had more serious psychiatric diagnoses and lower baselineGAF ratings on average than outpatients.

Service Use

Most veterans who receive mental health treatment from the VHA re-ceive services exclusively from general psychiatric clinics (Table 3). Asubstantial number of veterans receive specialized substance abuse ser-vices, particularly those veterans who were discharged from inpatientprograms. As would be expected, former inpatients were more likely toreceive specialized outpatient care of all types while new outpatientshad the lowest intensity of care.

GAF Baseline and Change Scores

Not surprisingly, given the targeting of these programs to severely illpatients, veterans in the MHICM, day hospital/treatment, and CRCprograms had the lowest baseline GAF scores (Table 4, upper panel).

The GAF change scores of former inpatients, regardless of the typeof program in which services were received were all significant andshowed the largest improvement. Increases were relatively smalleramong inpatients who received services from the CWT and MHICMprograms (Table 4, lower panel). Changes in GAF scores were alsostatistically significant for all new outpatient groups regardless of thetype of program in which services were received. Although the GAFchange scores were sizable for new outpatients in four programs andapproached that of discharged inpatients for CWT service recipients,they were small for new outpatients in the PTSD clinical team programand intermediate for new outpatients in the CRC programs and generalpsychiatric clinics.


Table 2Veteran Characteristics

DischargedInpatients New Outpatients

ContinuingOutpatients

Mean Std Dev Mean Std Dev Mean Std Dev

Age 50.8 10.0 55.3 12.7 55.5 11.7Blacks 26% 44% 13% 33% 15% 36%Hispanic 3% 17% 2% 15% 3% 18%Male 94% 24% 92% 27% 92% 27%Married 25% 44% 49% 50% 45% 50%Divorced or

Separated39% 49% 27% 45% 28% 45%

Annual Income (log) 7.81 3.54 8.25 3.39 8.60 2.96Service Connected

at <50%14% 35% 19% 39% 16% 37%

Service Connectedat >50%

27% 44% 25% 43% 40% 49%

Schizophrenia 30% 46% 11% 32% 20% 40%PTSD 39% 49% 33% 47% 40% 49%Drug Depen-

dence/Abuse54% 50% 15% 36% 14% 35%

Alcohol Depen-dence/Abuse

60% 49% 20% 40% 18% 38%

Bipolar 26% 44% 11% 31% 13% 34%Major Depression 36% 48% 30% 46% 28% 45%Dysthymia 56% 50% 52% 50% 42% 49%Anxiety 32% 47% 28% 45% 23% 42%Personality 18% 38% 6% 23% 6% 24%Dual Diagnosis 63% 48% 21% 41% 20% 40%Days Duration

Between GAFs133.4 26.7 131.5 27.5 133.4 27.9

Baseline GAF Score 41.91 13.07 53.9 11.2 53.1 10.8N = 9,184 63,455 160,697

GAF change scores were small and did not show significant changefor continuing outpatients receiving services from the PTSD ClinicalTeams, MHICM, or CRC programs. While the GAF change scores forcontinuing outpatients in the other specialized programs were statisti-cally significant due to the large sample size, they were relatively smallin magnitude.


Tab

le3

Ser

vice

Use

Per

cen

tage

wit

hA

ny

Cli

nic

Vis

it

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

N=

9,18

463

,455

160,

697

Gen

eral

Psy

chia

try

92.2

%94

.1%

91.2

%S

ubs

tan

ceA

buse

46.5

%11

.3%

11.1

%P

TS

DC

lin

ical

Tea

m14

.2%

8.8%

13%

MH

ICM

(AC

T)

5.9%

Not

App

lica

ble

1.8%

Com

pen

sate

dW

ork

Th

erap

y17

.3%

1.8%

2%D

ayH

ospi

tal

10.9

%1.

5%3.

9%C

omm

un

ity

Res

iden

tial

Car

e3.

2%N

otA

ppli

cabl

e1.

7%


Tab

le3

Con

tin

ued

Ave

rage

Nu

mbe

rof

Cli

nic

Vis

its

Am

ong

Th

ose

Wit

hA

ny

Cli

nic

Vis

its

NM

ean

Std

Dev

NM

ean

Std

Dev

NM

ean

Std

Dev

Gen

eral

Psy

chia

try

8,64

714

.023

.559

,705

4.7

7.8

146,

690

6.9

13.1

Su

bsta

nce

Abu

se4,

271

27.7

33.1

7,20

823

.532

.617

,984

23.7

34.5

PT

SD

Cli

nic

alT

eam

1,30

215

.427

.35,

608

5.2

9.8

20,8

807.

29.

9M

HIC

M(A

CT

)54

518

.623

.621

92,

931

23.1

22.9

Com

pen

sate

dW

ork

Th

erap

y1,

588

23.5

26.9

1,15

420

.125

.13,

269

26.4

31.3

Day

Hos

pita

l1,

007

20.9

29.2

973

19.2

30.7

6,20

827

.636

.1C

omm

un

ity

Res

iden

tial

Car

e29

54.

04.

627

73.

52.

752,

717

5.4

7.2

Ave

rage

Nu

mbe

rof

Cli

nic

Vis

its

35.8

8.2

12.0


Tab

le4

GA

FB

asel

ine

and

Ch

ange

Sco

res

Ave

rage

Bas

elin

eG

AF

Sco

reby

Pro

gram

type

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

Mea

nS

tdD

evN

Mea

nS

tdD

evN

Mea

nS

tdD

evN

Gen

eral

Psy

chia

try

41.9

13.1

8,46

754

.111

.259

,705

53.3

10.9

146,

690

Su

bsta

nce

Abu

se43

.712

.24,

271

49.2

11.1

7,20

851

.211

.417

,984

PT

SD

Cli

nic

alT

eam

42.6

11.9

1,30

250

.49.

925,

608

49.4

920

,880

MH

ICM

(AC

T)

37.5

12.3

545

42.4

10.9

2,93

1C

ompe

nsa

ted

Wor

kT

her

apy

46.8

12.1

1,58

848

.910

.41,

154

50.6

10.8

3,26

9D

ayH

ospi

tal

38.6

12.6

1,00

745

.111

.297

347

.010

.66,

208

Com

mu

nit

yR

esid

enti

alC

are

37.1

12.8

295

45.6

11.7

277

44.3

10.5

2,71

7


Tab

le4

Con

tin

ued

Ave

rage

GA

FC

han

geS

core

byP

rogr

amT

ype

Mea

nS

tdD

evN

Mea

nS

tdD

evN

Mea

nS

tdD

evN

Gen

eral

Psy

chia

try

6.3∗∗

14.4

8,46

71.

21∗∗

8.78

59,7

05.3

1∗∗7.

514

6,69

0S

ubs

tan

ceA

buse

5.5∗∗

13.8

4,27

12.

81∗∗

11.6

7,20

8.9

3∗∗9.

717

,984

PT

SD

Cli

nic

alT

eam

5.6∗∗

13.1

1,30

2.6

∗∗8.

585,

608

.02

6.7

20,8

80M

HIC

M(A

CT

)4.

8∗∗13

.654

5.1

88.

12,

931

Com

pen

sate

dW

ork

Th

erap

y4.

2∗∗14

.21,

588

4.08

∗∗11

.41,

154

1.06

∗∗9.

943,

269

Day

Hos

pita

l7.

1∗∗16

.31,

007

3.3∗∗

12.2

973

.45∗∗

8.4

6,20

8C

omm

un

ity

Res

iden

tial

Car

e6.

5∗∗12

.929

51.

7∗∗∗

10.1

277

.25

7.2

2,71

7∗∗

P<

.000

1in

dica

tin

ga

stat

isti

call

ysi

gnifi

can

tch

ange

.∗∗∗

P=

.005

.


Intensity/Continuity of Care and Outcomes

Multiple regression models indicated that greater intensity of care intwo of the specialized programs, CWT and specialized substance abusetreatment, was significantly and positively associated with GAF changescores, suggesting that relatively greater client improvement occurredin these programs than in general psychiatric programs (Table 5).More specifically, both medium and high levels of CWT and special-ized substance abuse outpatient care were found to be significantlyand positively associated with most measures of GAF change, acrossall three patient groups. Three specialized programs showed less im-provement than general psychiatric treatment. Both the MHICM andCRC programs showed significant negative associations between indi-cators of both medium and high service and the GAF change score in allthree patient groups. Additionally for both new and continuing outpa-tients, both medium and high levels of service use from PTSD ClinicalTeams were found to be significantly and negatively associated withGAF change scores. Measures of day hospital intensity were almost allnon-significant indicating no significant difference from general psy-chiatric programs. Similar results were found for the regularity of caremeasures.

DISCUSSION

In this study national client level administrative data were used toinvestigate whether outcomes in six specialized VHA mental healthoutpatient programs were superior to those in general VHA psychi-atric clinics, after controlling for available baseline measures of so-ciodemographic and clinical characteristics. The GAF change scoresof both discharged inpatients and new outpatients, regardless of theprogram in which care was received, showed significant improvement,with larger improvements occurring among discharged inpatients. Incontrast, GAF change scores of continuing outpatients were small andnon-significant for veterans receiving services in three of the special-ized programs. For continuing outpatients in the other programs GAFchange scores improved significantly but the changes were very smallin magnitude.

Veterans in two types of specialized programs, CWT and substanceabuse programs showed greater improvement than veterans receivingcare in general psychiatric programs while improvement among


Tab

le5

Rel

atio

nsh

ipof

Con

tin

uit

yof

Car

eIn

dic

ator

san

dC

han

ges

inth

eG

AF

Sco

reIn

ten

sity

ofC

are

Reg

ula

rity

ofC

are

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

Dis

char

ged

Inpa

tien

tsN

ewO

utp

atie

nts

Con

tin

uin

gO

utp

atie

nts

BP

BP

BP

BP

BP

BP

Med

ium

Su

bsta

nce

Abu

se.7

69.0

14−

.408

.007

.167

.045

.622

.052

−.6

45<

.000

1.2

08.0

11

Hig

hS

ubs

tan

ceA

buse

1.97

<.0

001

1.75

<.0

011.

37<

.000

12.

03<

.000

11.

88<

.000

11.

48<

.000

1

Med

ium

PT

SD

.604

.17

−.6

13.0

002

−.7

28<

.000

1.5

67.1

8−

.496

.001

−.8

04<

.000

1H

igh

PT

SD

−.2

12.6

5−

.965

<.0

001

−1.

05<

.000

1−

.105

.83

−1.

29<

.000

1−

1.13

<.0

001

Med

ium

MH

ICM

(AC

T)

−2.

53<

.000

1−

1.59

<.0

001

−2.

90<

.000

1−

1.92

<.0

001

Hig

hM

HIC

M(A

CT

)−

3.36

<.0

001

−2.

18<

.000

1−

3.02

<.0

001

−1.

88<

.000

1M

ediu

mC

WT

.914

.026

.714

.031

−.0

27.8

7.4

11.3

0.6

48.0

36.0

91.6

0H

igh

CW

T1.

96<

.000

11.

96<

.000

1.9

68<

.000

12.

50<

.000

12.

42<

.000

1.9

2<

.000

1M

ediu

mD

ayH

ospi

tal

−1.

46.0

03−

.401

.27

−.2

11.1

05−

.749

.091

.037

.91

−.0

31.8

1H

igh

Day

Hos

pita

l.2

52.6

1.5

27.1

5−

.137

.28

−.3

78.4

9.1

17.7

9−

.319

.015

Med

ium

CR

C−

2.51

.006

−1.

72.0

15−

1.77

<.0

001

−2.

21.0

06−

.765

.224

−1.

87<

.000

1H

igh

CR

C−

1.95

.018

−1.

34.0

34−

2.03

<.0

001

−2.

16.0

22−

2.29

.001

−1.

89<

.000

1N

=9,

169

63,4

1116

0,67

69,

169

63,4

1116

0,67

6


veterans treated in day hospitals did not differ significantly from thosetreated in general psychiatric clinics. In contrast veterans showedsignificantly less improvement on average in MHICM, CRC, and PTSDclinical teams than in general psychiatric programs. These resultswere unexpected, especially with regard to MHICM, since there isstrong evidence based on randomized clinical trials of the effectivenessof ACT programs (8). A 10-site randomized trial specifically conductedin the VHA also showed clinical benefits and cost savings with ACT (9).

The most likely explanation for these results is that patients whowere referred to these 3 specialized programs were different in crucialways from those seen in general clinics. For instance, it seems likely,as indicated both by the defined target populations of these programsand by their mean GAF baseline scores, that MHICM programs, PTSDclinical teams, and CRC programs treat poorer functioning clients withpoorer prognoses. Fidelity ratings collected annually show that theMHICM program, as implemented in the VHA, has high fidelity tothe established evidence-based ACT model, but also that patients inthis program have exceptionally high rates of hospitalization and highlevels of functional disability (28).

In the case of the CWT program, which had more positive results, itis also possible that selection biases affected the results since patientsare more likely to seek out the CWT program if they are ready andinterested in employment. Since this program is specifically focused onimproving vocational functioning and participants work at paying jobs,by definition, GAF ratings may also reflect legitimate improvement.

It is also likely that in the substance abuse programs, patients whorelapse and who are thus doing less well, are less likely to attendtreatment and may even be asked, as a matter of clinical policy, notto attend treatment when they are intoxicated from alcohol or high ondrugs. Thus, the association of substance abuse program participationwith greater GAF improvement may reflect withdrawal from treatmentor policy-related exclusion from treatment of patients who are usingalcohol or drugs, rather than the specific beneficial effects of continuityof care in these programs. Although an effort was made to control forselection bias with the inclusion of baseline values for the GAF scoreas well as sociodemographic and clinical characteristics it seems likelythat not enough information was available to adequately risk adjust theanalyses. This is a potential hazard in any non-experimental outcomemonitoring effort that relies on limited data especially, when comparingprograms that operate with very different admission criteria.

One possible alternative explanation for the results is that someprograms may have been poorly implemented, or that staff reductions


may have impaired their effectiveness. A related possibility is that in-dividuals in the sample may not have received the full set of servicesassociated with a program even though they had two or more clinicvisits. Some patients with two clinic visits may have in fact only beenscreened for program eligibility. However, the use of intensity mea-sures allowed us to differentiate low intensity treatment from higherintensity treatment, minimizing the impact of this threat to validity.

Although there have been no recent randomized trails of day hospi-tals, CRC, or PTSD clinical teams implemented in the VHA, it wouldnot have been expected for these programs to be less effective thangeneral psychiatric care programs. The most likely explanation here,too, is that the results reflect selection bias more than differential ef-fectiveness, since both CRC and day hospital programs are designed tosupport less functionally capable individuals.

Limitations

Before concluding, several methodological limitations deserve com-ment. As noted above, selection biases threaten the validity of non-randomized comparisons, especially in the presence of programatic dif-ferences in target populations, and in the absence of extensive measuresfor risk adjustment. In fact this study should stand as a cautionary taledemonstrating the limits of performance assessment using administra-tive data, especially when it comes to programs that are designed totreat the least well off.

Second, as with most administrative data sets, service utilizationmeasures do not reflect care received outside the VHA. However datafrom other studies (39, 40) suggest that veterans who receive servicesfrom VHA make little use of non-VHA sources of care.

Thirdly, the measures may not adequately represent client outcomes.The GAF is a single-item measure whose reliability and validity has notwell been demonstrated in this real world national practice setting. Norhas there been any system-wide training or standardization of ratingsacross the VHA. It is notable, however, that Moos and associates (35)found significant relationships between GAF ratings extracted fromthe same data file as the one used in this study, and other psychomet-rically sound measures. Greenberg and Rosenheck (24) also found thatmeasures derived from the GAF score appeared to have reasonablediscriminant validity.

A fourth limitation is that data are available only on relatively short-term changes in client functionality. Some specialized programs aredesigned to improve functioning over an extended period of time. The


multi-site VHA trial of MHICM treatment specifically found that sig-nificant differences in functional improvement favoring experimentalpatients did not emerge until the end of the second year of treatment(9).

Further, since GAF data were not available for all eligible veteransthese findings may not be representative of all VHA patients. A largenumber of clients did not have a second GAF either because it wasnot recorded or they did not have consistent outpatient care. Compara-tive analysis has shown that clients who had two GAFs and who weretherefore included in the sample were more seriously ill than others(41). These sicker veterans constitute a sample of particular interestand concern and selection biases of this type would tend to equalizethe problems of patients in specialized and general mental health pro-grams. It should also be noted that the reliance on a VHA sample mayalso limit the generalizability of the findings to other populations orhealth care systems. Veterans are overwhelmingly male and tend to beolder than patients served in other health care systems.

An additional limitation is that we only had program fidelity data forone of the specialized programs, although fidelity in that program wasquite good.

Finally, the guidelines we used in specifying clinically meaningfulchanges in the GAF score were based on only one study. More researchis needed to confirm the appropriateness of these guidelines.

Implications for Behavioral Health Services

The methodological limitations specified above preclude concludingthat care delivered in specialized VHA mental health programs is lesseffective than that provided in general clinics and suggest that theresults predominantly reflect programmatic differences in target pop-ulations.

There has been considerable enthusiasm for using outcomes moni-toring of real world programs to compare the effectiveness of variousprograms as implemented in natural settings (42–46). While the multi-variate analytic approach used here can usefully guide future efforts ofthis type; perhaps what is most notable about the findings is that theydemonstrate a major limit of endeavors such as this one. As one mighthave hoped, this study involved a comparison of a broad array of differ-ent types of specialized and general mental health programs using anespecially large and rich national data set. However, its findings pri-marily demonstrate the difficulties of adjusting for systematic selectionbias, especially when they are intrinsic to the design of the programs


being studied. This study thus draws attention to the extensive dataneeds for program comparisons without random assignment and illus-trates the difficulties of coming to credible conclusions in the absenceof adequate measures of patient characteristics needed to control forselection bias in non-experimental studies.

REFERENCES

1. Donohoe MT: Comparing generalist and specialty care: discrepancies, deficiencies,and excesses. Archives of Internal Medicine 158(15):1596–1608, 1998.

2. Engstrom S, Foldevi M, Borgquist L: Is general practice effective? A systematic lit-erature review. Scandinavian Journal of Primary Health Care 19(2):131–144, 2001.

3. Hedrick SC, Chaney EF, Felker B, et al: Effectiveness of collaborative care depressiontreatment in Veterans’ Affairs primary care. Journal of General Internal Medicine18(1):9–16, 2003.

4. Katon W, Von Korff M, Lin E, et al:. Collaborative management to achieve treatmentguidelines. Impact on depression in primary care. Journal of the American MedicalAssociation 273(13):1026–1031, 1995.

5. Bond GR, Resnick SG, Drake RE, Xie H, et al: Does competitive employmentimprove non-vocational outcomes for people with severe mental illness? Journal ofConsulting and Clinical Psychology 69(3):489–501, 2001.

6. Kashner TM, Rosenheck RA, Campinell AB, et al: Impact of work therapy onhealth status among homeless substance dependent veterans. Archives of GeneralPsychiatry 59(10):938–945, 2002.

7. Salyers MP, Becker DR, Drake RE, et al: A ten-year follow-up of a supportedemployment program. Psychiatric Services 55(3):302–308, 2004.

8. Phillips SD, Burns BJ, Edgar ER, et al: Assertive Community Treatment: Mov-ing an evidence-based intervention into standard practice. Psychiatric Services52:771–779, 2001.

9. Rosenheck RA, Neale MS: Cost-Effectiveness of intensive psychiatric commu-nity care for high users of inpatient services. Archives of General Psychiatry55(5):459–466, 1998.

10. Stein LI, Test MA: Alternative to mental hospital treatment: Conceptual model,treatment program, and clinical evaluation. Archives of General Psychiatry37(4):392–397, 1980.

11. Fontana A, Rosenheck RA: Effectiveness and cost of inpatient treatment ofPosttraumatic Stress Disorder. American Journal of Psychiatry 154:758–765, 1997.

12. Rosenheck RA, Fontana AF: Treatment of veterans severely impaired by PTSD, inEmotional Aftermath of the Persian Gulf War. Edited by Ursano RJ, Norwood AE.Washington DC: American Psychiatric Press, 1996.

13. Linn MW, Caffey EM: Foster placement for the older psychiatric patient. Journal ofGerontology 32(3):340–345, 1977.

14. Linn MW, Caffey EM, Klett CJ, et al: Hospital vs community (foster) care forpsychiatric patients. Archives of General Psychiatry 34(1):78–83, 1977.

15. Mares A, McGuire J. Reducing psychiatric hospitalization among mentally illveterans living in board-and-care homes. Psychiatric Services 51(7):914–921, 2000.


16. D’Agostino D, McCabe J, Sclar B: A psychiatric day hospital program in an HMO.HMO Practice 9(2):79–83, 1995.

17. Linn MW, Caffey EM Jr, Klett CJ, et al: Day treatment and psychotropic drugs in theaftercare of schizophrenic patients. Archives of General Psychiatry 36:1055–1066,1979.

18. Weisner C, Mertens J, Parthasarathy S, et al: The outcome and cost of alcohol anddrug treatment in an HMO: day hospital versus traditional outpatient regimens.Health Services Research 35(4):791–812, 2000.

19. Jerrell JM, Hu TW: Estimating the cost impact of three dual diagnosis treatmentprograms. Evaluation Review 20(2):160–180, 1996.

20. McLellan AT, Grissom GR, Brill P, et al: Private substance abuse treatments: aresome programs more effective than others? Journal of Substance Abuse Treatment10(3):243–254, 1993.

21. Moos RH, Finney JW, Federman EB, et al: Specialty mental health care improvespatients’ outcomes: Findings from a nationwide program to monitor the qualityof care for patients with substance use disorders. Journal of Studies on Alcohol61:704–713, 2000.

22. Rawson RA, Shoptaw SJ, Obert JL, et al: An intensive outpatient approach forcocaine abuse treatment. The Matrix model. Journal of Substance Abuse Treatment12(2):117–127, 1995.

23. Van Stone W, Henderson K, Moos RH, et al: In Nationwide implementation of GlobalAssessment of Functioning as an indicator of patient severity and service outcomesStandardized Evaluation in Clinical Practice. Edited by First MB. Washington, DC,American Psychiatric Publishing Inc, 2003.

24. Greenberg G, Rosenheck R: Using the GAF as a national mental health outcomemeasure in the Veterans Health Administration. Psychiatric Services 56(4):420–426,2005.

25. Greenberg G, Rosenheck R: Continuity of care and clinical outcomes in a nationalhealth system 56 (4): 427–433, 2005.

26. Sernyak M, Leslie DL, Rosenheck RA: Use of system-wide outcomes monitoringdata to compare the effectiveness of atypical neuroleptic medications. AmericanJournal of Psychiatry 160(2):310–315, 2003.

27. Resnick R, Baldino R, Corwel L, et al: Seventh Progress Report on the CompensatedWork Therapy/Veterans Industries Program. Fiscal Year 2003. West Haven, CT,Northeast Program Evaluation Center, 2004.

28. Neale M, Rosenheck R, Castrodonatti J, et al: Mental Health Intensive CaseManagement (MHICM): The Seventh National Performance Monitoring Report: FY2003. West Haven, CT, Northeast Program Evaluation Center, 2004.

29. Fontana A, Rosenheck R, Spencer H, et al: Long Journey Home XII: Treatment ofPosttraumatic Stress Disorder in the Department of Veterans Affairs: Fiscal Year2003. West Haven, CT, Northeast Program Evaluation Center, 2004.

30. Humphreys K, Huebsch PD, Moos RH, et al: The transformation of the veteransaffairs substance abuse treatment system. Psychiatric Services 50(11):1399–1401,1999.

31. Endicott J, Spitzer RL, Fleiss JL: The Global Assessment Scale: A procedure formeasuring the overall severity of psychiatric disturbance. Archives of GeneralPsychiatry 33:766–771, 1976.

32. Tracy K, Adler LA, Rotrosen J, et al: Interrater reliability issues in multicenter trials,Part 1: Theoretical concepts and operational procedures used in Department of Vet-erans Affairs cooperative study #394. Psychopharmacology Bulletin 33:53–57, 1997.


33. Rey JM, Starling J, Wever C, et al: Inter-rater reliability of global assessmentof functioning in a clinical setting. Journal of Child Psychology and Psychiatry36(5):787–792, 1995.

34. American Psychiatric Association: Diagnostics and Statistics Manual of MentalDisorders, 4th ed. Washington: DC, American Psychiatric Association, 1994.

35. Moos RH, Nichol AC, Moos BS: Global assessment of functioning (GAF) ratings:Global Assessment of Functioning ratings and the allocation and outcomes ofmental health services. Psychiatric Services 53(6):730–737, 2002.

36. Rosenheck RA, Cicchetti D: A Mental Health Program Performance MonitoringSystem for the Department of Veterans Affairs. West Haven, CT, Northeast ProgramEvaluation Center, 1995.

37. Rosenheck RA, Cramer J, Xu W, et al: A comparison of Clozapine and Haloperidolin hospitalized patients with refractory schizophrenia. New England Journal ofMedicine 337:809–815, 1997.

38. Bryk AS, Raudenbush SW: Hierarchical Linear Models. London, Sage, 1992.39. Desai RA, Rosenheck RA, Rothberg, A: Cross-system use among VA mental health

patients living in Philadelphia. Administration and Policy in Mental Health28:299–309, 2001.

40. Rosenheck RA, Frisman L, Essock S: Impact of VA bed closures on use of state psychi-atric services. Journal of Behavioral Health Services and Research 28:58–66, 2001.

41. Greenberg GA, Rosenheck RA: National Mental Health Program PerformanceMonitoring System: Fiscal Year 2003 Report. West Haven, CT, Northeast ProgramEvaluation Center; 2004.

42. Ellwood PM: Shattuck Lecture – Outcomes management: A technology of patientexperience. New England Journal of Medicine 318(23):1549–1556, 1988.

43. Epstein AM: The outcomes movement – will get us where we want to go? NewEngland Journal of Medicine 323(4):265–270, 1990.

44. Marshall MN, Shekelle PG, Leatherman S, et al: The public release of performancedata: What do we expect to gain? Journal of the American Medical Association283(14):1866–1874, 2000.

45. Relman AS: Assessment and accountability: The third revolution in medical care.New England Journal of Medicine 319(18):1220–1222, 1988.

46. Roper WL, Winkenwerder W, Hackbarth GM, et al:. Effectiveness in health care,An initiative to evaluate and improve medical practice. New England Journal ofMedicine 319(18):1197–1202, 1988.

USE OF NATIONWIDE OUTCOMES MONITORING DATA TO COMPARE CLINICAL OUTCOMES IN SPECIALIZED MENTAL HEALTH PROGRAMS AND GENERAL PSYCHIATRIC CLINICS IN THE VETERANS HEALTH ADMINISTRATION

Documents