MHHSC PROGRAM Constructing a Cross-Site Evaluation of Ethnic Minority HIV Mental Health Services The Mental Health HIV Services Collaborative (MHHSC) Program.

MHHSC PROGRAM

Constructing a Cross-Site Evaluation of Ethnic Minority

HIV Mental Health Services

The Mental Health HIV Services

Collaborative (MHHSC) Program

Michael Costa, AbtBarbara J. Silver, CMHSMaria Madison, AbtTandiwe Njobe, AbtGabriela Garcia, Abt

MHHSC PROGRAM

Presentation Goals

Describe the MHHSC Program Components Context Relevance

Convey: The process of this collaborative. utilization-

focused evaluation Accomplishments to date (products) Uses of the outputs of products

MHHSC PROGRAM

Substance Abuse Mental Health Services Administration

(SAMHSA)

Center for Mental Health Services (CMHS)

Center for Substance Abuse Prevention (CSAP)

Center for Substance Abuse Treatment (CSAT)

MHHSC PROGRAM

SAMHSA HIV/AIDS HISTORY

Prior to 2001: CSAT funds targeted capacity/HIV/AIDS Substance Abuse treatment programs for African American, Hispanic/Latino, and other racial/ethnic minorities.

2001: CMHS funds a similar targeted/expanded capacity program for community based organizations (CBOs) serving African American, Hispanic/Latino, and other racial/ethnic minorities.

MHHSC PROGRAM

HIV Infection among People with Severe Mental Illness

Across all published studies, the rate of HIV infection among psychiatric patients is 10%, 25 times higher than that of the general population.

Cournos & McKinnon, 1997;Krakow et al., 1998;Rosenberg et al., 2001

HIV Prevalence in Psychiatric and General Populations

10

0.4

0

2

4

6

8

10

12

Psychiatric General

MHHSC PROGRAM

Number of partners

Number of risky or anonymous partners

Frequency of sex trading

Rates of coerced sexMcKinnon et.al., 1996, 1999

COMPARED TO GENERAL POPULATION, PATIENTS HAVE FEWER COMPARED TO GENERAL POPULATION, PATIENTS HAVE FEWER EPISODES OF SEX WITH A PARTNER, BUT THEY HAVE:EPISODES OF SEX WITH A PARTNER, BUT THEY HAVE:

Sexual Risk Behavior Among People With Severe Mental Illness:

MHHSC PROGRAM

Elevated risk for HIV infection in psychiatric

Patients

Risk factors: Alcohol and other drug use Unsafe sex Environmental circumstances (poverty, institutionalization, etc.)

Substance use is associated with both psychiatric symptoms and HIV risk

Psychiatric Disorders and Risk for HIV Infection

MHHSC PROGRAM

MHHSC Program

21 Mental Health Service Sites – CBOs at least 2 years experience in behavioral

health care services MH Centers, Substance Abuse facilities, Primary Health Care &/or HIV/AIDS clinics

Abt Associates, Inc. = Coordinating Center

MHHSC PROGRAM

MHHSC Program

Congressional requirement (CBC & CHC): provide these new HIV/AIDS-related mental health services in both traditional and non-traditional settings.

Funding for mental health treatment services and related case management only.

However, grantees are required to develop comprehensive integrated individual treatment plans and monitor primary and substance use treatment.

MHHSC PROGRAM

WHO ARE THE SERVICE SITES?

New HIV/AIDS-Related Services New services (no prior HIV/AIDS-related MH services)– 5 sites Expanded services – 16 sites

Service Delivery Settings Traditional (primarily clinic-based) – 13 sites Non-traditional (e.g., mobile treatment, ) – 1 site Both settings – 8 sites

Target Populations African American -19 sites Hispanic/Latino – 14 sites Haitian – 1 site Native American – 1 site

MHHSC PROGRAM

Demographics – Gender

61.90%

37.10%

0.90%0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

Male Female Other

MHHSC PROGRAM

Hispanic Latino/a = 30.65%

Demographics – Race / Ethnicity

Native American3%

Asian0%Pacific Islander

0%

White17%

Other Race30%

African American/ Black50%

MHHSC PROGRAM

Demographics - Age

Age Category PercentLess than 20 years 1.72%

20 – 24 years 4.68%

25 – 29 years 7.41%

30 – 34 years 10.76%

35 – 40 years 20.83%

40 – 44 years 22.78%

45 – 50 years 14.51%

50 – 54 years 8.74%

55 years and older 6.01%

MHHSC PROGRAM

DSM IV Diagnoses

27.21%

5.39%

50.51%

16.89%

0%

10%

20%

30%

40%

50%

60%

Single Dx Dual Dx Triple Dx More than 3 Dx

MHHSC PROGRAM

DSM IV Diagnoses Categories

9.44%

13.88%

6.63%

45.71%

24.33%

6%

12%

18%

24%

30%

36%

42%

48%

AdjustmentDisorder

AnxietyDisorder

BipolarDisorder

DepressiveDisorder

OtherIncluding HIV

Dementia

MHHSC PROGRAM

MHHSC Program Goals

Expand

Effective

Culturally Competent

Mental Health Services

For PLWHIV

In Minority Communities

MHHSC PROGRAM

MHHSC Capacity Building

The Coordinating Center provides technical assistance to grantees:Local and regional trainings (e.g., ethics, neuropsychology, cultural competence)Expert speakers at national meetings on topics of mental health and psychiatry, gender issues, consumer involvement, etc.Assistance with data collection and management

MHHSC PROGRAM

CULTURAL COMPETENCE

Strategies to Promote Culturally-Competence Service Delivery Demographically similar staffing as target population(s) Treatment facilities reflect cultural interests of target

group(s) (e.g., artwork/decor, waiting room music, etc.) Diversity/cultural competence training Consumer feedback

This is being addressed in great detail by the MHHSC Cultural Competence Subcommittee

MHHSC PROGRAM

MHHSC Utilization-Focused Cross-site Evaluation

Clinically/Programmatically Relevant Evaluation meet the needs of the clinical and other program staff in

their efforts to better serve their clients

The MHHSC cross-site evaluation is voluntary, except for submission of required aggregate data critical that the cross-site evaluation be clinically and

programmatically relevant and valuable, otherwise no site would agree to participate

MHHSC PROGRAM

Evaluation as a Pain in the NeckMental health/social service providers’ historical experience with evaluation Intrusive reporting requirement Necessary to satisfy Local, State and Federal funding

requirements.

Often data are never reported back to programs

Concerns that data will used to make them look bad or draw inappropriate comparisons between sites and providers

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Engaging the key stakeholdersLocal site-specific evaluatorsLocal site cliniciansProgram AdministratorsConsumer Advisory Boards (CABS)

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

EvaluationSubcommittee

(ESC)

Logic ModelWorkgroup

Eval/ClinicianCommunication

Workgroup

Site ProgramLogic Models

Site-SpecificEvaluation Summaries

Determination ofEvaluation Foci

Across Sites

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Face-to-FaceMeeting w/MHHSC

Stakeholders

X-Site CandidateEvaluation

Foci

MHHSC PROGRAM

X-Site Candidate Evaluation Foci

Foci by Quantitative and Qualitative Methods

0

5

10

15

20

25

Fre

q o

f Met

ho

ds

Acr

oss

Site

Pla

ns

Quantitative Methods

Qualitative Methods

MHHSC PROGRAM

Quantitative Methods Used Across Sites

Quantitative Methods

0

5

10

15

20

25

Clie

nt

Sat

isfa

ctio

nS

urv

ey

Ad

min

istr

ativ

eC

lien

t C

on

tact

Dat

aA

bst

ract

ion

CM

HS

GP

RA

(wit

h a

nd

wit

ho

ut

aug

men

tati

on

)

Su

rvey

Dev

elo

ped

by

Eva

luat

or

Clin

ical

Rec

ord

sA

bst

ract

ion

Inta

ke a

nd

Ass

essm

ent

To

ol

Fre

q A

cro

ss S

ite

Pla

ns Quantitative Methods

MHHSC PROGRAM

Qualitative Methods Used Across SitesQualitative Methods

0

5

10

15

20

25

Focus Groups Interviews Observation ClinicalRecordReview

Fre

q A

cro

ss

Sit

e P

lan

s

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Breakout Groups(BOGs)

EvaluationDesign

Workgroup

MHHSCNationalMeeting

ConsumerNetwork

Committee

X-Site CandidateEvaluation

Foci

Face-to-FaceMeeting w/MHHSC

Stakeholders

MHHSC PROGRAM

MHHSC X-Site Evaluation Collaborative Process

Next Steps

Pilot Test

Instruments/Protocols

Final DraftField X-SiteEvaluation

Pray Everything Goes As Planned

Instruments/ProtocolsBeta Version

X-Site DesignSite Review

MHHSC PROGRAM

Final Cross Site Foci

Four main domains* of interest across sites:Mental HealthClient SatisfactionClient Retention and Service UtilizationQuality of Life & Medical Health

*Cultural Competence appears in all domains.

MHHSC PROGRAM

Common Questions of Interest

Who is being served? What are the barriers to care and to services for the target population? How do programs overcome these barriers? What services are being used by the target population? Are the services being provided in a culturally competent manner?

MHHSC PROGRAM

Mental Health

Given the target population being served, what are the Prevailing mental health diagnoses?Co-occurring disorders? (e.g., substance use)Changes in mental health symptoms?Disorders that may be more prevalent with HIV

positive status?Physiological HIV disease factors that

contribute to mental health symptoms?

MHHSC PROGRAM

Client Satisfaction

Questions relating to care received by clients.What care is being received?What is the frequency of care?What is the level of client involvement in care?What is the setting in which care is received? Is the client satisfied with the care?

MHHSC PROGRAM

Client Retention & Service Utilization

Client RetentionHow many cases become active/inactive in a

given time period?What is the site definition for active/ inactive

clients?What are the site policies for case closing? What

reasons are given for case closings?What retention strategies have been effective

with the target population?

MHHSC PROGRAM

Client Retention & Service Utilization - continued.

Service UtilizationWhat is the type, frequency and duration of

services used?How are services provided?

Referrals in and out of programAgency linkages - in-house and external - to other

providersService setting – traditional/ non-traditional

MHHSC PROGRAM

Quality of Life

How does the quality of life status and general medical health of a client impact uponTreatment compliance (medications, ITP)?Response to treatment?Client retention? Service utilization?

How does quality of life status and general medical health change over time with treatment?

MHHSC PROGRAM

Methods & Indicators1. CLIENT LEVEL INSTRUMENT

confidential – unique client id

Demographics

Social supports

Acculturation

Substance use and risk behavior

Trauma

Medical health

Medication adherence

Clinicians report

MHHSC PROGRAM

Methods & Indicators2. CLIENT SATISFACTION SURVEY

anonymous

Client characteristics

Service utilization

CareClient involvement in care Access/ barriers to care Cultural competence in care Overall satisfaction with care

MHHSC PROGRAM

Methods & Indicators3. FOCUS GROUPS

with clients on site

Focus groups will provide qualitative backdrop to analyze quantitative data from client satisfaction survey and client instrument

Types of services used Satisfaction with services Barriers to care Cultural competency

MHHSC PROGRAM

Methods & Indicators4. SITE VISIT DATA COLLECTION

MHHSC Coordinating Center Staff

Continuum of services

Location of site Geographic setting Site community

Service setting (traditional / non traditional)

Client retention strategies

Site activities to overcome identified barriers to mental health care

MHHSC PROGRAM

Utility of Cross-Site Analyses From the Sites’ Perspective

Site buy-in has been an on-going process

Clinicians and local evaluators participate in the evaluation design work group

They took ownership of the evaluation design

Made decisions on utility of collecting certain types of data across sites

MHHSC PROGRAM

Client Focused Domain

Client Characteristics

Clients’ Presenting Diagnosis

Barriers/Access to Care (enhance/maintain Client’s Quality of Life/Health)

MHHSC PROGRAM

Program Context Domains

Program StructureCBO VS Large System of care Staffing patterns (FT, PT)

Sustainability EffortsPlan in place

MHHSC PROGRAM

Client Focused DomainsClient Characteristics Site Uses

Systematic documentation type of clinical services offered to what target population across the sites

Tailor services to specific populations

Identify (ID) staff training & resource needs

Determine success or not in reaching target population

ID changing client populations

Ability to pool data across all sites by client characteristics

Identify types of clients (positively) impacted by type of treatment

Learn from sites’ experiences

ID promising practices

MHHSC PROGRAM

Client Focused Domains (cont.)Mental Health Site UsesExamine patterns/changes in symptomatology. Are symptoms unique to a given population

Better assess clients

Determine client case-mix

Document increased client needs

Gain insight on co-occurring disorders

Better understand client needs

Tailor services offered

Inform client retention strategies

Examine patterns of mental health disorders across sites

Understand impact of services

Inform site staff on cultural competence issues

MHHSC PROGRAM

Client Focused Domains (cont.)

Barriers / Access to Care Site Uses

Identify barriers to care specific to target population

Improve engagement

Improve retention

ID methods for overcoming stigma

Determine strategies for inter-agency collaboration

Gain insight into cultural competence issues

Gain better understanding of ease of access to mental health services

MHHSC PROGRAM

Program Context DomainsProgram Structure Site Uses

Understand contexts in which other sites operate

Determine relevance/applicability of findings across sites

Identify replicable program components

Understand uniqueness of site challenges

Learn from other sites' experiences about ways to overcome barriers to care

Identify promising practices

Create a menu of tools to address barriers

MHHSC PROGRAM

Program Context Domains (cont.)

Site Program Structure (cont’d)

Site Uses

Understand differences in the range of program services

Identify gaps in services

Identify staffing/resource needs

Inform program expansion and sustainability

Identify promising practices in client referrals

Improve client treatment fidelity

Improve client retention

Create a menu of tools to address barriers to care

MHHSC PROGRAM

Program Context Domains (cont.)

Sustainability Site Uses

Examine data on services provided to support funding opportunities

Develop sustainability plans at individual site or cross-site level if similarity in client needs

Develop plan to target specific funding sources

MHHSC PROGRAM

Purposes/Uses of Data

Cross-site analysis Attention to differences across sites, client

characteristics Careful & appropriate Nuanced, not reductionist Program context critical Use of qualitative and program-level data (e.g.,

differences in resources/capacity available)

MHHSC PROGRAM

Purposes/Uses of Data (Recap)UTILIZATION-FOCUSED EVALUATIONData/analysis to be provided to sites User friendly feedback A timely manner For program improvement /development of

better intervention strategiesNOT REPORT CARDS Individual clinicians Individual sites