Consumer Involvement in Evaluating a Recovery-based Systems Change Initiative Vicki Cousins, Director Office of Consumer Affairs South Carolina Department.

Consumer Involvement Consumer Involvement in Evaluating a Recovery-based in Evaluating a Recovery-based

Systems Change InitiativeSystems Change Initiative

Vicki Cousins, DirectorVicki Cousins, DirectorOffice of Consumer AffairsOffice of Consumer Affairs

South Carolina Department of Mental HealthSouth Carolina Department of Mental Health

Katherine M. Roberts, MPHKatherine M. Roberts, MPHCoordinator, Consumer-to-Consumer Evaluation TeamCoordinator, Consumer-to-Consumer Evaluation Team

South Carolina Department of Mental HealthSouth Carolina Department of Mental Health

Words to Live ByWords to Live Byin South Carolinain South Carolina

““Nothing About Us Nothing About Us Without Us”Without Us”

Consumer Involvement in Four Key Consumer Involvement in Four Key AreasAreas

PlanningPlanning Policy-makingPolicy-making Service ProvisionService Provision Program Evaluation Program Evaluation

Consumer Involvement in PlanningConsumer Involvement in Planningand Policy-makingand Policy-making

Central Administration Central Administration OfficesOffices

Regional Service Regional Service Sites Sites (Community Mental (Community Mental Health Centers in 17 Health Centers in 17 locations serving 46 locations serving 46 counties)counties)

Consumer Involvement in Planning Consumer Involvement in Planning and Policy-makingand Policy-making

New SCDMH New SCDMH consumer-centered consumer-centered

policies/directives policies/directives being developed being developed

as a result of as a result of our system’s our system’s emphasis on emphasis on

recovery.recovery.

Consumer Involvement Consumer Involvement in Service Provisionin Service Provision

Following Larry Fricks’ Georgia model, Following Larry Fricks’ Georgia model, South Carolina has developed a Medicaid-billable South Carolina has developed a Medicaid-billable

service description for peer supportservice description for peer support

40-hr. certification training for consumers to 40-hr. certification training for consumers to become credentialed as “CPSS”become credentialed as “CPSS”

Includes oral and written examinationIncludes oral and written examination Final approval from DHSS expected July 1, 2003Final approval from DHSS expected July 1, 2003 10 graduates to date; trainings to be year twice 10 graduates to date; trainings to be year twice

a yeara year

Consumer Involvement in Consumer Involvement in Program EvaluationProgram Evaluation

SCDMH Consumer-to-Consumer Evaluation TeamSCDMH Consumer-to-Consumer Evaluation Team

Several Years in the MakingSeveral Years in the Making

1996 …1996 …

… … 20022002 Found our place in the system’s Found our place in the system’s recovery initiativerecovery initiative

The Purpose of The Purpose of Consumer InvolvementConsumer Involvement

Surgeon General’s Report Surgeon General’s Report 19991999

Consumer staff, clients, and Consumer staff, clients, and the mental health system the mental health system allall benefit from consumer benefit from consumer

involvement.involvement.

The Purpose of The Purpose of Consumer Involvement Consumer Involvement a Mental Health Systema Mental Health System

Surgeon General’s Report 1999Surgeon General’s Report 1999

Consumer StaffConsumer Staff meaningful activity or meaningful activity or

employmentemployment role models of recoveryrole models of recovery enhance sensitivity of enhance sensitivity of

the service the service systemsystem

The Purpose of The Purpose of Consumer Involvement Consumer Involvement a Mental Health Systema Mental Health System

Service system benefits:Service system benefits: Increases in service satisfaction Increases in service satisfaction

as as related to people’s recovery needsrelated to people’s recovery needs Reduced hospitalizationsReduced hospitalizations Cost-savings in community Cost-savings in community

program program operating costsoperating costs(Chamberlin & Rogers, 1990; Carpinello & Knight, 1993)(Chamberlin & Rogers, 1990; Carpinello & Knight, 1993)

SCDMH Consumer-to-Consumer SCDMH Consumer-to-Consumer Evaluation TeamEvaluation Team

Project MissionProject Mission

To involve individuals who receive services, family members, independent To involve individuals who receive services, family members, independent

advocates and citizens-at-large in program evaluation and continuous quality advocates and citizens-at-large in program evaluation and continuous quality

improvement of the South Carolina Department of Mental Healthimprovement of the South Carolina Department of Mental Health. .

SCDMH Consumer-to-Consumer SCDMH Consumer-to-Consumer Evaluation TeamEvaluation Team

Office of Consumer Affairs, Division of Health Office of Consumer Affairs, Division of Health Care ReformCare Reform

Consumer-driven evaluation survey processConsumer-driven evaluation survey process Primary consumer team membersPrimary consumer team members Paid employees of SCDMHPaid employees of SCDMH

Participates in instrument developmentParticipates in instrument development Conducts interviews, compiles data, and Conducts interviews, compiles data, and

produces reportsproduces reports

History of Consumer Involvement in History of Consumer Involvement in EvaluationEvaluation

For several years in the late 1990s, the Team For several years in the late 1990s, the Team conducted open-ended surveys with over 900 conducted open-ended surveys with over 900 clients, striving to evaluate individual CMHC clients, striving to evaluate individual CMHC programs.programs.

Starting in 2002, a fast evolution began as the Starting in 2002, a fast evolution began as the emphasis shifted from program evaluation to emphasis shifted from program evaluation to system evaluation. Survey questions became system evaluation. Survey questions became closed ended.closed ended.

History of Recovery Measurement in History of Recovery Measurement in South CarolinaSouth Carolina

March 2002: Formal planning process for March 2002: Formal planning process for recovery system evaluation begins. recovery system evaluation begins. Expert researchers partner with the CCET.Expert researchers partner with the CCET.

June 2002: Consumer focus groups, staff June 2002: Consumer focus groups, staff surveyssurveys

August–October 2002: Surveying to test August–October 2002: Surveying to test for validity and reliabilityfor validity and reliability

InstrumentsInstruments

MHSIPMHSIP

Consumer-to-Consumer Evaluation Scale Consumer-to-Consumer Evaluation Scale (CCET)(CCET)

Recovery QuestionnaireRecovery Questionnaire

South Carolina Consumers South Carolina Consumers Define Recovery Define Recovery

Focus GroupsFocus Groups

What does it mean to be in recovery or What does it mean to be in recovery or recovered from mental illness?recovered from mental illness?

What helps a person recover from mental What helps a person recover from mental illness?illness?

What prevents a person from recovering from What prevents a person from recovering from mental illness?mental illness?

Recovery Questionnaire Domains Recovery Questionnaire Domains (10) from the Focus Groups(10) from the Focus Groups

Hope - Self-esteem - HousingHope - Self-esteem - Housing

Employment - Stigma - Self-management Skills Employment - Stigma - Self-management Skills

Empowerment/Independence/Self Empowerment/Independence/Self Responsibility/Control Responsibility/Control

Relationships - SocialRelationships - Social

Relationships - Family Relationships - Family Spirituality/Higher Power/ReligionSpirituality/Higher Power/Religion

Sample SizeSample Size

Plan A:Plan A: Sample consisted of 50 adult clients with Sample consisted of 50 adult clients with open cases in all 17 community mental health centers. open cases in all 17 community mental health centers.

ExpectationExpectation: 25 (425) would complete the initial survey : 25 (425) would complete the initial survey and half of that number would agree to be resurveyed 2 and half of that number would agree to be resurveyed 2 to 4 weeks later. to 4 weeks later.

Plan A Results: <10% available to participate.Plan A Results: <10% available to participate.

Plan B:Plan B: Asked for available adult clients with open Asked for available adult clients with open cases to volunteer. cases to volunteer.

MethodMethod

Informed consent Informed consent attained. attained.

Initial face-to-face Initial face-to-face interviews conducted by interviews conducted by CCET team members. CCET team members.

Follow-up surveys were Follow-up surveys were conducted either face-to-conducted either face-to-face or by phone. The face or by phone. The process was the process was the choicechoice of of the client.the client.

Incentives for Client ParticipationIncentives for Client Participation

Each client received a button, a $5 Each client received a button, a $5 McDonalds gift certificate, and a certificate McDonalds gift certificate, and a certificate of appreciation signed by the SC State of appreciation signed by the SC State Mental Health Director, George P. Gintoli, Mental Health Director, George P. Gintoli, for the initial interview.for the initial interview.

On the second interview, clients received On the second interview, clients received 2 more buttons and an additional gift 2 more buttons and an additional gift certificate.certificate.

ProcessProcess

400+ clients were surveyed/resurveyed from 400+ clients were surveyed/resurveyed from mid-August through mid-October 2002.mid-August through mid-October 2002.

Scannable forms were reviewed and prepared Scannable forms were reviewed and prepared for data entry. for data entry.

All analysis was conducted in a de-identified All analysis was conducted in a de-identified manner.manner.

AnalysisAnalysis

The psychometric properties of the MHSIP, The psychometric properties of the MHSIP, the Recovery Questionnaire, and the the Recovery Questionnaire, and the Consumer-to-Consumer Evaluation ScaleConsumer-to-Consumer Evaluation Scaleincluded:included:

Item-to-Scale CorrelationsItem-to-Scale Correlations

Test-Retest ReliabilityTest-Retest Reliability

ResultsResults

Based on the research, changes were made to the Based on the research, changes were made to the Recovery Questionnaire and to the Consumer-to-Recovery Questionnaire and to the Consumer-to-Consumer Evaluation Scale.Consumer Evaluation Scale. The Recovery Questionnaire is presently being used by The Recovery Questionnaire is presently being used by SC Share in South Carolina to evaluate their “Recovery for SC Share in South Carolina to evaluate their “Recovery for Life” program. Life” program.

Plans are underway by South Carolina’s Columbia Area Plans are underway by South Carolina’s Columbia Area MH Center, the largest community mental health center, toMH Center, the largest community mental health center, toutilize the Recovery Questionnaire in June 2003. utilize the Recovery Questionnaire in June 2003.

Lessons LearnedLessons Learned

Challenging to go to a full-system survey approach in a Challenging to go to a full-system survey approach in a limited time period – 2 times in 8 weeks - from limited time period – 2 times in 8 weeks - from individual program site assignments.individual program site assignments.

Dramatic increase in consumer hours, conflicted with Dramatic increase in consumer hours, conflicted with consumers’ Medicaid benefits, stressed Team consumers’ Medicaid benefits, stressed Team Members.Members.

But still … state car break-downs, a crash, But still … state car break-downs, a crash, getting lost, getting sick, losing surveyorsgetting lost, getting sick, losing surveyors

… … It was a terrific learning experience with It was a terrific learning experience with positive outcomes. We survived it all. positive outcomes. We survived it all.

Consumer Involvement in Evaluating Consumer Involvement in Evaluating a Recovery-based Systems Change Initiativea Recovery-based Systems Change Initiative

Vicki Cousins, DirectorVicki Cousins, DirectorOffice of Consumer AffairsOffice of Consumer AffairsSouth Carolina Department of Mental HealthSouth Carolina Department of Mental Health

Katherine M. Roberts, MPHKatherine M. Roberts, MPHCoordinator, Consumer-to-Consumer Evaluation TeamCoordinator, Consumer-to-Consumer Evaluation TeamSouth Carolina Department of Mental HealthSouth Carolina Department of Mental Health