Toolkit for Evaluation, Assessment, and Monitoring for ...

Toolkit for Evaluation, Assessment, and Monitoring for Adult Community Services and Supports Programs

Sacramento Regional MeetingMarch 22, 2017

Agenda Overview

Time Activity

11:30 – 12:00 Welcome and Introduction (Conference call begins at 11:30)Overview of Toolkit

12:00 – 1:00 Lunch (Presentations will resume at 12:30)Background and rationale of Toolkit

1:00 – 2:30Toolkit Components (Conference call ends at 2:30)

Preview of Toolkit componentsQuestions/Discussion

2:30 – 2:45 Break

2:45 – 3:45 Breakout groupsRound table discussions about specific components of the Toolkit

3:45 – 4:15 Wrap up

4:15 – 4:30 Final Comments

Project Team

Health Services Research Center (HSRC) Andrew Sarkin, PhD

Project Director

Frances Reyes, MAProject Manager

Gina Beal, MPA, MAProgram Evaluation Associate

Kyle Choi, MPHInformatics Project Manager

Edith Wilson, PhDResearch Associate

Lucyna Klinicka, BAProgram Evaluation Specialist

Jessica Rampton, MPHResearch Evaluation Specialist

Justine Zimiles, BAProgram Evaluation Specialist

Rami Hamid, MBBS, MPSResearch Assistant

Merissa Kado, BAResearch Assistant

Nicholas Panagon, MPHResearch Assistant

Project Team

Mental Health Services Oversight and Accountability Commission (MHSOAC)

Fred Molitor, PhDDirector of Research and Evaluation

Marijoyce Naguit, MPHSenior Researcher

Kayla Landry, BAResearch Associate

Toolkit Overview

What is the Toolkit for Evaluation, Assessment, and Monitoring(TEAM)?

A set of resources and information provided by MHSOAC to support and empower California counties and programs to: Determine readiness for adopting a tracking, monitoring,

and evaluation systems for collecting and reporting evaluation data

Implement the measures and data collection processes successfully, and provide training

Create and use evaluation reports on various data collected from their systems

Guide the program/county on which parts of the Toolkit should be used based on their specific needs

Toolkit Components

1. Introductory materials2. Feasibility checklist3. Decision support tools4. Guidance on integration into electronic data

collection systems5. Data-entry and reporting tools6. Training materials7. Supplemental documents

Why was the Toolkit for developed?

MHSOAC wants to empower all counties to evaluate their behavioral health services in meaningful ways, without adding additional state requirements.

Counties and programs need better resources to help them collect, process, and analyze data that will inform service improvement and resource planning.

Clinicians want data that is clinically meaningful and will contribute to caring for the people they serve.

Clients and their families want to have a strong voice in their own care.

The toolkit is designed to address these needs and more!

How can the Toolkit be used?

Implement all elements from the Toolkit to develop a plan, provide assessment measures, collect data, and report evaluation outcomes.

Use certain components of the Toolkit to align with local needs and resources.

Select features from the Toolkit that will supplement clinical work and evaluation measures already used.

Who should use the Toolkit?

Clinical or Medical Directors Program Directors and

Managers Mental Health Researchers Program Evaluation and

Quality Improvement Analysts

County or Program Informatics Coordinators

Behavioral Healthcare Providers

Presentation resumes at 12:30

Toolkit Background and Rationale

Continuous Program Improvement Identifying training and technical assistance needs

Test program changes or new programs

Identifying client groups that need attention

Accountability to Stakeholders Staff, Funders, Community, Clients and Families

Securing Future Funding

Importance of MeasurementWhy invest time in program evaluation activities?

Tracking Individual Progress and Treatment Plans Promoting an Integrated Recovery Orientation Facilitating Recovery and Communication

Between people receiving and providing services

Between service providers in an integrated system

Importance of MeasurementHow will this help our individual clients?

To better understand the people we are helping To comprehensively assess a person’s needs To aid in treatment planning and goal setting To assess recovery progress and outcomes Identify critical indicators Identify strengths that can aid in recovery Enhancing the therapeutic dialogue to

Increase client involvement in treatment planning

Set shared goals and monitor recovery together

Increase sharing of information between team members

Clinical Utility of Evaluation Data

PHASE ICSS Tracking, Monitoring, and Evaluation Project

Phase I: CSS Tracking, Monitoring, and Evaluation Project

TimeframeMay 2014 – June 2016 (completed successfully)

Goals Develop and pilot a standardized data collection system for adults

receiving non-FSP CSS services

Allows for evaluation of clients and services

Can be integrated into other Electronic Health Records or Data Collection and Reporting Systems (such as eBHS and Avatar).

Creation of policy and practice recommendations for how to improve upon current CSS services evaluation and quality improvement systems.

Phase I: CSS Tracking, Monitoring, and Evaluation Project

Primary Evaluation Questions Answered What statewide methods should be employed to ensure proper

tracking, monitoring, and evaluation of adults receiving CSS services?

What policies, practices, systems, and infrastructure should be created and/or modified to better track, monitor, and evaluate adults receiving CSS services?

What policies, practices, systems, and infrastructure should be created and/or modified to better use information to serve adults within the CSS component?

Domains and Data Elements

Review of relevant documents and guidelines Mental Health Services Act requirements and goals Bronzan-McCorquodale and other laws MHSOAC and DHCS goals National Behavioral Healthcare Quality Framework Relevant published work on measuring outcomes Results of other similar projects

MOQA, MHDATA DCR Tools, County-level efforts, SAMHSA, etc.

Stakeholders and the Evaluation Advisory Group provided input on what measures to implement and what measurement procedures to use

Review of current systems such as CSI, DCR, and Electronic Health Records and other Data Collection Systems utilized

Evaluation and comparison of validated health measures

Validated Measures Review

Contract requires the use of validated measures.

Data elements in the DCR have mostly not been validated, and some indicators have been shown to have significant validity problems.

No validated measure would cover everything, so we had to supplement with DCR-type items.

Validated measures allow for standardized comparisons to other programs, and data-based performance criteria.

Assessment Instrument Quality Checklist (AIQC)

Measures Viewer Survey with stakeholders

Maximize Usefulness to Staff

Usefulness to People Getting Services

Validity for Measuring Goals and Outcomes

Cultural Competence and Sensitivity

Minimize Burden to Staff

Burden to People Getting Services

Costs to the Programs

Selection of MeasuresQualities

Selection of MeasuresProcess

Review Available Measures

Relevance to Goals

Clinical Utility

Psychometric Validity

Cultural Competence

Cost, Copyright, and Translation Issues

Development of Some Questions

Identify needs

Preference to validated instruments

Meetings with Stakeholders

Program Directors

CORs and Administration

Local Experts and Academics

Staff delivering services

People getting services

Contributing Stakeholders

MHSOAC California Department of Health Care Services (DHCS) Racial and Ethnic Mental Health Disparities Coalition (REMHDCO) California Mental Health Planning Council County Administrators and Contract Supervisors Clinicians and Staff Subject Matter Experts Policy Makers (Bruce Bronzan, Rusty Selix, more) People with lived experience who use services Family members of people with lived experience Evaluation Advisory Group

Assembled information from stakeholders, current legislation, mental health agencies, existing data collection systems, outcome measures, and reviews of validated instruments

Developed a system to include outcome measures, assessment completion protocols, and data collection and reporting, that can work across multiple systems

Implemented a pilot system in fall 2015 to spring 2016

Gathered feedback from end user surveys, focus groups, and the Mental Health Services Evaluation Advisory Group (MHSEAG)

Developed or adapted measures to fill gaps expressed by stakeholders, such as a measure for family/friends and a strength-based self-report

Sample of Products from Phase I

Clinical Measure Recommendations Use of validated, recovery-oriented outcome measures A measure completed by family member or other close supporter Client-completed measures adapted for use with various populations

Data Collection System Recommendations Optional data elements that programs may choose to complete Data collection software that is integrated with other electronic health

record systems

Reporting Recommendations Customizable reporting formats designed to ensure the usability of

data collected for various audiences Training for clinicians in using the data to provide better services

Sample of Select Recommendations

PHASE II

Toolkit for Evaluation, Assessment, and Monitoring (TEAM)

Purpose of the Project

Development of a toolkit that will:

Provide guidance on how to prepare for a tracking, monitoring, and evaluation system

Help counties and programs determine what measures to implement, and how to implement them

Aid MHSOAC in providing assistance to counties planning to adopt a tracking, monitoring, and evaluation system

Improve mechanisms for the negotiation and sharing standardized client outcomes data

Enhance the clinical utility of new or current tracking, monitoring, and evaluation systems and improve how data is reported

Toolkit Development

Reviewed results and recommendations from Phase I

Developed the project plan and timeline in collaboration with the MHSOAC

Selected subject matter experts and key stakeholders

Formed a Toolkit Evaluation Advisory Group (TEAG) and conducted expert interviews

Developed a draft toolkit

Now holding these regional meetings for your input

Create final toolkit for distribution

Project Timeline October 2016 – July 2017

October

November

December

January

February

March

April

May

June

July

Regional Meetings

March 15‐31

Prepare Final Toolkit and Recommendations for

ImplementationDevelop Draft Toolkit

Deliverable 1: Work Plan

October 2016

Deliverable 2: Draft ToolkitFebruary 2017

Deliverable 3: Regional Meetings

ReportMay 2017

Deliverable 4: Final ToolkitJuly 2017

Toolkit Evaluation Advisory Group (TEAG)

Review toolkit materials

Ensure needs of staff and clients are addressed to increase clinical utility

Discuss strategies to maximize the utility of the toolkit for all stakeholders

Help to plan dissemination of the toolkit

TEAG Members

Dave Pilon Dawn Kaiser Joshua Morgan Lezlie Murch Liz Miles Marshall Lewis

Patricia Wentzel Rebecca Ballinger Rick Heller Saumitra SenGupta Sunjung Cho Tony Hobson

Toolkit Components

Toolkit Features

Materials to help counties and programs determine readiness for a data collection and

reporting system

Assessment measures for tracking, monitoring,

and evaluating client outcomes

Guidance on integrating a data

collection and reporting system into existing client health

record systems

Data entry templates and automated

dashboards depicting client progress

Materials that can be customized to meet

local needs

Recommendations for data analysis and

reporting

Toolkit Sections

Assessment Measures Integration into

Existing Data Collection Systems

Data Entry and Reporting

Preparation Promotional Materials

Toolkit Sections





Assessment Measures

The Toolkit includes a set of questionnaires and supplemental materials for tracking, monitoring, and evaluating client outcomes, including: Clinician and client-completed assessments Decision tools to aid in the selection of

measures and how to implement them Training materials with guidance on the

implementation of new measures

Minimize Burden to Staff Burden to People Getting Services Costs to the Programs

Maximize Usefulness to Staff Usefulness to People Getting Services Validity for Measuring Goals and Outcomes Cultural Competence and Sensitivity

Selection of Measures – Qualities

Selection of Measures – Process

Review Available Measures

Relevance to Goals

Clinical Utility

Psychometric Validity

Cultural Competence

Cost, Copyright, and Translation Issues

Development of Some Questions

Identify specific needs not covered by the validated measures

Preference to validated instruments when possible

Meetings with Stakeholders

Program Directors

County Contract Administrators

Local Experts and Academics

Staff delivering services

People getting services

Measures

Assessment Description

IntakeIntake InformationClient Information (Demographics)

Clinician

Key Indicators: Housing, Employment, Critical Events, and Health

Discharge Information

IMR, Goals, and MORS

CHAMPSSS(Integrated Self-Assessment)

Client-Rated OutcomesSatisfaction (brief)

IMR-FF Family/Friend-Rated Outcomes

Consumer Perception Survey Satisfaction (extended)

Clinician Assessment

Illness Management and Recovery Scales (IMR)

Milestones of Recovery Scale (MORS)

Progress towards goals

Housing level and stability

Employment

Frequency of critical events such as hospitalization

Demographics May already be collected in some systems and imported

Categories adhere to MHSA Innovations and PEI regulations

Illness Management and Recovery (IMR)

Captures clinician perspective of mental health recovery and wellness activities

15 Items, each with 5 anchored responses Measures:

Management of illness and symptom reduction Recovery-oriented behaviors and self-management Substance abuse

Administration Intake Follow-up Discharge

Sample IMR Items

Knowledge: How much do you feel s/he knows about symptoms, treatment, coping strategies (coping methods), and medication?

Not very much A little Some Quite a bit A great deal

Time in structured roles: How much time does s/he spend working, volunteering, being a student, being a parent, taking care of someone else or someone else’s house or apartment? That is, how much time does s/he spend in doing activities for or with another person that are expected of him/her? (This would not include self‐care or personal home maintenance.)

2 hours or less/week 3‐5 hours/week 6‐15 hours/week 16‐30 hours/week More than 30 hours/wk

Impairment of functioning: How much do symptoms get in the way of his/her doing things that s/he would like to do or need to do?

Symptoms really get in his/her way a lot

Symptoms get in his/her way quite a bit

Symptoms get in his/her way somewhat

Symptoms get in his/her way very little

Symptoms don't get in his/her way at all

Relapse of symptoms: When is the last time s/he had a relapse of symptoms (that is, when his/her symptoms have gotten much worse)?

Within the last month In the past 2 to 3 months

In the past 4 to 6 months


Hasn't had a relapse in the past year

Milestones of Recovery Scale (MORS)

1 item with 8 possible anchored responses

Assesses level of recovery/risk, engagement in treatment, and support needs

Administration Intake

Follow-up

Discharge

o 1. Extreme risko 2. Experiencing high risk/not engaged with mental health providerso 3. Experiencing high risk/engaged with mental health providerso 4. Not coping successfully/not engaged with mental health providerso 5. Not coping successfully/engaged with mental health providerso 6. Coping successfully/rehabilitating o 7. Early recoveryo 8. Advanced recoveryo Item not assessed

MORS

Goals

Clinician-completed items measuring seven goal areas

Administration Follow-up

Discharge

Goals

CHAMPSSS: Combined Health Assessment, Mental, Physical, Social, Substance, Strengths

Can be as short as one page with simple language

Includes PROMIS Global Health items

Data is comparable to a wide variety of state and national data being collected using NIH PROMIS

Screens and alerts for suicidality and relapse

Recovery-oriented and measures strengths

Optional substance abuse questions

Provides a general quality of life indicator

Supports cost effectiveness analysis using Quality Adjusted Life Years

CHAMPSSS Self-Report Domains

Strengths Mental Health

Physical Health

Social Health

Quality of Life Symptoms

Suicidality Substance Use

Family and Friend Version of the Illness Management and Recovery Scales

Only one page front and back with 15 items

Mirrors some questions on the clinician assessment

Recovery-oriented and measures strengths

Substance abuse questions and relapse alert from another view

Scales combine items to measure: Symptom Management

Participation in Wellness Activities

Substance Abuse

Sample IMR Items

Knowledge: How much do you feel s/he knows about symptoms, treatment, coping strategies (coping methods), and medication?

Not very much A little Some Quite a bit A great deal

Time in structured roles: How much time does s/he spend working, volunteering, being a student, being a parent, taking care of someone else or someone else’s house or apartment? That is, how much time does s/he spend in doing activities for or with another person that are expected of him/her? (This would not include self‐care or personal home maintenance.)

2 hours or less/week 3‐5 hours/week 6‐15 hours/week 16‐30 hours/week More than 30 hours/wk

Impairment of functioning: How much do symptoms get in the way of his/her doing things that s/he would like to do or need to do?

Symptoms really get in his/her way a lot

Symptoms get in his/her way quite a bit

Symptoms get in his/her way somewhat

Symptoms get in his/her way very little

Symptoms don't get in his/her way at all

Relapse of symptoms: When is the last time s/he had a relapse of symptoms (that is, when his/her symptoms have gotten much worse)?

Within the last month In the past 2 to 3 months



Hasn't had a relapse in the past year

Outcome Measures Decision Tool

Outcome Measures Decision Tool

Purpose of tool: Help counties/programs determine which measures and data elements to collect based on needs and program characteristics

Intended users: Teams or individuals from counties and programs who are deciding on the assessment measures

How to Use the Decision Tool

Example Based on the Clinician Assessment

Clinician Assessment

Are program staff available to complete

assessments?

Yes

Are mental health staff available (e.g., clinician, social worker,

peer specialist)?

YesGo to IMR, MORS, Goals, and Key Outcomes

No Go to Key Outcomes

No Clinician Assessment not completed

MORS

Is the evaluation of the level of

recovery needed? (MORS)

Yes

Are staff trained in MORS or are

resources available for

MORS training?

YesMORS completed (after training)

No MORS not completed

No MORS not completed

Clinician Assessment Example Should the program complete ….? Please start on the left.

Toolkit Sections





Electronic Data Capture (EDC) System Integration

The Toolkit provides technical assistance for integrating your EDC with Toolkit. Offers variations of integration and implementation of the

Toolkit Describes the steps for a data integration plan and

exporting data from your EDC Provides details for integrating the Toolkit into common

EDCs

Toolkit Integration FlowchartDecision-making tool to determine the best way to capture and integrate data

EDC Integration Pathways (Variation 1 of 4)

Direct Data Entry Best solution for

small counties Easiest set up Minimizes

paperwork


Entry Into Local EDC and Toolkit Spreadsheets Can use existing

EDC Use spreadsheet

for variables not captured in EDC


Using Existing EDC and Paper Forms Solution for systems that are unable to add new

data elements


Collection Using Forms and Entry into EDC and Toolkit Spreadsheets Option for

programs that must use paper forms

Great if computer access is limited

Data Integration Plan

Data Integration Plan

Integrating the Toolkit into Common EDC systems

Data Integration information available for: Anasazi & Cerner Community Behavioral Health Avatar Clinician’s Gateway CSI & DCR eBHS Echo mHOMS

Toolkit Sections






The Toolkit contains instruments designed to aid in the processing of data captured through assessment measures.

Data entry

Statistical guidance

Data dashboards

Data Entry Tools and Dashboards

Basic Functionality Data entry and automated analysis in Excel

Build to support the Toolkit Assessments

Data Integration

Manually into Excel Spreadsheet

Data Entry Forms in Excel

Incorporating into EDC

Import From External Data Sources

Data Entry Option 1: Assessment Worksheet IMR Example

There is a separate tab in Excel for each of the 5 different assessment forms

Data Entry Option 1: Data ValidationIMR Example

Message box shows valid codes for variable

Warning message if code is invalid

Data Entry Option 2: Data Entry FormsIMR Example

Data can be entered directly into form

Data goes directly into Excel spreadsheet (Data Entry Option 1)

Data Entry Option 3: Importing data from Electronic Data Capture (EDC)

Working with main EDCs

Adding toolkit data elements to EDCs

Extracting data from EHR to fit spreadsheet format

Data Analysis

Flexible Time Frames

Flexible Date Ranges

Change Over Time

Data Reporting

Dashboards

Frequency Tables

Dashboards

Clinician Rated Outcomes

• Intake and Discharge Information• Demographics• Goals• Milestones of Recovery (MORS)• Illness Management and Recovery (IMR) Scale

Clinician Rated Outcomes – Key

Indicators

• Housing• Employment• Critical Events• Health

Client Rated Outcomes

• Combined Health Assessment, Mental, Physical, Social, Substance, Strengths (CHAMPSSS)

• Satisfaction ‐Mental Health Statistics Improvement Program (MHSIP)

Family or Friends Rated Outcomes

• Illness Management and Discovery (IMR) Scale (Family or Friends

Reports‐ Frequency Tables: Current Assessment IMR Example

Each variable has a frequency table

For categorical variables: data is also displayed in rank order

Reports‐ Frequency Tables: Previous and Current Assessment IMR Example

Compares the current assessment to the previous assessment using matched data.

Toolkit Sections





Preparation

The Toolkit includes materials to inform planning and preparation for implementing a system.

Feasibility Checklist

Data Use Agreement Template

Plan

ImplementReview

Improve

Feasibility Checklist

Purpose Encourages counties and programs to consider

organizational characteristics and resources for implementing the Toolkit or other data collection and reporting system

Helps determine if obstacles prevent successful implementation

Received feedback from EAG

Feasibility Checklist Domains and Example Items

Leadership

Staff – Resources

Staff – Time

Training

Technical Requirements

County administration/Leadership prioritizes and values data collection for evaluation and quality improvement purposes.


Leadership

Staff – Resources

Staff – Time

Training


We have adequate staffing to oversee/manage the assessment completion process.


Leadership

Staff – Resources

Staff – Time

Training


Staff have time to complete a 10‐20 minute intake assessment for each client.


Leadership

Staff – Resources

Staff – Time

Training


Our staff members are already trained on/or have time to review training materials about the importance of data collection, quality, and completeness.


Leadership

Staff – Resources

Staff – Time

Training


Our staff members have routine access to data analysis and reporting software (e.g., Excel).

Scores range from 1 (strongly disagree) to 5 (strongly agree)

Instructions

Consider current and likely future situations (within one year)

Calculation of domain scores (higher scores – greater feasibility)

5 different domains (e.g.,

leadership)

Completed in Excel or paper form

Scoring and InterpretationScores: Domain scores and overall feasibility score

Decision making and planning: Scores help identify specific areas needing improvement.

Interpretation: Feasibility of implementing a system based on your program's or county’s existing situation (current score) or future situation (future score).

Supporting Materials – Next Steps to Overcome Barriers

Developing a Plan

Data Use Agreement The assessments in this toolkit capture personally identifiable protected health information (PHI).

All appropriate security measures must be taken when storing, managing, and sharing data between organizations.

Benefits of Toolkit:

Includes DUA Templates to facilitate the sharing of data between organizations in order to carry out larger evaluations

This DUA Template is recommended for counties or programs that do not have an existing DUA form. Please adhere to local protocols for establishing DUAs.

Toolkit Sections





Promotional Materials

Toolkit announcements flyers

Email templates

Communicating importance of assessments to clients

Example: Training Announcement Email

Example

Training Announcement Flyer

Example: CHAMPSSS Coversheet

Promotional Materials

What other promotional materials are needed?

Questions

What are your overall impressions? Would you use certain measures/components from the Toolkit or would you be

using it as a whole? Would you incorporate Toolkit components in your own EHR? What other things would you like to see in the Toolkit (e.g., more validation) that

are not included? What toolkits for behavioral health evaluation have you used or seen in the

past? If you have used or seen a toolkit before, what did you like about it? If you have

seen a toolkit before, what did you not like about it? What challenges did you experience?

Do you have any recommendations for how to maximize the adoptability of a toolkit? What are some barriers to using a toolkit or challenges to adoptability?

How do you see yourself using such a toolkit? What else would you like to see in the Toolkit?

15 Minute Break

Breakout Groups

Breakout Group Instructions

Each table is assigned a facilitator responsible for taking notes

Spend first 5-10 minutes going over the questions on your handouts

After thinking about your responses, discuss your responses amongst your table.

Round Table Conclusions

Wrap up

Discussion Questions

How can we encourage people to use Toolkit?

What may be some challenges to implementing the Toolkit?

What do you think the next steps are?

Meeting Highlights

Final Comments

Next Steps

Synthesize information from this meeting

Revise Toolkit based on feedback

Submit the final draft of the Toolkit

Contact Information

Andrew Sarkin, Ph.D.Director of Evaluation Research

UCSD Health Services Research Center

[email protected]

Thank you!

Toolkit for Evaluation, Assessment, and Monitoring for ...

Documents