Evaluating Innovation, Part2 012512 1 January 25, 2012 Cricket Mitchell, PhD CiMH Consultant Evaluating Innovation Part 2 of 2 Introduction • At the cohort meeting in December we reviewed key components of Innovation evaluation – Articulating program elements and goals, measurement, appropriate points of comparison, utilizing a Logic Model, and selecting evaluation strategies • Today we will take a closer look at applying these key components to representative examples within each of the three cohort areas 2 Overview • Brief review of Part 1 • Example of Innovation Evaluation: New roles for peers – A closer look at the Logic Model example from Part 1 • Example 2: Culturally relevant services • Example 3: Primary care and behavioral health integration • Questions and discussion 3
27
Embed
Evaluating Innovation - mhsaresources.orgmhsaresources.org/.../05/EvaluatingInnovationPart2.pdf · Evaluating Innovation, Part2 012512 2 A note… • Examples are provided for illustrative
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Evaluating Innovation, Part2 012512 1
January 25, 2012Cricket Mitchell, PhD
CiMH Consultant
Evaluating InnovationPart 2 of 2
Introduction• At the cohort meeting in December we
reviewed key components of Innovation evaluation– Articulating program elements and goals,
measurement, appropriate points of comparison, utilizing a Logic Model, and selecting evaluation strategies
• Today we will take a closer look at applying these key components to representative examples within each of the three cohort areas
2
Overview
• Brief review of Part 1
• Example of Innovation Evaluation: New roles for peers– A closer look at the Logic Model example
from Part 1
• Example 2: Culturally relevant services
• Example 3: Primary care and behavioral health integration
• Questions and discussion
3
Evaluating Innovation, Part2 012512 2
A note…
• Examples are provided for illustrative
purposes; they are not intended to be exhaustive or precise.
4
Summary of Part 1• Innovation evaluation combines both process
evaluation and outcome evaluation, with the ultimate goal of contributing to learning– Evaluation for all Innovation projects should include
measurement of both process and outcome learning goals
– And, in order to understand data collected, the most appropriate point(s) of comparison should be identified
• As a flow chart representing the components of an Innovation program, a Logic Model serves as a guide for evaluation
5
Building and Utilizing a Logic Model: Review
• A Logic Model is a pictorial representation,
a flow chart, of the hypothesized relationships between program Inputs,
Activities, Outputs, and Outcomes
– Provides a guide for measurement and data collection activities
6
Evaluating Innovation, Part2 012512 3
Building and Utilizing a Logic Model: Review
• Inputs– Resources necessary to achieve objectives
• Activities– What the program does with the resources to
meet the objectives
• Outputs– Direct products of program activities
• Outcomes– Changes that result from the program’s activities
and outputs
7
Building and Utilizing a Logic Model: Review
• Linking the findings from process and
outcome learning goals back to independent program elements is the
heart of evaluation for Innovation
– The ultimate goal is to identify program elements that are related to the achievement
of desired goals and can be adopted or replicated by others
8
Diverse stakeholder
input is important in
all aspects of
Innovation evaluation
9
Evaluating Innovation, Part2 012512 4
10
Example: Evaluating a New Roles for Peers Innovation Project
A Closer Look at the Logic Model Example from Part 1
Logic Model Example
• Let’s say we have an Innovation program
that aims to improve outcomes for youth leaving foster care by pairing a TAY peer
mentor with an older youth as they are
discharged from the foster care system...
11
Articulating Program Elements
• Questions to ask:
– Who is being served?
• Older youth discharging from foster care
– Is this Innovation intended to bring change to
anyone else / anywhere else?
• Publicly-funded service agencies?
– e.g., by coordinating available services and supports
• Young adults with histories of foster care experience (i.e., peer mentors)?
12
Evaluating Innovation, Part2 012512 5
Articulating Program Elements
• Questions to ask:
– What are this program’s activities?
• Pairing a peer mentor with older youth discharging from
foster care
– What are the key strategies or processes by which the activities will be achieved?
• How will youth and mentors be paired?
• What training and supervision are mentors provided?
• How will collaborating agencies work toward coordinated services and supports?
13
Articulating Program Elements• Questions to ask:
– What is the timeline?• Is there a limited or proscribed amount of time that the
mentor will work with a peer?
• When would one expect to see change as a result of this Innovation?
– Who delivers the activities?• What qualifications or attributes are necessary for peer
mentors to be hired?
• Who will oversee the program? Who will train and supervise the peer mentors?
• How will representatives from collaborating agencies be recruited for participation? What agencies will be involved?
14
• Process learning goals and outcome
learning goals combine to inform a broader set of
findings and recommendations from Innovation
programs
15
Articulating Program Goals
Evaluating Innovation, Part2 012512 6
Articulating Program Goals
• Possible outcome learning goals…– Does this increase knowledge about available
services and supports?
– Does this increase coordination of available services?
– Does this increase participation in available services and supports?
– Does this increase rates of employment, education/vocation, independent living?
16
Articulating Program Goals
• Possible process learning goals…– What mentor attributes are most likely to
contribute to success?
– What aspects of collaboration among community-based agencies are most likely to contribute to success?
– What information do older youth leaving foster care find most useful?
– What activities, services, and supports are identified as most helpful?
17
Logic Model Example
•TAY Peer Mentors
•Reps from Collaborating Agencies (e.g., CW/SS, MH/BH, Ed, WFTD)
•Guiding principles, policies, and training
TAY Peer Mentor paired with TAY at time of d/c from FC
Peer-provided assistance with forms
Peer-provided emotional support
Coordinated service options
Coordinated service delivery
Inputs Activities OutcomesOutputs
TAY engagement in emotional and informational supports from Peer Mentor
TAY engagement in coordinated services and supports from community-based agencies
Increased utilization of services and supports by TAY exiting the FC system
Increased enrollment in educational and/or vocational opportunities Increased rates of
Independent Living
Increased rates of employment
Improved Quality of Life
18
Evaluating Innovation, Part2 012512 7
Logic Model Example
– Inputs:
• TAY peer mentors
• Representatives from collaborating
agencies
• Guiding principles, policies and training
19
Examples of Measurement for Inputs
•TAY Peer Mentors
•Reps from Collaborating Agencies (e.g., CW/SS, MH/BH, Ed, WFTD)
•Guiding principles, policies, and training
Inputs •How many TAY Peer Mentors are hired?
•Are they retained/do they stay employed?
•How many agencies are represented in the collaboration?
•How many representatives participate from each agency?
•Are they retained/do they consistently participate?
•How are the Guiding Principles, Policies, and Training protocols
developed and enacted?20
Logic Model Example
– Activities:
• Pairing TAY with peer mentor at time of discharge from FC
• Peer-provided emotional support, information, and assistance with completing paperwork and forms
• Coordination of service options by collaborating agencies
• Coordination of service delivery by collaborating agencies
21
Evaluating Innovation, Part2 012512 8
Examples of Measurement for Activities
TAY Peer Mentor paired with TAY at time of d/c from FC
Peer-provided assistance with forms
Peer-provided emotional support
Coordinated service options
Coordinated service delivery
Activities •How is the older TAY discharging from Foster Care paired with a Peer Mentor?•How often does the Peer Mentor provide emotional support?•Does the emotional, informational, and instrumental support provided meet the needs of the TAY?•How are service options presented to the TAY?•Are services delivered in a coordinated manner?
22
Logic Model Example
– Outputs:
• TAY engagement in supports offered by peer mentor
• TAY engagement in coordinated services and supports from collaborating agencies
23
Examples of Measurement for Outputs
Outputs
TAY engagement in emotional and informational supports from Peer Mentor
TAY engagement in coordinated services and supports from community-based agencies
•Does the TAY discharging
from Foster Care access
and utilize the emotional,
information, and
instrumental supports
offered/provided by the Peer
Mentor?•Does the TAY find them
helpful?
•Does the TAY engage in any of the coordinated
service options presented?•Does the TAY stay
enrolled in any of the services?
24
Evaluating Innovation, Part2 012512 9
Logic Model Example
– Outcomes:
• Increased utilization of services and supports by TAY exiting the foster care system
• Increased enrollment in educational and/or vocational opportunities
• Increased employment
• Increased rates of independent living
25
Examples of Measurement for Outcomes
Outcomes
Increased utilization of services and supports by TAY exiting the FC system
Increased enrollment in educational and/or vocational opportunities Increased rates of
Independent Living
Increased rates of employment
Improved Quality of Life
•Does the TAY enroll in any
educational or vocational
opportunities?
•Does the TAY seek employment?•Does the TAY gain employment?
•Does the TAY retain employment?
•Does the TAY utilize available
community-based services and supports?
•What is the TAY’s living environment
status six months, 12 months, 24
months after discharge from Foster Care?
•What is the TAY’s perceptions of
his/her Quality of Life?
26
Process Goals
•TAY Peer Mentors
•Reps from Collaborating Agencies (e.g., CW/SS, MH/BH, Ed, WFTD)
•Guiding principles, policies, and training
TAY Peer Mentor paired with TAY at time of d/c from FC
Peer-provided assistance with forms
Peer-provided emotional support
Coordinated service options
Coordinated service delivery
Inputs Activities OutcomesOutputs
TAY engagement in emotional and informational supports from Peer Mentor
TAY engagement in coordinated services and supports from community-based agencies
Increased utilization of services and supports by TAY exiting the FC system
Increased enrollment in educational and/or vocational opportunities Increased rates of
Independent Living
Increased rates of employment
Improved Quality of Life
27
Evaluating Innovation, Part2 012512 10
• Hold focus groups, conduct interviews, and/or develop questionnaires to get the perceptions
of those involved with the Innovation on the hypothesized relationships between program
Inputs, Activities, Outputs, and Outcomes
– TAY participants
– Peer mentors
– Representatives from collaborating agencies
– Other stakeholders
Examples of Measurement and Data
Collection for Process Goals
28
• Tracking system (e.g., Excel workbook)– Include dates of critical activities
• e.g., recruiting representatives from partner agencies, hiring peer mentors, developing policies & procedures, conducting trainings, introduction of TAY and peer mentor, enrollment in services
– Include numbers• e.g., # of interviews for peer mentors, # of
successful hires, # of trainings, # of contacts/encounters between TAY and mentor, # of contacts by type of service or support
29
How Might These Data be Collected? (1 of 3)
How Might These Data be Collected? (2 of 3)
• Interviews– TAY
• What was helpful? What was not helpful? Did services appear to be coordinated? What else could have been done/offered/provided?
– Peer mentors• Did the TAY participant accept/engage in all
contacts? What seemed to be most successful? What seemed to be least successful? What else could have been done/offered/provided?
30
Evaluating Innovation, Part2 012512 11
How Might These Data be Collected? (3 of 3)
• Interviews
– Representatives from public agencies
• What processes have changed within and across agencies as a result of this Innovation? What worked? What didn’t?
• Standardized Quality of Life questionnaire
31
Appropriate Points of Comparison
• Historical data: Foster youth discharged in two years prior to implementation of Innovation– Rates of service utilization, enrollment in
educational and/or vocational opportunities, employment, independent living
• Current data: Foster youth discharging currently who don’t have access to or don’t participate in the Innovation– Rates as above, plus Quality of Life questionnaire
32
Employing Evaluation Strategies: Use of Descriptive Statistics
• Side-by-side comparisons of rates of desired outcomes (frequency bar charts)– Data from prior years and data since
Innovation implementation
– Data from Innovation group and current data from non-Innovation group
• Responses from participants regarding what worked and what didn’t (percentages)
33
Evaluating Innovation, Part2 012512 12
• Descriptive Statistic: Frequency Counts,
Bar Chart
34
4245
75
2 Years Prior to Innovation 1 Year Prior to Innovation 1 Year After Innovation
Gainful Employment of 20 hrs per Week or More
• Descriptive Statistic: Percentages, Pie
Chart
45%
35%
13%
7%
"How helpful was the peer mentor in assisting you with completing forms?"
Very helpful
Somewhat helpful
A little helpful
Not helpful
35
Utilizing Your Innovation Logic Model
• The Logic Model is not a static document
• Timelines should be established for review of the logic model, likely linked with the timing of measurement
– Document the progression of each review of the logic model - including evidence that supports proposed relationships as well as evidence that either does not support or suggests unexpected relationships
• Modify and update key components and proposed relationships in the Logic Model as data are collected and lessons are learned
36
Evaluating Innovation, Part2 012512 13
37
Questions and Discussion
38
Example 2: Evaluating a Culturally-Relevant Services Innovation
Project
Logic Model Example 2
• Let’s say we have an Innovation program
that aims to increase access and utilization of mental health services among
Native American youth...
39
Evaluating Innovation, Part2 012512 14
Articulating Program Elements
• Questions to ask:
– Who is being served?
• School-age Native American children and youth
– Is this Innovation intended to bring change to
anyone else / anywhere else?
• The Native American tribe/community
• Schools serving Native American children and youth
40
Articulating Program Elements• Questions to ask:
– What are this program’s activities?• Tribal leaders promote education, awareness, and
identification
• School staff promote education, awareness, and identification
• School-based screenings
– What are the key strategies or processes by which the activities will be achieved?
• What training materials will be used?
• How will gatekeepers be trained and supervised?
• Will there be ongoing technical assistance for tribal communities and schools?
• How will screenings be conducted and managed?
41
Articulating Program Elements
• Questions to ask:– What is the timeline?
• Is there a limited or proscribed amount of time that structured activities will include education and awareness information?
• When would one expect to see change as a result of this Innovation?
– Who delivers the activities?• How are champions at the tribal and school levels
selected?
• How are gatekeepers identified? Will they receive supervision, ongoing technical assistance?
• How will positive screenings and/or identified need for services be handled?
42
Evaluating Innovation, Part2 012512 15
Articulating Program Goals
• Possible outcome learning goals…– Does this increase knowledge and decrease
stigma about mental health issues?
– Does this increase identification of those in need of services?
– Does this increase access and utilization of available services and supports?
– Does this increase mental health and social role functioning?
43
Articulating Program Goals
• Possible process learning goals…– What types of information promoted increased
awareness and knowledge?• In what settings? Delivered by whom?
– What setting characteristics within the tribal community and schools contributed to greater success of the Innovation?
– What factors were most influential in youth pursuing (or not pursuing) referred services and supports?
44
Logic Model Example 2
•Tribal leaders as “Champions”
•Screening tools and educational materials adapted for cultural relevance
•Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”
Awareness and anti-stigma info woven in to structured tribal activities
School-based screenings
Inputs Activities OutcomesOutputs
Tribal-based identification of need and referrals to available services and supports
School-based identification of need and referrals to available services and supports
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved social role functioninge.g., improved academic performance
45
Awareness and anti-stigma info woven in to structured school-based activities
Evaluating Innovation, Part2 012512 16
Logic Model Example 2
– Inputs:
• Tribal leaders who will function as “champions”
• Screening tools and educational materials modified for cultural relevance
• Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”
46
Examples of Measurement for Inputs
•Tribal leaders as “Champions”
•Screening tools and educational materials adapted for cultural relevance
•Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”
Inputs
47
•How many tribal leaders will serve as
champions?•How are they recruited?•Do they have different responsibilities as champions?
•Who is involved with adapting materials
(screening tools, educational materials,
training curriculum) for cultural
relevance?•What is the process for determining or agreeing upon changes?•How are the materials tested?
•Who will be trained as gatekeepers?•How will their roles be tracked?
Logic Model Example 2
– Activities:
• Awareness and anti-stigma information woven into structured tribal activities
• School-based screenings
• Awareness and anti-stigma information
woven into structured school- based activities
48
Evaluating Innovation, Part2 012512 17
Awareness and anti-stigma info woven in to structured tribal activities
School-based screenings
Activities
49
Awareness and anti-stigma info woven in to structured school-based activities
Examples of Measurement for Activities
•How many tribal and school-
based events will include
awareness and anti-stigma
information?•What is their frequency?•Will different information be presented at different events?•Will the presenters vary?
•How many screenings are
conducted?•What is the frequency with which screenings will be conducted?
Logic Model Example 2
– Outputs:
• Tribal-based identification of need and referrals to available services and supports
• School-based identification of need and referrals to available services and supports
50
Outputs
Tribal-based identification of need and referrals to available services and supports
School-based identification of need and referrals to available services and supports
51
Examples of Measurement for Outputs•How many youth are identified
in need through members of
the tribal community?•How many youth are
referred?
•What referrals are provided?
•How many school-based
screenings identify need?•How many referrals result
from screenings?
•How many youth are identified
in need through school-based
gatekeepers?•How many youth are
referred?•What referrals are provided?
Evaluating Innovation, Part2 012512 18
Logic Model Example 2
– Outcomes:
• Increased access to available services and supports
• Increased engagement in available services and supports
• Increased utilization of available services and supports
• Improved mental health functioning
• Improved social role functioning– e.g., improved academic performance
52
Outcomes
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved social role functioninge.g., improved academic performance
53
Examples of Measurement for Outcomes
•How many youth access referred
services and supports?
•How many youth return after initial access?
•How many youth are retained in
services and supports until
identified goals are met?
•What is the average length of time
that youth utilize referred services and supports?
•What proportion of youth have
improved functioning after
utilization?•What is the average amount of improvement?
•In what areas/domains?
•Tribal leaders as “Champions”
•Screening tools and educational materials adapted for cultural relevance
•Culturally-relevant trainings for school- and community-based mental health professionals and “gatekeepers”
Awareness and anti-stigma info woven in to structured tribal activities
School-based screenings
Inputs Activities OutcomesOutputs
Tribal-based identification of need and referrals to available services and supports
School-based identification of need and referrals to available services and supports
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved social role functioninge.g., improved academic performance
54
Awareness and anti-stigma info woven in to structured school-based activities
Process Goals
Evaluating Innovation, Part2 012512 19
Appropriate Points of Comparison
• Historical data: Community and school-based data from prior years– Access and utilization of services
• Also compare to prevalence estimates
• Current data: Youth who access and utilize referred services as a function of Innovation– Academic performance, historical and current
– Mental health functioning, pre- and post-• i.e., using a standardized questionnaire
55
56
Questions and Discussion
57
Example 3: Evaluating a Primary Care and Behavioral Health
Integration Innovation Project
Evaluating Innovation, Part2 012512 20
Logic Model Example 3
• Let’s say we have an Innovation program
that aims to improve the physical and mental health of its community members
by implementing bi-directional integration
of physical and behavioral health services in a rural area...
58
Articulating Program Elements
• Questions to ask:– Who is being served?
• Primary care clients with mental and/or behavioral health issues
• Mental health clients with physical and/or behavioral health issues
– Is this Innovation intended to bring change to anyone else / anywhere else?
• Primary care clinics
• Mental health clinics
• Staff of varying disciplines within each set of clinics
59
Articulating Program Elements• Questions to ask:
– What are this program’s activities?• Trainings and ongoing technical assistance in
integration using an established curriculum– Implementation and sustainability activities
• Screening, assessment, interventions, referrals
– What are the key strategies or processes by which the activities will be achieved?
• How will clinics be recruited? Are there minimum characteristics or qualifications?
• How will trainings and T.A. be delivered and tracked?
• How will screenings, assessments, interventions, and referrals be coordinated and tracked?
60
Evaluating Innovation, Part2 012512 21
Articulating Program Elements• Questions to ask:
– What is the timeline?• How long is the training phase? How long is technical
assistance provided?
• When would one expect to see change as a result of this Innovation?
– Who delivers the activities?• Who are the expert trainers and advisors?
• What set of systems need to be in place in each clinic to provide oversight for the new activities?
– Who will be “in charge” of the Innovation in each clinic?
– What new hires are necessary?
61
Articulating Program Goals
• Possible outcome learning goals…– Does this increase identification of primary care
clients in need of mental health services and mental health clients in need of physical health services?
– Does this increase the capacity for clinics to provide comprehensive care?
– Does this increase access and utilization of available services and supports?
– Does this improve physical and mental health functioning of clients served?
62
Articulating Program Goals
• Possible process learning goals…– What aspects of the training, technical assistance,
implementation and sustainability activities are perceived as most helpful?
– What clinic characteristics are associated with increased likelihood of success?
– Are there differences across disciplines and/or clinics with regard to acceptance and implementation of integration?
– Are there client characteristics that are better served in one clinic vs. another? (e.g., primary care vs. mental health)
63
Evaluating Innovation, Part2 012512 22
Logic Model Example 3
•Primary care clinic(s) interested and willing to integrate mental health services
•Mental health clinic(s) interested and willing to integrate physical health services
•Cross-training and integration curriculum (with established history of success in rural areas)
Trainings and ongoing technical assistance to recruited clinics
Screenings and assessments
Inputs Activities OutcomesOutputs
Increased rates of identification of needed services and supports
Increased education and awareness of physical and behavioral health issues among patients and clients served
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved physical health functioning
64
Participation in implementation and sustainability activities with experts from established , successful bi-directional integration
Logic Model Example 3
– Inputs:
• Primary care clinic(s) interested and willing to integrate mental health services
• Mental health clinic(s) interested and willing to integrate physical health services
• Cross-training and integration curriculum with
an established history of success in rural areas
65
•Primary care clinic(s) interested and willing to integrate mental health services
•Mental health clinic(s) interested and willing to integrate physical health services
•Cross-training and integration curriculum (with established history of success in rural areas)
Inputs
66
Examples of Measurement for Inputs
•How many clinics will participate in the
Innovation?•How are they recruited?•What are the reasons sites choose to participate (or not)?•What is the staff make-up (e.g. by discipline) within each clinic?
•Are there any modifications or adaptations to the cross-training and integration curriculum?
•How are these decided?•How are they tested?
Evaluating Innovation, Part2 012512 23
Logic Model Example 3
– Activities:
• Trainings and ongoing technical assistance to
recruited clinics
• Screenings and assessments
• Participation in implementation and sustainability activities with experts from established, successful bi-directional
integration
67
Trainings and ongoing technical assistance to recruited clinics
Screenings and assessments
Activities
68
Participation in implementation and sustainability activities with experts from established , successful bi-directional integration
Examples of Measurement for Activities
•How many trainings, technical assistance contacts, and other implementation and sustainability activities are offered?
•What are the number of participants across disciplines and/or clinics for each?
•How many screenings are
conducted?•What is the frequency with which screenings are conducted?•What proportion of screenings lead to assessments?
Logic Model Example 3
– Outputs:
• Increased rates of identification of needed services and supports
• Increased education and awareness of physical and behavioral health issues among patients and clients served
69
Evaluating Innovation, Part2 012512 24
Outputs
Increased rates of identification of needed services and supports
Increased education and awareness of physical and behavioral health issues among patients and clients served
70
Examples of Measurement for Outputs
•How many assessments lead
to referrals?•How many referrals are pursued?
•How many clients report greater awareness of physical and mental health issues?
•How many clients report
increased understanding of how to manage / care for their
physical and/or mental health
issues?•How many clients report
better management of their
health?
Logic Model Example 3
– Outcomes:
• Increased access to available services and supports
• Increased engagement in available services and supports
• Increased utilization of available services and supports
• Improved mental health functioning
• Improved physical health functioning
71
Outcomes
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved physical health functioning
72
Examples of Measurement for Outcomes
•How many clients access referred
services and supports?
•How many clients return after initial access?
•How many clients are retained in
services and supports until
identified goals are met?
•What is the average length of time
that clients utilize referred services and supports?
•What proportion of clients
demonstrate improved health
status?•What is the average amount of improvement?
•In what areas/domains?
Evaluating Innovation, Part2 012512 25
•Primary care clinic(s) interested and willing to integrate mental health services
•Mental health clinic(s) interested and willing to integrate physical health services
•Cross-training and integration curriculum (with established history of success in rural areas)
Trainings and ongoing technical assistance to recruited clinics
Screenings and assessments
Inputs Activities OutcomesOutputs
Increased rates of identification of needed services and supports
Increased education and awareness of physical and behavioral health issues among patients and clients served
Increased utilization of available services and supports
Increased access to available services and supports
Improved mental heath functioning
Increased engagement
Improved physical health functioning
73
Participation in implementation and sustainability activities with experts from established , successful bi-directional integration
Process Goals
• Data from the established bi-directional integration in its original implementation– At the systems/clinic level
• Staff make-up (e.g., by discipline), numbers participating in training and T.A. events, systems changes to accommodate integration
– At the client or patient level• Numbers served by integration (screenings,
assessments, interventions, referrals), rates of improved self-management and/or improved physical and mental health functioning
74
Appropriate Points of Comparison (1 of 2)
Appropriate Points of Comparison (2 of 2)
• Patients and clients who participate in
integration activities
– Physical health functioning, historical and current
• e.g., using standardized measures of health status
– Mental health functioning, pre- and post-
• i.e., using a standardized questionnaire
75
Evaluating Innovation, Part2 012512 26
Summary
• A Logic Model should be used to guide
Innovation evaluation
– Clearly articulate program elements and program goals (process learning goals and outcome learning goals)
– Build a Logic Model by identifying the key components of inputs, activities, outputs, and outcomes, as well as the hypothesized
relationships among key components
76
Summary (cont’d)
– Utilize the Logic Model throughout the Innovation
• Collect data from multiple sources and examine the extent to which the proposed relationships in the Logic Model hold
– What works, what doesn’t
– What should be continued, what should be changed
• In evaluating outcomes of Innovation, ensure that appropriate points of comparison are employed
– Data in and of themselves don’t tell us much… Comparisons allow greater understanding and