Intended Uses of Evaluation Findings for Intended Users OAE Team Meeting October 2011 M E N T A L H E A L T H C E N T E R O F D E N V E R www.mhcd.org
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Intended Uses of Evaluation Findings for Intended Users
OAE Team Meeting
October 2011
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
“If you don’t measure results, you can’t tell success from failure.”
o Stakeholders: Who has a “stake?”o Making the case for “utility”o Research & Decision Makingo Resistance is futile!o Topics for Discussion
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Who are our Stakeholders & Intended Users?
“We are certain that the evaluation will completely meet the informational needs of absolutely everyone.”
Different intended users may have different and multiple information needs
“…to ignore Stakeholders is like ignoring a smoldering dynamite fuse…”
Intended users are more likely to use evaluations if they understand and feel ownership of the evaluation process and findings. Do we practice this?
Stakeholders’ commitment to evaluation is fragile. Do you agree?
“I would really like to know _________ about Program X at MHCD.”
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
KNOWING: “What’s So?”
ACTION: “So What?”
• Design studies• Methods &
Statistics• Collect data• Publish
Using Information in the Information Age
• Decision making• Standards:
o Utilityo Feasibilityo Proprietyo Accuracy
“…we have reached a level where we know what we are doing and have a track record of important contributions…”
Eminently Practical:Evaluations should be judged by their utility and actual use.How real people in the real world apply evaluation findings and experience the evaluation process.
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Evaluation Use: Both Challenge & Mandate
“We did not get the information we needed when we wanted it, and it wasn’t what we wanted when we got it.”
This scene is repeated over-and-over.A critical public issue (affects MILLIONS
of People and BILLIONS of Dollars).How are programs that combat societal
ills (poverty, drugs, etc.) to be judged? “…reports gather dust on bookshelves…”
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
What do we have to do to get results we can really use?
A man walking along the street notices another man with bananas in his ears. He Shouts, “Hey, mister, why do you have bananas in your ears?”
Receiving no response, he pursues the man, calling again as he approaches, “Pardon me, why do you have bananas in your ears?” Again there is no response. He catches up to the man, puts his hand on his shoulder, and says, “Do you realize that you have bananas in your ears?”
The gentleman in question stops, looks puzzled, takes the bananas out of his ears and says, “I’m sorry, what did you ask? I couldn’t hear you because I have bananas in m years?”
Can STAKEHOLDERS be blamed for NOT reading the reports?
Knowledge use: Getting people to apply what is already known (seatbelts, smoking, contraception, helmets).
• e.g., Medical compliance: Why so many people don’t follow their doctor’s orders?
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
“Agency Evaluation ReportsDisregarded by Legislators…”
Implementation Evaluation “Programs, like airplanes, need all their
parts to do what they are designed to do and accomplish what they’re supposed to accomplish. Programs, like airplanes, are supposed to be properly equipped to carry out their assigned functions and guarantee passenger (participant safety). Programs, like airplanes, are not always so equipped. Regular, systematic evaluations of inventory and maintenance checks help avoid disasters in both airplanes and programs.”
Knowledge Use
New Directions in Accountability
“What gets measured gets done.”
Outcomes
Desired
Long-term Results
Consumers’ Lives
Affected
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Problematic Outcomes: What’s in it for the Consumer?
“Develop a supportive, family-centered, culturally competent, capacity-building intervention for families and children.”
OUTCOME 1
“To continue implementation of a case management system to maintain contact with clients before, during, and after treatments.”
OUTCOME 3
“Adults with serious and persistent mental illness will engage in a process to function effectively in the community.”
OUTCOME 2
“Improve ability of adults with developmental disabilities to begin making decisions and exercise choice.”
OUTCOME 4
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Utilization Outcomes
Target subgroup Teenage fathers at Mumford High
Desired outcome Appropriate parenting knowledge and practices
Outcome indicator Score on parent practice inventory (knowledge and behavior subscales)
Data collection Pre- and posttest; 6 month follow-up by District
Performance target 75% of participants will complete the program and attain a passing score on posttest subscales
USE Task force of varied stakeholders will review results and decide if program should continue and be expanded to other schools; Recommendations forwarded
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
o Our project is uniqueo The program ‘aint broken’ o Evaluation could trigger threatening consequenceso Evaluation is a hassle o The study is seen as impossibleo It’s not in my job to participateo What’s in it for me?o Not enough money to do a good job (1% 5% ?)o Project plan/SOW is vague o Staff or leadership has changedo Been there, done that, we don’t need to do it again
Resistance to Evaluation
Resistance is Futile!
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Situational Dimensions Affecting Stakeholder & Evaluator Engagement
● Utility ● Feasibility ● Propriety ● Accuracy
Distant CloseRelationship w/ primary intended Users
Directed ConsultantControl of the evaluation process
Audience All AspectsScope of intended User involvement
None ConstituenciesNumber of Stakeholders engaged
Deadlines DevelopmentalTimeline
M E N T A L H E A L T H C E N T E R O F D E N V E Rwww.mhcd.org
Possible Future Discussion Topicso Who are the Stakeholders at MHCD?o Research vs. Evaluationo Must an evaluation be useful?o Ambiguity in outputs and outcomeso Are logic models useful?o Overcoming resistance to evaluationo Are MHCD evaluations useful?
How do we know?
Related Documents
