Quality In Action - May 2011
Post on 11-Nov-2014
708 Views
Preview:
DESCRIPTION
Transcript
Mentoring Partnership of Minnesota
Quality in Action
May 4, 2011
Mentor Program Evaluation
22
Webinar Logistics
Asking Questions & Sharing Comments During the Webinar
1. “Raise your hand” & MPM Organizers will unmute you
2. Or, type questions (and comments) in the question/answer section and submit; we will respond directly to you or possibly share your question with all attendees
When unmuted, please monitor your background noise
April Riordan, Director of Training and Community Partnerships
Recordings of past webinars are available on our YouTube Channel.
2
Overview Introduction & Best Practices
Elements of Effective Practice Evaluation Plan & Logic Model
Tips for Selecting Instruments Oregon Mentors Evaluation Toolkit
Important Rules for Administering Instruments
Additional Resources
3
What are Programs Measuring?
4
Mat
h Gra
des
Readin
g Gra
des
Scienc
e Gra
des
Social
Scie
nce
Grade
s
Overa
ll Gra
des
Schola
stic
Appea
l
Schola
stic
Compe
tenc
e
Educa
tiona
l Exp
ecta
tions
Absen
ces
Tardin
ess
Suspe
nsion
Quality
of W
ork
High S
choo
l Gra
duat
ion
0%
10%
20%
30%
40%
50%
60%
70%
80%
Percentage of Programs Measuring Academic Per-formance
What are Programs Measuring?
5
Misc
ondu
ct
Proac
tive
Behav
ior
Hyper
activ
ity
Youth
Sub
stan
ce U
se
Youth
Pre
gnan
cy/P
aren
ting
Youth
Gan
g In
volve
men
t
Positiv
e Rela
tions
hips
with P
aren
ts
0%
10%
20%
30%
40%
50%
60%
70%
80%
Percentage of Programs Measuring Behavior
What are Programs Measuring?
6
Emot
ional
Self-e
ffica
cy
Self C
ompe
tenc
y
Develo
pmen
tal A
sset
s
Emot
ional
Proble
ms
Peer
Relatio
nship
s with
Pee
rs
Mat
ch Q
uality
: Men
tors
help
cop
e with
pro
blem
s
Mat
ch Q
uality
: Unh
appin
ess
with m
ento
r
Mat
ch Q
uality
: Clos
enes
s to
men
tor
Relatio
nship
Qua
lity
Mat
ch Q
uality
: Clos
enes
s to
men
tee
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Percentage of Programs Measuring Psychological and Social Functioning
What are Programs Measuring?
7
Less than 5 5 through 10 10 through 15
15 through 20
20 through 25
Over 25 0%
5%
10%
15%
20%
25%
30%
Total Number of Indicators Measured (Out of 31)
Elements of Effective Practice
Ensure program quality and effectiveness:Develop a plan to measure program process:• Select indicators such as training hours,• meeting frequency and relationship duration; and• Develop a system for collecting and managing
specified data.Develop a plan to measure expected outcomes:• Specify expected outcomes;• Select appropriate instruments to measure outcomes• Select and implement an evaluation design.Create a process to reflect on and disseminate evaluation findings:• Refine the program design and operations based on
the findings; and• Develop and deliver reports to program constituents,
funders and the media
8
Mike GarringerAdvisor
National Mentoring Center Education Northwest
Celeste JanssenProgram Director
Oregon Mentors
Panelists
9
Measuring Effectiveness
How well you run
your program
How well your
volunteers mentor the
youth
Quality of match
relation-ships
Mentee’s attitudes
on certain topics
Youth behavior
(outcomes you wish
to achieve)
10
Tips for Evaluating Mentoring Programs
Let’s look at the mechanisms of mentoring…
11
Missed Opportunities in Mentoring Program Evaluation
1. Service deliveryDid they get the mentoring promised?
2. Participant satisfaction
3. Mentoring relationshipDid they form the bond needed to make any of this work?
4. Intermediary outcomes are OK!
12
What is a Logic Model?
Systematic picture of how you believe your program works
“…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”
13
Sample Vacation Logic Model
14
Sample YouthBuild Mentoring Logic Model
15
Tips for Selecting Good Instruments
• Try to use instruments that have been tested for reliability and validity
• Make sure they fit your logic model, especially in terms of proximal and distal outcomes
• Note the complexity of the tool: – Age range– Reading level– Concepts
• Can you provide a comparison group of some kind?
16
Quantitative Tools
Indicator: a variable with the purpose
to measure change in a phenomena or process
Valid: Does the tool measure
what it claims to measure? Reliable:
Does the tool give consistent results?
Pre and Post Test
17
Example
Pretest Posttest
Pretest Posttest
Live Demo - Oregon Mentors Evaluation Instrument Toolkit
Browse for specific instruments (19 “domains”)
Simple format for each instrument “record” Tips for using Links to other online repositories
18
Toolkit Domains of InterestLive Demo of Toolkit & Instruments
Personal Attitudes and Beliefs:Antisocial attitude scaleDecision making skills scaleGoal Setting Skills ScaleProsocial Involvement, Opportunities and Rewards
Mentoring relationship quality:Match Characteristics Questionnaire (MCQ)Youth Mentoring Survey (YMS)Youth Survey: Measuring the Quality of Mentor-Youth Relationships
19
Comprehensive Surveys:Youth Outcome SurveyYouth Risk Behavior Surveillance System
Connectedness:Hemingway Scale of Adolescent Connectedness
School and Academics:Education Expectations and Aspirations ScaleMotivated Strategies for Learning Questionnaire (MSLQ)
Tips for Administering Instruments
1. Plan for problems
2. Don’t hand-pick which questions to use
3. Don’t reword questions
4. Get parent permission
5. Provide a private space to administer
6. Don’t provide guidance
7. Administer the instruments in a consistent, uniform way
8. Keep the answers anonymous/ sort out any coding before administration
9. Plan for data storage and safety
20
Resources
22
Resources
W.K. Kellogg Foundation Evaluation Handbook
W.K. Kellogg Foundation Logic Model Development Guide
23
Stand by Me: The Risks and Rewards of Mentoring Today's Youth (The Family and Public Policy) by Jean E. Rhodes
Buying Information Reviews Endorsements
Handoutshttp://www.slideshare.net/traininginstitute
1. Using Standardized Instruments2. Comparing Program Outcomes to National
Standards (Appendix B – 36)3. Blank Logic Model Template4. Sample YouthBuild Mentoring Logic Model5. Using Your Logic Model to Focus the Evaluation6. Checklist for Selecting an Evaluator7. Sample Contact & Information Release Form8. Sample Evaluation Press Release9. Evaluating Mentoring Programs – P/PV Brief10. Additional Reading and Resources for Evaluating
Your Mentoring Program11. Glossary of Measurement and Evaluation Terms
24
Other Resources
This presentation & others www.slideshare.net/traininginstitute
MPM Training www.mpmn.org/traininginstitute
Web sites & PDFs www.delicious.com/traininginstitute
25
Next Quality in Action Webinar
June 1 – Eat Well, Stay Active, Have FunA new resource from the Harvard School of Public Health, "Eat Well, Stay Active, Have Fun: A Guide for Mentors," provides mentors with information and guidance on how to make healthy eating and physical activity natural parts of the mentor-mentee relationship. Join this webinar to learn more about the guide and how local mentors are using it.
26
Mentoring Program Evaluation & GoalsMonday, May 16 - 10:00-11:15am Pacific TimeGo to Friends for Youth's Mentoring Institute to
register online for this FREE webinar!
top related