Follow us @nccmt Suivez-nous @ccnmo Funded by the Public Health Agency of Canada | Affiliated with McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada. . Evaluability Assessments in Public Health Presenters: Laura C. Leviton, PhD Laura Kettel Khan, PhD November 10, 2017 1:00 – 2:30 PM ET
54
Embed
Spotlight Webinar: Evaluability assessments in public health
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
o Designing and using evaluation to improve intervention
■ Evaluability assessment: Assessing whether
o the intervention is ready to be managed for results
o what changes are needed to do so
o whether evaluation would contribute to improved performance
Definitions
Follow us @nccmt Suivez-nous @ccnmo
Resources
• Wholey, Hatry & Newcomer, Handbook of Practical Program
Evaluation, Wiley 2010
• Leviton et al., Evaluability assessment to improve public
health. In Annual Review of Public Health, Volume 31:213-
234.
• Leviton, Kettel Khan & Dawkins, New Directions in Evaluation,
No. 125, January 2010-- chapter 3 has templates, procedures
Follow us @nccmt Suivez-nous @ccnmo
Evaluations are Often Handed to Us
“Here, evaluate this.”
Follow us @nccmt Suivez-nous @ccnmo
Some of the Interventions Just Aren’t Very Good
“Here, evaluate this.”
Follow us @nccmt Suivez-nous @ccnmo
Why?
• Perhaps 95% of interventions are not fully developed at the
time of evaluation.
• Also, ongoing problems with measurement, design and
analysis.
• Wilson and Lipsey, 2001 review of 319 meta-analyses:
• Proportion of effect sizes associated with study features
• Variance accounted for:
Study methods = features of the intervention
Biggest sources: research design, operationalizing the dependent variable, sampling
error
Follow us @nccmt Suivez-nous @ccnmo
Don’t Rush to Summative Evaluation
Follow us @nccmt Suivez-nous @ccnmo
Steps in Evaluability Assessment
1. Involve intended users of evaluation information
2. Clarify the intended intervention
3. Explore intervention reality
4. Reach agreement on needed changes in activities or goals
5. Explore alternative evaluation designs
6. Agree on evaluation priorities and intended uses of information.
Follow us @nccmt Suivez-nous @ccnmo
It’s Cyclic,
Not Linear
Involve end users of
evaluation
Determine scope of project
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
no Create / revise logic model or
theory of change
Agree-ment on model?
yes
Feedback to program manager or policy maker
Interview staff; “scout” the
program reality
yes
Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,
availability)
Develop Program
Stop
Follow us @nccmt Suivez-nous @ccnmo
Early Steps
Involve end users of
evaluation
Determine scope of project
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
Feedback to program manager or policy maker
Stop
Follow us @nccmt Suivez-nous @ccnmo
Middle Steps
Review program
documents
Consult stakeholders
Agree-ment on goals?
no
no Create / revise logic model or
theory of change
Agree-ment on model?
yes
Feedback to program manager or policy maker
Develop Program
Stop
Follow us @nccmt Suivez-nous @ccnmo
Logic Models
Inputs Outputs Outcomes (Impact)
Resources
&
Staff
Activities,
Outreach to
Target Group
Products,
Target Group
Participation
Achieve
Short Term
Objectives
Achieve
Intermediate
Objectives
Achieve
Long Term
Objectives
Follow us @nccmt Suivez-nous @ccnmo
Theory of Change
Goals Strategies Target Group If-Then
Statements
Short Term
Outcomes
Long Term
Outcomes
Statements How goal will
be
accomplished
Define group If (activity)
then outcome
Measurement Was strategy
achieved?
Were
participants in
the target
group?
Did the
activity result
in the
outcome?
Was short
term objective
achieved?
Was long term
objective
achieved?
Data Sources E.g., program
records
E.g., survey of
group
Follow us @nccmt Suivez-nous @ccnmo
Later Steps
no Create / revise logic model or
theory of change
Agree-ment on model?
Interview staff; “scout” the
program reality
yes
Report on assessment of: Plausibility Areas for program development Evaluation feasibility Options for further evaluation Critique of data (quality,
availability)
Follow us @nccmt Suivez-nous @ccnmo
What Now?
• Intervention development
• Data collection to inform improvements
• When to do summative (outcome) studies:
• Logic model or TOC is sharpened and agreed to
• The model looks like the reality, and vice versa
• It’s plausible to achieve the outcome(s)
• Formative evaluation indicates intermediate steps are being
accomplished.
Follow us @nccmt Suivez-nous @ccnmo
Presenters
25
Dr. Laura Kettel Khan, PhD
Senior Scientist and Advisor
Centers for Disease Control and
Prevention
USA
Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
Systematic Screening and
Assessment: Past & Present
Examples from the Field
Follow us @nccmt Suivez-nous @ccnmo
Overview
•4 Examples from the Field
• Community Obesity Prevention – New York City Child
Care
• Healthy eating & Active living
• Hypertension Control
• Traumatic Brain Injury
•Coming full circle with SSA
• Childhood Obesity Declines
•Benefit for Public Health
27
Follow us @nccmt Suivez-nous @ccnmo
Obesity Prevention: NYC Child Care
Policies
Convene a panel of experts to identify and review
potential programs and policies
Assess programs and policies’ readiness for evaluation
Synthesize findings and share promising practices with the
field
Develop a network of professionals with the skills to
conduct evaluability assessments
Inform funders of programs and policies ready for
evaluation28
Follow us @nccmt Suivez-nous @ccnmo
Systematic Screening & Assessment
29
Inputs Steps Products
Guidance 1. CHOOSE priorities
2. SCAN environmental
interventions
3. REVIEW AND IDENTIFY
INTERVENTIONS that warrant
evaluability assessment
4. EVALUABILITY
ASSESSMENTS of priority
interventions
5. REVIEW & RATE for
promise/ readiness for eval
6. USE information
7. SYNTHESIZE what is known
Focus
Brief descriptions
Constructive feedback
Plan for rigorous
evaluation
List of
interventions
Nominations, existing
inventories, descriptions
Communicate with all
stakeholders
Expert review panel
Report of intervention and
evaluation issues
Ratings and Reports
List of Interventions
Brief DescriptionsNominations, existing
inventories, and
descriptions
Expert review panel
Network of
practitioners/research
Communicate with all
stakeholders
Expert review panel
Follow us @nccmt Suivez-nous @ccnmo
Systematic Screening
•Criteria expert panel used for screening:
– Potential impact
– Innovativeness
– Reach
– Acceptability to stakeholders
– Feasibility of implementation
– Feasibility of adoption
– Sustainability
– Generalizability/transportability
– Staff/organization capacity for evaluation
30
Follow us @nccmt Suivez-nous @ccnmo
Systematic Screening & Assessment
Results
Nominations
Met Inclusion
Criteria
Selected for
EA
After School/Day Care 167 61 23
Food Access 84 34 18
School District Local
Wellness Policies146 58 6
Comprehensive Physical
Activity39 7 2
Built Environment for
Physical Activity 22 14 4
TOTAL 458 174 53
31
Follow us @nccmt Suivez-nous @ccnmo
Evaluability Assessment
Review of documents• Draft logic model
2-3 day site visit• Interviews: program description, logic model,
Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the National Collaborating Centre for Methods and Tools:NCCMT website www.nccmt.ca