What Can Different Types of Evaluation Add to Individual-Focused Behavior
Change Interventions?
Margaret Handley, PhD MPH
Center for Vulnerable PopulationsDepartment of Epidemiology & Biostatistics
and Department of Medicine
Homework?
FINAL PROJECT FOR EPI 246DUE June 3, 2010 5pm
For the final project for the course, please prepare a 3-5 page document and an abstract (250-350 words) that draws on the materials and lectures from the course. You may choose to develop one of the following in an area that you are currently involved with or anticipate being involved with in the future: 1.Text for a grant proposal for a behavior change intervention or to describe behavior, using theory-informed approaches. 2.Text for a program you would develop, its description, setting and rationale, again linked to materials in class related to theory, tools or programs. 3. The rationale and sample materials for a training you would develop that would lead to behavior change, and how these programs would link to theory. Please be creative but do link the work to the theories, frameworks and models that have been presented in class, and include at least 3-4 citations from the literature. Also include at least one figure/diagram that relates to the course.
Outline
1. Evaluation Approaches and IDS Relevance
2. DIME and RE-AIM Frameworks
3. Examples
Tai-Chi Intervention- RE-AIMFidelity vs Flexibility in Practice-Based ResearchCBPR and Logic Model for Cancer Screening
Program Evaluation Can Help To…
• Measure intervention’s effectiveness on targeted process or outcome measures.
• Determine most efficient and effective strategy for implementation of intervention
Verify the mechanisms through which you believe
your intervention is working
Guide/support replication in other settings
Measure fidelity and adaption
• Align goals with system or stakeholder goals
• Determine cost-effectiveness & priority
How To Conceptualize Evaluation
Outcome Evaluation
Process Evaluation
Resource Evaluation
Relevant Perspectives
• Clinic/ Organization
• Public Health
• Policy
How do we get what we really want?
“Program evaluation is the systematic collection of data related to a program’s activities and outcomes so that decisions can be made to improve efficiency, effectiveness or adequacy”
- CDC, Practical Evaluation of Public Health Programs
?
http://www.cdc.gov/eval/evalcbph.pdf
Goals of Evaluation
Intuitive Goal Evaluation Concept
Am I making a difference?
What have we done?
How well have we done it?
What is the value of it?
What had biggest impact?
What could we get rid of?
How effective have we been, and for whom?
What are we going to do now that we have this info?
Describe/Summarize
Quality; Importance; Accountability; Cost-effectiveness
Effectiveness; Social equity
Evidence-based planning;
Outcomes-focusing
A Range of Evaluation Needs in IDS Research• We know research implementation is highly
dependent on local context and involved inter-related interactions across multiple groups, but we focus on measure of individual behavior change in most evaluations….
• Which does not give one much to go on for successful replication or on what are key pitfalls
Framework: DIME and Translating research into Practice (TRIPLaB)
Hanburry et al, Implementation Sciences 2010
Designed for UK National Health Service Program on TRIP and implementation – Collaborations for applied research(University and NHS partnerships)
1. Selecting the innovation=Develop2. Implement in Local Settings3. Evaluate >>> then conduct large RCT etc.
Develop, Implement and Evaluate (DIME)or
Build it with the evaluation in mind
Hanburry et al, Implementation Sciences 2010
3 Phases
1. Selecting the innovation=Develop
1. Implement in Local Settings
1. Evaluate
Develop, Implement and Evaluate (DIME)
Develop•Stakeholder consult to id innovations
•Conjoint analysis survey
•Mapping against theory-based characteristics
•Explore team/social network culture
Synthesize/ranking
Implement •Review acceptability to stakeholders in their local context
•Review of ‘policy’ cost effectiveness of different strategies
•Delivered to relevant groups
Evaluate•Pre-Post test change in outcomes
•Interrupted time series
•Cost-effectiveness of implementation
Hanburry et al, Implementation Sciences 2010
Develop, Implement and Evaluate (DIME)
Hanburry et al, Implementation Sciences 2010
Developing/Selecting the Innovation Phase
•Stakeholder consult to id innovations to target (e.g qualitative interviews/focus groups >Maternal mental health prioritization w/in MCH)
•Conjoint analysis survey of stakeholders(e.g. trade- offs of mix of attributes for scenarios, such as likely cost/patient and local expertise to implement, data resources, preferences gets ranked by stakeholders)
DIME cont.
Hanburry et al, Implementation Sciences 2010
•Mapping against theory-based characteristics resulting in scoring of the different possibilities (e.g. strength of evidence for innovations eg self-efficacy)
•‘Diagnostic analysis’ w/semi-quantitative surveys to see if the innovation proposed looks good regarding the local social networks and teams/networks/communication channels
•Synthesize to choose the innovation having been prioritized as a priority area for stakeholders, conjoint survey ranks high, and maps to evidence and practical considerations.
DIME cont.
Implementation Phase
•Piloting different strategies
•Review of ‘policy’ cost effectiveness of different strategies – costs, practical factors for each option
•Detailing of components for fidelity and uptake (eg details of numbers and types of sessions)
•Deliver to relevant groups
DIME cont.
Evaluation Phase
•Interrupted time series (e.g. snap-shots to examine the impact on processes of care and outcomes)
•Pre-Post test change in outcomes (surveys ind behavior change measures linked to theory-based constructs, team characteristics, qualitative doer/non-doer analyses>>’black-box’ evaluation)
•Cost-effectiveness of implementation (micro costs and extent of behavior change achieved to arrive at implementation cost-effectiveness)
RE-AIM TO HELP PLAN, EVALUATE, AND REPORT STUDIES
R Increase Reach
E Increase Effectiveness
A Increase Adoption
I IncreaseImplementation
M Increase Maintenance
Glasgow, et al. Ann Behav Med 2004;27(1):3-12
PURPOSES OF RE-AIM
• To broaden the criteria used to evaluate programs to include external validity
• To evaluate issues relevant to program adoption, implementation, and sustainability
• To help close the gap between research studies and practice by
Informing design of interventions
Providing guides for adoptees
Suggesting standard reporting criteria
RE-AIM Evaluation Qs
Area Evaluation question to include
Reach
Efficacy or Effectiveness
Adoption
What percent of potentially eligible participants a) were excluded, b) took part and c) how representative?
What impact on a) all participants who began the program; b) on process intermediate, and primary outcomes; and c) on both positive and negative (unintended), outcomes including quality of life?
What percent of settings and intervention agents within these settings (e.g., schools/educators, medical offices/physicians) a) were excluded, b) participated and c) how representative were they?
RE-AIM Evaluation Qs
Area Evaluation question to include
Implementation
Maintenance
What percent of settings and intervention agents within these settings (e.g., schools/educators, medical offices/physicians) a) were excluded, b) participated and c) how representative were they?
Were intervention components delivered as intended?
What were the long-term effects b) What was the attrition rate; were drop-outs representative; were different intervention components continued? b) How was the original program modified?
Example: Tai Chi Intervention in Community-Based Falls Prevention Program
Area Program Evaluation Measure
Reach
Effectiveness
Adoption
Implementation
Maintenance
Reach :Those who qualified for program divided by those who responded to the promotion materials
Representativeness: Demos of those who were in program compared to those coming to center, using admin data.
Change in functional status measures, QOL measure =SF12
Proportion of centers approached who agreed to participate
Did the trainers follow key elements of the protocol, adherence to plan, frequency of sessions, inds. Doing program at home, attendance level sustained
Plan to continue/actual continuation post-trial
Example: P4H Evaluation
Area Program Evaluation Measure
Reach
Adoption
Implementation
How did accommodating patients circumstances change the reach?
How were essential intervention components maintained, and how were protocols changed during implementation?
How did accommodating personnel costs affect implementation processes?
How did research team working relationships impact uptake?
Example: CBPR Approach to Evaluating a Program to Decrease Cancer Disparities in Southern US
Problem: cancer disparities between Af. Americans- whites
Goal: Improve early cancer detection and preventive behaviors
Evaluation Methods: Use a logic model and CBPR process to develop, implement and evaluate interventions that “capture the spirit of change”while maintaining measureable outcomes
Outcomes at multiple levels:
Process Evaluation-inputs and planning strategies
Impact Evaluation-immediate effects assessed
Outcome Evaluation-med. And long term outcomes assessed
Example: CBPR Approach to Evaluating a Program to Decrease Cancer Disparities in Southern US