Evaluating Physical Activity Intervention Programs
Thomas Schmid, PhDPhysical Activity and Health Branch
CDCAtlanta, Georgia, USA
Evaluation Definitions and Principles
CDC Physical Activity Evaluation Handbook – 6 Steps
Evaluation issues
Overview
The systematic examination and assessment of features of a program and its effects. Evaluation is designed to produce information that can be used by those who have an interest in the improvement or effectiveness of an initiative or program.
What is Evaluation?
To assess the effects and value of a program◦for accountability◦for progress ◦to identify best practice◦as a planning tool◦for research
Why Evaluate?
the program target groups and needs realistic achievable objectives is the program ready for evaluation?
Match Evaluation of Health Promotion Programs to:
Identify needs
State goals and measurable objectives
Assess progress towards objectives
Process of conducting evaluation
EfficacyIntervention tested under ideal conditions with optimal study designEffectivenesField based intervention testing whether the intervention works as intended for a defined population in the real worldEfficiencyResults in comparison to the efforts required to obtain them – often cost or cost effectiveness
Evaluation Terminology
EVALUATION
IMPACT
OUTCOME
Measures activities, program quality, reach
Measures long term effects (goals)
PROCESS
Formative Measures short term objectives
Develop program & resources
What is likely to work Best practices in health promotion Idea or concept development Developing and pilot testing materials Focus groups
Formative Evaluation
Formative Evaluation Focus groups and
interviews with partners about their needs
Reviewing best practices from other programs such as Agita Sao Paulo
Pilot testing in one company
Process evaluation◦Program implemented as intended ?◦ Quality of program components◦Attendance rates ◦Audience awareness of message◦ Appropriate message learned◦Participant satisfaction
Process Evaluation Companies enrolled Workshops held Promoters trained Companies
participating in community events
Impact & Outcome Evaluation◦ Short and long term effects of program◦ Hierarchy of effects : what level of effects
indicates program success◦ Assessing the impact upon those who attend or
participate◦ Community effects
Impact & Outcome Evaluation
Physical activity level of employees
PA level in community
Disease rates or costs
Program Evaluation FrameworkPhysical Activity Evaluation Handbook
Program Evaluation FrameworkPhysical Activity Evaluation Handbook
1. Engage Stakeholders1. Engage Stakeholders
Include people with a vested interest in the program: implementers, partners, participants, possible opponents and decision makers.
2. Clearly Describe the ProgramThe problem being addressed, descriptions of the planning, implementation and maintenance
phases of the program, and finally, defining and communicating how all phases work together (eg. the "logic model").
Program Evaluation FrameworkPhysical Activity Evaluation Handbook
3. Clearly Focus3. Clearly Focus
Asking a clear, straightforward question and ensuring linkage to the problem being addressed. Includes clarifying process and/or outcome evaluations of interest
4. Develop and Implement Data CollectionDesign of data collection efforts, selection of
appropriate control groups, designation of appropriate baseline values and decisions about how much time should be built in to adequately measure "change”
Program Evaluation FrameworkPhysical Activity Evaluation Handbook
5. Develop Conclusions5. Develop Conclusions
Data analyses and interpretation.
6. CommunicatePromotes the importance of tangible products from the evaluation, dissemination of those products, and follow-up (as appropriate).
www.iuhpe.org
20
All physical activity interventions should be evaluated. Evaluation begins with program planning
There are many good programs, but very few are well evaluated
Involve stakeholders Community – University partnerships can be
very useful
Observations on Evaluation
Process as well as outcome evaluation Quantitative and qualitative Tailor evaluation to stakeholders and situation Evaluation is often undervalued compared with “hard science” Adequate resources should be devoted to evaluation. 10% is recommended
Observations on Evaluation
Evaluation results should be used to
adjust programs and inform policy Share results with stakeholders Academic and practical training are
needed The CDC Physical Activity Evaluation
Handbook provides a good framework
Observations on Evaluation
Gracias/Thank you