Top Banner
Evaluating Physical Activity Intervention Programs Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA
24

Evaluating Physical Activity Intervention Programs

Feb 16, 2016

Download

Documents

katen

Evaluating Physical Activity Intervention Programs. Thomas Schmid, PhD Physical Activity and Health Branch CDC Atlanta, Georgia, USA. Overview. Evaluation Definitions and Principles CDC Physical Activity Evaluation Handbook – 6 Steps Evaluation issues . What is Evaluation?. - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript

PowerPoint Presentation

Evaluating Physical Activity Intervention ProgramsThomas Schmid, PhDPhysical Activity and Health BranchCDCAtlanta, Georgia, USAEvaluation Definitions and PrinciplesCDC Physical Activity Evaluation Handbook 6 StepsEvaluation issues Overview

The systematic examination and assessment of features of a program and its effects. Evaluation is designed to produce information that can be used by those who have an interest in the improvement or effectiveness of an initiative or program.What is Evaluation?

To assess the effects and value of a program

for accountabilityfor progress to identify best practiceas a planning toolfor researchWhy Evaluate?

the program target groups and needs realistic achievable objectives is the program ready for evaluation?Match Evaluation of Health Promotion Programs to:Identify needs

State goals and measurable objectives

Assess progress towards objectives Process of conducting evaluation

EfficacyIntervention tested under ideal conditions with optimal study designEffectivenesField based intervention testing whether the intervention works as intended for a defined population in the real worldEfficiencyResults in comparison to the efforts required to obtain them often cost or cost effectivenessEvaluation Terminology

EVALUATIONIMPACTOUTCOMEMeasures activities, program quality, reachMeasures long term effects (goals)PROCESSFormative Measures short term objectivesDevelop program & resourcesWhat is likely to workBest practices in health promotionIdea or concept developmentDeveloping and pilot testing materialsFocus groups

Formative Evaluation

9We conducted two sets of focus groups. One before the market opened to examine day, time, location, name of the market, and marketing materials and general interest in the market.

The second set were divided into users and non-users of the market to understand what we were doing right, what could be improved, and to gauge a qualitative reaction to the market as is.

We learned so much information from these groups. Our market had been enhanced because of them. Some of the changes we considered were not made because the groups did not support them.

Next slideFormative EvaluationFocus groups and interviews with partners about their needsReviewing best practices from other programs such as Agita Sao PauloPilot testing in one company

10We conducted two sets of focus groups. One before the market opened to examine day, time, location, name of the market, and marketing materials and general interest in the market.

The second set were divided into users and non-users of the market to understand what we were doing right, what could be improved, and to gauge a qualitative reaction to the market as is.

We learned so much information from these groups. Our market had been enhanced because of them. Some of the changes we considered were not made because the groups did not support them.

Next slideProcess evaluationProgram implemented as intended ? Quality of program componentsAttendance rates Audience awareness of message Appropriate message learnedParticipant satisfaction

Process Evaluation

Companies enrolled

Workshops held

Promoters trained

Companies participating in community events

12Our process evaluation involved a questionnaire given to shoppers at the market. We originally had a questionnaire for people who came to the market but did not purchase to find out why they werent buying. We stopped printing this questionnaire after we realized that no one was coming to the market just to look.

Most people said they would shop every week or every other week. Satisfaction with price, availability, service, location, and other issues ranged from 96-99%. We also asked when people were likely to come to the market, and what they wanted that wasnt available.

Our market is not a local farmer our vendor buys from the Georgia farmers market South of the city. He does not sell organically grown produce ( every now and then he has some). Many people in our Division asked about this but our evaluation showed little interest in organic, locally grown produce. Price was the most important factor in peoples decision to use the market. They told us that organically grown and locally grown would be nice if it didnt raise prices much. Our audience is the employees who dont have easy access to fresh produce which trumped the people who wanted something different but really werent our main audience.

Next slideImpact & Outcome Evaluation

Short and long term effects of program Hierarchy of effects : what level of effects indicates program success Assessing the impact upon those who attend or participateCommunity effectsImpact & Outcome EvaluationPhysical activity level of employees

PA level in community

Disease rates or costs

14We couldnt ask many outcome questions or track pre- and post-consumption of fruits and vegetables because of IRB issues. We did, however, ask employees a qualitative question about how likely the market would increase their consumption. Over 80% of respondents said it would increase their consumption. Our focus groups included stories of changes in snack consumption and children who couldnt wait for Wednesday because they knew theyd get fresh fruits and vegetables.

Finally, we collected sales data as an outcome of the market.

Next slide

Program Evaluation FrameworkPhysical Activity Evaluation Handbook Program Evaluation FrameworkPhysical Activity Evaluation Handbook 1.Engage StakeholdersEngage StakeholdersInclude people with a vested interest in the program: implementers, partners, participants, possible opponents and decision makers.

Clearly Describe the ProgramThe problem being addressed, descriptions of the planning, implementation and maintenance phases of the program, and finally, defining and communicating how all phases work together (eg. the "logic model").

Program Evaluation FrameworkPhysical Activity Evaluation Handbook 3.Clearly Focus3. Clearly FocusAsking a clear, straightforward question and ensuring linkage to the problem being addressed. Includes clarifying process and/or outcome evaluations of interest

4. Develop and Implement Data CollectionDesign of data collection efforts, selection of appropriate control groups, designation of appropriate baseline values and decisions about how much time should be built in to adequately measure "change

Program Evaluation FrameworkPhysical Activity Evaluation Handbook 5.Develop Conclusions5. Develop ConclusionsData analyses and interpretation.

6. CommunicatePromotes the importance of tangible products from the evaluation, dissemination of those products, and follow-up (as appropriate).

www.iuhpe.org

2020All physical activity interventions should be evaluated. Evaluation begins with program planningThere are many good programs, but very few are well evaluatedInvolve stakeholders Community University partnerships can be very usefulObservations on EvaluationProcess as well as outcome evaluationQuantitative and qualitativeTailor evaluation to stakeholders and situationEvaluation is often undervalued compared with hard scienceAdequate resources should be devoted to evaluation. 10% is recommended

Observations on EvaluationEvaluation results should be used to adjust programs and inform policyShare results with stakeholdersAcademic and practical training are neededThe CDC Physical Activity Evaluation Handbook provides a good framework

Observations on EvaluationGracias/Thank you