Evaluation Context & Questions PRT 595 Dr. Bob Barcelona
May 25, 2015
Evaluation Context & QuestionsPRT 595
Dr. Bob Barcelona
PRT 595: Program Evaluation
04/12/23 2Footer
Evaluation Context
There is no “one size fits all” approach to evaluationGood evaluations must fit the circumstances
Perspectives of stakeholdersPurpose of evaluation effortsKey guiding questionsPractical yet rigorous social science methods
PRT 595: Program Evaluation
04/12/23 3Footer
Who Determines Purpose & Questions?Program stakeholders (e.g. evaluation sponsor, funders, program management)Expect different perspectives (or conflict!) from different stakeholdersHow do evaluators deal with this?
Incorporate conflicting perspectives within the evaluation designFocus on only one perspectiveFacilitate negotiation of perspectives – build shared understanding
Balance stakeholder interests – see the case study by Morris (2003) and the commentary by Leviton (2003)
PRT 595: Program Evaluation
04/12/23 4Footer
Involving Stakeholders
Identify who your stakeholders areEstablish a structure for stakeholder involvementInvolve them earlyInvolve them continuouslyInvolve them actively
Reineke (1991)
PRT 595: Program Evaluation
04/12/23 5Footer
Questions for Stakeholders
Why is an evaluation needed? What is the evaluation’s purpose? What are the program’s goals and objectives?What are the important questions for the evaluation to answer? Are questions tied to a program’s theory or logic model?
PRT 595: Program Evaluation
04/12/23 6Footer
What’s the Purpose?Formative Evaluation: providing information that helps to guide program improvement
Summative Evaluation: providing information that helps to determine whether program resources are spent efficiently and effectively, and whether programs produce intended benefits
Knowledge Generation: providing information to describe the nature and effects of an intervention as a contribution to knowledge
PRT 595: Program Evaluation
04/12/23 7Footer
Purpose Statement Example
The purpose of this evaluation is to understand the impact of housing relocation on the quality of life, well-being, and economic self sufficiency of former and current residents of a severely distressed public housing development in Greenville, SC.
PRT 595: Program Evaluation
04/12/23 8Footer
Good Evaluation Questions
Help to focus the evaluationFacilitate the design of data collection proceduresMust be reasonable and appropriate – they are tied to the goals and objectives of the programMust be answerable – specific, concrete, practical, and measurableSet criteria for performance (i.e. 75% of children… a majority of residents…)
PRT 595: Program Evaluation
04/12/23 9Footer
Seventy-five percent (75%) of regular attendees will improve levels of fitness relative to the previous year by participating in health enhancing physical and recreational activities each day75% or more of the regular attendees will achieve at least 60 minutes of moderate or vigorous physical activity each day
Selected evaluation criteria, GoalPOST: Goal Oriented Performance in Out-of-School-Time Program, Clemson University
PRT 595: Program Evaluation
04/12/23 10Footer
Types of Evaluation Questions
Needs Assessment
Assessment of Program Process
Assessment of Program Theory
Impact Assessment
Efficiency Assessment
PRT 595: Program Evaluation
04/12/23 11Footer
Answering QuestionsSound social science methodology!Some examples (HFRP, 2004):1. Surveys and Questionnaires2. Interviews and Focus Groups3. Observations4. Tests and Assessments5. Document Reviews and Analysis6. Secondary Sources and Data Reviews
Each has strengths and weaknessesAdvantages of using multiple methods
PRT 595: Program Evaluation
04/12/23 12Footer
In Sum…Good Evaluations….Have defined purposesAccurately identify and involve stakeholdersAsk clear, relevant, and answerable questionsAre focused on assessing program need, developing program theory, assessing program process, assessing program impact, or assessing program efficiency Use practical and rigorous social science methodology and data collection techniques to answer questionsAre used by stakeholders in some tangible way