Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: Wayne M. Harding. Ed.M., Ph.D. , Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA [email protected]Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA [email protected]A Model for Using Technical A Model for Using Technical Assistance to Improve the Assistance to Improve the Evaluation Capacity of Local Evaluation Capacity of Local Programs Programs
23
Embed
Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by:
A Model for Using Technical Assistance to Improve the Evaluation Capacity of Local Programs. Perspectives on Impact Evaluation Cairo, Egypt March 29 – April 2, 2009 Presented by: - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Perspectives on Impact Evaluation
Cairo, Egypt March 29 – April 2, 2009
Presented by:
Wayne M. Harding. Ed.M., Ph.D. , Director of Projects, Social Science Research & Evaluation, Inc., Burlington, MA USA [email protected]
Cheryl Vince-Whitman, M.B.A., Ed.M., Senior Vice President, Education Development Center, Newton, MA USA [email protected]
A Model for Using TechnicalA Model for Using TechnicalAssistance to Improve the Evaluation Assistance to Improve the Evaluation
Capacity of Local ProgramsCapacity of Local Programs
2
Agenda
• Brief Overview of Service to Science (STS).
• Evaluation Findings About Service to Science
• Key Characteristics of The Technical Assistance Provided By Service To Science
Overview of Overview of Service to Service to ScienceScienceInitiativeInitiative
4
The Need for Service-to-Science
• Federal agencies and others in the USA have promoted the use of evidence-based programs to prevent substance abuse.
• The supply of “proven” programs is too limited to meet many local conditions.
• There are many other programs that may work, but lack evidence of their effectiveness
5
Service-to-Science Goals
• To help innovative prevention interventions that address substance abuse (or related issues) develop, improve, and document evidence of their effectiveness.
• To increase the pool of effective and appropriate interventions.
6
Organization of Service-to-Science
• Funded by Center for Substance Abuse Prevention (CSAP) in 2004.
• Implemented through 5 regional prevention centers (Centers for the Application of Prevention Technologies – CAPTs).
• In 2007, CSAP funded a related initiative for programs designed for Native Americans. Implemented by the Native American Center for Excellence (NACE).
7
Service-to-Science Activities
• CAPTs solicit prevention program nominations from State prevention officials.
• CAPTs assess the readiness of candidate programs.
• CAPTs conduct regional Service to Science Academies.
• CAPTs provide follow-up technical assistance.
8
Service-to-Science Activities (Continued)
• Service to Science Academy participants – eligible to compete for “subcontract awards” to further enhance evaluation capacity.
• CAPTs provide limited technical assistance post subcontract award.
9
Number of Programs Served Through Service to Science
• As of July 2009, the CAPTs had provided 294 prevention programs with technical assistance.
• Since 2006, 110 programs have received the subcontracts of up to $30,000.
Methods• Pilot study (2006) interviews and/or online survey
with program Directors. Target sample of 79 programs; 58 (73%) responded.
• Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants.
• 2007 online survey targeted 142 programs that had been participants for at least 15 months. 93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).
Participant Participant ProgramProgram
Characteristics Characteristics
13
Primary Behaviors Targeted for Change
Percent (a)
Alcohol Use/Abuse 80%
Illicit Drug Use/Abuse 60%
Tobacco Use 58%
Violence (including bullying) 37%
HIV/AIDS 8%
(a) Multiple responses allowed. N=93.
14
Ages of Population(s) Served
Percent (a)
Adolescents (ages 13-17) 71%
Childhood (ages 6-12) 51%
Young Adults (ages 18-25) 19%
Early Childhood (ages 0-5) 10%
Adults (ages 26-55) 10%
Older Adults (ages >55) 7%
No Distinction by Age 3%
(a) Multiple responses allowed. N=93.
15
Race/Ethnicity of Population(s) Served
Percent (a)
No Distinction 85%
Hispanic or Latino 9%
American Indian or Native Alaskan 5%
Black or African American 4%
White 4%
Asian American 3% Native Hawaiian or Other Pacific Islander 3%
(a) Multiple responses allowed. N=93.
16
Strategies Used
Percent (a) Percent
Improve Knowledge / Awareness
90% Mobilize Community 27%
Improve Skills 85% Improve Identification and Referral
25%
Healthy Alternatives 79% Improve Policies 17%
Improve Norms 59% Improve Laws 12%
Improve Mental / Physical Health
54%Improve Access to and Quality of Care
11%
Build Collaboration 36% Improve Enforcement 8%
(a) Multiple responses allowed. N=93.
Service To Service To ScienceScience
Outcomes Outcomes
18
Value of STS ServicesPercent (a)
Satisfied with CAPT TA(Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied)
72%
CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful)
91%
Would Recommend STS to Similar Programs (Definitely Yes or Probably Yes vs. Probably No or Definitely No)
91%
(a) N=93.
19
Factors That Facilitated Progress
When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs.
(a) From the 2006 pilot study. N=58.
20
TA Contributed to Program Capacity(a) in 5 Most Common Issues
Percent (b)
Improving understanding of program rating systems such as NREPP & NASADAD
95%
Improving understanding of evaluation 95%
Developing and/or improving a logic model 92%
Developing and/or implementing a more rigorous evaluation design
95%
Improving organizational commitment to evaluation
84%
(a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.” (b) N=93.
21
Seeking Recognition and/or Funding
Percent (a)
Plan to seek or are seeking recognition from NREPP or NASADAD
57%
Sought recognition from local, regional or other national organizations
13%
Prepared a journal article 13%
Presented evaluation findings at professional conference or meeting
13%
Used evaluation findings to help secure funding
23%
(a) N=93.
Defining Defining Features Of Features Of Service To Service To Science TAScience TA