Top Banner
CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jane Powers, Amanda Purington, Jenny Parise Jenny Parise ACT for Youth Center of Excellence ACT for Youth Center of Excellence
36

CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Dec 30, 2015

Download

Documents

Baldric Dean
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Page 1: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

CAPP EvaluationAugust 2011 Update

Jane Powers, Amanda Purington, Jenny Jane Powers, Amanda Purington, Jenny PariseParise

ACT for Youth Center of ExcellenceACT for Youth Center of Excellence

Page 2: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

OverviewOverview•Review Evaluation Plan: Documenting

EBP Implementation

•Ways to improve documentation

•Revisions to the Tools

•Next Steps

Page 3: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

CAPP Initiative CAPP Initiative

Goal 1: Promote healthy sexual behaviors and reduce the practice of risky sexual behaviors among adolescents

Core Strategy 1: Provide Provide comprehensive, age appropriate, comprehensive, age appropriate, evidence-based and medically evidence-based and medically accurate sexuality education to accurate sexuality education to promote healthy sexual behaviorspromote healthy sexual behaviors

Page 4: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

We know a lot about what works to We know a lot about what works to prevent teen pregnancyprevent teen pregnancy

EBP

•Increase age of first intercourse

•Increase use of condoms

•Decrease # sexual partners

•Decrease frequency of sex

DecreaseTeen Pregnancy

PromoteAdol SexualHealth

Page 5: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

What does it take to What does it take to implement EBPs in real implement EBPs in real

world settings???world settings???

Page 6: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Research to PracticeResearch to Practice While many EBPs have yielded positive While many EBPs have yielded positive

outcomes in research settings, the record outcomes in research settings, the record at the local level of “practice” is mixed at the local level of “practice” is mixed (Wandersman, 2009; Lesesne et al, 2008; (Wandersman, 2009; Lesesne et al, 2008; Fixsen, 2005). Fixsen, 2005).

Page 7: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

What do we know about What do we know about Implementation?Implementation?

• Implementation influences program outcomesImplementation influences program outcomes

• If EBPs are not implemented with fidelity and If EBPs are not implemented with fidelity and quality, not likely to result in outcomes observed quality, not likely to result in outcomes observed in researchin research

• Achieving good implementation increases Achieving good implementation increases chances of program success and stronger chances of program success and stronger benefits for participantsbenefits for participants

(Durlak and DuPre, 2008)(Durlak and DuPre, 2008)

Page 8: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Factors Affecting Factors Affecting ImplementationImplementation

• Community LevelCommunity Level

• Organizational Capacity Organizational Capacity

• Program CharacteristicsProgram Characteristics

• Training and TATraining and TA

• Facilitator CharacteristicsFacilitator Characteristics

Page 9: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

CAPPCAPPAn Opportunity to Learn!!!An Opportunity to Learn!!!

Page 10: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Opportunity to learn:Opportunity to learn:

• How do EBPs work in different settings?How do EBPs work in different settings?

• How do EBPs work with diverse youth How do EBPs work with diverse youth populations?populations?

• What adaptations are needed to make EBPs What adaptations are needed to make EBPs work more effectively?work more effectively?

• Are EBPs successful in getting to outcomes?Are EBPs successful in getting to outcomes?

Page 11: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Documenting Implementation is Documenting Implementation is critical!critical!

• Helps us learn how the intervention Helps us learn how the intervention was deliveredwas delivered

• Helps us understand challenges facedHelps us understand challenges faced

• Helps us learn about what does and Helps us learn about what does and does not work re implementing EBPsdoes not work re implementing EBPs

Page 12: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Tools to evaluate Tools to evaluate ImplementationImplementation

• Fidelity Checklist: document what you Fidelity Checklist: document what you did, adaptations made, did, adaptations made, successes/challengessuccesses/challenges

• Attendance Record: who, where, Attendance Record: who, where, when, how muchwhen, how much

Page 13: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Role of the Fidelity Checklists

Page 14: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Role of the Fidelity Role of the Fidelity ChecklistsChecklists

• Document:Document:– What you didWhat you did– TimingTiming– Adaptations made and whyAdaptations made and why– Challenges and successesChallenges and successes

Page 15: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Role of the Fidelity Role of the Fidelity ChecklistsChecklists

Help you Prepare!!Help you Prepare!!

Page 16: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Using Fidelity Checklist to PlanUsing Fidelity Checklist to Plan

• Review activitiesReview activities

• What equipment is needed?What equipment is needed?

• TimingTiming

• Anticipating issuesAnticipating issues

• Questions that might come upQuestions that might come up

• Prepare for next cyclePrepare for next cycle

Page 17: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

What have we learned?What have we learned?

Page 18: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Ways to Improve Ways to Improve DocumentationDocumentation

1. Provide More Details!1. Provide More Details!

– WHAT WHAT was changed?was changed?

– WHYWHY was it changed?was it changed?

Page 19: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

GREAT exampleGREAT example

“ “We took the entire class period to implement and We took the entire class period to implement and process this activity. Although it is timed at 15 process this activity. Although it is timed at 15 minutes, it took ten minutes alone to break the minutes, it took ten minutes alone to break the class of 32 students into groups, hand out cards class of 32 students into groups, hand out cards and explain the activity. We also needed more and explain the activity. We also needed more time to allow groups to respond as we physically time to allow groups to respond as we physically had more groups than if we were working in a had more groups than if we were working in a small group setting.”small group setting.”

Page 20: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Good exampleGood example““Due to the age of the participants and the Due to the age of the participants and the material in the Robert Townsend video clip, the material in the Robert Townsend video clip, the teacher did not find it to be appropriate for the teacher did not find it to be appropriate for the classroom. Instead we discussed the information classroom. Instead we discussed the information in the clip and used a short role-playing exercise in the clip and used a short role-playing exercise to make the information clear.”to make the information clear.”

– Why wasn’t it appropriate?Why wasn’t it appropriate?

– What was the role-playing exercise?What was the role-playing exercise?

Page 21: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Need more detailsNeed more details

““ran out of time”ran out of time”•Tells us WHY but not WHAT was changedTells us WHY but not WHAT was changed

•Was the activity shortened (if so, how?) or Was the activity shortened (if so, how?) or skipped altogether? skipped altogether?

““switched the order of these activities”switched the order of these activities”•Tells us WHAT the change was but not WHY Tells us WHAT the change was but not WHY

it was madeit was made

Page 22: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

2. Document ALL 2. Document ALL adaptationsadaptations

Was Activity Carried Out According to Direction in the Facilitator’s Curriculum?

•Leave something outLeave something out

•Change orderChange order

•Add new contentAdd new content

•Shorten activityShorten activity

•Substitute activitySubstitute activity

•Modify activityModify activity

Page 23: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

2. Document ALL adaptations 2. Document ALL adaptations cont.cont.

– Include info on pre-approved adaptationsInclude info on pre-approved adaptations– Were NO adaptations at all made??Were NO adaptations at all made??– ExampleExample

• NO adaptations madeNO adaptations made

• General Comment: “General Comment: “It would have been nice It would have been nice to have more time for each activity/moduleto have more time for each activity/module””

– Were activities completed according to the Were activities completed according to the curriculum?curriculum?– Were some activities omitted or shortened?Were some activities omitted or shortened?

Page 24: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

3. Complete the Fidelity 3. Complete the Fidelity Checklist right away!Checklist right away!

– Capture this information when the Capture this information when the experience of the module and the experience of the module and the activities are fresh in your mindactivities are fresh in your mind

Page 25: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

4. Complete the right number of 4. Complete the right number of toolstools

One One set of tools per cycle per set of set of tools per cycle per set of participantsparticipants

Page 26: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Class 1

Class 3

Class 4

Class 5

Fidelity Checklist Attendance Record

Class 2 Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Fidelity Checklist Attendance Record

Page 27: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Teacher 1

Teacher 2

Teacher 3

Fidelity Checklist Attendance Record

Page 28: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Common AdaptationsCommon Adaptations

• Common adaptations include:Common adaptations include:– Updating videos, or using something more Updating videos, or using something more

relevant in place of videosrelevant in place of videos– Changes to accommodate timing issuesChanges to accommodate timing issues– Changing language/names to be more inclusiveChanging language/names to be more inclusive

• How can the COE support these common How can the COE support these common adaptations?adaptations?– Providing/linking to updates on medical Providing/linking to updates on medical

informationinformation– Identifying alternative videos (updated)Identifying alternative videos (updated)– Developing alternatives to the condom Developing alternatives to the condom

demonstrationdemonstration

Page 29: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Documentation DetailsDocumentation Details• Block names of individual participantsBlock names of individual participants• Do not use acronyms – spell these outDo not use acronyms – spell these out• Write legiblyWrite legibly• Review your documentation for typos Review your documentation for typos

beforehandbeforehand• Bundle all evaluation materials for each Bundle all evaluation materials for each

cycle (fidelity checklists and attendance cycle (fidelity checklists and attendance records) and mail to:records) and mail to:

Amy BreeseAmy BreeseCornell UniversityCornell University

ACT for Youth Center of ExcellenceACT for Youth Center of ExcellenceBeebe HallBeebe Hall

Ithaca, NY 14853Ithaca, NY 14853

Page 30: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Submit tools electronicallySubmit tools electronically

• Save the fidelity check list and Save the fidelity check list and attendance record under a different attendance record under a different name and input the informationname and input the information

• Preserve the template for future usePreserve the template for future use

• Send the documents to Amanda Send the documents to Amanda Purington (Purington ([email protected]@cornell.edu))

Page 31: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Revisions to the Fidelity Revisions to the Fidelity ChecklistsChecklists

1.1. Clarified the adaptations description field:Clarified the adaptations description field:““If Changed, If Changed, WHATWHAT was changed and was changed and WHYWHY? Please ? Please

be specific: describe things you left out, added, be specific: describe things you left out, added, or changed and WHY.”or changed and WHY.”

2.2. Added “Were changes (if any) pre-approved?” Added “Were changes (if any) pre-approved?”

3.3. Added a comments section after each Added a comments section after each module:module:

““Please use this space if you have comments on this Please use this space if you have comments on this module or any of its activities:” module or any of its activities:”

Page 32: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.
Page 33: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.
Page 34: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Revisions to the Attendance Revisions to the Attendance RecordRecord1. Added “Age” column for participants

Page 35: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Next StepsNext Steps• Additional tools:Additional tools:

– Participant Feedback FormParticipant Feedback Form– Facilitator Feedback ToolsFacilitator Feedback Tools– Pre/Post TestsPre/Post Tests– Other needs?Other needs?

• Tools and presentations on Tools and presentations on www.actforyouth.net - Search for - Search for “CAPP” “CAPP” in the site search box at the top of the in the site search box at the top of the home page.home page.

• Revise Tools for Year 2 Revise Tools for Year 2

Page 36: CAPP Evaluation August 2011 Update Jane Powers, Amanda Purington, Jenny Parise ACT for Youth Center of Excellence.

Contact InformationContact Information

Amanda Purington: Amanda Purington: [email protected] 607-255-1861 607-255-1861

oror

Jane Powers: Jane Powers: [email protected]

607-255-3993607-255-3993