Even the President Is Talking about Evidence- based Programs!

Even the President Is Talking about Evidence-based Programs!

Why Use Evidence-based Programs?

• Mandate - Funding may require EBP

• Efficiency - More effective use of scarce resources

• Accountability - More confidence that what we do will make a difference

• Why reinvent the wheel?

• Take advantage of existing expertise (prevention science)

• Programs that aren’t evidence-based may actually be harmful

Summary of Benefits and Costs (2003 Dollars)Summary of Benefits and Costs (2003 Dollars)Dollars Per Youth (PV lifecycle) BenefitsBenefits CostsCosts B B -- C CEarly Childhood Education $17,202 $7,301 $9,901

Nurse Family Partnership $26,298 $9,118 $17,180

Functional Family Therapy $16,455 $2,140 $14,315

Life Skills Training $746 $29 $717

Seattle Soc. Dev. Project $14,246 $4,590 $9,837

Guiding Good Choices $7,605 $687 $6,918

Multi-D Treat. Foster Care $26,748 $2,459 $24,290

Intensive Juv. Supervision $0 $1,482 -$1,482

Selected Findings Selected Findings ((www.wa.gov/wsipp)

Aggression Repl. Trng. $9,564 $759 $8,805

S. Aos

Programs can be placed along a continuum of

“proof” of effectiveness

Not much

confidence

• Best Practices“We’ve done it and we like it”

• Promising Approaches

“We really think this will work…but we need time to prove it”

• Research-based“This program is based on sound theory informed by research”

• Evidence-based“This program has been rigorously evaluated and shown to work”

Very much confidence

Evidence-based ProgramsTheoretically sound interventions that

have been evaluated using a well-designed study (randomized controlled trial or quasi-experimental design) and have demonstrated significant improvements in the targeted outcome(s).

Evidence is strengthened by independent replication and sustained benefits.

Federal Working Group Standard for EB Certification*

Experimental Design/Randomized Control Trial

Effect sustained for at least 1 year post- intervention

At least 1 independent replication with RCTRCT’s adequately address threats to

internal validityNo known health-compromising side effects

*Adapted from Hierarchical Classification Framework for Program Effectiveness, Working Group for the Federal Collaboration on What Works, 2004.

EBP work, but ..EBP still represent the minority of prevention programs in

useResearch has shown that many (most?) aren’t being

implemented with sufficient quality or fidelityThere is tension between advocates of strict fidelity and

those who encourage local adaptationVery few programs measure or monitor implementation

fidelity and quality Policy work is needed to capture & redistribute dollars

saved through preventionPrograms are often “pull-out” and operate singularlySustainability remains a challenge – no permanent

infrastructure

Evidence-based programs are most effective when they are implemented with fidelity

Fidelity = the practitioners use all the core intervention components skillfully

IMPLEMENTATION IS IMPORTANT

Dissemination of information by itself does not lead to successful implementation.

Training alone, no matter how well done, does not lead to successful implementation. It must contain on-going oversight

Improving fidelity locallyWhat gets measured mattersImprove practitioner knowledge of

prevention scienceUse adaptation discussion as a tool for

training on the logic model of an intervention

Build a sustainable infrastructure for monitoring implementation fidelity and quality

Build internal capacity AND desire

Adaptation happens…Between 23% and 81% of program activities

may be omitted during implementation. (Durlak, 1998)

Only 19% of schools implement research-based curricula with fidelity. (Hallfors & Godette, 2002)

Only about 75% of the students received 60% or more of the Life Skills Training Program. (Botvin, et al., 1995)

Is adaptation Inevitable/necessary?Research shows that a high degree of

fidelity is attainable (Project TND, PROSPER, Blueprints)

There is little empirical support for cultural adaptation of EVPsMost have shown similar effects across

gender, ethnicity/race, SESStudies of prospective cultural adaptations

have failed to yield positive outcomes

Many Sources for EBPBlueprints for Violence Prevention *

www.colorado.edu/cspv/blueprints

SAMSHAwww.modelprograms.samsha.gov

OJJDP Model Programs Guidewww.dsgonline.com/mpg2.5/mpg_index.htm

The International Campbell Collaboration*http://www.campbellcollaboration.org/Fralibrary.html

Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and Emotional Learning Programs http://www.CASEL.org

What Works Clearinghouse (U.S. Department of EducationInstitute of Education Sciences)http://ies.ed.gov/ncee/wwc/index.asp

Top Tier Evidence Initiative www.toptierevidence.org

Center for Disease Control Effective Programshttp://www.cdc.gov?healthyYouth.partners/registries.htm

Child Trends What Works *(www.childtrends.org )

Guide to Community Preventive Services www.thecommunityguide.org

Promising Practices Networkwww.promisingpractices.net

Social Programs that Work www.evidencebasedprograms.org

Best Evidence Encyclopedia (K-12 Education) www.bestevidence.org

A Google search on “Evidence Based Programs” returned over 63 million hits

Professionals who will select the EBP, train or be trained, and/or implement the program

Considerations:Knowledge of programsAwarenessAttitudeMotivation to change or adopt something

newBehavior routines- Can the existing

structure be changed?

Barriers – What do you think?

MoneyResistance to

changeBuy-inRed tape TimeEgosOrganization’s

traditions

Insufficient professional development

ContractsUnions Cost or availability

of training for a program

Sustainability

Have you heard this one before?

I already have too much to do.I’ve heard it doesn’t work with all

audiences.I am not giving up what I believe works to

try something new that might not work.Most of the program is OK but there are

parts that I can make better.Not possible to get individuals/families to

participate in a program like this. It takes too much time, too many sessions.

Overcoming the Barriers Summary of Barriers Strategies for Success Difficult to find the right

programFear of changePeople are invested in the

status quoDevalues professional

expertiseOrganizational barriers:

personnel rules, staff turnover

Limits to the flexibility of the system

Cultural competency

Administrative supportTeacher/educator/

stakeholder supportSecure financial resourcesProvide high-quality training

to ensure program fidelityAlign intervention with

school/community goals, policies and programs

Make program outcomes visible

Develop strategy for staff turnover

Questions to Ask…How strong is the evidence of each program?Do we need it?What are the other options?Does the evidence apply to our population?Is it worth the investment, and can we afford

it?Can we assemble the necessary resources and

stakeholders?Will our community find it acceptable?How broad might the impact be?Can the developer support our site?Do we know others who have used it?What will it take to sustain it?

ResourcesAddressing Barriers to Learning. (2007). Evidence-based practices

in schools: Concerns about fit and implementation. UCLA Center, 12(3).

Dunifon, R., Duttweiler, M., Pillemer, K., Tobias, D., & Trochim, W.M.K. (2004). Evidence-based extension. Journal of Extension, 42(2).

Forman, S.F., Olin, S.S., Hoagwood, K.E., Crowe, M., & Saka, N. (2008). Evidence-based interventions in schools: Developers’ Views of implementation barriers and facilitators. School Mental Health, 1.

Henderson, M.L., Mathias-Humphrey, A., & McDermott, M.J. (2008). Barriers to effective program implementation: Rural school-based probation. Federal Probation, 72(1).

Identifying barriers to evidence-based uptake. (2006). National Institute of Clinical Studies, Melbourne VIC.

Research Development Associates. Evidence-based Practices. Power Point retrieved from http://www.resourcedevelopment.net/projects/workshops.html