Implementing What Works Daniel Perkins & Brian Bumbarger Pennsylvania State University.

Implementing What Works

Daniel Perkins & Brian Bumbarger

Pennsylvania State University

Importance of Research Standards

There could be no wiser investment in our country than a commitment to foster the prevention of mental disorders [or problem behaviors] and the promotion of mental health through rigorous research with the highest of methodological standards. Such a commitment would yield the potential for healthier lives for countless individuals and the general advancement of the nation's well-being.

Institute of Medicine- 1994

What are Evidence-based Programs?

The Gold Standard• Strong evidence of effectiveness

– Randomized controlled trials, well designed and implemented

– Trials showing effectiveness in two or more settings (including a setting similar to that of school/classroom implementing the program) (at least 300 students or 50-60 classrooms.)

Quality + Quantity = “Strong” Evidence

Why is a Randomized Clinical Trial (RCT) Convincing?

We know unequivocally if a program is effective

Not due to pre-test differences

Not due to other changes that might explain effects

Replication of effects using RCT greatly increases confidence that the program causes the changes

Examples: Nurse Family Partnership Program; PATHS

Life Skills Training; SFP 10-14; Early Head Start

What is Convincing?

The choice of research methodologies is a major issue in examining preventive interventions and research trials designed to determine their outcomes. It determines whether evidence is compelling. The ideal design is a randomized controlled trial.

Institute of Medicine- 1994

RCTs, Service, & Ethics Tension between rigorous science and providing

services.– Without evidence of positive impacts then we cannot be certain

that what we are doing is working, or worse, is not causing harm. – Not providing services to all feels like we are not being truthful to

community. Agencies who provide innovative programs

almost uniformly believe that their programs work

There is a need for a clear framework of accountability

The answer is compromise in terms of providing the best service and at the same time doing the most rigorous possible science.

Why is a Randomized Clinical Trial Not Sufficient by Itself?

There is a need for replication

There is a need to show effects across different populations Ethnicity, Urban/Rural, Levels of Education, Types of

Communities

There is a need for a carefully developed set of training procedures to ensure fidelity when disseminated

There is a need to learn how to flexibly adapt some aspects of the model to the “culture” of different communities

In the past…• 20 years ago, there were NO empirically-

validated prevention programs• Efforts were guided primarily by “good

intentions” and “gut instinct”• Hundreds of millions of dollars were spent

without any accountability• Prevention was considered more “art” than

“science”

Now…• Two decades of rigorous scientific research have

informed our knowledge of epidemiology, etiology, methodology, and prevention practice

• We have learned more about what causes and what works to preventing youth problem behaviors and promoting positive youth development in the last 20 years than we did in the previous 200 years

• We have tested theories of changes (public health model) that guide our program

• Today, there are many programs that have been proven effective in well-designed studies and have been independently replicated

• There is clearly a “science” of prevention!

Why Evidence-based Programs?

• Required use of “scientifically-based research” to decide which interventions to use and those that will be funded

• Accountability• To ensure the smart use of

LIMITED resources

When are Evidence-based Programs Needed?

• When you want to increase the likelihood that your program will have expected impacts (long-term).

• When there is support to implement an evidence-based program with rigor by the collaborators. (Evidence-based programs take a lot a time to implement if done right)

The Impact of Programs that Workmodelprograms.samhsa.gov

• Life Skills Training cut tobacco, alcohol, and marijuana use 50% - 75%

• Nurse Home Visitation reduced alcohol use by 56% in children 15 years after the intervention

• Project TND found a 26% reduction in regular hard drug use

• All-Stars reduced poly-drug use 40-60% at immediate post-test

• Project Alert reduced marijuana use initiation by 30% and regular marijuana use by 60%

Prevention is Cost-effectivewww.wa.gov/wsipp

(measured benefits and cost per youth)

Program Benefits Costs B-C

Nurse Home Visitation* $26,298 $9,118 $17,180

Guiding Good Choices* $7,605 $687 $6,918

Strengthening Families 10-14* $6,656 $851 $5,805

Project Northland* $1,575 $152 $1,423

LifeSkills Training* $746 $29 $717

Project TND* $279 $5 $274

All-Stars* $169 $49 $120

Functional Family Therapy* $16,455 $2,140 $14,315

Multisystemic Therapy* $14,996 $5,681 $9,316

DARE $0 $99 -$99

Intensive Supervision Probation $0 $1,482 -$1,482

Scared Straight -$11,002 $54 $-11,056

…Still more work to do

• Most prevention programs being utilized are not EBIs

• Research has shown that most are not being implemented with fidelity

• There is tension between rigorous science design and providing services to all

Next Steps With Research-Based Programs: Ensure Implementation

Quality When Communities Adopt Research-Based Programs

the Central Concerns Are: Maintaining High Fidelity Understand What Factors Influence Implementation

Quality

Leads to A New Generation of Research Questions– What factors influence the quality of implementation?– How does implementation quality effect outcome?

““Our work must Our work must emphasize deliberate emphasize deliberate investment in positive investment in positive

factors that research has factors that research has shown to be closely tied shown to be closely tied

to reduced levels of to reduced levels of negative behaviors as negative behaviors as

well as well as increased levels of increased levels of thriving [resiliency] thriving [resiliency]

attitudes and behaviors.”attitudes and behaviors.”(Blyth, 2000)

Sites Programs

1998 12 17

1999 8 9

2000 21 24

2001 19 21

2002 21 24

2005 14 24

2006 14 15

2007 15 15

Total 124 149

Replication of Effective Programs:PA’s Blueprints Initiative

Risk-focused Prevention PlanningRisk-focused Prevention Planning(the CTC model)(the CTC model)

Collect local data on risk and protective factors

Use data to identify priorities

Select and implement evidence-based program that targets those factors

Re-assess prevalence of risk and protective factors

Issues & Challenges

• Readiness/Program Selection

• Understanding of program and what is required

• Buy-in of key stakeholders

• Training availability/access

• Cost, timeliness and turnover

• Fidelity

• Ongoing TA

• Monitoring/measurement

• Measurement of program impact

• Sustainability

Why does fidelity matter?

• Research has clearly linked fidelity with positive outcomes

• Higher fidelity is associated with better outcomes across a wide range of programs and practices (PATHS, MST, FFT, TND, LST and others)

• Fidelity enables us to attribute outcomes to the intervention, and provides information about program feasibility

The reality….

• While possible, fidelity is not a naturally occurring phenomenon – adaptation (more accurately program drift) is the default

• Most adaptation is reactive rather than proactive

• Most adaptation weakens rather than strengthens the likelihood of positive outcomes

Adaptation happens…

• Between 23% and 81% of program activities may be omitted during implementation. (Durlak, 1998)

• Only 19% of schools implement research-based curricula with fidelity. (Hallfors & Godette, 2002)

• Only about 75% of the students received 60% or more of the Life Skills Training Program. (Botvin, et al., 1995)

Adaptation as a Function of TrainingAdaptation as a Function of Training(formal training by the developer)(formal training by the developer)

0

20

40

60

80

100

trained not trained

Per

cen

t

Fidelity Adaptation

Is adaptation inevitable/necessary?

• Research shows that a high degree of fidelity is attainable (Project TND, PROSPER, Blueprints)

• There is little empirical support for cultural adaptation of EVPs– Most have shown similar effects across gender,

ethnicity/race, SES– Studies of prospective cultural adaptations have

failed to yield positive outcomes

Improving fidelity locally

• What gets measured matters• Improve practitioner knowledge of

prevention science• Use adaptation discussion as a tool for

training on the logic model of an intervention

• Build a sustainable infrastructure for monitoring implementation fidelity and quality

• Build internal capacity AND desire

Building internal capacity and motivation

• Approach fidelity from a practical, accountability perspective – don’t make it a research issue

• The goal is to develop local intrinsic motivation for monitoring fidelity and quality of program delivery – it must be tied to outcomes

• Involve local practitioners/implementers in the development and conduct of evaluation

– Process evaluation is fidelity monitoring

Practical strategies• Peer coaching, peer observation• Schedule regular opportunities for reflective

practice and de-briefing• Never let the initial training be the only training• Data in must ALWAYS require data out – create

feedback loops and safe environments for reflection

• Foster internal competition• Emphasize the importance of a clear understanding

of a program’s logic model

Where to find evidence-based interventions

• The What Works Clearinghouse (http://www.,w-w-c.org/)

• SAMHSA National Registry of Effective Prevention Programs((http://www.modelprograms.samhsa.gov))

• The Promising Practices Network (http://www.promisingpractices.net/)• Blueprints for Violence Prevention (http://www.colorado.edu/cspv/blueprints/index.html)• The International Campbell Collaboration

(http://www.campbellcollaboration.org/Fralibrary.html)• Safe and Sound: An Educational Leader’s Guide to Evidence-Based Social and

Emotional Learning Programs (http://www.CASEL.org)• Social Programs that Work (

http://www.excel.gov.org/displayContent.asp?Keyword=prppcSocial)• Center for Disease Control Effective Programs

(http://www.cdc.gov?healthyYouth.partners/registries.htm)

When one has no stake in the way things

are, when one’s needs and opinions are

provided no forum, when one sees

oneself as the object of unilateral

actions, it takes no particular wisdom to

suggest that one would rather be

elsewhere.

-S. Sarason, 1990

Successful Community Engagement

• Use data about strengths and needs of community to inform your

selection strategies

• Agencies and staff buy-in is critical;

– Participation in decision-making & understanding of overall logic

model

• Support for the community member engagement on the

management teams

– Advisory board that engages the support of local Champions and

community leaders

– Opportunities for community members to complete meaningful tasks

• Ongoing communication among staff and

agencies– Learning Communities

– Recognize successes

• Social Marketing Strategy to obtain support of

citizens (timing)

Critical Elements of Youth on Management teams

• Adult support

• Youth-friendly environment

• Opportunities to complete meaningful

tasks

• Opportunities to learn and use new

skills.

We are Guilty of many errors and many faults, but our worst crime is abandoning the

children, neglecting the fountain of life. Many of the things we need can wait.

The child cannot. Right now is the time his bones are being formed, his blood is being

developed.To him we cannot answer ‘Tomorrow.’

His name is ‘Today.’Gabriela Mistral, Nobel Prize-winning Poet