Prepared by the Justice Research and Statistics Association IMPLEMENTING EVIDENCE-BASED PRACTICES.

Prepared by the Justice Research and Statistics Association

IMPLEMENTING EVIDENCE-BASED PRACTICES

2

THE IMPORTANCE OF EVIDENCE-BASED PROGRAMS

• The evidence-based movement is one of the most significant developments to occur in criminal and juvenile justice in recent years.

• Evidence-based programs (EBPs) are important because they can be expected to produce results and a sound return on taxpayer investment.

• Funding sources are increasingly promoting the use of EBPs.

Prepared by the Justice Research and Statistics Association

3

IMPLEMENTATION MATTERS

• Identifying and adopting an EBP is not enough to achieve successful outcomes.

• An evidence-based program still has to be delivered with fidelity/integrity in diverse and complex real-world settings.

Prepared by the Justice Research and Statistics Association

4

HOW ARE EBPS IDENTIFIED?

• EBPs are deemed to be effective based on rigorous scientific research.

• While specific criteria may vary, all organizations assess the quality and consistency of evidence when determining program effectiveness.

Prepared by the Justice Research and Statistics Association

5

THE IMPORTANCE OF THEPROGRAM MODEL

• Conclusions about a program’s effectiveness are derived from evaluations of a specific set of program components and activities, typically referred to as the program model.

• An EBP should be implemented in a manner that is consistent with its program model.

• Implementing an EBP in a way that deviates from the program model changes the program and invalidates expectations regarding the program’s effectiveness.

Prepared by the Justice Research and Statistics Association

6

IMPLEMENTATION FIDELITY

• Implementation fidelity refers to the degree to which a program’s real-world implementation matches the program model.

• Research has shown that programs implemented with a high degree of fidelity are more likely to be successful.

• Deviating from the program model not only can degrade program effectiveness, but also can result in an otherwise effective program doing more harm than good.

Prepared by the Justice Research and Statistics Association

7

ORDINARY CIRCUMSTANCE CAN BE AN OBSTACLE TO IMPLEMENTATION

• What appears to be simple and straightforward in the implementation process often turns out to be more complex than anticipated.

• We often underestimate the number of steps involved, the number of separate decisions that have to be made or the number of participants whose preferences have to be taken into account.

• Because of the complexity involved, implementation, even under the best circumstances, is exceedingly difficult.

Prepared by the Justice Research and Statistics Association

8

HIGH-QUALITY IMPLEMENTATION IS DIFFICULT TO ACHIEVE

• A 2011 Westat study found that only about 3.5% of all school-based programs to prevent youth substance abuse and school crime are research-based and well-implemented.

• A 2013 EPISCenter study found that nearly half of the Blueprint program implementations in PA involved adaptation, and the majority (53%) of these adaptations were predicted to negatively impact program effectiveness.

Prepared by the Justice Research and Statistics Association

9

IMPLEMENTATION SCIENCE

Research-based insights and tools that can be used

to support high-quality

implementation in complex

real-world settings

National Implementation Research Network (NIRN)

Prepared by the Justice Research and Statistics Association

10

IMPLEMENTATION SCIENCE: LESSONS LEARNED

• Implementation is a process, not an event.• Implementation occurs in stages:

– Exploration: acquisition of information– Installation: preparation, behind-the-scenes tasks– Initial implementation: small scale, trial change in practice– Full implementation: the innovation becomes accepted

practice; full staffing and client loads

• Entire process takes 2-4 years.

Prepared by the Justice Research and Statistics Association

11

TRADITIONAL STRATEGIES TO PROMOTE EBPS ARE INSUFFICIENT TO ACHIEVE

SOUND IMPLEMENTATION

• Diffusion and dissemination of information

• Passing laws or using other mandates

• Providing funding, or using funding incentives

• Training

• Organizational developmentPrepared by the Justice Research and Statistics Association

12

FACTORS THAT AFFECTIMPLEMENTATION FIDELITY

• Complexity of the program

• Specificity of information in the program model

• Facilitation strategies available to the implementing organization

• Responsiveness of clients and staff

Prepared by the Justice Research and Statistics Association

13

CORE PROGRAM COMPONENTS

• Core components are the parts of a program that are vital for program success and should not be eliminated or altered prior to or during implementation

• When core components are known, organizations should ensure they are implemented with the highest degree of fidelity possible.

Prepared by the Justice Research and Statistics Association

14

ADAPTATION

• The deliberate or accidental modification of a program – Deletions, additions or enhancements of program

components– Changes in the manner or intensity of administration of

program components– Cultural and other modifications required by local

circumstances

• Excessive program adaptation is a common problem.

Prepared by the Justice Research and Statistics Association

15

ADAPTATION

• Finding the right balance between fidelity and adaptation is difficult.

• An evidence-based program should be implemented with the highest degree of fidelity possible.

• Adaptation is likely to be advantageous only when it is highly strategic, pursued with caution, and monitored to prevent potentially harmful effects.

Prepared by the Justice Research and Statistics Association

16

SYSTEMS TRUMP PROGRAMS

• Systems are not very accommodating to innovation and change.

• Systems will exert pressures to alter a new initiative so it fits into existing structures with the least amount of change.

Prepared by the Justice Research and Statistics Association

17

IMPLEMENTATION DEPENDS ON:

• Core components of implementation that facilitate implementation fidelity

• Based on the commonalities found among successfully implemented programs

• Integrated and compensatory in nature

Prepared by the Justice Research and Statistics Association

18

IMPLEMENTATION DRIVERS

• Staff selection• Pre-service and in-service training• On-the-job coaching• Staff and program evaluation to identify problems

and guide mid-course corrections• Facilitative administrative support to address

technical and adaptive problems

Prepared by the Justice Research and Statistics Association

19

OVERCOMING RESISTANCE

• Implementation teams– Must have the knowledge, skill, freedom, and authority to

act within the larger organization or a collaboration of agencies

– Actively support and facilitate implementation on a daily basis

• Communities of practice– Mechanisms for practitioners to interact and exchange craft

knowledge

• Practice to policy feedback loops

Prepared by the Justice Research and Statistics Association

20

ASSESSING IMPLEMENTATION FIDELITY

• Program structure – the framework for service delivery – The presence of key program elements – Participation of requisite partner organizations– Resources allocated to the program– Staffing levels and characteristics

• Program processes - the way services are delivered– Adherence to program protocols or procedures– Frequency and intensity of services provided– The quality with which program staff

delivers services

Prepared by the Justice Research and Statistics Association

21

ASSESSING IMPLEMENTATION FIDELITY

• Context - prerequisites that must be in place for the program to operate– Staffing qualifications and numbers– Practitioner-consumer ratio (i.e., caseload size)– Completion of requisite training

• Compliance with the intervention components prescribed in the program model

• Competence or level of skill shown by the program staff

Prepared by the Justice Research and Statistics Association

22

ASSESSING IMPLEMENTATION FIDELITY

• Adherence to the program model - is the program being delivered as it was designed?

• Dosage or Exposure - do program participants receive the amount of the intervention prescribed in the program model?

• Quality of program delivery - do staff members deliver the program with skill?

• Participant responsiveness - are program participants engaged by program activities?

Prepared by the Justice Research and Statistics Association