EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH: THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS Maria Scannapieco, Ph.D., Professor & Director Center.

EVIDENCE BASED PRACTICE ALONE IS NOT ENOUGH:  THE USE OF IMPLEMENTATION SCIENCE TO ENSURE SUCCESS

Maria Scannapieco, Ph.D., Professor & Director

Center for Child Welfare

Mountains & Plains Child Welfare Implementation Center

25 April 2011

Overview

Introduction to Implementation Centers

Basic understanding of Implementation Science

Reflections on lessons learned in systems change and implications for states

Introduction to the Implementation Centers

Implementation Centers and National Resource Centers4

Improve child welfare administration and practice Meet Federal performance standards and

implement effective programs Pursue sustainable and positive systems change Improve outcomes for children, youth & families

Funded by the Children’s Bureau to provide States and Territories, Tribes, and Tribal Consortia individualized training and technical assistance (T/TA) to:

Implementation Centers: Filling the Gap

5

States and Tribes are sometimes without the resources necessary to implement comprehensive strategic plans

National Resource Centers have limited resources to provide intensive, long-term TA

Implementation Centers provide in-depth and long-term consultation and peer networking opportunities to States and Tribes

Project Partner RolesBuilding Capacity to Implement Sustainable Systems

Change

Implementation Center Child Welfare AgencyConduct organizational and readiness assessment to identify potential barriers to implementation

Develop and implement strategies to address organizational barriers and improve readiness for implementation

Guide planning process Develop project plan

Provide and coordinate technical assistance resources to support organizational change and facilitate implementation

Provide and coordinate organizational/ human resources to manage change and support implementation

Provide technical assistance to promote best practices in organizational and child welfare systems change

Utilize best practices to align people, process, structure, measurement/rewards, and technology around the desired systems change

Facilitate implementation Implement plan

Develop methodology (e.g. benchmarks, tools) to monitor implementation progress

Collect project data; use data reports to monitor implementation and inform /refine implementation strategies

Evaluate project outcomes Use findings and lessons learned to inform future implementation efforts

Geographic Assignments

Who are the Implementation Centers? 8

Implementation Center Organization Northeast & Caribbean Implementation Center (NCIC)

University of Southern Maine, Muskie School of Public Service

Atlantic Coast Child Welfare Implementation Center (ACCWIC)

University of Maryland School of Social Work

Midwest Child Welfare Implementation Center (MCWIC)

University of Nebraska-Lincoln, Center on Children, Families & the Law

Mountains & Plains Child Welfare Implementation Center (MPCWIC)

University of Texas at Arlington, partnering with the University of Denver and The Native American Training Institute (NATI)

Western & Pacific Child Welfare Implementation Center (WPIC)

American Institutes for Research, partnering with National Indian Child Welfare Association, Center for the Study of Social Policy, National Technical Assistance Center for Children’s Mental Health at Georgetown University, and the Florida Mental Health Institute at the University of South Florida

Mountains and Plains Child Welfare Implementation Center

MPCWIC

Geographical Service Area

Region 6: Arkansas, Louisiana, New Mexico, Oklahoma, Texas

Region 8: Colorado, Montana, North Dakota, South Dakota, Utah, Wyoming

MPCWIC Projects The Skun-eyah (Garden) Project is a collaborative project to be

implemented by two tribes, Three Affiliated Tribes (Mandan, Hidatsa and Arikara Nations) and Turtle Mountain Band of Chippewa

The Osage Nation implementation project goal is three-fold: develop a business mapping model, develop and articulate a culturally based family centered practice model, and create a decision support data system to help facilitate the first two

Colorado Department of Human Services, Division of Child Welfare– Child Welfare Practice Reform. This three-year project will explore, define and implement a Colorado practice model, inclusive of specific practice strategies, methods and tools to improve outcomes for children and families. Denver, Colorado

Developing a New Mexico Child Welfare Practice Modelwill be focused on the development and implementation of a clearly articulated practice framework, inclusive of vision, mission, values and operating principles, to guide all of the change initiatives underway in the State

Key Elements of Systems Change

Change Management Model

Awareness of the need for change Desire to participate and support

the change Knowledge of how to change Ability to implement the required

skills/behaviors Reinforcement to sustain the

change

Competencies for Leading Change

Creativity and innovation External awareness Flexibility Strategic Thinking Vision

Understanding Implementation Science

Implementation is not a Hobby!

There is a body of research around effective implementation.

Child welfare is slow to apply the science.

Implementation Centers are utilizing methodology to support project work with states and tribes.

What do we know?

Implementation Science National Implementation Research

Network (NIRN) Phases of Change

John Kotter, Harvard Business School Managing Change

ADKAR Model (Awareness, Desire, Knowledge, Ability and Reinforcement)

Implementation Frameworks

Brief overview of the science of implementation

Practice, program and systems change through… Multi-dimensional, fully integrated

use of Implementation Stages Implementation Drivers Implementation Teams

Applying Frameworks

The value of frameworks is To promote the ability to generalize

beyond the immediate project or initiative

To enhance communication among partners (e.g. better understanding of one another)

To more easily share and apply improvements

To increase the relevance of the “lessons learned”

Building the Implementation Platform

Brief overview of the science of implementation

Multi-dimensional, fully integrated Implementation Stages Implementation Drivers Transformation Zones

Science to Service

SCIENCE SERVICEGAPIMPLEMENTATION

What Do We Mean by Implementation?

A specified set of activities designed to put into practice a policy, activity, or program of known dimensions

Implementation processes are purposeful and defined in sufficient detail such that independent observers can detect the presence and strength of these “specified activities”

Overall Challenge Science to Service Gap

What is known often is not what is adopted to help children, families, and caregiversImplementation GapThere are not clear pathways to implementation

What is adopted often is not used with fidelity and good effect

What is implemented often disappears with time and staff turnover

From the Synthesis of the Literature We Know That…

Implementation issues are common across widely diverse domains Human service prevention and

intervention (e.g. child welfare, substance abuse, mental health, violence prevention, education)

Advanced manufacturing technologies

Research-based clinical guidelines Engineering (e.g. bridge

maintenance) Hotel management National franchise operations Cancer prevention and treatment

Positive Intervention Outcomes

≠ ImplementationImplementation has not been achieved by doing more or better research on interventions or on curricula

The usability of a program or practice has nothing to do with the weight of the evidence regarding that program

– “Evidence” on effectiveness helps you select what to implement for whom

– “Evidence” on these outcomes does not help you implement the program or practice

Insufficient Methods

Implementation by laws/ compliance by itself does not work

Implementation by “following the money” by itself does not work

Implementation without changing supporting roles and functions does not work

Fixsen, Naoom, Blase, Friedman, Wallace, 2005

Insufficient Methods

Diffusion/dissemination of information

by itself does not lead to successful

implementation

Training alone, no matter how well

done, does not lead to successful

implementation Fixsen, Naoom, Blase, Friedman, Wallace,

2005

So what does work?

What Works

Effective NOT Effective

Effective

NOT Effective

IMPLEMENTATION – The HOW

INT

ER

VE

NT

ION

Th

e W

HA

T

Performance Implementation (High Fidelity)

Paper ImplementationProcedure Implementation (Low Fidelity)

Getting Here and Staying Here is the Challenge!

Pre-Requisite: Defining “IT”

From what current state to what future state? The “it” must be operationalized

whether it is: An evidence-based practice or program A best practice Initiative A broad systems change initiative

OperationalizePart of Speech:  verb Definition:  to

define a concept or variable so that it can be measured or expressed quantitativelyWebster's New Millennium™ Dictionary of English, Preview Edition (v 0.9.7)Copyright © 2003-2008 Lexico Publishing Group, LLC

Child Welfare Practice Classification System

Well-Supported, Efficacious Practice Evidenced Based Practice (EBP)

Promising and Acceptable Practice Best Practices

Innovative or Novel Practice Concerning Practice

Stages of Implementation

Exploration

Design/Installation

Initial Implementatio

n

Full Implementatio

n

Sustainability

Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

Stages of ImplementationCore Activities

Exploration

• Conduct organizational assessment

• Garner leadership and stakeholder support

• Review literature

• Solicit multiple perspectives on needs and opportunities

• Establish baseline performance

• Develop intervention model

Design/Installation

• Develop plans:• Implementation

• Communication

• Change management

• Define goals and benchmarks

• Engage stakeholders

• Conduct readiness assessment

• Align organization systems/supports

Initial Implementation

• Build staff competency

• Communicate project status

• Monitor fidelity of implementation

• Monitor implementation climate

• Identify and address barriers to implementation

• Provide performance feedback to staff, management, and stakeholders

Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

Stages/Activities

Full Implementation•Apply lessons

learned from initial implementation to full operation•Track fidelity through quality assurance and performance evaluation data

•Solicit feedback from multiple stakeholders and consumers•Evaluate impact on child and family outcomes

Innovation

• Adapt or adjust intervention model to reflect lessons learned from stakeholder feedback, tracking, evaluation

• Communicate changes and rationale

• Re-build competency around modified model

Sustainability

• Establish long term funding sources

• Align ongoing quality assurance and performance measurement with model

• Promote visibility of new practice and successful outcomes

Adapted from Implementation Research: A Synthesis of the Literature (2008) The National Implementation Research Network

Selection

Training

Coaching

Performance Assessment

(Fidelity)

Systems Intervention

LEADERSHIP

Decision Support Data

System

Integrated &

Compensatory

CO

MPE

TEN

CY

ORG

AN

IZATI

ON

Facilitative Administration

© Fixsen & Blase, 2008

National Implementation Research Network: Implementation Drivers

© Fixsen & Blase, 2008

Performance Assessment

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Adaptive

Technical

Integrated & CompensatoryC

ompet

ency

Drive

rs

Com

pet

ency

Drive

rs

Org

anization

Drivers

Org

anization

Drivers

LeadershipLeadership

Improved outcomes for children and families

Graphics by Steve Goodman,2009

Implementation Drivers

System Change Demonstrations or “pilots” are a

place to start for innovations (“it’s possible!”)

Don’t usually lead to sustainable service and system change Random acts of innovation Person and passion dependent Can “ghost” system its way to

success Executed by the “extraordinary” No replicable implementation

infrastructure

Transformation Zone

Focus on innovations And implementation

infrastructure development And organizational change And systems change processes

Transformation Zone

Use Innovations

Develop Implementation Infrastructure

Change System

Transformation Zone A “vertical slice” of the service

system (from the front-line to the Capitol) The “slice” is small enough to be

manageable The “slice” is large enough to

include all aspects of the system (court, case work, service system, urban, rural, frontier, diverse communities)

The “slice” is large enough to “disturb the system” – a “ghost” system won’t work.

Transformation Zone

Transformation Zones provide opportunities to… Manage the risks (most innovations

don’t work at first) and experience intended and unintended outcomes

Limit the damage (quick detection, recovery, repair)

Document “what works” Consider the implications of scaling-

up

Process for Implementing Change

TA Outcomes Evaluation Questions:

Research Question Indicator Process Measures

To what extent do implementation project stakeholders view MPCWIC technical assistance as having increased their State/Tribal capacity to meet project goals?

Perceptions of increased state/tribal implementation capacity

Implementation Capacity Assessment

Key Stakeholder Interviews

What components of NIRN are applicable in implementation of child welfare system change projects?

Stages/Drivers alignment with implementation success

Implementation Process Analysis

Global Attainment Scaling Driver Analysis

TA Outcomes Evaluation Questions:

Research Question Indicator Process Measures

To what extent do implementation project stakeholders view MPCWIC technical assistance as helpful in addressing state/tribal systems barriers?

Perceptions of increased state/tribal implementation capacity to address systems barriers

Key Stakeholder Interviews

To what extent do implementation project stakeholders view the NIRN framework as helpful in addressing system barriers?

Perceptions of Stages/Drivers installation related to implementation success

Key Stakeholder Interviews

Implementation Capacity Assessment

Implementation Process Analysis

MPCWIC Outcome Evaluation

Individual Implementation Project Evaluation Do State and Tribal systems improve with respect to the issue

addressed in their individual implementation projects? Do State and Tribal systems make significant advancements

toward the development of successful practice models? How do State and Tribal characteristics or organizational

factors contribute to the development of successful practice models?

What is the impact of the implementation project activities and integrations of the practice model on State or Tribal CSFR and PIP outcomes?

Individually, do State and Tribal partners accomplish the goals set forth in their multi-year strategic plans for sustainable system’s change?

Are State and Tribal partners successfully able to use practice model frameworks with fidelity to guide daily practice?

MPCWIC Outcome Evaluation

Cross- Implementation Project Outcomes As a group, do State and Tribal partners

accomplish goals set forth in their multi-year strategic plans for sustainable systems change?

Do organizations develop successful practice models?

What organizational factors are associated with successful practice models?

Do project type and organizational characteristics differentially affect outcomes?

What is the relationship between TA intensity and type and overall outcomes, particularily those related to culture, climate leadership, supervision and practice?

Summary Stage related activities prepare the

system for a successful change process Competence needs to be developed and

sustained Selection, training, coaching, fidelity measures help

change and support new practitioner behavior and skills

Organizations and systems need to change Data systems need to be used to make decisions Facilitative administrative practices & systems

interventions create hospitable environments Policy enables new practice but practice needs to

inform policy