Karen A. Blase Barbara Sims Michelle A. Duda Dean L. Fixsen Jonathan Green Ron Dughman, RRCP Jeanna Mullins, RRCP Presented by Jeanna Mullins, Mid-South.

Karen A. BlaseBarbara Sims

Michelle A. DudaDean L. Fixsen

Jonathan GreenRon Dughman, RRCP

Jeanna Mullins, RRCP

Presented by Jeanna Mullins, Mid-South RRC

Eugene, Oregon

October 31-November 1,

2013

Building Implementation Capacity

Western RRC Part B Leadership Forum

“A serious deficiency is the lack of expertise to implement best practices and innovations effectively and efficiently to improve student outcomes.”Rhim, Kowal, Hassel, & Hassel (2007)

Developing the Capacity to Implement Effectively

GOALS OF THE SESSION

• Increase knowledge of the ‘active implementation frameworks’

• Increase familiarity with tools and processes for using the frameworks

• Identify connections to the SSIP development and implementation

Complex environments

Unpredictable people

Competing demands

Shifting priorities

Various points of view

The Challenge

5 years of turnaround work

< 10% out of improvement status

> 90% still “in improvement”

—Stuit (2011; Are bad schools immortal?)

The Challenge

Recognizing Gaps

Science to Service Gap What is known to be effective is not what is selected to

help students

Implementation Gap What is selected is not used with fidelity and good

outcomes What is used with fidelity is not sustained for a useful

period of time What is used with fidelity is not used on a scale sufficient

to broadly impact student outcomes

The Challenge

“Implementation science is the systematic study of variables and conditions that lead to full and effective use of evidence-based programs and other effective innovations in typical human service settings.”

—Blase and Fixsen, 2010 National Implementation Research Network

Implementation Science

Business as Usual: Impact

Best data show these methods, when used alone Do not Result in Implementation as Intended Diffusion/ Dissemination of information Training Passing laws/ mandates/ regulations Providing funding/ incentives Organization change/ reorganization

5 to 10% return on investmentNECESSARY BUT NOT SUFFICIENT


HTTP://NIRN.FPG.UNC.EDU

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Implementation Research: A Synthesis of the Literature


It is not a “school problem”

District is the point of entry for sustainable school improvement Use short-term infusion of resources Establish long-term, district-based capacity for

quality

Plan for Change

“Making It Happen”

To successfully implement and sustain evidence-based and evidence-informed interventions, we need to know:

WHAT to doWhat is the intervention (e.g. effective instruction, effective assessment)?

HOW to do itActive and effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults)

WHO will do itOrganized, purposeful, & active implementation support from linked implementation teams

Plan for Change

Q. HOW?

A. Effective Implementation• Changing the behavior of educators and administrators

• Creating the setting conditions to facilitate these changes

• Creating the processes to maintain and improve these changes in both setting conditions and behavior of well-intentioned adults

• So that students benefit

Active Implementation Frameworks

Usable InterventionsImplementation Stages

Implementation DriversImprovement Cycles

Implementation Teams

• An intervention needs to be teachable, learnable, doable, and be readily assessed in practice.

Usable Interventions

UsableInterventions

USABLE INTERVENTIONS

Purposeful matching of critical implementation activities to the stage of the process

UsableInterventions

ClearDescription

Clear Description

• Philosophy, Values and Principles

• Inclusion and Exclusion Criteria



ClearDescription


Operational Definitions

ClearDescription

PerformanceAssessment

EssentialFunctions


EssentialFunctions

Essential Functions

• Clear description of the features that must be present to say that a program exists in a given location

• Core components


EssentialFunctions




• Describe each core component in terms that can be taught, learned, done in practice, and assessed in practice

• Practice Profiles





Performance Assessment

• Provides evidence that the program is being used as intended and is resulting in the desired outcomes

• Fidelity

• Practical enough to repeat time and time again





ClearDescription


EssentialFunctions

Usable InterventionsTools You Can UseHexagon Tool Practice Profiles

UsableInterventions

IMPLEMENTATION STAGES

Purposeful matching of critical implementation activities to the stage of the process

Stages AND Drivers

Implementation Takes

Time: 2 – 4 Years

EXPLORATIO

N

IN

STALL

ATION

INIT

IAL

IMPLE

MENTA

TION

FULL

IMPLE

MENTA

TIONDrivers

Drivers

Drivers

“DRIVERS”

Stages of Implementation

EXPLORATIONC

ompe

tenc

y D

river

s Organization D

rivers

Leadership Drivers

Integrated & Compensatory

—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

“Pay now or Pay later”

Exploration

Goals

• Create readiness for change• Changing hearts and minds

• Examine degree to which the proposed strategies and practices meet the needs of our State and our students

• Determine whether the strategies, practices, and implementation are desirable and feasible

Implementation Stages

Exploration Stage

What happens during Exploration Stage?

• Determine Need and Identify Options

• Assess “Fit” and Feasibility

• Structural and functional changes identified

• Promote “Buy in” for the innovation and for implementation supports

• Make recommendations (go/no go)

• Implementation “Team” identified


The HexagonAn EBP Exploration Tool

NEED

FIT

RESOURCES

EVIDENCE

CAPACITY

READINESS

Fit with current Initiatives• School, district , state priorities• Organizational structures

Community values

Need in school, district, state• Academic & socially significant Issues• Parent & community perceptions of need• Data indicating need

Resources and supports for:• Curricula & Classroom• Technology supports (IT dept.)• Staffing• Training• Data Systems• Coaching & Supervision• Administration & system

Evidence• Outcomes – Is it worth it?• Fidelity data• Cost – effectiveness data• Number of studies• Population similarities• Diverse cultural groups• Efficacy or Effectiveness

Capacity to Implement• Staff meet minimum qualifications• Able to sustain Imp Drivers

• Financially • Structurally

• Buy-in process operationalized• Practitioners • Families

Readiness for Replication• Qualified purveyor• Expert or TA available• Mature sites to observe• Several replications• How well is it operationalized?• Are Imp Drivers operationalized?

The “Hexagon” can be used as a planning tool to evaluate evidence-based programs and practices during the Exploration Stage of Implementation.

Download available at:www.scalingup.org/tools-and-resources

EBP:

5 Point Rating Scale:High = 5; Medium = 3; Low = 1.Midpoints can be used and scored as a 2 or 4.

High Med Low

Need

Fit

Resource Availability

Evidence

Readiness for Replication

Capacity to Implement

Total Score

© National Implementation Research Network 2009-2012 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University

of Maryland

Reflection Creating Readiness

• What role can you play in developing readiness for the EBP you have in mind?

• What are 2 things your team could do tomorrow to asses your current infrastructure?

The Hexagon Tool (Analysis of Evidence- based Programs or Practices)

SupportingNew Ways of Work


Com

pete

ncy

Dri

vers

Organization D

rivers

Leadership Drivers


INST

ALL

ATIO

N

—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

“If you build it, they will come”. . .

but you actually have to build it!

Installation

Goals

• Structural and functional changes are made to support implementation

• Staff selection protocols developed

• First ‘practitioners’ selected

• Define and initiate training of first cohort of practitioners

• Develop coaching system and plans

• Evaluate readiness and sustainability of data systems (e.g. fidelity, outcomes)


Installation

What’s Needed

• High-level protection, problem solving, and support

• Reduced expectations and higher costs during start up

• Help in evolving organizational supports at every level

• Help in establishing new implementation supports for the EBP



Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

EXPLORATION

INSTALLATIO

N

INITIAL

IMPLEMENTATION

“Get Started, then Get Better.”

Com

pete

ncy

Dri

vers

Organization D

riversLeadership Drivers


Initial Implementation

• Work through the Awkwardness• New skills are fragile and uncomfortable

• Implementation supports require new thinking/doing

• Organization/system change is scary

• Provide training and coaching on the evidence-based practice, re-organization of school roles, functions and structures

• Make use of improvement cycles to resolve systems issues


Initial Implementation

“Get started, then get better!”

• Learn from mistakes (detect and correct)

• Celebrate participation and progress

• Continue “buy-in” efforts

• Make organization and systems changes

• Manage expectations, “buy time in order to get better”

All the components of the program or innovation are at least partially in place and the implementation supports begin to function



2 - 4

Years

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

FULL

IMPLEMENTATION

“The only thing worse than failing and not

knowing why you failed, is succeeding and

not knowing why you succeeded.”

Full Implementation

• Maintaining and improving skills and activities throughout the system

• Components integrated, fully functioning

• Skillful practices by front line staff, supervisors, administrators (50% meet performance criteria)

• Changes in policy are reflected in practice at all levels

• Ready to be evaluated for expected outcomes


Full Implementation

“What Change? This is our way of work!”• Skillful Teaching and School Practices

• Skillful Use of the Drivers

• Drivers experience their own Improvement Cycles

• Data systems are in use, reliable and efficient, and are used for decision-making at multiple levels to regenerate and improve

• Practice to Policy Feedback Cycles are working and policies are reviewed regularly and changed to support improved practices and outcomes


ReflectionImplementation Stages

• What are you already doing that is “stage-based” relative to SSIP?

• What are the facilitators and barriers to doing stage-based work for the SSIP?

Stages of Implementation Analysis


Stages of Implementation Analysis

• Purpose Help Team plan for and/or assess the use of stage-based

activities to improve the success of implementation efforts for EBPs or evidence-informed innovations (action planning/anticipatory guidance)

The tool can be used to Self-Assess current stage related activities (e.g. “We are in the midst of Exploration”) or past efforts related to a stage (e.g. “We just completed most of Installation? How did we do? What did we miss?) (manage expectations)

StagesTools You Can Use

Stages of Implementation Analysis Template

Exploration Stage Guiding Questionsf

Hexagon Tool and Instructions

http://sisep.fpg.unc.edu/sites/sisep.fpg.unc.edu/files/resources/SISEP-GuidingImplementationQuestions-03-2012.pdf

IMPLEMENTATION DRIVERS

Common features of successful supports to help make full and effective use of a wide variety of innovations

Core Implementation

Components

© Fixsen & Blase, 2008

Positive Outcomes for Students

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Effective Educational Practices

How:

What:

Why:

Professional Development/Professional

Learning

Professional Development/Professional

Learning

Capacity to provide direction and vision

Staff capacity to support children/families with the selected practices

Institutional capacity to support teachers & staff in implementing practices with fidelity

Performance Assessment(Fidelity)

Coaching

Training

Selection Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers


Implementation Drivers


(Fidelity)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data SystemC

ompe

tenc

y D

river

s

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers


Improved educational outcomes

Interventions meet

Implementation

Consistent Use of

Educational Innovations



Purposes

“Are we doing what we said we would do?”Measure fidelityMotivate implementationReinforce staff and build on strengthsInterpretation of Outcome DataFeedback on functioning of

» Recruitment and Selection Practices» Training Programs (pre and in-service)» Supervision and Coaching Systems


Com

pete

ncy

Driv

ers

Performance Assessment Challenges

Drivers are ‘in service’ to a defined “it”

• No definition of “it”

• No definition of ‘fidelity’ to the “it”

What’s the way forward?

Performance Assessment processes are weak

• Content

• Context

• No Competency indicators


Blaming the teacher



Com

pete

ncy

Driv

ers


Coaching

Training

Selection Com

pete

ncy

Driv

ers



Selection

Purposes• Select for the “tough to teach traits”

• Screen for pre-requisites

• Set expectations for new hires – use of data, coaching

• Allow for mutual selection

• Improve likelihood of retention after “investment”

• Improve likelihood that training, coaching, and supervision will result in implementation


Com

pete

ncy

Driv

ers

Selection Driver Challenges

“We have who we have…this doesn’t apply to us!”



Com

pete

ncy

Driv

ers


Coaching

Training

Selection Com

pete

ncy

Driv

ers



Training Driver Challenges

“But our staff deserve professional development opportunities – we have to trust them to make use of the information.”


“Training events aren’t meeting our expectations for supporting implementation!” (e.g. no pre/post test, no practicing skills) “Now what?”



Com

pete

ncy

Driv

ers

Training

Purposes • Continue “Buy-in” process

• Knowledge acquisition

• Skill Development

• Form a ‘community’

• Get started…get better


Com

pete

ncy

Driv

ers


Coaching

Training

Selection Com

pete

ncy

Driv

ers



Training and CoachingImplementation Drivers

OUTCOMES% of Participants who Demonstrate Knowledge, Demonstrate

New Skills in a Training Setting, and Use new Skills in the Classroom

TRAINING

COMPONENTS

Knowledge SkillDemonstration

Use in theClassroom

Theory and Discussion 10% 5% 0%

..+Demonstration in Training 30% 20% 0%

…+ Practice & Feedback in Training 60% 60% 5%

…+ Coaching in Classroom

95% 95% 95%

—Joyce and Showers, 2002

Coaching

Purposes• Ensures fidelity

• Ensures implementation

• Develops application judgment in their setting

• Provides feedback to selection and training processes

• Grounded in “Best Practices”

• Must include direct observation and feedback


Com

pete

ncy

Driv

ers

Coaching

Common Challenges• Discomfort with ‘active skill development’ –

“Let’s reflect.”

• Multiple coaches for multiple initiatives.

• Lack of PDSA process – acts of random advice

• Process skills weighted more heavily than innovation knowledge and vice versa

• Resources – Who will do this? How will we fund it?

Pair up, Pick one (or create your own), Discuss ideas for addressing the challenge.


Com

pete

ncy

Driv

ers


Coaching

Training

Selection



Decision Support Data System

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers



Organization Drivers

• Change Organizations and Systems

• Create and sustain hospitable organizational and system environments for effective services

• Develop functional data systems that can be used to inform decision-making


Organization D

rivers

© Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008

All organizations [and systems] are designed, intentionally or unwittingly, to achieve precisely the results they get.“

—R. Spencer DarlingBusiness Expert

Organizational Change

System Change

Creating Capacity for Competent Change• New innovations do not fare well in current

organizational structures and systems

• Develop new position descriptions and job functions in State Departments of Education and in Regional and District systems.

“Systems trump programs.”—Patrick McCarthy, Annie E. Casey Foundation


Coaching

Training

Selection




Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers



Decision Support Data Systems

Purposes• Monitor and improve student outcomes through

data-based decisions

• Provide information to assess effectiveness of intervention and prevention practices

• Analyze the relationship of fidelity to outcomes

• Guide further program development Detect discrete issues as well as systemic issues

• Engage in continuous quality improvement Of the Intervention and the Drivers

• Celebrate success

• Be accountable to parents, Board of Education taxpayers, and other funders


Organization D

rivers


Coaching

Training

Selection




Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers




Purposes• Creates an internally hospitable environment for

the new way of work – at the level of the “agency” (e.g. school, District)

• Facilitates the installation, implementation, and improvement of the Drivers for each innovation

• Takes the lead on Systems Interventions

• Looks for ways to make the direct work of practitioners (e.g. teachers, school staff) and administrators more effective and less “burdensome”!!


Organization D

rivers


Coaching

Training

Selection




Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers




Purposes• Identify and “lift up” systemic barriers and

facilitators to the next level to improve support for the new way of work

• Create an externally “hospitable” environment for the new way of work

• Embed facilitators and strengths

• Contribute to cumulative learning in multi-site projects


Organization D

rivers


Coaching

Training

Selection




Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers

Organization D

rivers

Organization D

rivers


© Fixsen & Blase, 2008Leadership

Adaptive Technical

Leadership Drivers

Purpose

Identifying “wicked” problems and applying effective strategies to address those problems


Leadership Drivers

Wicked problems

Each attempted solution permanently alters the nature of the problem.

“The problem” is a moving target Attempted “solutions” often make the problem

worse, not better Legitimate But Competing Alternatives:

“Solutions” as defined by one group are seen as “calamitous failures” by other groups

Leadership Drivers

Different challenges call for different strategies

• Technical Strategies

• Adaptive Strategies According to Ron Heifetz and his colleagues at Harvard’s Kennedy School of Government, one of the biggest mistakes “leaders” make is to incorrectly identify the type of challenge they are facing

Using technical approaches for adaptive issues (and vice versa)


Leadership Drivers

Leadership Drivers

Technical Challenges

• Perspectives are aligned (views, values)

• Definition of the problem is clear

• Solution and implementation of the solution is relatively clear

• There is reasonable confidence that if the solution is implemented there will be resolution

• There can be a “primary” locus of responsibility for organizing the work


Leadership Drivers

Leadership Drivers

Adaptive Challenges

• Legitimate, yet competing, perspectives emerge

• Definition of the problem is unclear

• There are different perspectives on the “issue” at hand

• Solution and implementation are unclear and require learning

• Primary locus of responsibility is not a single entity or person


Leadership Drivers

Leadership Drivers

Strategies for Adaptive Work• Terms of Reference

• Nominal Group Process

• Criteria Referenced Problem-Solving What are the legitimate but competing agendas in

play? How do we define the challenge? What are the criteria for a good solution? Options? How do options address each of the criteria? Agreement to “next right steps” and next assessment

points


Leadership Drivers


Coaching

Training

Selection




Adaptive

Technical


Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers


Improved educational outcomes

Interventions meet

Implementation

Consistent Use of

Educational Innovations


Benefits of Driver-Based Action Planning

• Infrastructure needed becomes visible to all

• Strengths and progress get celebrated

• Next right steps are planned and results measured

• Resources can be aligned and re-purposed to improve implementation

Implementation Drivers Action Plans

DriversTools You Can Use-Drivers Best Practices -Locus of Responsibility for Drivers

Full Suite of Drivers’ Assessments

Drivers’ Best Practices – Overview Assessment

Drivers’ Strategic Analysis – Starter Discussion

Reflection Implementation Drivers

Do the Competency Drivers have applicability to SSIP work?o How?o Why not?o Under what conditions?

Do the Organization Drivers have applicability to SSIP work?o Why?o Why not?o Under what conditions

Does adaptive and technical leadership have application to SSIP work?o Why?o Why not?o Under what conditions?


Changing on Purpose

• New practices do not fare well in existing organizational structures and systems

• Effective innovations are changed to fit the system, as opposed to existing systems changing to support effective innovations.

• People, organizations, and systems. . .• Cannot change everything at once (too big;

too complex; too many of them and too few of us)

• Cannot stop and re-tool (have to create the new in the midst of the existing)

• Cannot know what to do at every step (we will know it when we get there)

• Many outcomes are not predictable (who knew!?)

Improvement Cycles

Types of Improvement Cycles

Plan-Do-Study-Act Cycles

• Rapid cycle problem solving (Shewhart; Deming)

• Transformation Zone

• Usability testing (Neilson; Rubin)

• Practice-policy communication loops

Improvement Cycles

Rapid Cycle Problem SolvingImprovement Cycles

Plan

DoStudy

Act

Usability TestingImprovement Cycles

Plan

DoStudy

Act Plan

DoStudy

Act Plan

DoStudy

Act

Usability vs. Pilot TestingImprovement Cycles

Usability

• Clear description of the program

• Trial and learning approach

• Small number of participants (N = 3 - 5)

• Multiple iterations to detect and correct problems as they arise

• Learn HOW to do the work effectively

Pilot

• Clear description of the program

• Trial and assessment approach

• Sufficient number of participants for statistical power (N= 20 – 50)

• Sufficient time to realize potential results

Transformation Zone

A “vertical slice” of the service system (from the classroom to the Capitol)

• The “slice” is small enough to be manageable

• The “slice” is large enough to include all aspects of the system

• The “slice” is large enough to “disturb the system” – a “ghost” system won’t work.

Improvement Cycles

Practice-Policy Communication Cycle

Policy

Practice

Po

licy En

ables P

ractices

Plan

DoEx

tern

al I

mp

lem

enta

tio

n S

up

po

rt Policy

Practice

Structure

Procedure

Pra

ctic

e In

form

s P

oli

cy

Fee

db

ack

Stu

dy - A

ct

FORM SUPPORTS FUNCTION

Implementation Team

StateManagement

Team

TeachersInnovations

Students

Sys

tem

C

han

ge

SIS

EP

Sy

ste

m C

han

ge

Su

pp

ort

Pra

ctic

e-

Po

licy

C

om

mu

nic

ati

on

Cy

cle

Po

licy

Su

pp

orts

Effe

ctive

Pra

ctice

System Alignment

Improvement CyclesTools You Can Use

Improvement Cycles – Analysis Worksheet

Transformation Zone Functions and Structure

Reflection Improvement Cycles and Communication Loops

• How can we make use of improvement cycles in developing and implementing our improvement activities?

• Linking Communication Protocols


IMPLEMENTATION TEAMS

Organized, expert assistance to develop and sustain an accountable and effective structure

Implementation Teams

• Provide accountable and effective structure to move intervention through stages of implementation

• Scope of the initiative determines the number of teams and the linked communication protocols needed

• Focus is on Ongoing “buy-in” and readiness Installing and sustaining the Implementation Drivers Fidelity & Outcomes Systems Alignment and Stage-based work Problem-solving and sustainability

“We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved.”

—Senge, 1990

Linked Team StructuresImplementation Teams

School-basedImplementation Team

School-basedImplementation Team

District-basedImplementation Team

District-basedImplementation Team

Regionally-basedImplementation Team

State-basedImplementation Team

State-basedImplementation Team

Reflection Implementation Teams

• In your experience, who supports the change process?

• How is the transition made from external expertise to building internal capacity?


Implementation Teams Tools You Can Use & Learn More

Terms of Reference and Linking Communication Protocols

SISEP Teams, Roles, Functions

Your Implementation Resource

Active Implementation Hub http://implementation.fpg.unc.edu/

http://sisep.fpg.unc.edu/

http://implementation.fpg.unc.edu/





Summary

“Making it Happen” for students and families: Purposeful selection of an effective and feasible “What”

Conceptualize a change process so that effective interventions for children and families can become embedded and sustained in socially complex settings “stage-matched activities” to guide the process “implementation drivers” to build the infrastructure

Improvement processes are critical the work is never done because the environment is in motion

Invest in the development of organized, “expert” implementation support

Reflection Implementation Action Planning

• What are the best next steps for your team relative to applying implementation frameworks to the work around SSIP?


Karen A. Blase Barbara Sims Michelle A. Duda Dean L. Fixsen Jonathan Green Ron Dughman, RRCP Jeanna Mullins, RRCP Presented by Jeanna Mullins, Mid-South.

Documents

effective implementation

effective innovations

effective use of evidence

effective instruction

effective assessment

crosssector evidence

improvement stuit

student outcomes