Circe Stumbo, West Wind Education Policy Inc. and CCSSO
CCSSO/SCEE National Summit on
Educator Effectiveness
April 10, 2013
Implementation Science:
Closing Gaps Between Policy and Practice
Acknowledgements
Special thanks to Dean Fixsen, Karen Blase, and the National Implementation Research Network for their generosity in sharing their findings and their passion for all things implementation. This PowerPoint slide deck is built off their original work, with permission.
2
Sixth Year of Working with SEAs
State Implementation & Scaling-up of Evidence-based Practices Center
scalingup.org
3
Implementation Science`
A policy is one thing
Implementation of a policy is a very different thing
Students cannot benefit from a policy they do not experience
An Implementation Failure labeled as an Intervention Failure
Example: $500 million for Homebuilders (Family Support Services) 1993-1998
Funding only for interventions
No fidelity criteria insisted upon by the developers
National evaluation = not effective
Chapin Hall Center for Children, 2002
An Implementation Failure labeled as an Intervention Failure
Example: $500 million for Homebuilders (Family Support Services) 1993-1998
Funding only for interventions
No fidelity criteria insisted upon by the developers
National evaluation = not effective But, over 25% was spent on in-office
interventions with parents or children(< 0 fidelity)
Chapin Hall Center for Children, 2002
Longitudinal Studies of CSR Programs
Source: Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006
Evidence-Base For Effectiveness
Every Teacher Trained
Every Teacher Continually Supported
Actual SupportsYears 1-3
Fewer than 50% of the teachers received some training
Fewer than 25% of those teachers received support
Fewer than 10% of the schools used the CSR as intended
Vast majority of students did not benefit
OutcomesYears 4-5
Formula for Success
8
Socially Significant Outcomes
Effective Policies/
Interventions
Effective Implementation
Methods
Enabling Contexts
Formula for Success
9
WHY: Socially
Significant Outcomes
Effective Policies/
Interventions
Effective Implementation
Methods
Enabling Contexts
(Vision!)
Formula for Success
10
WHY: Socially
Significant Outcomes
WHAT: Effective Policies/
Interventions
Effective Implementation
Methods
Enabling Contexts
Formula for Success
11
WHY: Socially
Significant Outcomes
WHAT: Effective Policies/
Interventions
Effective Implementation
Methods
WHERE: Enabling Contexts
Formula for Success
12
WHY: Socially
Significant Outcomes
WHAT: Effective Policies/
Interventions
HOW & WHO: Effective
Implementation Methods
WHERE: Enabling Contexts
Table Discussions
Share a situation you have experienced/ observed where one of the variables in the formula was zero
Implementation Science
Excellent evidence for what does not work
Implementation without changing supporting roles and functions
Implementation by edict
Diffusion/dissemination of information by itself
Implementation by “following the money”
Training alone, no matter how well done
Paul Nutt (2002). Why Decisions Fail
What does work? An Implementation Framework
Implementation Teams
Implementation Drivers
Improvement Cycles
Implementation Stages
http://sisep.fpg.unc.edu/learning-zone/science-of-implementation/implementation-frameworks
An Implementation Framework
Implementation Teams
Implementation Drivers
Improvement Cycles
Implementation Stages
http://sisep.fpg.unc.edu/learning-zone/science-of-implementation/implementation-frameworks
IMPLEMENTATION
Implementation Science
Effective NOT Effective
Effective
NOTEffectiveIN
TE
RV
EN
TIO
N Student Benefits
(Institute of Medicine, 2000; 2001; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)
Poor outcomes
Inconsistent, not sustainable, poor outcomes
Poor outcomes; sometimes
harmful
IMPLEMENTATION
Implementation Teams
Effective
INT
ER
VE
NT
ION
80%, 3 Yrs 14%, 17 Yrs
Impl. Team NO Impl. Team
NOTEffective
Balas & Boren, 2000 Green, 2008
Fixsen, Blase, Timbers, & Wolf, 2001
Implementation Team
A group that knows:
the intervention/policy
implementation
improvement cycles
Simultaneous, Multi-Level Interventions
School
Management (leadership, policy)
Administration (HR, structure)
Supervision (nature, content)
Teacher
Region
District
Imp
lem
enta
tio
n T
eam
State
Partners
Iowa’s Collaborating for Kids (C4K)
22
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Cascading Logic Model
An Implementation Framework
Implementation Teams
Implementation Drivers
Improvement Cycles
Implementation Stages
http://sisep.fpg.unc.edu/learning-zone/science-of-implementation/implementation-frameworks
Improvement Cycles
Practice-Policy Communication Loops
Plan-Do-Study-Act cycles
Usability Testing
25
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Polic
y Ena
bled
Pract
ice
(PEP)
Pract
ice
Info
rmed
Polic
y (P
IP)
Practice/Policy Communication
Loops
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Polic
y Ena
bled
Pract
ice
(PEP)
Pract
ice
Info
rmed
Polic
y (P
IP)
Practice/Policy Communication
Loops
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Polic
y Ena
bled
Pract
ice
(PEP)
Pract
ice
Info
rmed
Polic
y (P
IP)
Practice/Policy Communication
Loops
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Feed
back
Loop
s
Feed
back
Loop
s
North Carolina Example
30
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Authentic Work Informs the
PEP-PIP Cycle
Plan-
Do-
Study
-Act
Plan-
Do-
Study
-Act
Students & Families
Students & Families
Building Teachers and Staff
Building Teachers and Staff
Building Leadership and Implementation
Team
Building Leadership and Implementation
Team
District Leadership and Implementation
Teams
District Leadership and Implementation
Teams
Regional Implementation
Teams
Regional Implementation
Teams
State Transformation
Team
State Transformation
Team
StateDepartment of
Education Leadership
StateDepartment of
Education Leadership
Authentic Work Informs the
PEP-PIP Cycle
Usabi
lity
Studi
es
Usabi
lity
Studi
es
Table Discussions
How do teams at each level of your system interact?
What work are the teams doing with each other?
What feedback loops exist between teams?
How might you increase interaction among teams?
Have/might you incorporate improvement cycles into pilots and policy innovations?
33
Growing Implementation Capacity Using
Transformation Zones
Implementation Team
Minimum of 3 people (4-5 preferred)
Tolerate turnover; sustainable
State Transformation Team
First School Implementation Team
N = 10 Schools
Growing Implementation Capacity
(Phase 1)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 9 Staff
Invest up front to build
capacity
State Transformation Team
First School Implementation Team
N = 10 Schools
Growing Implementation Capacity
(Phase 2)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 3 Staff
Subsequent Regional Implementation Team (3 Staff)
Subsequent Regional Implementation Team (3 Staff)
State Transformation Team
First School Implementation Team, Expanded
N = 20 Schools
Growing Implementation Capacity
(Phase 2)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 3 Staff
Subsequent Regional Implementation Team (3 Staff)
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent Regional Implementation Team (3 Staff)
State Transformation Team
First School Implementation Team, Expanded
N = 20 Schools
Growing Implementation Capacity
(Phase 2)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 3 Staff
Subsequent Regional Implementation Team (3 Staff)
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent Regional Implementation Team (3 Staff)
State Transformation Team
First School Implementation Team, Expanded
N = 20 Schools
Growing Implementation Capacity
(Phase 2)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 3 Staff
Subsequent Regional Implementation Team (3 Staff)
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent Regional Implementation Team (3 Staff)
State Transformation Team
First School Implementation Team, Expanded
N = 20 Schools
Growing Implementation Capacity
(Phase 2)
Ext
ern
al S
up
po
rt &
2 F
TE
s
First Regional Implementation Team
N = 3 Staff
Subsequent Regional Implementation Team (3 Staff)
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent School Implementation Teams
N=20 Schools
Subsequent Regional Implementation Team (3 Staff)
STAFF
Table Discussions
What responsibility will your state take to ensure implementation of your policies?
How can you grow implementation capacity within your state?
How can you rethink staffing in order to invest in the early stages of implementation?
42
Poll
One key take-away
OR
What questions do you have?
43
Possibilities to consider….
44
SCEE webinars
SCEE Topical Meetings
SCEE Discussion Groups
GIC
Individual state follow-up
Thank you
45