Tony Bryk - Bristol - Joining Improvement Science to NICs

Anthony S. Bryk

University of Bristol May 21, 2014

2

Triple Aims of Educational Improvement

EFFICIENCY

EFFECTIVENESS

ENGAGEMENT

Be0er Use of Resources

Deeper Learning For All Students

More Relevant

Last Decade: Performance Management Using Accountability to Drive Change

–  Set targets

– Create incentives

– Collect data/dashboards

– Hold individuals accountable “Go figure it out or else…”

Last Decade: Evidence-based Practice Movement

An academic has an idea

He/she design and fine tunes an intervention

An RCT field trial (5 years later)

Evidence it can work

Reviewed by What Works Clearing House Goes on an “approved list”

Districts required or “incented” to buy only from approved list

Educators “Implement with Fidelity”

Practice Improves!

But there is a problem, actually many problems.

Last Decade: Communities of Practice

5

Great Dynamism, But PuLng All of This Together

6

Resembles a Tower of Babel

We have to find a be0er way to accelerate learning to improve.

I. Analogical Scavenger in hot pursuit!

9

We can accomplish more together, than

even the best of us can do alone.

Complex systems problems that we now seek to solve

A Better Way: Integrating Two Big Ideas

•  Improvement Science Disciplines Efforts

joined to

•  Structured Networks that Accelerate the Learning

A Shi& Toward Learning Fast to Implement Well.

II.

Two Networked Improvement Communities

! 1. Extraordinary high failure rates in developmental mathematics courses in community colleges—a gatekeeper to opportunity. – Can colleges dramatically increase the success of

these students to and through college-level math in one year of instruction?

! 2. The weak and incoherent supports by which individuals learn to teach in public school settings. –  Can districts learn how to support new teachers to

learn faster, better and hold on to them?

13

III. Forming As a Community: Building Education’s Capacity to Improve

The Six Core Principles

14

! 1. Make the work problem-specific and user-centered.

! 2. Variation in performance is the core problem to address.

! 3. See the system that produces the current outcomes.

! 4. We cannot improve at scale what we cannot measure.

! 5. Anchor practice improvement in disciplined inquiry.

! 6. Accelerate improvements through networked communities.

Taken Together

15

•  Disciplined Inquiry

•  Rudiments a scientific community

•  Aim: systematic practice improvement

The Six Core Principles

16

! 1. Make the work problem-specific and user-centered.

! 2. Variation in performance is the core problem to address.

! 3. See the system that produces current outcomes.

! 4. We cannot improve at scale what we cannot measure.

! 5. Anchor practice improvement in disciplined inquiry.

! 6. Accelerate improvements through networked communities.

How Do We Heal Medicine? Atul Gawande April, 2012

See the System

18

Gawande’s Closing Observation

Making systems work is the great task of my generation of physicians and scientists. But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.

And at the heart of making systems work is the problem of complexity…

When you are confronted by any complex social system with things about it that you’re dissatisfied with and anxious to fix, you cannot just step in and set about fixing with much hope of helping.

This realization is one of the sore discouragements … You cannot meddle with one part of a complex system without the almost certain risk of setting off disastrous events that you hadn’t counted on.

If you want to fix something you are first obliged to seek to understand it…the whole system.

-‐ Lewis Thomas, 1974 Biologist and Essayist

What We Should Work On:

Task complexity Organiza\onal complexity

Complexity of work in 21st century ins\tu\ons

Need to “see the system”

The Invisible Complexity Schooling

22

The Invisible Complexity of Schooling



Consequences: •  Breakdowns •  Wide variability

in outcomes






in outcomes

Need to iden\fy high-‐leverage problems

Design/develop/refine quality work processes



I. Problem-‐ & User-‐Centered

•  What we tend to do now: a general issue comes into view and

we jump on solu\ons.

!  What is the specific problem we’re trying to solve?

60-‐70% Students assigned to developmental math

course.

80% Percent of these

students that never get past this gate.

500,000 students

in every cohort will never complete a college math

requirement.

26

A Specific High-Leverage Problem to Solve

What We Should Work On: How We Work On It:



in outcomes



Inherent indeterminism

Must learn our way into improvement; “Change it to understand

it.”



V. Engage in Disciplined Inquiry

•  Design-‐development orienta\on, itera\ve cycles

–  The Driving Principle: Quick, minimally intrusive, and an empirical warrant

–  Mantra: Learn Fast, Fail Fast, Improve Fast!

Framing the Learning to Improve Challenge

Current Situation Resistant Indifferent Ready

Low confidence: Good idea, But how to

make it work ????

Limited Capacity

Very Small Scale Test



Good base Capacity



Small Scale Test

High

confidence: Good idea

èè

Can execute

Limited Capacity


Small Scale Test

Large Scale Test

Good base Capacity

Small Scale Test

Large Scale Test

Implement




make it work ????

Limited Capacity




Good base Capacity



Small Scale Test

High


èè

Can execute

Limited Capacity


Small Scale Test

Large Scale Test

Good base Capacity

Small Scale Test

Large Scale Test

Implement




make it work ????

Limited Capacity




Good base Capacity



Small Scale Test

High


èè

Can execute

Limited Capacity


Small Scale Test

Large Scale Test

Good base Capacity

Small Scale Test

Large Scale Test

Implement




make it work ????

Limited Capacity



Small Scale Test

Good base Capacity


Small Scale Test

Modest Scale Test

High


èè

Can execute

Limited Capacity

Small Scale Test

Modest Scale Test

Large Scale Test

Good base Capacity

Modest Scale Test

Large Scale Test

Implement

Wide-spread

33

Of Educational Reform

Going Fast, but Learning Slow.

34

We Are in Good Company

Shil to Learning Fast to Implement Well




in outcomes





it.”



Developing a Quality Process Reliably at Scale

Develop A Change

Test under mul\ple condi\ons

Test under increasingly varied

condi\ons

Make the change permanent

Ini@al Hunches

System Changes

1 school 1 administrator

5 schools Many administrators

En\re ver\cal team A more diverse group of administrators

District Wide All administrators

Seeing Task Complexity

Seeing Organiza\onal Complexity

Learning to improve feedback conversa\ons between

principals and new teachers PLAN DO

ACT STUDY




in outcomes





it.”

Can’t always see all the

consequences of what we

do.

Centrality of measurement: How will you know Δ is an improvement?



You Cannot Improve at Scale What You Cannot Measure

•  Need measureable targets – But, you just can not stand at the end of the line.

•  We need process measures \ed to intermediate targets and key process changes.

Trad

ition

al S

eque

nce

Stat

way

Effects: Time to Complete a College Level Math Course 1 Year 2 Years

Triple the

success rate in half the time.

6%

51%

15%

Produc\ve Persistence

Belonging Uncertainty: Suppor\ve social \es

Target: How do we measure it? Mindsets about the

value of math

Mindsets about poten\al to learn

math

Anxiety Regula\on

Study Skills Conceptual Task: reduce 40+ concepts to 5 core ideas focus on underlying malleable causes + change evidence

Prac@cal Measurement:

reduce 900 items to 26 “you have 3 minutes”

A Primary Driver: key intermediate outcome

41

!

Testing a Change Idea: A Starting Strong Package to Enhance Productive Persistence




in outcomes





it.”

Can’t always see all the

consequences of what we

do.

The Goal: Quality reliably at scale

•  Can we get an idea to work? •  What will it take to make it

work in many other contexts?

Centrality of measurement: How will you know Δ is an improvement?



II. Variation in Performance Is the Problem to Solve

•  Cri\cal ques\on is not: “What Works?” But rather:

“How to advance quality among diverse teachers engaging varied popula\ons of students and

working in different organiza\onal contexts?”

Goal: Achieve quality reliably at scale.

Belonging Uncertainty and Stereotype

12% 13% 14%

28%

40%

7% 11% 14%

50%

71%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Never Hardly Ever Some\mes Frequently Always

Pathways Drop

out

All students Black students

“How olen, if ever, do you wonder: ‘Maybe I don't belong here?’”

N = 714 math students

Variation in Pathways Success Rates by College (n=19)

45

1

2 3

4 5

6 7

8

9

11

12 13

14

15

17

18 19

-50%

0%

50%

100%

0% 50% 100%

Stat

way

Stu

dent

s

Non-Statway Matched Comparisons

No improvement line

We also have a failure, why? What can we learn?

Triple success rate line

VI. Accelera@ng Improvement: Tap the Power of Networks

•  A source of innova\on

•  Mul\ple fast tests/refinements

•  Improvement diffusion A Learning Educa/onal System

A A

Improvement Networks: Accelerate Learning in Prac\ce for Improvement

A

B

A A A B

A A A B

A A A B

C

(Englebart,1994)

A System of Social Learning to Improve: Embracing Disciplined Inquiry

Transla@onal Research

Interven@ons (Alpha Labs)

Will it work in our

context and with out students, and if so,

how?

Expert Prac@@oner Knowledge (Subnet)

Building robust clinical

knowledge about effec\ve materials and instruc\onal prac\ces

(PDSA).

Learning from Network Data (Hub Analy@cs)

Learning from observed variability.

Discerning the unseen.

The Network Improvement Paradigm

49

Researchers vs Users “Knowers” “Doers”

All Improvers

What Works!

How to Make It Work! Replicability as the new

Gold Standard.

“Script it” vs. “Every situation is unique”

Develop Quality Processes to

Support Complex Work

Individual Autonomy As Most Prized Norm

Working Together We Can Accomplish More

"

Implement Fast and Scale Wide

Learn Fast to Implement Well

"

Focus on Standard Effect Size

Focus on Sources of Variability in Performance

"

"

"

"

Developing Practice-Based

Evidence

The Network Improvement Paradigm

“The problem that is managing quality is not just an intellectual endeavor; it is a pragmatic one. The point is not just to know what makes things better or worse; it is to make things actually better.”

–Dr. Don Berwick, Founder Institute for Healthcare Improvement

Learning Fast to Implement Well to Achieve Quality Reliably at Scale.

50

It is all about accelerating how we learn in and through practice to improve.

Our thanks to our foundation partners in this work: Carnegie, Gates, Hewlett, Kresge and Lumina.

Also the Institute of Education Sciences and the National Science Foundation

Tony Bryk - Bristol - Joining Improvement Science to NICs

Education

problem of complexity

core problem

work problemspecific

anchor practice improvement

communities of practice

systems work

complex social system

fidelity practice