Measurement for Improvement 18 March 2008 Mike Davidge.

Measurement for Improvement


18 March 2008

Mike Davidge


The prologue

What actually is

the question?


What are we trying toaccomplish?

How will we know that achange is an improvement?What changes can we make

that will result in the improvements that we seek ?

Model for improvement

Act Plan

Study Do

aims

measurements

interventions


How does measurement help us?

• By showing the variation that lives within our data (and therefore our processes)

• By indicating whether the way we react to that variation is taking us in the right direction

Source: Robert Lloyd IHI 2006


The measurement journey


Why are we measuring?

Resea

rch

Resea

rch

JudgementJudgement

Improvem

ent

Improvem

ent

The answer to this question will guide our entire The answer to this question will guide our entire measurement journeymeasurement journey


Measurement for judgement

0.00

5.00

10.00

15.00

20.00

25.00

35

37

39

41

43

45

47

49

51 1 3 5 7 9

11

13

15

17

19

21

23

25

27

29

31

33

35

% waiting over 4 hours in A&E

England by week 2003/04

target

The week Trusts were

measured for performance

ratings


Outpatient waiting times

The rise and fall of outpatient long waiters(waiting over 13 weeks)

0

1

2

3

4

5

6

Jun-96

Dec-96

Jun-97

Dec-97

Jun-98

Dec-98

Jun-99

Dec-99

Jun-00

Dec-00

Jun-01

Dec-01

Nu

mb

er

(00

,00

0s

)


The 3 reasons for measurement


Why is measurement

for improvement

different?


Take two numbers

29

12

0

5

10

15

20

25

30

35

Q1 Q2


Take two numbers

1229


We have 2 quarterly data points - is this an improvement?

Executive Time Series

0

20

40

60

80

100

J F M A M J J A S O N D

Months

So

me

thin

g I

mp

ort

an

t

Higher is better


Are we assuming something like this?

Executive Time Series - linear trend

0

20

40

60

80

100


Months

So

meth

ing

Im

po

rtan

t


But it could be like this ...

Executive Time Series - no trend

0

20

40

60

80

100


Months

So

meth

ing

Im

po

rtan

t


Or this ...

Executive Time Series - seasonal dip

0

20

40

60

80

100


Months

So

me

thin

g I

mp

ort

an

t


Or this!

Executive Time Series - one month blip

0

20

40

60

80

100


Months

So

me

thin

g I

mp

ort

an

t


Case study: Housing repairs

Example courtesy of John Seddon in ‘Freedom from Command & Control’


The scenario

• A Local Authority manages a large stock of housing• They manage repairs using 2 measures

BudgetTime to repair

• ‘Time to repair’ is a government mandated BVPI (best value performance indicator) measured in %% emergencies completed in 24 hrs% urgents completed in 7 days% routines completed in 28 days

• The LA achieving all BVPI’s


Hunky dory?

• What does the customer want?

• Repair done quickly and properly

• How do we know we are delivering this?

• Let’s measure the end to end time – from request to job complete

• But there’s a problem – the IT system couldn’t provide the data – they only had the BVPI percentages so had to track each job manually


SPC chart showing true performance

Total Elapsed Time of Repairs

0

20

40

60

80

100

120

140

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Consecutive jobs

Days t

o c

om

ple

te

Job time Average (31) UCL (85)


Not so hunky dory …

• Special causes not different – system unstable

• Achieve targets by distorting the data & systemIgnore if tenant is outReclassifying emergencies as urgentsSplitting a job into several, each one opened & closed

• Got a problem? Have a management restructure!Actually destabilised the system even more

Total Elapsed Time of Repairs

0

20

40

60

80

100

120

140

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49

Consecutive jobs

Days t

o c

om

ple

te

Job time Average (31) UCL (85)


So what next?

• They understood they had a problem

• They looked at the process and discovered it was broke40% ‘failure’ demand at the call centreTradesmen scheduling work to maximise earningsQueuing to get the right materials

• Led to redesigning the process

• Within weeks all repairs completed within 8 days


They didn’t know they had a problem until they measured the right thing

in the right way!


And that reminds me of …

“What gets measured gets managed”

“even when it’s pointless to measure and manage it and even if it harms the

purpose of the organisation to do so.”


The 3 reasons for measurement


Why is measurement

for improvement

different?






Act Plan

Study Do

aims

measurements

interventions

Source: IHI

Save 1000 lives and 50,000 episodes of harmHSMR and Global Trigger Tool

6 Content Areas

national






Act Plan

Study Do

aims

measurements

interventions

Use the measures and interventions for each of the content areas

local


How to measure?

CC

AA

RR

ollect – the right data in a sustainable way

nalyse – the right measures presented in a helpful way

eview – whether your changes are having the desired effect

Start collecting the data and using it for local decision making


The ‘How’ questions

• How many of your project leads can:Collect the relevant dataAnalyse data effectively (create run charts)Review it regularly (interpret charts and make the right

decisions)

• How much time do your Information analysts have to support them?

• How many of the content area measures can be obtained via your corporate IT systems?


In summary

• Measures follow aims – clarity here means you know what 42 means

• Measurement for improvement is different from the other types of measurement we are used to

• Plot your data over time to understand current situation and track changes

• Ensure your project teams have the right measurement skills

• Re-align your Information department to provide longer term sustainability

Measurement for Improvement 18 March 2008 Mike Davidge.

Documents

right data

right measures

right way

data systemignore

measurement journeywhy

right materialsled

right directionsource

right thing