Evaluating new models of care: Improvement Analytics Unit Martin Caunt, Improvement Analytics Unit Project Director, NHS England Adam Steventon, Director of Data Analytics, The Health Foundation
Evaluating new models of care: Improvement Analytics Unit
Martin Caunt, Improvement Analytics Unit Project Director, NHS
England
Adam Steventon, Director of Data Analytics, The Health
Foundation
www.england.nhs.uk
The NHS will take decisive steps to break down
the barriers in how care is provided
England is too diverse for a ‘one size fits all’ care
model to apply everywhere. Different local health
communities will be supported by the NHS’s national
leadership to choose from amongst a small number of
radical new care delivery options
…we will expand NHS operational research, RCT
capability and other methods to promote more
rigorous ways of answering high impact questions in
health services redesign
NHS England’s programmes of work have their roots in the Five
Year Forward View
www.england.nhs.uk
What is the problem?
What are the options?
What is the theory
underpinning the proposed
model?
What is causing these
changes?
What is happening on the
ground? What progress is
being made?
What should we be
replicating and spreading?
What difference are the
care models making?
How do you best
implement x, y, z?
Dashboard / outcome metrics;
Improvement Analytics Unit; local evaluations; Impact studies;
Independent DH study
Evidence reviews;
Logic models;
Independent DH study Enabler metrics;
Impact studies;
Local evaluations
Local evaluations
Impact studies
Local evaluations; Impact
studies; Independent DH
study
Local evaluations;
Evidence reviews
NHS England has developed a response to core
questions for the New Care Model programme
A set of questions from around the policy cycle, being posed more
frequently by senior stakeholders.
www.england.nhs.uk
• Complex interventions, with no implementation blueprint
• Understanding of impact is not sufficient
• Traditional or piloting and trialling approaches to evaluation are
not appropriate
• Data available nationally is limited in its timeliness, quality and
coverage
Significant national challenges to assessing the
impacts of innovations to the delivery of care
www.england.nhs.uk
• Data problems
• Getting person-level data on intervention and
comparison groups is hard (but possible)
• Statistical problems
• In the absence of randomisation, the treatment
groups usually differ at baseline, meaning that a
straightforward comparison between their
outcomes is biased
6
Problems with evaluation
www.england.nhs.uk
• Regression adjustment
• Matching methods
• Propensity score matching
• Predictive risk score matching
• Genetic matching
• Doubly robust methods
• Instrumental variables
• Regression discontinuity
7
Statistical methods exist to estimate
treatment effects without randomisation
These make different
assumptions
Pick the one with the
most reasonable set
of assumptions
www.england.nhs.uk
• A telemonitoring service in a large rural county
• Open to people with chronic obstructive pulmonary
disease, congestive heart failure or diabetes, and a
history of associated inpatient admission
• Controls were selected from within the county using
genetic matching, based on predictive risk score,
demographics, diagnosed health conditions and prior
hospital utilisation
• Some checks for unobserved confounding
8
A quick example
Steventon A, Ariti C, Fisher E, Bardsley M. Effect of telehealth on hospital utilisation and mortality in routine
clinical practice: a matched control cohort study in an early adopter site. BMJ Open 2016:e009221.
www.england.nhs.uk
Matching to find a similar control group
0%
20%
40%
60%
80%
Telemonitoring patients Matched controls
www.england.nhs.uk
Telemonitoring was associated with more emergency hospital
admissions than usual care in this particular study
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Emergencyadmission
Admission forambulatory
care sensitivecondition
A&E visit Electiveadmission
Outpatientattendance
Death
Higher utilisation
amongst
telemonitoring
patients than
controls
Lower utilisation
amongst
telemonitoring
patients than
controls Not suggestive of
unobserved
confounding
www.england.nhs.uk
Another check for unobserved confounding
But the observed
confounders are
not nearly as
strongly
associated
To have
obscured a
reduction in
emergency
admissions, an
unobserved
confounder would
need to have
been this strongly
associated with
outcome /
treatment
www.england.nhs.uk
• Data problems
• Can be solved
• Statistical problems
• Can be solved (subject to certain assumptions)
• Can we develop teams in the NHS to apply these
methods on a regular basis to improve care?
12
Problems with evaluation
www.england.nhs.uk
Additional challenge 1: Timeliness
13
Data Data Data
Evaluation
findings Evaluation
findings Evaluation
findings
www.england.nhs.uk 14
Continue?
Stop?
Refine the logic model?
Spread the change more widely?
Data Data Data
Evaluation
findings Evaluation
findings Evaluation
findings
Additional challenge 2: Informed decision making
www.england.nhs.uk
• Rapid feedback (ultimately end-to-end
within 2-3 months)
• Focus on achieving scale (‘engineering
not research’)
• Embedding analytics within the ‘data
bloodstream’
• Partnership between NHS England and
The Health Foundation
• Joint team (5 people from The Health
Foundation, 5 from NHS England)
• Using nationally available data
The Improvement Analytics Unit will help
determine if discernible change has happened
Aim by 2019 to create a unit to
provide rapid feedback on the
progress being made to improve care
and efficiency as part of national
programmes in England
www.england.nhs.uk
We are working with two pilots to inform subsequent roll
out to more local areas and programmes
www.england.nhs.uk
• Each care home having an assigned GP practice
• A named Community Matron / Senior Nurse for each care home
• Dedicated Allied Health Professional support for falls
• Community follow-up 48 hours post hospital admission
• In-reach of specialist community services/professionals including geriatrician, dietetics, heart failure and respiratory
• Regular proactive resident reviews, including medication
• Reviews of new residents within five days and a Comprehensive Geriatric Assessment (CGA) within two weeks
• Standardised long-term conditions planning and increased identification of dementia
• Explicit end of life care discussions with resident, family and carers
• Peer support for care homes staff
• Direct access to district nurse on call at weekends
• Advocacy support to residents and families including ‘worry catcher’
Source: Principia vanguard team
Working with Principia Partners in Health vanguard (MCP) to help
assess its model of enhanced support to Care Homes
www.england.nhs.uk
Working with Northumberland vanguard (PACS) to help assess the impact
of creation of their integrated primary and acute care system
Source: Northumberland vanguard team
www.england.nhs.uk
Mapping processes
Reporting
Producing
analysis
data set
Analysis Site
selection Design
www.england.nhs.uk
Analysis phase in more detail
Estimate
treatment
effects (e.g.
by fitting
regression
models)
Pre-process
data set (e.g.
by producing
matched
control
groups)
Diagnostics
on final pre-
processed
data set (e.g.
balance after
matching)
QA to confirm
data are as
expected
Analysis data
set received
from data
management
team
Report
diagnostics to
site
Derive
additional
variables if
needed (e.g.
by conducting
risk-
adjustment)
QA to confirm
the process
involved in
producing the
estimated
treatment
effects
Conduct
planned
subgroup
analyses
Conduct
sensitivity
analyses
Prepare site
report (see
separate flow
diagram)
Report
estimated
treatment
effect to site
www.england.nhs.uk
12Q1-12Q4 12Q2-13Q1 12Q3-13Q2 12Q4-13Q3 13Q1-13Q4 13Q2-14Q1 13Q3-14Q2 13Q4-14Q3
14Q1-14Q4 14Q2-15Q1 14Q3-15Q2 14Q4-15Q3 15Q1-15Q4
Tow
er H
amle
ts IP
P
Prin
cipi
a Pa
rtne
rs in
Hea
lth
Lake
side
Sur
geri
es
Sout
hern
Ham
pshi
re
Natio
nal
Erew
ash
MCP
(Der
bysh
ire)
Whi
tsta
ble M
edic
al P
ract
ice
Cald
erda
le H
&SC
Eco
nom
y
Prim
ary C
are C
hesh
ire
MCP
Vita
lity
NHS
Dudl
ey C
CG
Wes
t Wak
efie
ld H
&W
Ltd
Fyld
e Co
ast L
ocal
Hea
lth E
cono
my
Bette
r H&
C fo
r Sun
derl
and
Stoc
kpor
t Tog
ethe
r
Tow
er H
amle
ts IP
P
Prin
cipi
a Pa
rtne
rs in
Hea
lth
Lake
side
Sur
geri
es
Sout
hern
Ham
pshi
re
Erew
ash
MCP
(Der
bysh
ire)
Whi
tsta
ble M
edic
al P
ract
ice
Cald
erda
le H
&SC
Eco
nom
y
Prim
ary C
are C
hesh
ire
Vita
lity
NHS
Dudl
ey C
CG
Wes
t Wak
efie
ld H
&W
Ltd
Bette
r H&
C fo
r Sun
derl
and
Stoc
kpor
t Tog
ethe
r
MCP
Fyld
e Co
ast L
ocal
Hea
lth E
cono
my
0
20
40
60
80
100
120
140
Emer
genc
y Ad
miss
ions
-Ac
tivity
(per
1,0
00 p
opul
atio
n)
Emergency Admissions - Activity (per 1,000 population)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP Other Vanguards National
NEW CARE MODELS EVALUATION AND MEASUREMENT NATIONAL METRICS DASHBOARD
MAIN MENU
-
5.0
10.0
15.0
20.0
25.0
30.0
35.0
2012Q1
2012Q2
2012Q3
2012Q4
2013Q1
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
Emer
genc
y Ad
miss
ions
-Ac
tivity
(per
1,0
00 p
opul
atio
n)
Emergency Admissions - Activity (per 1,000 population)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP National
0.70
0.75
0.80
0.85
0.90
0.95
1.00
1.05
1.10
1.15
2012Q1
2012Q2
2012Q3
2012Q4
2013Q1
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
Emer
genc
y Ad
miss
ions
-Ba
selin
e Ra
tio (2
012Q
1)
Emergency Admissions - Baseline Ratio (2012Q1)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP National
Tow
er H
amle
ts IP
P
Whi
tsta
ble M
edic
al P
ract
ice
Cald
erda
le H
&SC
Eco
nom
y
Vita
lity
Natio
nal
Prin
cipi
a Pa
rtne
rs in
Hea
lth
NHS
Dudl
ey C
CG
Wes
t Wak
efie
ld H
&W
Ltd
MCP
Sout
hern
Ham
pshi
re
Prim
ary C
are C
hesh
ire
Erew
ash
MCP
(Der
bysh
ire)
Bette
r H&
C fo
r Sun
derl
and
Stoc
kpor
t Tog
ethe
r
Fyld
e Co
ast L
ocal
Hea
lth E
cono
my
Lake
side
Sur
geri
es
Tow
er H
amle
ts IP
P
Whi
tsta
ble M
edic
al P
ract
ice
Cald
erda
le H
&SC
Eco
nom
y
Vita
lity
Prin
cipi
a Pa
rtne
rs in
Hea
lth
NHS
Dudl
ey C
CG
Wes
t Wak
efie
ld H
&W
Ltd
Sout
hern
Ham
pshi
re
Prim
ary C
are C
hesh
ire
Erew
ash
MCP
(Der
bysh
ire)
Bette
r H&
C fo
r Sun
derl
and
Stoc
kpor
t Tog
ethe
r
Lake
side
Sur
geri
es
MCP
Fyld
e Co
ast L
ocal
Hea
lth E
cono
my
0
200
400
600
800
1,000
1,200
Bed
Days
-Ac
tivity
(per
1,0
00 p
opul
atio
n)
Bed Days - Activity (per 1,000 population)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP Other Vanguards National
12Q1-12Q4 12Q2-13Q1 12Q3-13Q2 12Q4-13Q3 13Q1-13Q4 13Q2-14Q1 13Q3-14Q2 13Q4-14Q3
14Q1-14Q4 14Q2-15Q1 14Q3-15Q2 14Q4-15Q3 15Q1-15Q4
-
50.0
100.0
150.0
200.0
250.0
300.0
2012Q1
2012Q2
2012Q3
2012Q4
2013Q1
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
Bed
Days
-Ac
tivity
(per
1,0
00 p
opul
atio
n)
Bed Days - Activity (per 1,000 population)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP National Average
MET
RIC
1 M
ETRI
C 2
0.70
0.75
0.80
0.85
0.90
0.95
1.00
1.05
1.10
1.15
2012Q1
2012Q2
2012Q3
2012Q4
2013Q1
2013Q2
2013Q3
2013Q4
2014Q1
2014Q2
2014Q3
2014Q4
2015Q1
Bed
Days
-Ba
selin
e Ra
tio (2
012Q
1)
Bed Days - Baseline Ratio (2012Q1)Vanguard: Fylde Coast Local Health Economy - NCM: MCP
Fylde Coast Local Health Economy MCP National Average
For more information please visit:
http://www.health.org.uk/improvement-analytics-unit-
faqs
Questions?