Public Service Productivity Measurement:Macro or Micro?
Aileen Simkins, Department of HealthCo-Director of the Atkinson Review
What I will Talk About
• Some concepts
• Macro productivity measurement: the Atkinson Review (and Gershon)
• Micro productivity measurement: some NHS examples
• Challenges for the future
Productivity: What do we mean?
Getting more for the same, or the same for less
Technical efficiencyallocative efficiency
value for moneyeffectiveness
economycuts
Increase in output divided by
increase in volume of input
Improving services while money is tight
Measuring Productivity: Who By, What for?
Do you get what you pay for?
Accountability to the tax payer
Just try harder
Performance management
Can’t we get another one in? - Local drivers to expand/ improve
The XX industry is inefficient by international standards
Evidence of use of resources
Best of breed have 3 times more throughput than bottom quartile
Benchmarking, competitive pressures
Some key questions
• Can Government measures of public service productivity be valid?
• Is measuring productivity the same as measuring performance?
• Improving productivity matters – is measurement a side alley?
• Does measurement of efficiency create distortions?
The Atkinson Review
How to use National Accounts
to measure public service productivity
and
How to do it better
How National Accounts Measure Public Services
• Traditionally, output = input• From 1997, System for National Accounts
changed: measure outputs via activities• UK early implementer – health (cost weighted
activity index), schools (pupil days and quality adjustment), social services, social security admin
• ONS began publishing productivity articles
ONS Output Measure, 2003
Better Measures for the National Accounts
• Queries on basis of public service measures
• National Statistician set up Review Dec 2003, chaired Sir Tony Atkinson
• ‘Measurement of Government Output and Productivity for the National Accounts’
Atkinson Report Jan 2005
• 9 principles ‘output should be measured in a way that is
adjusted for quality, taking account of the attributable incremental contribution of the service to the outcome’
• Specific recommendations for improvement to measures used in NA
• Encouraged ONS productivity articles• UK Centre for Measurement of
Government Activity
ONS Health Productivity Oct 2004
80
85
90
95
100
105
110
1995 1996 1997 1998 1999 2000 2001 2002 2003
Output w ithout quality; inputs: drugs deflated by cost of all items; capitalservices; missing years estimated as average of last 3 years
Output w ithout quality; inputs: drugs deflated by Paasche Price Index;capital consumption; missing years estimated as previous year
DH Press Release Oct 04
John Reid (Secretary of State for Health) says
“ it is absurd to measure NHS output without taking account of quality”
Quality as part of NHS Output• How many domains of quality?
– Health gain– Patient experience
• What can we measure?• How can we link quality measures to the NHS
output index?• How should we weight different aspects of
quality?• How valid is a partial story?
Accounting for Quality Change
Average over last 5 years:• Value of health 1.5%• Value weight for statins 0.81%• York/NIESR adjustment 0.17%• Patient experience* 0.07%• Blood pressure control * 0.05%• Heart attack survival 0.01%Total ** 2.68% Quality adjusted output growth 6.29%
ONS Health Productivity 2006
80
85
90
95
100
105
110
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Output w ith quality and value of health; inputs: drugs deflated by cost of allitems; capital consumption, direct labour method
Output w ith quality and value of health; inputs: drugs deflated by PaaschePrice Index; capital services, indirect labour method
Education and Other Areas• DfES action following Atkinson – new use of
pupil attainment tests and GCSE grades• DfES article, ONS education productivity
article• Controversial issues – grade drift, ‘real
earnings’ effect• Also productivity articles for adult social
care, social security administration
Value and Validity of Atkinson-based Productivity Measures• Designed to compare total outputs with total
inputs• Focus on attributable impact on outcomes
and quality change• Data incomplete; biased towards areas of
attention / improvement; analysis by Depts• Major developmental issues on techniques• ONS consultation document Sept 2006
Performance = Productivity?• A perfect Atkinson measure captures all
activities, all contributions to outcomes
• Is this the same as measuring organisation performance?
• If not – choices about key priorities, value judgements
• Should those values inform the aggregate productivity measure?
Another macro measure: Gershon
• Gershon includes savings from better procurement, Atkinson doesn’t
• Specific changes in use of inputs valued as £m
• Better use of productive time – evidence from outputs and outcomes?
Gershon and Atkinson
• Gershon efficiency improvements are managed, not just measured
• Strong delivery support for Gershon, changes evident
• Measured to deliver target? – NAO scrutiny• Will measured productivity gains match? –
HMT, OGC and UKCeMGA should be able to explain
Micro Measures: NHS
• NHS Institute for Innovation and Improvement, with DH
• Developing and issuing key metrics for health authorities, PCTs, Trusts, FTs
• Systematic focus on key areas for improvement – volume, costs, variation
• Clinical engagement, benchmarking
First Metrics: 22 Sept 2006
• Pdfs and web based tool• Finance, workforce, procurement • Clinical productivity:
– Potential bed days saved through reduced length of stay
– Day case rate– Reduce pre-operative bed days
Reduce wasted bed days
Percentage of patients admitted on the day of operation
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Acute hospital trusts
390,000 bed days£100m
Benchmarking
• Evidence based assessment of performance• Addressing variations and helping outliers• Providing a tool kit• Opportunity to test performance against
peers• Stimulus to learn how to improve practice• Recognition of success and challenge to do
better
XXX NHS Trust
Value and Power of Micro Productivity Indicators
• Practical engagement in what to change• Service based, clinical focus• Small steps on long journey• Risks that change happens, savings don’t• Check for unintended perverse incentives• Tipping points for cutting overheads
Impact of Micro Changes on Macro Measure
NHS III says:1 – Reduce Avoidable Emergency Admissions 2 – Reduce Unnecessary Outpatient Appointments,
Follows-up, and DNAs 3 – Avoid Unnecessary Procedures 4 – Improve Day Case Performance 5 – Reduce Wasted Bed Days6 – Accurate Clinical Coding7 – Reduce Variation in Length of Stay8 – Improve Staff Productivity9 – Actively Manage Staff Costs
If NHS follows NHS III advice..• Lower growth in some hospital activities• Shifts in unit costs• Shifts in NHS spending to primary care,
community, public health (harder to count)• Probably, slower growth in output• Measurable change in quality, outcomes?• Productivity measure?
Measures, not Targets
• Macro and micro productivity measures both need careful interpretation
• Micro are more useful for managers, front line, actionability
• Macro should help with ‘do you get what you pay for’ – but may mislead
• Use of any efficiency measures changes behaviours – intended, maybe also unintended
Lessons of History
‘For the first decade of its existence (1982-92) the HCHS efficiency index received little adverse comment either from the NHS or policy analysts. However, once it was converted from a retrospective analytical exercise into a management tool it attracted a barrage of criticism….attaching rewards to counts of particular activities led to reporting drift, reclassification and at worst statistical dishonesty’
Clive Smee ‘Speaking Truth to Power’ 2005
Dilemma: Measuring with Right Impact
• Productivity is Govt business: accountability • Many drivers in public service to improve
productivity• Accountability needs metrics• Metrics are almost always partial, may
mislead or have unintended incentives • Choice of metrics affects what is delivered• Moral: use mix of macro and micro
indicators, with intelligence & transparency