A Framework for Understanding Statistical Performance Paul Askew CONFERENCE 2-5 SEPTEMBER 2013 NEWCASTLE
Jun 24, 2015
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
1. Introduction
Operational Delivery
Methodological Leadership
1. Scope….A framework for
Managing Statistics about performance
(rather than performance of statistical techniques)
2. Operational Origins
• More about practical drivers and process
• Utility….target setting, performance improvement
3. Distilling application and development across sectors….
• Criminal justice, regulation, education, health
• It really matters….safety, housing, education….
1. Introduction
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
Data
Eviden
ce
Decisi
ons
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
“Burglary is down compared to last month”
“Yes but it’s up compared the same month last year”
“Yes but it’s down overall for the financial year to date”
“Yes but its’ up for the calendar year so far”
“Yes but we’re still less better than our neighbours”
“Yes but they are reducing faster than we are this year”
“Yes but we’re still under (over) target”.
etc………….
1. It actually matters to people – safety, home, education
2. Performance Regime – broad scope, high profile, deep drill down
3. “Multi-multi” dimensional – both of measures and assessments
4. Statistics meaning – datum, summary, technique,
5. Targets - legal, audited, collaborative!
6. Performance Pantomime
7. Less about techniques, more about process
8. Operational Delivery – police, health, regulation…
2. Why - Operational Drivers
Smoothed Data or Real Data
Highs and lowsHigh and low
DecreasingIncreasing Increasing convergence
Decreasing convergence
Three month step Six month stepTwo month step
Smoothed Data – 12 month rolling average
Smoothed Data
Example Real Data
Notes: Real data for 12 months, previous 12 months is exactly the same, to create 12 month rolling average (mean).
This smoothed data is derived from any of these underlying
raw data examples.
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Scope - Detail - Volume
Data.gov…10K
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Words
Numbers
Data
Eviden
ce
Decisi
ons
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
Skills for Life Survey 2011 (England)Department for Business Innovation and Skills
• Literacy Improving while Numeracy declining
Numeracy • 26% to 22% (7.5m
adults) with GCSE+
• 17m adults at primary school level
The numeracy challenge is big and getting bigger…%
Adu
lts a
t GC
SE
+ L
evel
s
A Framework for Understanding Statistical Performance
Paul Askew
1. Data, Evidence, Decisions… Impact, Value.
2. Big & Open & Now data
3. Tactical vs. Strategic focus
4. Key Strategies…Communication emphasis - ONS, RSS…
5. Underlying Numeracy and statistical literacy
6. Policy Perception Gap
7. Data Science – Shakespeare review, Open Data, UKSA…
8. Austerity World - Effective (right thing) & Efficient (right way)
2. Why - Current and Strategic Drivers
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro level
Analytical level
Outline
3. How - Macro
DATA - inputs -
ANALYSIS- process -
PRODUCTS- outputs -
INSIGHT- outcomes -
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
II = (d)D x (t)T x s(S) x (c)C x (q)Q x (i)IThe factors:
D Data: Right data? Enough of it? Good enough?T Tools: Have any? Right ones?S Skills: Have any? Right ones?C Capacity: How much? Realistic?Q Question: Specific question to answer or issues to address I Inclination: Desire and drive to want to address the issues
d,t,s,c,q,i Relative weights
Askew Analytical Insight Index
1. Purpose
2. Require-
ments
4. Design
3. Constr-
aints
5. Defiine
6. Specify
8. Record
7. Collect
9. Entering
10.Process
12. Storage
11. Validate
2.Tools
1. Data
3.Skills
4.Capacity
5.Question
6.Inclinat-
ion
Plan
Implement
Manage
3.Target
2. Bench-mark
1. Trend
ANALYSISComp-
aritorsTime
Periods
DATA
Time Periods
Keys Message
INSI
GH
T
PRODUCTS
Cover the
angles
Lift Pitch EvidenceSummary
Comms
Stake-holders
Synthesis
- inputs -
- process --
- outputs -
- out
com
es - Analysis Strategy
Words
Graphics
Numbers
OPE
NOPEN
OPEN
OPEN
3.Target
2. Bench-mark
1. Trend
Comp-aritors
Time Periods
Time Periods
3. How – Analytical Level
0. Snapshot
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
0. Snapshot - we have a number which is important to us
1. Trend - what’s happening to our measure over time
2. Benchmark – how this compares to others
2a. Trend for the comparison to others
3. Target – the trajectory for our measure
3a. – the comparison trajectory 3.2. 1. 0.
3. How – Analytical Level
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
0. Snapshot – we have a number which is important to us
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
1. Trend – what’s happening over time
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2. Benchmark – how this measures compares to others
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
2a. Trend for the comparison to others
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3. Target - the trajectory for our measure
t-9 t-8 t-7 t-6 t-5 t-4 t-3 t=2 t-1 t=now t+1 t+2 t+3 t+40
20
40
60
80
100
120
140
160
Value
Time
3a. Target - Trajectory for the comparison to others
1. Introduction
2. Framework – the “Why”
Operational Drivers
Current Strategic Drivers
3. Framework – the “How”
Macro
Analytical
Outline
A Framework for Understanding Statistical Performance
Paul Askew
CONFERENCE2-5 SEPTEMBER 2013
NEWCASTLE
Thank You