Case Study Delivering Major Public Sector Change Managing The Portfolio Realising The Benefits 2 Stephen Jenner Version 2

Managing the Portfolio, Realising the Benefits

BPUG Congress, 10th February 2009

Stephen Jenner

[email protected] OR [email protected]

Framing statement

“I love money.

I love money more than the things it can buy.

There’s only one thing I love more than money.

You know what that is?”

Framing statement

“OTHER PEOPLE’S MONEY”

Danny DeVito as ‘Larry the Liquidator’

1991, Warner Bros Films

Change programmes - the rationale

“The fundamental reason for beginning a programme is torealise the benefits through change.”Office for Government Commerce (OGC), ‘Managing Successful Programmes’

“It is only possible to be sure that change has worked if we can measure the delivery of the benefits it is supposed to bring.”UK Cabinet Office, ‘Successful IT: Modernising Government in Action’

But the track record is not good…

“Implementation of IT systems has resulted in delay, confusion and inconvenience to the citizen and, in many cases poor value for money to the taxpayer.”

Public Accounts Committee, 2000

“30-40% of systems to support business change delivery no benefits whatsoever.”

Office of Government Commerce, 2005

“the committee has no confidence that the amounts being assessed have any relationship to the benefits anticipated to be returned.”

US Senate Appropriations Committee, 2007

And ‘horror’ stories abound…

Cost overruns

• Concorde (1100%) • Channel Tunnel (80% for construction and 140% for financing costs)• Scottish Parliament – 10 times over budget and late• Jubilee line extension costs underestimated by £1.4bn

Benefits shortfalls

• The Dome visitors down by 5-6 million• Eurostar passenger forecast in yr1 of full operations = 15.9m, actual = 2.9m• Bangkok skytrain – demand 2.5 times over-estimated• Humber Bridge – traffic 25% overestimated in yr1 of full operation.

And empirical research concludes…

“There is a demonstrated, systemic tendency for project appraisers to be overly optimistic. This is a worldwide phenomenon that affects both the private and public sectors…appraisers tend to overstate benefits, and underestimate timings and costs.” HM Treasury

“it is found with overwhelming statistical significance that the cost estimates used to decide whether such projects should be built are highly and systematically misleading.”

Flyvbjerg

“Delusional optimism: we overemphasise projects’ potential benefits and underestimate likely costs, spinning success scenarios while ignoring the possibility of mistakes.”

Lovallo and Kahneman

Four explanations…

1. The technical explanation - our tools are inadequate.

2. The psychological explanation – Optimism bias, derived from cognitive biases.

3. The economic explanation – continuing with a project creates work for project managers, suppliers etc.

4. The political explanation – “Strategic misrepresentation”?

Delusion or Deception?

“the planned, systematic, deliberate misstatement of costs and benefits to get projects approved.” In short, “that is lying”. And that’s because, “Lying pays off, or at least economic agents believe it does.”

Flyvbjerg et al

“Figures don’t lie, but liars can figure”Sharpe and Keelin

38% of respondents in one survey openly admitted to overstating benefits to get funding with the traditional investment appraisal process being, “seen as a ritual that must be overcome before any project can begin”

Ward

The Result - Business Cases contain ‘assumptions masquerading as facts’

The Solutions

1. Be clear about the benefits you are buying.

2. Spend more time doing your homework.

3. Triangulate and Validate project proposals.

4. Appraise ‘Attractiveness’ in the context of ‘Achievability’.

5. ‘Gates with teeth’.

6. Independent review.

7. A ‘Clear line of Sight’ from Strategic Intent through to Benefits Realisation.

The Solutions: 1. Be clear about the benefits you are buying

Beware staff time savings – they are vouchers, they only have a value when they are cashed or the time is re-deployed to other value adding activity.

The Solutions: 1. Be clear about the benefits you are buying

Beware ‘Strategic alignment’ as an investment justification,

“Our CEO defines ‘strategic projects’ as expensive projects without a businesscase.”

Corporate Executive Board paper

The Solutions: 1. Strategic Contribution AnalysisBe clear about the benefits you are buying

Vision Strategies Metrics

To change public services so they

more often meet the needs of people and businesses, rather than the needs of

government, and by doing so reduce the

frustration and stress of accessing them.

The result will be services that are

better for the customer, better for front line staff and

better for the taxpayer (Service Transformation

Agreement, October 2007).

Service Transformation Programme – High Level Strategy Map

Learning from citizens and businesses – Customer Insight, Customer Journey

Mapping, Customer Satisfaction Measurement

Grouping services in ways that are meaningful to the customer e.g. Tell Us Once, Directgov and Businesslink.gov

Rationalising services for efficiency and service improvement – online

(Directgov and Businesslink.gov), phone (contact centre accreditation);

face to face and helplines

Making better use of customer information public sector already holds:a. Strategic level – Home Office lead on

Identity mgt and MoJ on information sharing

b. Tactical level – Tell Us Once

Linking local and central government

Engaging front line staff

Government will monitor the customer experience through journey mapping and customer satisfaction tracking

mechanisms at the front line (STA Oct 07)

Key progress measures 1 - Reduction in the amount of avoidable contact. To achieve a 50% reduction by 31.3.11.

(STA Oct 07).

Key progress measure 2 - Building better online services – citizen and

business e-services content migrated to Directgov and Businesslink.gov. More than 95% of websites to have migrated

by 31.3.11. (STA Oct 07)

Efficiency savings – the value in recording the level of savings achieved by departments is recognised, and the

CO will track these as this STA is delivered. (STA Oct 07).

Face-to-Face – The LGDC will develop a progress measure reflecting the

FOSS approach for later inclusion in this Agreement. (STA Oct 07)

The Solutions:2. Spend more time doing your homework

“It’s the Business Case, stupid”

Failing projects don’t have brilliant business cases

The Solutions: 3. Triangulate and Validate

Use more than one appraisal method, or ‘value lens’:

• Combine financial metrics with multi-criteria analysis and decision conferencing.

• Validate/book the benefits with the recipients – before investment.

• Establish effective accountability for delivery and benefits realisation.

The Solutions:4. Appraise ‘Attractiveness’ in the context of ‘Achievability’

The Solutions: 5. ‘Gates with teeth’

• Regularly requiring project teams and business sponsors to ‘sing for their supper’.

• Presumption – if you exceed tolerance, funding ceases.

• Concludes with a formal re-commitment to benefits realisation.

• Challenge and Support.

The Solutions: 6. Independent review

“humans not only are prone to make biased predictions, we’re also damnably overconfident about our predictions and slow to change them in the face of new evidence. In fact, these problems of bias and overconfidence become more severe the more complicated the prediction.”

Ayers

The Solutions: 6. Independent review

Ayers suggests an, “’Advocatus Diaboli’…whose job it is to poke holes in pet projects. These professional “No” men could be an antidote to overconfidence bias.”

Davidson Frame proposes the use of “murder boards” to pull a proposal apart to, “make sure that arguments in support of project ideas do not have built into them the seeds of their own destruction.”

Steve Jenner - I suggest a ‘fool’ to ask the questions others don’t dare to ask and identify those, ‘assumptions that masquerade as facts’.

The Solutions:7. A Clear line of sight from strategic intent

to benefits realisation

• Investment Appraisal report

• Benefits Scorecard

• Portfolio Analysis • Proving Model report

SR2004 CSR07 10 Year TotalQuality of

Benefit Forecast

Scale of Benefits Forecast

Quality ofRealisation

PlanningLikelihood ofRealisation

Confidence Victims and Witnesses OBTJ Enforcement Re-offending Q1 03/04 - Q4

05/06 2006/07 2007/08NSPIS Custody & Case Prep 79.08 199.82 348.06 AMBER AMBER AMBER AMBER Realised Plan ActualPentiP - 5.08 14.06 GREEN GREEN AMBER AMBER Efficiency Cashable - - 0.39 0.71 COMPASS CMS (50%) 10.95 20.79 52.74 GREEN AMBER GREEN AMBER HD=1 Efficiency Opportunity 4.95 2.69 26.136 50.59 NWNJ IT tool (WMS) 5.70 9.66 25.02 GREEN GREEN GREEN AMBER D=2 HD=1 NK=1 HD=1 Effectiveness 0.26 0.24 0.94 1.12 SOCA 0.61 1.21 3.43 GREEN GREEN GREEN GREEN M=1 Total 5.21 2.93 - 27.473 52.42 Libra application (includes benefits jointly delivered with 3a) 18.15 81.70 154.31 GREEN RED AMBER AMBER M=1 NK=1

OASys 79.52 93.18 239.12 GREEN GREEN GREEN GREEN MC=2Q1 03/04 - Q4

05/06 2006/07 2007/08NOMIS (70%) 5.06 50.00 162.33 GREEN GREEN GREEN GREEN MC=1 HD=1 Realised Plan ActualViSOR 0.93 2.21 6.08 GREEN GREEN GREEN GREEN Efficiency Cashable - - - - CJSE Release 1a (NSPIS-CMS) 1.63 9.76 17.88 GREEN GREEN GREEN AMBER Efficiency Opportunity 31.40 8.88 35.50 39.04 CJSE Release 1b (NSPIS - Libra) 0.87 19.08 37.29 AMBER GREEN AMBER AMBER Effectiveness 0.13 0.39 1.56 2.83 XHIBIT/CJSE Release 2a&b (XHIBIT Portal) 9.29 7.11 21.15 AMBER RED GREEN AMBER M=1 D=1 M=1 Total 31.53 9.27 - 37.06 41.87 PROGRESS 1.88 12.02 26.04 GREEN GREEN GREEN GREEN M=4 HD=1 D=2

Secure Email/Emailing Securely 1.49 1.90 4.78 AMBER RED AMBER AMBER D=2Q1 03/04 - Q4

05/06 2006/07 2007/08CJSE Release 3a 0.00 0.01 0.02 AMBER AMBER AMBER Realised Plan ActualCJSE Release 3b 1.10 2.13 6.07 GREEN AMBER AMBER Efficiency Cashable - 0.70 17.45 COMPASS infrastructure (50%) 85.95 95.79 252.74 see COMPASS CMS Efficiency Opportunity 2.59 0.53 2.67 1.70 Libra enabled 30.17 34.26 95.54 see LIBRA application Effectiveness - 0.34 2.54 4.42 NOMIS infrastructure (30%) 2.17 21.43 69.57 see NOMIS application Total 2.59 0.87 - 5.91 23.57 OMNI infrastructure 1.03 3.11 6.48 OMNI cost effectiveness - 9.36 36.37

LINK enabled 8.68 11.79 28.49 Q1 03/04 - Q4

05/06 2006/07 2007/08

Shared Access 5.85 7.80 24.05 D=2 Realised Plan ActualEquip direct 16.50 16.50 55.00 Efficiency Cashable - - - - Equip enabled 41.41 149.10 311.00 Efficiency Opportunity 26.77 5.68 22.70 24.36

CEI 9.78 21.43 45.54 D=9 M=1 HD=3 D=5 M=1 HD=11 D=3 M=1 D=5 M=1 NK=1 D=1 M=1 Effectiveness 12.40 5.26 21.06 21.21 Economic/Social Value Benefits 15.61 113.53 291.21 Total 39.17 10.94 - 43.76 45.57

433.41 999.74 2,334.36

1274.65 KEY

250.18 Q1 03/04 - Q4

05/06 2006/07 2007/08

12.90% 2003/04 2003-05 2003-06 2003-07 2003-08Benefits as %

of Cost Realised Plan Actual57.09 90.37 142.59 183.78 202.01 Efficiency Cashable - 0.53 1.63 5.21 32.69 85.10 Efficiency Opportunity 0.11 0.05 0.21 0.37

- - - - Effectiveness - 0.53 1.63 5.21 32.69 85.10 42% Total 0.11 0.05 - 0.21 0.37

111.21 194.71 269.22 408.09 519.61 5.50 15.38 39.29 83.26 129.20 - 8.55 10.12 19.22 49.96 Q3 05/06 Q4 05/06

5.50 23.93 49.40 102.47 179.16 34% No. % No. %25.05 58.82 97.87 132.70 170.63 6 6% 2 2%

- 2.05 31.53 68.59 110.46 26 27% 35 36%- 2.05 31.53 68.59 110.46 65% 13 13% 19 20%

154.55 283.61 426.40 512.40 585.28 52 54% 41 42%- 0.30 2.59 8.50 32.07

3.23 8.43 18.80 32.10 47.28 3.23 8.73 21.39 40.60 79.35 14%

3Ring Fence costs from 2003-06 and Delivery Plan RF budget from 2006-08. Full benefits by recipient used. Corrections includes YJB.

RAG

1

2

3

Summary of Key Mitigating Actions

1. Settlement Letter Conditions/Hurdle rates 2. Root Cause Model 3. Social Value Research 4. Analysis of benefits enabled by CJS IT funded infrastructure

1. Quarterly Benefits Integrity Check 2. Benefits Eligibility Framework

Risk Description

2006/07 Q1

Difficulty with tracking/measure.

Benefits Rating

Prog

RAG of Benefits in SR2004 Benefits Realisation Plans

2006/07 Q1

YJB Benefits Realisation Plan

Direct Benefits

2006/07 Q1

HMCS Benefits Realisation Plan

2006/07 Q1

Inf

rast

ruct

ure

Scale of CJS IT benefits forecast

1. Process Modelling 2. CJO Benefit Realisation Plans approved by OB and BWG 3. Project Benefit Realisation Plans approved by BWG CJSIT benefits realisation

Total 10 year CJS IT Application benefits Ring Fence only

CJS IT Application NPV Ring Fence only

CJS IT Application IRR Ring Fence only Cost Benefit Analysis3

Quality of CJS IT benefits forecast

Risk Register - CJS IT Benefits Management

CJS IT BENEFITS SCORECARD June 2006This table shows the full CJS IT Programme benefits picture, including ring fence benefits and benefits enabled by the ring fence infrastructure (£m)

App

licat

ions

CJS IT Benefits Realisation Plans

CJS IT Projects

Forecast Benefit Values Self assessment of RAG status Police Benefits Realisation Plan

TOTAL BENEFITS

Total CPS RF Cost/BudgetDirect & Enabled BenefitsTotal CPS Benefits

Total Corrections RF Cost/BudgetDirect BenefitsEnabled BenefitsTotal Corrections Benefits

Total Police RF Cost/Budget

Enabled BenefitsTotal Police Benefits

Total HMCS Benefits

Total HMCS RF Cost/BudgetDirect BenefitsEnabled Benefits

Forecast

2. Independent Scores assessed by Proving Services

Benefit on track/ahead of scheduleBenefit not yet due for realisation

Benefit behind schedule

NOTE: Benefits shown only include quantified, validated benefits but other enabled benefits have been identified

and will be included as further work is undertaken

Forecast

Forecast

Forecast

Forecast

CPS Benefits Realisation Plan

2006/07 Q1

NOMS Benefits Realisation Plan

1. Based on BRP submissions & agreed with Strand Board leads, this shows the number of benefits to the Strands in terms of value (ie. mission critical, highly desirable etc)

MC = Mission Critical, HD = Highly Desirable, D = Desirable, M = Minimal, NK = Not Known

Strand Board Alignment1

Relative Contribution to

Strategic Drivers2

Benefits to the CJS and Society

Home CJO, 76%

x-CJS, 9% Social Value, 16%

Home CJO x-CJS Social Value

Benefits By Type

Efficiency Opportunity

61%

Efficiency Cashable

9%

Effectiveness30%

Efficiency Cashable Efficiency Opportunity Effectiveness

ATTRACTIVENESS ANALYSIS ECONOMIC ANALYSIS* NPV (£M) IRR (%) PAY BACK PERIOD

(YEARS) EXCLUDING OPTIMISM BIAS

INCLUDING OPTIMISM BIAS

CONFIRMATION THAT THE ABOVE FIGURES ARE COMPLIANT WITH BENEFITS ELIGIBILITY FRAMEWORK (BEF) Yes No

CONFIRMED BY: COMMENTS:

BENEFITS ANALYSIS OVER 10 YEAR PROJECT LIFE CYCLE (£M) STATE 10 YEAR PERIOD COVERED OR PROJECT LIFE SPAN IF LESS THEN 10 YEARS: RECIPIENT EFFICIENCY –

CASHABLE EFFICIENCY – OPPORTUNITY VALUE

EFFECTIVENESS - CASHABLE

EFFECTIVENESS – OPPORTUNITY VALUE

CJO: Police CJO: CPS CJO: DCA CJO: NOMS CJO: YJB CROSS CJS: BEYOND THE CJS(SPECIFY):

TOTALS:

INC

LUD

ED

IN

BR

P

AG

REE

D IN

PR

INC

IPLE

N

OT

YET

AG

REE

D

SOURCE OF CONFIRMATION OR PROPOSED ACTIONS & TIMESCALES TO ENSURE BENEFITS ARE INCLUDED IN BENEFITS REALISATION PLAN

BENEFITS INCLUDED IN THE POLICE BENEFITS REALISATION PLAN (BRP) OR AGREED IN PRINCIPLE BY THE POLICE BENEFITS REALISATION LEAD(BRL)

BENEFITS INCLUDED IN THE CPS BRP OR AGREED IN PRINCIPLE BY THE CPS BRL

BENEFITS INCLUDED IN THE DCA BRP OR AGREED IN PRINCIPLE BY THE DCA BRL

BENEFITS INCLUDED IN THE NOMS BRP OR AGREED IN PRINCIPLE BY THE NOMS BRL

BENEFITS INCLUDED IN THE YJB BRP OR AGREED IN PRINCIPLE BY THE YJB BRL

ACHIEVABILITY ANALYSIS

PROVING SERVICES ACHIEVABILITY SCORE BREAKDOWN CONCEPT CASE (COMPLETE AS APPROPRIATE)

Stakeholder Analysis (Achievability)Indication Achievability

Complexity Analysis

Score

Achievability Total

BUSINESS CASE (COMPLETE AS APPROPRIATE)

Complexity AnalysisProcesses & Capability

Ownership & AccountabilityClarity & Perception

Score

Achievability Total

Benefits Realisation ManagementStakeholders Analysis

ASSESSMENT OF DEGREE OF BUSINESS CHANGE REQUIRED TO REALISE BENEFITS RECIPIENT BRL ASSESSMENT COMMENTS

HIG

H

MED

IUM

LO

W

N/A

Police CPS DCA NOMS YJB Others [specify]

PROVING MODEL ASSESSMENT DATE OF CURRENT ASSESSMENT: ACHIEVABILITY SCORE: PROGRESS SINCE LAST ACHIEVABILITY ASSESSMENT:

(1) TOTAL PROJECT COSTS YEAR RESOURCE

(£M) CAPITAL

(£M) TOTAL (£M)

TOTAL

(2) RESOURCE/CAPITAL CURRENTLY ALLOCATED FROM THE RING FENCE

(3) OTHER FUNDING SOURCES

YEAR RESOURCE (£M)

CAPITAL (£M)

TOTAL (£M)

RESOURCE (£M)

CAPITAL (£M)

TOTAL (£M)

TOTAL

RESOURCE/CAPITAL GAP (1-(2+3)) RESOURCE/CAPITAL REQUIRED FROM THE RING FENCE

YEAR RESOURCE (£M)

CAPITAL (£M)

TOTAL (£M)

RESOURCE (£M)

CAPITAL (£M)

TOTAL (£M)

TOTAL

FUNDING – OTHER ISSUES AVAILABILITY CONFIRMED? FUNDING AVAILABLE TO REALISE BUSINESS CHANGE? Yes

No BY: DATE

CONFIRMATION THAT OVERSPENDS WILL BE MET BY DEPARTMENTAL BASELINES

Yes No

BY: DATE

RECYCLING OF COST SAVINGS YEAR LEGACY SYSTEM

SAVINGS SAVINGS

RECYCLED? AMOUNT BY WHICH THE RING FENCE

REQUEST CAN BE REDUCED TOTAL

• Recipient Organisation Benefits Realisation Report

• Contributor project benefits report

RecipientCost Avoidance

Self Assessment Cashable Opportunity Cashable Opportunity £ %Crown Courts 12.700 35.620 0.000 25.920 0.000 74.240 6.7%

Quality of Benefits Forecast Other CJOs 0.000 117.820 0.000 52.760 0.000 170.580 15.3%Scale of Benefits Forecast Cross CJS 0.000 0.000 0.000 173.710 0.000 173.710 15.6%Quality of Realisation Planning Outside CJS 0.000 0.000 0.000 694.860 0.000 694.860 62.4%Likelihood of Realisation Total £ 12.700 153.440 0.000 947.250 0.000 1113.390 100.0%

Benefits Realised this financial year £m Last report NowMonetary Value of Benefits Realised YTD 0.01 NPV £m* 154.301 -

4.10.0%

Variance -4.09

Performance £ M

From 1 month after Go-live at each court centre

Reduced waiting time claim valuesBenefit not yet due for realisation10 Reduction in solicitor waiting - 12.7 Efficiency cashable

PSA Targets 2, 3 and 5

Crown Court

Following rollout to all prisons in a CJA

Benefit not yet due for realisation9 NOMS Benefits - 36.9 Effectiveness opportunity

PSA Target 5

NOMS

Benefit ahead of or in line with plan10

Benefit ahead of or in line with plan

No of bail forms sent and time taken

9

3 months after go live in a CJ Area

No of enquiries made to CCBenefit ahead of or in line with plan

No of daily/warned lists sent and time taken

8 Police Benefits - 115.52 Efficiency opportunity

PSA Target 5

Police

3 months after go live in a CJ Area

Number of enquiries made to CCBenefit ahead of or in line with plan

No of daily lists faxed/time taken.

7 CPS Benefits - 2.3 Efficiency opportunity

PSA Target 5

CPS

From 1 month after Go-live at each court centre

Backlog countBenefit ahead of or in line with plan

Number of disposals

6 Crown Court Efficiency benefits - 35.61 Efficiency opportunity

PSA Targets 2, 3 and 5

Crown Court

From 3 months after Go-live at each court centre

Number of disposalsBenefit ahead of or in line with plan5 Court time saving (Same day delays) - 5.98 Effectiveness opportunity

PSA Targets 2, 3 and 5

Crown Court

From 6 months after Go-live at each court centre

Percentage of ineffective hearingsBenefit not yet due for realisation4 Court time saving ( Ineffective Hearings)

Assumption of improvement of 1-10% 31.78 Effectiveness opportunity

PSA Targets 1, 2, 3 and 5 Crown Court, Police, CPS, Probation

From Go-live at each court centre

No. of adjournments as a result of no PSRBenefit ahead of or in line with plan3 Court time saving (PSR adjournments)

Improvement from 15% to 3.4% 4.06 Effectiveness opportunity

PSA Targets 2, 3 and 5

Crown Court

To start 6 months after completion in a CJ Area

Time from CC result to PNC update

2 Reduction in the cost of crime

PNC update down to 3 days 694.86 Effectiveness opportunity

PSA Targets 1 and 3

UK economy

To start 6 months after completion in a CJ Area

Time from CC result to PNC updateBenefit realisation behind Plan

Realisation Ramp Up Key Measures/Indicators of Benefit

Benefits Realisation

Status

1 Reduction in the cost of crime

PNC update down to 3 days 173.71 Effectiveness opportunity

PSA Target 5

CJS

94.2*NPV discounted at 3.5%

Project Life Cycle Benefits Profile (Top 5-10 benefits over 10 years)

Benefit Description

Impact

Benefit Category Strategic AlignmentMain Recipient

Name(s) of recipient agreed with

Q4 Assessment

Q4 forecast for Q1

Forecast in latest published Delivery Plan YTD IRR %

Approved by SRO:

Benefit not yet due for realisation

CJS IT BENEFITS PROGRESS REPORT - QUARTER 4 2004-05 FOR CJS EXCHANGE XHIBIT PORTAL

10 Year Analysis

Efficiency £ M Effectiveness Total

Q3 Assessment

Q3 Forecast for Q4

Cumulative Cost and Benefit Forecast 2003-2013

-

200

400

600

800

1,000

1,200

2003-0

4

2004-0

5

2005-0

6

2006-0

7

2007

-08

2008

-09

2009

-10

2010

-11

2011

-12

2012

-13

Year

£'m

Cumulative Benefits Cumulative Costs

And remember…

1. Spend more time doing your homework.2. Be clear about what benefits you are buying.3. Track performance – to learn and to inform

investment decision.4. Triangulate and Validate – ‘trust everyone, but cut the

deck’.5. Use ‘Gates with teeth’. 6. Joint accountability - planning for success, with a

forward looking perspective.7. Use summary documentation – size is the enemy of

understanding.8. Ensure there is a ‘clear line of sight’ from strategic

intent through to benefits realisation.9. Manage benefits from an enterprise perspective.10. Dare to employ a ‘fool’!

Want to find out more?

Available from www.academic-publishing.org