1 Session 3: Benefits Definition and Measurement v4.0 May 2009 eHealth Benefits Management Toolkit.

1

Session 3:

Benefits Definition and Measurement

v4.0May 2009

eHealth Benefits Management Toolkit

2 Benefits Management

Session 3 Outline

Benefits measurement categorisation

Benefits profile

template

Benefits Realisation

Plan

Transition to business

as usual


At the end of the session you will have an understanding of:

• what methods can be used to define quantifiable benefits and estimate their value;

• how to finalise detailed benefit definitions using SMARTT and ensure the benefits are measurable;

• how to pull all the outputs of the previous sessions together to create the Benefits Realisation Plan; and

• what further activities need to take place at the end of the project.

Session 3 purpose


Benefits Measurement Categorisation


Benefits profile

template


Plan


as usual


Specific Definitions of Benefits

Benefits should be:• clearly and succinctly defined• aligned with the projects objectives and the overall NHS

strategy• focused on improving service and patient care, not focused on

IT

Effective measurement and monitoring will:• indicate the extent to which benefits are being realised;• give early warning of potential problems;• create the opportunity to adapt the benefits or changes to

enable the overall objectives; and• ensure that achieved benefits are measured, reported and

communicated.


Measurable Outcomes

The next step is to:

• devise measures that will show when the benefits have been delivered and to what degree;

• determine which benefits can be quantified;

• determine the financial value of benefits where possible;

• investigate additional data capture activities required; and

• establish baseline, targets and benchmarking figures.


Defining Measures

• Use existing measures where possible• Agree all new measures with those concerned• Ask yourself:

– Is it achievable? – Is it an appropriate and valid indication that the benefit has been

delivered? – Is it directly attributable to a particular change in the project?

• Consider the effort and cost of capturing and collating the data• Decide who will be responsible for ensuring that the data is available• Ensure the measures encourage the right type of behaviour• Define when and how often the measurements should be taken


Benefits Categorisation(JW)

Degree of explicitness of measures

Financial By applying a cost, price or other valid financial formula to a quantifiable benefit, a financial value can be calculated.

Quantifiable Sufficient evidence exists to forecast how much improvement / benefit should result from the changes.

Measurable This aspect of performance is currently being measured or an appropriate measure could be implemented. But it is not possible to estimate by how much performance will improve.

Observable By use of agreed criteria, specific individuals or groups will decide to what extent the benefits has been realised, based on their experience and judgement.


Non Quantifiable Benefits

Observable benefits • lowest category of benefits in term of robust measuring• often the softer / people benefits• can help get buy-in essential to the implementation of changes and the

realisation of their resulting benefits• agree realisation criteria and who will say whether they are met

Measurable benefits • performance is easily measured; but• potential improvement won’t be known until after the event • take baseline measurements before any changes are made for

comparison


Quantifiable Benefits

Quantifiable benefits

• aspect of performance easily and consistently measured

• improvement can be predicted and targets defined

• usually applicable to embedded processes where performance is well documented and predictable

Financial benefits

• highest category of benefits in term of robust measuring• focus on benefits that can easily be translated in terms of financial gain

or reduced cost: if it is too complicated or not robust don’t assign a monetary value to it

• don’t confuse value with savings


Progressing Non Quantifiable Measures to Quantifiable Measures

The following methods can be used to get from measurable to quantifiable measures:

• Modelling or simulation: extrapolate expected benefits from existing data. It may be necessary to establish trends/times series.

• Benchmarking: evaluate changes in relation to “best practices” in the comparable organisations.

• References sites: can also be used to establish a comparative base but assess relevance and feasibility in your own organisation.

• Pilots: test the new way of working and system on a small scale.


Examples for Benefits Template

Category Benefit Descriptions and Numbers Measure Value / Improvement Level

Financial e.g. “increase utilisation of theatres resources by allowing patient-cancelled appointments to be filled at short notice from reserve list” expected increase in utilisation = 1 in 3 cancellations filled = 200 slots = £1000 per slot = £200,000

cancellations filled x resource value (£1,000)

£200,000

Quantifiable e.g. “elimination of referrals from SCI gateway being printed onto paper for transfer to secondary care system” number of referrals, and therefore those being printed, known before implementation; integration will eliminate these

% of referrals processed through new system

100%

Measurable e.g. “reduce unnecessary x-rays by avoiding repeat request” not known how many x-ray requests are repeats, but assumption is that these will be eliminated, reducing overall number of x-ray requests

number of x-ray requests

N/A

Observable e.g. “reduce staff frustration due to need for double data entry” staff satisfaction survey with regard to duplicate data entry

% of satisfied or very satisfied

N/A


Benefits Profile Template


Benefits profile

template


Plan


as usual


SMARTT BenefitsDetailed benefit descriptions contain the following categories:

Benefit Characteristic

Definition Example

SpecificDescription of the benefits Number of re-xray forms completed.

Quality dimensions enabled Safety, efficiency, timeliness

Measureable Variable to be evaluated Number of forms filled in per month

Achieveable

How high a priority is the benefit?

Scored on a scale of 1-5 (where 5 is the highest priority)

How likely is the benefit to be realised?

Scored on a scale of 1-5 (where 5 is the highest likelihood)

RealisticWhat is the benefit dependent on?

Strategic direction, process, information management, skills/ behaviours, organisation/ roles, infrastructure/ physical environment, technology integration

Time-bound

Realisation timeframe E.G. Measure benefit from 3 months pre-go-live until 1 year post go-live

Measurement interval recommendation

E.G. Per minute, per hour, per week, etc.

Frequency of measurements

E.G. One week out of each month, one month out of each year, etc.

TargetedBenchmark E.G. 20 forms/month

Target E.G. 20% reduction


Filling in the Benefit Profiles

ACTION:Expand on the benefits information from the previous workshops in the template for each priority benefit identified:

•Utilise measurements that are already being used by the Health Board where possible•Take into account pre-existing government targets of efficiency estimates when setting benchmarks and targets for benefits•Ensure that dependencies are taken into account when setting realisation timelines

Benefit IDEnter ID number from BDN

Benefit descriptionEnter benefit description

Benefit ownerEnter the name of the person or group who will receive the benefit

Benefit sponsorEnter name and role of the person who will ensure that the benefit is achieved and instigate remedial actions if required

SafeTick if relevant

EffectiveTick if relevant

TimelyTick if relevant

EfficientTick if relevant

Patient-centredTick if relevant

EquitableTick if relevant

Measure/Variable to be evaluatedEnter description of measure to be collected

Benefit explicitnessEnter category: financial (high), quantifiable, measurable, observable (low)

Value/Level of improvementOnly applicable for quantifiable and financial benefits

ImpactHow important is the benefit? Scale of 1 (low) to 5 (high)

LikelihoodHow likely is the benefit to be realised? 1 (low) to 5 (high)

Strategic directionEnter ID and description of objective(s) the benefit contributes to

Changes requiredEnter ID and owner of required changes (see change profile for details)

IM&T requiredEnter ID and description of required IM&T capabilities

Benefit realisation dateEnter date the benefit will be realised (or dates and degree of realisation for staged roll-out)

Measurement timeframeEnter start and end dates of monitoring and reporting

Measure intervalEnter description of the period of time the measurement covers

BaselineEnter value, source and date if applicable

Benchmark or targetEnter value, source and date if applicable


Preparing to Measure

• Are there pre-set requirements for benefits realisation timelines?

• Who is going to be collecting the measurements?

• How will the measurement be collected? (data sources)

• Who will oversee the benefits collection?

• Where will the results be reported to?

• Who will conduct benefits reviews?

• Who will be responsible for communicating about the benefits side of the project?


Preparing for Benefits Monitoring

In order that the benefits monitoring is as productive as possible:

• define a robust mechanism to monitor progress and ensure that reports are distributed to the appropriate people;

• ensure that the Benefits Management templates are up to date;• hold regular benefits reviews to track the progress of benefits realisation

against the project and programme objectives;• use benefits reviews as an opportunity to highlight areas where benefits

are not being achieved, and identify the barriers to achieving them;• for those benefits that are under-realised, refer to the dependencies

shown in the BDN for potential root causes;• assign responsibilities for taking and coordinating corrective actions

where benefits are not being met; and• identify and communicate successes too!


Benefits Realisation Plan


Benefits profile

template


Plan


as usual


Formalising it

The Benefit Realisation Plan provides evidence to:

• Justify investment decisions• Prioritise competing work in context of finite resources• Establish clear ownership / governance arrangements• Realise benefits• Demonstrate value and effective use of resources

Store the Benefit Realisation Plan with the other key project management work.


Benefits Realisation Plan (BRP)

When drafting the Benefit Realisation Plan remember to:

• Consider the dependencies identified between benefits, changes and IS&T capabilities;

• Identify timeline and milestones that take those dependencies into account;

• Work with stakeholders towards minimising the negative impact of identified disbenefits; and

• Extend the plan beyond the system implementation date (often viewed as the end of the project) to schedule regular reviews of benefits realised and consider potential further benefits.


BRP Content

The Benefit Realisation Plan should include:

• the completed Benefits Dependency Network (BDN)• the completed benefit profiles, including dates for capture of baseline

figures at an early stage• the completed change templates• the completed stakeholders analysis

Prioritise the benefits so that the most important always has the most focus. This ensures that the project makes the greatest impact.

Identify dates for expected delivery of the benefits.


Benefits Strategy

A benefits strategy should also be prepared for the business case, including:

• the rationale linking the drivers, objectives and benefits• defined roles and responsibilities• a mechanism for regular measurements and reporting• a mechanism for regular reviews and corrective action• arrangements for the transfer of responsibilities at close of project


Providing all this, the business case will demonstrate that:• benefits outweigh disbenefits;• a shared vision is held which is strategically aligned;• the BDN clearly shows how the shared vision will be achieved;• the change required, when considered in relation to the overall schedule

of change planned, is not overwhelming for any stakeholders; and,• the benefits realised will be worth the effort required to achieve them.

It will also show that:• there is a clear process for identifying, monitoring and realising the

benefits; and• the baseline benefits position has been recorded to enable comparison

with projected targets for monitoring the achievement of benefits.

Linking Business Case & Benefits Management


Transition to Business as Usual


Benefits profile

template


Plan

Transition to business as usual


Close of project activities

Following full implementation of information systems, the achievement of the business case should be reviewed.

The review will produce 3 reports:

• formal end-of-project review

• lessons learned

• potential further benefits


End of Project Review

The end of project review will:

• assess to what extend the project was successful in delivering the benefits;

• decide what further actions are required, by whom and when they should be reviewed; and

• detail handover arrangements for any tasks outstanding (including benefits monitoring and reporting).

The Benefits Plan should be updated as relevant to ensure a reference document is available to handover to the “business as usual” team.


Lessons Learned

The purpose of the “lessons learned report” is to:

• reflect on the lessons learned for future investments;• disseminate useful lessons learned during the project for the benefit of

other projects, internal and external;• provide a useful control as part of the functions of an independent quality

assurance group;• consider how much effort was required to complete the benefits

management documentation and monitor the benefits; and• make recommendations for future enhancement or modification of the

benefits management method.


Further Benefits

The Review Team should also consider and communicate:

• what opportunities exist for the identification and realisation of additional benefits; and

• who will take the business case for them forward.

Additional benefits should be defined, planned and implemented following the same methodology as the initial project benefits.


Transition to Business As Usual

• Leave no loose ends: each outstanding activity needs to be allocated to a named individual

• Have end dates for decommissioning of legacy systems to ensure the new IM&T are used – this will ensure new processes are used too and related benefits are delivered

• Monitor the implementation of change over a period: – people may revert to the “old ways” once the initial enthusiasm has

worn off– people may find work-arounds where the technology or new

processes are not working instead of reporting the problems– address teething problems


Summary of Benefits Management LifecycleProject lifecycle Benefits management lifecycle Outputs

Projectinitiation

Identify and structure benefits Benefits Dependency Network (BDN)Identify change activities required to realize the

benefits

Prepare benefit plan and business case Benefit profiles Change profiles

Stakeholder analysis

Run first benefits analysis Baseline data

Delivery management

Use benefit plan to inform project management, system development and change management

activities

Regularly review project progress against benefits analysis

Updated figures

Make adjustment to system capabilities, business changes or benefit definitions

as required

Amended templates and BDN

CloseRun end of project benefits analysis (formal

review)Formal report to

Board

Business as usual

Identify need for further reviews and potential for further benefits


References and Further Reading

• The Clinical Indicators Team has produced a presentation on statistical process control (SPC):

http://www.indicators.scot.nhs.uk/Workshops/SPC.html

• HM Treasury’s “Green Book” provides examples of methods used to estimate the financial value of activities or resources:

http://www.hm-treasury.gov.uk/d/green_book_complete.pdf

• No Delays Scotland have devised a methodology for measuring benefits:

http://www.nodelaysscotland.scot.nhs.uk/ServiceImprovement/Tools/Pages/IT211_Methodology_for_measuring_benefits.aspx

• Statistics for NHS England and Wales for benchmarking on:

http://www.ic.nhs.uk/statistics-and-data-collections (cont’d)


References and Further Reading (cont’d)

• Benchmark statistics (Scotland) are available from ISD:

http://www.isdscotland.org/isd/782.html

• Quarterly and annual figures on patient safety are published by NPSA:

http://www.npsa.nhs.uk/nrls/patient-safety-incident-data/quarterly-data-reports/

• The Department of Health publishes annual and quarterly healthcare performance statistics:

http://www.dh.gov.uk/en/Publicationsandstatistics/Statistics/Performancedataandstatistics/index.htm

• Population numbers by NHS Boards – births, deaths, totals, by age, by gender – are available from GRO:

http://www.gro-scotland.gov.uk/statistics/publications-and-data/population-estimates/index.html


Congratulations on completing the benefits

management programme and on all your efforts

to ensure that maximum benefit is gained from

eHealth Systems.

1 Session 3: Benefits Definition and Measurement v4.0 May 2009 eHealth Benefits Management Toolkit.

Documents

expected benefits

resulting benefits

benefits definition

financial value of benefits

quantifiable measures

measures financial

usual slide

comparison slide