Risk based cost estimating for water infrastructure projects

www.advisian.com

Michael Quinnell – Senior Associate, Advisian

November 2016

Risk-based cost estimating for water

infrastructure projects – does it stack up?

Inaccurate cost estimates can lead to suboptimal project selection, inefficient budget management and disruption to the delivery of other essential projects.

To overcome these issues and improve the management of their investment programs, a number of NSW water authorities have adopted a stochastic approach to estimating: Risk-based cost estimating (RBCE).

Risk-based Cost Estimating

What is it?A stochastic process utilising Monte-Carlo Simulation modelling to reflect the degree of uncertainty (risk and opportunity) associated with a cost estimate.

Base estimateInherent risk:Uncertainty within base estimate, defined assumptions and known scope

Contingent risk:Uncertainty from unplanned events

Risk workshop

Monte Carlo Simulation

Sample output of RBCE process

BASE COST ESTIMATE

COST

BASE RISK

CONTINGENCY RANGE

CUMULATIVE PROBABILITY

FORECAST EXPECTED COST

(MEAN OR MEDIAN [P50])

MAKE FINANCIAL PROVISION FOR WORSE COST

OUTCOME

RESIDUAL RISK EXPOSURE

RANGE20%

40%

60%

80%

100%

PROB

ABIL

ITY

(CUM

ULAT

IVE)

Why use the RBCE process?

1.Identify, quantify and incorporate risks and opportunities into the project budget

2.Incorporate project knowledge and experience from a wide stakeholder group through the risk workshop process

3.Provides organisations the information required for decision makers to set project budgets that match the risk tolerance

4.A more sophisticated approach to allocating appropriate project contingency

Understanding the performance of the RBCE

process

water infrastructure projects studied

2002 – 2012range in total actual cost values

23 $4.2m to $191.7m

Projects delivered by several water authorities in NSW over the period

Study objectives:Obtain a better understanding of the performance of the RBCE process

Identify areas where the process can be further improved in the future

1.2.

Data collected

Final actual project cost

Actual cost incurred by the water authority at

completion of the project; includes initial contract

price, any variations paid and client costs

Data collected from each project included:

Project timing data

Date when the estimate was prepared, contract

award date and the date of practical completion

Original cost estimate

Prepared at project appraisal stage at the time

where the decision was made to go ahead with the

project

Data AdjustmentsAdjustments were made to ensure a ‘like-for-like’ comparison between budget and actual project cost. These included:

Escalation adjustment to account for the difference between base period of

estimate and actual project delivery timeframe

Increased or decreased scope change

So… does it stack up?

Findings 1: How accurateare RBCE budgets?• The water projects in the studies

sample performed well compared to other studies

• Cost underruns were found to occur more frequently than overruns

• Weighted average of projects under budget was 8.5%

• Out of the 23 projects:

• 17% of projects experienced cost overruns

• 83% of projects were on or under budget

Difference (%) =

Actual Project Cost (adjusted to ensure consistent comparison)

Project Budget (raw)X 100

% overrun/underrun vs. Estimate date

20%

10%

0%

-10%

-20%

-30%

-40%

-50%A B C D E F G H I J K L M N O P Q R S T U V W%

ove

rrun/

unde

rrun

vs. E

stim

ate

date

Projects sorted by estimate preparation date

Older Newer

Findings 2: RBCE success in forecasting outcomesThe main output from the RBCE process is not a single estimate; it is a range of potential outcomes.

• The range of outcomes were between the P1 to P99 value

• Only 57% of the sample fell within the expected range

Possible causes:

• Project Managers ‘anchor’ on the base estimate (i.e. they’re not willing to adjust or expand the range to capture the best and worst case scenario)

• Results in the forecasted range are too narrow to capture the true level of uncertainty

50%57% of actual project costs were within the

expected range

CostPr

obab

ility

(cum

ulat

ive)

100%

“It was expected that 98% of projects would fall within the defined

range”

Findings 3: Reliability of confidence from RBCEA key feature of the RBCE process is that it provides a level of confidence for each value in the range of potential project outcomes.

• Between P50 and P90, confidence levels in the process were reliable

However…

120%

100%

80%

60%

40%

20%

0%P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90

Well calibrated

results (i.e. accurate representation of p-value)

P Value

% o

f pro

ject

at o

r bel

ow b

udge

t (un

derru

n)

Expected % Underrun

Actual % Underrun

Findings 3: Reliability of confidence from RBCEBelow the P50 and above the P90 level there was a wide difference between the expected and the actual result.

This suggests:

• The process isn’t placing enough focus on identifying opportunities

• The process isn’t capturing the ‘extreme risks’ (low likelihood and/or very high consequence)

A possible explanation could be that it

may not be worth the effort of accurately estimating figures outside of the P50 and P90 range (which are used the most often).

120%

100%

80%

60%

40%

20%

0%P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90

Extreme risks not adequately considered

P Value

% o

f pro

ject

at o

r bel

ow b

udge

t (un

derru

n)

Expected % Underrun

Actual % Underrun

Results indicate value <P50 too conservative

Findings 4: The Portfolio PerspectivePublic agencies delivering projects face the dual challenge of neither exceeding their allocated funding nor significantly underspending the allocated funding.

• Results highlighted an opportunity for more efficient contingency allocation

• Agencies could set a less conservative P-value as the budget and reduce the portfolio cost underrun

• Means accepting that a greater number of projects will overrun vs individual project budgets, but the overall portfolio is sufficient to deliver all projects

• Provides ability to invest this released contingency on other valuable projects or agency priorities

% deviation from budgetTotal portfolio actual costTotal portfolio budget

P01 P99P10 P20 P30 P40 P50 P60 P70 P80 P90

1,400

1,200

1000

800

600

400

200

-

10.00%

5.00%

0.00%

-5.00%

-10.00%

-15.00

Milli

ons

% d

evia

tion

from

bud

get

Had the program budget been set at P50, the percentage

underrun would have reduced from 8.5% under the budget to

4%

Areas for improvement

Issues relating to cognitive bias or delusions - such as anchoring and adjustment - affected the results of the process.

This was particularly highlighted by the limitations to fully capture the range of expected outcomes and in producing reliable confidence levels below the P50 and above the P90 levels.

This can be mitigated by placing more emphasis on identifying key opportunities rather than only focusing on risks, as well as calibration techniques and training.

1. 2. 3.

DISCLAIMERThis presentation has been prepared by a representative of Advisian.The presentation contains the professional and personal opinions of the presenter, which are given in good faith. As such, opinions presented herein may not always necessarily reflect the position of Advisian as a whole, its officers or executive.Any forward-looking statements included in this presentation will involve subjective judgment and analysis and are subject to uncertainties, risks and contingencies—many of which are outside the control of, and may be unknown to, Advisian. Advisian and all associated entities and representatives make no representation or warranty as to the accuracy, reliability or completeness of information in this document and do not take responsibility for updating any information or correcting any error or omission that may become apparent after this document has been issued.To the extent permitted by law, Advisian and its officers, employees, related bodies and agents disclaim all liability—direct, indirect or consequential (and whether or not arising out of the negligence, default or lack of care of Advisian and/or any of its agents)—for any loss or damage suffered by a recipient or other persons arising out of, or in connection with, any use or reliance on this presentation or information.