Good Practices and Common Mistakes - EpiX Analytics · Preventing common mistakes: 1. Multiply your efforts when multiplying 2. Software is no substitute for thinking… 3. Value

www.epixanalytics.com

Good Practices and Common Mistakes

Huybert Groenendaal, PhD, MBA

EpiX Analytics

© EpiX Analytics LLC

� Financial industry

� Health / Food safety

� Energy, oil & gas

� Many others….

Risk analysis: many applications

1. EpiX Analytics specialized risk analysis and modeling company

2. Focus: Quantitative risk analysis & modeling

Goal: Improving decision-making

Services: Consulting, training & research

3. Experience in a wide range of fields:� Pharma

� Mining

� Manufacturing

� Transportation

� Insurance


• Cost and schedule estimation

• Asset allocation

• Sales Forecasting

• Financial Analysis (NPV)

• Anti-microbial risk analysis

• Portfolio forecasting

• Pricing decisions

• Commission Estimation

• Reliability

• Inventory Optimization

• Evaluating of mining prospects

• Mine production forecasting

• Import risk analysis

• Electricity Price Forecasting

• R&D portfolio optimization

• Business Development Deal

Structuring

• Pharmacoeconomics

• Health and Epidemiology

• Budgeting

• Financial options

• Venture capital

• Hedging & VaR calculation

• Early drug development

• Many more…..


Monte Carlo & @RISK:Many applications and case-studies

Free Monte Carlo and @RISK training and reference tool:

http://www.epixanalytics.com/ModelAssist.html

� $500 Risk Analysis Training tool

� Based on 20+ years of risk analysis consulting and training expertise

� Within ModelAssist, page numbers are Mxxxx. For example, M0407 will get you to the “Selecting the appropriate distributions for your model”


ModelAssist for @RISK

So, now you are an @RISK user, but…

� What is important?

How do I start and what should I focus on?

� Do I maybe make mistake?

What should I look out for?

Goal of this seminar:

Share important good practices and common

mistakes (and how to prevent them)

Goal of this seminar:

Share important good practices and common

mistakes (and how to prevent them)


Preventing common mistakes:

1. Multiply your efforts when multiplying

2. Software is no substitute for critical thinking…

3. Value your relationships!

Good practices:

1. Start with a ‘blue-print’

2. Check/verify your model!

3. Educate and engage your executives!

Good practices and common mistakesQuick overview


� What is the mistake?

Using probability distributions as if they were fixed

numbers (e.g. multiplying, subtracting) M0089

� A simple example:

� Bank offering loans

� Number of clients taking a loan uncertain

� Size of the loan vary quite widely

� What will be total annual interest revenue?

Multiply your efforts when multiplying Preventing common mistake #1






� Some other examples:1. Forecasting demand on services team (e.g. hours) based

on number of demands and length of calls in order to forecast and optimizing hiring needs;

2. Total exposure to credit risk, operational risks, etc.

3. Total costs of building 5 pumping stations

when building a 100-mile natural gas pipeline

4. Total amount of pig meat eaten by families who own pigs themselves


Case Study: Large Oil and Gas companyIncorrect multiplying


� Company developed

Monte Carlo model to

evaluate alternative

portfolios of natural gas

wells

� Incorrectly used

multiplication of

distribution and as a

result the company

greatly overestimated the

risk in its portfolio

Background:

� Corrected the Monte Carlo model using Central Limit Theory (CLT)

� Included efficient use of Stochastic Optimization (i.e. OptQuest)

� Company now uses Monte Carlo and Stochastic Optimization to optimize its use of resource (budget, manpower, potential wells)

Solution:


Rely too much on Monte Carlo software without

thinking critically

� A simple example:

� You have some data on historical

annual returns of some global real estimate fund;

� Open up @RISK, click “Fit to data” M0198

� ‘Automatically’ select the distribution that has the highest

GOF statistic M0211

Software is no substitute for thinking…Preventing common mistake #2


-4 -2 0 2 4 6 8

10

12

14

Software is no substitute for thinking… Preventing common mistake #2

� Some other examples:

� ‘Batch fitting’ distributions to data

(Does fitting a parametric distribution even make sense?)

� Using rank-order correlations because it is the only

correlation pattern that is explicitly including within @RISK;

� Using a GARCH (General AutoRegressive Conditional

Heteroskedasticity ) time-series to forecast sales based on

sales of last 5 years



Omit including relationships within Monte Carlo

models

Important rule: Every iteration (all 10,000) has to be a

possible future scenario

� Quick examples:

� Forecasting investment portfolio in multiple assets;

� Financial analysis of new product with uncertain price,

demand, level of competition, etc.

� Evaluating project cost and schedule;

Value your relationships!Preventing common mistake #3


Value your relationships! Preventing common mistake #3

� Important:

� Omitting to include relationships typically results in the

under-estimation of risk;

� Many @RISK user only use the rank-order correlation, but

there are many alternative methods to include relationships:

� Envelop method M0146

� Lookup tables; M0275

� Logical relationships (IF-statements etc.) M0097

� The bootstrap M0444

� Relationship based on a regression (linear or non-linear)


General recommendation: Only use rank-order correlation

if no other correlation methods can be used!

Value your relationships! Preventing common mistake #3


Envelop method

Lookup tables

Regression Bootstrap

Conditional logic



2. Software is no substitute for critical thinking…


Good practices:


2. Check/verify your model!


Good practices and common mistakesQuick overview


� Too often, risk analysis is equated to just “adding

@RISK to an existing Excel model

� Why a blueprint?

� Forces the team to think through the model before

building it;

� Allows everyone to provide inputs/data/ideas before

constructing model

� Ensures that the decision-questions are being answered;

� Results in better constructed models;

� Often greatly reduces time to build a model

Start with a blueprint!Good practice #1


� Why also a blueprint?

Start with a blueprint!Good practice #1


0

0Size of the model / amount of detail

Perceived accuracy

Complexity

Time & effort

Accuracy

Transparency

Case Study: Pharmaceutical firmMore detail can back-fire


� Company tried to use

Monte Carlo to support

R&D / drug development

decision, but developed

very complex model.

� While realistic, the Monte

Carlo model was too

difficult for users to

employ and not

transparent for

management

Background:

� Focused on decision-

questions

� Involved management in

model design

� Built Monte Carlo model

in close collaboration with

users

Company now uses Monte

Carlo simulation to

analyze the risks around

every new product

Solution:

� Remember, every iteration has to represent a

possible further scenario

� All too often models are built, but no time is spent

on verifying that the Monte Carlo model works as

expected/required

Check/verify your model!Good practice #2


� Some very simple steps to build robust models and

verify they work as expected M0295

� Separate inputs (e.g. min, ml, max) from the model;

� Once the model is built, add graphs!

� Before running 10,000 MC simulation, review 10-20

individual iterations (F9 button)

� Documenting models pays off!

Check/verify your model!Good practice #2


Educate and engage your executives!Good practice #3


� Support from senior management critically important

� No understanding � no trust in model �

No support of decision � risk analysis useless!

� Involve management in framing the decision-questions,

determining alternative actions and looking at possible

data or sources of information

Educate and engage your executives!Good practice #3


Presenting results:

� Limit use of hard-to-understand statistical measures

� Range of visual graphics for clear communication

Start simple, identify one or two important projects that can serve as pilot project to show the use of @RISK

Don’t focus on the model, but encourage communicating and thinking in probabilistic terms!!

Summary:




2. Software is no substitute for thinking…


Good practices:


2. Check your model!


Dr. H.GroenendaalManaging partner

EpiX Analytics [email protected]

P: 303 440 8524


Questions?

Good Practices and Common Mistakes - EpiX Analytics · Preventing common mistakes: 1. Multiply your efforts when multiplying 2. Software is no substitute for thinking… 3. Value

Documents