Turning PIGs into BACON: Estimating consequences properly Dr Edward Lewis School of Engineering and Information Technology University of NSW Canberra .

Turning PIGs into BACON: Estimating consequences properly

Dr Edward LewisSchool of Engineering and Information Technology

University of NSW Canberrawww.layrib.com

© Layrib 2014

2

Overview

1. What is needed for responding to risk2. What is available3. What is wrong with it4. What can we do better

Do not worry about keeping up. This presentation will be available from www.layrib.com

Preamble

I want to cover the latest thoughts about risk assessment, as expressed in IEC/ ISO 31010: 2009 – Risk management – Risk assessment techniques, but it is being revised at the moment.

The working group (and me)providing that revision met in London last week …

• Preparing working draft for 31010, with another meeting in Prague in October – trying to group techniques and line them up with tasks or questions to aid selection

• Preparing draft standard for Open Systems Dependability(how may are aware of “dependability” standards, such as Fault Trees or Root Cause Analysis?)

What is needed: Risk as control

Performance

TimeNow Then

Risk is “effect of uncertainty upon objectives”orPossible deviation from desired performance (current or future value)that is of concern

“Risk management:coordinated activities to direct and control an organization with regard to risk”ISO 31000, Guide 73

So emphasis should be upon control, which is:• Set the target level of

performance• Monitor deviation from

that level• Respond if the deviation

exceeds limits

risk

What is needed: It is all about risk responses

We talk a lot aboutrisk assessmentNot much about treatment(or response)

But isn’t that the whole point about risk management … doing something about it?

6

What is needed: characteristics of methods

What we need is a method that enables usquickly enoughto determine:• If a response is needed because the deviation is of concern• What response is possible – where to intercede• Which response is ‘best’

So we need methods for risk assessment that are ‘close enough’ to raise a red flag.

With sufficient resolution (even resolving) power to tell the difference between • Risk of no action vs risk of action• Alternative responses

What is Available: Risk Assessment methods

We do have many risk assessment methods (ISO 30010), as shown in the Handout.

Most of them are variations on a theme, developed by different disciplines over time.We are trying to reconcile them but it is a political fight.For example, • Beta distributions in Bayesian analysis,

with links to the use of Matthews correlations of the effects of risk indices;

• Fault and event trees are subsets of logic trees, useful for the analysis of Bowties:

• LOPA and Accimaps are similar approaches for identifying possible responses

Many are often misused and even faulty, because they have no scientific or theoretical basis – bit like boiling the quinine tree to get anti-malarial medicine.

So we need to see if we can improve what we’ve got in risk assessment and then go on to the design and evaluation methods … one day.

Get ready 0.1

x                                                                    

Plan the plan   1 x                                                                  Establish Risk Policy     1 x                                                                Establish responsibilities       1 x                                                              Gather resources         1 x                                                            Establish the Context           1 x                                                          Determine objectives, if given             1 x                                                        Consider circumstances from the external context that could influence setting or meeting objectives (valency: threats or opportunities)

              1 x                                                      

List desires of key stakeholders from their concerns about circumstances (valency: strenths or weaknesses/ vulnerabilities)

                1 x                                                    

Consider conditions from the internal context that could influence meeting objectives or values

                1   x                                                  

Establish time frame                       x                                                List objectives and values                 1 1 1 1 x                                              Establish risk criteria                         1 x                                            Form measures of consequence (determine outcomes, metrics, means of measurement)

                      1     x                                          

Determine how to estimate level of risk (measure likelihood of event, extent of circumstance)

                            1 x                                        

Establish acceptable levels of risk                               1 x                                      Determine how to aggregate risks (risk factors/ profiles)

                                1 x                                    

Assess risk: Anticipate need for action 0.1

                                  x                                  

Identify risk                                     0.1

x                                

List sources of risk (causes of uncertainty), drivers of events)??

                                      1 x                              

Determine events (changes in conditions or circumstances)

                                        1 x                            

Determine consequences of events                                           1 x                          Analyse risk                                     0.

1        x                        

Consider detail of causes, including interdependence and confidence in information about them

                                        1     1 x                      

Consider the effect of existing controls over the time span of interest

                                                1 x                    

Estimate anticipated risk level                                   1         1     1 x                  Evaluate risk                                     0.

1                x                

Compare level of risk with risk criteria                                 1 1                 1 1 x              Determine whether action in response is warranted

                                                        1 x            

Treat Risk 0.1

                                                          x          

Determine one or more options for modifying risk or possible mix of options

                                                          1 1 x        

Assess residual risk                                                               1 x 1    Compare costs, disbenefits of options                                                         1     1   x    Prepare treatment plan, showing priority for treatments

                                                                1   x  

Monitor treatment measures to determine if risks introduced elsewhere or residual risk grows above tolerable level

                                                                    1 x

Brainstorming         1       1 1 1 1 1   1   1       1 1 1   1 1           1 1   1  

** Nominal Group Technique         1       1 1 1 1 1   1   1       1 1 1   1 1     1 1   1 1   1  Interviews         1     1 1 1 1 1 1   1 1 1       1 1 1   1 1     1 1   1 1   1  Delphi                 1 1 1 1 1   1           1 1 1   1 1     1     1 1      Check-lists           1     1   1                   1                     1        Primary hazard analysis                 1                       1                              Hazard and operability studies                 1                       1                              Hazard Analysis and Critical Control Points (HACCP)

                1                       1                              

Environmental risk assessment                 1                       1                              Structure What If (SWIFT)                 1                       1                              Scenario Analysis                 1                       1                              Business Impact Analysis                                           1 1                          Failure mode effect analysis FMEA/FMECA)                                         1 1 1     1           1        Fault tree analysis                                         1 1       1                    Root cause analysis                                         1 1 1     1           1        Event tree analysis                                         1 1 1     1 1                  Cause and consequence analysis                                         1 1 1     1           1        Cause-and-effect analysis                                         1 1 1                          Layer protection analysis                                         1 1 1                 1        Decision tree                                         1 1 1       1         1   1    Human reliability analysis                                         1 1 1     1           1        Bow tie analysis                                         1 1 1     1           1        Reliability centred maintenance                                         1 1 1                 1        Sneak circuit analysis                                         1 1 1                          Markov analysis                                           1 1       1                  Monte Carlo simulation                                           1 1       1                  Bayesian statistics and nets                                           1 1       1                  FN curves                                           1 1       1                  Risk indices                                                     1     1            Consequence/ probability matrix                                                     1   1 1            ** Real option pricing                                                         1         1    Cost/benefit analysis                                                         1              Multi-criteria decision analysis                                                         1         1    ** Multiple objectives utility technqiue                                                         1         1    

8

What is available: PIGs

We are most (too) familiar with the Risk Matrix

Or

Probability x Impact Grid (PIG)

So what can we do about it.

This example is from Military Risk Management, Defence Instruction (Army) Ops 68/1 of 31 Aug 2011.It is good to see risk management introduced in accordance with the principles of ISO 31000. Pity they have not got it right, as we will see.

9

What is wrong with it: Problems with PIGs

1. compare only a small fraction of randomly selected pairs of hazards

2. mistakenly assign identical ratings to quantitatively different risks

3. mistakenly assign higher qualitative ratings to quantitatively smaller risks, lead to worse-than-random decisions

4. mistakenly allocate resources, as effective allocation of resources to risk treatments cannot be based on the categories provided by risk matrices

5. Categorizations of severity cannot be made objectively for uncertain consequences

1. risk matrices are still one of the best practical tools that we have: widespread (and convenient)

2. promote robust discussion (the discussion often being more useful than the actual rating)

3. provide some consistency to prioritizing risks4. help keep participants in workshop on track5. focus decision makers on the highest priority risks6. present complex risk data in a concise visual

fashion7. prioritizing the allocation of resources is not the

role of the risk matrix – that role belongs to the selection of risk treatments

8. any risk assessment tool can assign identical ratings to quantitatively different risks

9. no tool can consistently correctly and unambiguously compare more than a small fraction of randomly selected pairs of hazards

10.if a risk is in the ‘High’ or the ‘Top 10’ list it requires attention and whether it is third or fourth on the list is not likely to be significant

11.subjective decision making will always be a part of the risk assessment process no matter what tool is used

12.risk matrices are a tool which support risk informed decisions, not a tool for making decisions

13.last but not least, most of the flaws listed above only exist if risk matrices are used in isolation, which is rarely the case

√Justin Talbot , What’s Right with Risk Matrices, http://www.jakeman.com.au/media/knowledge-bank/whats-right-with-risk-matrices, 30 Aug 13

Oh?

10

My view of PIGs

Using a PIG is like giving a loaded revolver to a child

Another way of saying it is …A fool with a tool is still a fooland a PIG is a foolish tool

Wrong scales

Wrong combination

Wrong use

Art from Clkr.com, Creative Commons

11

Wrong scales

Wording means different things to different people – “estimative words”.

Can lack resolution power: cannot tell one level of risk from another because the impact scale is too coarse.• How much difference is there between

‘multiple fatalities’ and ‘fatality or permanent disability’?

• Going from 5 to 50 to 100 to 200M in large, uneven jumps means that 6M is as ‘bad’ as 49, 51 is as ‘bad’ as 99 and ‘much worse’ than 49

12

Wrong combination

Where things really go wrong is when probabilityis combined with impact Probability of impact is f(probability of event, probability of impact if event occurs)

in PIG, have range of extent of consequence (same as likelihood of getting estimate), range of likelihood of event: that’s OK

But often get P x I wrong

Often use scales that cannot be – must not be - combined mathematically

What do these numbers mean?

1 2 3 9 16

1 x1

1 x 2

1 x 3 1 x 9?1 x 3^2?

1 x 161 x 4^2?

3 4 8 14 21

1+2 2+2

2+62^32x4?

I give up

…

Is an Occasional loss of $99M a ‘worse’ level of risk than a Likely loss of $49M?

Looks like assigning numbers (and colours) to successive cells.

13

Wrong use 1/2

This example shows a consistent use of probability x impact (additive, as suitable for log scales).Shows different probability scales for different events.

But can still be misused (not saying that Justin does so)• Showing difference in

level of risk to choose treatments/ controls/ response

• Adding levels of risk over criteria as a score for different treatments

They are not precise enough or in the right mathematical form to do more than flag the need for attention. Often not even suitable for setting priority.

14

Wrong use 2/2: Enclosing PIGs in Risk Registers

The other problem with existing practice is the use of Risk Registers.

Risk Registers lead to, even enforce, a silo approach.They suggest that there are single causes or consequences.They suggest that there is one response to one risk.

Regard as pens for the PIGsFrom ISACA (2012), COBIT 5 Implementation

15

What we can do better: Link bow ties into chains

Firstly, turn PIGs free to run in herds,because drivers, events, and consequences entwineThe consequence of one event can drive another event.

16

Examples of the richness of risk links 1/2

Alan McLucas (2011), Failures to Learn, presentation to Risk Engineering Society Workshop, Canberra, Nov

17

Examples of the richness of risk links 2/2 (Consequence chains)

Lewis and Masroof work on risk management of the use of Cloud Computing by the ATO, 2012

18

What can we do better: Design Course of Action - Cedes

Cedes are risk-responses that change the coupling between antecedents – behaviour – consequences:• Either to reduce ‘bad’ things• Or make the most of ‘good’ things

Others would call these cedes ‘controls’,as shown in bowties.

Others would call these cedes ‘controls’,as shown in bowties.

Course of action = combination of cedes

Pick point where cede has most cumulative effect upon set of consequences

What can we do better: Bayesian Analysis of Consequences 1/2

1. Set up risk criteria2. Determine Willingness to Pay for levels of desired performance (concerns)

3. Set Act level of concern = extent of consequence that if exceeded indicates immediate need for actionWatch level of concern

= extent of consequence if exceeded suggests that action will be needed if trajectory of effect continues, anticipating likely effect of controls and their

timing4. Replace risk registers with bow ties or even consequence chains5. Analyse cascade of effects of circumstances – conditions – consequences

using fuzzy Bayesian analysis of ANDs, ORs links6. Estimate consequence, based upon set of entwined conditions, using

Pearson-Tukey: . Mean est = .185 (5%) + .625 (50%) + .185 (95%). Variance est =[ (1 (95%) – 1(5%))/3.2]2

7. Determine if estimate exceeds Act level by, say one standard deviation8. Design course of action that intercedes at most effective points in chain9. Determine response with the best Risk Adjusted Price

RAP = cost of resources needed for course of action + WTP for remaining level of concern

Worth

Performance

1 2

3

4

5

6

7

8

9

1

0 In

jury

(H

osp

ital

days

)1

2

3

4

W

TP

(0

00

,00

0)

Sufficient to recreate most cdfs

What can we do better: Bayesian Analysis of Consequences

Need WHS policy

Legal

People

Money

Lose skill

Less Fund for

service

Less Fund for

facility

Protect staff

Avoid inciden

t

Keep staff safe

Keep data secur

e

1 2

3

4

5

6

7

8

9

1

0 In

jury

(H

osp

ital

days

)1

2

3

4

5

6

7

8

9

1

0 B

reach

(N

am

es

00

0s)

12

3

4

5

6

WT

P (

00

0,0

00

)

1 2

3

4

W

TP

(0

00

,00

0)

Limit access

Have Backup

Act now

Watch C

ed

e b

C

ed

e c

C

ed

e

a

C

ed

e d

Change in internal condition with energy to have an effect

External driver

Sub-objectives/ values

Objectives/ values

Performance measure

Worth measure

What can we do better: Other ways of showing Registers

Linking these items

Generates this tree And shows this Risk List

First link Far linkEvent with most effect

22

BACON

So turn PIGS in pens into Bayesian Analysis of (cedes in chain) of CONsequences

23

What you can do to help

I am convening the Working Group in SA/NZS OB 007 for preparing the Handbook “Making Decisions with Risk”,need help in gathering ideas and judgementsSo send in the cards and letters about what you know and how you can help

Meanwhile, have a look at this presentation andRisk-Response: User Manual for Strategic and Systemic Thinking atwww.layrib.comor chat [email protected]