YOU ARE DOWNLOADING DOCUMENT

Please tick the box to continue:

Transcript
Page 1: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Analytical Variance in Qualitative

Bowtie Risk Analysis

Phillip McKenzie

Research oral presentation submitted in fulfilment of the

Degree of Master of Risk Management by Research

1

Page 2: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research Part 1

Page 3: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research problem

Research objectives

What is bowtie analysis

What is analytical variance

Analytical variance sources and types

System based model of analytical variance

3

Research part 1 agenda

Page 4: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

4

Companies routinely apply risk assessment tools

and methodologies in their risk management

systems. One methodology that is growing in use

is qualitative bowtie analysis.

It has been observed that qualitative bowtie

analysis often produces inconsistent analytical

results (‘analytical variance’).

This is concerning as it calls into question the

reliability and validity of the methodology.

Research problem

Page 5: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Qualitative bowtie analysisISO 2000

5

Page 6: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Quantitative bowtie analysisISO 2000

6

Page 7: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Analytical

Process

Analytical

Subject

Analytical

Results

Simplified model of the analytical process7

The research premise is that given a common

‘analytical subject’; and a common ‘analytical

process’; you would expect consistent ‘analytical

results’ across multiple analyses.

However, in practice the analytical results appear to

be highly at variance from each other.

Page 8: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research objectives8

Objective 1:

To identify and describe the

antecedent factors inherent in the

qualitative bowtie analysis process

which cause the observed analytical

variance.

Page 9: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Literature review

A literature review was performed on ‘qualitative

bowtie analysis’ and ‘analytical variance’.

This explored the factors occurring throughout the

risk analysis process which may be sources of the

observed analytical variance.

The findings of the literature review has produced a

typology of analytical variance which demonstrates

the sources of analytical variance and the related

types of variance factors.

9

Page 10: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

No. Analytical Element Common Analytical Element Synonyms

1 Hazard threat energy

2 Top Event hazardous event

3 Causes mechanisms, threats

4 Outcomes consequences

5 Controls barriers, safeguards, defences, mitigations

6 Defeating factors escalation factors, preconditions, active failures

7 Defeating factor controls escalation factor controls

10

Typical qualitative bowtie analysis in practice

Page 11: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

11

Evolution of accident modelsHollnagel & Goteman 2004

Model type Search principle Analysis goals Example

SequentialSpecific causes and well-

defined links

Eliminate or contain

causes

Linear chain of events

domino, Trees / networks

EpidemiologicalCarriers, barriers, and

latent conditions

Make defences and

barriers stronger

Latent conditions,

Carrier-barriers,

Pathological systems

SystemicTight couplings and

complex interactions

Monitor and control

performance variability

Control theory models,

Chaos models,

Stochastic resonance

Qualitative bowtie analysis employs an

epidemiological modelling approach. This

approach is associated with high information

complexity arising from barrier ‘control’ analysis.

Page 12: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

12

Analytical variance

Variance: “the fact or quality of being different,

divergent, or inconsistent” Oxford University Press 2014

Variance is the actual state of difference between

two or more things. The term ‘analytical variance’

therefore refers to the inconsistent results of

multiple comparative analyses.

In the case of this research, analytical variance is

the inconsistency observed in the analytical results

of qualitative bowtie analysis.

Page 13: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Omissions of relevant analytical elements

Inclusions of irrelevant analytical elements

Differences in characterisations of the same analytical elements

Differences in classifications of the same analytical elements

Differences in relationships between the analytical elements

13

Observed analytical variance manifestation in

qualitative bowtie analysis

cause

outcome

top

event

hazard

defeating

factor

control

Page 14: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

14

Variance typologies ANS and IEEE 1983; Ferson & Ginzburg 1996; Regan, Colyvan & Burgman 2002; Carey &

Burgman 2008; Markowski, Mannan & Bigoszewska 2009; Ferdous et al. 2012; Shahriar, Sadiq &

Tesfamariam 2012; Ferdous et al. 2013

Analytical variance is discussed in the literature as

resulting from either ‘uncertainty’ or ‘variability’; with

uncertainty being the most prevalent term used.

A review of the literature on the concepts of

‘uncertainty’ and ‘variability’ typologies within the

domain of risk analysis was undertaken.

This revealed a very wide spectrum of typologies

which use divergent terminology and describe

many different ‘types’ of uncertainty and variability.

Page 15: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Variability(Aleatory Uncertainty)

Analytical Subject Analytical Methodology Human Analysts

· Data (parameter) uncertainty

Amount of data, Diversity of data

sources, Accuracy of data sources

· Completeness uncertainty

List of initiating events, system failure

contributors, accident sequence,

definition of system damage states, list

of system interactions, accounting of

human factors

· Model uncertainty

Limitations of binary logic models

· Model uncertainty

Skill and accuracy of analyst,

Misapplication of method rules

ANS and IEEE 1983

· Variability (objective uncertainty)

Heterogeneity, stochasticity

· Ignorance (epistemic uncertainty)

Systematic measurement error,

incomplete information

Ferson & Ginzburg

1996

· Epistemic uncertainty

Measurement error, Systematic error,

Natural variation, Inherent randomness

· Epistemic uncertainty

Model uncertainty

· Linguistic uncertainty

Vagueness, Context dependence,

Ambiguity, Underspecificity,

Indeterminacy of theoretical terms

· Epistemic uncertainty

Subjective judgement

Regan, Colyvan &

Burgman 2002

· Variability

Naturally occurring, unpredictable

change

· Incertitude

Lack of model parameter knowledge,

Lack of model relationship knowledge

· Linguistic uncertainty

Ambiguity, Vagueness, Underspecificity,

Context dependenceCarey & Burgman

2008

· Objective uncertainty

Variability, Random behaviour

· Subjective uncertainty

Lack of knowledge

· Parameter uncertainty

Imprecise data, Vague data, Missing

data, Inadequate data

· Completeness uncertainty

Have all significant phenomena and

relationships been considered

· Modelling uncertainty

Inadequacies and deficiencies in

formulation of accident scenario

structure

· Subjective uncertainty

Vagueness in interpretation

Markowski, Mannan &

Bigoszewska 2009

· Aleatory uncertainty (variation)

Stochastic, Objective, Irreducible,

Random

· Epistemic uncertainty (knowledge)

Imprecise, Incomplete, Ambiguous,

Ignorance, Inconsistent, Vague

Ferdous et al. 2012

· Data uncertainty (epistemic)

Impreciseness, Vagueness, Lack of

knowledge, Incompleteness

· Model uncertainty

Interdependency of event relationshipsShahriar, Sadiq &

Tesfamariam 2012

· Aleatory uncertainty

Natural variation, Random behaviour of

a system

· Epistemic uncertainty

Lack of knowledge, Incompleteness

· Data uncertainty

Incomplete, Inconsistent or imprecise

data, Missing or unavailable data, Multi-

source data, Vagueness or inadequacy

in input data

· Quality uncertainty

Knowledge deficiency about a system

· Model uncertainty

Model adequacy, Mathematical and

numerical approximations in the model,

Assumptions or validation of the model

· Quality uncertainty

Error in hazard identification,

Incorrectness in identification of

consequences and their interactions

Ferdous et al. 2013

Uncertainty(Epistemic Uncertainty)

Variability(Aleatory Uncertainty)

Analytical Subject Analytical Methodology Human Analysts

· Data (parameter) uncertainty

Amount of data, Diversity of data

sources, Accuracy of data sources

· Completeness uncertainty

List of initiating events, system failure

contributors, accident sequence,

definition of system damage states, list

of system interactions, accounting of

human factors

· Model uncertainty

Limitations of binary logic models

· Model uncertainty

Skill and accuracy of analyst,

Misapplication of method rules

ANS and IEEE 1983

· Variability (objective uncertainty)

Heterogeneity, stochasticity

· Ignorance (epistemic uncertainty)

Systematic measurement error,

incomplete information

Ferson & Ginzburg

1996

· Epistemic uncertainty

Measurement error, Systematic error,

Natural variation, Inherent randomness

· Epistemic uncertainty

Model uncertainty

· Linguistic uncertainty

Vagueness, Context dependence,

Ambiguity, Underspecificity,

Indeterminacy of theoretical terms

· Epistemic uncertainty

Subjective judgement

Regan, Colyvan &

Burgman 2002

· Variability

Naturally occurring, unpredictable

change

· Incertitude

Lack of model parameter knowledge,

Lack of model relationship knowledge

· Linguistic uncertainty

Ambiguity, Vagueness, Underspecificity,

Context dependenceCarey & Burgman

2008

· Objective uncertainty

Variability, Random behaviour

· Subjective uncertainty

Lack of knowledge

· Parameter uncertainty

Imprecise data, Vague data, Missing

data, Inadequate data

· Completeness uncertainty

Have all significant phenomena and

relationships been considered

· Modelling uncertainty

Inadequacies and deficiencies in

formulation of accident scenario

structure

· Subjective uncertainty

Vagueness in interpretation

Markowski, Mannan &

Bigoszewska 2009

· Aleatory uncertainty (variation)

Stochastic, Objective, Irreducible,

Random

· Epistemic uncertainty (knowledge)

Imprecise, Incomplete, Ambiguous,

Ignorance, Inconsistent, Vague

Ferdous et al. 2012

· Data uncertainty (epistemic)

Impreciseness, Vagueness, Lack of

knowledge, Incompleteness

· Model uncertainty

Interdependency of event relationshipsShahriar, Sadiq &

Tesfamariam 2012

· Aleatory uncertainty

Natural variation, Random behaviour of

a system

· Epistemic uncertainty

Lack of knowledge, Incompleteness

· Data uncertainty

Incomplete, Inconsistent or imprecise

data, Missing or unavailable data, Multi-

source data, Vagueness or inadequacy

in input data

· Quality uncertainty

Knowledge deficiency about a system

· Model uncertainty

Model adequacy, Mathematical and

numerical approximations in the model,

Assumptions or validation of the model

· Quality uncertainty

Error in hazard identification,

Incorrectness in identification of

consequences and their interactions

Ferdous et al. 2013

Uncertainty(Epistemic Uncertainty)

Variability(Aleatory Uncertainty)

Analytical Subject Analytical Methodology Human Analysts

· Data (parameter) uncertainty

Amount of data, Diversity of data

sources, Accuracy of data sources

· Completeness uncertainty

List of initiating events, system failure

contributors, accident sequence,

definition of system damage states, list

of system interactions, accounting of

human factors

· Model uncertainty

Limitations of binary logic models

· Model uncertainty

Skill and accuracy of analyst,

Misapplication of method rules

ANS and IEEE 1983

· Variability (objective uncertainty)

Heterogeneity, stochasticity

· Ignorance (epistemic uncertainty)

Systematic measurement error,

incomplete information

Ferson & Ginzburg

1996

· Epistemic uncertainty

Measurement error, Systematic error,

Natural variation, Inherent randomness

· Epistemic uncertainty

Model uncertainty

· Linguistic uncertainty

Vagueness, Context dependence,

Ambiguity, Underspecificity,

Indeterminacy of theoretical terms

· Epistemic uncertainty

Subjective judgement

Regan, Colyvan &

Burgman 2002

· Variability

Naturally occurring, unpredictable

change

· Incertitude

Lack of model parameter knowledge,

Lack of model relationship knowledge

· Linguistic uncertainty

Ambiguity, Vagueness, Underspecificity,

Context dependenceCarey & Burgman

2008

· Objective uncertainty

Variability, Random behaviour

· Subjective uncertainty

Lack of knowledge

· Parameter uncertainty

Imprecise data, Vague data, Missing

data, Inadequate data

· Completeness uncertainty

Have all significant phenomena and

relationships been considered

· Modelling uncertainty

Inadequacies and deficiencies in

formulation of accident scenario

structure

· Subjective uncertainty

Vagueness in interpretation

Markowski, Mannan &

Bigoszewska 2009

· Aleatory uncertainty (variation)

Stochastic, Objective, Irreducible,

Random

· Epistemic uncertainty (knowledge)

Imprecise, Incomplete, Ambiguous,

Ignorance, Inconsistent, Vague

Ferdous et al. 2012

· Data uncertainty (epistemic)

Impreciseness, Vagueness, Lack of

knowledge, Incompleteness

· Model uncertainty

Interdependency of event relationshipsShahriar, Sadiq &

Tesfamariam 2012

· Aleatory uncertainty

Natural variation, Random behaviour of

a system

· Epistemic uncertainty

Lack of knowledge, Incompleteness

· Data uncertainty

Incomplete, Inconsistent or imprecise

data, Missing or unavailable data, Multi-

source data, Vagueness or inadequacy

in input data

· Quality uncertainty

Knowledge deficiency about a system

· Model uncertainty

Model adequacy, Mathematical and

numerical approximations in the model,

Assumptions or validation of the model

· Quality uncertainty

Error in hazard identification,

Incorrectness in identification of

consequences and their interactions

Ferdous et al. 2013

Uncertainty(Epistemic Uncertainty)

Summary of uncertainty and variability typologies

within the domain of risk analysis15

Variability(Aleatory Uncertainty)

Analytical Subject Analytical Methodology Human Analysts

· Data (parameter) uncertainty

Amount of data, Diversity of data

sources, Accuracy of data sources

· Completeness uncertainty

List of initiating events, system failure

contributors, accident sequence,

definition of system damage states, list

of system interactions, accounting of

human factors

· Model uncertainty

Limitations of binary logic models

· Model uncertainty

Skill and accuracy of analyst,

Misapplication of method rules

ANS and IEEE 1983

· Variability (objective uncertainty)

Heterogeneity, stochasticity

· Ignorance (epistemic uncertainty)

Systematic measurement error,

incomplete information

Ferson & Ginzburg

1996

· Epistemic uncertainty

Measurement error, Systematic error,

Natural variation, Inherent randomness

· Epistemic uncertainty

Model uncertainty

· Linguistic uncertainty

Vagueness, Context dependence,

Ambiguity, Underspecificity,

Indeterminacy of theoretical terms

· Epistemic uncertainty

Subjective judgement

Regan, Colyvan &

Burgman 2002

· Variability

Naturally occurring, unpredictable

change

· Incertitude

Lack of model parameter knowledge,

Lack of model relationship knowledge

· Linguistic uncertainty

Ambiguity, Vagueness, Underspecificity,

Context dependenceCarey & Burgman

2008

· Objective uncertainty

Variability, Random behaviour

· Subjective uncertainty

Lack of knowledge

· Parameter uncertainty

Imprecise data, Vague data, Missing

data, Inadequate data

· Completeness uncertainty

Have all significant phenomena and

relationships been considered

· Modelling uncertainty

Inadequacies and deficiencies in

formulation of accident scenario

structure

· Subjective uncertainty

Vagueness in interpretation

Markowski, Mannan &

Bigoszewska 2009

· Aleatory uncertainty (variation)

Stochastic, Objective, Irreducible,

Random

· Epistemic uncertainty (knowledge)

Imprecise, Incomplete, Ambiguous,

Ignorance, Inconsistent, Vague

Ferdous et al. 2012

· Data uncertainty (epistemic)

Impreciseness, Vagueness, Lack of

knowledge, Incompleteness

· Model uncertainty

Interdependency of event relationshipsShahriar, Sadiq &

Tesfamariam 2012

· Aleatory uncertainty

Natural variation, Random behaviour of

a system

· Epistemic uncertainty

Lack of knowledge, Incompleteness

· Data uncertainty

Incomplete, Inconsistent or imprecise

data, Missing or unavailable data, Multi-

source data, Vagueness or inadequacy

in input data

· Quality uncertainty

Knowledge deficiency about a system

· Model uncertainty

Model adequacy, Mathematical and

numerical approximations in the model,

Assumptions or validation of the model

· Quality uncertainty

Error in hazard identification,

Incorrectness in identification of

consequences and their interactions

Ferdous et al. 2013

Uncertainty(Epistemic Uncertainty)

Page 16: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Analytical subject(knowledge, complexity, randomness)

Analytical methodology(elements, terminology, format, rules, tools)

Human analysts(language, skill, experience, cognition)

16

Variance sources

Variability(Aleatory Uncertainty)

Analytical Subject Analytical Methodology Human Analysts

· Data (parameter) uncertainty

Amount of data, Diversity of data

sources, Accuracy of data sources

· Completeness uncertainty

List of initiating events, system failure

contributors, accident sequence,

definition of system damage states, list

of system interactions, accounting of

human factors

· Model uncertainty

Limitations of binary logic models

· Model uncertainty

Skill and accuracy of analyst,

Misapplication of method rules

ANS and IEEE 1983

· Variability (objective uncertainty)

Heterogeneity, stochasticity

· Ignorance (epistemic uncertainty)

Systematic measurement error,

incomplete information

Ferson & Ginzburg

1996

· Epistemic uncertainty

Measurement error, Systematic error,

Natural variation, Inherent randomness

· Epistemic uncertainty

Model uncertainty

· Linguistic uncertainty

Vagueness, Context dependence,

Ambiguity, Underspecificity,

Indeterminacy of theoretical terms

· Epistemic uncertainty

Subjective judgement

Regan, Colyvan &

Burgman 2002

· Variability

Naturally occurring, unpredictable

change

· Incertitude

Lack of model parameter knowledge,

Lack of model relationship knowledge

· Linguistic uncertainty

Ambiguity, Vagueness, Underspecificity,

Context dependenceCarey & Burgman

2008

· Objective uncertainty

Variability, Random behaviour

· Subjective uncertainty

Lack of knowledge

· Parameter uncertainty

Imprecise data, Vague data, Missing

data, Inadequate data

· Completeness uncertainty

Have all significant phenomena and

relationships been considered

· Modelling uncertainty

Inadequacies and deficiencies in

formulation of accident scenario

structure

· Subjective uncertainty

Vagueness in interpretation

Markowski, Mannan &

Bigoszewska 2009

· Aleatory uncertainty (variation)

Stochastic, Objective, Irreducible,

Random

· Epistemic uncertainty (knowledge)

Imprecise, Incomplete, Ambiguous,

Ignorance, Inconsistent, Vague

Ferdous et al. 2012

· Data uncertainty (epistemic)

Impreciseness, Vagueness, Lack of

knowledge, Incompleteness

· Model uncertainty

Interdependency of event relationshipsShahriar, Sadiq &

Tesfamariam 2012

· Aleatory uncertainty

Natural variation, Random behaviour of

a system

· Epistemic uncertainty

Lack of knowledge, Incompleteness

· Data uncertainty

Incomplete, Inconsistent or imprecise

data, Missing or unavailable data, Multi-

source data, Vagueness or inadequacy

in input data

· Quality uncertainty

Knowledge deficiency about a system

· Model uncertainty

Model adequacy, Mathematical and

numerical approximations in the model,

Assumptions or validation of the model

· Quality uncertainty

Error in hazard identification,

Incorrectness in identification of

consequences and their interactions

Ferdous et al. 2013

Uncertainty(Epistemic Uncertainty)

Page 17: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Knowledge amount (inadequate, source diversity)

Knowledge accuracy (errors, imprecise, inconsistent)

Knowledge completeness (missing, ignorance, incomplete)

Knowledge clarity (vague, ambiguous)

Subject complexity (heterogeneity, irreducibility)

Subject randomness (stochasticity, natural variation, unpredictability)

17

Variance source: analytical subject

Page 18: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Socio-technical systems (subject complexity)Bostrom & Heinen 1977

Page 19: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

19

Model of control analysis (subject complexity)UKOOA 1999; ISO 2009a; ISO 2009b; Standards Australia 2004; Sklet 2006; NOPSEMA 2014

Context Type (1)

Risk Aversion (1)

Risk Types (2)

Cost (3)

Risk Targets (4)

Risk Level (2)

Compatibility (4)

Survivability (6)

Maintainability (6)

Ownership (6)

Equity (4)Authority (1) Consequences (4)Acceptability (4)Bases (1) Alternatives (4)

Reliability (5) Adequacy (5)Availability (5) Means Class (5)Objective Class (5)

3

41

2

Robustness (5)

Functionality (5)

Operating Status

Selection Decision

Operating Effect

Selection Context Control

Control evaluation

Define the context Monitor and review

Stakeholders (1)Efficiency (5)

(2)

(2)

(2)

(2)

Dependencies (5)

Specificity (5)

Control analysis

Page 20: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Elements (hazards, top-events, causes, controls, outcomes, defeating factors)

Terminology (element definitions, element names, element characteristics)

Format (structure, graphical presentation)

Rules (logic, element identification criteria, element classification criteria)

Tools (software, formulae)

Propagation

20

Variance source: analytical methodology

Page 21: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Language - ambiguity

Language - vagueness

Language - underspecificity

Language - context dependence

Human performance - skill

Human performance - experience

Human performance - cognition

21

Variance source: human analysts

Page 22: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

22

Human errorReason 1990, 1997, 2008

Error

Unintended Actions Intended Actions

Slips Lapses Mistakes Violations

Ro

utin

e

Op

timis

ing

Ne

ce

ssa

ry

Ru

le B

ase

d

Kn

ow

led

ge

Ba

se

d

Me

mo

ry F

ailu

res

Re

co

gn

ition

Fa

ilure

s

Mis

ide

ntific

atio

n

No

n-d

ete

ctio

ns

Wro

ng

De

tectio

ns

Inp

ut F

ailu

res

Sto

rag

e F

ailu

res

Re

trieva

l Fa

ilure

s

Atte

ntio

n F

ailu

res

Stro

ng

Ha

bit In

trusio

n

Inte

rfere

nce

Go

od

Ru

le M

isa

pp

lied

Ba

d R

ule

Ap

plie

d

Go

od

Ru

le N

ot A

pp

lied

Good rules Bad rules No rules

Correct

performance

Correct

compliance

Correct

violation

Correct

improvisation

Erroneous

performanceMisvention Mispliance Mistake

Page 23: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

23

Typology of analytical variance sources and factors

Analytical Subject Analytical Methodology Human Analysts

Variability(Aleatory Uncertainty)

Uncertainty(Epistemic Uncertainty)

· Language - ambiguity

· Language - vagueness

· Language - underspecificity

· Language - context dependence

· Performance - skill

· Performance - experience

· Performance - cognition

· Limits - elements

· Limits - terminology

· Limits - format

· Limits - rules

· Limits - tools

· Propagation

· Knowledge - amount

· Knowledge - accuracy

· Knowledge - completeness

· Knowledge - clarity

· Variability - randomness

· Variability - complexity

Analytical Variance

Page 24: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

· Language - ambiguity

· Language - vagueness

· Language - underspecificity

· Language - context dependence

· Performance - skill

· Performance - experience

· Performance - cognition

· Limits - elements

· Limits - terminology

· Limits - format

· Limits - rules

· Limits - tools

· Variance propagation

· Knowledge - amount

· Knowledge - accuracy

· Knowledge - completeness

· Knowledge - clarity

· Variability - randomness

· Variability - complexity

Human

Error

Knowledge

Uncertainty

Knowledge

Variability

Control

Analytical

Methodology

Input

Analytical

Subject

Processing

Human

Analyst

Output

Analytical

Result

Methodology

Limits

Analytical Process

Variance

Propagation

Analytical

Variance

24

Systems based model of the process leading to

analytical variance

Page 25: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research Part 2

Page 26: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research objectives

Measurement theory

Data types and difficulties

Statistical measures of qualitative variance

Indices of analytical variance

Conclusions

Future work

26

Research part 2 agenda

Page 27: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Research objectives27

Objective 2:

To develop a simple and practical

methodology and tool for the

measurement of the analytical

variance between comparable

qualitative bowtie analyses.

Page 28: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Literature review

A broad exploration of the fundamentals of

measurement and statistical analysis was

undertaken during the literature review.

This review was focused on understanding ‘what

needs to be measured’, ‘by what means’ and ‘for

what purpose’.

The literature review also undertook an exploration

of the fundamentals of measurement theory.

28

Page 29: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Management by MeasurementDeming 2000

It is often said that “you can't manage

what you don't measure”. Measurement

provides the foundation for important risk

management decisions.

Deming’s ‘system of profound knowledge’

identifies a concept called the ‘knowledge

of variation’ as being a fundamental

principle and practice of good

management.

This requires managers to understand

both the ‘range’ and ‘causes’ of the

variation through application of statistical

methods for measurement.

29

Page 30: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Measurement scales and permissible statisticsStevens 1946; Sutcliffe 1958; Velleman & Wilkinson 1993; Michell 1986.

30

Measurement scale Empirical operations Permissible statistics

Nominal Determination of equality

· Number of cases

· Mode

· Contingency correlation

Ordinal Determination of greater or less· Median

· Percentiles

IntervalDetermination of equality of

intervals or differences

· Mean

· Standard deviation

· Rank-order correlation

· Product-moment correlation

Ratio Determination of ratios · Coefficient of variation

Page 31: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Data types and data difficultiesStevens 1946; McCrum-Gardner 2008; Regan, Colyvan & Burgman 2002; Carey & Burgman

2008.

The type of nominal or categorical data within

bowtie analysis creates some unique difficulties and

dominates considerations of the ‘meaningful’

measurements that can be made on the data.

Linguistic data uncertainty

Non-discrete data ranges

Multiple data samples

Different data groups

Logic diagram data order and sequence

31

Page 32: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Measuring qualitative varianceWilcox 1967; Lieberson 1969; Perry & Kader 2005; Kader & Perry 2007; Agresti 2007, 2014;

Magurran 2004; Mueller & Schuessler 1961; Gordon 1986.

The current statistical methods reviewed were

limited for a variety of reasons such as:

focuses on measurement of categorical data

in a discrete or finite range of possible values.

focuses on variation of data around statistical

functions related to measures of central

tendency.

only addresses variance within a limited

number of non-related categorical variables.

32

Page 33: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Methodology and tool for measurement of

analytical variance

A methodology has been

developed for the measurement

of analytical variance within

qualitative bowtie analysis.

The methodology provides a

simple and practical means of

processing the qualitative data

and arranging it for statistical

operations.

A measurement tool has also

been developed to assist in the

performance of the statistical

operations.

33

Select a number of comparable qualitative bowtie analyses

for use as individual data samples

Create a total data population sample which includes all

unique categories included within all individual data samples

Calculate the total number of unique categories within the

total data population sample

Calculate the frequency of each unique category within all

individual data samples

Calculate the variance of each unique category within the

total data population sample

Calculate the cumulative or total analytical variance for all

unique categories within the total data population sample

Page 34: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Processing individual data samples34

Page 35: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Total data population (extract of causes)35

ID Category Sample 1 Sample 2 Sample 3 Frequency (𝒇𝒌)

A Contaminated fuel 1 1 2

B Pilot incapacity 1 1 1 3

C Mechanical failure 1 1

D Extreme weather 1 1 2

E Fire on helicopter 1 1

F Hazmat release 1 1 2

G Bird strike 1 2

Number of unique categories (𝑘) = 7; Number of individual data samples (𝑠) = 3.

Page 36: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Indices of analytical variance

Three indices of analytical variance have been

developed which are capable of measuring the

analytical variance. The indices satisfy Wilcox’s

(1967) four formal properties of an index of

qualitative variance.

Index of total analytical variance (and group)

Index of category analytical variance

Index of sample analytical variance (and group)

36

Page 37: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Index of total analytical variance

Where:

𝑘 is the number of unique categories in the total data population

𝑓𝑘 is the frequency of the unique category within the total data population

𝑠 is the total number of individual data samples

37

Page 38: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Index of category analytical variance38

Where:

𝑘 is the number of unique categories in the total data population

𝑓𝑘 is the frequency of the unique category within the total data population

𝑠 is the total number of individual data samples

Page 39: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Total analytical variance - worked example39

𝐤 Categories 𝐒𝟏 𝐒𝟐 𝐒𝟑 𝐟𝐤Category

Variance

1 Hazard - helicopter transportation 1 1 1 3 0.00

2 Top event - loss of aircraft control 1 1 2 0.50

3 Top event - unable to reach destination 1 1 1.00

4 Cause - pilot error 1 1 1 3 0.00

5 Cause - contaminated fuel 1 1 2 0.50

6 Cause - severe weather 1 1 2 0.50

7 Cause - fire on helicopter 1 1 1.00

8 Cause - navigation failure 1 1 2 0.50

9 Outcome - ditch into ocean 1 1 1.00

10 Outcome - crash on land 1 1 2 0.50

11 Outcome - survivor drowning 1 1 2 0.50

12 Outcome - survivor hypothermia 1 1 2 0.50

Total Analytical Variance 0.5417

Page 40: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Group total analytical variance - worked example40

𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑 𝑓𝑘Category

Variance

1 Hazard - helicopter transportation 1 1 1 3 0.00

Group total analytical variance – Hazard categories 0.00

1 Top event - loss of aircraft control 1 1 2 0.50

2 Top event - unable to reach destination 1 1 1.00

Group total analytical variance – Top event categories 0.75

1 Cause - pilot error 1 1 1 3 0.00

2 Cause - contaminated fuel 1 1 2 0.50

3 Cause - severe weather 1 1 2 0.50

4 Cause - fire on helicopter 1 1 1.00

5 Cause - navigation failure 1 1 2 0.50

Group total analytical variance – Cause categories 0.63

1 Outcome - ditch into ocean 1 1 1.00

2 Outcome - crash on land 1 1 2 0.50

3 Outcome - survivor drowning 1 1 2 0.50

4 Outcome - survivor hypothermia 1 1 2 0.50

Group total analytical variance – Outcome categories 0.63

Page 41: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Index of sample analytical variance41

Where:

𝑘𝑠 is the number of unique categories in the comparison data sample

𝑘𝑡 is the number of unique categories in the total data population

Page 42: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Sample analytical variance - worked example42

𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑

1 Hazard - helicopter transportation 1 1 1

2 Top event - loss of aircraft control 1 1

3 Top event - unable to reach destination 1

4 Cause - pilot error 1 1 1

5 Cause - contaminated fuel 1 1

6 Cause - severe weather 1 1

7 Cause - fire on helicopter 1

8 Cause - navigation failure 1 1

9 Outcome - ditch into ocean 1

10 Outcome - crash on land 1 1

11 Outcome - survivor drowning 1 1

12 Outcome - survivor hypothermia 1 1

Sample Analytical Variance 0.4167 0.25 0.4167

Page 43: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Group sample analytical variance - worked example43

𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑

1 Hazard - helicopter transportation 1 1 1

Group sample analytical variance – Hazards 0.00 0.00 0.00

1 Top event - loss of aircraft control 1 1

2 Top event - unable to reach destination 1

Group sample analytical variance – Top events 0.50 0.50 0.50

1 Cause - pilot error 1 1 1

2 Cause - contaminated fuel 1 1

3 Cause - severe weather 1 1

4 Cause - fire on helicopter 1

5 Cause - navigation failure 1 1

Group sample analytical variance – Causes 0.40 0.20 0.40

1 Outcome - ditch into ocean 1

2 Outcome - crash on land 1 1

3 Outcome - survivor drowning 1 1

4 Outcome - survivor hypothermia 1 1

Group sample analytical variance – Outcomes 0.50 0.25 0.50

Page 44: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Validation testing44

Scenario A Scenario B Scenario C

s1 s2 s3 s1 s2 s3 s1 s2 s3

k1 1 1 1 0.00% k1 1 100.00% k1 1 100.00%

k2 1 1 1 0.00% k2 1 100.00% k2 1 100.00%

k3 1 1 1 0.00% k3 1 100.00% k3 1 100.00%

0.00 1.00 1.00

Scenario D Scenario E Scenario F

s1 s2 s3 s1 s2 s3 s1 s2 s3

k1 1 1 1 0.00% k1 150.00% k1 1 1 50.00%

k2 150.00% k2 150.00% k2 1 1 50.00%

k3 150.00% k3 150.00% k3 1 1 50.00%

1.00 1.50 0.50

Analytical variance Analytical variance Analytical variance

Analytical variance Analytical variance Analytical variance

Page 45: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Conclusions

Analytical variance occurs in qualitative bowtie

analysis from a number of identifiable sources and

factors which are inherent within the analytical

process. There are three sources of analytical

variance within the analytical process:

The analytical subject

The analytical methodology

The human analyst

45

Page 46: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Conclusions

These three sources produce analytical variance

through five primary analytical variance factors

which have corresponding subclass manifestations:

Subject knowledge (amount, accuracy, completeness, clarity)

Subject variability (randomness, complexity)

Methodology limits (elements, terminology, format, rules, tools)

Human language (ambiguity, vagueness, underspecificity, context)

Human performance (skill, experience, cognition)

46

Page 47: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Conclusions

Analytical variance can be understood by

performing simple statistical operations which

produce quantitative measurements or indices:

The index of total analytical variance

The index of category analytical variance

The index of sample analytical variance

The index of group total analytical variance

The index of group sample analytical variance

47

Page 48: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Future work

Reliance upon literature review for the development of the

model of analytical variance. The model is therefore only

considered preliminary and further work is required to

validate this model.

Validation testing of the measurement methodology was

limited to hypothetical data only; hence further validation

testing is required with real world data samples.

Experimental based research is needed to investigate the

significance of analytical variance factors by measuring

how analytical variance (dependent variable) is effected

by controlling variance factors (independent variables).

48

Page 49: RSK80016 - Phil McKenzie (6900593) - Research Oral Presentation

Thank you …49


Related Documents