Analytical Variance in Qualitative
Bowtie Risk Analysis
Phillip McKenzie
Research oral presentation submitted in fulfilment of the
Degree of Master of Risk Management by Research
1
Research Part 1
Research problem
Research objectives
What is bowtie analysis
What is analytical variance
Analytical variance sources and types
System based model of analytical variance
3
Research part 1 agenda
4
Companies routinely apply risk assessment tools
and methodologies in their risk management
systems. One methodology that is growing in use
is qualitative bowtie analysis.
It has been observed that qualitative bowtie
analysis often produces inconsistent analytical
results (‘analytical variance’).
This is concerning as it calls into question the
reliability and validity of the methodology.
Research problem
Qualitative bowtie analysisISO 2000
5
Quantitative bowtie analysisISO 2000
6
Analytical
Process
Analytical
Subject
Analytical
Results
Simplified model of the analytical process7
The research premise is that given a common
‘analytical subject’; and a common ‘analytical
process’; you would expect consistent ‘analytical
results’ across multiple analyses.
However, in practice the analytical results appear to
be highly at variance from each other.
Research objectives8
Objective 1:
To identify and describe the
antecedent factors inherent in the
qualitative bowtie analysis process
which cause the observed analytical
variance.
Literature review
A literature review was performed on ‘qualitative
bowtie analysis’ and ‘analytical variance’.
This explored the factors occurring throughout the
risk analysis process which may be sources of the
observed analytical variance.
The findings of the literature review has produced a
typology of analytical variance which demonstrates
the sources of analytical variance and the related
types of variance factors.
9
No. Analytical Element Common Analytical Element Synonyms
1 Hazard threat energy
2 Top Event hazardous event
3 Causes mechanisms, threats
4 Outcomes consequences
5 Controls barriers, safeguards, defences, mitigations
6 Defeating factors escalation factors, preconditions, active failures
7 Defeating factor controls escalation factor controls
10
Typical qualitative bowtie analysis in practice
11
Evolution of accident modelsHollnagel & Goteman 2004
Model type Search principle Analysis goals Example
SequentialSpecific causes and well-
defined links
Eliminate or contain
causes
Linear chain of events
domino, Trees / networks
EpidemiologicalCarriers, barriers, and
latent conditions
Make defences and
barriers stronger
Latent conditions,
Carrier-barriers,
Pathological systems
SystemicTight couplings and
complex interactions
Monitor and control
performance variability
Control theory models,
Chaos models,
Stochastic resonance
Qualitative bowtie analysis employs an
epidemiological modelling approach. This
approach is associated with high information
complexity arising from barrier ‘control’ analysis.
12
Analytical variance
Variance: “the fact or quality of being different,
divergent, or inconsistent” Oxford University Press 2014
Variance is the actual state of difference between
two or more things. The term ‘analytical variance’
therefore refers to the inconsistent results of
multiple comparative analyses.
In the case of this research, analytical variance is
the inconsistency observed in the analytical results
of qualitative bowtie analysis.
Omissions of relevant analytical elements
Inclusions of irrelevant analytical elements
Differences in characterisations of the same analytical elements
Differences in classifications of the same analytical elements
Differences in relationships between the analytical elements
13
Observed analytical variance manifestation in
qualitative bowtie analysis
cause
outcome
top
event
hazard
defeating
factor
control
14
Variance typologies ANS and IEEE 1983; Ferson & Ginzburg 1996; Regan, Colyvan & Burgman 2002; Carey &
Burgman 2008; Markowski, Mannan & Bigoszewska 2009; Ferdous et al. 2012; Shahriar, Sadiq &
Tesfamariam 2012; Ferdous et al. 2013
Analytical variance is discussed in the literature as
resulting from either ‘uncertainty’ or ‘variability’; with
uncertainty being the most prevalent term used.
A review of the literature on the concepts of
‘uncertainty’ and ‘variability’ typologies within the
domain of risk analysis was undertaken.
This revealed a very wide spectrum of typologies
which use divergent terminology and describe
many different ‘types’ of uncertainty and variability.
Variability(Aleatory Uncertainty)
Analytical Subject Analytical Methodology Human Analysts
· Data (parameter) uncertainty
Amount of data, Diversity of data
sources, Accuracy of data sources
· Completeness uncertainty
List of initiating events, system failure
contributors, accident sequence,
definition of system damage states, list
of system interactions, accounting of
human factors
· Model uncertainty
Limitations of binary logic models
· Model uncertainty
Skill and accuracy of analyst,
Misapplication of method rules
ANS and IEEE 1983
· Variability (objective uncertainty)
Heterogeneity, stochasticity
· Ignorance (epistemic uncertainty)
Systematic measurement error,
incomplete information
Ferson & Ginzburg
1996
· Epistemic uncertainty
Measurement error, Systematic error,
Natural variation, Inherent randomness
· Epistemic uncertainty
Model uncertainty
· Linguistic uncertainty
Vagueness, Context dependence,
Ambiguity, Underspecificity,
Indeterminacy of theoretical terms
· Epistemic uncertainty
Subjective judgement
Regan, Colyvan &
Burgman 2002
· Variability
Naturally occurring, unpredictable
change
· Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Linguistic uncertainty
Ambiguity, Vagueness, Underspecificity,
Context dependenceCarey & Burgman
2008
· Objective uncertainty
Variability, Random behaviour
· Subjective uncertainty
Lack of knowledge
· Parameter uncertainty
Imprecise data, Vague data, Missing
data, Inadequate data
· Completeness uncertainty
Have all significant phenomena and
relationships been considered
· Modelling uncertainty
Inadequacies and deficiencies in
formulation of accident scenario
structure
· Subjective uncertainty
Vagueness in interpretation
Markowski, Mannan &
Bigoszewska 2009
· Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible,
Random
· Epistemic uncertainty (knowledge)
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
Ferdous et al. 2012
· Data uncertainty (epistemic)
Impreciseness, Vagueness, Lack of
knowledge, Incompleteness
· Model uncertainty
Interdependency of event relationshipsShahriar, Sadiq &
Tesfamariam 2012
· Aleatory uncertainty
Natural variation, Random behaviour of
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
Incomplete, Inconsistent or imprecise
data, Missing or unavailable data, Multi-
source data, Vagueness or inadequacy
in input data
· Quality uncertainty
Knowledge deficiency about a system
· Model uncertainty
Model adequacy, Mathematical and
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of
consequences and their interactions
Ferdous et al. 2013
Uncertainty(Epistemic Uncertainty)
Variability(Aleatory Uncertainty)
Analytical Subject Analytical Methodology Human Analysts
· Data (parameter) uncertainty
Amount of data, Diversity of data
sources, Accuracy of data sources
· Completeness uncertainty
List of initiating events, system failure
contributors, accident sequence,
definition of system damage states, list
of system interactions, accounting of
human factors
· Model uncertainty
Limitations of binary logic models
· Model uncertainty
Skill and accuracy of analyst,
Misapplication of method rules
ANS and IEEE 1983
· Variability (objective uncertainty)
Heterogeneity, stochasticity
· Ignorance (epistemic uncertainty)
Systematic measurement error,
incomplete information
Ferson & Ginzburg
1996
· Epistemic uncertainty
Measurement error, Systematic error,
Natural variation, Inherent randomness
· Epistemic uncertainty
Model uncertainty
· Linguistic uncertainty
Vagueness, Context dependence,
Ambiguity, Underspecificity,
Indeterminacy of theoretical terms
· Epistemic uncertainty
Subjective judgement
Regan, Colyvan &
Burgman 2002
· Variability
Naturally occurring, unpredictable
change
· Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Linguistic uncertainty
Ambiguity, Vagueness, Underspecificity,
Context dependenceCarey & Burgman
2008
· Objective uncertainty
Variability, Random behaviour
· Subjective uncertainty
Lack of knowledge
· Parameter uncertainty
Imprecise data, Vague data, Missing
data, Inadequate data
· Completeness uncertainty
Have all significant phenomena and
relationships been considered
· Modelling uncertainty
Inadequacies and deficiencies in
formulation of accident scenario
structure
· Subjective uncertainty
Vagueness in interpretation
Markowski, Mannan &
Bigoszewska 2009
· Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible,
Random
· Epistemic uncertainty (knowledge)
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
Ferdous et al. 2012
· Data uncertainty (epistemic)
Impreciseness, Vagueness, Lack of
knowledge, Incompleteness
· Model uncertainty
Interdependency of event relationshipsShahriar, Sadiq &
Tesfamariam 2012
· Aleatory uncertainty
Natural variation, Random behaviour of
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
Incomplete, Inconsistent or imprecise
data, Missing or unavailable data, Multi-
source data, Vagueness or inadequacy
in input data
· Quality uncertainty
Knowledge deficiency about a system
· Model uncertainty
Model adequacy, Mathematical and
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of
consequences and their interactions
Ferdous et al. 2013
Uncertainty(Epistemic Uncertainty)
Variability(Aleatory Uncertainty)
Analytical Subject Analytical Methodology Human Analysts
· Data (parameter) uncertainty
Amount of data, Diversity of data
sources, Accuracy of data sources
· Completeness uncertainty
List of initiating events, system failure
contributors, accident sequence,
definition of system damage states, list
of system interactions, accounting of
human factors
· Model uncertainty
Limitations of binary logic models
· Model uncertainty
Skill and accuracy of analyst,
Misapplication of method rules
ANS and IEEE 1983
· Variability (objective uncertainty)
Heterogeneity, stochasticity
· Ignorance (epistemic uncertainty)
Systematic measurement error,
incomplete information
Ferson & Ginzburg
1996
· Epistemic uncertainty
Measurement error, Systematic error,
Natural variation, Inherent randomness
· Epistemic uncertainty
Model uncertainty
· Linguistic uncertainty
Vagueness, Context dependence,
Ambiguity, Underspecificity,
Indeterminacy of theoretical terms
· Epistemic uncertainty
Subjective judgement
Regan, Colyvan &
Burgman 2002
· Variability
Naturally occurring, unpredictable
change
· Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Linguistic uncertainty
Ambiguity, Vagueness, Underspecificity,
Context dependenceCarey & Burgman
2008
· Objective uncertainty
Variability, Random behaviour
· Subjective uncertainty
Lack of knowledge
· Parameter uncertainty
Imprecise data, Vague data, Missing
data, Inadequate data
· Completeness uncertainty
Have all significant phenomena and
relationships been considered
· Modelling uncertainty
Inadequacies and deficiencies in
formulation of accident scenario
structure
· Subjective uncertainty
Vagueness in interpretation
Markowski, Mannan &
Bigoszewska 2009
· Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible,
Random
· Epistemic uncertainty (knowledge)
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
Ferdous et al. 2012
· Data uncertainty (epistemic)
Impreciseness, Vagueness, Lack of
knowledge, Incompleteness
· Model uncertainty
Interdependency of event relationshipsShahriar, Sadiq &
Tesfamariam 2012
· Aleatory uncertainty
Natural variation, Random behaviour of
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
Incomplete, Inconsistent or imprecise
data, Missing or unavailable data, Multi-
source data, Vagueness or inadequacy
in input data
· Quality uncertainty
Knowledge deficiency about a system
· Model uncertainty
Model adequacy, Mathematical and
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of
consequences and their interactions
Ferdous et al. 2013
Uncertainty(Epistemic Uncertainty)
Summary of uncertainty and variability typologies
within the domain of risk analysis15
Variability(Aleatory Uncertainty)
Analytical Subject Analytical Methodology Human Analysts
· Data (parameter) uncertainty
Amount of data, Diversity of data
sources, Accuracy of data sources
· Completeness uncertainty
List of initiating events, system failure
contributors, accident sequence,
definition of system damage states, list
of system interactions, accounting of
human factors
· Model uncertainty
Limitations of binary logic models
· Model uncertainty
Skill and accuracy of analyst,
Misapplication of method rules
ANS and IEEE 1983
· Variability (objective uncertainty)
Heterogeneity, stochasticity
· Ignorance (epistemic uncertainty)
Systematic measurement error,
incomplete information
Ferson & Ginzburg
1996
· Epistemic uncertainty
Measurement error, Systematic error,
Natural variation, Inherent randomness
· Epistemic uncertainty
Model uncertainty
· Linguistic uncertainty
Vagueness, Context dependence,
Ambiguity, Underspecificity,
Indeterminacy of theoretical terms
· Epistemic uncertainty
Subjective judgement
Regan, Colyvan &
Burgman 2002
· Variability
Naturally occurring, unpredictable
change
· Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Linguistic uncertainty
Ambiguity, Vagueness, Underspecificity,
Context dependenceCarey & Burgman
2008
· Objective uncertainty
Variability, Random behaviour
· Subjective uncertainty
Lack of knowledge
· Parameter uncertainty
Imprecise data, Vague data, Missing
data, Inadequate data
· Completeness uncertainty
Have all significant phenomena and
relationships been considered
· Modelling uncertainty
Inadequacies and deficiencies in
formulation of accident scenario
structure
· Subjective uncertainty
Vagueness in interpretation
Markowski, Mannan &
Bigoszewska 2009
· Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible,
Random
· Epistemic uncertainty (knowledge)
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
Ferdous et al. 2012
· Data uncertainty (epistemic)
Impreciseness, Vagueness, Lack of
knowledge, Incompleteness
· Model uncertainty
Interdependency of event relationshipsShahriar, Sadiq &
Tesfamariam 2012
· Aleatory uncertainty
Natural variation, Random behaviour of
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
Incomplete, Inconsistent or imprecise
data, Missing or unavailable data, Multi-
source data, Vagueness or inadequacy
in input data
· Quality uncertainty
Knowledge deficiency about a system
· Model uncertainty
Model adequacy, Mathematical and
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of
consequences and their interactions
Ferdous et al. 2013
Uncertainty(Epistemic Uncertainty)
Analytical subject(knowledge, complexity, randomness)
Analytical methodology(elements, terminology, format, rules, tools)
Human analysts(language, skill, experience, cognition)
16
Variance sources
Variability(Aleatory Uncertainty)
Analytical Subject Analytical Methodology Human Analysts
· Data (parameter) uncertainty
Amount of data, Diversity of data
sources, Accuracy of data sources
· Completeness uncertainty
List of initiating events, system failure
contributors, accident sequence,
definition of system damage states, list
of system interactions, accounting of
human factors
· Model uncertainty
Limitations of binary logic models
· Model uncertainty
Skill and accuracy of analyst,
Misapplication of method rules
ANS and IEEE 1983
· Variability (objective uncertainty)
Heterogeneity, stochasticity
· Ignorance (epistemic uncertainty)
Systematic measurement error,
incomplete information
Ferson & Ginzburg
1996
· Epistemic uncertainty
Measurement error, Systematic error,
Natural variation, Inherent randomness
· Epistemic uncertainty
Model uncertainty
· Linguistic uncertainty
Vagueness, Context dependence,
Ambiguity, Underspecificity,
Indeterminacy of theoretical terms
· Epistemic uncertainty
Subjective judgement
Regan, Colyvan &
Burgman 2002
· Variability
Naturally occurring, unpredictable
change
· Incertitude
Lack of model parameter knowledge,
Lack of model relationship knowledge
· Linguistic uncertainty
Ambiguity, Vagueness, Underspecificity,
Context dependenceCarey & Burgman
2008
· Objective uncertainty
Variability, Random behaviour
· Subjective uncertainty
Lack of knowledge
· Parameter uncertainty
Imprecise data, Vague data, Missing
data, Inadequate data
· Completeness uncertainty
Have all significant phenomena and
relationships been considered
· Modelling uncertainty
Inadequacies and deficiencies in
formulation of accident scenario
structure
· Subjective uncertainty
Vagueness in interpretation
Markowski, Mannan &
Bigoszewska 2009
· Aleatory uncertainty (variation)
Stochastic, Objective, Irreducible,
Random
· Epistemic uncertainty (knowledge)
Imprecise, Incomplete, Ambiguous,
Ignorance, Inconsistent, Vague
Ferdous et al. 2012
· Data uncertainty (epistemic)
Impreciseness, Vagueness, Lack of
knowledge, Incompleteness
· Model uncertainty
Interdependency of event relationshipsShahriar, Sadiq &
Tesfamariam 2012
· Aleatory uncertainty
Natural variation, Random behaviour of
a system
· Epistemic uncertainty
Lack of knowledge, Incompleteness
· Data uncertainty
Incomplete, Inconsistent or imprecise
data, Missing or unavailable data, Multi-
source data, Vagueness or inadequacy
in input data
· Quality uncertainty
Knowledge deficiency about a system
· Model uncertainty
Model adequacy, Mathematical and
numerical approximations in the model,
Assumptions or validation of the model
· Quality uncertainty
Error in hazard identification,
Incorrectness in identification of
consequences and their interactions
Ferdous et al. 2013
Uncertainty(Epistemic Uncertainty)
Knowledge amount (inadequate, source diversity)
Knowledge accuracy (errors, imprecise, inconsistent)
Knowledge completeness (missing, ignorance, incomplete)
Knowledge clarity (vague, ambiguous)
Subject complexity (heterogeneity, irreducibility)
Subject randomness (stochasticity, natural variation, unpredictability)
17
Variance source: analytical subject
Socio-technical systems (subject complexity)Bostrom & Heinen 1977
19
Model of control analysis (subject complexity)UKOOA 1999; ISO 2009a; ISO 2009b; Standards Australia 2004; Sklet 2006; NOPSEMA 2014
Context Type (1)
Risk Aversion (1)
Risk Types (2)
Cost (3)
Risk Targets (4)
Risk Level (2)
Compatibility (4)
Survivability (6)
Maintainability (6)
Ownership (6)
Equity (4)Authority (1) Consequences (4)Acceptability (4)Bases (1) Alternatives (4)
Reliability (5) Adequacy (5)Availability (5) Means Class (5)Objective Class (5)
3
41
2
Robustness (5)
Functionality (5)
Operating Status
Selection Decision
Operating Effect
Selection Context Control
Control evaluation
Define the context Monitor and review
Stakeholders (1)Efficiency (5)
(2)
(2)
(2)
(2)
Dependencies (5)
Specificity (5)
Control analysis
Elements (hazards, top-events, causes, controls, outcomes, defeating factors)
Terminology (element definitions, element names, element characteristics)
Format (structure, graphical presentation)
Rules (logic, element identification criteria, element classification criteria)
Tools (software, formulae)
Propagation
20
Variance source: analytical methodology
Language - ambiguity
Language - vagueness
Language - underspecificity
Language - context dependence
Human performance - skill
Human performance - experience
Human performance - cognition
21
Variance source: human analysts
22
Human errorReason 1990, 1997, 2008
Error
Unintended Actions Intended Actions
Slips Lapses Mistakes Violations
Ro
utin
e
Op
timis
ing
Ne
ce
ssa
ry
Ru
le B
ase
d
Kn
ow
led
ge
Ba
se
d
Me
mo
ry F
ailu
res
Re
co
gn
ition
Fa
ilure
s
Mis
ide
ntific
atio
n
No
n-d
ete
ctio
ns
Wro
ng
De
tectio
ns
Inp
ut F
ailu
res
Sto
rag
e F
ailu
res
Re
trieva
l Fa
ilure
s
Atte
ntio
n F
ailu
res
Stro
ng
Ha
bit In
trusio
n
Inte
rfere
nce
Go
od
Ru
le M
isa
pp
lied
Ba
d R
ule
Ap
plie
d
Go
od
Ru
le N
ot A
pp
lied
Good rules Bad rules No rules
Correct
performance
Correct
compliance
Correct
violation
Correct
improvisation
Erroneous
performanceMisvention Mispliance Mistake
23
Typology of analytical variance sources and factors
Analytical Subject Analytical Methodology Human Analysts
Variability(Aleatory Uncertainty)
Uncertainty(Epistemic Uncertainty)
· Language - ambiguity
· Language - vagueness
· Language - underspecificity
· Language - context dependence
· Performance - skill
· Performance - experience
· Performance - cognition
· Limits - elements
· Limits - terminology
· Limits - format
· Limits - rules
· Limits - tools
· Propagation
· Knowledge - amount
· Knowledge - accuracy
· Knowledge - completeness
· Knowledge - clarity
· Variability - randomness
· Variability - complexity
Analytical Variance
· Language - ambiguity
· Language - vagueness
· Language - underspecificity
· Language - context dependence
· Performance - skill
· Performance - experience
· Performance - cognition
· Limits - elements
· Limits - terminology
· Limits - format
· Limits - rules
· Limits - tools
· Variance propagation
· Knowledge - amount
· Knowledge - accuracy
· Knowledge - completeness
· Knowledge - clarity
· Variability - randomness
· Variability - complexity
Human
Error
Knowledge
Uncertainty
Knowledge
Variability
Control
Analytical
Methodology
Input
Analytical
Subject
Processing
Human
Analyst
Output
Analytical
Result
Methodology
Limits
Analytical Process
Variance
Propagation
Analytical
Variance
24
Systems based model of the process leading to
analytical variance
Research Part 2
Research objectives
Measurement theory
Data types and difficulties
Statistical measures of qualitative variance
Indices of analytical variance
Conclusions
Future work
26
Research part 2 agenda
Research objectives27
Objective 2:
To develop a simple and practical
methodology and tool for the
measurement of the analytical
variance between comparable
qualitative bowtie analyses.
Literature review
A broad exploration of the fundamentals of
measurement and statistical analysis was
undertaken during the literature review.
This review was focused on understanding ‘what
needs to be measured’, ‘by what means’ and ‘for
what purpose’.
The literature review also undertook an exploration
of the fundamentals of measurement theory.
28
Management by MeasurementDeming 2000
It is often said that “you can't manage
what you don't measure”. Measurement
provides the foundation for important risk
management decisions.
Deming’s ‘system of profound knowledge’
identifies a concept called the ‘knowledge
of variation’ as being a fundamental
principle and practice of good
management.
This requires managers to understand
both the ‘range’ and ‘causes’ of the
variation through application of statistical
methods for measurement.
29
Measurement scales and permissible statisticsStevens 1946; Sutcliffe 1958; Velleman & Wilkinson 1993; Michell 1986.
30
Measurement scale Empirical operations Permissible statistics
Nominal Determination of equality
· Number of cases
· Mode
· Contingency correlation
Ordinal Determination of greater or less· Median
· Percentiles
IntervalDetermination of equality of
intervals or differences
· Mean
· Standard deviation
· Rank-order correlation
· Product-moment correlation
Ratio Determination of ratios · Coefficient of variation
Data types and data difficultiesStevens 1946; McCrum-Gardner 2008; Regan, Colyvan & Burgman 2002; Carey & Burgman
2008.
The type of nominal or categorical data within
bowtie analysis creates some unique difficulties and
dominates considerations of the ‘meaningful’
measurements that can be made on the data.
Linguistic data uncertainty
Non-discrete data ranges
Multiple data samples
Different data groups
Logic diagram data order and sequence
31
Measuring qualitative varianceWilcox 1967; Lieberson 1969; Perry & Kader 2005; Kader & Perry 2007; Agresti 2007, 2014;
Magurran 2004; Mueller & Schuessler 1961; Gordon 1986.
The current statistical methods reviewed were
limited for a variety of reasons such as:
focuses on measurement of categorical data
in a discrete or finite range of possible values.
focuses on variation of data around statistical
functions related to measures of central
tendency.
only addresses variance within a limited
number of non-related categorical variables.
32
Methodology and tool for measurement of
analytical variance
A methodology has been
developed for the measurement
of analytical variance within
qualitative bowtie analysis.
The methodology provides a
simple and practical means of
processing the qualitative data
and arranging it for statistical
operations.
A measurement tool has also
been developed to assist in the
performance of the statistical
operations.
33
Select a number of comparable qualitative bowtie analyses
for use as individual data samples
Create a total data population sample which includes all
unique categories included within all individual data samples
Calculate the total number of unique categories within the
total data population sample
Calculate the frequency of each unique category within all
individual data samples
Calculate the variance of each unique category within the
total data population sample
Calculate the cumulative or total analytical variance for all
unique categories within the total data population sample
Processing individual data samples34
Total data population (extract of causes)35
ID Category Sample 1 Sample 2 Sample 3 Frequency (𝒇𝒌)
A Contaminated fuel 1 1 2
B Pilot incapacity 1 1 1 3
C Mechanical failure 1 1
D Extreme weather 1 1 2
E Fire on helicopter 1 1
F Hazmat release 1 1 2
G Bird strike 1 2
Number of unique categories (𝑘) = 7; Number of individual data samples (𝑠) = 3.
Indices of analytical variance
Three indices of analytical variance have been
developed which are capable of measuring the
analytical variance. The indices satisfy Wilcox’s
(1967) four formal properties of an index of
qualitative variance.
Index of total analytical variance (and group)
Index of category analytical variance
Index of sample analytical variance (and group)
36
Index of total analytical variance
Where:
𝑘 is the number of unique categories in the total data population
𝑓𝑘 is the frequency of the unique category within the total data population
𝑠 is the total number of individual data samples
37
Index of category analytical variance38
Where:
𝑘 is the number of unique categories in the total data population
𝑓𝑘 is the frequency of the unique category within the total data population
𝑠 is the total number of individual data samples
Total analytical variance - worked example39
𝐤 Categories 𝐒𝟏 𝐒𝟐 𝐒𝟑 𝐟𝐤Category
Variance
1 Hazard - helicopter transportation 1 1 1 3 0.00
2 Top event - loss of aircraft control 1 1 2 0.50
3 Top event - unable to reach destination 1 1 1.00
4 Cause - pilot error 1 1 1 3 0.00
5 Cause - contaminated fuel 1 1 2 0.50
6 Cause - severe weather 1 1 2 0.50
7 Cause - fire on helicopter 1 1 1.00
8 Cause - navigation failure 1 1 2 0.50
9 Outcome - ditch into ocean 1 1 1.00
10 Outcome - crash on land 1 1 2 0.50
11 Outcome - survivor drowning 1 1 2 0.50
12 Outcome - survivor hypothermia 1 1 2 0.50
Total Analytical Variance 0.5417
Group total analytical variance - worked example40
𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑 𝑓𝑘Category
Variance
1 Hazard - helicopter transportation 1 1 1 3 0.00
Group total analytical variance – Hazard categories 0.00
1 Top event - loss of aircraft control 1 1 2 0.50
2 Top event - unable to reach destination 1 1 1.00
Group total analytical variance – Top event categories 0.75
1 Cause - pilot error 1 1 1 3 0.00
2 Cause - contaminated fuel 1 1 2 0.50
3 Cause - severe weather 1 1 2 0.50
4 Cause - fire on helicopter 1 1 1.00
5 Cause - navigation failure 1 1 2 0.50
Group total analytical variance – Cause categories 0.63
1 Outcome - ditch into ocean 1 1 1.00
2 Outcome - crash on land 1 1 2 0.50
3 Outcome - survivor drowning 1 1 2 0.50
4 Outcome - survivor hypothermia 1 1 2 0.50
Group total analytical variance – Outcome categories 0.63
Index of sample analytical variance41
Where:
𝑘𝑠 is the number of unique categories in the comparison data sample
𝑘𝑡 is the number of unique categories in the total data population
Sample analytical variance - worked example42
𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑
1 Hazard - helicopter transportation 1 1 1
2 Top event - loss of aircraft control 1 1
3 Top event - unable to reach destination 1
4 Cause - pilot error 1 1 1
5 Cause - contaminated fuel 1 1
6 Cause - severe weather 1 1
7 Cause - fire on helicopter 1
8 Cause - navigation failure 1 1
9 Outcome - ditch into ocean 1
10 Outcome - crash on land 1 1
11 Outcome - survivor drowning 1 1
12 Outcome - survivor hypothermia 1 1
Sample Analytical Variance 0.4167 0.25 0.4167
Group sample analytical variance - worked example43
𝒌 Categories 𝑺𝟏 𝑺𝟐 𝑺𝟑
1 Hazard - helicopter transportation 1 1 1
Group sample analytical variance – Hazards 0.00 0.00 0.00
1 Top event - loss of aircraft control 1 1
2 Top event - unable to reach destination 1
Group sample analytical variance – Top events 0.50 0.50 0.50
1 Cause - pilot error 1 1 1
2 Cause - contaminated fuel 1 1
3 Cause - severe weather 1 1
4 Cause - fire on helicopter 1
5 Cause - navigation failure 1 1
Group sample analytical variance – Causes 0.40 0.20 0.40
1 Outcome - ditch into ocean 1
2 Outcome - crash on land 1 1
3 Outcome - survivor drowning 1 1
4 Outcome - survivor hypothermia 1 1
Group sample analytical variance – Outcomes 0.50 0.25 0.50
Validation testing44
Scenario A Scenario B Scenario C
s1 s2 s3 s1 s2 s3 s1 s2 s3
k1 1 1 1 0.00% k1 1 100.00% k1 1 100.00%
k2 1 1 1 0.00% k2 1 100.00% k2 1 100.00%
k3 1 1 1 0.00% k3 1 100.00% k3 1 100.00%
0.00 1.00 1.00
Scenario D Scenario E Scenario F
s1 s2 s3 s1 s2 s3 s1 s2 s3
k1 1 1 1 0.00% k1 150.00% k1 1 1 50.00%
k2 150.00% k2 150.00% k2 1 1 50.00%
k3 150.00% k3 150.00% k3 1 1 50.00%
1.00 1.50 0.50
Analytical variance Analytical variance Analytical variance
Analytical variance Analytical variance Analytical variance
Conclusions
Analytical variance occurs in qualitative bowtie
analysis from a number of identifiable sources and
factors which are inherent within the analytical
process. There are three sources of analytical
variance within the analytical process:
The analytical subject
The analytical methodology
The human analyst
45
Conclusions
These three sources produce analytical variance
through five primary analytical variance factors
which have corresponding subclass manifestations:
Subject knowledge (amount, accuracy, completeness, clarity)
Subject variability (randomness, complexity)
Methodology limits (elements, terminology, format, rules, tools)
Human language (ambiguity, vagueness, underspecificity, context)
Human performance (skill, experience, cognition)
46
Conclusions
Analytical variance can be understood by
performing simple statistical operations which
produce quantitative measurements or indices:
The index of total analytical variance
The index of category analytical variance
The index of sample analytical variance
The index of group total analytical variance
The index of group sample analytical variance
47
Future work
Reliance upon literature review for the development of the
model of analytical variance. The model is therefore only
considered preliminary and further work is required to
validate this model.
Validation testing of the measurement methodology was
limited to hypothetical data only; hence further validation
testing is required with real world data samples.
Experimental based research is needed to investigate the
significance of analytical variance factors by measuring
how analytical variance (dependent variable) is effected
by controlling variance factors (independent variables).
48
Thank you …49