1 Copyright © 2006 M. E. Kabay. All rights reserved. Rational Risk Management: Balancing Costs and Benefits of Security Measures Cybersecurity Conference.

1 Copyright © 2006 M. E. Kabay. All rights reserved.

Rational Risk Management: Balancing Costs and Benefits of Security

MeasuresCybersecurity Conference

26 July 2006 – 13:00-16:00M. E. Kabay, PhD, CISSP-ISSMP

Assoc. Prof. Information AssuranceDivision of Business & Management, Norwich University

Program Director, MSIASchool of Graduate Studies, Norwich University

mailto:[email protected] V: 802.479.7937


TopicsPart 1: Risk Assessment *

DefinitionsObjectives of Risk AssessmentLimits of QuestionnairesA Model of RiskRisk Mitigation

Part 2: Risk Assessment TechniquesQuestionnairesFocus GroupsInterviewsAnalytical Tools

____________________________________________

* Based in part on Robert Jacobson’s chapter in CSH4 (Bosworth & Kabay’s Computer Security Handbook, 4th edition – Wiley, 2002)


Definitions

Risk: possibility of suffering harm or loss

Risk ManagementRisk assessmentRisk mitigationSecurity managementSecurity auditing

Feedback ensures corrective actions back into process – continuous process improvement

Security is a process, not a state.


Objectives of Risk Assessment

Help to select subset of security measures given limitations on resources

Every system will have unique security requirements

Risk assessment must provide appropriate information aboutPossible losses (costs of damage and of

recovery)Estimated probability of specific events or

classes of events


A Model of Risk

Fundamental Risk ModelTwo Inconsequential Risk ClassesTwo Significant Risk ClassesReal-World Risks & the ALE


Fundamental Risk Model

“Jacobson’s Window”

Low High

ConsequencesH

ighOc

cu

rre

nc

es Lo

w


Two Inconsequential Risk Classes

Low High

ConsequencesH

ighOc

cu

rre

nc

es

Don’tcare

Doesn’thappen

Lo

w


Two Significant Risk Classes

Low High

ConsequencesH

ighOc

cu

rre

nc

es

Major fire,long power outage,

flooding,cash fraud,

….

Power transient,minor sw bug,

keystroke error,….

Lo

w


Real-World Risks & the ALE

To compare risks, we use the annualized loss expectancy (ALE):

E(x) = piciWhere

E(x) = ALE of strategy xpi = probability of occurrence ici = cost of occurrence i = add up the products

i


Example of ALE Calculation

Keystroke errors (Jacobson’s example with slight modifications)

100 errors per operator per hour

100 operators2,000 hours per

operator per year= 20,000,000 errors per yearDetection rate 99.9% at no costThus p = 0.001 failure rate of missed errorsErrors corrected later @ $1 eachSo E(X) = 0.001 * 20,000,000 * $1 = $20,000


Another ALE Calculation

Major fire (also Jacobson’s example)

Probability “p” of major fire in a year = 0.0001

Cost of major fire estimated at $100M

Therefore E(x) = 0.0001 x $100M = 10-4 x $108 = $104 = $10,000


ALE of an Insurance PolicyCustomer bets insurance company he will die

this year (probability 0.1%)Bets (pays) $750 in “premium” If customer dies, insurance company pays

$500,000 to widow Insurance company bets that customer lives –

keeps premium, pays nothing.p1 = 0.001 c1 = -$500,000 (a gain to widow

and a loss to the insurance company)p2 = 0.999 c2 = +$750 (a loss to family and a

gain to the insurance company)E(x) = pici = 0.001 x -$500,000

+ 0.999 x +$750 = +$249.25 (a loss to the family and a gain to the company)


Risk Mitigation

Difficulties Applying ALE Estimates

Risk Managers’ Goals

Mitigating Infrequent Risks

Summary of Risk-Mitigation Strategies


Difficulties Applying ALE Estimates

Information about information assurance risks is very poorLittle or no mandatory reportingNo centralized databanksTherefore no actuarial statistics

Jacobson’s 30-Year LawPeople dismiss risks not personally

experienced in last 30 yearsKabay’s Paradox of Security

The better the security, the less direct evidence there is to support security measures


Risk Managers’ Goals Imagine wide range of risksTry to estimate consequences / costsAttempt to determine probabilities Identify risk-mitigation strategies and their costsCompute ALEs to estimate appropriate return on

investment (ROI)Generally focus on loss-avoidanceHowever, some loss-avoidance can reduce

costs to such a point as to provide overall increase in profitability

Also consider secondary effects such as improved customer relations, marketability, visibility in competitive marketplace….


Three Risk-Management Regions


Where ROI-Based Risk Mitigation is Effective

Works well for high-probability, low-cost risk exposuresRealistic appraisal by managersData are credible

Does not work well for low-probability, high-cost risk exposuresUpper management rarely understand

implications of information technology risks

“Who would have thought….” common reaction by upper management


Four Reasons for Adopting a Mitigation Strategy

1. Required by law or regulations

2. Cost trivial but significantly lowers probability

3. Addresses low-probability, high-cost event with unacceptable SOL (single-occurrence loss); e.g., consequence that wipes out organization

4. Cost of mitigation is more than offset by expected reduction in ALE (i.e., positive ROI overall compared with doing nothing)


Mitigating Infrequent RisksReduce magnitude of high SOLs*

Transfer risks using insuranceDisperse risk exposure (e.g., multiple ops

centers)Reduce vulnerability (e.g., BCP)

Mitigation selection processChoose low-cost measuresIgnore low risksUse insurance to spread cash flow over

years

_______________* Single-occurrence losses


Summary of Risk-Mitigation Strategies (1)


Summary of Risk-Mitigation Strategies (2)

IT staff may be unable to reduce ALE of high-probability/low-consequence risks

Midrange risks can be handled using mitigation measures chosen by evaluating their ROI using ALE calculations

Low-probability/high-cost risks involve evaluations of SOLs and mitigation measures to reduce probabilities further or reduce costs through planning and preparation

Ideally, risk management should bePerformed by expertsIndependent of IT managementReported to senior management directly


Risk Assessment Techniques

Aggregating Threats and Loss PotentialsBasic Risk-Assessment AlgorithmsLoss-PotentialRisk Event ParametersVulnerability Factors, ALE, SOL EstimatesSensitivity TestingSelecting Risk-Mitigation Measures


Aggregating Threats and Loss Potentials

Calculations of ALE can be increased in precision using aggregation of individual ALEs for specific components of systemsE.g., if manufacturers provide failure rates for

specific components (e.g., servers), these data can be helpful in estimating overall failure rates

One useful rule: probability P of failure of a system with independent units “i” where each has probability pi of failing is

P = 1 - (1-pi) which reduces to P = 1 – (1-p)n

for systems where all the units have the same pi


Loss-Potential

Loss potential can include costs of

Property damageLiabilityService interruption


Risk Event Parameters

Occurrence rate estimationRates often change after problems occurDon’t count events twice; e.g., if a power

failure causes a system crash, be careful not to count both of these separately

Look for external source of actuarial dataOutage duration affects costs

Service interruption increasingly important with e-commerce growing

EDI, Web purchases, multiple competitors….


Vulnerability Factors, ALE, SOL Estimates

Validating the estimates is importantCheck all the individual data and calculations

before basing decisions on mathLook for the risk event/loss potential pairs

that generate ~80% of total ALECheck assumptions – discuss with team

membersLook for outliers – extraordinarily large

contributors – and double-check them


Sensitivity Testing

Estimates of probability and costs are unlikely to be point-estimates

Can use range estimatesTry high, medium and low

If probability distributions are available, try Monte Carlo simulationRun random trials selecting values from

parameter distributionsPlot range of resulting ALEs to see central

tendenciesLook out for chaotic systems


Selecting Risk-Mitigation Measures

Address intolerable SOLsDiscard mitigation with negative ROIs (but

remember that insurance always has a short-term negative ROI)

Rank measures by descending benefits, costs, ROI


BREAK 17’13”


Methods for Qualitative Risk Assessment and Prioritization

QuestionnairesFocus groups InterviewsDelphi TechniqueComputer-Aided

Consensus


Limits of Questionnaires

Could a security questionnaire suffice as a risk assessment?Ask people for their opinionsCollate the results

ProblemsAmbiguities in use of words (“serious”,

“expensive”….Many questions prompt yes/no answers

but need more subtle distinctionsQuestionnaires miss points that arise in

open discussion with back-and-forth exchange of ideas


Designing Effective Questionnaires (1)

People unconsciously try to please othersMay give what they think/feel is expected

answerTend to answer “Yes” to whatever is asked

Therefore avoid leading questions“Do you think that the most important

issue in our security plans is employees?”Try “What is the most important issue in

our security plans?”


Designing Effective Questionnaires (2)

Some respondents will automatically check all the high or low answersAvoid having all scales in the questions

pointing same way (1 low – 5 high)Some respondents will lie

Introduce internal validationAsk same question in two different ways in

different parts of the questionnaireQ14 “Which of the following is the

lowest risk?”Q72 “Which of the following is the

highest risk?”


Designing Effective Questionnaires (3)Sometimes questions influence answers to

neighboring questionsE.g., giving a high estimate in Q22 may lead

to a higher answer in Q23Therefore prepare different versions of the

questionnaire which have different question sequences

Be careful about closed vs open questionsPre-determined scales may influence

answers (e.g., “Estimate the total cost / $1K , $10K, $100K, $1M, $10M” will skew results)

Can simply ask “Estimate the total cost” and let respondent choose range of answer


Focus Groups

Small group of people brought together to discuss thoughts, feelings, analyses of specific problem

Can be highly productiveNormally recorded and

analyzed in detail later Important to keep

atmosphere positive and open to all ideas

Can also use brainstorming techniques


BrainstormingGoal: generate ideas to solve

problemSeparate ideation from analysis

2 phases: find/create ideas then organize

IdeationSet numerical goal (e.g., “100 ideas on how

to. . .”No critical (negative or positive) responsesWrite every idea down on large paper

Including silly onesPost sheets on walls

100, now let`s find 10 more!”


Brainstorming (cont’d)

Facilitator encourages ideationScribe writes everything down

Also ideal to record discussionsParticipants

Should have means for making notes – avoid losing new ideas

Should not go into any detailCryptic suggestions are good

HitchhikingWhen an idea sparks a new one, use

hand signal to indicate priority (to avoid forgetting)


InterviewsOne-on-one discussions with appropriate

people“I need your help. Can you work with me

to identify key areas where we need to improve security from your perspective?”

Individuals know their own work better than anyone elseCan lead to deep insights

nto processOften have unspoken ideas on

problems and possible solutions


Analyzing Results from Focus Groups & Interviews

Sometimes have enormous mass of materialMay not know where to start in making sense

of findingsOffer proposals to panel of experts and ask

them to use Delphi Technique to come to consensus

Can also refine brainstorming using Computer-Aided Consensus™ (CAC)

Can use Computer-Aided Thematic Analysis™ (CATA) to sort through masses of ideas


Delphi Technique

RAND Corporation, 2nd World WarDevelop quantitative estimates using expert

opinionAsk top and bottom

quartiles to explain reasons

Share reasonsEstimate again Iterate to stability

Est

imat

es

Days

Top Quartile


Computer-Aided ConsensusReal-time Delphi TechniqueNeed a spreadsheet & printer or network1st, determine operational scale of importance

How much time?How much money?When to start?

Agree on simple scale; e.g.,1 = start this week OR spend $02 = this month $1,0003 = this quarter $10,0004 = this year $100,0005 = never $1,000,000


Computer-Aided Consensus (cont’d)

Lay out results of brainstorming or other listUse spreadsheet1 idea/proposal per row

Define 1 column per participant

Enter each participant’s estimate of importance / priority / value in column beside ideas / proposals

Can collect scores using printouts or using networked computers to fill in spreadsheets

Idea Bob Jane Karim Robbie

Javelin 2 4 3 1Halberd 3 3 3 1Morningstar 2 5 3 4Broadsword 5 2 3 2Pike 1 3 2 5Ballista 2 2 2 2Retarius 1 5 3 4Bombard 4 3 3 3

Idea Bob Jane Karim Robbie

Javelin 2 4 3 1Halberd 3 3 3 1Morningstar 2 5 3 4Broadsword 5 2 3 2Pike 1 3 2 5Ballista 2 2 2 2Retarius 1 5 3 4Bombard 4 3 3 3


Computer-Aided Consensus (cont’d)Calculate average and varianceSort descending by priority / importance

Idea Bob Jane Karim Robbie avg var

Morningstar 2 5 3 4 3.5 1.7Halberd 3 6 4 1 3.3 2.9Broadsword 5 3 4 3 3.0 2.0Ballista 2 2 2 2 3.0 0.7Pike 1 5 3 5 2.8 2.9Bombard 4 1 2 1 2.0 2.0Retarius 1 3 2 2 2.0 0.7Javelin 2 1 1 1 1.3 0.3



Group roughly by class of priority / importanceSort downward by variance within subgroupDiscuss reasons for greatest variation in

estimated priority / importance among most important proposals / ideas

Idea Bob Jane Karim Robbie avg var

Halberd 3 6 4 1 3.3 2.9Broadsword 5 3 4 3 3.0 2.0Morningstar 2 5 3 4 3.5 1.7Ballista 2 2 2 2 3.0 0.7

Pike 1 5 3 5 2.8 2.9Bombard 4 1 2 1 2.0 2.0

Retarius 1 3 2 2 2.0 0.7Javelin 2 1 1 1 1.3 0.3

Arbitrarily definedtop (most important)group

Why so much disagreement

?



Spend most time on important issues where people disagree

Discussing differences reveals new information about why people diverge:different assumptionsdivergent prioritiesunshared or contradictory informationdifferent reasoningerrors

Sharing info and resolving differences on important issues speeds consensus



Extremely important not to generate hostilityBest if spreadsheet visible for everyone

ProjectorNetwork with net-meeting software

Keep track of explanations for divergencesUse brainstorming techniques

Make process as dynamic as you canChange priorities in spreadsheet as often

as neededRecalculate and sort again


Computer-Aided Thematic Analysis (CATA)

How can we organize non-quantitative knowledge without imposing extrinsic framework

Extrinsic frameworks (preconceived notions)Can interfere with development of novel

insightMay mask data that don’t fit preconceptions

Intrinsic frameworksDevelop by examination of data themselvesWork with existing frameworks but go beyond

conventional ideas


Overview of CATA

Define themesWrite one theme per line in spreadsheetKeep track of originDevelop intrinsic framework for classificationApply preliminary classificationSortClassify again using finer granularityRepeat sort/classify until stableReport using synthetic paragraphs


Define Themes

“Theme”Any expression of fact, opinion or feeling“This project started 18 months ago.”“This project has been running too long.”“I hate this project.”

Break down all sentences at punctuation marks (. , ; : ! ? ) at some conjunctions (and, but)

Insert hard-return (line-break) to demarcate themesUse global find-replace function


Enter Themes into SpreadsheetPaste (copy) text into spreadsheet

One theme per line in spreadsheetKeep track of origin

1 column per source / person


Develop Intrinsic Framework for ClassificationRead through the (huge) list of themesJot down any word that

Occurs spontaneously to youCould help you organize themes

Look through categories or metathemesOrganize, order, number metathemesStick to 6-10 metathemes if possibleE.g., 1 Current status

2 Policy development3 Awareness program4 Psychological issues


Apply Preliminary Classification & Sort

Insert 2 columns to left of themesGenerate sequence number for

each lineTo keep connected themes

togetherClassify each theme by noting number (or

letter) of the appropriate metathemeSort entire list (including origin columns) by

Metatheme; andBy sequence number within metatheme

See next slides for illustrations


Preliminary Classification


Sort

By metathemeAnd by sequence number


Classify Again Using Finer Granularity & Sort

Examine each group of items under one metatheme

As required, subdivide metathemesGrouping helps identify subdivisionsOriginal metathemes become “n – 0”

Introduce another column into listingRate each theme according to two-part

metathemesDuplicate lines for more than 1 metathemeReorganize metathemes as needed

Can use find-replace function and sort to move whole blocks


Report Using Synthetic Paragraphs

When satisfied at organization of data, can start writing report

“-0” items serve as headings and sub-headings

Summarize findings and combine quotations that are in the same direction as one synthetic paragraph; e.g.,

“Most employees felt that management were listening to their feelings:

Managers really listen to us. They pay attention to our suggestions.”


DISCUSSION

1 Copyright © 2006 M. E. Kabay. All rights reserved. Rational Risk Management: Balancing Costs and Benefits of Security Measures Cybersecurity Conference.

Documents

model of risk risk mitigation

definitions risk

ale slide

risk mitigation difficulties

riskmitigation strategies

rational risk management

mitigation strategies

p i c i