(Risk based) test prioritization
Torbjørn Skramstad
How to prioritize
There are several ways to prioritize tests. We will, however, focus on risk based prioritization.
This lecture will cover• An introduction to risk assessment• How to use risk as a prioritization mechanism
for tests• A small example
Risk-Based Testing
• Risk-based testing is not a new concept. Most companies that develop or buy software think risk, albeit often in an unstructured, and undocumented manner.
• Only companies with a tradition for using risk analysis will use systematic methods for handling risk through analysis and testing.
Risk and stakeholders• Risk has only meaning as a relationship
between a system and its environment. Thus, what is a risk, how often it will occur and how important it is, will vary between stakeholders
• Even though the probability of an event is a system characteristic, the consequences will vary depending on the stakeholder’s perspective.
• Thus, we need to identify and prioritize all stakeholders before we start to discuss and analyze risk.
Stakeholders - 1
We have two main groups of stakeholders, each with their own concerns if a risk is allowed to become a problem, e.g.:
• Customers: lose money, either directly – e.g. an expensive failure – or indirectly – losing business.
• The company that develops the system: lose business – e.g. lose marked shares.
Stakeholders - 2
All stakeholders must be involved in the risk assessment process. They will have different areas of expertise and different experience. Thus, the methods we use must be:
• simple to use – no long training period needed.
• simple to understand – people don’t have confidence in something they don’t understand.
Risk identification • We can start with the system’s use cases.
Prepare the use case diagram for the function to be analyzed. Each participant should familiarize himself with the use case diagram.
• Start with a warm-up exercise, for instance going through results from previous risk identification processes. The warm-up exercise will help us to clear up misunderstandings and agree on a common process
ReviewTreatment
Plan
Doctor
ReviewDrug Data
ReviewDocuments
ReviewDiagnoses
OrderTests
LabSendTest
Results
Use case – high level example
Re(schedule) train
Update existingschedule
Control centeroperator
Change the scheduleof a train
Create newschedule
conflicting schedules
<<extends>>
Change the scheduleof a track maintenance activity
<<extends>>
Use case – detailed example
Flowchart level
Risk identification from use cases or flowchart/code
For each function we need to consider:• How can this function fail? Identify failure modes
and what part of the code will contribute to this failure mode.
• What is the probability that there are defects in this part of the code?
• Which consequences can this failure have for the stakeholders?
• Document results in a consequence table.• Assess the severity of each failure mode.
Subsystem: Consequences
Function Failure mode description Code involved User Cust. Dev.
What the user wants to achieve
How the function may fail
System parts involved.
How likely is it that they will cause the failure mode
Consequence Table
13
Risk assessment - 1
• Even though risk assessment is a subjective activity it is not about throwing out any number that you want.
• To be useful, a risk assessment must be– Based on relevant experience.– Anchored in real world data.– The result of a documented and agreed-upon
process.
14
Risk assessment - 2• Risk assessment uses the participants’
experience and knowledge to answer questions such as– Can this really happen; e.g. has it happened
before?– Can we describe a possible cause - consequence
chain for the event?– How bad can it get?– How often has this happened in the past?
How to make an assessment
We will look at the two main methods for risk assessment:
• Qualitative risk assessment based on– The probability / consequence matrix– The GALE (Globally At Least Equivalent) method
• Quantitative risk assessment – based on the CORAS model
16
Qualitative assessment
• We can assess consequences, probabilities and benefits qualitatively in two ways. We can use:– Categories – e.g.
• High, • Medium and • Low
– Numbers – e.g. values from 1 to 10. Note that this does not make the assessment quantitative – it is just another way to document the assessments.
17
Categories – 1
• When using categories, it is important to give a short description for what each category implies. E.g. it is not enough to say “High consequences”. We must relate it to something already known, e.g.– Project size– Company turn-over– Company profit
18
Categories – 2
Two simple examples:• Consequences: we will use the category
“High” if the consequence will gravely endanger the profitability of the project.
• Probability: we will use the category “Low” if the event can occur but only in extreme cases.
19
Consequences and probability - 1
Consequences
Probability H M L
H H H M
M H M L
L M L L
20
Consequences and probability - 2
• The multiplication table is used to rank the risks. It can not tell us how large they are.
• In the general case, we should only use resources on handling risks that are above a certain, predefined level – e.g. M or H.
Risk and consequences • Consequences should not be used alone for
ranking purposes – we also need the probability or frequency of occurrence. This should, however, be the same for all stakeholders.
• As long as we use qualitative measures for consequences and frequency or probability, we can use the same multiplication rules.
22
The GALE method – 1 • GALE – Globally At leas Equivalent• The GALE method is a method for making
decision about introducing or not introducing a change in e.g. a process or a construction.
• The principle is used to show that the changes have not increased the total risk
• The scoring scheme focuses on deviations from the current average. This is reasonable, given that the method is mainly concerned with comparing status quo to a new situation.
The GALE method – 2 • We will only use the scoring scheme for risk
assessment. This will not, however, free us from the need to compare the current system to something else - see statements such as:
• “The portion of occurring problems that have serious consequences is much larger than average”
• We will always need something to relate the problem to.
The GALE risk indexThe GALE risk index is computed based on our
assessment of an incident’s • frequency score – how often will the event occur.
The event here is a defect that has not been removed.
• probability score – what is the likelihood that the event will cause a problem
• Severity score – how serious is the problem. The risk index I = FE + PE + S (NB: logarithmic)
25
Frequency score for event Frequency
class Occurrences per project FE
Very frequent 200 Every project 6
Frequent 100 Every few projects 5
Probable 40 Every 10th project 4
Occasional 10 Every 100th project 3
Remote 1 A few times in the company’slifetime 2
Improbable 0.2 One or two times during thecompany’s lifetime 1
Incredible 0.01 Once in the company’slifetime 0
26
Probability score for problem
Classification Interpretation PE
Probable It is probable that this event, if it occurs, will cause a problem 3
Occasional The event, if it occurs, will occasionally cause a problem 2
Remote There is a remote chance that this event, if it occurs, will cause a problem
1
Improbable It is improbable that this event, if it occurs, will cause a problem 0
27
Severity score for event Severity
class Interpretation S
Severe The portion of occurring problems thathave serious consequences is muchlarger than average
2
Average The portion of occurring problems thathave serious consequences is similarto our average
1
Minor The portion of occurring problems thathave serious consequences is muchlower than average
0
28
The CORAS model – 1
• CORAS was developed as a framework for assessment of security risks.
• What should concern us here, however, is how CORAS relates the qualitative risk categories, not to absolute values, but to the company’s turn-over.
The 8 steps in CORAS• Step 1: The initial preparations for a risk analysis. • Step 2: The introductory meeting with the customer on the behalf of
which the analysis is conducted. • Step 3: Aims to ensure a common understanding of the target of analysis,
including its focus, scope and main assets. • Step 4: Ensure that the background documentation is correct and
complete • Step 5: Risk identification. • Step 6: Determine the risk level of the risks that are represented by the
identified unwanted incidents. • Step 7: Aims to decide which of the identified risks are acceptable, and
which of the risks must be further evaluated for possible treatment. • Step 8: Is concerned with the identification and analysis of treatments.
30
The CORAS model – 2 • Quantitative risks and opportunities give us
real values. The usefulness of this is, however, limited since it is – difficult to find real values for all risks. – not obvious how we can compare qualitative and
quantitative risks.
• When we use the CORAS tables, it is important to remember that developers, customers and users will have different values – e.g. different company incomes.
31
The CORAS consequence table
Consequence values
Category Insignificant Minor Moderate Major Catastrophic
Measured loss
related toincome
0.0 – 0.1% 0.1 – 1.0% 1 – 5% 5 – 10% 10 – 100%
Measuredloss due toimpact onbusiness
No impact onbusiness. Minor delays
Lost profits
Reduce theresources of oneor moredepartmentsLoss of a coupleof customers
Close downdepartments orbusinesssectors
Out ofbusiness
32
The CORAS frequency table - 1
• As we will see on the next slide, CORAS allows us to interpret frequency in two ways:– The number of unwanted incidents per year – e.g.
the number of times a function will fail.– The failing portion of demands – related to e.g.
number of service demands to a system.
33
The CORAS frequency table - 2Frequency values
Category Rare Unlikely Possible LikelyAlmostcertain
Number ofunwantedincidents peryear
1/100 1/100 – 1/50 1/50 - 1 1 - 12 > 12
Number ofunwantedincidents perdemand
1/1000 (1/500) 1/50 (1/25) 1/1
Interpretationof number ofdemands
Unwantedincidentneveroccurs
Eachthousandtime thesystem isused
Each fivetimes thesystem isused
Each tenthtime thesystem isused
Everysecondtime thesystem isused
IEC61508 – safety integrity levels and probability of incident
CORAS - Example
• Company annual income: NOK 100 000 000.— • Consequence of failure: Minor => 0.001 to 0.01• Frequency of failure: Possible => 1 per year – 2 per
100 year. • Max risk= 108 * 0.01 * 1 = NOK 1 000 000.—
– Minor – high end
• Min risk = 108 * 0.001 * 1/50 = NOK 2 000.—– Possible
An alternative consequence table
• We have been introducing risk based testing in a Norwegian company that develops software that is used in health care. This company have made a table including:– Service receiver – the patient– Service provider – the hospital– Developing company – e.g. the software
developers
Consequence level
Service receiver Service provider Developing company
High – 3
One person killed or several persons seriously injured
Several persons killed or many seriously injured
Bad pressLose customer
Medium – 2One person seriously injured
User looks bad to his superior(s) or large sums lost
Dissatisfied customers or
users
Low – 1 Minor irritant
Minor irritant, small amount of money lost or wasted
-
Consequence Levels
Earlier test experiences – Worry list
• Testing is a sampling process. Thus, if we find a lot of defects in a component or sub-system, we should conclude that this component probably has many defects.
• The conclusion will not necessarily be the same if we conclude based on the number of defects found in an inspection.
The Worry List
Consequences
High (3) Medium (2)
Low (1)
Number of errors
registered
High 10 -
Medium 3 – 9
Low0 - 2
Testing and resources
• We will always have limited resources for testing. This is made worse by the fact that testing often starts late in the project (often delayed)
• We make best use of the available resources by trying to minimize the total system-related risk.
Risk-Based Testing – 1
Risk-based testing has the following steps• Identify how the product interact with its
environment. This is necessary to understand failure consequences
• Identify and rank risks – probability and consequence. If this is a white box or grey box test we should also identify possible cause-event chains to understand the failure mechanisms.
Risk-Based Testing – 2
• Define tests that can be used to ensure that the code defect probability is low.– Black box test – apply e.g. random testing or
domain testing.– White box or grey box test – make sure that all
identified cause-event chains are exercised. • Run the tests – highest risk first
ATM example – 1
ATM example – 2 Subsystem: ATM transaction Consequences
Function Failure mode description Code involved
User Cust Dev.
Withdrawal Withdraw more than allowed
Wrong amount registered
Wrong account
W-1, M-1
Acc-1, M-1
Acc-2
L
H
L
M
H
H
H
H
H
Deposit Wrong amount registered
Wrong account
M-1, V-1
Acc-2
H
H
H
H
H
H
Inquiry Wrong account
Wrong value returned
Acc-2, V-1
V-1, M-1
M
M
L
M
M
M
ATM example – 3Frequency
class Occurrences per project FE
VeryFrequent
200 Every project 6
Frequent 100 Every few projects 5
Probable 40 Every 10th project 4
Occasional 10 Every 100th project 3
Remote 1 A few times in thecompany’s lifetime 2
Improbable 0.2 One or two times duringThe company’s lifetime 1
Incredible 0.01 Once in the company’slifetime 0
Classification Interpretation PE
Probable It is probable that this event, if it occurs, will cause a problem 3
Occasional The event, if it occurs, will occasionally cause a problem 2
Remote There is a remote chance that this event, if it occurs, will cause a problem
1
Improbable It is improbable that this event, if it occurs, will cause a problem 0
Function Components S FE PE IDeposit – wrong amount registered
M-1, V-1 2 4 3 9
Inquiry – wrong account
Acc-2, V-1 1 3 2 6
46
Severity score for event Severity
class Interpretation S
Severe The portion of occurring problems thathave serious consequences is muchlarger than average
2
Average The portion of occurring problems thathave serious consequences is similarto our average
1
Minor The portion of occurring problems thathave serious consequences is muchlower than average
0