Reporting Software Quality Mark Fewster, Grove Consultants
Jan 13, 2015
Reporting Software Quality Mark Fewster, Grove Consultants
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.2
Measuring software quality
what are we doing? assigning a number to something accuracy – guess / ruler / precision instrument
in testing we measure number and severity of defects found extent (coverage) of the software/system tested percentage of tests run (each cycle) number of test – debug – retest cycles time and effort taken non-functional attributes (e.g. performance, usability)
some measurements are more exact / objective than others
Measuring – lessons for testing
software quality has many aspects functionality, non-functional characteristics, defects
one measure doesn’t fit all!
choose the quality of your measurement gold / silver / bronze testing
reporting we report how bad – why not how good?
tests passed and predict future defect levels
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.5
How good is the software?
2.6
But this doesn’t tell us anything about the quality of the software now!
What about the unknown bugs?
We’ve tested it and we found 40 bugs.
38 bugs have been fixed. There are 2 bugs outstanding.
What about the tests that passed?
Known knowns and unknown unknowns
2.7
Known knowns: Unknown unknowns: bugs found bugs not found bugs shown not to exist (non-bugs) number of bugs not found
Knowing the unknowns
2.8
So which is it?
How can we tell?
Predict by using DDP!
Quote
2.9
“There is no point in asking how good the software is
until we know how good the testing was.”
Defects found in testing
Defects found in testing
Start Release
Not found - yet
How effective are we at finding faults?
Defects found after testing
or
Benchmark point
Defects found afterwards
“Really good!” “OK”
“Not bad” “Could be better”
“Poor” “Terrible!”
50 100 Defects found
after testing: Total defects
found:
Release
50 0
50 45 40 35 30 25 20 15 10 5 0
Defects Found
Time
62 12 62 81
69 19 69 72
74 24 74 68
77 27 77 65
85 35 85 59
87 37 87 57
88 88 38 57
10 Defects found
in testing: 42 50
Effectiveness at finding defects
100% 90% 80% 70% 60% 50% 40% 30% 20% 10%
0%
DDP
50 DDP = % =
Prediction of remaining bugs
20
10
66%
20
20
50%
20
5
80% Bugs found so far
DDP
Predicted bugs not found yet
Known knowns and known unknowns
2.13
Known knowns: bugs that have been found
Known unknowns: estimated bugs that have not been found
DDP 80%
Quote
2.14
Watts Humphrey, MIT
“If you want quality software out of testing,
you have to put quality software in to testing.”
Quality of testing
coverage has testing ‘covered’ all the important bits?
the functionality the non-function issues (performance, usability, etc.)
thoroughness has testing dug deep enough in each area?
typical data extreme data combinations
prioritisation has testing focused most on highest priorities?
2.15
Some stakeholders are not interested in bugs
bugs fixed are an historical detail they have been dealt with not about to influence what happens next
outstanding bugs are a ‘technical detail’ not always meaningful impact difficult to assess
consider car service
predicted bugs are hard to understand not meaningful
“something will go wrong, we don’t know what and we don’t know when”
impact impossible to assess 2.16
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.17
Risk-based testing
(product) risk is the possibility of a product failure describes possible consequences of product failures can express the impact on the business
identify product risks what could go wrong?
assess identified product risks how likely is it that the failure will happen? what would the impact be?
impact on end-user/business
use testing to mitigate highest level risks more thorough testing appropriate type of testing
2.18
Generic factors:
frequency of use business loss potential financial, ecological or social losses or liability civil or criminal legal sanctions safety concerns
fines, loss of license lack of reasonable workarounds visibility of the feature visibility of failure leading to negative publicity and damage to image loss of customers
1.19
Factors influencing business risk
¥
€
£ $
¥ €
£
$ ¥ €
£ $
Risks and tests
2.20
maintain traceability between risks and tests
when all tests for a risk have passed risk is mitigated
Risk 1
Risk 2
Risk 3
Test condition
Test condition
Test condition Test
condition Test
condition Test
condition
Test case
not run
passed failed
not run
passed failed
not run
passed failed
not run
passed failed
Test case
Test case
Test case
Risk 1: mitigated
Risk 2: outstanding
Risk 3: outstanding not run
passed
failed
passed
Risk-based reporting
2.21
Progress through the test plan
today end date
residual risks of releasing
TODAY
start
Source: “Risk Based E-Business Testing”, Paul Gerrard & Neil Thompson
all risks ‘open’ at the start
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.22
Benefit based test reporting
2.23
Open
Closed
Ris
ks
Open
Open
Closed
Closed
Open
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Ben
efit
Benefits available for release
Ben
efit
Ben
efit
Focusing on certain risks will deliver different benefits
Closed
Source: “Risk Based E-Business Testing”, Paul Gerrard & Neil Thompson
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.24
Agile approach
a user story describes functionality that will be valuable to a user or purchaser three parts (usually hand-written):
description notes of conversations acceptance tests
3.25 Front
Users can view list of account transactions for a selected period. Richard said to show transactions up to 12 months old.
Back
Users can view list of account transactions for a selected period. Richard said to show transactions up to 12 months old.
• Try with 1000’s of transactions. • Try with no transactions. • Try a 1-day period.
User story testing
each user story includes acceptance tests
completion criteria non-functional criteria an indication of size and complexity: story points
a small number of user stories are implemented within a single iteration
designed, developed, tested and demonstrated total number of story points = velocity
3.26
Content
measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary
2.27
Reporting software quality: summary
defect-based reporting is a negative view, can overlook what works is often a technical view
not always meaningful to the business
risk-based reporting assessment based on impact on business reporting mitigated risks more positive
benefit-based reporting focus on what can be delivered successfully
delivery-based reporting agile focused on user stories – what the user can do incremental approach, proof by demonstration
3.28
A final thought
3.29
“The accuracy of the reporting is dependent on the
quality of the testing.”