8/4/2019 Test Process Improvement - Paul Gerrard
1/27
2000 Systeme Evolutif Ltd Slide 1
Test Process Improvement
Paul Gerrard
Systeme Evolutif Limitedemail: [email protected]://www.evolutif.co.uk
2000 Systeme Evolutif Ltd Slide 2
Agenda
u Introduction
uWhat Can Be Improved?
uTest Methodology
uTest Organisation Maturity (TOM)
and Testing Assessment
u Implementation.
8/4/2019 Test Process Improvement - Paul Gerrard
2/27
2000 Systeme Evolutif Ltd Slide 3
Introduction
2000 Systeme Evolutif Ltd Slide 4
Is testing a strategic issue?
uTesting, in its broadest sense, comprises
more than 50% of your development cost
and has influence over all these risks
uSoftware development productivity isincreasing, testing productivity decreasing
uTester psychology at odds with developer
and most management mentality
uDemand for, appreciation of testing and
testers is increasing.
8/4/2019 Test Process Improvement - Paul Gerrard
3/27
2000 Systeme Evolutif Ltd Slide 5
Testing perceptions
uTesting is innate - anyone can do it
everyone tests, but it takes a good tester to find
bugs in good code
anyone can find bugs in poor quality code
uTester psychology is counter-intuitive,
destructive, all-wrong
uGood testers are awkward employees
uSome code doesnt need testing
uTesting doesnt add value.
2000 Systeme Evolutif Ltd Slide 6
Testing perceptions (2)
uSoftware is getting easier to test
uSoftware quality is improving
uTesting has little value - destructive,
pointless
uTesting is random and cannot be
systematised
test techniques are prone to mathematical
treatment
development is still a craft, in comparison.
8/4/2019 Test Process Improvement - Paul Gerrard
4/27
2000 Systeme Evolutif Ltd Slide 7
Some of the problems to be
addressed
uToo many errors being found in production
or being reported by customers
uTesting is taking too long, and delaying
delivery
uTesting is too expensive
uDifficulty in finding volunteers to test
uTesters dont see a career path in your
company.
2000 Systeme Evolutif Ltd Slide 8
What can be Improved?
8/4/2019 Test Process Improvement - Paul Gerrard
5/27
2000 Systeme Evolutif Ltd Slide 9
Testing know-how
uTraining is the quick-win stream of work
uFocus on principles, process and techniques
uTraining and qualification schemes are
emerging (ISEB scheme in the UK):
grades of certification
industry-relevant syllabus
accredited training organisations, courses and
instructors
examination schemes.
2000 Systeme Evolutif Ltd Slide 10
Standards and procedures
uServe four main purposes:
a consistent baseline for the quality of testing
across the organisation
alignment with your development anddeployment processes
appropriate use of industry best practices
provide a shortcut to using the best, most
appropriate techniques.
8/4/2019 Test Process Improvement - Paul Gerrard
6/27
2000 Systeme Evolutif Ltd Slide 11
Organisation
uStreamlining testing usually implies
fostering closer liaisons between
projects and software users
developers, maintainers and support groups
uA common improvement is to create an
independent test group
set up to offer testing services to other
departments, projects.
2000 Systeme Evolutif Ltd Slide 12
Environment
uExisting test environments are often
inadequate and need improvement
scale - too small to run some tests
control - chaos is a barrier to systematic testing isolation - some tests too risky to implement
dedication - shared environments problematic
uTools can make testers more productive
the only realistic way to regression test.
8/4/2019 Test Process Improvement - Paul Gerrard
7/27
2000 Systeme Evolutif Ltd Slide 13
Improvement investment
uAll improvements require investment
uPay back by ensuring resources are focused
uBenefits vary with type of improvement and
by installation
e.g. one site found that ...[There was] a seven-
fold pay-back for the cost of the training over a
three month period.
2000 Systeme Evolutif Ltd Slide 14
Test Methodology
8/4/2019 Test Process Improvement - Paul Gerrard
8/27
2000 Systeme Evolutif Ltd Slide 15
Typical test strategy
Integration
System
Acceptance
Unit
Ad hoc
2000 Systeme Evolutif Ltd Slide 16
V model: waterfall and locks
Requirements
Logical
Design
Physical
Design
Code and test
Integration
System test
Acceptance
test
8/4/2019 Test Process Improvement - Paul Gerrard
9/27
2000 Systeme Evolutif Ltd Slide 17
System
Integration
Typical test practice
Acceptance
Unit
Ad hoc x
2000 Systeme Evolutif Ltd Slide 18
Time/Cost
Development LiveRunning
Economics of errors
8/4/2019 Test Process Improvement - Paul Gerrard
10/27
2000 Systeme Evolutif Ltd Slide 19
Front-loading
uThe principle is to start testing early
uReviews, walkthroughs and inspections of
documents during the definition stages are
examples of early tests
uStart preparing test cases early. Test case
preparation tests the document on which
the cases are based
uPreparing the user manual tests the
requirements and design.
2000 Systeme Evolutif Ltd Slide 20
Front-loading advantages
uRequirements, specification and design
errors are detected earlier and are therefore
less costly
uRequirements more accurately captured
uTest cases are a useful input to designers and
programmers
uSpreads the workload of test preparation
over the whole project.
8/4/2019 Test Process Improvement - Paul Gerrard
11/27
2000 Systeme Evolutif Ltd Slide 21
Early test case preparation
Requirements
Logical
Design
Physical
Design
Code and test
Integration
System test
Acceptance
test
Prepare
tests
Preparetests
Prepare
tests
2000 Systeme Evolutif Ltd Slide 22
Testing throughout the lifecycle: the W model
AcceptanceTest
Code
BuildSoftware
Install
Unit Test
Write
LogicalDesign
PhysicalDesign
Test the
Test theDesign
Test theSpecs
IntegrationTest
System
Test
BuildSystem
RequirementsRequirements
8/4/2019 Test Process Improvement - Paul Gerrard
12/27
2000 Systeme Evolutif Ltd Slide 23
Acceptance
Test
Code
BuildSoftware
Install
Unit Test
Wri te
LogicalDesign
Physical
Design
Test the
Test the
Design
Test theSpecs
IntegrationTest
SystemTest
BuildSystem
RequirementsRequirements
Static test techniques
Inspections StaticAnalysis
Requirements
Animation
BehaviourAnalysis
ScenarioWalkthroughs
Reviews
Inspections
EarlyTest CasePreparation
2000 Systeme Evolutif Ltd Slide 24
AcceptanceTest
Code
BuildSoftware
Install
Unit Test
Wri te
LogicalDesign
Physical
Design
Test the
Test the
Design
Test theSpecs
IntegrationTest
SystemTest
BuildSystem
RequirementsRequirements
Dynamic test techniques
FeaturesTesting
Performance,
TestingVolume, Stress
Restart & Recovery
TestingInstallation
PathTesting
LoopTesting
Input ValidationTesting
Multi-userTesting
SecurityTesting
Equivalence PartitioningBoundary Value Analysis
Cause-effect Graphing
Transaction Flows
8/4/2019 Test Process Improvement - Paul Gerrard
13/27
2000 Systeme Evolutif Ltd Slide 25
Test Process Assessment
2000 Systeme Evolutif Ltd Slide 26
Information gathering
uNeed to determine:
where you are today
where you want to go tomorrow
uData gathering using interviews
managers, developers, testers, users
checklist type and open-ended questions
uOther research
examination of project and test plans, test
records, fault reports.
8/4/2019 Test Process Improvement - Paul Gerrard
14/27
2000 Systeme Evolutif Ltd Slide 27
The challenges of testing
improvement
uMajor barriers are organisational and
personal, not technical
changing management perceptions so they
support testing and improvement
overcoming management and practitioner
resistance to change
design and implementation of workable
processes and management controls
uPeople, not tools, implement test strategies.
2000 Systeme Evolutif Ltd Slide 28
Problems or symptoms?
uNeed to separate problems from symptoms:
management doesn't understand the objectives of
testing
the cost of testing is high but difficult to pindown
developers, testers, users may never have been
trained
the quality of the product delivered into testing
is poor, so takes longer to system test.
8/4/2019 Test Process Improvement - Paul Gerrard
15/27
2000 Systeme Evolutif Ltd Slide 29
Improvement mix
uA mix of improvements is most likely to be
required:
management awareness
tester training
improved definition of the test stages and their
objectives
measurement of the quality of the product at
each stage etc. etc.
2000 Systeme Evolutif Ltd Slide 30
Staged improvements
uNot all improvements are a good idea
straight away
some improvements are expensive
some save time, but bring dramatic change some improve the quality of the testing, but take
longer to implement
uVery few improvements save time, improve
quality, cause minimal change and pay back
after two weeks.
8/4/2019 Test Process Improvement - Paul Gerrard
16/27
2000 Systeme Evolutif Ltd Slide 31
TOM
uRecommended improvements must take
account of broader organisational objectives,
constraints and priorities
uTest Organisation Maturity model.
2000 Systeme Evolutif Ltd Slide 32
Test Organisation Maturity
(TOM)
and Testing Assessment
8/4/2019 Test Process Improvement - Paul Gerrard
17/27
2000 Systeme Evolutif Ltd Slide 33
Process maturity
uProcess maturity is a product-independent
measure of an organisations capability
uFramework for process improvements
uWhat are the most sensible things to improve
next?
uAssessment: where are we now?
uMonitoring: are we there yet?
2000 Systeme Evolutif Ltd Slide 34
Capability Maturity Model (CMM)
uCMM for software widely adopted
uUsed to assess capability and identify
improvements
uMaturity 'levels' define a graduated scale and
provide a roadmap of improvements
uCMM represents a high-level, process-
oriented description of the capability of a
development organisation.
8/4/2019 Test Process Improvement - Paul Gerrard
18/27
2000 Systeme Evolutif Ltd Slide 35
CMM (2)
uHeritage of the CMM is large, long term,
defence projects in the US
uSEI agenda remains high-integrity, but has
little to say about testing
uRelevance to commercial IT organisations is
often tenuous
uMost organisations at levels 1 (0, -1?)
uWhose process is it anyway?
2000 Systeme Evolutif Ltd Slide 36
Several testing maturity modelsexist
uTest Process Improvement (TPI) Model,
Koomen, Pol (Holland)
20 key areas scored, improvement suggestions
uTestability Maturity Model, Gelperin, USA 20 key areas, score between 1-2
uTesting Maturity Model, Burnstein,
Suwannasat and Carlson, USA
aligned with the CMM
8/4/2019 Test Process Improvement - Paul Gerrard
19/27
2000 Systeme Evolutif Ltd Slide 37
Problems with existing models
uRemedy oriented, not problem-oriented
here are the pills, which are you taking?
heres the solution, does it solve the problem?
uLittle guidance on priorities or constraints
hackers and high integrity get same treatment
assumes a click and go mentality and culture
no comparison of costs v benefits
uFor process hypochondriacs - you didnt
know you had a problem until now.
2000 Systeme Evolutif Ltd Slide 38
TOM
uTest Organisation Maturity model - TOM
uAssessment
self assessment or consultant-assessment
questions based on symptoms of poor testing
assessment score from 20-100
uBuilt-in improvements model
improvements selected, based on assessment
improvements prioritised, based on assessment.
8/4/2019 Test Process Improvement - Paul Gerrard
20/27
2000 Systeme Evolutif Ltd Slide 39
TOM assessment to action plan
Symptoms(current maturity)
Priorities(target maturity)
Action Plan
u Document objectives and
constraints and prioritise
u Identify the testing-related
problems
u Which problems cause
most concern?
u Select improvementswhich best meet
objectives.
Objectives/
Constraints
2000 Systeme Evolutif Ltd Slide 40
Assessment questions
uQuestions focus on organisational rather than
technical issues
can be answered by management or practitioners
(try both and compare!)uQuestions relate directly to the symptoms
how bad is the problem? (the score)
how much does it hurt? (the priority).
8/4/2019 Test Process Improvement - Paul Gerrard
21/27
2000 Systeme Evolutif Ltd Slide 41
Improvement objectives and
constraints
uDecrease time required to test
uDecrease cost of testing
u Increase quality of testing (and systems)
uMinimal change to current practice
uQuick payback
uGive each a priority of 1-5 (low-high).
2000 Systeme Evolutif Ltd Slide 42
Example assessment symptom
Score 1 Tests are not based on requirements or design documents,
there are no test inventories or means of measuring coverage
against requirements or specifications
Score 3 Test inventories are used to define the scope of system and
acceptance tests and cross-reference requirements, formal
test techniques are sometimes used to design black-box test
cases.
Score 5 Test inventories are used for all testing and are reviewed
against requirements and specifications, formal test
techniques are used for test case design, tools are used to
measure code coverage
There are gaps in the testing - features of thesystem may be released untested
8/4/2019 Test Process Improvement - Paul Gerrard
22/27
2000 Systeme Evolutif Ltd Slide 43
Score 1 Errors are found in acceptance tests which should have
been found in sub-system and system tests
Score 3 Errors are found in system tests which should have been
found in sub-system tests
Score 5 Errors found would not be expected to have been
detected earlier
Of the errors that are found, there is a
perception (based on evidence) that manyshould have been found in earlier test stage(s).
Example assessment symptom (2)
2000 Systeme Evolutif Ltd Slide 44
Potential improvements
uTrain developers, system testers in testing
u Improve test design by adopting techniques
u Involve users in definition of system tests
uMotivate developers to do better testing
uSeparate code and test activities in plans.
8/4/2019 Test Process Improvement - Paul Gerrard
23/27
2000 Systeme Evolutif Ltd Slide 45
Objective/Constraint Score Description
Decrease time required to test 0 Increase dev.test
Decrease sys.test
Decrease cost of testing 0 No change
Increase quality of testing(and systems)
+1 Yes
Minimal change to currentpractice
-1 Change likely
Quick payback +1 Yes
Improvement: train developers,
system testers in testing
2000 Systeme Evolutif Ltd Slide 46
Implementation
8/4/2019 Test Process Improvement - Paul Gerrard
24/27
2000 Systeme Evolutif Ltd Slide 47
Pilot projects
uTest the improvements and gain experience
uPilot projects should be:
low risk, in a familiar area, low criticality
in their earliest stages - budget yet to be set
project managed by someone willing and
capable of managing additional responsibilities
staffed by people willing to learn and apply new
techniques, and capable of providing objectivefeedback.
2000 Systeme Evolutif Ltd Slide 48
Pilot project objectives
uGet consistent and comprehensive incident
and defect information for a project
where are the bugs found?
where should the bugs have been found?
u Identify where costs are incurred in testing
uMeasure the amount of testing done
uReduce or eliminate duplicated tests
uReduce the frustrations of testing.
8/4/2019 Test Process Improvement - Paul Gerrard
25/27
2000 Systeme Evolutif Ltd Slide 49
Pilot projects - preparation
uProject manager needs to be briefed and
measurable objectives for the pilot agreed
uProject staff
need to be aware of the purpose of doing things
differently
training in the use of unfamiliar process,
techniques and tools
uNeed to provide support to the projectmanager and staff involved.
2000 Systeme Evolutif Ltd Slide 50
Reviewing the results of the pilot
u If the pilot objectives were met and the
changes accepted, then the changes are ready
for roll-out to other projects
uOtherwise, the conduct of the pilot should be
investigated:
were the objectives too ambitious?
could the changes ever have delivered the
benefits anticipated?
8/4/2019 Test Process Improvement - Paul Gerrard
26/27
2000 Systeme Evolutif Ltd Slide 51
Roll-out - refinement and full
implementation
uRefinement and re-issue of materials based
on pilot experiences
u Internal publicity of the results of the pilot
and the imminent roll-out of the changes
uSchedule and conduct training
uNominate members of the pilot project and
implementation team as consultants to other
projects.
2000 Systeme Evolutif Ltd Slide 52
Post-implementation review
uWere benefits achieved? If not why not?
u If you exceeded expectations, celebrate!
uAre there new opportunities for additional
improvements?
uAre additional refinements required?
uWhat would you do different? The same?
u Is testing now providing the information to
provide an informed decision to release?
8/4/2019 Test Process Improvement - Paul Gerrard
27/27
2000 Systeme Evolutif Ltd Slide 53
TOM assessment forms can be
downloaded and completed at
www.evolutif.co.uk