Why Risk-Based Testing And TPI® Fails
Ruud Teunissen, Polteq, The NetherlandsW17
www.polteq.com 2
Why risk-based testing and test process improvement fails
Ruud TeunissenPolteq IT Services BV – The Netherlands
Manchester – December 6th 2006
www.polteq.com 3
Become a successful test manager by becoming a test leader
or
“All you ever wanted to know about becoming a test leader but everyone was afraid to tell you”
www.polteq.com 4
W e’re all fam iliar w ith these “T esting P rinciples”:
• Exhaustive testing is testing everything!
• Testing everything is impractical!
• Detect the most important defects as soon as possible (at the lowest cost)
• Prioritize testing so that whenever you stop testing you have done the best testing in the time available
www.polteq.com 5
S o w e stop testing if and w hen…
… the required level of confidence has been achieved
… the residual risks are acceptable
… reaching the test completion criteria (exit/success criteria) in term s of costs, tim e, coverage, … …
… the costs of finding the next defect exceed the costs the defect w ill cause “life”
www.polteq.com 6
H m m m , w hat w ill a defect cost “life”?
W hat if w e …… loose business… loose customers… loose reputation… loose trust… get bad publicity!… break the law…
Let’s ask the business and organise a w orkshop
Operational
Financial
Customer Satisfaction
Security
Regulatory
… …
www.polteq.com 7
W e cluster risks and m anage test activities, …
StakeholderTest Level End User IT Department …
Unit Test Changeability
Integration test Stability
System Test Accuracy RecoverabilityAnalysability Traceability
User Acceptance Test
SuitabilityLearnability Connectivity
Production Acceptance Test Time Behaviour Operability
Resource Behaviour
www.polteq.com 8
Low Medium High CriticalAccuracy• P rocessing DCoT
Error GuessingDCoT ECT
DCoTECT
• U ser Interface Error guessing Sample SEMError guessing
Sample both SEM and SYN
SEM and sample SYN
Efficiency The thoroughness of the RLT is variable and will thus be determined by the rating available as a consequence. and the amount of hours that comes
Security Error Guessing SEM sample user profiles
SEMuser profiles
SEMuser profiles and overall system
Usability Error Guessing and SUMI
Use Cases or PCT and SUMI
Use Cases or PCT and SUMI
PCT and Usability Laboratory Test
DCoT Data Combination Test SYN Syntax TestECT Elementary Comparison Test RLT Real Life TestSEM Semantic Test PCT Process Cycle TestSUMI Software Usability Measurement Inventory
… , choose the right T est D esign T echniques, …
www.polteq.com 9
… and create a test plan!
Product /System Part
Quality Characteristic
Test Priority
Test Design Techniques
InventoryAccuracy Critical
ECTSEM and sample SYN
Usability Medium PCT and SUMI
Efficiency Medium RLT
Invoicing Accuracy Low DCoTError guessing
CRMAccuracy Medium
DCoTSample SEMError guessing
Interoperability Medium PCT
Complete system Security High SEM, user profiles
www.polteq.com 10
B ut som ehow w e fail…
• TMapTest Strategy (Master) Test Plan TDT
• RRBTTest Strategy Clustering Priorities TD
• P rince 2, C obit, …“T esting should be perform ed and m anaged”
• F D A , …“T esting m ust be perform ed and m anaged”
www.polteq.com 11
… in im plem enting the chosen strategy!
Test Design Techniques
Test Preparation and Execution
ECTSEM and sample SYN
DCoTExploratory Testing
PCT and SUMI Questionnaire
RLT …
DCoTError guessing
DCoTExploratory Testing
DCoTSample SEMError guessing
DCoTExploratory Testing
PCT Trial and Error
SEM, user profiles User profiles
www.polteq.com 12
T he “reasons” are know n and plentiful
• Not enough time• Not enough budget• Involved too late• Specifications unclear and ambiguous• Bad software quality
- Have they even performed a unit test?• Inappropriate test environment• Reorganisation• … …
www.polteq.com 13
So we start to improve!!!
• Optimize the quality, costs and lead time of the total software development process
- S P I, C M M (I), S pider, B ootstrap, R U P , … …
• (Continuously) optimizing the quality, costs and lead time of the test process, in relation to the total information services
- Quality:• Timeliness, Coverage, Insight, Control
- Costs and lead time:• Risk-based, Early detection, Reuse, Collaboration,
Automation
www.polteq.com 14
W e use T M M or …
Level 2 Phase Definition•Institutionalize basic testing• techniques and methods•Initiate a test planning process•Develop testing and debugging goals
Level 3 Integration•Control and monitor the test process•Integrate testing into the lifecycle•Establish a technical training program•Establish a software test organisation
Level 4 Management & Measurement•Software quality evaluation•Establish a test measurement program•Establish an organisation-wide review program
Level 5 Optimization, Defect Prevention & QC•Test process optimization•Quality Control•Application of process data for defect prevention
Initial Testing Maturity Model and TMM are service marks of Illinois Institute of Technology
www.polteq.com 15
Key Area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 131 Test strategy A B C D2 Life-cycle model A B3 Moment of involvement A B C D4 Estimating and planning A B5 Test specification techniques A B6 Static test techniques A B7 Metrics A B C D8 Test automation A B C9 Test environment A B C
10 Office environment A11 Commitment and motivation A B C12 Test functions and training A B C13 Scope of methodology A B C14 Communication A B C15 Reporting A B C D16 Defect management A B C17 Testware management A B C D18 Test process management A B C19 Evaluation A B20 Low-level testing A B C
PlanImplement
improvement actions
Awareness
AssessmentDefine
improvementactions
Goal, scope and approach Evaluation
… the T P I m odel1 or even …
1 - TPI® is a registered trademark of Sogeti Nederland B.V.
www.polteq.com 16
… w hat w e learned from IS E B P ractitioner…
• Test policy• Project test plan • Phase test plan • Test strategy
Test policy
Project test plan
PTP PTP
Project test plan
PTP
Test strategy
www.polteq.com 17
B ut som ehow w e fail again…
• TMM• TPI• ISEB Practitioner• … …
• CMM(I)• SPI(DER)• BootsTrap• … …
www.polteq.com 18
→
• Management Expectations• Reality• Worst case
Expectations versus reality
www.polteq.com 19
“F ailures” and “successes”
• D id you or didn’t you involve the people w ho do the “real thing” in the test (im provem ent) process?
• A nd how did you involve them …
www.polteq.com 20
Y our testers are your success factor…
• They are the most important part of your test process• They have to learn to apply the new methods, tools and
templatesThey need to change the most!
• So create a training program aimed at - young, “eager”, am bitious, highly educated people
• But also include a coaching program for - the senior business testers, “heroes” and “fire fighters”
www.polteq.com 21
→
• Management Expectations• Reality• Worst case• “A chievable”
Maturity level
www.polteq.com 22
F irst steps : test m anagem ent…
• Make the test organisation more professional• Develop a test strategy• Follow a lifecycle: plan, prepare, specify and execute• Select the right tools• S elect adequate test design techniques …• According to the book!
Take into account the capabilities and potential of your test engineers.
Listen to what they have to say.Find out what they can achieve!
www.polteq.com 23
Required
Minimum
Possible
Actual status
Optimum
– Time– Budget– Business push– Quality RQMS– Technology push– # projects– Metrics
Maturity level
Accelerators/Limitations
TPI, accelerators and limitations
– Awareness/buy-in – M aturity of the organisation (R U P , P R IN C E 2, … )– Knowledge and skills– Ability to change
Plan
Do
Check
Act
www.polteq.com 24
Become test leaderD on’t stay behind as test m anager
The manager administers, the leader innovatesThe manager maintains, the leader develops
The manager relies on systems and tools, the leader on peopleThe manager counts on control, the leader counts on trust
The manager does things right, the leader does the right thing
www.polteq.com 25
Ruud TeunissenPolteq IT Services B.V. – The Netherlands
Manchester – December 6th 2006