Why Risk-Based Testing And TPI Failsconference.eurostarsoftwaretesting.com/wp-content/uploads/w17-3.pdfFirst steps : test m anagem ent… • Make the test organisation more professional

Why Risk-Based Testing And TPI® Fails

Ruud Teunissen, Polteq, The NetherlandsW17

www.polteq.com 2

Why risk-based testing and test process improvement fails

Ruud TeunissenPolteq IT Services BV – The Netherlands

Manchester – December 6th 2006

www.polteq.com 3

Become a successful test manager by becoming a test leader

or

“All you ever wanted to know about becoming a test leader but everyone was afraid to tell you”

www.polteq.com 4

W e’re all fam iliar w ith these “T esting P rinciples”:

• Exhaustive testing is testing everything!

• Testing everything is impractical!

• Detect the most important defects as soon as possible (at the lowest cost)

• Prioritize testing so that whenever you stop testing you have done the best testing in the time available

www.polteq.com 5

S o w e stop testing if and w hen…

… the required level of confidence has been achieved

… the residual risks are acceptable

… reaching the test completion criteria (exit/success criteria) in term s of costs, tim e, coverage, … …

… the costs of finding the next defect exceed the costs the defect w ill cause “life”

www.polteq.com 6

H m m m , w hat w ill a defect cost “life”?

W hat if w e …… loose business… loose customers… loose reputation… loose trust… get bad publicity!… break the law…

Let’s ask the business and organise a w orkshop

Operational

Financial

Customer Satisfaction

Security

Regulatory

… …

www.polteq.com 7

W e cluster risks and m anage test activities, …

StakeholderTest Level End User IT Department …

Unit Test Changeability

Integration test Stability

System Test Accuracy RecoverabilityAnalysability Traceability

User Acceptance Test

SuitabilityLearnability Connectivity

Production Acceptance Test Time Behaviour Operability

Resource Behaviour

www.polteq.com 8

Low Medium High CriticalAccuracy• P rocessing DCoT

Error GuessingDCoT ECT

DCoTECT

• U ser Interface Error guessing Sample SEMError guessing

Sample both SEM and SYN

SEM and sample SYN

Efficiency The thoroughness of the RLT is variable and will thus be determined by the rating available as a consequence. and the amount of hours that comes

Security Error Guessing SEM sample user profiles

SEMuser profiles

SEMuser profiles and overall system

Usability Error Guessing and SUMI

Use Cases or PCT and SUMI

Use Cases or PCT and SUMI

PCT and Usability Laboratory Test

DCoT Data Combination Test SYN Syntax TestECT Elementary Comparison Test RLT Real Life TestSEM Semantic Test PCT Process Cycle TestSUMI Software Usability Measurement Inventory

… , choose the right T est D esign T echniques, …

www.polteq.com 9

… and create a test plan!

Product /System Part

Quality Characteristic

Test Priority

Test Design Techniques

InventoryAccuracy Critical

ECTSEM and sample SYN

Usability Medium PCT and SUMI

Efficiency Medium RLT

Invoicing Accuracy Low DCoTError guessing

CRMAccuracy Medium

DCoTSample SEMError guessing

Interoperability Medium PCT

Complete system Security High SEM, user profiles

www.polteq.com 10

B ut som ehow w e fail…

• TMapTest Strategy (Master) Test Plan TDT

• RRBTTest Strategy Clustering Priorities TD

• P rince 2, C obit, …“T esting should be perform ed and m anaged”

• F D A , …“T esting m ust be perform ed and m anaged”

www.polteq.com 11

… in im plem enting the chosen strategy!

Test Design Techniques

Test Preparation and Execution

ECTSEM and sample SYN

DCoTExploratory Testing

PCT and SUMI Questionnaire

RLT …

DCoTError guessing

DCoTExploratory Testing

DCoTSample SEMError guessing

DCoTExploratory Testing

PCT Trial and Error

SEM, user profiles User profiles

www.polteq.com 12

T he “reasons” are know n and plentiful

• Not enough time• Not enough budget• Involved too late• Specifications unclear and ambiguous• Bad software quality

- Have they even performed a unit test?• Inappropriate test environment• Reorganisation• … …

www.polteq.com 13

So we start to improve!!!

• Optimize the quality, costs and lead time of the total software development process

- S P I, C M M (I), S pider, B ootstrap, R U P , … …

• (Continuously) optimizing the quality, costs and lead time of the test process, in relation to the total information services

- Quality:• Timeliness, Coverage, Insight, Control

- Costs and lead time:• Risk-based, Early detection, Reuse, Collaboration,

Automation

www.polteq.com 14

W e use T M M or …

Level 2 Phase Definition•Institutionalize basic testing• techniques and methods•Initiate a test planning process•Develop testing and debugging goals

Level 3 Integration•Control and monitor the test process•Integrate testing into the lifecycle•Establish a technical training program•Establish a software test organisation

Level 4 Management & Measurement•Software quality evaluation•Establish a test measurement program•Establish an organisation-wide review program

Level 5 Optimization, Defect Prevention & QC•Test process optimization•Quality Control•Application of process data for defect prevention

Initial Testing Maturity Model and TMM are service marks of Illinois Institute of Technology

www.polteq.com 15

Key Area / Scale 0 1 2 3 4 5 6 7 8 9 10 11 12 131 Test strategy A B C D2 Life-cycle model A B3 Moment of involvement A B C D4 Estimating and planning A B5 Test specification techniques A B6 Static test techniques A B7 Metrics A B C D8 Test automation A B C9 Test environment A B C

10 Office environment A11 Commitment and motivation A B C12 Test functions and training A B C13 Scope of methodology A B C14 Communication A B C15 Reporting A B C D16 Defect management A B C17 Testware management A B C D18 Test process management A B C19 Evaluation A B20 Low-level testing A B C

PlanImplement

improvement actions

Awareness

AssessmentDefine

improvementactions

Goal, scope and approach Evaluation

… the T P I m odel1 or even …

1 - TPI® is a registered trademark of Sogeti Nederland B.V.

www.polteq.com 16

… w hat w e learned from IS E B P ractitioner…

• Test policy• Project test plan • Phase test plan • Test strategy

Test policy

Project test plan

PTP PTP

Project test plan

PTP

Test strategy

www.polteq.com 17

B ut som ehow w e fail again…

• TMM• TPI• ISEB Practitioner• … …

• CMM(I)• SPI(DER)• BootsTrap• … …

www.polteq.com 18

→

• Management Expectations• Reality• Worst case

Expectations versus reality

www.polteq.com 19

“F ailures” and “successes”

• D id you or didn’t you involve the people w ho do the “real thing” in the test (im provem ent) process?

• A nd how did you involve them …

www.polteq.com 20

Y our testers are your success factor…

• They are the most important part of your test process• They have to learn to apply the new methods, tools and

templatesThey need to change the most!

• So create a training program aimed at - young, “eager”, am bitious, highly educated people

• But also include a coaching program for - the senior business testers, “heroes” and “fire fighters”

www.polteq.com 21

→

• Management Expectations• Reality• Worst case• “A chievable”

Maturity level

www.polteq.com 22

F irst steps : test m anagem ent…

• Make the test organisation more professional• Develop a test strategy• Follow a lifecycle: plan, prepare, specify and execute• Select the right tools• S elect adequate test design techniques …• According to the book!

Take into account the capabilities and potential of your test engineers.

Listen to what they have to say.Find out what they can achieve!

www.polteq.com 23

Required

Minimum

Possible

Actual status

Optimum

– Time– Budget– Business push– Quality RQMS– Technology push– # projects– Metrics

Maturity level

Accelerators/Limitations

TPI, accelerators and limitations

– Awareness/buy-in – M aturity of the organisation (R U P , P R IN C E 2, … )– Knowledge and skills– Ability to change

Plan

Do

Check

Act

www.polteq.com 24

Become test leaderD on’t stay behind as test m anager

The manager administers, the leader innovatesThe manager maintains, the leader develops

The manager relies on systems and tools, the leader on peopleThe manager counts on control, the leader counts on trust

The manager does things right, the leader does the right thing

www.polteq.com 25

Ruud TeunissenPolteq IT Services B.V. – The Netherlands

Manchester – December 6th 2006