“The Future of Software Testing” Structured or Agile, … › wp...Structured or Agile, The Best of Both Worlds Derk-Jan de Grood, Collis, The Netherlands Europe’s Premier Software

Structured or Agile, The Best of Both

WorldsDerk-Jan de Grood,

Collis, The Netherlands

Europe’s Premier Software Testing EventWorld Forum Convention Centre, The Hague, Netherlands

WWW.QUALTECHCONFERENCES.COM

“The Future of Software Testing”

Leiden Singapore Minneapolis Edinburgh

Best of both worlds

By: Derk-Jan de Grood

Date: November 2008

Location: EuroSTAR

Exploratory testing in a structured way

www.collis.nl 3

Objectives for this presentation

Some basic principles of exploratory and structured testing

Insight in the way different approaches can benefit from each other

Answer to the question „Why should I ?‟

Mail exchange

DJ: I got the idea, also during discussion with John Bach, that in the US a lot of Exploratory testing is done as a separate thing. Thus, although structured it is not embedded in a test method. Do you disagree?

Lee: Depends on the organization –some see it as two different approaches (even done by different people). With others ET is very integrated. It‟s difficult to generalize.

www.collis.nl 4

Regional differences ?

www.collis.nl 5

Administrative systems

Procedural testingProducts

Technical testing

6

Schools of testing

Analytic School

Aka Structural testing

Code coverage

Verification

Standard School

Sees testing as a way to measure progress with emphasison cost and repeatable standards

Make sure each requirement is tested

Quality School

Emphasizes process and acts as Gatekeeper

Protect users from bad software

Testing is a stepping stone to process improvement

Context-Driven School

Emphasizes people,seeking bugs that stakeholders care about

Exploratory testing

Commercial market driven software

Agile School

Uses testing to prove that development is complete; emphasizes automated testing

Test Driven development

[Pettichord 2007]

7

Which school would you apply ?

Which school would apply for testing :

Auto-pilot system for airplane (analytical)

SAP system (standard)

Website supporting a campaign (context)

Innovative system (agile)

Mobile phone (standard?)

Off-shore build system (quality?)

No specs available (context or agile)

Game (context)

Result-driventesting

8

Schools of testing

Analytic Standard

Quality

Context

Agile

[Pettichord 2007]

9

N-dimensions

Maturity

Organization

Formality

Aim of testing

Technology

Development

method

Organisation type

Testteam

(skills)Key risk area’s

(important quality attributes)

Time to

market

3 approaches for testing

Unstructured

Scripted

Exploratory

www.collis.nl 10

www.collis.nl 11

Exploratory Testing

Scripted vs. Exploratory

www.collis.nl 12

100 % Exploratory 100 % Scripted

13

What is Exploratory Testing?

An approach for unscripted testing based upon skills and experience of the tester. ET is a risk based technique using a formal procedure, test charters and heuristics.

The no 1. excuse for not having to prepare our test design in full detail: We do exploratory testing !

“Exploratory testing is simultaneous learning, test design, and test execution.”James Bach

Real live situation

A Colleague said:

“The specs are outdated and incomplete. There is a need for manual regression scripts. In preparation phase our test scripts only contains the basic info (test purposes). During execution we write down the details”

www.collis.nl 14

www.collis.nl 15

A test design technique

16

Traditional techniques

Error in the s/wCoverage of the testdesign

Error that remains undiscovered

Error found with used techniques

17

ET

Error in the s/wCoverage of the testdesign

1. Points of Interest (POI)

2. First tests executed

3. Plan next step basedupon test results

4. Define new POI

5. Cont. with next POI

6. Conclusion thus finished !

Process and charters

www.collis.nl 18

Assign Charters

Preparation Test execution Session evaluation

Debriefing

Test Charters

Define Charters

www.collis.nl 19

Best of both worlds

20

Test plan

Result

General

Test

strategy

Test

estimate &

planning

Test Risk

Analysis Detail Test plan

(test project

specific test

strategy)

Test strategy

www.collis.nl 21

Which information is

available?

cWhat kind of error

is expected ?

What is the impact

when this error occurs

while operational ?

Select a technique

that covers the

right domain

Select a technique

with right test

depth

TRA

Test basis

Test technique

Select technique

for which

information is

available

22

Building our test design

Syntax

PCT

BVA

EP

Exploratory

“the puzzle changes the puzzling.”James Bach

23

ET fits y/n ?

Y Rapid feedback

Extension to scripted tests

Find most important bug in shortest time

Check work of other tester

Little or no specifications

Domain knowledge available

N Feedback loop breaks down

Detailed calculations

(life) Critical functions

Testing of reliability/performance

Test ware is important

Testers are less skilled

Sourc

e:

ISEB P

ractitioner

–Im

pro

ve Q

S

Test Risk Analysis

www.collis.nl 24

Traceability

www.collis.nl 25

Registration

www.collis.nl 26

Test charter 123

Test reporting

www.collis.nl 27

Befit based reporting

www.collis.nl 28

Progress reporting

www.collis.nl 29

www.collis.nl 30

Why should I ?

31

Evaluation

The ET session gave us clear understanding of the quality of the system. This was achieved in a very short period.

The fun about ET is that its fundamentals are easily understood.

Jaap Azier (KPN)

In order to use ET effectively we need to take the lessons learned into account. In special the logging and scenario testing.

Still I am glad we did ET. It enabled other people to get insight in the quality of the system.

Carin Smits (KPN)

Exploratory testing is testing on the edge.ET means taking the most out of people, this implies your dealing with people issues.It is exciting to find the edge of „we have tested all the essential‟.

Hugo Achthoven Implementation Manager (KPN)

The project went well, great team working. Together we worked towards the best working method. In the end, we certainly have found it. This resulted in clear and traceable test results.

Jasper OvergaauwTestexpert (Collis)

Close collaboration between users and test expert provided a judgement on the quality of the system, in a only short term.

Matthijs JorissenTestexpert (Sogeti)

Focus on result

www.collis.nl 32

Exploratory Testing

A nice extension of your tool-box

Don‟t let your message get blurred

www.collis.nl 33

www.collis.nl 34

Wrap-up

www.collis.nl 35

Questions?

Derk-Jan de [email protected]

CollisDe Heijderweg 1 2314 XZ LeidenThe Netherlands

www.testgoal.com

Leiden Singapore Minneapolis Edinburgh