Exploratory Testing

Exploratory Testing – an improved approach

Jayapradeep Jiothis

Narendra Kumar

by

Agenda

Existing formalized methods.

Introduction.

Improvements.

Orthogonal arrays, Decision tables,State diagrams.

Team Management.

Applying in Corporate world.

Introduction

How is it different from Ad hoc Testing ?

What is Exploratory Testing ?

Where does ET fit ?

Characteristics of Exploratory tester.

Learning styles and ET.

Styles of Exploration.

Pros and Cons of ET.

What is ET ?

"Exploratory testing involves simultaneously learning,

"Exploratory testing is an interactive process of concurrent

planning, running tests, and reporting / troubleshootingresults."

- Dr. Cem Kaner.

product exploration, test design and test execution. To the

extent that the next test we do is influenced by the result

of the last test we did, we are doing exploratory testing.”

- James Bach.

How is it different from Ad hoc Testing ?

Ad hoc Testing:

“It is an impromptu and unrehearsed testing to uncover

the bugs with no mission or goal.”

0 100 Documentation(%)

Pure Scripted testingAd hoc testing Increasing degree of ET

Charters Vague Scripts

Ad Hoc Vs ET

Where does ET fit ?

No time to specify,script and test.

No specifications or little documentation available.

Thread Pulling.

ET can be associated with scripted testing to improve

To help assess risks.

When there is open communication between

When the application does not have any future releases.

the coverage.

development and testing teams.

Design and Excel :

An exploratory tester is first and foremost a Test

Characteristics of Exploratory Tester :

Designer.

The excellent exploratory tester is able to crafttests that systematically explore the product. One way to approach it is to consider it a questioning process. To design a test is to craft a question for a product that will reveal vital information.

Hawk eyes :

Excellent exploratory testers are excellent observers .

Characteristics of Exploratory Tester :…contd

The exploratory tester must watch for anything unusualmysterious, or otherwise relevant to the testing.Exploratory testers must be careful to distinguishobservation from inference(even under pressure) andavoid to allow preconceived assumptions to blind them.

Critical Thinking :


Break the Rules :

Excellent exploratory testers are able to review and

Exploratory tester clears their mind from the shackles of

explain their logic, looking for errors in their ownthinking.

the past, thinks creatively and laterally, to produce betterideas.Use heuristics to accomplish this.

Race up the learning curve :


Rich Resources :

Excellent exploratory testers climb learning curves more

Excellent exploratory testers build a deep inventory of

quickly than most. Intelligence helps but its more a matterof skill and practice that is build over a period of time.

tools,information sources, test data, and friends to drawupon. While testing, they remain alert for opportunities toapply those resources to the testing at hand.

Manage and Succeed :

Characteristics of Exploratory Tester : …contd

Keep Track :

Excellent exploratory testers know the value of their

Excellent exploratory tester keeps track of what was

precious time.They differentiate between a dead end and apromising lead. They follow their main mission whiletaking deviations to ensure the coverage.

tested, techniques used and risks the tests were intendedto explore, and how they relate to the mission of testing.

Learning Styles and ET:

Visual/Verbal : Preferred mode of information received.

Sensory/Intuitive : Preferred type of information.

Active/Reflective : Preferred organization of information.

Inductive/Deductive : Information processing approach.

Sequential/Global : Preferred way of understanding.

•This is based on an informal survey of a small number of testers by Andy Tinkham and Cem Kaner.

Reference “Learning Styles and Exploratory Testing”.

A learning style is a characteristic strengths and preferencesin the ways they take in and process information.

Sensory/Intuitive A sensory learner preferentially perceives information by the external senses.Examples :Sights , Sounds , Physical sensations.etc. The sensory-based person focuses on the details gained on actual observations. Inclined to develop a standard pattern for approaching exploratory testing.

An intuitive learner perceives information relies on the internal information (generated from memory, conjecture, and interpretation) and intuition. The intuitor focuses on the internal model , applies risk-based approach and different theories of error to the software.

Learning Styles and ET : …contd

Visual/Verbal

Visual learners perceive information from visual images such as pictures, movies, diagrams or demonstrations.They tend to work off an internal model that is picture-based.They make diagrams and pictures for their notes as they explore.

Verbal learners perceive information hearing (or reading) to lectures, written words, and mathematical formulas.They use a textual model like descriptions for their testing.

Learning Styles and ET : …contd

Inductive/Deductive

An inductive learner prefers to work from specifics and derive the generalities.They gather as many specifics as possible along with the historical data and generalize them to the application.

A deductive learner starts with the generalities and applies them to the specific situations they encounter.They apply guidelines and heuristics immensely in their approach.

Learning Styles and ET :…contd

Active/Reflective :

Active learners experiment with the information they receive and like to work in groups and find solutions. They do very rapid hands-on testing.They solicit the feedback from the teams.

Reflective learners prefer to think about information before they use it and tend to work alone.They have a thought process that precedes each test.They make up for their lack of speed by executing the “good” tests that are most likely to find bugs.


Sequential/GlobalSequential learners learn in a logically structured incremental manner. They follow “linear reasoning processes”. The sequential tester will seem to get off to a faster start and test plans will grow in complexity over time as the understanding deepens.

Global learners learn in chunks and have a holistic view of the information. A global tester will get off to a slower start. They need demonstrations on the application.Once the understanding is complete , he creates complex tests and can catch the bugs which are normally not seen.


Styles of Exploration :

Hunches

Examples

Models

Invariances

Interference

Error handling

Troubleshooting

Group Insight

Specifications

Hunches :

Intuitions

Interrogative

Analogous

Hearsay

Trace Changes

Styles of Exploration :…contd

Examples :

Use Cases

Simple Walkthroughs

Positive Thinking

Scenarios

Soap Operas


Models :

Architecture diagrams

Bubble diagrams

Relationships

Requirements definition

Functional relationship (for regression testing)

Failure models


Invariances :

Tests that change things that should have no impact

Example:

Printing a word document or a power point.


Interferences :

Interrupt

Alter

Terminate

Suspend

Switch

Contend.


Error Handling :

Check if errors and exceptions are handled correctly.

Specifications :

Heuristics

User Manual

Active reading


Group Insights :

Troubleshooting :

Bug Analysis.

Bug regression.

Brainstormed test lists

Group discussion of related components

Fishbone analysis

Paired Exploratory Testing


Pros and Cons of ET:

Responsive to changing scenarios.

Does not require extensive documentation.

Under tight schedules, testing can be more focused

Improved coverage.

Pros :-

depending on the bug rate or risks.

Pros and Cons of ET:

Test tracking not concrete.

Dependent on the tester’s skills.

More prone to human error.

No contingency plan if the tester is unavailable.

Cons :-

Formalised Methods of ET

Scripted Testing vs Formalised ET

What is Formalised ET ?

What does ET achieve ?

Approaches used for Formalised ET

What is Formalised ET ?A structured and reasoned approach to exploratory

Using the systematic approach, arrive at an outline of

Applying planning to Exploratory Testing, we have

what to achieve

what to attack

what would be the scope what effort to spend

testing is termed as Formalised ET. This approachconsists of specific tasks, objectives, and deliverablesthat makes it a systematic process.

Exploratory Planning.

Scripted Testing Vs ET

0 100 Documentation(%)

Pure Scripted testingAd hoc testing

Increasing degree of formalised ET

Charters Vague Scripts

What does Formalised ET achieve?

How much planning is done with ET?

How do you execute the tests with ET?

What kind of documentation is produced with ET?

How to use Pair Testing with ET?

Approaches used for Formalised ET

Identify the application domain

Identify the workflows

Identify the primary and secondary functions

Identify the UI standards

Identify the purpose

Identify the break points

Record failures

Identify the expected results

Identify the the interfaces with other interfaces/external

Document the issues and questions

Charter – states the goal and the tactics to be used

Decomposing the activites to smaller tasks

applications

Approaches used for Formalised ET…contd

Session Based Test Management (SBTM)

It’s the concept of Testing in Sessions

A session, is not a test case or bug report, is the basic

Reference :“Session-Based Test Management” by Jonathan Bach (first published in Software Testing and Quality Engineering magazine, 11/00)


testing work uninterrupted unit of reviewable,chartered test effort. It operates within a documentedprocess structure and produces records that complement verification documentation.

UnInterrupted : No significant disturbances during the testing( e.g mails, calls, meetings etc.).

Reviewable : A report , typically called a Session Sheet is produced on the testing performed which can be comprehended by competent authority and client.

Chartered : States the ‘What’ and ‘How’ of testing and the issues to look for.It is not suggested to make it a detailed plan.


We can separate test sessions into broadly 3 tasks(called the TBS metrics) :

…contdApproaches used for Formalised ET

Test design and Execution: Scanning the product and looking for problems.

Bug investigation and Reporting : Begins when the tester stumbles on something what typically looks to be a fault.

Session Setup : Setting up the test environment and improving the understanding of the product.

SBTM uses a very generalized light weight report called the Session Sheet.

A Session sheet consists of:

Session charter

Tester name(s)

Date and time started

Task breakdown (the TBS metrics)

Data files

Test notes

Issues

Bugs

…contdApproaches used for Formalised ET

Each session is debriefed.

During debriefing the manager reviews the session sheets, makes improvements to the charters, gets feedback from the testers and estimates and plans made for further sessions.


Agenda for debriefing session is :PROOF

Past : What happened during the session?Results : What was achieved during the session?

Obstacles : What got in the way of good testing?Outlook : What still needs to be done?

Feelings : How does the tester feel about all this?

Defect Driven Exploratory Testing:


*Reference: Defect Driven Exploratory Testing, Ananthalakshmi H.V,SISL

Defect Driven Exploratory Testing (DDET) is a goal oriented

Scripted Tests Bug Reports DDET

approach focused on the critical areas identified on the Defectanalysis study based on Procedural Testing results.


The Pre-Requisites for productive DDET are : In-depth knowledge of the product.

Procedural Testing has to be carried out. Defect Analysis based on Scripted Tests.

Key advantages of DDET are : Tester has clear clues on the areas to be explored.

No wastage of time. Goal oriented approach , hence better results.

DDET:


DDET:

2

3

7

12

8

3

3

3

200

0

246

81012

BUG COUNT

A B C D

ELEMENT/MODULE

HIGH

MODERATE

LOW

Scripted Testing


DDET:

2

4

9

022

220

3

5

4

0

5

10

15

BUG COUNT

A B C D

ELEMENT/MODULE

HIGH

MODERATE

LOW

Applying DDET

DDET:


Tim

e

Defects

Normal ET

Formalized ET

DDET

Improvements

Test Session Sheets

Metrics

Improved Approaches using ET

Using DDET / RDET.

Improvements

Test Session Sheets

Charter \ Mission Areas Covered Strategy Used Session and Tester details TBS metrics Test Notes Anomalies Questions Issues

Improvements…contd

Microsoft Word

Document

Session Sheet

Why “Test Session Sheets” ?

Managers can have track of quantum of testing.

Client can have track of billable hours.

Documented record for regulatory purposes.

Provides the structure for organization and management.

Prevents overlapping of the areas covered.

Acts as a feedback medium.

Standardized reporting mechanism across the organization.


Metrics

Helps determine Test Coverage details.

Helps in debriefing sessions. Helps the manager to plan resource and work allocation.

Provides a simple medium for traceability. Provides an insight into the quality of testing.

Helps collect important data for organizational purposes.


Microsoft Word

Document

Test Metrics

Coverage Metrics

Session Division :

Session Name

Date of session

Time Duration

S1

07/01/2004

09:00 Hrs

90 mins

S2

S3

S4


Group Division:

User groups :


Group Testers

G1 Joe, Mary

G2

G3

G4

User Group Acronym

Application Roles Description of the Role

U1 Admin Administrator

U2 User End user

Module/Element – Session Tracker:

(This table is used to track modules covered by different groups in the different sessions).


Session id/name

Module/Element Tested

M1 M2 M3 M4

S1 G1 G3 G4 G2S2 G2/G3 G1 G3 S3 G4 G2 G1 G3S4 G3 G4 G2 G1

Group – Role Tracker :


UserGroup

Groups

G1 G2 G3 G4

U1 X X

U2 X X

For Session S1:

Test Summary Metrics:


Can be used for designing better sessions and estimation in future .

Helps in knowing the maturity of the existing process and team.

Top management can track the progress of the project.

Microsoft Word

Document

Test Summary Metrics


What are Q-patterns ?

A set of related questions on a software requirement which

*Reference : www.WhatIsTesting.com by Vipul Kocher.

provides you with various alternatives that can beenhanced and reused.


Structure of a Q-Pattern

Name of the pattern Intent/Explanation/definition

Questions related to:

Administration

Performance

Usage/User UI Security

Examples Associated patterns

Specialization

ScriptedTesting

Using DDET / RDET:

Q patterns

DDET /RDET

ETScripted

Test cases

Review


Using Orthogonal Arrays, Decision tables and State Diagrams

Decision Tables

Orthogonal Arrays

State Diagrams

Orthogonal Arrays

Orthogonal Arrays Testing Strategy:The Orthogonal Array Testing Strategy (OATS) is a

It provides representative (uniformly distributed) coverageof all variable pair combinations.

systematic, statistical way of testing pair-wise interactions and deriving a suitable small set of test cases(from alarge number of possibilities).

Orthogonal Arrays…contd

Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array an even distribution of all the pair-wise combinations of values in the array can be tested.

Terminology for working with orthogonal arrays:Runs: The number of rows in the array which translates to the number of test cases that will be generated by the OATS technique.


Factors : The number of columns in an array. This directly translates to the maximum number of variables that can be handled by this array.

Levels : The maximum number of values that can be taken on by any single factor. An orthogonal array will contain values from 0 to Levels-1.

Strength : The number of columns it takes to see each of the Levels Strength possibilities equally often.Orthogonal arrays are represented by the pattern LRuns(LevelsFactors).


Steps to create Orthogonal Arrays:

Identify the independent variables that will be tested for interaction. This will map to the Factors of the array.

Identify the maximum number of values that each independent variable will take on. This will map to the Levels of the array.


Choose values for any "left over" Levels.

Transcribe the Runs into test cases, adding any particularly suspicious combinations that aren't generated.

Map the Factors and values onto the array.

Identify a suitable orthogonal array with the smallest number of Runs.(A suitable array is one that has at least as many Factors as needed from Step 1 and has at least as many levels for each of those factors as decided in Step 2.)


Example : Consider a screen with three drop downs list with three values.OA for the case is shown below

FACTORS

RUNS

Factor1 Factor2 Factor3

1 1 11 2 21 3 32 1 32 2 12 3 23 1 23 2 33 3 1

Decision tables

What are decision tables ?Decision table is a tabular representation of

Ideal for describing situations where varying conditions

combinational logic used to derive test cases andprovide a means of tracking complex combinations of conditions and resulting actions.

produce a combination of different actions.

Why use a Decision Table?

Supports manual or automated generation of test cases.

Provides a representation of requirements where aresponse or subset of the response is selected by evaluating many related conditions.

Effective at finding faults in both implementation andspecification.

Can support test design at all scopes from method tosystem.

Decision tables…contd

Decision tables

Add effects to the table

Steps to create Decision Table:

List all causes in the decision table

Calculate the number of possible combinations

Fill columns with all possible combinations

Reduce test combinations

Check covered combinations


Example : Decision Table of a credit card billing system


State Diagrams

What are State Diagrams ?State diagrams show the dynamic behaviour of a system.

Terminology for State Diagrams:

State: The state object is a snapshot of an object at a particular point in its life. A state may have an activity describing the function being performed.

The diagram shows the various states that an object canget into and the transitions that occur between the states.

Initial State : The initial state is the starting state of the object with reference to the behaviour that the diagram explains. Each state diagram should have only one initial state.

Final State : Each final state is the ending state of the object with reference to the behaviour that the diagram explains. There may be multiple final states for an object.

State Diagrams …contd

Transition : The transition link represents the relationship between different states of an object. The transition guard is a condition which limits the cases in which a transition can occur. The transition action is performed during the transition and cannot be interrupted.


Initial State:ATM

Machine

Enter PIN

Could not read the

card.

Transaction successful

“Tech” Failure

Transaction declined

Application Connected

NO Connectio

n

Amount Dispensed

Connection failed.

Transaction COMPLETE

transction ABORTED

Card inserted

Request Accepted

Request sentServer

Responding

Applic. negotiate

NO. Connectio

n

Appl. Failure

“Com.”

Failure

Card inserted

Valid PIN

Invalid PIN

ATM Cash withdrawal :


Team Management

Role of QA manager/Lead

Building a ET team

Pair Testing

Maintaining Morale & Accountability

Building an ET team

Look for diverse skilled resources.

Find the right mix

Convincing the experienced testers to join the ET team.

Technical and Domain knowledge.

Experience.

Learning Styles.

Role of QA manager/Lead

Decide the risk factors involved.

Decide the charters and sessions.

Hold debriefing sessions.

Deciding the strategies for ET.

Prepare the metrics.

Maintaining the morale of the team.

Updating the Q-Patterns.

Resource allocation.

Pair testing is a way of approaching a test design process

two people test the same thing at the same time and place

continuously exchanging ideas

high productivity, high creativity an effective complement to individual testing

one works the PC, the other makes creative suggestions

and writes deviation reports

generation of more and different ideas than either tester islikely to produce on his own

Pair Testing

Successful pair testing requires three specific conditions:

At least one tester is available who can be trusted to test

without supervision.

Another tester is available who can join the first tester for

a session of test design.

The two testers are otherwise capable of workingtogether.

Pair Testing …contd

Factors affecting Pair Testing:

Idea Exchange

Personal factors

Social factors

Attention flow

Test strategy

Pair Testing …contd

Maintaining Morale & Accountability

Develop an whale done attitude towards the tester.

Change the pairs regularly to avoid dependency.

Informal QC groups meetings to discuss the learnings, experience and suggestions for improvement.

Assigning challenging and varied tasks to avoidmonotony.

QA lead or the manager involved in testing/Coachingthe novice testers.

Exploratory testing in corporate world

Resource Time

Experience,Skill and knowledge

Available Documentation

Frequent Requirement changes

Cost and Quality

Morale

Personnel

Infrastructure

Any Questions please

Exploratory Testing – an improved approach Jayapradeep Jiothis Narendra Kumar by.

Documents

excellent exploratory

mission of testing

contd slide

unrehearsed testing

testing teams

vs et slide

internal information