HELSINKI UNIVERSITY OF TECHNOLOGY T-76.5613 Software Testing and Quality Assurance Lecture 10, 11.10.2006 Exploratory Testing (ET) vs. Test-Case Based (scripted) Testing Juha Itkonen SoberIT
HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.5613 Software Testing and Quality Assurance
Lecture 10, 11.10.2006
Exploratory Testing (ET)vs.
Test-Case Based (scripted) Testing
Juha ItkonenSoberIT
2Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Testing without predefined test casesManual testing
Based on experience, knowledge and skills of the testerWithout pre-documented test steps (detailed test cases)
Exploring the software or systemGoal is to expose quality-related informationAllowing tester to adjust plans and follow hunches
Based on the results and knowledge gained from the testing
Minimizing time spent on (pre)documentationTesting (execution) is not something that anyone masters
Exploratory Testing (ET) is
3Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory Testing vs. Scripted Testing
ET is an approachMany testing techniques can be used in exploratory wayExploratory testing vs. scripted testing are the ends of a continuum
Freestyle exploratory “bug hunting”
Pure scripted(automated) testing
Manual scripts
High leveltest cases
Charteredexploratory testing
4Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Scripted vs. Exploratory Testing
In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.
In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.
You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave
Tests
Product
Tests
James Bach, Rapid Software Testing, 2002
5Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Describing Exploratory Testing
In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.
In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.
James Bach, Rapid Software Testing, 2002
To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory testing. James Bach, 2001
Exploratory testing involves simultaneously learning, planning, running tests, troubleshooting and reporting results. Cem Kaner, 2001
6Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Characterizing Exploratory Testing
1. Tests are not defined in advance as detailed test scripts or test cases
Exploration with a general mission without specific step-by-step instructions on how to accomplish the missionAn exploratory tester uses any available information of the target of testing
requirements document, user manual, marketing brochure, …
2. Exploratory testing is guided by the results of previously performed tests and the gained knowledge from them
3. The focus in exploratory testing is on finding defects by exploration
Instead of systematically producing a comprehensive set of test cases for later use
4. Exploratory testing is simultaneous learning of the system undertest, test design, and test execution
5. The effectiveness of the testing relies on the tester’s knowledge, skills, and experience
7Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
ET and dimensions of testing
Pure ad hoc “bug hunting”
Completely scriptedtesting (automated)
No planningRigorously plannedon the level of individualtest cases
No tracking Coverage metrics andprogress tracking byplanned and executed testcases
No reporting or documentation
Detailed reports oneach executed test andall results, defects, issuesand comments
Destructive,revealing defects
Constructivedemonstrating benefitsbuilding confidence
ET
8Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exploring is asking questions and questioning
In exploratory testing tests are like questionsQuestioning skills are at the heart of ETExploratory tester proceeds by asking questions of the software under test
And uses the answers to guide testing and find the next questions
Lets play “20 questions”I think of some “thing” and you try to find out that “thing” by asking 20 questions that I can answer either ‘yes’ or ‘no’
Imagine having to write down those questions beforehand
9Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Scripted vs. Exploratory TestsMine-field analogy
bugs fixes
10Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Lateral thinking
Allowed to be distractedFind side paths and explore interesting areasPeriodically check your status against your mission
11Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Dimensions of testing
Testing combines techniques that focus onTesters: who does the testingCoverage: what gets testedPotential problems: why you're testing (what risk you're testing for)Activities: how do you testEvaluation: how to tell whether the test passed or failed
All testing involves all five dimensionsHowever, each testing technique typically covers only one or two
Black Box Software Testing (Course Notes, Academic Version, 2004) www.testingeducation.org
Focus of the Exploratory Testing approach
12Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
ET as part of test lifecycle
Planning & Control
Prep
arat
ion
Spec
ificat
ion
Exec
utio
n
Com
plet
ion
Pol & Veenendal. Software Testing A guide to TMap Approach. 1998.
Planning & Control
Prep
arat
ion
Spec
ificat
ion
Exec
utio
n
Com
plet
ion
Planning & ControlPr
epar
atio
n
Spec
ificat
ion
Exec
utio
n
Com
plet
ion
Prep
arat
ion
Spec
ificat
ion
Exec
utio
n
Com
plet
ion
Planning & Control
ET
13Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Well-timed test design
Early test designtest design finds faultsfaults found early are cheaper to fixmost significant faults found firstfaults prevented, not built inno additional effort
re-schedule test designtest design causes requirement changes
Not too early test case designDesign in implementation order
All tests do not have to be designed before implementation startsStart test design from the most completed and best understood features
Avoiding anticipatory test design and deprecated, incorrect test cases that are not based on the actual featuresTest planning and design must begin early, test case designnot necessarily
14Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Information objectives
Exploratory testing is good for finding new informationYour testing can serve different information objectives
Find defectsMaximize bug countHelp managers make release decisionsBlock releasing premature softwareAssess qualityConform to regulationsFind safe scenarios to use the product…
A good exploratory tester will write down test ideas and use them in later test cycles
to create stronger and more efficient tests for the application in the future
15Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory Function Testing
Use list of functions to give structure and high level guide to your testing
Requirements specificationFunctional specificationUser manual
Explore creatively each individual function and interactions of functions
Cover side paths, interesting and suspicious areasTests are designed simultaneously with test executionUse the list of functions to get back on track
Coverage and progress is planned and tracked by functions
Not by test cases
16Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory testing vs. ad-hoc testing?
Exploratory testing can be a planned, goal driven and managed process – not an ad-hoc processTesting approach, goal, and coverage can be well plannedDetailed test log is created during the testing and defects are reported
Test report should be reviewable by another person in order to determine what was tested, how and what are the results
Exploratory test design and execution easily seem like ad-hoc activities
ET is ad hoc at the level of test case designAd hoc actually means “fit for purpose and situation”
NOT unmanaged, sloppy, vague, or randomSession-based testing is an approach to manage this creative and flexible process
17Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Session Based Test Management (SBTM)
CharterTime BoxReviewable ResultDebriefing
18Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Session-Based Testing
Enables planning and tracking exploratory testingWithout detailed test (case) designsUses time-boxed testing sessions as the planning and tracking mechanism instead of test cases and suites
Planning and tracking testing work in small chunks Dividing testing work in smaller tasksTracking testing work in time-boxed sessions
Can help getting testing done when resources are scarceEfficient – no unnecessary documentationUtilizes tester’s skills and experience during test executionAgile – testing is easy to focus to most important areas based on the test results and other information
Changes in requirements, increasing understanding, revealed problems, …
19Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Charter
Designing the Charters is test planning and designBrief information / guidelines on:
What should be tested?Areas, components, features, …
Why do we test this?Goals, focus
How to test (approach)?Specific techniques or tactics to be usedTest data to be used
What problems to look for?
Might include guidelines on:Tools to useWhat risks are involvedDocuments to examineDesired output (information objectives) from the testing
20Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Time Box
Focused test effort of fixed durationNo interruptions or other simultaneous tasks
Brief enough for accurate reportingBrief enough to allow flexible schedulingBrief enough to allow course correctionLong enough to get solid testing doneLong enough for efficient debriefingsBeware of overly precise timing
Short: 60 minutes (+-15)e.g. Normal: 90 minutes (+-15)
Long: 120 minutes (+-15)
21Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Reviewable results
CharterTest notes
What was tested, howReproducibilityWhat needs more testing, new test ideas, etc.
Bugs and issuesData filesEffort breakdown
Duration (hours:minutes)Test design and execution (percent)Bug investigation and reporting (percent)Session setup (percent)Charter / opportunity (percent/percent)
22Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Debriefing
The test lead reviews session notes to assure that he understands it and that it follows the process
Results of the sessionWhat was tested and howOther quality information
The tester answers any questionsSession metrics are checkedCharter may be adjustedSession may be extendedNew sessions may be charteredCoaching / mentoring happens
23Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory Testing in Literature
No reliable scientific research resultsAcademic research on software testing focuses on test case design techniques, automation, and optimization
24Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Literature study:Applicability of Exploratory Testing
When the test cases cannot be determined in advanceRapid feedback or learning of the product is neededTesting from the end-user viewpointProviding more diversity to scripted testing
When test scripts become “tired”, i.e., they are not detecting many defects anymore.
Exploring the size, scope, and variations of a found defect Regression testing based on defect reportsInvestigating the status of particular risksNot enough time for systematic approaches(When competent testers are not available)
25Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Literature study:Claimed Benefits of Exploratory Testing
Utilises creativity and skills of a testerUtilizing experienceFinding important defectsYou see more when not following a script
Increased effectiveness in terms of number and importance of found defects
Optimized to find bugs (fast)Minimizing time spent on documentation (low-cost)
Simultaneous learning of the systemHelp testers to come up with more powerful tests
FlexibilityMinimized documentation before test executionRe-focusing on the most promising risk areasFollowing hunchesBeneficial when requirements or design changes
Ability to perform testing without finished or comprehensive documentationRapid flow of feedback from testing
To both developers and testers
26Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Literature study:Identified Shortcomings of Exploratory Testing
Difficulty of tracking progressIndividual testersThe whole testing process
Tracking, planning, and measuring coverageExploratory testing has no ability to prevent defects through early test design
27Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Results of a multiple case studyItkonen and Rautiainen. "Exploratory Testing: A Multiple Case Study", ISESE, 2005.
What practitioners from industry think about ETHow they use ETWhy they use ET
What benefits they feel it provides
What challenges they face using ET
28Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Ways of utilizing ET
Session-based exploratory testingFunctional testing of individual featuresExploratory smoke testingFreestyle exploratory testing
Unmanaged ET as part of other dutiesExtending test cases
Outsourced exploratory testingAdvanced users, strong domain knowledgeReal use scenarios
Exploratory regression testingby verifying fixes or changes
29Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Reasons for using ET
Writing detailed test cases for everything is difficult and laborious
The software can be used in so many ways or there are so many combinations between different features
With ET it is easy to test from the user’s viewpointET is a natural way of testing, which enables utilizing the testers’ experience and creativity during testingET enables quick feedback on new features from testers to developersET adapts well to changing situations
Where the requirements and the tested features change often, and the specifications are vague or incomplete
ET enables learning about the systemWhich can be utilized in other tasks, such as customer support and training
30Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Perceived benefits of ET
ET is more versatile and goes deeper into the tested features
Testing things that would not be documented as test casesDependencies with existing features based on experience and knowledge
Test cases typically based on new requirementsRegression testing fixes
ET is an effective way of finding defectsRequires experienced testerEven if test cases are used the defects are found by exploringEasier to achieve a destructive attitudeIn the long run might be inefficient
ET gives a good picture of the overall quality of the system in a short timeET enables testing the look and feel of the system
31Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Perceived shortcomings of ETCoverage was the biggest shortcoming
Planning and selecting what to testPrioritizing and allocating time of the domain expertsTracking the coverage of performed tests
Are all new features coveredAre all ways the user could use the system covered
Reliance on the expertise and skills of an individual tester
Human errorsDifferent ways of testing (also a strength)Difficulty of finding good domain experts
Repeatability of found defectsLong sequences of actions that lead to failureMemory leaks and other “non-deterministic” failures
32Juha ItkonenSoberIT/HUTHELSINKI UNIVERSITY OF TECHNOLOGY
Conclusions
ET is an accepted approach to testing in industryET seems to be an effective way of finding defects
At least when a session-based approach is used
Support for the benefit of utilizing testers’ creativity and experienceWe did not find any intentionally applied techniques or strategies used in ETUsing ET to learn the system for other purposes than testing was a new finding
Customer service and training
Biggest shortcoming is the challenges of planning and tracking test coverage