Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January
Post on 15-Mar-2018
223 Views
Preview:
Transcript
1
A Rapid Introduction to Rapid Software Testing
Michael BoltonDevelopSenseSTeP-IN Conference
Bengalooru, IndiaJanuary 2007
2
Who I Am
Michael Bolton(not the singer, not the guy in Office Space)
DevelopSense, Toronto, Canada
mb@developsense.com+1 (416) 992-8378
http://www.developsense.com
3
Acknowledgements
James Bach, senior author of this course− http://www.satisfice.com− Co-author of Lessons Learned in Software Testing− James and I both teach this course, of which I am co-author
Much of this presentation is influenced by the work of our colleagues and mentors, including Jerry Weinberg, Elisabeth Hendrickson, Doug Hoffman, Payson Hall
4
Acknowledgements
Cem Kaner, Ph.D., J.D.− http://www.kaner.com− http://www.testingeducation.org
− Senior author of Lessons Learned in Software Testing− Cem presents an online course on Black Box Software Testing,
written by himself and James Bach− http://www.testingeducation.org/BBST/index.html− university level− online videos, course notes, supplementary reading
−free
5
Rapid Testing
Rapid testing is a mind-setand a skill-set of testing
focused on how to do testingmore quickly,
less expensively, with excellent results.
This is a general testingmethodology. It adapts to
any kind of project or product.
6
KEY IDEA
The tester’s job is
to think critically about the software,
so that we can provide information,
so that management can make informed decisions
about the product and the project.
7
Quality is value to some person.
A bug is anything that threatens the valueof the product.
These definitions are designed to be inclusive.Inclusive definitions minimize the chance that testers will inadvertently overlook an important problem.
Testers light the way.
8
This is our role.We see things for what they are.
We make informed decisions about quality possible, because we think critically about software.
Testers light the way.
45
9
The “questions” consist of various ways of configuring and operating the product. (Asking questions about the product without operating it is usually called review rather than testing)The product “answers” by exhibiting behavior, which the tester observes and evaluates.To evaluate a product is to infer from its observed behavior howit will behave in the field, and to identify important problems in the product.
Testing is questioning a product in order to evaluate it.
Testers light the way.
10
Questions = tests; right answers = credible observationsTraditionally, testing has been more focused on getting the right answers, through carefully documented, controlled, and repeated tests (confirmation)BUT… there are so many questions worth asking (investigation), we have to consider the tradeoffs between ultimate care in getting the right answers and the danger of simply not doing enough testing.
Asking the right questionsvs.
Getting the right answers
Two Sides of Testing
11
What is a Test?
The test is what youthink and do!
The test is not the test script.
Only a fraction of a skilled tester’s thinkingand behavior can be represented in a script
(about which more later).
12
KEY IDEA
“Rapid Testing” means
the fastest,
least expensive testing
that fulfills the mission.
13
The Triangle Program
This program takes three numbers as input.The numbers represent the dimensions of a triangle.When you click on the check button, the program tells you what kind of triangle the sides represent:
− scalene (no side equal to any other)− isosceles (two sides are equal)− equilateral (all sides are equal)
I want you to test this program.Any questions?
14
Some Follow-Up Questions
What was your mission?What problems did you find?How did you know they were problems?How did you find them?What questions did you ask?Can you report credibly?Did you need more time?Is the Triangle program good?
15
The Universal Test Procedure, v1.0
“Try it and see if it works.”
Get itSet it upRun itRun it again, maybe
Choose where to lookSee what’sthereSee what’s not there
Read the specSee if the product matches the specFind problems……especially the bad ones
Procedures OraclesCoverage
16
Tests
ProjectEnvironment
ProductElements
QualityCriteria
PerceivedQuality
A Heuristic Test Strategy Model
17
ProjectEnvironment
Tests
ProductElements
QualityCriteria
PerceivedQuality
A Heuristic Test Strategy Model
18
The Universal Test Procedure, v2.0“Try it and see if it works.”
There are numerous waysof doing these
things. Patterns thatdescribe how to do
this are called“test techniques.”
1. Model the test space.2. Determine coverage.3. Determine oracles.4. Determine test procedures.5. Configure the test system.6. Operate the test system.7. Observe the test system.8. Evaluate the test results.9. Report test results.
19
Assumptions of Rapid Testing
Testers have at least some control over the design of tests and some time to create new tests.One goal of testers is to find important bugs fast.Testing is done under conditions of uncertainty and severe time pressure.Testers have control over how they think and what they think about.Testers are, or become, very good at software testing.
20
A Rapid Tester’s Mandate
To be able to test a productwhen it has to be tested right now,under conditions of uncertainty,
in a way that stands up to scrutiny.
21
A Rapid Tester’s Mandate
Testers should always be thinkinglike expert testers.
Do that well, and they’ll beable to handle any testing situation.
22
The Themes of Rapid Testing
Put the tester's mind at the center of testing.Learn to deal with complexity and ambiguity.Be a service to the project community, not an obstacle.Consider cost vs. value in all your testing activity.Develop your testing skills.Use heuristics to guide and structure your process.Diversify your testing and your test team.Your context should drive your choices, both of which evolve over time.
23
-Idea-Idea
…
What Does Rapid Testing Look Like?Skill + Heuristics Makes Powerful Testers
1. Do this2. Then do this3. Then do this4. Then do this5. And then this…
This…
…not this.Hey! Testing is not
a clerical process.
A heuristic is a falliblemethod for solving
a problem.
24
Rapid testing may not be exhaustive,but it is thorough enough and quick enough. It’s less work than ponderous testing. It might be less work than slapdash testing.
It fulfills the missionof testing.
How does Rapid Testing Compare with Other Kinds of Testing?
Management likes to talkabout exhaustive testing, butthey don’t want to fund it andthey don’t know how to do it.
You can always test quickly...
But it might be poortesting.
When testing is turned into an elaborate set of rote tasks,
it becomes ponderous without really being thorough.
Mor
e W
ork
&
Tim
e(C
ost)
Better Thinking & Better Testing(Value)
SlapdashMuch faster, cheaper,
and easier
PonderousSlow, expensive,
and easier
RapidFaster, less expensive,
still challenging
ExhaustiveSlow, very expensive,
and difficult
25
Is Rapid Testing Agile?
Two cheers for AgilityWe love the idea that programmers write unit tests− that saves us from having to report silly bugs that could be, should
be caught early
We like the idea that programmers are even talking about testing
Testing is not merely rote confirmationTesting is also (and, I believe, more importantly) active investigation
26
KEY IDEA
27
Do You Test UnderTime Pressure?
You have less time than you’d like.Faster is better.Your plans are often interrupted.You are not fully in control of your time.
What is this like?
28
Excellent rapid technical workbegins with you…
When the ball comes to you…Do you know you have the ball?
Can you receive the pass?
Do you know what yourrole and mission is?
Do you know whereyour teammates are?
Do you know your options?
Is your equipment ready?
Can you read the situation on the field?
Are you aware of the criticality of the situation?
Are you ready to act, right now?
Can you let your teammates help you?
29
Rapid test teams are about diverse talents cooperating− We call this the elliptical team, as opposed to the well-rounded team.− Some important dimensions to vary:
− Technical skill− Domain expertise− Temperament (e.g. introvert vs. extrovert)− Testing experience− Project experience− Industry experience− Product knowledge− Educational background− Writing skill
− Diversity makes exploration far more powerful− Your team is more powerful because of your unique individual contribution
…but you don’t have to be great at everything.
30
The Tester is at the Center
skill
procedureenvironment
supervisionThe greater the skill of
the tester, the lessprescribed procedure,
supervision, orfavorable environment
is required.
31
Rapid Testing MindsetAssess Cost vs. Value
VALUE− Coverage. The value of the elements of the product that are exercised by the test.
− Power. The probability that if a problem exists in the area covered, the test will reveal it.
− Pop. The potential for it to reveal unanticipated problems as well as those you anticipate.
− Integrity. The degree to which the test can be trusted, because its results are valid and consistentacross multiple executions of that test.
− Technical Necessity. The degree to which the information we want to discover exists; and the degree to which that information matters.
− Business Necessity. The degree to which a test is expected to be performed by your clients.
COSTS (remember that time = money)− Opportunity Cost. Performing it may prevent you from doing other tests.
− Development. The cost of getting ready to perform the test.
− Execution. The cost of performing it as needed.
− Transfer. The cost of getting someone else ready to run that test.
− Maintenance. The cost of keeping it running.
− Accounting. The cost of explaining the test, justifying it, show that it was performed.
32
Happy PathDocumentation TourSample Data TourVariability TourComplexity TourContinuous Use
A quick test is a cheap test that has some valuebut requires little preparation, knowledge,
or time to perform.
Cost vs. ValueTry quick tests as well as careful tests
InterruptionsUnderminingAdjustmentsDog PilingFeature InteractionsAsk for Help
33
Input Constraint AttackClick FrenzyShoe TestBlink TestError Message HangoverKeyboard Tour
A quick test is a cheap test that has some valuebut requires little preparation, knowledge,
or time to perform.
Cost vs. ValueTry quick tests as well as careful tests
Folder TourResource StarvationMultiple InstancesCrazy ConfigsCheap ToolsBuild Your Own Tools
34
Blink Tests
Informed by Malcolm Gladwell’s BlinkBlink tests are based on removing some information and using human pattern recognitionBlink test example: ProSum− thank you, Earl Everett!
Blink test example: Excel SheetBlink test example: another Excel Sheet
30
35
Blink Precepts
Snap judgments are a central part of our decision-making processSnap judgments are vulnerable to corruption by forces we don’t recognize or understandSometimes we can improve the quality of our judgments by removing informationInstead of changing the decision-maker, change the context in which the decision is made
30
36
Motivations for Quick Tests
Finding important bugs quicklyAchieving broader coverage quicklySmoke (or “build acceptance”) testingDeveloping, discovering, or refining a test planBuying time for more testingWhen investigation is more important than confirmation
37
Thinking Scientifically
Ability to pose useful questions.Ability to observe what’s going on.Ability to describe what you perceive.Ability to think critically about what you know.Ability to recognize and manage bias.Ability to form and test conjectures.Ability to keep thinking despite already knowing.Ability to analyze someone else’s thinking.
40
38
General Systems
systems are made up of parts comprising a whole, with dynamic connections and relationships between theminput, output, control, and feedback influence the behaviour of systemssystems can be decomposed into parts or subsystems, which are themselves systemssystems are components of larger systems
39
General Systems Thinking
General systems thinking is a way of observing and determining the way things (tend to (tend to)) workA means of analyzing, mastering, and learning to live with complexityA means of simplifying problems in useful waysAny view of a system is necessarily a model− “All models are wrong; some are useful.”
− George Box− “The map is not the territory.”− “When the map and the territory disagree, believe the territory.”
− Jerry Weinberg, quoting the Swedish Army
40
General Systems in Testing
The product is composed of functional componentsThe product runs on platforms that are also systemsThe product is part of a business processThe business process is part of the work done by an organizationThe organization is part of a communityThe community is part of a societyDo your tests reflect these ideas?− many testers stop at the first line!
41
Heuristics:Generating Solutions Quickly
adjective:“serving to discover.”
noun:“a fallible method for finding a solution.”
“Heuristic reasoning is not regarded as final and strictbut as provisional and plausible only, whose purpose
is to discover the solution to the present problem.”- George Polya, How to Solve It
42
Some Everyday Heuristics
It’s dangerous to drink and drive.A bird in hand is worth two in the bush.Nothing ventured, nothing gained.Sometimes people stash their passwords near their computers. Try looking there.Stores are open later during the Holidays.If your computer is behaving strangely, try rebooting. If it’s very strange, reinstall Windows.If it’s a genuinely important task, your boss will follow-up, otherwise, you can ignore it.
43
How Heuristics Differ fromOther Procedures or Methods
A heuristic is not an edict. Heuristics work implicitly under the control of a competent practitioner.Heuristics are presumed fallible and context-dependent.Heuristics aid or focus an open-ended solution search.Heuristics can substitute for complete and rigorous analysis.
44
KEY IDEA
45
Oracles
An oracle is the principle or mechanismby which you recognize a problem.
“..it works”
“...it appeared at least once to meet some requirement to some degree.”
46
All Oracles Are Heuristic
There is no single oracle that can tell us whether a program (or a feature) is working correctly at all times and in all circumstances.Oracles are fallible and context-dependent.Multiple oracles may increase our confidence, but even combinations of oracles are fallible.Oracles can be contradicted by other oracles.Recognizing a different problem usually requires a different oracle.A test designer doesn’t need to be aware of an oracle in advance of the observation, unless the test is designed to be run by rote.Any time you see a problem, you must be using an oracle…so what is it?
47
General Oracles: Consistency
History: Present system is consistent with past versions of the same system.
Image: System is consistent with an image that the organization wants to project.
Comparable Products: System is consistent with comparable systems.
Claims: System is consistent with what people say it’s supposed to be.
User’s Expectations: System is consistent with what we think users want.
Product: Elements of the system are consistent with comparable elements within the same system.
Purpose: System is consistent with its purposes, both explicit and implicit.
Statutes: System elements are consistent with applicable laws.
Use consistency heuristics to support inferences about possible bugs.
48
Was the file saved?
You’re testing Microsoft Word.You work with a file, save, and close it. You note that a file with the expected name appears in the expected folder.You reopen the file immediately, and the reopened document looks the same as it was just before you saved it.
49
Does Word save files properly?MAYBE…
we’re opening an identical file from the wrong folder.the file wasn’t saved at all, and we’re opening an identical file from thisfolder.we weren’t warned, even though we were replacing an existing file.the file we opened was in the old Word file format, but we weren’t asked to save the file in the most recent Word file format.the file did get saved, but clobbered another file in the process.the new document did get saved, but the default filename that Word offered didn’t match the first few words of our file. the new document did get saved to the right filename, but left a copy of ~WRD####.tmp lying around on the disk.the file got saved, but it took an unacceptably long time to do it.the “protect document” features are reset, instead of preservedthe bit of the document that we looked at is okay, but the rest of it is messed up − we looked at the text, but we forgot to look at the graphics− some of the file contents got saved, but the macros didn’t.− some of the file contents got saved, but unauthorized new macros
got saved too.− some of the file got saved, but the change tracking information
didn’t get saved.− some of the file got saved, but the font data was saved incorrectly
somewhere in the file.− the font data got saved, but the colors didn’t.− all the font information got saved, but the styles didn’t.− the text got saved, but all the graphics were missing.− it looks identical to a casual glance, but on a pixel-by-pixel basis,
the graphics quality has degraded− the file didn’t save any auto-text entries.
the file got saved, but the metadata under the File / Propertiesmenu didn’t get savedthe file got saved, but the metadata in the file system is inconsistent with File / Propertiesthere was no room on the disk, so the save operation should havefailedif we saved it with a password, we didn’t get prompted for a password when we reopened the filethe file got saved as an .RTF file, but WordPad can’t read it; only this version of Word can read it.the file can be reopened, but not edited.the file got saved, but when we press Shift-F5, the cursor doesn’t return to the point where it was when the document was saved.you weren’t prompted to update a linked documentyou were not prompted to save the Normal template, even though the option to save the Normal template was checked.you can open the file from the document management system where you saved it, but you won’t be able to save subsequent changes.when you saved it, Word only let you save the file as a template.the saved file did not get listed in the Recent Documents folder.the saved file did not get added to the “most recently used” list on the File/Open menu.Word didn’t maintain case-sensitivity in saving the file name.the autorun macros contained in the document did not execute, even though you weren’t holding the Shift key.
50
Simplicity vs. Complexity
Sometimes simple functions can conceal enormous complexity.Sometimes complex things can be simplified.
Let’s look at DiskMapperDiskMapper is a utility designed to display the relative amounts of space taken up by your files.A DiskMapper display can display hundreds of rectangles, each with a specific size and color.How can we look at a DiskMapper display and know that it is correct?
51
DiskMapper
52
Tests Are Oracles PLUS Coverage
“We haven't tried very hard to make it fail, and we haven't been running it very long or under very diverse conditions, but so far we haven't seen any failures, though we haven't
been looking too closely, either.”
According to Jerry Weinberg, “it works”
53
KEY IDEA
54
Coverage
There are as many kinds of coverageas there are ways to model the product.
StructureFunctionalDataPlatformOperationsTime
Product coverage is the proportion of theproduct that has been tested.
CapabilityReliabilityUsabilitySecurity
Scalability
PerformanceInstallability
CompatibilitySupportability
Testability
MaintainabilityPortability
Localizability
Product Elements:Structural Coverage
platform
input outputTest what it’smade of.
Print testing example− Files associated with printing− Code modules that implement printing− Code statements inside the modules− Code branches inside the modules
Product Elements:Functional Coverage
input output
platform
functionsTest what it does.
Print testing example− Print, page setup and print preview− Print range, print copies, zoom− Print all, current page, or specific range
functions
Product Elements:Data Coverage
functions&
structureinput output
platform
Test whatit does it to.
Print testing example− Types of documents− Items in documents, size and structure of documents− Data about how to print (e.g. zoom factor, no. of copies)
Product Elements:Platform Coverage
Test what itdepends upon.
Print testing example− Printers, spoolers, network behavior− Computers− Operating systems− Printer drivers
functions&
structureinput output
platform
Product Elements:Operations Coverage
Test howit’s used.
Print testing example− Use defaults− Use realistic environments− Use realistic scenarios− Use complex flows
input output
platform
Product Elements:Time Coverage
Test howit’s affected
by time.
Print testing example− Try different network or port speeds− Print one document right after another, or after long intervals− Try time-related constraints--spooling, buffering, or timeouts− Try printing hourly, daily, month-end, and year-end reports− Try printing from two workstations at the same time
input output
platform
61
Sometimes your coverage is disputed…
“No user would do that.”
“No user I can think of, who I like,would do that on purpose.”
Who aren’t you thinking of?Who don’t you like who might really use this product?
What might good users do by accident?
62
KEY IDEA
63
Thirty-Six Testing Heuristics
CustomersInformationDeveloper relationsTeamEquipment & toolsScheduleTest ItemsDeliverables
StructuresFunctionsDataPlatformsOperationsTime
CapabilityReliabilityUsabilitySecurityScalabilityPerformanceInstallabilityCompatibilitySupportabilityTestabilityMaintainabilityPortabilityLocalizability
Function testingDomain testingStress testingFlow testingScenario testingClaims testingUser testingRisk testingAutomatic testing
15
64
General Test Techniques
Function testing Domain testingStress testingFlow testingScenario testingUser testingRisk testingClaims testingAutomatic Testing
For any one of these techniques,there’s somebody, somewhere,
who believes that is theonly way to test.
65
MISSION:The most important part
Find important problemsAssess qualityCertify to standardFulfill process mandatesSatisfy stakeholdersAssure accountability
Advise about QAAdvise about testingAdvise about qualityMaximize efficiencyMinimize timeMinimize cost
The quality of testing depends on which of thesepossible missions matter and how they relate.
Many debates about the goodness of testingare really debates over missions and givens.
66
Good Enough Software...
…with what level of confidence?…to meet ethical obligations?…in what time frame?…compared to what?…for what purpose?…or else what?…for whom?
Perspectiveis
Everything
67
Heuristic Model:The Four-Part Risk Story
Victim. Someone that experiences the impact of a problem. Ultimately no bug can be important unless it victimizes a human.
Problem: Something the product does that we wish it wouldn’t do.
Vulnerability: Something about the product that causes or allows it to exhibit a problem, under certain conditions.
Threat: Some condition or input external to the product that, were it to occur, would trigger a problem in a vulnerable product.
Someone may be hurt or annoyedbecause of something that might go wrong while operating the product,
due to some vulnerability in the productthat is exploited by some threat.
68
KEY IDEA
69
Test Design and Execution
How do you effectivelyguide a tester?
TestIdea
How do youarrive at excellent
test design?
70
Exploratory Testing Defined
Exploratory testing is simultaneouslearning, test design, and test execution.
pure scripted freestyle exploratory
When I say “exploratory testing” and don’t qualify it, I mean anythingon the exploratory side of this continuum.
chartersvague scriptsfragmentarytest cases roles
71
Exploratory Testing Is…
an approach to software testing…
that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work…
by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project.
(applicable to any test technique)
(optimize how?)
72
Test DesignTesting to Learn vs. Testing to Search
Testing (primarily) to Learn− Forming a mental model of the product.− Learning what the product is supposed to do.− Inventorying the product elements you may want to test.− Looking at consistency relationships and trying various oracles.− Generating test data.− Considering testability and trying potentially useful tools.− Experimenting with different ways of testing it.− Reporting bugs that you find.
Testing (primarily) to Search− Using your detailed product knowledge, and any relevant tools, to
systematically exercise the product in every important way.− Using careful observation and good oracles to detect important bugs.− Modifying and diversifying your tests to make them more powerful.− Reporting bugs that you find.
73
Test DesignTesting to Learn vs. Testing to Search
− Starting a new project.
− Seeing a new feature for the first time.
− Testing a product deeply to reveal important bugs.
− Investigating a particular bug.
− Re-testing a product after a change.
− Repeated execution of detailed procedural test scripts.
LL
LSSS
Compare these Situations:
L
L
S
S
S
74
The Process of Test Design
75
How to Start?Pick a Useful, Fun, or Easy Starting Point.
What do I need to produce for my client?What has already been done?
What kind of testingam I good at?
What obstacles threatenmy success?
What do I know?What do I need to know?
Do I have a product?What can I do with
the product?
76
Contrasting Approaches
In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.
In exploratory testing, tests are designed and executed at the same time, and they are not necessarily recorded, but may be.
Product
TestScripts
Test Ideas
77
Contrasting Approaches
Scripted testing is likemaking a prepared speech.It is guided by pre-conceived ideas.
Exploratory testing is like having a conversation. It is self-guided.
Product
TestScripts
Test Ideas
78
Contrasting Approaches
Scripted testing is aboutcontrolling test execution.
Exploratory testing is about improving test design.
Product
TestScripts
Test Ideas
79
Contrasting Approaches
Scripted testing emphasizes predictability, accountability and decidability.
Exploratory testing emphasizes adaptability, credibility and learning. Product
TestScripts
Test Ideas
80
The Cognitive Processof the Tester Matters!
The test is what youthink and do!
The test is not the test script.
Only a fraction of a skilled tester’s thinkingand behavior can be represented in a script.
81
Why Script Anything?
IF any of the following are both true and important…− An agent needs to do something specific…− Those actions are known prior to the time of performing the action…− The agent would not otherwise know what to do unless told…− The agent might know what to do, but forget…− The script organizes information for more efficient, reliable access…− The script helps speed up execution…− The script helps coordinate the behavior of multiple agents…− The script helps multiple stakeholders come to a shared decision…− The scripts defines a pass/fail decision rule…− The script helps account for or assess what happens…− The script can be created at a reasonable cost…− The script is itself a product, not just a tool…
…THEN scripting an action MIGHT be a good idea.
82
Why Work Without a Script?
An agent DOES NOT need to do something very specific…Specific actions are NOT known prior to the time of performing the action…The agent knows what to do WITHOUT being told…There’s LITTLE danger of forgetting what to do…When a script DOESN’T organize information more accessibly…When a script DOESN’T help speed up execution…When there are NO multiple agents, or they are ALREADY coordinated…When NO shared decision is necessary…When you DON’T need to decide pass/fail criteria in advance…When sufficient accounting and assessment ALREADY occurs…When a good script would be TOO EXPENSIVE or take TOO LONG…When the script is NOT otherwise a product, but just a tool.When unscripted testing would be far more productive.When a script is a hindrance to the future instead of a help.
83
Blending Scripted & Exploratory
Generic scripts: specify general test procedures and apply them to different parts of a test coverage outline.Vague scripts: specify a test step-by-step, but leave out any detail that does not absolutely need to be pre-specified.Improvisation: have scripts, but encourage deviation from them, too.Fragmentary cases: specify tests as single sentences or phrases.Test Coverage Outline: use outline of product elements and have tester construct tests from it on the fly.Risk Catalog: specify types of problems to look for, then construct tests on the fly to find each one.Exploratory Charters: specify 90 minutes of testing in two sentences or less.Roles: Give each tester a standing role to test a certain part of the product. Leave the rest up to them.Heuristics: Train exploratory testers to use standardized test design heuristics.SBTM: Consider using Session-Based Test Management, a formalized method of exploratory test management. (http://www.satisfice.com/sbtm).
84
Test Design and Execution
Guide testers with personal supervision and concise documentation of test ideas. Meanwhile, train them so that they can guide themselves and be accountable for increasingly challenging work.
TestIdeas
Achieve excellent test design by exploring different test designs
while actually testing Product
Productor spec
85
KEY IDEA
86
Common Claims AboutTest Documentation
It’s needed for new testers.It’s needed to deflect legal liability.It’s needed to share testing with other testers.It’s needed for “repeatability.”It’s needed for accountability.It forces you to think about test strategy.
Something is wrong with these claims. Can you see the problem?
87
The First Law of Documentation
“That should be documented.”
“That should be documentedif and when and how it serves our purposes.”
Who will read it? Will they understand it?Is there a better way to communicate that information?
What does documentation cost you?
88
Common Problems withTest Documentation
Distracts, and even prevents, testers from doing the best testing they know how to do.Authors don’t understand testing.Authors don’t own format.Templates obscure poor thinking. Full of fluff.Fluff discourages readers and increases cost of maintenance.No one knows documentation requirements.Too much formatting increases the cost of maintenance.Reference and tutorial information mixed together.Catering to rare audiences instead of everyday user.Disregard for the social life of information.Long term and short term goals conflict.
89
Documentation: Product or Tool?
ProductTool
Audience:
Self Team Customers Regulators
Purpose:Recollection Organization Communication Demonstration
Paradigm:
5
90
Rapid Testing Documentation
Determine where your documentation is on the continuum: product or tool?Keep your tools sharp and lightweightObtain consensus from others as to what’s necessary and what’s excess in productsEliminate unnecessary clerical work
91
Accountability for Rapid Testing:Session-Based Test Management
Charter− A clear, concise mission for a session of testing
Time Box− 90-minute (+/- 30), long enough for setup and focused work; short
enough to make sure things don’t get off track
Reviewable Result− next slide!
Debriefing− conversation between tester and manager− problems, bugs and issues can be discussed− new risks can be identified− coaching and mentoring can happen
92
Reviewable Results
Charter− #AREAS
Start TimeTester Name(s)Breakdown− DURATION− TEST DESIGN AND EXECUTION− BUG INVESTIGATION AND
REPORTING− SESSION SETUP− CHARTER/OPPORTUNITY
Data Files
Test NotesBugs− BUG
Issues− ISSUE
CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk.
#AREASOS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional Analysis
START-----------------------------------------------5/30/00 03:20 pm
TESTER-----------------------------------------------Jonathan Bach
TASK BREAKDOWN-----------------------------------------------
#DURATIONshort
#TEST DESIGN AND EXECUTION65
#BUG INVESTIGATION AND REPORTING25
#SESSION SETUP20
A scannable session sheet
93
KEY IDEA
94
How Do We Justify Rapid Testing?
Recognize that you’re using it already− but that you might not realize it…− …or if you realize it, you might not be admitting it…− …to important people…− …who deserve to know if you’re producing good
results by avoiding their mythodologies
Ask people what they value− more documentation, or more testing?− they might well value the documentation− if so, are they ways to make that more rapid?
95
How Do We Justify Rapid Testing?
Do a bit of it using disposable timeTry a pilot project of a few hours a week− and then deploy some of those metrics, of which certain
managers seem to be so fond
Recognize that some of the best testing and development work is, at its heart, exploratory and investigative… and rapid
96
Additional References
Our Web sites and blogs− http://www.developsense.com− http://www.satisfice.com
My Test Connection column in Better Software MagazineLessons Learned in Software Testing− Cem Kaner, James Bach, Bret Pettichord
How To Break Software− James Whittaker
How To Break Software Security− James Whittaker
The Hacking Exposed Series− Various authors
Blink− Malcolm Gladwell
ANYTHING by Jerry Weinberg− Gerald M. Weinberg
Still more references available via email or my USB key
top related