This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
AcknowledgementSome of this material is from the Third Los Altos Workshop on Software Testing (LAWST), February 7 and 8, 1998. I founded the LAWST and was co-organizer of LAWST 3. At LAWST, we discussed test documentation (test planning strategies and materials). The agenda item was:
• How do we know what test cases we have? How do we know which areas of the program are well covered and which are not? • How do we develop this documentation EFFICIENTLY? As many of you know, I despise thick test plans and I begrudge every
millisecond that I spend on test case documentation. Unfortunately, some work is necessary. My question is, how little can we get away with, while still preserving the value of our asset?
The following people attended LAWST 3: Chris Agruss, James Bach, Karla Fisher, David Gelperin, Kenneth Groder, Elisabeth Hendrickson, Doug Hoffman, III, Bob Johnson, Cem Kaner, Brian Lawrence, Brian Marick, Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Johanna Rothman, Jane Stepak, Melora Svoboda, Jeremy White, and Rodney Wilson.
These slides are distributed under the Creative Commons License.
In brief summary, you may make and distribute copies of these slides so long as you give the original author credit and, if you alter, transform or build upon this work, you distribute the resulting work only under a license identical to this one.
For the rest of the details of the license, see http://creativecommons.org/licenses/by-sa/2.0/legalcode.
– Such as lists of fields, error messages, DLLs• Outlines: An outline organizes information into a hierarchy of
lists and sublists– Such as the testing objectives list later in the course notes
• Tables: A table organizes information in two dimensions showing relationships between variables.– Such as boundary tables, decision tables, combination test tables
• Matrices: A matrix is a special type of table used for data collection.– Such as the numeric input field matrix, configuration matrices
» Refer to Testing Computer Software, pages 217-241. For more examples, see page Testing Computer Software, page 218.
This table defines 6 standard configurations for testing. In later tests, the lab will set up a Config-1 system, a Config-2 system, etc., and will do its compatibility testing on these systems. The variables might be software or hardware choices. For example, Var 1 could be the operating system (V1-1 is Win 2000, V1-2 is Win ME, etc.) Var 2 could be how much RAM on the computer under test (V2-1 is 128 meg, V2-2 is 32 meg., etc.). Var 3 could be the type of printer, Var 4 the machine’s language setting (French, German, etc.). Config planning tables are often filled in using the All Pairs algorithm.
• COMPLETE SCRIPTING is favored by people who believe that repeatability is everything and who believe that with repeatable scripts, we can delegate to cheap labor.
1 ____ Pull down the Task menu 2 ____ Select First Number3 ____ Enter 34 ____ Enter 25 ____ Press return6 ____ The program displays 5
Never Do ANY kind of Scripting?• Checklists (as distinct from scripts) have their place. For
example, think of releasing a product:– Many very different tasks– All of them must be done– The task is rarely done, so many steps may be forgotten.
• I prefer to tell testers what to test (what issues to cover), not how to do the tests. Teaching people “how” is a matter of training, not something that I record time and time again in the test plan. A checklist will sometimes be the right way to present the list of issues.
• I think the level of detail is sufficient if I can successfully pass the section to a reasonably experienced tester who is a little familiar with the program and be confident that she can figure out what the test cases are and how to run them.
– What is the documentation cost per test case?– What is the maintenance cost of the documentation, per test case?– If software design changes create documentation maintenance
costs, how much inertia do we build into our system? How much does extensive test documentation add to the cost of late improvement of the software? How much should we add?
– What inertia is created in favor of invariant regression testing?– Is this incompatible with exploratory testing? Do we always want
• The basic thinking underlying ISO 9000 goes like this:– Decide what you’re going to do (create and document a process).– Do what you said you’d do.– Document what you’ve done in a way that lets an auditor check
what you’ve done against your process.• Risks
– It can be difficult to build in room for exploratory testing. To some degree (perhaps a large degree), this depends on the auditor.
– The style and extent of documentation needed to clue in a third party (auditor) might go beyond your internal needs. On finite budgets, where do we draw the line?
– What is the best tradeoff between process development and test execution? This must be a practical question, not a religious question.
– There are many different notions of what a good set of test documentation would include. Before spending a substantial amount of time and resources, it’s worth asking what documentation should be developed (and why?)
– Test documentation is expensive and it takes a long time to produce. If you figure out some of your main requirements first, you might be able to do your work in a way that achieves them.
– Stakeholders, interests, actions, objects• Who would use or be affected by test documentation?• What interests of theirs does documentation serve or disserve?• What will they do with the documentation?• What types of documents are of high or low value?
– Asking questions– Context-free questions– Context-free questions specific to test planning– Evaluating a plan
What is your group’s mission?• Find important problems• Assess quality• Certify to standard• Fulfill process mandates• Satisfy stakeholders• Assure accountability
• Advise about QA• Advise about testing• Advise about quality• Maximize efficiency• Minimize time• Minimize cost
The quality of testing depends on which of thesepossible missions matter and how they relate.
Many debates about the goodness of testingare really debates over missions and givens.
• Try to describe your core documentation requirements in one sentence that doesn’t have more than three components.
• Examples:– The test documentation set will primarily support our efforts to
find bugs in this version, to delegate work, and to track status.– The test documentation set will support ongoing product and test
maintenance over at least 10 years, will provide training material for new group members, and will create archives suitable for regulatory or litigation use.