CS2103/T-Aug2013 156 [Handout for L9P2] Heuristics for Better Test Case Design Test case design approaches Exploratory testing vs scripted testing Previously, we learned about two alternative approaches to test case design: exploratory vs scripted. That distinction was based on when test cases are designed. In scripted approach, they are designed in advance. In the exploratory approach, they are designed on the fly. Given next are alternative approaches based on some other aspects of testing. Black-box vs white-box This categorization is based on how much of SUT (software under test) internal details are considered when designing test cases. In the black-box approach, also known as specification- based or responsibility-based testing, we design test cases exclusively based on SUT’s specified external behaviour. In the white-box approach, also known as glass-box or structured or implementation-based testing, we design test cases based on what we know about the SUT’s implementation i.e. the code. Knowing some important information about the implementation can actually help in black-box testing. This kind of testing is sometimes called gray-box testing. For example, if the implementation of a sort operation uses an algorithm to sort lists shorter than 1000 items and another to sort lists longer than 1000 items, we can add more meaningful test cases to verify the correctness of both algorithms. Test case design techniques If we want to test all possible variations of using the SUT, we may have to write an infinite number of test cases. For example, consider test case for adding a String object to a Collection Add an item to an empty collection. Add an item when there is one item in the collection. Add an item when there are 2, 3, .... n items in the collection. Add an item that has an English, a French, a Spanish, … word. Add an item that is same as an existing item. Add an item immediately after adding another item. Add an item immediately after system startup. ... Exhaustive testing of this operation can take many more test cases. As you can see from the example above, except for trivial systems, exhaustive testing is not possible! Program testing can be used to show the presence of bugs, but never to show their absence! --Edsger Dijkstra Every test case adds to the cost of testing. In some systems, a single test case can cost thousands of dollars (e.g. on-field testing of flight-control software). Therefore, we have to design our test cases to make the best use of our testing resources. In particular: Testing should be effective, i.e. it finds a high % of existing bugs. A set of test cases that finds 60 defects is more effective than a set that finds only 30 defects in the same system.
14
Embed
[Handout for L9P2] Heuristics for Better Test Case Designcs2103/files/[Handout for... · 2013-10-16 · [Handout for L9P2] Heuristics for Better Test Case Design Test case design
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
CS2103/T-Aug2013
156
[Handout for L9P2]
Heuristics for Better Test Case Design
Test case design approaches Exploratory testing vs scripted testing Previously, we learned about two alternative approaches to test case design: exploratory vs
scripted. That distinction was based on when test cases are designed. In scripted approach, they
are designed in advance. In the exploratory approach, they are designed on the fly.
Given next are alternative approaches based on some other aspects of testing.
Black-box vs white-box This categorization is based on how much of SUT (software under test) internal details are
considered when designing test cases. In the black-box approach, also known as specification-
based or responsibility-based testing, we design test cases exclusively based on SUT’s specified
external behaviour. In the white-box approach, also known as glass-box or structured or
implementation-based testing, we design test cases based on what we know about the SUT’s
implementation i.e. the code.
Knowing some important information about the implementation can actually help in black-box
testing. This kind of testing is sometimes called gray-box testing. For example, if the
implementation of a sort operation uses an algorithm to sort lists shorter than 1000 items and
another to sort lists longer than 1000 items, we can add more meaningful test cases to verify the
correctness of both algorithms.
Test case design techniques
If we want to test all possible variations of using the SUT, we may have to write an infinite
number of test cases. For example, consider test case for adding a String object to a Collection
Add an item to an empty collection.
Add an item when there is one item in the collection.
Add an item when there are 2, 3, .... n items in the collection.
Add an item that has an English, a French, a Spanish, … word.
Add an item that is same as an existing item.
Add an item immediately after adding another item.
Add an item immediately after system startup.
...
Exhaustive testing of this operation can take many more test cases. As you can see from the
example above, except for trivial systems, exhaustive testing is not possible!
Program testing can be used to show the presence of bugs, but never to show their absence!
--Edsger Dijkstra
Every test case adds to the cost of testing. In some systems, a single test case can cost thousands
of dollars (e.g. on-field testing of flight-control software). Therefore, we have to design our test
cases to make the best use of our testing resources. In particular:
Testing should be effective, i.e. it finds a high % of existing bugs. A set of test cases that finds 60 defects is more effective than a set that finds only 30 defects in the same system.
CS2103/T-Aug2013
157
Testing should be efficient, i.e. it has a high rate of success (bugs found/test cases). A set of 20 test cases that finds 8 defects is more efficient than another set of 40 test cases that finds the same 8 defects.
That is why we have to learn various tools and techniques to improve E&E (Effectiveness and
Efficiency) of testing. Next, let us look at some useful techniques to improve E&E of testing.
Equivalence partitioning
Let us consider testing of the following operation.
isValidMonth (int m): boolean
Description: checks if m is in the range [1..12]. returns true if m is in the range, false
otherwise.
It is inefficient and impractical to test this method for all integer values [-MIN_INT to MAX_INT].
Fortunately, we need not test for all possible input values. For example, if the input value 233
failed to produce the correct result, the input 234 is likely to fail too. We need not test both. In
general, most SUTs do not treat each input in a unique way. Instead, they process all possible
inputs in a small number of distinct ways. That means a range of inputs is treated the same way
inside the SUT. Testing one of the inputs for a given range should be as good as exhaustively
testing all inputs in that range.
Equivalence partitioning is a technique that uses the above observation to improve the efficiency
and effectiveness of testing. If we divide possible inputs into groups which are likely to be
processed similarly by the SUT, we do not have to test every possible input in each group. Such
groups of input are called equivalence partitions (or equivalence classes). Equivalence
partitioning can minimize test cases that are unlikely to find new bugs.
Equivalence partitions are usually derived from the specification of the SUT. Preconditions and
postconditions can also help in identifying partitions. For example, these could be equivalence
partitions for the isValidMonth example:
[MIN_INT ... 0] (below the range that produces ‘true’)
[1 … 12] (the range that produces ‘true’)
[13 … MAX_INT] (above the range that produces ‘true’)
Note that equivalence partitioning technique does not tell us how many test cases to pick from
each partition. It depends on how thoroughly we want to test.
Here’s an example from an OO system. Consider the Minesweeper system we explored
previously. What are the equivalence partitions for newGame() operation of Logic component? In
general, we have to consider equivalence partitions of all data participants9 that take part in the
operation, such as,
the target object of the operation call, input parameters of the operation call, and other data/objects accessed by the operation such as global variables.
Since newGame() does not have any parameters, the only participant is the Logic object itself.
Note that if we are using the glass-box or the grey-box approach, we might even include other
associated objects that are involved in the operation as participants. For example, Minefield
9 We use the term “data participants” to mean both objects and primitive values. Note that this is not a standard term
CS2103/T-Aug2013
158
object can be considered as another participant of the newGame() operation. Here, we assume
the black-box approach.
Next, we identify equivalence partitions for each participant. Will the newGame operation
behave differently for different Logic objects? If yes, how will it differ? Yes, it might behave
differently based on the game state. Therefore, out equivalence partitions are: PRE_GAME (i.e.
before the game is starts, minefield does not exist yet), READY (i.e. a new minefield has been
created and waiting for player’s first move), IN_PLAY, WON, LOST
Here’s another example. Consider the markCellAt(int x, int y) operation of the Logic component.
Applying the technique described above, we can obtain the equivalence partitions for markCellAt
operation. The partitions in bold represent valid inputs. We assume a Minefield of size WxH :
Logic: PRE_GAME, READY, IN_PLAY, WON, LOST
x: [MIN_INT..-1] [0..(W-1)] [W..MAX_INT]
y: [MIN_INT..-1] [0..(H-1)] [H..MAX_INT]
Cell at (x,y): HIDDEN, MARKED, CLEARED
Here’s another example. Let us consider the push operation from a DataStack class.
Operation: push(Object o): boolean
Description: Throws MutabilityException if the global flag FREEZE==true
Else, throws InvalidValueException if o==null.
Else, returns false if the stack is full.
Else, puts o at the top of the DataStack and returns true.
Here are the equivalence partitions for the push operation.
DataStack object (ds): [full] [not full]
o: [null] [not null]
FREEZE: [true][false]
A test case for the push operation can be a combination of the equivalence partitions. Given
below is such a test case.
id: DataStack_Push_001
description: checks whether pushing onto a full stack works correctly
[Q3] Given below is the overview of the method dispatch(Resource, Task), from an emergency management system (e.g. a system used by those who handle emergency calls from the public about incidents such as fires, possible burglaries, domestic disturbances, etc.). A task might need multiple resources of multiple types. For example, the task ‘fire at Clementi MRT’ might need two fire engines and one ambulance.
dispatch(Resource r, Task t):void
Overview: This method dispatches the Resource r to the Task t. Since this can dispatch
only one resource, it needs to be used multiple times should the task need multiple
resources.
Imagine you are designing test cases to test the method dispatch(Resource,Task). Taking into
account equivalence partitions and boundary values, which different inputs will you combine to
test the method?
[A3] Test input for r Test input for t
A resource required by the task
A resource not required by the task
A resource already dispatched for
another task
null
A fully dispatched task
A task requiring one more resource
A task with no resource dispatched
Considering the resource types required
A task requiring only one type of
resources
A task requiring multiple types of
resource
null
[Q4] Given below is an operation description taken from a restaurant booking system. Use
equivalence partitions and boundary value analysis technique to design a set of test cases for it.
boolean transferTable (Booking b, Table t)
Description: Used to transfer a Booking b to Table t, if t has enough room.
Preconditions: t has room for b , b.getTable() != t
Postconditions: b.getTable() == t
[A4] Equivalence partitions
Booking:
Invalid: null, not null and b.getTable==t
Valid: not null and b.getTable != t
Table:
Invalid: null, not vacant, vacant but doesn’t have enough room,