January 27, 2002 ECEN5033 University of Co lorado -- Class Testing 1 Specifying interactions Remainder of slides assume Operations defined by a class are specified by preconditions, post conditions, and class invariants contract design approach
Dec 24, 2015
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
1
Specifying interactions Remainder of slides assume
Operations defined by a class are specified by preconditions, post conditions, and class invariants
contract design approach
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
2
Contract vs. defensive approach & testing Contract More responsibility on
human designer Less class-level testing
due to fewer paths due to less error-checking code
MORE interaction testing required to ensure human designer complied with client side of contract using precondition constraints – (are preconditions in receiver met by sender?)
“illegal” for test cases to violate preconditions
Defensive More responsibility on
error-checking code More class-level testing
due to increased paths due to more code
Once the class-level error-checking code is tested, the interaction testing is simplified since there is no need to test if preconditions are met
Appropriate to violate preconditions to see if receiver catches that
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
3
Sampling Exhaustive testing: every possible test case
covering every combination of values Not reasonable much of the time
Want to select the ones that will find the faults in which we are the most interested Without prior information, random selection is
as good as we can do A sample is a subset of a population (e.g. all
possible test cases) that has been selected based on some probability distribution
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
4
Use profile (operational profile) Gave us a way to associate each use of the
system with a frequency These can be ranked by frequency Ranks can be turned into probabilities The higher the frequency of use, the larger the
probability of selection
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
5
Operational profile – stratified sample Stratified sample is a set of samples in which each
sample represents a specific subpopulation Example
Select test cases that exercise each component of the architecture
Divide a population of tests into subsets so that a subset contains all of the tests that exercise a specific component.
Sample on each subset independent of the others Need a basis for stratifying
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
6
A basis for stratification of test cases Use the actors from the use cases Select a sample of test cases from the uses of each actor
in the use cases Each actor uses some subset of the possible uses with
some frequency; rank by frequency Stratifying the test case samples by each actor provides a
way to increase the reliability of system Running selected tests
uses the system the way it will be used in typical situations;
finds defects likely to be found in typical use gives largest possible increase in reliability with least
effort
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
7
Test data generation for a range of values Generate test data first using the specification as
basis Select values based on boundary values
just within and just on the boundaries if contract just outside the boundaries also if defensive
Select values within intervals by sampling
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
8
Test data generation for a range of values, cont. Consider a random function over range 0.. N
such as int (random( ) * N) which generates a pseudo random value between 0 and 1 according to a uniform distribution. +: many different values will be tested over
many iterations Need to log actual values used in case of failure
so that the failed test case can be re-created. A randomly chosen value causing failure is
explicitly added to test suite and used to test the repaired software
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
9
Systematic sampling (+ boundary value testing)
Want to be able to increase level of coverage systematically. We use boundary values coupled with sampling
over ranges We never omit boundary value test data
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
10
Issues about sampling for interaction testing
Sampling issues Different states can cause two objects from the
same class to behave differently A class family is a subset of a larger family
Possibility of combinatorial explosion in the number of test configurations
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
11
Impact of inheritance hierarchy on s.t.d.’s Each member of the family may have different states
that can cause two objects from the same class to behave differently.
In families of classes, the state transition tables are related along the lines of the inheritance hierarchy. as we look down the inheritance hierarchy, there
will be the same number of states or more states in the derived class as in the base class
should cover the states defined for each class, emphasizing the new states added at that level in inheritance hierarchy.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
12
Class families as subsets of others
If a class family is a subset of a larger family, after the tests are designed, they can be applied to any of the classes in the family, assuming the substitution principle has been followed during design (see next slide)
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
13
Substitution Principlea reminder
Only the following changes are allowed in defining the behavior associated with a new subclass: Preconditions for each operation must be the
same or weaker – less constraining – than those of the superclass
Post conditions for each operation must be the same or stronger – do at least as much as defined by the superclass
Class invariant – must be the same or stronger; add more constraints
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
14
Orthogonal array testing applied to OO Orthogonal array testing is a sampling
technique to limit the explosion of test cases Define pair-wise combinations of a set of
interacting objects because most faults result from interactions due to two-way interactions
An orthogonal array is an array of values in which each column represents a factor.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
15
Why PAIR-wise Goal: choose a fairly small subset of test cases
that finds a great many defects. (duh!) How? Test all pairs of whatever it is that is
varying. Pairs are fewer: Suppose 4 input parameters,
each of which could take on 3 values. Number of combinations is 3^4 = 81. Can cover all pairwise input combinations in 9 tests
13 input params, each of which as 3 values; number of comb.’s is 3^13. Can cover all pairwise input combinations in 15 tests.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
16
More on pairwise AT&T wanted to test a LAN-based electronic mail
system. Original plan was to develop and execute 1500 test cases. Switched to pairwise testing – 50 percent less effort and found 28% more defects.
NIST reviewed 15 years of defect data on recalled medical devices. They concluded: 98% of the reported software flaws would have been detected by testing all pairs of parameter settings.
Why does it work so well?
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
17
Orthogonal array testing applied to OO An orthogonal array is an array of values in
which each column represents a factor. A factor is
a variable in an experiment; in our case, a specific class family in the software system
Or states of that class family Each variable takes on a certain set of values
called levels (rows); in our case, specific classes or states of those classes
Each cell is an instance of the class, that is, a specific object or a particular state
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
18
What are orthogonalarrays
Consider numbers 1 and 2 – how many pair combinations are there?
An orthogonal array is a 2-dimensional array of numbers that has this property: Choose any two columns of the array; all the pairwise combinations of its values will occur in every pair of columns.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
19
Example
Subscript is number of rows in the array. Superscript is number of columns. 2 says each value is a 1 or a 2.
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
22
How to use Identify the variables Determine the number of choices for each variable Locate an orthogonal array with a column for each
variable and values within the columns corresponding to the number of choices.
Map the test problem onto the array Construct test cases with these combinations
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
25
When mapped to all of array Each row becomes a test case Decode the level numbers for a row in the array
back to the individual lists for each factor Sometimes, you don’t need all of the columns
We can ignore them because we don’t need them.
Or we may decide we can include one more class family and its states in the future
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
26
What about mismatches? When there is a difference in the number of levels,
we can use a column that matches or exceeds the maximum
What about a class with 2 states but there can be a 3 in the array we picked Interpret the 3 as if it were either 1 or 2. What
do you want to emphasize in the testing? Arbitrary assignment High frequency High risk
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
27
Other what-ifs? See the Copeland excerpt on Class Lecture
materials page of web site
January 27, 2002 ECEN5033 University of Colorado -- Class Testing
28
OATS adequacy criteria Ability to vary how completely the software under
test is covered Exhaustive – all combinations of all factors Minimal – only interactions between the base
classes from each hierarchy are tested Random – tester haphazardly selects cases from
several of the classes; not statistically random Representative – uniform sample ensures every
class is tested to some level Weighted representative – adds cases to the
representative approach based on relative importance or risk associated with the class