When to Test Less Presented by Lan Guo
When to Test Less
Presented by Lan Guo
Introduction (1)
• Methods of software testing: functional, coverage, and user-oriented
• Phases of software testing: unit, integration, product, or regression
• White-box (coverage) testing uses the structure of the software to measure the quality of testing.
Introduction (2)
– Statement coverage– Decision coverage– Data flow coverage– Mutation testing
• Black-box testing aims at the specific features of the software.– Generate an operational profile, which is a list of
occurrence probabilities of each element in the input domain of the program
Introduction (3)
– The operational profile is dependent on the input from the customer.
• Difficulties in black-box testing– new software, new feature, no customer base– inadequate test set– coarse features, interacting features– multiple and unknown user groups
Introduction (4)
• There is a saturation effect associated with ALL testing methods.
Testing effort
Reliability
functional
decisionData flow
mutation
Introduction (5)
• Elaborate and expensive testing does not tend to find more faults than inexpensive manual or simple automatic testing schemes.
Theory of Testing (1)
• In black-box testing, the probability to find a fault in N random tests is
y = 1 - (1 - x)N
x: the probability of finding a fault for a random input• Black-box testing can be expensive. If x =
0.0001, 46,000 tests are needed to be 99% sure to reveal a fault.
Theory of Testing (2)
• Formal-methods testing is even more expensive compared to black-box testing.
• While-box testing only slightly reduces the number of tests required by black-box testing. In addition, the partition cost is very high.
Reality of Testing (1)
• Software-in-the-small projects produce software without rigorous testing.
• The software does not crash more often.
• A program’s shape is the shape of the pathways within the program. The more tangled the pathways, the more complex the program shape.
Reality of Testing (2)
• The simple shapes are common, which is demonstrated by du-pathways, control-flow diagrams and saturation effect.
• Saturation effect is consistent with programs containing either many portions with simple shapes--so the portions are easily reached-- or many portions that are so twisted in shape that we will never reach them.
Average Shape of Software
• Right shape: the program is either too complex to test or too simple to test elaborately.
• In either case, there is no point to conduct a lengthy and expensive test.
• Most programs are indeed the right shape for simple testing.
Proof: A mathematical model of building a pathway across a program from randomly selected inputs to some randomly selected state.
Assumption 1
• Programs are networks connecting system concepts. (NAYO)
• Optimizing compiler builds a network (control and data-flow graphs) from the code.– An execution trace of a procedural program shows what
statements are executed, which subroutines are care called, and in what order.
– A verification condition generator creates a proof tree (execution tree), including preconditions and post-conditions
Assumption 2
• A fault explanation tree is a tree whose leaves are inputs and whose root is a fault.
• If we arrive at a state which should not happen in this tree, it is a bug in our system.
Assumption 3
• Testing is a process of trying to generate a fault explanation tree from the program network.
• If we can’t generate such tree, we are confident that there is no fault in our program.
Assumption 4
• NAYO tree (explanation tree)
N: no edge
A: and node
Y: yes edge
O: or node
• No edge only connects or nodes
• Yes edge connects both and nodes and or nodes
Assumption 5
• Testing is the process of extracting NAYO trees (the explanations) from NAYO networks (the program).
Mathematics Equations (1)
• Given in inputs to a NAYO network with V nodes, the probability to find a fault is:
xo = in/V (A)
• The probability of reaching an and node with andp parents is the probability of reaching all its parents:
xand = xiandp (B)
Mathematics Equations (2)
• The probability of reaching an or node with orp parents is the probability of not missing all of its parents:
xor = 1 - (1 - xo)orp (C)• If the ratio of and nodes in a NAYO tree is andf,
then the ratio of or nodes in the same tree is orf = 1 - andf. The probability to reach some random node xj is:
xj = andf * xand + orf * xor (D)
Mathematics Equations (3)
• Rearrange Equation A to isolate the number of tests required to be 99% sure of finding a fault with probability xj is:
y = 0.99 = 1- ((1 - xj)N)
Therefore, N =log(1- 0.99)
log(1-xj)
Implementation
• We can approximately categorize the tree’s shape by the number of tests N required to reach that tree’s root to find a fault.
• Results from the Simulation RunsClassification Threshold Percent
Simple
Moderate
Hard
Overly complex
0 < N < 102
102 <=N<104
104<=N<106
N >= 106
36
19
25
20
Conclusion
• Black-box mathematics is blind to the internal structures of the program. If we include internal structure in the analysis, the result is more optimistic.
• The average case analysis of the shape of programs indicates that simple and RANDOM testing can be very valuable.
Related Research
• Simple and random testing can probe non-determinate systems.
• Limited testing is adequate for knowledge based systems.
Potential Future Research
• Extending static code analysis tools (optimizer and verification condition generator) to extract the parameters in the mathematical model from real-world programs, in order to monitor the change of program’s shape.
• Offering design guideline to avoid hard-shaped programs.
References (1)
• T. Menzies and B. Cukic, “When to Test Less”, IEEE Software, September/October 2000, pp. 107-112
• D. Hamlet and R. Taylor, “Partition Testing Does not Inspire Confidence”, IEEE Trans. Software Eng., Vol. 16, No. 12, Dec. 1990, pp. 1402-1411
• J. Horgan and A. Mathur, “Software Testing and Reliability”, The Handbook of Software Reliability Eng., M.R. Lyu, ed., McGraw-Hill, New York, 1996, pp. 531-565.
References (2)
• T. Menzies and B. Cukic, “Adequacy of limited testing for knowledge based systems”, International Journal on Artificial Intelligence Tools, Vol. 9, No. 1, 2000, pp. 153-172
• T. Menzies, B. Cukic, H. Singh, and J. Powell, “Testing Nondeterminate Systems”