1 Testing Course notes for CEN 4010
1
Testing
Course notes forCEN 4010
2
Outline
Introduction: terminology and philosophy
Factors that influence testing
Testing techniques
3
Why Do We Test?
We test to find bugs “Testing is the process of executing a program
with the intent of finding errors” A good test intends to find bugs
“A good test case is one that has a high probability of detecting an as-yet undiscovered error”
A test that finds no bug is a failure “A successful test case is one that detects an
as-yet undiscovered error”
4
Testing vs. Debugging
Testing != Debugging
While testing may reveal a symptom of an error, it may not uncover the exact cause of the error
Debugging is the process of locating the exact cause of an error, and removing that cause
5
Our Very Real Problem
We can never say for an arbitrary program: This software has no errors This software works only as intended This software is safe
Testing proves the presence, not the absence, of bugs
Absence of evidence is not evidence of absence
6
4 dimensions of the Testing
Outputs (i.e. Results)In
pu
ts
Valid InvalidV
alid
Inva
lid
Good Goal(includes
exceptions)
BUG!!!
Shouldalways
produceexceptions!
7
Where Does Testing Fit?
Requirements Analysis Design Implementation Test
Client's viewof theProblem andone or moreSolutions
Description of the PROBLEM:"WHAT is the business domain;"WHAT" are the entities involved?
Logical description of the SOLUTION:"HOW will we solve the problem:"HOW will we meet the environment, etc.constraints?
The computablerepresenation of thelogical solution.
Activity of verifyingand validating thesolution from differentperspectives.
8
When to test ?
Requirements Analysis Design Implementation Test
Code
Unit /Component
Test
Code Code
Unit /Component
Test
Unit /Component
Test
9
What other Tests Are There?
Analysis Design Implementation Test
IntegrationTest
SystemTest
AcceptanceTest
Testing of combinationsof components.Tested from theproducer's perspective.
Testing of totality ofcomponents.Tested from theproducer'sperspective.
Testing of totality ofcomponents.Tested from theclient's perspective.
10
Test activities: Analysis
Test the Models against the Problem specification “Play computer” through the models
Analysis models must accurately describe the problem to be solved, and the boundaries of the problem domain
11
Test activities: Design
Test the Models against the Solution domain. “Play computer” through the models Incorporating all required “design” classes
GUI frameworks Third-party software and class libraries Collection/Container classes Operating environment classes (wrappers) Interface classes to external resources. E.g.,
RDBMS, network, communications.
12
Test Metrics
Traditional Metrics: No. of bugs per 10,000 Source Lines of Code IEEE Standard 982.1 McCabe Complexity Measure Halstead Software Science Measures
…
13
Test Techniques: The “Best of all Possible Worlds” technique:
Run it...and make sure it doesn’t crash, then ship it!
For the rest of us: Desk Checking Inspections &Walkthroughs
Black-box/White-box testing Boundary testing Path testing Code coverage Error message testing Top-down & Bottom-up …
14
Desk Checking
Simple, labor-intensive Procedure
Developer has his/her design or code manually reviewed by another developer
“a second pair of eyes” Developer must justify decisions to
reviewer. Developer reviews/accepts/rejects
reviewer’s recommendations.
15
Inspections & Walkthroughs
Formal or informal as needed. Well-known technique. Procedure:
Group gather to review artifacts (docs to code). Group composed of “role” representatives (e.g.
“user”, “QA”, “Standards”, etc.). Rules of conduct apply to participants Responsibility for verification and validation of
artifact is removed from the developer.
16
Inspections
Author narrates, statement by statement, the logic of his program.
Reviewers listen to the author, raise questions and ask for clarification in their attempt to uncover errors.
Ironically, most of the errors are discovered by the author as he/she “teaches” the program to the group.
17
Walkthroughs Programmer does not narrate to the group. One of the group plays the role of “tester”. Tester comes to review armed with a small
set of simple test cases to apply to the program.
Group “play computer” by performing manual simulation of the system using the test data.
Data are important as a vehicle to stimulate discussion.
Most errors are uncovered by questioning of the author rather than “execution” of data.
18
Inspections & Walkthroughs When should I perform one?
Anytime, so long as the product under review is complete (i.e. tangible, understandable, and objective).
After each milestone with delivered artifacts (e.g. Requirements Spec., Analysis Models, Design Spec., pseudo-code or flowcharts prepared, program code written, etc.).
Yourdon’s milestones [YOUR79]: after the design artifacts are completed after the code is prepared but not compiled after the first compilation after the first “clean” compilation after the first test data set have been executed successfully after the programmer thinks all test cases have been executed
successfully
19
Inspections & Walkthroughs
Who should participate? Author/developer/presenter/producer Moderator/coordinator Scribe/secretary Standards Bearer Maintenance Expert User Representative GUI Expert
20
White-Box Testing
White-box testing is the testing of the underlying implementation of a piece of software (e.g., source code) without regard to the specification (external description) for that piece of software.
The goal of white-box testing of source code is to identify such items as (unintentional) infinite loops, paths through the code which should be
allowed, but which cannot be executed, and dead (unreachable) code.
21
Black-Box Testing
Black-box testing is the testing of a piece of software without regard to its underlying implementation.
Specifically, it dictates that test cases for a piece of software are to be generated based solely on an examination of the specification (external description) for that piece of software.
The goal of black-box testing is to demonstrate that the software being tested does not adhere to its external specification.
(Note that if there is no "external specification" it will be difficult to conduct black-box testing.)
22
Test Categories
Black Box Tests Boundary-value Analysis Equivalence Partitioning Cause-Effect Graphing Error Guessing Error Message Generation
from [MYER79]
White Box Tests Statement Coverage Decision/Condition
Coverage Multiple-Condition
Coverage
23
Black Box Tests Boundary-value Analysis
Uses test cases “generated on, and immediately around, the boundaries of the input an output for a given piece of software.”
Equivalence Partitioning Divide test sets into equivalence partitions: “collections of
items which can all be regarded as identical at a given level of abstraction. I.e., a set of data items which will all evoke the same general behavior from a given software module.”
Cause-Effect Graphing Generates test sets based on input combinations, specified
by a combinatorial logic network or Boolean relations, and the expected outputs from those input combinations.
24
Black Box Tests
Error Guessing An intuitive and empirical technique of
selecting test cases which elicit failures. A.k.a. “a gift”.
Error Message Generation Selecting test cases which will elicit the
error messages defined in the software under test.
25
White Box Tests
Statement Coverage a.k.a. “basis path testing” Use test cases which cause each executable
statement to be executed at least once.
Decision/Condition Coverage All statements are executed at least once, and
all binary decisions have a TRUE and FALSE outcome at least once, all exceptions are raised at least once, and all possible interrupts are forced to occur at least once.
26
White Box Tests
Multiple-Condition Coverage Use test cases such that all possible
combinations of condition outcomes in each decision, and all points of entry, are invoked at least once.
27
Path Testing
tests all possible paths
the flowchart on the right has approximately 100 trillion possible paths
at 1ms/test, working 24 hrs/day, 365 days/year: it will take 3170 years
Loop
<=
20 ti
mes