1
Software Testing StrategiesSoftware Testing Strategies
based onbased onChapter 13 - Chapter 13 - Software Engineering: A Practitioner’s Software Engineering: A Practitioner’s
Approach, 6/eApproach, 6/ecopyright © 1996, 2001, 2005
R.S. Pressman & Associates, Inc.For University Use Only
May be reproduced ONLY for student use at the university levelwhen used in conjunction with Software Engineering: A Practitioner's Approach.
Any other reproduction or use is expressly prohibited.
2
Software Software TestingTestingTesting is the process of exercising a program with the Testing is the process of exercising a program with the
specific intent of finding errors prior to delivery specific intent of finding errors prior to delivery to the end user.to the end user.
errorserrors
requirements conformancerequirements conformance
performanceperformance
an indicationan indicationof qualityof quality
3
Who Tests the Who Tests the Software?Software?
developerdeveloper independent testerindependent tester
Understands the system Understands the system
but, will test "gently"but, will test "gently"
and, is driven by "and, is driven by "deliverydelivery""
Must learn about the system,Must learn about the system,but, will attempt to but, will attempt to breakbreak it it
and, is driven by and, is driven by qualityquality
4
Levels of TestingLevels of Testing Unit testingUnit testing Integration testingIntegration testing Validation testingValidation testing
Focus is on software requirementsFocus is on software requirements System testingSystem testing
Focus is on system integrationFocus is on system integration Alpha/Beta testingAlpha/Beta testing
Focus is on customer usageFocus is on customer usage Recovery testingRecovery testing
forces the software to fail in a variety of ways and verifies that recovery is properly performedforces the software to fail in a variety of ways and verifies that recovery is properly performed Security testingSecurity testing
verifies that protection mechanisms built into a system will, in fact, protect it from improper verifies that protection mechanisms built into a system will, in fact, protect it from improper penetrationpenetration
Stress testingStress testing executes a system in a manner that demands resources in abnormal quantity, frequency, or volumeexecutes a system in a manner that demands resources in abnormal quantity, frequency, or volume
Performance TestingPerformance Testing test the run-time performance of software within the context of an integrated systemtest the run-time performance of software within the context of an integrated system
5
Unit TestingUnit Testing
modulemoduleto beto betestedtested
test casestest cases
resultsresults
softwaresoftwareengineerengineer
interface interface
local data structureslocal data structures
boundary conditionsboundary conditions
independent pathsindependent paths
error handling pathserror handling paths
6
Integration Testing StrategiesIntegration Testing StrategiesOptions:Options:
•• the “big bang” approachthe “big bang” approach•• an incremental construction strategyan incremental construction strategy
• Top Down, bottom-up, sandwich Top Down, bottom-up, sandwich IntegrationIntegration
7
OOT OOT StrategyStrategy class testing is the equivalent of unit testingclass testing is the equivalent of unit testing
operationsoperations within the class are tested within the class are tested the the state behaviorstate behavior of the class is examined of the class is examined
integration applied three different integration applied three different strategies/levels of abstractionstrategies/levels of abstraction thread-basedthread-based testing—integrates the set of testing—integrates the set of
classes required to respond to classes required to respond to one input or eventone input or event use-baseduse-based testing—integrates the set of classes testing—integrates the set of classes
required to respond to required to respond to one use caseone use case clustercluster testing—integrates the set of classes testing—integrates the set of classes
required to demonstrate required to demonstrate one collaborationone collaboration
…if there is no nesting of classes
…this is pushing…
8
test casestest cases
resultsresults
DebuggingDebugging
suspectedsuspectedcausescauses
identifiedidentifiedcausescauses
correctionscorrections
regressionregressionteststests
new testnew testcasescases
Debugging: A Diagnostic ProcessDebugging: A Diagnostic Process
9
Software Testing Software Testing TechniquesTechniques
based onbased onChapter 14 - Chapter 14 - Software Engineering: A Practitioner’s Software Engineering: A Practitioner’s
Approach, 6/eApproach, 6/e copyright © 1996, 2001, 2005
R.S. Pressman & Associates, Inc.For University Use Only
May be reproduced ONLY for student use at the university levelwhen used in conjunction with Software Engineering: A Practitioner's Approach.
Any other reproduction or use is expressly prohibited.
10
What is a “Good” Test?What is a “Good” Test? a high probability of finding an errora high probability of finding an error not redundant.not redundant. neither too simple nor too complexneither too simple nor too complex
"Bugs lurk in corners "Bugs lurk in corners and congregate at and congregate at boundaries ..."boundaries ..."
Boris BeizerBoris Beizer
OBJECTIVE:OBJECTIVE:
CRITERIA:CRITERIA:
CONSTRAINT:CONSTRAINT:
to uncover errorsto uncover errors
in a complete mannerin a complete manner
with a minimum of effort and timewith a minimum of effort and time
11
Exhaustive TestingExhaustive Testing
loop < 20 Xloop < 20 X
There are approx. 10 possible paths! If we execute oneThere are approx. 10 possible paths! If we execute onetest per millisecond, it would take 3,170 years totest per millisecond, it would take 3,170 years to test this program!!test this program!!
1414
Where does 10 14 come from?
White-Box or Black-Box?White-Box or Black-Box?
12
RE in V ModelRE in V Model
systemrequireme
nts
systemintegration
softwarerequireme
ntspreliminar
ydesign
detaileddesign
code &debug
acceptancetest
softwareintegratio
ncomponen
ttest
unittest
Time
Leve
l of
ab
stra
cti
on
analyze an
d d
esign
test and in
tegrate
[Chung]
13
Software TestingSoftware Testing
White-Box TestingWhite-Box Testing
... our goal is to ensure that ... our goal is to ensure that allall statements and conditionsstatements and conditions have have
been executed at least been executed at least onceonce ... ...
Black-BoxBlack-Box Testing Testing
requirementsrequirements
eventseventsinputinput
outputoutput
14
BasisBasis Path Path TestingTesting
First, we compute the cyclomatic complexity:
number of simple decisions + 1
or
number of enclosed areas + 1
In this case, V(G) = 4
White-Box TestingWhite-Box Testing
15
BasisBasis Path Path TestingTestingNext, we derive the Next, we derive the
independent paths:independent paths:
Since V(G) = 4,Since V(G) = 4,there are four pathsthere are four paths
Path 3: 1,2,3,6,7,8Path 3: 1,2,3,6,7,8Path 2: 1,2,3,5,7,8Path 2: 1,2,3,5,7,8Path 1: 1,2,4,7,8Path 1: 1,2,4,7,8
Path 4: 1,2,4,7,2,4,...7,8Path 4: 1,2,4,7,2,4,...7,8
Finally, we derive testFinally, we derive testcases to exercise these cases to exercise these paths.paths.
11
22
3344
55 66
77
88
White-Box TestingWhite-Box Testing
A number of industry studies have indicated A number of industry studies have indicated that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.
16
Loop TestingLoop Testing
Nested Nested LoopsLoops
ConcatenatedConcatenated Loops Loops Unstructured Unstructured
LoopsLoops
Simple Simple looploop
White-Box TestingWhite-Box Testing
Why is loop testing important?
17
Equivalence Partitioning & Equivalence Partitioning & Boundary Value AnalysisBoundary Value Analysis
If x = 5 then …
What would be the equivalence classes?
If x > -5 and x < 5 then …
White-Box TestingWhite-Box TestingBlack-Box Black-Box
TestingTesting
18
Comparison TestingComparison Testing
Used only in situations in which the reliability of Used only in situations in which the reliability of software is absolutely critical (e.g., human-rated software is absolutely critical (e.g., human-rated systems)systems) Separate software engineering Separate software engineering teamsteams develop develop
independentindependent versionsversions of an application using the of an application using the same specificationsame specification
Each version can be tested with the Each version can be tested with the same test datasame test data to to ensure that all provide identical output ensure that all provide identical output
Then all versions are Then all versions are executed in parallelexecuted in parallel with real- with real-time comparison of results to ensure consistencytime comparison of results to ensure consistency
Black-BoxBlack-Box Testing Testing
19
OOT—Test Case OOT—Test Case DesignDesignBerard [BER93] proposes the following approach:Berard [BER93] proposes the following approach:
1.1. Each test case should be uniquely identified and should be explicitly Each test case should be uniquely identified and should be explicitly associated with the associated with the classclass to be tested, to be tested,
2.2. A list of testing steps should be developed for each test and should A list of testing steps should be developed for each test and should contain [BER94]:contain [BER94]:
a.a. a list of specified a list of specified statesstates for the object that is to be tested for the object that is to be tested
b.b. a list of a list of messages and operationsmessages and operations that will be exercised as a that will be exercised as a consequence of the testconsequence of the test how can this be done?how can this be done?
c.c. a list of a list of exceptionsexceptions that may occur as the object is tested that may occur as the object is tested
d.d. a list of a list of external conditionsexternal conditions (i.e., changes in the environment external (i.e., changes in the environment external to the software that must exist in order to properly conduct the test)to the software that must exist in order to properly conduct the test)
{people, machine, time of operation, etc.}
20
OOT Methods: Behavior TestingOOT Methods: Behavior Testing
emptyacctopen setup Accnt
set upacct
deposit(initial)
workingacct
withdrawal(final)
deadacct close
nonworkingacct
deposit
withdrawbalance
creditaccntInfo
Figure 14.3 State diagram for Account class (adapted from [ KIR94])
The tests to be The tests to be designed should designed should achieve achieve all state all state coveragecoverage [KIR94]. [KIR94]. That is, the That is, the operation operation sequences should sequences should cause the Account cause the Account class to make class to make transition through transition through all allowable statesall allowable states
Is the set of initial input data enough?
21
Omitted Omitted SlidesSlides
22
TestabiliTestabilityty
OperabilityOperability—it operates cleanly—it operates cleanly ObservabilityObservability—the results of each test case are readily observed—the results of each test case are readily observed ControllabilityControllability—the degree to which testing can be automated and —the degree to which testing can be automated and
optimizedoptimized DecomposabilityDecomposability—testing can be targeted—testing can be targeted SimplicitySimplicity—reduce complex architecture and logic to simplify tests—reduce complex architecture and logic to simplify tests StabilityStability—few changes are requested during testing—few changes are requested during testing UnderstandabilityUnderstandability—of the design—of the design
23
Strategic IssuesStrategic Issues
Understand the Understand the usersusers of the software and develop a of the software and develop a profile for each user profile for each user categorycategory..
Develop a Develop a testing plantesting plan that emphasizes “rapid cycle testing.” that emphasizes “rapid cycle testing.”
Use effective Use effective formal technical reviewsformal technical reviews as a filter prior to testing as a filter prior to testing
Conduct Conduct formal technical reviewsformal technical reviews to assess the test strategy and test cases to assess the test strategy and test cases themselves. themselves.
24
Counting BugsCounting Bugs
Sometimes reliability requirements take the form:Sometimes reliability requirements take the form:"The software shall have no more than X bugs/1K LOC""The software shall have no more than X bugs/1K LOC"But how do we measure bugs at delivery time?But how do we measure bugs at delivery time?
Bebugging Process - based on a Monte Carlo technique for statistical analysis of random Bebugging Process - based on a Monte Carlo technique for statistical analysis of random events.events.1. before testing, a known number of bugs (seeded bugs) are secretly inserted.1. before testing, a known number of bugs (seeded bugs) are secretly inserted.2. estimate the number of bugs in the system2. estimate the number of bugs in the system3. remove (both known and new) bugs.3. remove (both known and new) bugs.
# of detected seeded bugs/ # of seeded bugs = # of detected bugs/ # of bugs in the system # of detected seeded bugs/ # of seeded bugs = # of detected bugs/ # of bugs in the system # of bugs in the system # of bugs in the system = = # of seeded bugs x # of detected bugs# of seeded bugs x # of detected bugs / /# of detected seeded # of detected seeded bugsbugs
Example: Example: secretely seed 10 bugssecretely seed 10 bugsan independent test team detects 120 bugs (6 for the seeded)an independent test team detects 120 bugs (6 for the seeded)# of bugs in the system # of bugs in the system = = 10 x 120/6 10 x 120/6 = = 200200# of bugs in the system after removal = 200 - 120 - 4 = 76# of bugs in the system after removal = 200 - 120 - 4 = 76
But, deadly bugs vs. insignifant ones; not all bugs are equally detectable; ( Suggestion But, deadly bugs vs. insignifant ones; not all bugs are equally detectable; ( Suggestion [Musa87]:[Musa87]:
"No more than X bugs/1K LOC may be detected during testing""No more than X bugs/1K LOC may be detected during testing""No more than X bugs/1K LOC may be remain after delivery,"No more than X bugs/1K LOC may be remain after delivery,
as calculated by the Monte Carlo seeding technique"as calculated by the Monte Carlo seeding technique"
NFRs:NFRs:ReliabilityReliability
[Chung, RE Lecture Notes]][Chung, RE Lecture Notes]]
25
Cyclomatic Cyclomatic ComplexityComplexityA number of industry studies have indicated A number of industry studies have indicated
that the higher V(G), the higher the probability that the higher V(G), the higher the probability or errors.or errors.
V(G)V(G)
modulesmodules
modules in this range are modules in this range are more error pronemore error prone
White-Box TestingWhite-Box Testing