1 Lecture 14: Testing Techniques and Strategies SACHIN
Nov 12, 2014
1
Lecture 14: Testing Techniques and Strategies
SACHIN
Today’s TopicsChapters 17 & 18 in SEPA 5/eTesting Principles & TestabilityTest CharacteristicsBlack-Box vs. White-Box TestingFlow Graphs & Basis Path TestingTesting & Integration Strategies
2SACHIN
Software TestingOpportunities for human error
Specifications, design, codingCommunication
“Testing is the ultimate review”Can take 30-40% of total effortFor critical apps, can be 3 to 5 times all
other efforts combined!
3SACHIN
Testing ObjectivesExecute a program with the intent of
finding errorsGood tests have a high probability of
discovering errorsSuccessful tests uncover errors‘No errors found’: not a good test!Verifying functionality is a secondary goal
4SACHIN
Testing PrinciplesTests traceable to requirementsTests planned before testing Pareto principle: majority of errors traced
to minority of componentsComponent testing first, then
integrated testingExhaustive testing is not possibleIndependent tests: more effective
5SACHIN
Software Testability
OperabilityObservabilityControllabilityDecomposability
6
Characteristics that lead totestable software:
• Simplicity• Stability• Understandability
SACHIN
Operability
System has few bugsNo bugs block execution of testsProduct evolves in functional stages
7
The better it works, the more efficientlyit can be tested
SACHIN
Observability
Distinct output for each inputStates & variables may be queriedPast states are loggedFactors affecting output are visibleIncorrect output easily identifiedInternal errors reportedSource code accessible
8
What you see is what you test
SACHIN
Controllability
All possible outputs can be generated by some input
All code executable by some inputStates, variables directly controlledInput/output consistent, structuredTests are specified, automated, and
reproduced
9
The better we can control the software,the more the testing can be automated
SACHIN
Decomposability
Independent modulesModules can be tested separately
10
By controlling the scope of testing, wecan more quickly isolate problems andperform smarter retesting
SACHIN
Simplicity
Minimum feature setMinimal architectureCode simplicity
11
The less there is to test, the morequickly we can test it
SACHIN
Stability
Changes made to system:are infrequentare controlleddon’t invalidate existing tests
Software recovers from failure
12
The fewer the changes, the fewer thedisruptions to testing
SACHIN
Understandability
Design is well-understoodDependencies are well understoodDesign changes are communicatedDocumentation is:
accessiblewell-organizedspecific, detailed and accurate
13
The fewer the changes, the fewer thedisruptions to testing
SACHIN
Test CharacteristicsGood test has a high probability of finding
an errorGood test is not redundantA good test should be “best of breed”A good test is neither too simple nor too
complex
14SACHIN
Test Case Design‘Black Box’ Testing
Consider only inputs and outputs‘White Box’ or ‘Glass Box’ Testing
Also consider internal logic paths, program states, intermediate data structures, etc.
15SACHIN
White-Box TestingGuarantee that all independent paths have
been testedExercise all conditions for ‘true’ and ‘false’Execute all loops for boundary conditionsExercise internal data structures
16SACHIN
Why White-Box Testing?More errors in ‘special case’ code which is
infrequently executedControl flow can’t be predicted accurately
in black-box testingTypo errors can happen anywhere!
17SACHIN
Basis Path TestingWhite-box method [McCabe ‘76]Analyze procedural designDefine basis set of execution pathsTest cases for basis set execute every
program statement at least once
18SACHIN
Basis Path Testing [2]
19
Flow Graph: Representation of Structured Programming Constructs
[From SEPA 5/e]
SACHIN
20
Cyclomatic Complexity
V(G)=E-N+2 = 4
Independent Paths1: 1,112: 1,2,3,4,5,10,1,113: 1,2,3,6,8,9,10,1,114: 1,2,3,6,7,9,10,1,11
V(G): upper bound on number of teststo ensure all code has been executed
[From SEPA 5/e]
SACHIN
Black Box TestingFocus on functional requirementsIncorrect / missing functionsInterface errorsErrors in external data accessPerformance errorsInitialization and termination errors
21SACHIN
Black Box Testing [2]How is functional validity tested?What classes of input will make good test
cases?Is the system sensitive to certain inputs?How are data boundaries isolated?
22SACHIN
Black Box Testing [3]What data rates and volume can the system
tolerate?What effect will specific combinations of data
have on system operation?
23SACHIN
Comparison TestingCompare software versions“Regression testing”: finding the outputs
that changed Improvements vs. degradationsNet effect depends on frequency and
impact of degradationsWhen error rate is low, a large corpus can
be used
24SACHIN
Generic Testing StrategiesTesting starts at module level and moves
“outward”Different testing techniques used at
different timesTesting by developer(s) and independent
testersTesting and debugging are separate
activities
25SACHIN
Verification and ValidationVerification
“Are we building the product right?”Validation
“Are we building the right product?”Achieved by life-cycle SQA activities,
assessed by testing“You can’t create quality by testing”
26SACHIN
27
Organization of Testing
[From SEPA 5/e]
SACHIN
28
Logarithmic Poissonexecution-time model
With sufficient fit,model predicts testingtime required to reachacceptable failure rate
[From SEPA 5/e]
SACHIN
29
[From SEPA 5/e]
SACHIN
30
PRO: Higher-level (logic) modules tested earlyCON: Lower-level (reusable) modules tested late
[From SEPA 5/e]
SACHIN
31
PRO: Lower-level (reusable) modules tested earlyCON: Higher-level (logic) modules tested late
[From SEPA 5/e]
SACHIN
Hybrid ApproachesSandwich Integration: combination of top-
down and bottom-upCritical Modules
address several requirementshigh level of controlcomplex or error pronedefinite performance requirements
Test Critical Modules ASAP!
32SACHIN
33SACHIN