Dynamic Testing Techniques
1 Principles 2 Lifecycle
4 Dynamic testtechniques
3 Static testing
5 Management 6 Tools
Software Testing
ISTQB / ISEB Foundation Exam Practice
Chapter 4
Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice
Why dynamic test techniques?
Exhaustive testing (use of all possible inputs and conditions) is impractical- must use a subset of all possible test casesmust use a subset of all possible test cases- must have high probability of detecting faultsmust have high probability of detecting faults
Need thought processes that help us select test cases more intelligently- test case design techniques are such thought test case design techniques are such thought
processesprocesses
What is a testing technique?
a procedure for selecting or designing tests based on a structural or functional model of
the software successful at finding faults 'best' practice a way of deriving good test cases a way of objectively measuring a test effort
Testing should be rigorous, thorough and systematicTesting should be rigorous, thorough and systematic
Using techniques makes testing much more effectiveUsing techniques makes testing much more effective
Advantages of techniques
Different people: similar probability find faults- gain some independence of thoughtgain some independence of thought
Effective testing: find more faults- focus attention on specific types of faultfocus attention on specific types of fault- know you're testing the right thing know you're testing the right thing
Efficient testing: find faults with less effort- avoid duplicationavoid duplication- systematic techniques are measurablesystematic techniques are measurable
Measurement
Objective assessment of thoroughness of testing (with respect to use of each technique)- useful for comparison of one test effort to anotheruseful for comparison of one test effort to another
E.g.
Project A60% Equivalence
partitions50% Boundaries75% Branches
Project B40% Equivalence
partitions45% Boundaries60% Branches
Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice
Three types of systematic technique
Static (non-execution)
• examination of documentation,examination of documentation,source code listings, etc.source code listings, etc.
Functional (Black Box)
• based on behaviour /based on behaviour /functionality of softwarefunctionality of software
Structural (White Box)
• based on structurebased on structureof softwareof software
Some test techniquesStaticStatic DynamicDynamic
StructuralStructural
BehaviouralBehavioural
FunctionalFunctionalNon-functionalNon-functional
ReviewsReviews
WalkthroughsWalkthroughs
Desk-checkingDesk-checking
DataFlowDataFlow
SymbolicExecutionSymbolicExecution
Definition-UseDefinition-Use
StatementStatement
Branch/DecisionBranch/Decision
Branch ConditionBranch Condition
Branch ConditionCombinationBranch ConditionCombination
LCSAJLCSAJ
ArcsArcs
EquivalencePartitioningEquivalencePartitioning
BoundaryValue AnalysisBoundaryValue Analysis
Cause-Effect GraphingCause-Effect Graphing
RandomRandom
UsabilityUsability
PerformancePerformance
Static AnalysisStatic AnalysisInspectionInspection
ControlFlowControlFlow
etc.etc.
etc.etc.
etc.etc.
etc.etc.
etc.etc.
State TransitionState Transition
Black box versus white box?
IntegrationIntegration
ComponentComponent
AcceptanceAcceptance
SystemSystem
Black box appropriateat all levels butdominates higherlevels of testing
White box usedpredominatelyat lower levelsto complimentblack box
Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice
Black Box test design and measurement techniques
Techniques defined in BS 7925-2- Equivalence partitioningEquivalence partitioning- Boundary value analysisBoundary value analysis- State transition testingState transition testing- Cause-effect graphingCause-effect graphing- Syntax testingSyntax testing- Random testingRandom testing
Also defines how to specify other techniques
= Yes= No
Also a measurementtechnique?
Equivalence partitioning (EP)
- divide (partition) the inputs, outputs, etc. into areas divide (partition) the inputs, outputs, etc. into areas which are the same (equivalent)which are the same (equivalent)
- assumption: if one value works, all will workassumption: if one value works, all will work- one from each partition better than all from oneone from each partition better than all from one
1 100 1010
valid invalidinvalid
Boundary value analysis (BVA)
- faults tend to lurk near boundariesfaults tend to lurk near boundaries- good place to look for faultsgood place to look for faults- test values on both sides of boundariestest values on both sides of boundaries
1 100 1010
valid invalidinvalid
Example: Loan application
Customer Name
Account number
Loan amount requested
Term of loan
Monthly repayment
Term:
Repayment:
Interest rate:
Total paid back:
6 digits, 1stnon-zero
£500 to £9000
1 to 30 years
Minimum £10
2-64 chars.
Customer name
Number of characters:
2 64 65invalid valid invalid
1
Conditions ValidPartitions
InvalidPartitions
ValidBoundaries
InvalidBoundaries
Customername
2 to 64 charsvalid chars
< 2 chars> 64 charsinvalid chars
2 chars64 chars
1 chars65 chars0 chars
Valid characters:Anyother
A-Za-z-’
space
Account number
5 6 7invalid
validinvalid
number of digits:
first character:invalid: zero
valid: non-zero
Conditions ValidPartitions
InvalidPartitions
ValidBoundaries
InvalidBoundaries
Accountnumber
6 digits1st non-zero
< 6 digits> 6 digits1st digit = 0non-digit
100000999999
5 digits7 digits0 digits
Loan amount
500 9000 9001
invalid valid invalid
499
Conditions ValidPartitions
InvalidPartitions
ValidBoundaries
InvalidBoundaries
Loanamount
500 - 9000 < 500>90000non-numericnull
5009000
4999001
Condition template
Conditions ValidPartitions
Tag InvalidPartitions
Tag ValidBoundaries
Tag InvalidBoundaries
Tag
Customername
2 - 64 charsvalid chars
V1V2
< 2 chars> 64 charsinvalid char
X1X2X3
2 chars64 chars
B1B2
1 char65 chars0 chars
D1D2D3
Accountnumber
6 digits1st non-zero
V3V4
< 6 digits> 6 digits1st digit = 0non-digit
X4X5X6X7
100000999999
B3B4
5 digits7 digits0 digits
D4D5D6
Loanamount
500 - 9000 V5 < 500>90000non-integernull
X8X9X10X11X12
5009000
B5B6
4999001
D7D8
Design test cases
TestCase
Description Expected Outcome New TagsCovered
1
2
Name: John SmithAcc no: 123456Loan: 2500Term: 3 years
Name: ABAcc no: 100000Loan: 500Term: 1 year
Term: 3 yearsRepayment: 79.86Interest rate: 10%Total paid: 2874.96
Term: 1 yearRepayment: 44.80Interest rate: 7.5%Total paid: 537.60
V1, V2,V3, V4,V5 .....
B1, B3,B5, .....
Why do both EP and BVA?
If you do boundaries only, you have covered all the partitions as well- technically correct and may be OK if everything works technically correct and may be OK if everything works
correctly!correctly!- if the test fails, is the whole partition wrong, or is a if the test fails, is the whole partition wrong, or is a
boundary in the wrong place - have to test mid-partition boundary in the wrong place - have to test mid-partition anywayanyway
- testing only extremes may not give confidence for testing only extremes may not give confidence for typical use scenarios (especially for users)typical use scenarios (especially for users)
- boundaries may be harder (more costly) to set upboundaries may be harder (more costly) to set up
Test objectives?
For a thorough approach: VP, IP, VB, IB Under time pressure, depends on your test objective
- minimal user-confidence: VP only?minimal user-confidence: VP only?- maximum fault finding: VB first (plus IB?)maximum fault finding: VB first (plus IB?)
Condition ValidPartition
Tag InvalidPartition
Tag ValidBoundary
Tag InvalidBoundary
Tag
Decision tables
explore combinations of inputs, situations or events,
it is very easy to overlook specific combinations of input
start by expressing the input conditions of interest so that they are either TRUE or FALSE
– record found– file exists– code valid
– policy expired– account in credit– due date > current date
Example: student access
A university computer system allows students an allocation of disc space depending on their projects.
If they have used all their allotted space, they are only allowed restricted access, i.e. to delete files, not to create them. This is assuming they have logged on with a valid username and password.
What are the input and output conditions?What are the input and output conditions?
List the input and output conditions
• list the ‘output conditions’ under the input conditions
Input ConditionsValid usernameValid passwordAccount in creditOutput ConditionsLogin acceptedRestricted access
• list the ‘input conditions’ in the first column of the table
Determine input combinations
add columns to the table for each unique combination of input conditions.
each entry in the table may be either ‘T’ for true, ‘F’ for false.
Input ConditionsValid username T T T T F F F FValid password T T F F T T F FAccount in credit T F T F T F T F
Rationalise input combinations
some combinations may be impossible or not of interest
some combinations may be ‘equivalent’ use a hyphen to denote “don’t care”
Input ConditionsValid username F T T TValid password - F T TAccount in credit - - F T
Complete the table
determine the expected output conditions for each combination of input conditions
Input ConditionsValid username F T T TValid password - F T TAccount in credit - - F TOutput ConditionsLogin accepted F F T TRestricted access - - T F
Determine test case groups each column is at least one test case
Input ConditionsValid username F T T TValid password - F T TAccount in credit - - F TOutput ConditionsLogin accepted F F T TRestricted access - - T F
Tags A B C D
Design test cases
usually one test case for each column but can be none or several
Test Description Expected Outcome Tag1 Username BrbU Invalid username A2 Username
usernametoolongInvalid username A
3 Username BobUPassword abcd
Invalid password B
4 Valid user, no discspace
Restricted access C
5 Valid user with discspace
Unrestricted access D
Rationalising outputs
X T F F
Y F T FZ F F T Output X Y Z
if outputs or effects are mutually exclusive, I.e. T occurs in only one place in each column, we can combine them
for example:
is equivalent to:
Rationalising dangers
rationalising is based on assumptions assumptions may be wrong! assumptions should be stated assumptions may change over time be aware of the dangers
- filling in the full table may find errors which will be filling in the full table may find errors which will be missed if you rationalisemissed if you rationalise
- it is possible to rationalise too farit is possible to rationalise too far
Extending decision tables
Entries can be more than just ‘true’ or ‘false’- completing table needs to be done carefullycompleting table needs to be done carefully- rationalising becomes more importantrationalising becomes more important
E.g.
Code = 1, 2, or 3 1 1 1 1 2 2 2 2 3 3 3 3Exp.date < now T T F F T T F F T T F FClass A product T F T F T F T F T F T F
Decision Tables in relation to EP and BVA
Input value Output value
FALSE
TRUE
Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice
White Box test design and measurement techniques
Techniques defined in BS 7925-2- Statement testingStatement testing- Branch / Decision testingBranch / Decision testing- Data flow testingData flow testing- Branch condition testingBranch condition testing- Branch condition combination testingBranch condition combination testing- Modified condition decision testingModified condition decision testing- LCSAJ testingLCSAJ testing
Also defines how to specify other techniques
= Yes= No
Also a measurementtechnique?
Stronger structural techniques (different structural elements)
More tests
Using structural coverage
Increasing coverage
Coverage OK?
What'scovered?
Results OK?
Enoughtests?
Spec
Spec SoftwareSoftware
Tests
More testsMore testsMore testsMore testsMore testsMore testsMore tests
The test coverage trap
Structure exercised,insufficient function
Structure exercised,insufficient function
Function exercised,insufficient structure
Function exercised,insufficient structure
better testingbetter testing
% Statement % Decision % Condition Combination
Structural testedness
Functionaltestedness
100% coverage doesnot mean 100% tested!
100% coverage doesnot mean 100% tested!
Coverage is notThoroughness
Coverage is notThoroughness
Statement coverage
percentage of executable statements exercised by a test suite
number of statements exercisednumber of statements exercised
total number of statementstotal number of statements example:
- program has 100 statementsprogram has 100 statements- tests exercise 87 statementstests exercise 87 statements- statement coverage = 87%statement coverage = 87%
=
Typical ad hoc testing achieves 60 - 75%Typical ad hoc testing achieves 60 - 75%
Statement coverageis normally measured
by a software tool.
Statement coverageis normally measured
by a software tool.
?
Example of statement coverage
Testcase
Input Expectedoutput
1 7 7
As all 5 statements are ‘covered’ bythis test case, we have achieved
100% statement coverage
read(a)
IF a > 6 THEN
b = a
ENDIF
print b
1
2
3
4
5
Statementnumbers
Decision coverage(Branch coverage)
percentage of decision outcomesexercised by a test suite
number of decisions outcomes exercisednumber of decisions outcomes exercised
total number of decision outcomestotal number of decision outcomes example:
- program has 120 decision outcomesprogram has 120 decision outcomes- tests exercise 60 decision outcomestests exercise 60 decision outcomes- decision coverage = 50%decision coverage = 50%
Typical ad hoc testing achieves 40 - 60%Typical ad hoc testing achieves 40 - 60%
=
Decision coverageis normally measured
by a software tool.
Decision coverageis normally measured
by a software tool.
True
False?
Paths through code
? ?
1 2
? ?
1231 2 ?
?
1234
Paths through code with loops
?
1 2 3 4 5 6 7 8 ….
for as many times as itis possible to go roundthe loop (this can beunlimited, i.e. infinite)
End
Selecttrans...
Yes
Example 1
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
IF PIN is valid THEN
select transaction
ELSE (otherwise)
display “PIN invalid”
ELSE (otherwise)
reject card
End
Display“Enter..
Yes
ValidPIN?
No
Rejectcard
Display“PIN in..
No
Wait for card to be insertedWait for card to be inserted
IF card is a valid card THEN
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
IF PIN is valid THEN
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
IF PIN is valid THEN
select transaction
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
IF PIN is valid THEN
select transaction
ELSE (otherwise)
display “PIN invalid”
Wait for card to be inserted
IF card is a valid card THEN
display “Enter PIN number”
IF PIN is valid THEN
select transaction
ELSE (otherwise)
display “PIN invalid”
ELSE (otherwise)
reject card
Validcard?
Wait
Read AIF A > 0 THEN IF A = 21 THEN
Print “Key” ENDIFENDIF
IF A > 0 THEN
ENDIF
Print “Key”
IF A = 21 THEN
ENDIF
Example 2
- Cyclomatic complexity: _____Cyclomatic complexity: _____- Minimum tests to achieve:Minimum tests to achieve:
• Statement coverage: ______Statement coverage: ______
• Branch coverage: _____Branch coverage: _____
3
13
Read A
YesA=21
Yes
No
End
A>0
No
Read
Example 3
- Cyclomatic complexity: _____Cyclomatic complexity: _____- Minimum tests to achieve:Minimum tests to achieve:
• Statement coverage: ______Statement coverage: ______
• Branch coverage: _____Branch coverage: _____
Read ARead BIF A > 0 THEN IF B = 0 THEN
Print “No values” ELSE
Print BIF A > 21 THEN Print AENDIF
ENDIFENDIF
4
24
PrintYes
A>21
No
No
B=0
Yes
Yes
Read
A>0
End
No
- Cyclomatic complexity: _____Cyclomatic complexity: _____- Minimum tests to achieve:Minimum tests to achieve:
• Statement coverage: ______Statement coverage: ______
• Branch coverage: _____Branch coverage: _____
Read ARead BIF A < 0 THEN Print “A negative”ELSE Print “A positive”ENDIFIF B < 0 THEN Print “B negative”ELSE Print “B positive”ENDIF
3
22
Example 4 Read
End
No
PrintYes
B<0
No
PrintYes
A<0
Note: thereare 4 paths
End
No
- Cyclomatic complexity: _____Cyclomatic complexity: _____- Minimum tests to achieve:Minimum tests to achieve:
• Statement coverage: ______Statement coverage: ______
• Branch coverage: _____Branch coverage: _____
Read ARead BIF A < 0 THEN Print “A negative”ENDIFIF B < 0 THEN Print “B negative”ENDIF
3
12
Example 5 Read A<0 PrintYes
No
B<0 PrintYes
- Cyclomatic complexity: _____Cyclomatic complexity: _____- Minimum tests to achieve:Minimum tests to achieve:
• Statement coverage: ______Statement coverage: ______
• Branch coverage: _____Branch coverage: _____
Read AIF A < 0 THEN Print “A negative”ENDIFIF A > 0 THEN Print “A positive”ENDIF
3
22
Example 6 Read PrintYes
No
End
No
A>0 PrintYes
A<0
Contents
What is a testing technique?
Black and White box testing
Black box test techniques
White box test techniques
Error Guessing
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice
Non-systematic test techniques
Trial and error / Ad hoc Error guessing / Experience-driven User Testing Unscripted Testing
A testing approach that is onlyrigorous, thorough and systematic
is incomplete
A testing approach that is onlyrigorous, thorough and systematic
is incomplete
Error-Guessing
always worth including after systematic techniques have been used can find some faults that systematic
techniques can miss a ‘mopping up’ approach supplements systematic techniques
Not a good approach to start testing withNot a good approach to start testing with
Error Guessing: deriving test cases
Consider:- past failurespast failures- intuitionintuition- experienceexperience- brain stormingbrain storming- ““What is the craziest thing we can do?”What is the craziest thing we can do?”
Summary: Key Points
Test techniques are ‘best practice’: help to find faults
Black Box techniques are based on behaviour
White Box techniques are based on structure
Error Guessing supplements systematic techniques
Dynamic Testing Techniques
1 2 3
4 5 6
ISTQB / ISEB Foundation Exam Practice