Defect testing l Testing programs to establish the presence of …scharff/cs389/ref/ch20cs389.pdf · 2002-02-26 · Chapter 20 Slide 1 Defect testing l Testing programs to establish
Post on 12-Aug-2020
0 Views
Preview:
Transcript
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 1
Defect testing
l Testing programs to establishthe presence of system defects
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 2
Objectives
l To understand testing techniques that are gearedto discover program faults
l To introduce guidelines for interface testing
l To understand specific approaches to object-oriented testing
l To understand the principles of CASE toolsupport for testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 3
Topics covered
l Defect testing
l Integration testing
l Object-oriented testing
l Testing workbenches
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 4
The testing process
l Component testing• Testing of individual program components
• Usually the responsibility of the component developer (exceptsometimes for critical systems)
• Tests are derived from the developer’s experience
l Integration testing• Testing of groups of components integrated to create a system or
sub-system
• The responsibility of an independent testing team
• Tests are based on a system specification
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 5
Testing phases
Componenttesting
Integrationtesting
Software developer Independent testing team
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 6
Defect testing
l The goal of defect testing is to discover defects inprograms
l A successful defect test is a test which causes aprogram to behave in an anomalous way
l Tests show the presence not the absence ofdefects
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 7
l Only exhaustive testing can show a program isfree from defects. However, exhaustive testingis impossible
l Tests should exercise a system's capabilitiesrather than its components
l Testing old capabilities is more important thantesting new capabilities
l Testing typical situations is more important thanboundary value cases
Testing priorities
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 8
l Test data Inputs which have been devised totest the system
l Test cases Inputs to test the system and thepredicted outputs from these inputs if thesystem operates according to its specification
Test data and test cases
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 9
The defect testing process
Design testcases
Prepare testdata
Run programwith test data
Compare resultsto test cases
Testcases
Testdata
Testresults
Testreports
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 10
Black-box testing
l An approach to testing where the program isconsidered as a ‘black-box’
l The program test cases are based on the systemspecification
l Test planning can begin early in the softwareprocess
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 11
Black-box testing
Ie
Input test data
OeOutput test results
System
Inputs causinganomalousbehaviour
Outputs which revealthe presence ofdefects
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 12
Equivalence partitioning
l Input data and output results often fall intodifferent classes where all members of a class arerelated
l Each of these classes is an equivalence partitionwhere the program behaves in an equivalent wayfor each class member
l Test cases should be chosen from each partition
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 13
Equivalence partitioning
System
Outputs
Invalid inputs Valid inputs
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 14
l Partition system inputs and outputs into‘equivalence sets’• If input is a 5-digit integer between 10,000 and 99,999,
equivalence partitions are <10,000, 10,000-99, 999 and >10, 000
l Choose test cases at the boundary of thesesets• 00000, 09999, 10000, 99999, 10001
Equivalence partitioning
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 15
Equivalence partitions
Between 10000 and 99999Less than 10000 More than 99999
999910000 50000
10000099999
Input values
Between 4 and 10Less than 4 More than 10
34 7
1110
Number of input values
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 16
Search routine specification
procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;
Pre-condition-- the array has at least one elementT’FIRST <= T’LAST
Post-condition-- the element is found and is referenced by L( Found and T (L) = Key)
or -- the element is not in the array( not Found and
not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 17
l Inputs which conform to the pre-conditions
l Inputs where a pre-condition does not hold
l Inputs where the key element is a member ofthe array
l Inputs where the key element is not a memberof the array
Search routine - input partitions
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 18
Testing guidelines (sequences)
l Test software with sequences which have only asingle value
l Use sequences of different sizes in different tests
l Derive tests so that the first, middle and lastelements of the sequence are accessed
l Test with sequences of zero length
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 19
Search routine - input partitionsArray ElementSingle value In sequenceSingle value Not in sequenceMore than 1 value First element in sequenceMore than 1 value Last element in sequenceMore than 1 value Middle element in sequenceMore than 1 value Not in sequence
Input sequence (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 29, 21, 23 17 true, 141, 18, 9, 31, 30, 16, 45 45 true, 717, 18, 21, 23, 29, 41, 38 23 true, 421, 23, 29, 33, 38 25 false, ??
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 20
l Sometime called white-box testing
l Derivation of test cases according to programstructure. Knowledge of the program is used toidentify additional test cases
l Objective is to exercise all program statements(not all path combinations)
Structural testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 21
White-box testing
Componentcode
Testoutputs
Test data
DerivesTests
Binary search (Java)
class BinSearch {
// This is an encapsulation of a binary search function that takes an array of// ordered objects and a key and returns an object with 2 attributes namely// index - the value of the array index// found - a boolean indicating whether or not the key is in the array// An object is returned because it is not possible in Java to pass basic types by// reference to a function and so return two values// the key is -1 if the element is not found
public static void search ( int key, int [] elemArray, Result r ){
int bottom = 0 ;int top = elemArray.length - 1 ;int mid ;r.found = false ; r.index = -1 ;while ( bottom <= top ){
mid = (top + bottom) / 2 ;if (elemArray [mid] == key){
r.index = mid ;r.found = true ;return ;
} // if partelse{
if (elemArray [mid] < key)bottom = mid + 1 ;
elsetop = mid - 1 ;
}} //while loop
} // search} //BinSearch
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 23
l Pre-conditions satisfied, key element in array
l Pre-conditions satisfied, key element not inarray
l Pre-conditions unsatisfied, key element in array
l Pre-conditions unsatisfied, key element not inarray
l Input array has a single value
l Input array has an even number of values
l Input array has an odd number of values
Binary search - equiv. partitions
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 24
Binary search equiv. partitions
Mid-point
Elements < Mid Elements > Mid
Equivalence class boundaries
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 25
Binary search - test cases
Input array (T) Key (Key) Output (Found, L)17 17 true, 117 0 false, ??17, 21, 23, 29 17 true, 19, 16, 18, 30, 31, 41, 45 45 true, 717, 18, 21, 23, 29, 38, 41 23 true, 417, 18, 21, 23, 29, 33, 38 21 true, 312, 18, 21, 23, 32 23 true, 421, 23, 29, 33, 38 25 false, ??
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 26
Path testing
l The objective of path testing is to ensure that theset of test cases is such that each path through theprogram is executed at least once
l The starting point for path testing is a programflow graph that shows nodes representingprogram decisions and arcs representing the flowof control
l Statements with conditions are therefore nodes inthe flow graph
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 27
l Describes the program control flow. Each branchis shown as a separate path and loops are shownby arrows looping back to the loop conditionnode
l Used as a basis for computing the cyclomaticcomplexity
l Cyclomatic complexity = Number of edges -Number of nodes +2
Program flow graphs
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 28
l The number of tests to test all controlstatements equals the cyclomatic complexity
l Cyclomatic complexity equals number ofconditions in a program
l Useful if used with care. Does not implyadequacy of testing.
l Although all paths are executed, all combinationsof paths are not executed
Cyclomatic complexity
Binary search flow graph
1
2
3
4
65
7
while bottom <= top
if (elemArray [mid] == key
(if (elemArray [mid]< key8
9
bottom > top
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 30
l 1, 2, 3, 8, 9
l 1, 2, 3, 4, 6, 7, 2
l 1, 2, 3, 4, 5, 7, 2
l 1, 2, 3, 4, 6, 7, 2, 8, 9
l Test cases should be derived so that all of thesepaths are executed
l A dynamic program analyser may be used tocheck that paths have been executed
Independent paths
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 31
Integration testing
l Tests complete systems or subsystems composedof integrated components
l Integration testing should be black-box testingwith tests derived from the specification
l Main difficulty is localising errors
l Incremental integration testing reduces thisproblem
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 32
Incremental integration testing
T3
T2
T1
T4
T5
A
B
C
D
T2
T1
T3
T4
A
B
C
T1
T2
T3
A
B
Test sequence1
Test sequence2
Test sequence3
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 33
Approaches to integration testing
l Top-down testing• Start with high-level system and integrate from the top-down
replacing individual components by stubs where appropriate
l Bottom-up testing• Integrate individual components in levels until the complete
system is created
l In practice, most integration involves acombination of these strategies
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 34
Top-down testing
Level 2Level 2Level 2Level 2
Level 1 Level 1Testingsequence
Level 2stubs
Level 3stubs
. . .
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 35
Bottom-up testing
Level NLevel NLevel NLevel NLevel N
Level N–1 Level N–1Level N–1
Testingsequence
Testdrivers
Testdrivers
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 36
Tetsing approaches
l Architectural validation• Top-down integration testing is better at discovering errors in
the system architecture
l System demonstration• Top-down integration testing allows a limited demonstration at
an early stage in the development
l Test implementation• Often easier with bottom-up integration testing
l Test observation• Problems with both approaches. Extra code may be required to
observe tests
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 37
l Takes place when modules or sub-systems areintegrated to create larger systems
l Objectives are to detect faults due to interfaceerrors or invalid assumptions about interfaces
l Particularly important for object-orienteddevelopment as objects are defined by theirinterfaces
Interface testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 38
Interface testingTestcases
BA
C
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 39
Interfaces types
l Parameter interfaces• Data passed from one procedure to another
l Shared memory interfaces• Block of memory is shared between procedures
l Procedural interfaces• Sub-system encapsulates a set of procedures to be called by
other sub-systems
l Message passing interfaces• Sub-systems request services from other sub-systems
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 40
Interface errors
l Interface misuse• A calling component calls another component and makes an
error in its use of its interface e.g. parameters in the wrong order
l Interface misunderstanding• A calling component embeds assumptions about the behaviour
of the called component which are incorrect
l Timing errors• The called and the calling component operate at different speeds
and out-of-date information is accessed
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 41
Interface testing guidelines
l Design tests so that parameters to a calledprocedure are at the extreme ends of their ranges
l Always test pointer parameters with null pointers
l Design tests which cause the component to fail
l Use stress testing in message passing systems
l In shared memory systems, vary the order inwhich components are activated
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 42
Stress testing
l Exercises the system beyond its maximum designload. Stressing the system often causes defects tocome to light
l Stressing the system test failure behaviour..Systems should not fail catastrophically. Stresstesting checks for unacceptable loss of service ordata
l Particularly relevant to distributed systemswhich can exhibit severe degradation as anetwork becomes overloaded
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 43
l The components to be tested are object classesthat are instantiated as objects
l Larger grain than individual functions soapproaches to white-box testing have to beextended
l No obvious ‘top’ to the system for top-downintegration and testing
Object-oriented testing
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 44
Testing levels
l Testing operations associated with objects
l Testing object classes
l Testing clusters of cooperating objects
l Testing the complete OO system
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 45
Object class testing
l Complete test coverage of a class involves• Testing all operations associated with an object
• Setting and interrogating all object attributes
• Exercising the object in all possible states
l Inheritance makes it more difficult to designobject class tests as the information to be tested isnot localised
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 46
Weather station object interface
l Test cases are needed for alloperations
l Use a state model to identifystate transitions for testing
l Examples of testingsequences
• Shutdown ♦ Waiting ♦ Shutdown
• Waiting ♦ Calibrating ♦ Testing ♦Transmitting ♦ Waiting
• Waiting ♦ Collecting ♦ Waiting ♦Summarising ♦ Transmitting ♦ Waiting
identifier
reportWeather ()calibrate (instruments)test ()startup (instruments)shutdown (instruments)
WeatherStation
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 47
Object integration
l Levels of integration are less distinct in object-oriented systems
l Cluster testing is concerned with integrating andtesting clusters of cooperating objects
l Identify clusters using knowledge of the operationof objects and the system features that areimplemented by these clusters
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 48
Approaches to cluster testing
l Use-case or scenario testing• Testing is based on a user interactions with the system
• Has the advantage that it tests system features as experienced byusers
l Thread testing• Tests the systems response to events as processing threads
through the system
l Object interaction testing• Tests sequences of object interactions that stop when an object
operation does not call on services from another object
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 49
Scenario-based testing
l Identify scenarios from use-cases and supplementthese with interaction diagrams that show theobjects involved in the scenario
l Consider the scenario in the weather stationsystem where a report is generated
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 50
Collect weather data:CommsController
request (report)
acknowledge ()report ()
summarise ()
reply (report)
acknowledge ()
send (report)
:WeatherStation :WeatherData
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 51
Weather station testing
l Thread of methods executed• CommsController:request ♦ WeatherStation:report ♦
WeatherData:summarise
l Inputs and outputs• Input of report request with associated acknowledge and a final
output of a report
• Can be tested by creating raw data and ensuring that it issummarised properly
• Use the same raw data to test the WeatherData object
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 52
Testing workbenches
l Testing is an expensive process phase. Testingworkbenches provide a range of tools to reducethe time required and total testing costs
l Most testing workbenches are open systemsbecause testing needs are organisation-specific
l Difficult to integrate with closed design andanalysis workbenches
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 53
A testing workbench
Dynamicanalyser
Programbeing tested
Testresults
Testpredictions
Filecomparator
Executionreport
Simulator
Sourcecode
Testmanager Test data Oracle
Test datagenerator Specification
Reportgenerator
Test resultsreport
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 54
Tetsing workbench adaptation
l Scripts may be developed for user interfacesimulators and patterns for test data generators
l Test outputs may have to be prepared manuallyfor comparison
l Special-purpose file comparators may bedeveloped
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 55
Key points
l Test parts of a system which are commonly usedrather than those which are rarely executed
l Equivalence partitions are sets of test cases wherethe program should behave in an equivalent way
l Black-box testing is based on the systemspecification
l Structural testing identifies test cases which causeall paths through the program to be executed
©Ian Sommerville 2000 Software Engineering, 6th edition. Chapter 20 Slide 56
Key points
l Test coverage measures ensure that all statementshave been executed at least once.
l Interface defects arise because of specificationmisreading, misunderstanding, errors or invalidtiming assumptions
l To test object classes, test all operations,attributes and states
l Integrate object-oriented systems around clustersof objects
top related