Jan 20, 2016
June 4, 1997 OO Software Testing 2
A Brief Meditation on the Essential Nature of Software Testing
Thus spoke the master: “Any program, no matter how small, contains bugs.”The novice did not believe the master’s words. “What if the program were so small that itperformed a single function?” he asked.
“Such a program would have no meaning,” said the master, “but if such a one existed, theoperating system would fail eventually, producing a bug.”But the novice was not satisfied. “What if the operating system did not fail?” he asked.
“There is no operating system that does not fail,” said the master, “but if such a one existed,the hardware would fail eventually, producing a bug.”The novice still was not satisfied. “What if the hardware did not fail?” he asked.
The master gave a great sigh. “There is no hardware that does not fail,” he said, “but if sucha one existed, the user would want the program to do something different, and this too is abug.A program without bugs would be an absurdity, a nonesuch. If there were a program withoutany bugs then the world would cease to exist.”
Geoffrey James, The Zen of Programming
June 4, 1997 OO Software Testing 3
Object-Oriented Testing Papers
• Object-Oriented Integration Testing– Jorgensen and Erickson– CACM Sept. 1994
• Object-Oriented Software Testing - A Hierarchical Approach– Shel Siegel– J. Wiley and Sons, 1996
June 4, 1997 OO Software Testing 4
Structure vs. Behavior
• most popular notations in software development portray software structure
• examples: E/R models, data flow diagrams, structure charts, etc.
– used to show relationships between components, interfaces, control and dataflow, etc.
– not very useful for testing
• SW testing is fundamentally concerned with behavior, i.e. what it does
June 4, 1997 OO Software Testing 5
Traditional Testing Notions
• Traditional software is– written in an imperative language– described by functional decomposition– developed in a waterfall life cycle– separated into three levels of testing
• unit level, integration level, system level
• these points do often not apply directly to object-oriented software
June 4, 1997 OO Software Testing 6
Imperative vs. Declarative
• Imperative languages lend themselves to a description as a directed graph
• declarative languages suppress sequentiality
• the event-driven nature of OO systems forces a “declarative spirit” on testing– not evident at unit level since most object-
oriented languages are imperative– but pronounced at integration and system levels
June 4, 1997 OO Software Testing 7
Unit Testing Overview
• Units can be defined in several ways• a single, cohesive function
• a function which, when coded fits on one page
• the smallest separately compilable segment of code
• the amount of code that can be written in 4 to 40 hours
• a task in a work breakdown structure
• code that is assigned to one person
• code that one person designs, codes and test in a three-month period
June 4, 1997 OO Software Testing 8
Unit Testing Overview (cont.)
• Many organizations have not chosen their definition of a software unit
• Units are always tested “by itself”– stubs and drivers are used to emulate inputs
and outputs– goal: verification that, taken by itself, the unit
functions correctly
June 4, 1997 OO Software Testing 9
Integration Testing Overview
• Once the units are separately tested, they are integrated together
• gradual replacement of stubs and drivers by separately tested units
• pairwise integration, i.e. each unit is integrated with its adjacent units
• bottom-up integration guided by functional decomposition tree
• “big bang” integration where all units are thrown together at once
June 4, 1997 OO Software Testing 10
System Testing Overview
• Conducted exclusively in terms of inputs and outputs that are visible at the port boundary of a system
• Tester only has access to those port events that are visible to the customer/user
• Can occur on the target platform only
June 4, 1997 OO Software Testing 11
System Testing Overview (cont.)
• Examples of system testing activities• sequences of machine instructions
• sequences of source instructions
• scenarios of normal usage
• system-level test cases
• stimulus/response pairs
• behavior that results from sequences of system-level inputs
• sequences of transitions in a state machine description of the system
June 4, 1997 OO Software Testing 12
Object-Oriented Software
– within an object, individual methods are imperative– declarative aspect because of event-driven nature– dynamic binding creates indefiniteness that
resembles declarative programs– it is impossible to ever know the full set of
“adjacent” objects with which a given object may be composed
• two object may be correct, but errors might result when they are composed
June 4, 1997 OO Software Testing 13
Composing Errors Illustrated
June 4, 1997 OO Software Testing 14
Levels of OO Testing
• Unit testing– method testing
• Integration testing– Message quiescence– Event quiescence
• System testing– Thread testing– Thread interaction testing
June 4, 1997 OO Software Testing 15
Integration Testing
• Method executions are linked by messages
• Quiescence provides natural endpoints
• Message quiescence– Method/Message path (MM-Path)
• sequence of message executions linked by messages• starts with a method• ends when method is reached that does not issue any
messages of its own– endpoint is called message quiescence
June 4, 1997 OO Software Testing 16
Integration Testing (cont.)
• Atomic System Function (ASF)• input port event,
• followed by a set of MM-Paths,
• terminated by output port event
– ASFs constitute the point at which integration and system testing meet
– elemental function visible at the system level
June 4, 1997 OO Software Testing 17
MM-Paths and ASFs Illustrated
June 4, 1997 OO Software Testing 18
ATM System Class Hierarchy
June 4, 1997 OO Software Testing 19
Example: ATM System
• Examples of– MM-Path: Digit entry– ASF: PIN Entry– Thread (sequence of ASFs):
• simple transaction: PIN entry, transaction type selection, account detail presentation, operation execution, result reporting
– Thread sequence:• a session consisting of multiple simple transactions
June 4, 1997 OO Software Testing 20
Digit Entry MM-Path Illustrated
June 4, 1997 OO Software Testing 21
PIN Entry ASF Illustrated
June 4, 1997 OO Software Testing 22
Observations
• Many-to-many relationships dominate (see E/R diagram)– an object may be involved in
many threads
– threads entail many objects
– an object may be involved with many ASFs
– ASFs may entail many objects
June 4, 1997 OO Software Testing 23
Observations (cont.)
• The five levels of OO testing result in distinct, useful testing goals
• Bottom-up testing order
• Seamless juncture between OO integration testing and system testing
• Constructs are directly usable as basis for test coverage metrics
• Works well with composition
June 4, 1997 OO Software Testing 24
The Hierarchical Approach
• Everything is a system
• Defines and applies testing standards for several levels of software components– objects– classes– foundation components– systems
June 4, 1997 OO Software Testing 25
Hierarchical Approach Illustrated
June 4, 1997 OO Software Testing 26
Safe Components
• Components that meet the testing standards for that kind of components
• only safe components can be integrated with other safe components to produce the next-level component
• Safe is a relative state– depends on risk management practices, attitude
toward risk, what is enforced, the application, etc.
June 4, 1997 OO Software Testing 27
Foundation Components
• Can be one complete class hierarchy
• or some cluster of classes– that performs a core function,– that represents a logical or physical
architectural component, etc.
• After testing to a safe level, can be integrated with other foundation components
June 4, 1997 OO Software Testing 28
Integration Testing of Foundation Components
• Needs to address the interconnections between the foundation components and any new composite functionality ONLY
• Eliminates the need to test all of the combinations of states during integration testing
• Improves productivity
June 4, 1997 OO Software Testing 29
Integration Testing Problem Illustrated
June 4, 1997 OO Software Testing 30
Test Suites
• Conditional Test Suite
• Hierarchical Incremental Test Suite
• Integration Test Suite
• System Test Suite
• Regression Test Suite
June 4, 1997 OO Software Testing 31
Conditional Test Suite
• Four types of test scripts– Assertions: abort if condition fails– Exceptions: raise exception if condition succeeds– Concurrent test operations: monitor condition– Message polling: evaluate condition on demand
• Conditions– class invariant, precondition, postcondition, loop
invariant
June 4, 1997 OO Software Testing 32
Pre/Postconditions Illustratedclass STACK2 [T] export
push, pop, top, empty, nb_elements, fullfeature
implementation: ARRAY [T]max_size: INTEGER;nb_elements: INTEGER;
Create (n: INTEGER) is-- Allocate stack for a maximum of n elements-- (or for no elements if n < 0)
doif n > 0 then max_size := n end;implementation.Create (1, max_size)
end; -- Create
empty: BOOLEAN is-- is stack empty?
doResult := (nb_elements = 0)
end; -- empty
full: BOOLEAN is-- Is stack full?
doResult := (nb_elements = max_size)
end; -- empty
pop is-- Remove top element
requirenot empty – i.e. nb_elements > 0
donb_elements := nb_elements – 1
ensurenot full;nb_elements = old nb_elements – 1
end; -- pop
top: T is-- Top element
requirenot empty – i.e. nb_elements > 0
doResult := implementation.entry (nb_elements)
end; -- top
push (x: T) is-- Add x on top
requirenot full
-- i.e. nb_elements < array_size in this-- representation
donb_elements := nb_elements + 1;implementation.enter (nb_elements, x)
ensurenot empty;top = x;nb_elements = old nb_elements + 1
end; -- pushend – class STACK2
June 4, 1997 OO Software Testing 33
Hierarchical Incremental Test Suite
• Structural models– control-flow model
• models class as series of control-flow graphs deriving from the method implementations
– data-flow model• models class as series of control-flow graphs with
data annotations showing the flow of data values through the method implementations
June 4, 1997 OO Software Testing 34
Hierarchical Incremental Test Suite (cont.)
• Functional models– state-transition model, aka OMT dynamic model
• models states and transitions for a class in the context of its position in the inheritance hierarchy
– transaction-flow model• models object life cycle as transaction from construction to
destruction
– exception model• specifies what methods raise what exceptions under what
conditions
June 4, 1997 OO Software Testing 35
Integration Test Suite
• Functional Test Suite– tests vertical integration of a cluster of
components, i.e. the way the cluster implements specific functions from the requirements
• Abstraction Test Suite– tests horizontal integration of a cluster of
components, i.e. the way the cluster abstracts a service contract into an API
June 4, 1997 OO Software Testing 36
Integration Test Suite (cont.)
• Possible test scripts– class-to-class test script
• interaction between classes based on message relationship
• protocol-interface, synchronization, exception handling tests
– UI test script• tests UI components of cluster and the components on
which they depend
• true-time, character-recognition, widget-playback tests
June 4, 1997 OO Software Testing 37
System Test Suite
• System Acceptance Test Suite– derived from the set of use cases covering the
system requirements
• User Options Test Suite– based on probable usage patterns of the features
and functions delivered with the system– designed to simulate the way the end users will
combine the features and functions
June 4, 1997 OO Software Testing 38
Regression Test Suite
• Series of tests run against a previous version of the CUT
• Almost always automated– performing manual tests is usually not cost
effective
June 4, 1997 OO Software Testing 39
Epilog
“Testing proves the presence, not the absence, of bugs.”
E.W. Dijkstra
“Absence of evidence is not evidence of absence.”
Source unknown
From: E.V. Berard, Issues in the Testing of OO Software
Slides available at http://www.ics.uci.edu/~jfeise/ICS224A