Copyright 1993-1998 Timothy Korson. All rights reserved.Borland
Conference, Denver, August 11, 19981Testing Across the
Lifecyclepresented atNinth Annual BorlandConference August 11,
1998Denver ColoradobyTimothy D. Korson, Ph. d.Senior Partner,
Software ArchitectsandDean, School of ComputingSouthern Adventist
University(423)
238-3288korson@software-architects.comwww.software-architects.comEmail:
[email protected]: (423) 238-3288Web:
www.software-architects.comFax: (423) 238-3289Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 19982Section 1Course OverviewIn this section we will
discuss:Course GoalsTesting ContextRisk AnalysisCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-3Development Goals Affect the Testing Process Reuse
- requires thorough testing across the complete specification.
Shorter development time - requires an automated technique for
class, system, and regression testing. Higher quality - requires
systematic testing over the entire development process.
Extensibility - requires the inclusion of hypothetical test cases
to search for problems encountered in extending the design.
Copyright 1993-1998 Timothy Korson. All rights reserved.Borland
Conference, Denver, August 11, 1998-4Why Do We Test?Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-5Testing Testing consists of all activities
that increase our confidence that the system will do what it should
and not do what it shouldnt. More specifically, testing is a
process of reducing risk by comparing what is and what should be.
In the object oriented paradigm, faults may be in: The analysis
model The design model The implementation Those faults are the
result of defective development processes which allow (and in some
cases even foster): Improper analysis Improper design Improper
implementationCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-6Risk Analysis
Risk analysis is the process used to decide where to allocate
limited testing resources. In the object oriented paradigm, risk
analysis becomes the basis for determining the level of testing
classes, clusters and systems. Use risk analysis to determine the
types of errors that are more critical to system success and focus
resources in these areas. Develop a standard risk scale that
developers can use to categorize their components.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-7Risk Analysis Establish risk criteria:
Complexity of idea Stability of specification Maturity level of
class Risk of injury - financial, safety, etc. Use risk analysis
to: Prioritize component tests Allocate testing resources Choose
the number of test cases Decide which features to cover more
completely Re-evaluate as classes mature Reallocate resources over
iterations Risk should be reduced over iterationsCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-8Classic Waterfall DevelopmentCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-9System
TestingDomainAnalysisApplicationAnalysisArchitecture &High
Level DesignDetailed DesignClass
DevelopmentApplicationAssemblySystem
TestingDomainAnalysisApplicationAnalysisArchitecture &High
Level DesignDetailed DesignClass
DevelopmentApplicationAssemblySystem
TestingDomainAnalysisApplicationAnalysisArchitecture &High
Level DesignDetailed DesignClass
DevelopmentApplicationAssemblyIncrementsIterationIterative
IncrementalDevelopment ProcessCopyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
1998-10The Object-Oriented Process Model The iterative incremental
process model is a widely used approach for building
object-oriented software systems. The dual goals of the
object-oriented development process are: To generate
applicationsAND To develop an inventory of components and
frameworks for use in future projects How do these dual goals
affect testing?Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-11Myths About
Testing Object-Oriented
SystemsMyth:Object-orientedprogramsareeasiertotestbecause once a
method is tested, it never needs tobe tested again.Reality: Not
necessarily so.Myth: Object-oriented programs are harder to
testbecause the use of dynamic binding means that anyobject can
send a message to any other object.Reality: Not necessarily
so.Testing object-oriented systems is not nec-essarily more or less
difficult, it is simply DIFFERENT!Copyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
1998-12What is Different Adapting to a use case requirements
process Understanding and interacting with domain and application
level class diagrams Adapting to the impact of OO language
constructs on unit testing Increased importance of unit testing
Establishing a new metrics program Learning how to create a
parallel architecture for component testing Learning how to use
orthogonal arrays to test framework interactions Implementing
support for hierarchical incremental testing for inheritance trees
Adjusting to an incremental/iterative software development process
Creating a certification process for standard components and
frameworks Learning the impact of object distribution mechanisms on
testing distributed and web-enabled applications Adapting ISO
requirements to the new OO development processes New ways of
interacting with software development teams which may involve
organizational adjustments. Defining an effective regression
testing process to match the iterative development
process.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-13What do We
Test? Requirements Models Architectures and Frameworks Designs
ComponentsClassCluster Increments Applications ProcessesCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-14Testing Requirements Prototypes Use
CasesUnfortunately most testing organizaions take requirements as a
given. Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-15Testing
ModelsDo we understand the business?Pedestrian TCS Vehicle
TCSClockControllerPhase Plan
PhaseIntersectionRoadDetectorLaneTrafficTraffic Control
SignalPressureDetectorMagneticDetector Push Button Detector
RadioDetectorPedestrian VehicleTraffic ControlOfficerCrosswalkLeft
Turn ThruOther Detector Types.. .Other Types of
Lanes...2+1+1+1+1+1+1+TrafficIntersection Domain Analysis Object
ModelCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-16Testing
FrameworksConsider all possible combinations of Polymorphic
Substitutionsm(P)ABCXYZPQ RCopyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-17Testing DesignsWhat If1. Meets best current practices and
guidelines2. is the most robust against What If?Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-18Testing Components PACT-- Parrallel Architecture
for Component Testing IT--Hierarchiacal Incremntal TestingSoftware
developers should do the unit testing.Copyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
1998-19Testing Increments Stubs SimulationGrey Box TestingIncrement
5System?Unit?Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-20Testing
ApplicationsNot much new here...Except Use Case ---> Test Case
First N increments already testedCopyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
199821Testing Process Testing effectiveness is determined by the
percentage of total defects found at each step in the testing phase
Our goal is to find a larger percentage in the early stages and
smaller percentages in later phases. Effectiveness of the overall
testing process is defined as: Faults found during system testing
divided by total number of faults found by all testing activities
(including the testing done by customers that we call
production).Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 199822Section
2Testing Object-Oriented ModelsIn this sectionwe will discuss:
Static Testing Reviews Inspections Walkthroughs Inspecting
Requirements Inspecting Analysis ModelsCopyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
1998-23Static Testing Static testing is the testing of a product
without actually executing it. Static testing is used to identify
errors in requirements, analysis and design models, coding, plans,
and tests. Static tests consist of: Reviews Walkthroughs
Inspections (the primary focus of this chapter) The goals of static
testing are: To detect defects as early aspossible in the
developmentprocess To improve the software development
processCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-24Inspections
inObject-Oriented Projects Good news: The use of domain terminology
in the development process makes it easier for clients to
effectively participate in inspections. The iterative/incremental
development process produces smaller chunks of work products
allowing inspections to be more thorough in their coverage. Bad
news: The iterative/incremental approach requires careful
coordination and scheduling of inspections.Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-25C3 - Criteria for Testing Models Correctness It
is judged to be equivalent to a reference standard believed to be
an infallible source of truth, i.e. a domain expert. Completeness
No required elements are missing. It is judged complete if the
knowledge recorded is sufficient to support the goals of the
current portion of the system being developed. Consistency There
are no contradictions among the elements within the work product
(internal) or between work products (external).Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 199826Create Domain-level Object Interaction Diagrams A
sequence diagram1 is a diagram that formally describes a scenario.
Each class is shown as a vertical line; each event as a horizontal
arrow from the sender class to the receiver class; time flows from
the top to the bottom of the diagram.1.Event trace or object
interaction diagramInstructor Student Work
GraderStudentGradeCreateSubmit (Student Work)DetermineRecord
(Grade)CreateAssignmentBookGradeAssign (Assignment)Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 199827Domain Analysis Summary The minimal set of
UML deliverables includes use cases, class diagrams, and sequence
diagrams. Of these, the class diagram is core.n The sequence
diagrams show how the use cases are played out in the class
diagramCategoryStudent WorkActual Student Work
AnnotationGuardianStudentAssignmentGrade******Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 199828Section 3Testing ComponentsIn this section we will
discuss: A philosophy of testing Class testing- Functional testing-
Structural testing- Interaction testing Subclass testing Cluster
testing Framework testing-Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-29Levels of Testing Object-Oriented Systems Class testing -
the smallest unit that should be tested. This level of testing
combines traditional unit testing with some aspects of integration
testing. Cluster testing - a set of closely interacting classes.
The focus at this level is the interactions between objects in the
cluster. System testing - a complete system that could be
represented as a single class. The focus at this level is
demonstrating the required functionality. Regression testing -
becomes an integral part of the development process because each
iteration refines products from the previous iteration and these
products should be retested.Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11, 1998-30Goals
for a Testing Process The first goal in developing a testing
process is to reduce the number of test cases that must be
generated. The second goal is to minimize the number of test cases
that must be executed and the results validated. Testing algorithms
should provide a volume control approach that supports increasing
the level of confidence in a systematic way.Minimal
ExhaustiveAdequate RepresentativeTest Coverage Volume
ControlWeightedRepresentativeCopyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
1998-31Types of Objects Passive Also known as Abstract Data Types
(ADT) or primitives No change in behavior based on state Active
Also known as Finite State Machines (FSM) Significant changes in
behavior depending upon stateCopyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
1998-32Types of Test Cases Three types of test cases allow the
tester to know that various aspects of the component or system
under test are correct: Functional test cases are created to verify
the product against the specification. Structural test cases are
created to fully exercise the code. Interaction test cases are
created to determine the correctness of class interaction.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-33Definition of Terms A test case is a
sequence of messages and an expected result. A test script first
creates the environment for the execution of test case(s); it
creates the OUT (object under test), tests it, and finally deletes
it. A test suite is a grouping of test scripts that achieve some
level of test coverage.For example, the functional suite covers the
class specification.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-34Topics A
philosophy of testing Class testing Functional testing Structural
testing Interaction testing Subclass testing Cluster testing
Framework testingCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-35Functional
Test Cases Constructed by analyzing the specification of the class
- the aggregation of the specification of each method. Coverage may
be expressed in terms of the percentage of post-conditions, or the
percentage of transitions in the state representation, covered with
the selected test cases. Impossible to guarantee any level of
coverage of the underlying implementation. Synonyms: (1)
Specification-based; (2) Black-box testing.Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-36Constructing Functional Test Cases Each
pre-condition is used to establish the appropriate testing
environment for object. Each post-condition is a logical statement
constructed as a sequence of if-then clauses. These may be linked
together with disjunctive (or) clauses. Each disjunctive clause may
have several conjunctive (and) clauses. Example for the pop method
of a Stack class:(If the stack contains items, then pop will return
the last item added to the stack and the stack will contain one
less item) or (if the stack is empty initially, then pop will raise
an UnderFlow exception.)Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-37Constructing Functional Test Cases Create a test case for
each or clause. For each or clause check the resulting environment
to determine that every and clause within the or clause has been
satisfied. Create a test case for each exception. Create test cases
for obvious boundary conditions such as empty stacks, full stacks,
a stack with only one element.Copyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
1998-38Topics A philosophy of testing Class testing Functional
testing Structural testing Interaction testing Subclass testing
Cluster testing Framework testingCopyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
1998-39Structural Class Testing Use techniques similar to those for
procedurally-oriented analysis to identify test cases that exercise
the paths through the code. Use partially implemented classes to
speed integration testing. Complete stubbing is difficult since
many lines of code will be messages to other objects.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-40Topics A philosophy of testing Class
testing Functional testing Structural testing Interaction testing
Subclass testing Cluster testing Framework testingCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-41Interaction Test Cases - Intraclass Test
cases are identified by considering methods that access a common
attribute or send messages to other methods within the object.
Coverage would be expressed as percentage of interactions
tested.int widthset_width()get_width()Class RectangleHere the two
methods are in the same classCopyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
199842Interaction Test Planning Matrix Intra-class interactions are
identified as either one method invoking another (M), or as two
methods messaging the same object (O), or two methods sharing data
(D).startstartstopstopread_read_ lapsed lapsedresetresetSimple
TimerCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-43Basic Test
Case Execution Sequence Develop a functional test suite that covers
the complete class specification. Develop state-based test cases
for all of the transitions in the dynamic model. Develop structural
test cases to cover every line of code and every conditional.
Develop test cases that test the interactions between methods
within the class. Develop test cases that cover the interactions
between classes.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-44Topics A
philosophy of testing Class testing Functional testing Structural
testing Interaction testing Subclass testing Cluster testing
Framework testingCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-45Constructing
Test Cases for a Set of
SubclassesTestcaseconstructionstartsatthetopoftheinheritance
hierarchy and progresses
downward.Thisexpandsthenumberoftestcasesastheinterfaceoftheclassgrows.Thisprovidesopportunities
for reuse of test cases.ListQueue StackDequeCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-46Testing Abstract Classes An abstract class should
not be, and in some languages can not be, instantiated. Test cases
should be constructed even for abstract classes. For each method,
use the pre- and post-conditions of the methods to construct
functional test cases. Fully implemented methods in an abstract
class can have complete structural test suites developed and
executed. It should be possible to static check (i.e. compile) the
abstract class to test for interface errors. In C++, abstract (pure
virtual) methods can be changed to provide a null or stubbed
implementation, an object can be created, and some portion of the
class can be tested.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-47Questions
Regarding Subclass Testing Questions: Can the test cases from the
parent classes be used as test cases for the subclass? What parts
of the subclass must be tested or retested? Important Question:
Does the subclass conform to strict inheritance?Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-48Hierarchical Incremental Testing (HIT) Once a
superclass is tested, its subclasses can benefit from that testing.
HIT provides a technique for determining just which pieces of the
subclass must be retested.Superclass Not Defined Redefinable
Redefinable Abstract AbstractSubclass New Code No New CodeNew Code
No New CodeNew CodeFunctionalTestsWrite? yes no no no noExecute?
yes no yes no yesStructuralTestsWrite? yes no yes no yesExecute?
yes no yes no yesInteractionTestsWrite? yes no*yes*no yesExecute?
yes yes yes no yesm(){...} m(){...}m(){...}m(){...}m() m(){...}
m()Copyright 1993-1998 Timothy Korson. All rights reserved.Borland
Conference, Denver, August 11, 1998-49Topics A philosophy of
testing Class testing Functional testing Structural testing
Interaction testing Subclass testing Cluster testing Framework
testingCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-50Cluster
Interface A cluster is a set of closely related classes that
interact more with each other than other classes. The specification
for a cluster could take the same form as that of a class. There
are a limited number of messages that come into the cluster from
outside. These are the methods in the specification of the
cluster.ClusterInterfaceClusterRest ofSystemCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-51Clusters Pre- and post-conditions can be derived
for each method in the interface by using the conditions stated for
those methods in their respective classes. The test cases are
derived in the same way as for a class. Sometimes it is more
sensible for cluster testing to be the first level of testing
rather than testing the individual classes.Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-52Topics A philosophy of testing Class testing
Functional testing Structural testing Interaction testing Subclass
testing Cluster testing Framework testingCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-53Using Frameworks Frameworks reverse the usual
idea of the direction of reuse: In reusing a component, our code
calls the reused component. In reusing a framework, the framework
calls our code that specializes some operation for which the
framework contains a default or virtual
placeholder..frameworkapplicationCopyright 1993-1998 Timothy
Korson. All rights reserved.Borland Conference, Denver, August 11,
1998-54Testing When Using Frameworks A framework contains many
abstract classes with many partially implemented methods. A
framework contains many alternatives so there are many possible
paths through the classes. The classes that we add to the framework
are tested individually prior to the framework being tested (but
remember, they inherit from the framework classes so they are not
testable in isolation). Techniques such as HIT become important
since the developer of an application will subclass from classes
written by the framework developers. HIT assists in determining
what portion of the behavior inherited from the framework should be
retested.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 199855Too Many
Tests Consider the following system structure. How many test cases
do we need to cover all of the combinations?m(P)ABCXYZPQ RCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 199856OATS to the Rescue Orthogonal Array
Testing is a statistical technique that has been borrowed from
manufacturing. The purpose of OATS is to assist in the selection of
appropriate combinations of factors to provide maximum coverage
from a test case with a minimum number of cases. OATS selects test
cases so as to test the interactions between independent measures,
called factors. Each factor has a finite set of possible values,
called levels. Each column in the array corresponds to a factor.
Each row corresponds to a test case. The rows are created to
provide all possible pairwise combinations of possible levels for
the factors.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 199857Section
4Testing State-Based ClassesIn this section we will discuss:
Testing Strategies Case Study - Timer ClassesCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 199858State-Based Testing All systems have state, but
object-oriented systems decompose states into sufficiently small
state machines to be manageable.Global state decomposedinto
encapsulated states. Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-59Many Classes
Are State Machines Here is a Queue
classEmptyNotEmptyaddremove/itemaddremove/itemcreateremove/exceptionCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-60Adequacy Criteria for State-Based
Structural Testing All methods
exercised.Thisleveloftestingassuresthatnonecessarymethodsaremissing,butdoesnotguaranteethatallrequired
states are present. All states visited.This level of testing
assures that no states are missing, but does not guarantee that all
required methodsare present. All transitions exercised.This level
of testing would identify any missing states or methods. It would
also identify some extrastates, if they are present. N-way switch
cover.This level tests combinations of transitions. It will find
additional extra states and some corruption. All paths
followed.This level of testing would identify all missing and extra
states and methods. It is seldom practical andoften not
possible.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 199861Section
5Parallel Architecture for Component TestingIn this section we will
discuss: An Architecture for Managing Test CasesCopyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-62Architecture for Managing Test Cases Test Cases
as Data Store sequence of messages as a file Interpret the messages
and apply Test Cases as Methods Each test case is a method in a
test harness class Take advantage of inheritance and other
object-oriented
featuresClassTesterOfClassmethod1()method2()testMethod1()testMethod2()testAll()Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-63Automating Testing A simple main method
may be used or a more sophisticated one could be built:void main()
{TesterOfX foo;foo.testAll();}orvoid main() {TesterOfX
bar;bar.functionalSuite();bar.structuralSuite();bar.interactionSuite();}Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-64Parallel Architecture For Component
TestingTesterOfClass1TesterOfClass2TesterOfClass3Class1Class2
Class3GenericTest
HarnessTesterOfClass4TesterOfClass5TesterOfClass6Class4Class5Class6ProductionArchitecturePACTCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-65An Alternative
ApproachClassmethod1()testMethod1()method2()testMethod2()...testAll()
Advantages? DisadvantagesCopyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-66Generic Test Harness Class Methods in the Generic Test
Harness include: Test scripts that sequence test execution Logging
and reporting mechanisms Methods that catch exceptions Methods that
watch for memory leakage Methods that access test hardwareCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 199867Section 6System TestingIn this session we
will discuss: Strategies for System Testing Constructing System
Test Cases Measuring the Effectiveness of System TestingCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-68Object-Oriented System Testing System
testing is basically independent of the paradigm used to create the
system. Why? However, system testing of object-oriented systems is
different in the following ways: Because modeling is a vital part
of the object-oriented development process, system testing can
begin very early in development as those models are tested. The use
cases built as part of analysis are tested early in the development
process. These use cases then serve as the foundation for system
test cases. Testing of models and use cases involves a closer
interaction between clients and developers and testers.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-69Object-Oriented System Testing System
testing of object-oriented systems is different in the following
ways (continued): The creation of PACT structures serves as the
foundation for system and regression test suites. The use of
frameworks may mean that a significant portion of system testing
has already been done by the framework author.Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-70Role of the System Tester System testing can
begin much earlier in the development cycle. System testers should:
Develop the project test strategy during project planning phase.
Participate in all system-level tests including tests of
requirements models, design models, and the system implementation.
Develop system test cases from use cases and from additional system
requirements such as performance and capacity requirements. Audit
developers class and cluster test processes. Take the lead in
building regression tests.Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11, 1998-71Use
Cases and Test CasesSystemTest CasesDesign TestCasesUseCases
Object-oriented projects often employ a use case model to represent
the majority of system requirements. This model represents both the
users of the system and their requirements for the system.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-72Constructing Test CasesUse CaseScenario
Scenario Scenario ... ...Test Case ... Test Case ... Test Case Each
use case may produceseveral scenarios. Each scenario in turn may
produceseveral test cases.Requirement RequirementCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-73Percentage of Defects Found Testing
effectiveness is determined by the percentage of total defects
found at each step in the testing phase Our goal is to find a
larger percentage in the early stages and smaller percentages in
later phases. Effectiveness of the overall testing process is
defined as: Faults found during system testing divided by total
number of faults found by all testing activities (including the
testing done by customers that we call production).Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-74Iterative Measurement of Effectiveness
The effectiveness ratios should be calculated after the testing of
each iteration. The progression of the values from one iteration to
another provides trends that indicate the effectiveness of the
testing process. Expect more defects to be found in the early
testing phases and fewer in the later testing phases as the
increment matures. Do the number of errors always approach zero as
the component matures?Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
199875Section 7SummaryCopyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-76Higher Reliance onFunctional Testing Functional tests are
derived from the class/method specifications. Classes and methods
are specified using pre-conditions, post-conditions, and
invariants. At least one test should be created for each
post-condition statement. Use equivalence classes to minimize the
number of test cases created. Select test cases to verify the
correct operation at the boundary conditions.Copyright 1993-1998
Timothy Korson. All rights reserved.Borland Conference, Denver,
August 11, 1998-77Subclass Testing Follow strict inheritance
Pre-conditions same or less Post-conditions same or more Invariants
same or more Hierarchical Incremental TestingSuperclassNot
DefinedVirtual Virtual Abstract AbstractSubclassNew CodeNo New
CodeNew CodeNo New CodeNew CodeFunctionalTestsWrite? Yes No No No
NoExecute? Yes No Yes No YesStructuralTestsWrite? Yes No Yes No
YesExecute? Yes No Yes No YesInteractionTestsWrite? Yes No Yes No
YesExecute? Yes Yes Yes No YesCopyright 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
1998-78Parallel Test Harness
DevelopmentTesterOfClass1TesterOfClass2TesterOfClass3Class1Class2
Class3GenericTest
HarnessTesterOfClass4TesterOfClass5TesterOfClass6Class4Class5Class6ProductionArchitecturePACTCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-79Interaction Testing Use OATS to select a
random but equally distributed selection of test cases.L4 (23)1 2
31 1 1 12 1 2 23 2 1 24 2 2 1L8 (27)1 2 3 4 5 6 71 1 1 1 1 1 1 12 1
1 1 2 2 2 23 1 2 2 1 1 2 24 1 2 2 2 2 1 15 2 1 2 1 2 1 26 2 1 2 2 1
2 17 2 2 1 1 2 2 18 2 2 1 2 1 1 2L9 (34) 1 2 3 41 1 1 1 12 1 2 2 23
1 3 3 34 2 1 2 35 2 2 3 16 2 3 1 27 3 1 3 28 3 2 1 39 3 3 2
1Copyright 1993-1998 Timothy Korson. All rights reserved.Borland
Conference, Denver, August 11, 1998-80System Testing System testing
is less dependent on the paradigm used in creating the system than
component testing. The difference in system testing of
object-oriented systems is the availability and applicability of
use cases. Structured use cases become the basis of structured test
cases. System testers should be involved from the beginning in:
Creating and executing system tests Auditing the developers ongoing
testing Emphasizes the importance of regression testing. Measure
the effectiveness of system testing as input to the process
improvement process.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-81Organizing
for Testing Must have a complete testing architecture
defined.TestGroupDeveloper-ledTestingTestGroupTimeFocusCopyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-82Thanks On behalf of Software Architects,
thanks for attending this tutorial. Remember us for future
object-oriented training or consulting. Let us know about your
testing experience.Wed like to hear about your successes and your
difficulties. My e-mail address is:
[email protected] 1993-1998 Timothy Korson.
All rights reserved.Borland Conference, Denver, August 11,
199883Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11,
199884BibliographyCopyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11,
1998-85Bibliography[1]SoftwareQualityManagementII.BuildingQualityintoSoftware.Comp.Mech.Publications,Southampton,
UK, 1994.[2] Marick, Brian. Testing Object-oriented Software. ACM,
New York, NY, USA,
1995.[3]R.D.Ammann,M.H.;Cameron.Measuringprogramstructurewithinter-modulemetrics.IEEEComput.
Soc. Press, Los Alamitos, CA, USA, 1994.[4] Thomas R. Arnold and
William A. Fuson.Testing in a perfect world. CACM, 37(9):78-86,
Sep-tember 1994.[5] R.J.; Wright C.; Zweig-D. Banker, R.D.;
Kauffman. Automating output size and reuse metrics in
arepository-basedcomputer-aidedsoftwareengineering(case)environment.IEEETransactionsonSoftware
Engineering, 20(3):169-87, March
1994.[6]A.Barbey,S.;Strohmeier.Theproblematicsoftestingobject-orientedsoftware.Comp.Mech.Publications,
Southampton, UK,
1994.[7]B.Beizer.SoftwareTestingTechniques.VanNostrandReinholdCompany,Inc.,NewYork,1990.[8]
Robert V. Binder. Design for testability in object-oriented
systems. CACM, 37(9):87-101, Septem-ber
1994.[9]J.ProwseBrownlie,RandM.S.Phadke.RobusttestingofAT&Tpmx/starmailusingoats.AT&T
Technical Journal, pages 41-47, May-June 1992.[10]D.M. Chao, B.P.;
Smith. Applying software testing practices to an object-oriented
software devel-opment. OOPS Messenger, 5(2):49-52, April
1994.[11]Thomas J. Cheatham and Lee Mellinger.Testing
object-oriented software systems. In 1990 ACMEighteenth Annual
Computer Science Conference Proceedings, pages 161-165,
1990.[12][13]In Sang Chung. Methods of comparing test criteria for
object-oriented programs based on
sub-sumption,powerrelationandtestreusability.JournaloftheKoreaInformationScienceSociety,22(5):693-704,
May
1995.[14]M.-C.Chung,C.-M.;Lee.Object-orientedprogrammingtestingmethodology.InternationalJournal
of Mini and Microcomputers,
16(2):73-81.[15]Z.E.Cusack.Inheritanceinobjectorientedz.InEuropeanConferenceonObject-OrientedPro-gramming,
pages 167-179, 1991.[16]S.R. Davis.Armor cladding c++ classes.C++
Report vol.6, no.8, p.36-9, 41,Oct. 1994.[17]P. K. Devanbu, D. S.
Rosenblum, and A. L. Wolf. Automated construction of testing and
analysistools.In Proceedings of 16th International Conference on
Software Engineering, pages 241-250,
LosAlamitos,CA,1994.IEEEComput.Soc.Tech.CommitteeonSoftwareEng.,IEEEComput.Soc.Press.Copyright
1993-1998 Timothy Korson. All rights reserved.Borland Conference,
Denver, August 11, 1998-86[18]R.K. Doong and Phyllis G. Frankl.
Case studies on testing object-oriented programs. In Proceed-ings
of the Fourth Symposium on Testing, Analysis and
Verification(TAV4), pages 165-177, 1991.[19]R. J. DSouza and R. J.
Leblanc Jr.Class testing by examining pointers. Journal of
Object-Orient-ed Programming, 7(4):33-39, July-Aug 1994.[20]D. Duke
and Roger Duke.Towards a semantics for object-z.In VDM and Z-Formal
Methods inSoftware Development, pages 244-259,
1990.[21]DawnStockbridge Dunaway andEugenia Gillan. Applying
object-orienteddesign principles todeveloping a test strategy. In
11th International Conference and Exposition on Testing Computer
Soft-ware, pages 341-368. ASQC and STSC, June 13-16
1994.[22]M.FelderandP.SanPietro.Testingbyexecutinglogicspecification.InA.D.Halang,W.A.;Stoyenko,
editor, Proceedings of the NATOAdvanced Study Institute on Real
Time Computing, pag-es 683-684, Berlin, Germany, 1994.
Springer-Verlag.[23]S.P. Fielder. Object-oriented unit testing.
Hewlett-Packard Journal, April 1989.[24]S. Frost. Modeling for the
rdbms legacy. Object Magazine, 4(5):43-4, 46, 48-51, Sept.
1994.[25]T.J. Gattis, G.F.; Cheatham. Testing object-oriented
software. ACM, New York, NY, USA,
1995.[26]MaryJeanHarroldandJohnD.McGregor.Incrementaltestingofobject-orientedclassstruc-tures.InProceedingsoftheInternationalConferenceonSoftwareEngineering,pages68-80,May1992.[27]P.
A. Hausler, R. C. Linger, and C. J. Trammell. Adopting cleanroom
software engineering with aphased approach. IBM Systems Journal,
33(1):89-109, 1994.[28]J.H. Hayes. Testing of object-oriented
programming systems (oops), 1994.[29]Ivar Jacobson, Magnus
Christerson, Patrik Jonsson, and Gunnar Overgaard. Object-Oriented
Soft-ware Engineering. Addison-Wesley,
1991.[30]PaulC.JorgensenandCarlErickson.Object-orientedintegrationtesting.CACM,37(9):30-38,September
1994.[31]P. Juettner, P. Zimmerer, U. Naumann, and S. Kolb.A
complete test process in object-orientedsoftware development. In
The 7th International Software Quality Week in San Francisco, pages
3-A-2.Software Research, Inc., May 17-20 1994.[32]P. Juttner, S.
Kolb, S. Sieber, and P. Zimmerer. Testing major object-oriented
software systems.Siemens Review (Special Issue), pages25-29, Spring
1994.[33]S. Kanjulal, S. T. Chakradhar, and V. D. Agrawal. A test
function architecture for interconnectedfinite state machines. In
Proceedings of 7th International Conference on VLSI Design, pages
113-116,Los Alamitos, CA, 1994. VLSI Soc. India, IEEE Comput. Soc.
Press.[34]Mark Lorenz.Object-Oriented Software Development:a
practical guide. PTR Prentice Hall, En-glewood Cliffs, N.J.,
1993.[35]T. J. McCabe and A. H. Watson.Combining comprehension and
testing in object-oriented devel-opment. Object Magazine,
4(1):63-64,March-April 1994.Copyright 1993-1998 Timothy Korson. All
rights reserved.Borland Conference, Denver, August 11,
1998-87[36]R.L. McGarvey. Object-oriented test development in
ABBET. IEEE,New York, NY, USA, 1994.[37]J.D. McGregor and D. Dyer.A
note on inheritance and state machines. Technical report,
ClemsonUniversity, 1993.[38]John McGregor. Constructing functional
test cases using incrementally derived state machines.In11th
International Conference and Exposition on Testing Computer
Software, pages 377-386. ASQCand STSC, June 13-16 1994.[39]John D.
McGregor. Selecting functional test cases for a class. In The7th
International SoftwareQuality Week in San Francisco, pages
2-T-2.Software Research, Inc., May 17-20 1994.[40]John D. McGregor
and Douglas M. Dyer.Selecting specification- based test cases for
object-ori-ented systems.Technical Report, Clemson University,
1993.[41]John D. McGregor and Tim Korson. Understanding
object-oriented: A unifying paradigm.Com-munications of the ACM,
33(9):40-60, September 1990.[42]John D. McGregor and Timothy D.
Korson.Integrating object-oriented testing and
developmentprocesses.CACM, 37(9):59-77, September 1994.[43]M. D.
Smith and D.J. Robson. A framework for testing object-oriented
programs. Journal of Ob-ject-Oriented Programming, 5(3):45-53, June
1992.[44]Gail C. Murphy, Paul Townsend, and Pok Sze
Wong.Experiences with cluster and class testing.CACM, 37(9):39-47,
September 1994.[45]Udo Naumann.Experiences in testing
object-oriented software.In 11th International Conferenceand
Exposition on Testing Computer Software, pages 373-376. ASQC and
STSC, June 13-16 1994.[46]A. Parrish and D. Cordes. Applying
conventional unit testing techniques to abstract data type.
In-ternational Journal of Software Engineering and Knowledge,
4(1):103-122, March 1994.[47]A. Parrish, D. Cordes, and M.
Govindarajan. Removal from object- oriented modules.In The
7thInternational Software Quality Week in San Francisco, pages
2-T-1. Software Research, Inc., May 17-20 1994.[48]D.E. Perry and
G.E. Kaiser.Adequate testing and object-oriented
programming.Journal of Ob-ject-Oriented Programming,
2:13-19,Jan/Feb 1990.[49]M.; Staples-G. Petruv, V.; Ross. Testing
for tomorrows professionals, 1994.[50]Robert M. Poston. Automated
testing from object models. CACM, 37(9):48-58, September
1994.[51]Marc Rettig. Testing made palatable. Communications of the
ACM,34(5):25-29, May
1991.[52]P.G.Roong-KoDoong;Frankl.Theastootapproachtotestingobject-orientedprograms.ACMTransactions
on Software Engineering and Methodology, 3(2):101-30, April
1994.[53]M.J.HarroldandRothermel,G.;Selectingregressiontestsforobject-orientedsoftware.IEEEComput.
Soc. Press, Los Alamitos, CA, USA, 1994.[54]S.J. Samadzadeh, M.H.;
Khan.Stability, coupling, and cohesion ofobject-oriented software
sys-tems. ACM, New York, NY, USA, 1994.[55]A. Serra and P.
Nesi.Object-oriented approach for a non-invasive, automatic testing
tool.In The7th International Software Quality Week in San
Francisco, pages 4-A-3. Software Research, Inc., May17-20
1994.Copyright 1993-1998 Timothy Korson. All rights
reserved.Borland Conference, Denver, August 11, 1998-88[56]S.
Siegel. Automating integration testing for oo projects. In The 7th
International Software QualityWeek in San Francisco, pages 4-A-4.
Software Research, Inc., May 17-20 1994. Paper not included inthe
conference proceedings.[57]A.R. Siepmann, E.; Newton.Tobac:a test
case browser for testing object-oriented software. SIG-SOFT
Software Engineering Notes special issue, pages 154-68,
1994.[58]M.D. Smith and D.J. Robson.Object-oriented programming -
theproblems of validation. In Pro-ceedings of the Conference on
Software Maintenance, pages 272-281, 1990.[59]Bob Stahl. How to
test object-oriented software. In 11th International Conference and
Expositionon Testing Computer Software, pages 369-372. ASQC and
STSC, June 13-16 1994.[60]S. Subramanian, Wei-Tek Tsai, and S. H.
Kirani. Hierarchical dataflow analysis for o-o programs.Journal of
Object-Oriented Programming, 7(2):36-46, May 1994.[61]C.D. Turner
and D.J. Robson.The testing of object-oriented programs. Technical
report, Univer-sity of Durham, England, 1992.[62]J.; Mraz R. Von
Mayrhauser, A.; Walls.Sleuth : a domain basedtesting tool. In Int.
Test Confer-ence, Altoona, PA, USA, 1994.[63]Chi-Ming Chung;
Ming-Chi Lee; Ching-Chian Wang.Inheritance testing for
object-oriented pro-gramming by transitive, closure strategies.
Advances in Modelling & Analysis B, 31(2), 1994.[64]K.;
Srivastava A.; Krueger-J. Weber, R.; Thelen. Automated validation
test generation.
1994.[65]StuartH.Zweben.Testingformallyspecifieddata-orientedmodulesusingprogram-basedtestdata
adequacy criteria.Technical report, Ohio State University,
1989.