04/22/22 04/22/22 1 Testing: A Roadmap Testing: A Roadmap Mary Jean Harrold Mary Jean Harrold Sunita Bodigutla Sunita Bodigutla CS 589 Fall 2003 CS 589 Fall 2003
04/10/2304/10/23 11
Testing: A RoadmapTesting: A Roadmap
Mary Jean HarroldMary Jean Harrold
Sunita BodigutlaSunita Bodigutla
CS 589 Fall 2003CS 589 Fall 2003
04/10/2304/10/23 22
DefinitionDefinition
A verification A verification method that applies method that applies a controlled set of a controlled set of conditions and conditions and stimuli for the stimuli for the purpose of finding purpose of finding errors.errors.
The paper says:The paper says: It’s a process that is It’s a process that is
performed to support performed to support quality assurancequality assurance
04/10/2304/10/23 33
Quality AssuranceQuality Assurance
ISO 9000:2000 ISO 9000:2000 "Part of quality management focused on "Part of quality management focused on
providing confidence that quality requirements providing confidence that quality requirements will be fulfilled“will be fulfilled“
It is a planned program consisting of: all actions It is a planned program consisting of: all actions necessary to provide adequate confidence that necessary to provide adequate confidence that an item or product an item or product conformsconforms to established to established technical requirements; and, activities designed technical requirements; and, activities designed to evaluate the to evaluate the processprocess by which products are by which products are developed or manufactured.developed or manufactured.
A product does what it is expected to.A product does what it is expected to.
04/10/2304/10/23 44
TestingTesting
Designing test casesDesigning test cases executing the software with those executing the software with those
test casestest cases examining the results produced by examining the results produced by
those executionsthose executions
04/10/2304/10/23 55
ExpensiveExpensive
Testing accounts Testing accounts for for 50% 50% of the cost of the cost of software of software developmentdevelopment
Higher for critical Higher for critical systemssystems
Efficient ways to Efficient ways to perform testing to perform testing to control the costcontrol the cost
04/10/2304/10/23 66
What this paper is aboutWhat this paper is about
A report written in 2000A report written in 2000 assesses the state of the art in assesses the state of the art in
software testingsoftware testing outlines future directions in software outlines future directions in software
testingtesting gives pointers to software testing gives pointers to software testing
resourcesresources
04/10/2304/10/23 77
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 1. Software Testing Roadmap
04/10/2304/10/23 88
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 2
04/10/2304/10/23 99
Component Based SystemsComponent Based Systems
Increased size and Increased size and complexity of complexity of software systems software systems has led to has led to component based component based systemssystems
Component Component Developer vs. Developer vs. Component userComponent user
04/10/2304/10/23 1010
Component UserComponent User
Component
ReusableService
Semantics
Repository
System
System Under Assemble
Select
Composible
Executable
Understand
Assembly
Compile
fig 3
04/10/2304/10/23 1111
Component Based Systems Component Based Systems (contd.)(contd.)
COTS COTS : Commercial off the shelf software: Commercial off the shelf software e.g. Microsoft Active X, MFC, Java Applets e.g. Microsoft Active X, MFC, Java Applets
etcetc Why COTS-Why COTS-
saves moneysaves money saves timesaves time component reusecomponent reuse standardizationstandardization simplified designsimplified design
04/10/2304/10/23 1212
Problems with Testing COTSProblems with Testing COTS
Unavailability of source codeUnavailability of source code Technical HeterogeneityTechnical Heterogeneity Enterprise HeterogeneityEnterprise Heterogeneity Dynamic evolutionDynamic evolution Component Trust ProblemComponent Trust Problem
04/10/2304/10/23 1313
Component Based SoftwareComponent Based Software
Techniques for testing by Component Techniques for testing by Component provider- a number of themprovider- a number of them
Techniques available to component Techniques available to component user-very fewuser-very few
Theory of Theory of test adequacytest adequacy by by RosenblumRosenblum
04/10/2304/10/23 1414
Test AdequacyTest Adequacy
A test adequacy criterion is a systematic criterion that is used to determine whether a test suite provides an adequate amount of testing for a component under test
C-adequate-for-P for adequate unit testing of a component
C-adequate-on-M for adequate integration testing of a component-based system
fig 4
04/10/2304/10/23 1515
Future DirectionsFuture Directions
Efficient techniques and tools forEfficient techniques and tools for testing aspects like security, dependability and safetytesting aspects like security, dependability and safety representing and computing the types of testing representing and computing the types of testing
information (meta-data) that the component-user needsinformation (meta-data) that the component-user needs Java Beans: BeanInfoJava Beans: BeanInfo
storing the informationstoring the information using the informationusing the information A Framework for component deployment [1] called A Framework for component deployment [1] called
‘component deployment testing (CDT) framework’‘component deployment testing (CDT) framework’ component runtime introspectioncomponent runtime introspection reflection API of Javareflection API of Java
04/10/2304/10/23 1616
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 5
04/10/2304/10/23 1717
Integration test strategies and architectural Integration test strategies and architectural stylesstyles
fig 6
04/10/2304/10/23 1818
Fault type and Integration Test StrategyFault type and Integration Test Strategy
fig 7
04/10/2304/10/23 1919
Testing based on Architecture Testing based on Architecture specificationsspecifications
Formal notation for architecture specification Formal notation for architecture specification ‘‘CHAMCHAM’ : Chemical Abstract Machine.’ : Chemical Abstract Machine. Architecture: a set of components (molecules)Architecture: a set of components (molecules) states and interactions governed by rules states and interactions governed by rules
(reactions)(reactions) CHAM is specified by defining molecules m1,m2,…, CHAM is specified by defining molecules m1,m2,…,
solutions s1, s2,….. and transformation rules T, T’,solutions s1, s2,….. and transformation rules T, T’,…..…..
Solutions are multisets of molecules interpreted as Solutions are multisets of molecules interpreted as defining the states of CHAMdefining the states of CHAM
Transformation rules describe how the system Transformation rules describe how the system dynamically evolves through reaction stepsdynamically evolves through reaction steps
04/10/2304/10/23 2020
Testing based on Architecture specifications Testing based on Architecture specifications (contd.)(contd.)
[4] Using the formal description of SA, a transition [4] Using the formal description of SA, a transition graph depicting all possible behaviors of the graph depicting all possible behaviors of the system is arrived at.system is arrived at.
A suitable set of reduced graphs are identified, A suitable set of reduced graphs are identified, each subgraph specific architectural properties of each subgraph specific architectural properties of the system.the system.
For each subgraph, test cases are derived to For each subgraph, test cases are derived to cover all the transitions in it.cover all the transitions in it.
04/10/2304/10/23 2121
Future DirectionsFuture Directions
Techniques for Techniques for automatic generation of test cases from automatic generation of test cases from
formal specification of software formal specification of software architecturearchitecture
evaluating software architectures for evaluating software architectures for testabilitytestability
04/10/2304/10/23 2222
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 8
04/10/2304/10/23 2323
Testing Evolving SoftwareTesting Evolving Software
Regression Testing: Regression Testing: ensuring that no ensuring that no new errors are introduced into new errors are introduced into previously tested codepreviously tested code
ExpensiveExpensive Selective Regression Testing: which Selective Regression Testing: which
subset of a test suite must be rerun subset of a test suite must be rerun to test a new version of the systemto test a new version of the system
04/10/2304/10/23 2424
Future DirectionsFuture Directions
Techniques toTechniques to assist in managing the test suitesassist in managing the test suites reduce the size of test suitesreduce the size of test suites prioritize test cases prioritize test cases
04/10/2304/10/23 2525
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 9
04/10/2304/10/23 2626
Test CriteriaTest Criteria
Effective for certain classes of faultsEffective for certain classes of faults data flowdata flow complex boolean expressions: Modified complex boolean expressions: Modified
Condition/decision coverageCondition/decision coverage
04/10/2304/10/23 2727
Modified Condition/Decision CoverageModified Condition/Decision Coverage
Definition:Definition: Every point of entry and exit in the Every point of entry and exit in the
program has been invoked atleast onceprogram has been invoked atleast once every condition in a decision in the every condition in a decision in the
program has taken on all possible program has taken on all possible outcomes atleast onceoutcomes atleast once
each condition has been shown to each condition has been shown to independently affect the decisions independently affect the decisions outcomeoutcome
04/10/2304/10/23 2828
Modified Condition/Decision Modified Condition/Decision Coverage (contd.)Coverage (contd.)
Simple exampleSimple example AA and and BB Extension of truth Extension of truth
table to pairs tabletable to pairs table The test set { (T T),The test set { (T T),
(T F), (F T)} is (T F), (F T)} is adequate for MC/DC adequate for MC/DC coveragecoverage
NumbNumberer ABAB ResulResul
ttAA BB
11 T TT T TT 33 22
22 T FT F FF 11
33 F TF T FF 11
44 F FF F FF
fig 10
04/10/2304/10/23 2929
Future directionsFuture directions
Find ways Find ways to determine interaction between the to determine interaction between the
various test- selection criteria and find various test- selection criteria and find ways to combine them to perform more ways to combine them to perform more effective testingeffective testing
04/10/2304/10/23 3030
Testing Componentbased Systems
Testing based on Precode artifacts
Demonstrating effectiveness
of testing techniques
Testing Evolving Software
creating effective testing processes
Other testing approaches
Using testing artifacts
Methods and Tools
Empirical Studies
Practical Testing
Methods, Tools, and Processes
forDevelopment of
High-QualitySoftware
Fundamental Research
Status of Testing
Methods,Tools,and
Processes
fig 11
04/10/2304/10/23 3131
Effective Processes for Effective Processes for TestingTesting
Process used for planning and implementing Process used for planning and implementing itit
Test plan Test plan Perpetual testingPerpetual testing Microsoft uses a different modelMicrosoft uses a different model
synchronize and stabilizesynchronize and stabilize Future directionsFuture directions
validate the existing modelsvalidate the existing models integration of various quality techniques and toolsintegration of various quality techniques and tools
04/10/2304/10/23 3232
Using Testing ArtifactsUsing Testing Artifacts
Execution TracesExecution Traces Information about whichInformation about which
statements were executedstatements were executed paths were executedpaths were executed values particular variables acquiredvalues particular variables acquired
Artifacts can be stored for use in retesting the Artifacts can be stored for use in retesting the software after it is modifiedsoftware after it is modified dynamic program slicesdynamic program slices heuristic approachesheuristic approaches List of invariantsList of invariants Regression TestingRegression Testing Visualization techniqueVisualization technique
04/10/2304/10/23 3333
Other topicsOther topics
Methods and ToolsMethods and Tools ScalabilityScalability Computational tradeoffsComputational tradeoffs Automatic generationAutomatic generation
Empirical StudiesEmpirical Studies Testing ResourcesTesting Resources
04/10/2304/10/23 3434
ConclusionConclusion
OverviewOverview List of techniquesList of techniques No comparison No comparison Future directionsFuture directions Testing is a wide areaTesting is a wide area Dynamic fieldDynamic field
04/10/2304/10/23 3535
[1] Antonia Bertolino, Andrea Polini, “A Framework for Component Deployment [1] Antonia Bertolino, Andrea Polini, “A Framework for Component Deployment Testing”Testing”
[2] A.Orso, M.J.Harrold and D.Rosenblum, “Component Metadata for Software [2] A.Orso, M.J.Harrold and D.Rosenblum, “Component Metadata for Software Engineering Tasks”, in W.Emmerich and S.Tai(Eds) EDO 2000, LNCS 1999, pp. 129-Engineering Tasks”, in W.Emmerich and S.Tai(Eds) EDO 2000, LNCS 1999, pp. 129-144144
[3] J.A.Stafford and A.L.Wolf, “Annotating components to support component based [3] J.A.Stafford and A.L.Wolf, “Annotating components to support component based static analysis of software systems “, static analysis of software systems “, Proc. the Grace Hopper Celeb. of women in Proc. the Grace Hopper Celeb. of women in computing 2001computing 2001
[4] Antonia Bertolino, Paola Inverardi, Henry Muccini, Andrea Rosetti, “An Approach [4] Antonia Bertolino, Paola Inverardi, Henry Muccini, Andrea Rosetti, “An Approach to Integration Testing based on Architectural Descriptions”to Integration Testing based on Architectural Descriptions”
References
04/10/2304/10/23 3636
fig 12. Sequential Multiphase Compiler Architecture
04/10/2304/10/23 3737
fig 13.Transition Graph for the Sequential Compile
04/10/2304/10/23 3838
fig 14. Concurrent Reduced Graph
04/10/2304/10/23 3939