Chapter 13: Regression Testing Omar Meqdadi SE 3860 Lecture 13 Department of Computer Science and Software Engineering University of Wisconsin-Platteville.
Post on 03-Jan-2016
235 Views
Preview:
Transcript
Chapter 13: Regression Testing
Omar Meqdadi
SE 3860 Lecture 13Department of Computer Science and Software Engineering
University of Wisconsin-Platteville
2
Topic Covered
What is Regression Testing? Why Regression testing? Regression Testing Process Object-Oriented Regression Testing
3
What is Regression Testing?
Testing activities occur after software changes During software maintenance phase
Execution of a set of test cases on a program in order to ensure that implemented changes do not produce unintended faults
Regression Testing Levels
Unit Testing Integration Testing Acceptance Testing
Who Perform Regression Testing
Developers: regression testing at the unit level or integration
Test Engineers : regression testing at the function level
Quality Engineers: regression testing at the system level
Why Regression Testing?
Validate and Verify the modified software Validate and verify the new features Ensure existing features are still working(change impact)
Small changes in one part of a program may have indirect undesired effects in other unrelated parts of the program
Fixing could introduce new bugs
Why Regression Testing?
8
Regression Testing Process
Input: Given program P, its modified version P’, and a test set T
that was used to previously test P Goal:
Find a way to utilize T to gain sufficient confidence in the correctness of P’
Regression Testing Process
Activities: Regression Test Selection
Select T’ ⊆ T, a set of test cases to execute on P’ Test Suite Execution
Test P’ with T’, establishing correctness of P’
Regression Test Selection
Selecting a subset of the existing test suite based on information about: Original Version (P) Modified Version (P’) Original Test Suite (T)
Approaches: Select All Modification-Based Test Selection Coverage-Based Test Selection
Test Set Minimization Test Case Prioritization
Select All
Traditional approach Select all the test cases in T to test P’ Advantage:
Safe Problem:
Too expensive Note: the new and the old programs could produce
different outputs
Modification-Based Test Selection
Select set of test cases T that covers only modified or affected portions of P
Based on Module-Based Firewall A changed software module and a closure of all possible affected
modules and related integration links in a program based on the static and dynamic analyses (e.g.: a control-flow graph).
We can come out different kinds of firewalls based on various test models such as: Control Firewall Data Firewall Function (method) Calling Firewall Class Firewall (Object-Oriented Regression Testing)
Modification-Based Test Selection
Test selection concept based om Module Firewall re-test of the changed module and its affected modules
within the firewall re-test all of related integration links in the firewall
Main
M1 M3M2
M7M6M5M4
M8
Modification-Based Test Selection
: Control Flow
Mi : Module
Main
M1 M3M2
M7M6M5M4
M8
Changed Module
Modification-Based Test Selection
Main
M1 M3M2
M7M6M5M4
M8
Changed Module
Modification-Based Test Selection
A control based module firewall: M5, M1, M8 , and Main
Main
M1 M3M2
M7M6M5M4
M8
1
2
3
Changed Module
Modification-Based Test Selection
A control based module firewall: M5, M1, M8 , and MainRe-testing at the unit level: M5
Re-testing at the integration level: 1, 2, 3
Modification-Based Test Selection
Problems: Still too many test cases
Need to further reduce the number of regression tests Coverage: the risk of missing test cases that are able to
reveal defect not yet discovered
Test Set Minimization
The test suite is reduced to a minimal subset that maintains the same coverage as the original test suite with respect to a given coverage criterion
Goal: Find a minimal subset T’of T such that each testable entity (function, statement,…) is covered by at least one test in T’
What we need ? Coverage (Execution Flow Trace) per test case
Test Set Minimization
Steps: Step1: From old version P : determine the coverage for
each test t ⊆ T, determine which of the k testable entities is covered.
Step2: Find a minimal subset T’ of T such that each testable entity (function, statement,…) is covered by at least one test in T’by using Test Case Prioritization
Note: T’is not unique
Test Case Prioritization
Tests are prioritized based on some criteria. Highest priority test are executed earlier Test Prioritization Approaches:
Total statement coverage # of statements executed
Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher
number of not yet executed statements Total function coverage
# of functions executed Additional function coverage
Test Set Minimization
Example1: Test prioritization based on total function coverage Suppose
P contains two functions: main and foo suppose that P is tested using test cases t1 and t2 During testing it was observed that :
t1 causes the execution of main but not of foo t2 causes the execution of both main and foo
Now suppose that P’ is obtained from P by making some modification
Which of the two test cases should be included in the regression test suite?
regression test suite consists of only t2 (minimization)
Test Set Minimization
Example2: Test prioritization based on total statement coverage Assume P contains two functions main and f1, where the
flow charts are as follow
Test Set Minimization
Example2: Assume the following test cases
Test case Coverage
T1 main: {1, 2, 3} f1: {1, 3}T2 main: {1, 3} f1: {1, 3}T3 main: {1, 3} f1: {1, 2, 3}
A minimal test set for regression testing is {t1, t3}
Test Case Prioritization
Additional statement coverage((Heuristic #1) A higher priority is assigned to a test that covers the higher
number of not yet executed statements
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3
t2 : S1, S5, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t8 : S1, S2, S3, S4, S7
t9 : S1, S6
t10 : S1, S2
S: t8
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements t2 : S1, S5, S8, S9 S1, S2, S3, S4, S7
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements
S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2
Additional Statement Coverage
Executed statements for each test t1: S1, S2, S3 Covered statements
S1, S2, S3, S4, S5, S7, S8, S9
t3 : S1, S5, S7
t4 : S1, S5, S3, S4
t5 : S1, S2, S7
t6 : S1, S2
t7 : S1, S2, S4
t9 : S1, S6
t10 : S1, S2
S: t8, t2 , t9
Test Set Minimization/ Prioritization
Problem:
Test minimization discards test cases.
There is a chance that, if P’ were executed against a
discarded test case, it would reveal an error in the
modification made
Object-Oriented Regression Testing
Typical Cases Inheritance:
Getting a new subclass working without exercising the superclass is usually easy
No need to re-test the superclass We assume that inherited features just work
Component based development (Re-use) OO technologies offer strong support for reuse but they
cannot prevent regression bugs No unit-testing We need integration testing
Object-Oriented Regression Testing
Dependency: Dependencies occur for many reasons and so regression
faults can occur in many ways Assume a dependency is between B and D
D has a side effect on B: B fails because a new action of D is inconsistent with B’s requirements and assumptions with respect to D
D is a client of B: D sends a message that violates B’s invariant/precondition. B is not defensive and accepts the incoming message, leading to failure
Object-Oriented Regression Testing
Dependency (contd.): D is a server of B: B sends a message to D. D has been
changed (or contain bugs). D returns a value that causes a violation of B’s invariant . B fails or returns an invalid value to another baseline component C. C fails
Object-Oriented Regression Testing
OO- Regression Testing Concepts Each class must be tested when it is developed Each class needs a regression test Regression tests need to have standard interfaces Thus, we can build the regression test when building the
class and have a better, more stable product for less work
Object-Oriented Regression Testing
OO- Regression Testing Strategy Based on the Class Firewall Concept
Identify changed classes Identify affected classes (using dependency relation) Apply the Class Test Order strategy to perform class re-
testing at the unit level Use the Class Test Order to re-integrate classes together. Select, reuse, and define test cases
Example: A Changed Class in Class Diagram
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
Example: A Class Firewall in Class Diagram
X
Y
D
T
S
I
U
A C
B
M
N
R V
L
1
1.1
1.2
1.3
1.42.1
2
2.2
3 3.1
3.2
Example: Class Test Order for The Class Firewall
4.1
4
top related