Multi- Objective Test Suite Optimization for Incremental Product Family Testing ( joint work with Hauke Baller, Sascha Lity and Ina Schaefer). 03.03.2014, Copenhagen. Contents. „ What is a good ordering for testing the members of a product family ?“ - PowerPoint PPT Presentation
Welcome message from author
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Minimum Partial Weighted Set Cover Problem [Könemann et al., 2011]
Given:
Set of Weighted Requirements R
Set of Weighted Test Cases T
Profit goal kp
Find:
Sub set with minimal such that
pTS ≥ kp
where and
TSt
TTS twc
ØTSTRr
iRTS
i
i
rwp
ES – Real-Time Systems Lab9 | 03.03.2014 | PLE
Product Line Analysis
ES – Real-Time Systems Lab10 | 03.03.2014 | PLE
Sample-based Product Line Analysis
[Cohen et al. 2006], [Gustafsson 2007], [Oster et al., 2010], [Perrouin et al, 2010], [Kim et al. 2011], [Johansen et al., 2012], [Henard et al., 2013], [Haslinger et al., 2013],
….
ES – Real-Time Systems Lab11 | 03.03.2014 | PLE
Family-based Product Line Analysis
[Krishnamurti et al., 2006], [Gruler et al., 2010], [Classen et al. 2010], [Gnesi et al. 2011], [Cordy et al., 2013], …
𝑂 (?∗𝜑 )
ES – Real-Time Systems Lab12 | 03.03.2014 | PLE
Incremental Product Line Analysis
∅
[Batory et al. 2007], [Engström et al. 2011], [Lochau et al. 2012], …
ES – Real-Time Systems Lab13 | 03.03.2014 | PLE
T={t1, …, t15}
t7=(r1r6r8r7)
t8=(r1r4r5r2)
t11=(r1r4r7)
t12=(r9r10r8)
t13=(r3r11)
t14=(r1r4r11)
t15=(r3r7)}
Family-based Product Line Test Suite Design
4 Product Variants
r1(2)
r2(1)
r3(2)
r4(1) r5(2)
r6(1)
r7(2)r8(4)
r9(2)
r10(2)
r11(8)
ES – Real-Time Systems Lab14 | 03.03.2014 | PLE
Complete Minimal Product Line Test Suite [Cichos et al. 2011]
Assumptionsif then satisfies on
Given:
Set of Requirements R
Set of Test Cases T
Set of Product Variants P
Sub sets satisfying Requirement
Sub sets executable on product variant
Sub sets relevant for product variant
Find:
Minimum sub set such that
ES – Real-Time Systems Lab15 | 03.03.2014 | PLE
Constraint Minimum Set Cover Problem
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4 … Test Cases
Test Requirements
p1
Product Variants
p2 p3
5 2 3Product Costs/Profits ppTS
2 1 4 3 1 2
4 2 4 1 Testing Costs
Testing Profits
+ + = ≥ Profit Goal kpp
pTS ≥ kp
rpTS ≥ krp
ES – Real-Time Systems Lab16 | 03.03.2014 | PLE
Constraint (Minimum) Partial Weighted Set Cover Problem
Given:
Set of Weighted Requirements
Set of Weighted Test Cases
Set of Weighted Product Variants
Profit Goals
(Cost Bound
Find:
Sub set (with minimal such that
and and is an Adequat Product Line Test Suite
Ø:, TSTTRrPpRr
iRTS
ijj
ij
i
rwrp
Ø: TSTTRrPp
jpTS
ijj
i
j
pwpp
ES – Real-Time Systems Lab17 | 03.03.2014 | PLE
Integer Linear Programming (ILP)
(Integer) Decision variables Linear objective functionLinear constraints
ES – Real-Time Systems Lab18 | 03.03.2014 | PLE
ILP for Product Line Test Suite Optimization
Product pj selected xj=1
Test Case t selected into TS yt=1
Requirement ri satisfied zi=1
Binary Variables
ppTS
Product selected Requirements must be satisfied
pj select Test Case from Test Case Set
NOT pj OR One Test Case from Test Case Set
tp
For all Requirements
(of a Product)
Requirement satisfied For all Products, a Test Case must be selected
Again (ri Set) (NOT ri OR Set)
r t
Test Cases must be applied to a Product
t pPriotization
ppTS ≥ kpp ppTS - Δpp = kpp
ppTS ≥ kpp
cTS ≤ kc
rpTS ≥ krp
Differences must not be negativ
Normalization Value
ES – Real-Time Systems Lab19 | 03.03.2014 | PLE
Re-Formulation as Product Selection Problem
ILP finds Adequate Test Suite (if exists), but Problem is NP-hardAnother Point-of-View: Incremental Product Selection
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4
p1 p2 p3
rpTS
cTS
ppTS
ES – Real-Time Systems Lab20 | 03.03.2014 | PLE
Adequate Product Sub Set
Given:
Set of Weighted Requirements
Set of Weighted Test Cases
Set of Weighted Product Variants
Profit Goals
(Cost Bound
Find:
Sub set P such that an adequate test suite exists with
ES – Real-Time Systems Lab21 | 03.03.2014 | PLE
Incremental Product Selection
p1
p2p3
p4
𝑃 𝑆𝑖 𝑃 𝑆𝑖+1
p1
p2p3
p4
p5
P∖𝑃 𝑆𝑖
p7
p6p8
p9
p5
is adequate
is adequate
𝑇 𝑆𝑖
+
𝑇 𝑆𝑖
+
𝑇
profit
goals reached?
select next product
cost bounds
exceeded?
no adequate solution
found
ES – Real-Time Systems Lab22 | 03.03.2014 | PLE
Product Selection Heuristic1. Are Profit Goals reached? (rpTS ≥ krp and ppTS ≥ kpp)2. No:
a. For each unselected Product i. While unsatisfied Requirements exist, select one:
With high profit and uniqueness/rareness Select a satisfying Test Case :
With the ability to satisfy many other open Requirements for many other Products and low cost
b. Rank Products and Select By: Product profit, resulting Test Case costs, and Requirement profit (weighted)
c. Recalculate ppTS, rpTS and cTS d. Is Test budget excelled (cTS > kc)
i. Yes: “Error” Ende. Goto 1.
3. Yes: “Success” End
r
n
?
?
tn
?
?
?
cf. [Harrold et al., 1993]
[Harrold et al., 1993]
+ Products
ES – Real-Time Systems Lab23 | 03.03.2014 | PLE
Experimental ResultsVarying: Number of: Requirements, Test Cases, and Products Mapping: Requirements/Test Cases and Test Cases/Products Weights
In Total: 51 generated Data SetsTime Limit: 24h
t1
r1 r2 r3 r4 r5 r6
t2 t3 t4
p1 p2 p3
50, 100, and 500 Test Cases
50, 100, and 500 Products
50, 100, and 500 Requirements
Mean and Variance
Mean and Variance
-/o/+
-/o/+
Heuristic is 21.5 times faster
68.6% less
accuracy
Efficency improvement factor: 127.48
ES – Real-Time Systems Lab24 | 03.03.2014 | PLE
Discussion & Future WorkSummary Heuristic produces useful results and scales Feasible tradeoff between accuracy and efficiency
Ongoing and Future Work Backtracking if bounds are exceeded Search-based optimization techniques Methods and criteria to find appropriate weights and bounds Beyond ILP: Dynamic weight function, On-the-fly generation of test artifacts, …
Application to Real-World Case Studies
ES – Real-Time Systems Lab25 | 03.03.2014 | PLE
Industrial Case Study: Heidelberg Ion-Beam Therapy Center