University of Southern California Center for Systems and Software Engineering Value-Based Software Test Prioritization Annual Research Review CSSE-USC Qi Li, Barry Boehm {qli1, boehm}@usc.edu March 7, 2012
Dec 31, 2015
University of Southern California
Center for Systems and Software Engineering
Value-Based Software Test Prioritization
Annual Research ReviewCSSE-USC
Qi Li, Barry Boehm{qli1, boehm}@usc.edu
March 7, 2012
University of Southern California
Center for Systems and Software Engineering
Outline
• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion
04/19/2023 ARR 2012 2
University of Southern California
Center for Systems and Software Engineering
Research Motivation• Value-neutral SE methods are increasingly risky [Boehm, 2003]
– Every requirement, use case, object, test case, and defect is equally important– “Earned Value” Systems don’t track business value– System value-domain problems are the chief sources of software project failures
• Testing & Inspection resources are expensive and scarce– 30%-50%, even higher for high reliability projects [Ramler, 2005]– Time-to-market [Boehm, Huang, 2005]
• Empirical Findings [Bullock 2000, Boehm & Basili 2001 ]– About 20 percent of the features provide 80 percent of business value– About 80 percent of the defects come from 20 percent of the modules– …
• Value-based Software Engineering 4+1 theorem [Boehm, 2005]
04/19/2023 ARR 2012 3
University of Southern California
Center for Systems and Software Engineering
Outline
• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion
04/19/2023 ARR 2012 4
University of Southern California
Center for Systems and Software Engineering
Value-Based Software Test Prioritization
What to be prioritized? •Testing items: Testing Scenarios, Testing Features, Test Cases
How to prioritize?•Value-Based (Business Importance, Risk, Cost)•Dependency Aware
How to Measure?•Average Percentage of Business Importance Earned (APBIE)
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based• Risk Exposure (RE)
– Where Size (Loss) is the risk impact size of loss if the outcome is unsatisfactory, Pro (Loss) is the probability of an unsatisfactory outcome
• Risk Reduction Leverage (RRL)
– Where REbefore is the RE before initiating the risk reduction effort and REafter is the RE afterwards.
– RRL is a measure of the cost-benefit ratio of performing a candidate risk reduction or defect removal activity
604/19/2023 ARR 2012
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based
– Business Case Analysis– Stakeholder
Prioritization
– Impact of Defect
– Experience Base
04/19/2023 ARR 2012 7
– Business Value
Defect Criticality
– Defect-prone Components, Performers
• Value-Based Prioritization Drivers:
– Size of Loss
– Probability of Loss
Risk Exposure
• Testing items are to be ranked by how well they can reduce RE
University of Southern California
Center for Systems and Software Engineering
Research Method: Value-Based• Combining with the testing items’ relative costs• =>Priority Trigger:
• This proposed strategy enables them to be prioritized in terms of Risk Reduction Leverage (RRL) or ROI
• Supposed to improve the lifecycle cost-effectiveness of defect removal techniques
04/19/2023 ARR 2012 8
University of Southern California
Center for Systems and Software Engineering
Animated displays
Secondary application functions
User amenities
Main application functions
Natural speech input
Tertairy application functions
Basic application functions
Data management system
Operating System
Investment High-payoff Diminishing returns
Cost of software product [Boehm, 1981]Va
lue
of
so
ftw
are
pro
du
ct
to o
rga
niz
ati
on
Research Method: Dependency Aware
04/19/2023 9ARR 2012
University of Southern California
Center for Systems and Software Engineering
Research Method: Dependency Aware• Dependency:
– Example: dependencies among test cases to be executed
– Solution: Prioritization Algorithm (greedy alg)• Select the one with the
highest RRL• Check dependency
04/19/2023 ARR 2012 10
9->3->9->5->9->4->7
University of Southern California
Center for Systems and Software Engineering
Research Method: Metrics• Testing Cost Effectiveness
– Average Percentage of Business Importance Earned (APBIE)
04/19/2023 11ARR 2012
University of Southern California
Center for Systems and Software Engineering
Outline
• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion
04/19/2023 ARR 2012 12
University of Southern California
Center for Systems and Software Engineering
Case Studies Results• Exercise Test Prioritization based on Risk
Reduction Level (RRL)software testing scenarios to be walked through in
Galorath.Incsoftware features to be tested in a Chinese companysoftware test cases to be executed in USC SE course
projects
All of them show preliminary positive results
04/19/2023 ARR 2012 13
University of Southern California
Center for Systems and Software Engineering
Case Studies Results (Galorath Inc.)Prioritize testing scenarios to be walked through• Galorath Inc. (2011 Summer)
04/19/2023 ARR 2012 14
APBIE-1 70.99%
APBIE-2 10.08%
APBIE-3 32.10%
– Value-based prioritization can improve the cost-effectiveness of testing
8 10 12 14 16 18 20 22 24 26 28 30
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
35.80%
39.51%
45.68%
51.85%
58.02%61.73%
87.65%
4.94%6.17%
9.88%
16.05%22.22%
25.93%
58.02%
74.07%
77.78%
83.95%
90.12% 93.83%95.06%
100.00%
PBIE-1
PBIE-2
PBIE-3
Stop Testing
Value-based
Value-neutral
Value-inverse (worst case)
University of Southern California
Center for Systems and Software Engineering
Outline
• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion
04/19/2023 ARR 2012 15
University of Southern California
Center for Systems and Software Engineering
Automated Tool Support (Beta Version)
04/19/2023 ARR 2012 16
University of Southern California
Center for Systems and Software Engineering
Tool Demo
04/19/2023 ARR 2012 17
Website: http://greenbay.usc.edu/dacs/vbt/testlink/index.php
University of Southern California
Center for Systems and Software Engineering
Future Features• Establish the traceability matrix between the requirement
specifications and test cases to automatically obtain test case business importance ratings
• Establish the traceability matrix between test cases and defects in order to automatically predict the fail probability based on the collected historical defect data via incorporating the-state-of-art defect prediction techniques
• Experiment sensitivity analysis for reasoning and judging the correctness of factors’ ratings, or how ratings changes will impact the testing order
04/19/2023 ARR 2012 18
If you would like to be a beta version tester, please contact me at [email protected]
University of Southern California
Center for Systems and Software Engineering
Outline
• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion
04/19/2023 ARR 2012 19
University of Southern California
Center for Systems and Software Engineering
Conclusion• Propose a Real “Earned Value” System to Track Business Value
of Testing and Measure Testing Efficiency in terms of APBIE• Propose a Systematic Strategy for Value-based, Dependency
Aware Test Processes• Apply This Strategy to a Series of Empirical Studies with
different granularities of Prioritizations• Elaborate Decision Criteria of Testing Priorities Per Project
Contexts, Which are Helpful for Real Industry Practices• Implement an automatic tool for its application on large-scale
industrial projects
04/19/2023 ARR 2012 20