1 06/23/22
Dec 29, 2015
104/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP TestingApril 19, 2023 2
Software Systems – A Part of Daily LifeBanking
applications
Home appliances
Educational applications
Cars and other vehicles
Communicati-on Media
Software is now
omnipresent
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
• Reduction of errors• Increasing confidence of end users• Ensuring safe operations• Contributes to quality• Reduces cost of defect fixing
3
Role of Testing
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP TestingApril 19, 2023 4
Testing: Perspective shift…
• Establish confidence that a program does what it is supposed to do (Hetzel, 1973)
ISQT’s STEP-AUTO 2011: Conference on ERP TestingApril 19, 2023 5
Testing: Perspective shift…
• The process of executing a program or system with the intent of finding errors (Myers, 1979)
ISQT’s STEP-AUTO 2011: Conference on ERP TestingApril 19, 2023 6
Testing: Perspective shift…
• Verifying that a system satisfies its specified requirements and identifying the differences between expected and actual results
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 7
The Lacuna
04/19/23
SQT’s STEP-AUTO 2011: Conference on ERP Testing04/19/23 8
Where we stand now…
You can manage, what you can measureYou can manage, what you can measure
You can measure, what you can defineYou can measure, what you can define
You can define, what you understand.You can define, what you understand.
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
bug prediction and tracking
904/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Introduction
• Uses baseline data of the organization and size of the current project
• Predicts the expected number of bugs and the testing schedule for the current project
• Cumulative bug trend curve resembles the curve of a first order response system
• The actual cumulative bug trend is tracked against the estimated
• Any notable deviation from the bug prediction curve, analysis will be triggered and corrective actions taken accordingly.
1004/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 11
Cumulative Bug Trend Curve
• A curve with:– The days of
testing marked on X-Axis
– The cumulative number of bugs caught on Y-Axis
• The slope of the curve gives the rate of bug capture
050100150200250300350400
24-Jan 29-Jan 3-Feb 8-Feb 13-Feb 18-Feb 23-Feb 28-Feb 5-Mar 10-MarC
umul
ativ
e B
ugs
Day
CumulativeBug-TestDay Chart
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
• Monitoring– Bug Prediction Curve– Weekly Status Report– % Schedule Completed– Resource Utilization
Process In Testing - Monitoring
Schedule
ODB
ProjectTeam
No. Testing
Days
DefectDensity
Size
Defect Prevention measures
Expected Defects & Schedule
1204/19/23
CumulativeBug-TestDay Chart
0
100
200
300
400
24-Jan 3-Feb 13-Feb 23-Feb 5-Mar 15-Mar
Day
Cu
mu
lati
ve B
ug
s
Est vs Act Total Bugs Curve
0
50
100
150
200
0 2 4 6 8 10
No. Days of Testing
Cu
mu
lati
ve N
o. B
ug
s
EstDefectActDefectsULEst
LLEst
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 13
Bug Prediction Curve
Number of days
Cum
ulati
ve n
umbe
r of b
ugs
Helps in tracking the test execution
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 14
Bug Prediction Curve
0
50
100
150
200
0 2 4 6 8 10
Cumu
lative
No.
Bugs
No. Days of Testing
Bug Trend Curve
Est Defect
ULEst
LLEst
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 15
Possible Outcomes
• The actual curve - within expected trend• The actual curve - outside expected trend– The number of bugs in the initial quarters is much less
than estimated– The number of bugs in the initial quarters is much more
than estimated
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 16
The Scenario Diagram
Situations
Bugs in initial stage :Less than what is
estimated
Bugs in initial stage :More than what is
estimated
Stable Product InadequateTesting
UnstableProduct
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 17
Inadequate Testing
0
50
100
150
200
0 2 4 6 8 10
Cumu
lative
No.
Bugs
No. Days of Testing
Est vs Act Total Bugs Curve
Est Defect
Act Defects
ULEst
LLEst
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing 18
Unstable Product
0
50
100
150
200
0 5 10
Cumu
lative
No.
Bugs
No. Days of Testing
Est vs Act Total Bugs Curve
Est Defect
Act Defects
ULEst
LLEst
04/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Challenges in ERP Testing
• Wide range of stakeholders that use the system• Impact of changes across various modules and higher
risk of issues to be found at integration• Inherent slippages that is accumulated over various
phases of the development lifecycle• Tracking and Managing multiple test iterations in parallel• High pressure to ensure delivery dates as well as the
quality of delivery
1904/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
In the ERP context
• Bug trend curve can be calculated for different iterations• Specification limits can be defined for each iterations• Actual bug trend curve can be plotted and monitored for each
iterations
2004/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Bug Trend Curve for multiple iterations
2104/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Test Status Indicator
2204/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Test Status Indicator
• Root mean square of the variation between the estimated and actual bug trend curve
• The variation between the planned and actual value of the bugs can be calculated on a daily basis.
• The overall TSI values of the total projects in an organization can be plotted in a control chart for monitoring.
• The specification limits can be defined and from the control chart, the points which fall outside the limits can be analyzed by the top management.
2304/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
SPC for TSI
2404/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
SPC for TSI in different iterations
2504/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Advantages
2604/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Case Study
Project Language Plat form Start Date Cost of a Bug (Hrs/Bug)
% Testing Effort
Before implementation of the method
Project 1 VC Win NT 13-Mar-11 1.11 11%
Project 2 VC Win NT 21-Dec-10 1.28 37%
Project 3 C# Win NT 13-Mar-11 1.86 25%
After implementation of the method
Project 4 VC Win NT 16-May-11 0.68 13%
Project 5 VC Win NT 11-Jun-11 0.94 7%
Project 6 C# .net 29-Apr-11 0.63 7%
• Reduction in % effort for testing
• Implementation of Bug Prediction and Tracking method ensures quantitative monitoring and control
• Critical phases tracked in an efficiently and effectively
2704/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Questions
2804/19/23
ISQT’s STEP-AUTO 2011: Conference on ERP Testing
Thank you
2904/19/23