Yan Ma, Bojan Cukic Yan Ma, Bojan Cukic Lane Department of Computer Science and Electrical Lane Department of Computer Science and Electrical Engineering Engineering West Virginia University West Virginia University May 2007 Adequate Evaluation of Adequate Evaluation of Quality Models Quality Models in Software Engineering in Software Engineering Studies Studies CITeR CITeR The Center for Identification Technology Research www.citer.wvu.e du An NSF I/UCR Center advancing integrative research
19
Embed
Adequate and Precise Evaluation of Predictive Models in Software Engineering Studies
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
Yan Ma, Bojan Cukic Yan Ma, Bojan Cukic
Lane Department of Computer Science and Electrical EngineeringLane Department of Computer Science and Electrical Engineering
West Virginia UniversityWest Virginia University
May 2007
Adequate Evaluation of Quality Models Adequate Evaluation of Quality Models
in Software Engineering Studiesin Software Engineering Studies
CITeRCITeR The Center for Identification Technology Researchwww.citer.wvu.eduAn NSF I/UCR Center advancing integrative research
Evaluating Defect Models
• Hundreds of research papers. – Most offer very little one can generalize and
reapply.– Initial hurdle was the lack of data, but not any
longer:• Open source repositories, NASA MDP, PROMISE
datasets.
• How to evaluate defect prediction models?
Software defect data: Class Imbalance
• A few modules are fault-prone.– A problem for supervised learning algorithms, which
typically try to maximize overall accuracy.
Software Defect Data: Correlation
MDP-PC1: Pearson correlation coefficients
LOC TOpnd V B LCC N
LOC 1.000 0.908 0.937 0.931 0.545 0.924
TOpnd 0.908 1.000 0.976 0.971 0.464 0.996
V 0.937 0.976 1.000 0.995 0.468 0.987
B 0.931 0.971 0.995 1.000 0.468 0.982
LCC 0.545 0.464 0.468 0.468 1.000 0.473
N 0.924 0.996 0.987 0.982 0.473 1.000
Software Defect Data: Correlation (2)
MDP-KC2: Pearson correlation coefficients
LOC UOp V IV.G TOp LOB
LOC 1.000 0.632 0.986 0.968 0.991 0.909
UOp 0.632 1.000 0.536 0.577 0.615 0.636
V 0.986 0.536 1.000 0.970 0.990 0.887
IV.G 0.968 0.577 0.970 1.000 0.972 0.836
Top 0.991 0.615 0.990 0.972 1.000 0.912
LOB 0.909 0.636 0.887 0.836 0.912 1.000
Software Defect Data: Correlation (3)
• Five “most informative attributes”
Software Defect Data: Module Size
• Defect-free modules are smaller.• In MDP, modules are very small.
The 90th percentile of LOC for the collection of defect modules and defect-free modules
KC1 KC2 PC1 JM1 CM1
Defect-free 42 55 47 72 55
Defect 99 167 114 165 131
Software Defect Data: Close Neighbors
• The “nearest neighbor” of most defective modules is a fault free module. – Measured by Euclidian distance between module
metrics
Project% of defect modules whose
nearest neighbor is a majority class instance
% of defect modules that has 2 among the three nearest
neighbors in the majority class
KC1 66.26% 73.62%
KC2 58.33% 58.33%
JM1 67.90% 75.46%
PC1 75.32% 85.71%
CM1 73.47% 97.96%
Implications on Evaluation
• Many machine learning algorithms ineffective.– But one would never know by reading the literature.