Industry - Estimating software maintenance effort from use cases an industrial case study

Estimating Software Maintenance Effort from Use Cases: an Industrial Case Study

Yan Ku, Jing Du, Ye Yang, Qing Wang

Institute of Software, Chinese Academy of Sciences

2011-09-29

*This work is supported by the National Natural Science Foundation of China under Grant Nos. 90718042, 60873072, 60903050 and

61073044; the National Hi-Tech Research and Development Plan of China under Grant Nos. 2007AA010303, 2007AA01Z186 and

2007AA01Z179.

Agenda

� Motivation

� Industrial Setting

� Methodology

� Case Study

� Discussion

� Tool Implementation

22

Motivation

� An anecdote in software engineering domain*:

� Elephant: Discipline

� Monkey: Agile

� Elephant & Monkey: Practical

33

*Barry W. Boehm, Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Aug, 2003

*Picture Source: www.image.baidu.com

Motivation

� “Speaking of the problems we are facing during software lifecycle, hmmm, there is no way a short list. … I hate to, but I have to say that effort estimation especially for maintenance project, if not totally impossible, is really a big challenge

-----A Project Manager

� Formal estimation models: time-consuming, requiring

enough information and data---- the elephant

� Expert Judgment: expert-dependent, easily lead to

thumb-up---- the monkey

44*Picture Source: http://www.zcool.com.cn

Industrial Setting

� The industrial research is inspired by the estimation

dilemma a mentioned before.

� The problem occurred in developing the leading product

of a medium-size software enterprise with CMMI ML4

in China.

� The product, named QONE, is a commercial software

process management tool.

� It has contributed to the process improvement for more

than 300 small and medium-sized software companies

and organizations in China.

55

Industrial Setting (Con)

� Since 2004, QONE has released several major versions as well as branches for special customized ones in succession.

� Several of the evolving versions are maintenance projects.

� Expert estimates were mainly used in the past effort estimation of QONE.

� The estimation results are not so stable due to the objectivity and other issue.

� The actual effort and other data including use case documents have been accumulated by QONE itself.

66

2009-3-202008-9-1v7

2008-8-212008-3-20v6

2008-7-312007-12-10v5

2007-10-312007-5-28v4

2007-3-302006-1-16v3

2005-11-302005-7-11v2

2004-11-152004-10-8v1

End DateBegin DateVersions

Methodology

� Goal: Achieving the balance of simplicity, early-estimating and accuracy in one effort estimate .

� Methodology Principles:

� Apply use cases as the size metric and introduce requirement elaboration factors to make the estimate in advance

� Introduce adjusted factors as few as possible in order to reduce the complexity

�Take advantage of the history data to help improve the estimation accuracy.

77

Maintenance task type is not distinguished

due to the difficulty for effort classification

Modeling Process

88

Get the lowest level

requirementsUse the same

data unit

Count Data

� Use case: number of use cases � newUC: new-added

� modUC: modified

� reuUC: reused without modified

� delUC: delete

99

Modeling Process

1010

Count Data & Construct Model

� Weight � Wmod/ Wreu / Wdel :effort ratio of

modified/reused/deleted use case to a new-added one

1111

Sizeadjusted = newUC + Wmod * modUC

+Wreu * reuUC + Wdel * delUC

Effort = A * (Sizeadjusted) B

Where

Effort is the maintenance effort;

Sizeadjusted is the adjusted product size;

A is the multiplicative calibration constant;

B is the exponential calibration constant

Modeling Process

1212

Validation

� Metric Definition

� MRE = |predictive effort – actual effort| / actual effort

� Referred Measures

� MMRE: mean magnitude relative erro

� MdMRE: median magnitude relative error

� PRED25: the % of the data points with RE<=0.25

� PRED30: the % of the data points with RE<=0.30

1313

Prediction Process� Data collection

� Prediction

� To apply in different phases during lifecycle,

elaboration factors are referred to estimate size input.

� Sizere-adjusted = EF * Sizeadjusted,where EF is the elaboration factor between higher and lower-level requirements

� whereNRi is the number of higher-

level requirements and NUCi is the number of lower-leverlrequirements in ith data point

1414

∑=

=

n

1i iNRiNUC

n

1EF

*Picture Source: Alistair Cockburn, Writing Effective Use Cases

Case Study

� Historical data are the projects mentioned in

slice 6.

� Requirements are described in three level:

� capability goals (CG, least detailed)

� capability requirements (CR)

� use cases (UC, most detailed)

1515

16164366815Use cases(UC)

05133Capability requirements(CR) 7166

01830Capability goals(CG)

v7


032202Capability requirements(CR) 14903.6


v6




v5




v4




v3




v2




v1

Actual Effort

(person/hour)

Number of

deleted

Number of

reused

Number of

modified

Number

of newRequirements LevelsVersions

Case Study

� Weight:

� Wmod = 0.2, Wreu = 0.05, Wdel = 0

1717

Sizeadjusted = newUC + 0.2 * modUC +0.05 * reuUC

0.913862Adjusted R2

0.928219R2

0.000481P-value

1.192651B

96.9396A

ValuesParameters

Effort = 96.9396 * (Sizeadjusted) 1.192651

Validation Result

� Leave-one-out cross validation is applied.

� Elaboration factors are referred from A. A. Malik’s research* since the data used there is the subset of our dataset.

1818

*A. A. Malik, B. W. Boehm, Y. Ku, and Y. Yang, “Comparative Analysis of Requirements Elaboration of an Industrial Product,”

Proceedings of the 2nd International Conference on Software Technology and Engineering(ICSTE 2010), Oct. 2010, pp. 46-50

0.85710.71430.5714PRED30

0.71430.57140.4286PRED25

20.01%24.13%27.09%MdMRE

26.94%26.18%36.87%MMRE

UCCRCGMetrics

UC yields the best

result

Validation Result (Cont)

1919

7.506628.5125716634.9v7

20.0111921.178614903.658.55v6

17.946136.34557477.533.25v5

86.3019263.978010340.173.65v4

24.7823276.772730945111.5v3

4.393767.9859394121.75v2

27.682916.89252284.515.8v1

MRE (%)Predictive EffortActual EffortAdjusted SizeVersions

1. COTS has been used

2. Increased productivity through

requirement management

Method Comparison

� Analogy � Database used for analogy is from China Software

Benchmarking Standard Group (CSBSG)

� 999 software project data from 140 organizations distributed in 15 regions across China

� COCOMO2000

2020

0.85710.714320.01%26.94%Use case

0.57140.571415.47%32.47%COCOMO

0.28570.285731.59%33.09%Analogy

PRED30

PRED25

MDMREMMREMethods

Discussion

� Lessons Learned:

� Use Case Metrics

� Requirement Elaboration Factors

� Advantages of use-case based estimation

� Linear vs. Exponential relationship between effort and use case

� Threats to Validity

� Internal threats:

� Weak outlier tolerance

� Complexity of use cases

� External threats:

� Use case weight

2121

Tool Implementation

2222

Than You！Our Lab: http://itechs.iscas.ac.cn

Industry - Estimating software maintenance effort from use cases an industrial case study

Technology

effort estimate

use case documentsand2006

effort classificationintroduce

software lifecycle

number of use cases

past effort estimationof

history data

data points