Sandeep Krishnan, ISU Robyn R. Lutz, ISU & JPL Katerina Goseva-Popstojanova, WVU Dept. of Computer Science, Iowa State University, MSR, May 22, 2011 1.

1

Empirical Evaluation of Reliability Improvement in an Evolving Software

Product Line

Sandeep Krishnan, ISURobyn R. Lutz, ISU & JPL

Katerina Goseva-Popstojanova, WVU

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

This research is supported by NSF grants 0916275 and 0916284

2

BackgroundProduct line – “A family of products designed to take advantage of

their common aspects and predicted variabilities” [Weiss and Lai 1999]

e.g. Nokia cellphones, HP printers, etc.

Products -

Commonalities – Shared by all products. e.g. Platform

Variabilities – Differentiate the productsHigh-reuse variation – JDT, PDE, Mylyn, Webtools, etc. Low-reuse variation – CDT, Datatools, Java EE tools.

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

3

MotivationDoes reliability improve as a product line evolves?

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

PL Evolution

Reliability

Reliability – Continuity of correct service; measured in terms of post-deployment failures [Avizienis, et al., 2001]

4

Research QuestionsFailure trends: Do serious failures decrease over time

as the common/high-reuse variation/ low-reuse variation components evolve over releases?

Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components?

Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components?

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

5

FindingsFailure trends: Do serious failures decrease over time as the

common/high-reuse variation/ low-reuse variation components evolve over releases?

Common High-reuse Low-reuse

Change trends: Does the percentage of new files and/or modifications to the source code decrease across releases for common/ high-reuse variation/ low-reuse variation components?

Common High-reuse Low-reuse

Failure/Change relationships: Does the number of serious failures normalized for source-code changes decrease over time for the common/ high-reuse variation/ low-reuse variation components?

Common High-reuse Low-reuse Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

6

Approach

Source of failure reports-

Failure cateogries –

Raw number of severe failures and percentage.Source of change reports – CVS repository of Eclipse.We look at change to existing files measured by KChanges

and change via new files [Mockus et al., 2000].Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

Blocker

Critical Major

Normal Minor

2007 2008 2009 2010 2011

Europa

Ganymede

Galileo

Helios

Blocker

Critical Major

7

Observations

Number of severe failures decreases over time as expected

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

1. A. Failure trends – Common components

8

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

Percentage of severe failures tends to stabilize and even increases gradually

1. A. Failure trends – Common components

9

Observations

Number of severe failures show mixed behavior, contrary to our expectations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

1. B. Failure trends – High-reuse variation components

10

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

Percentage of severe failures tends to stabilize and even increases gradually, contrary to our expectations

1. B. Failure trends – High-reuse variation components

11

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

Number of severe failures do not monotonically decrease; rather show mixed behavior

1. C. Failure trends – Low-reuse variation components

12

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

Percentage of severe failures show mixed behavior and not a decreasing trend

1. C. Failure trends – Low-reuse variation components

13

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

2. Change trends

• Common components: Percentage of new files decreases.• High-reuse variation components: Percentage of new files

tends to remain stable.• Low-reuse variation components: Percentage of new files

shows decreasing trend.• All components show relatively high modification rate.

14

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

3. Failure/evolution relationship

• Failures/new-file o Common components: For two subcomponents, release 1 has

lower failures/new-file than release 4. o High-reuse variation components: Release 1 always has higher

failures/new-file than release 4. o Low-reuse variation components: No specific trend.

15

Observations

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

3. Failure/evolution relationship

• Failures/Kchangeso No stable behavior for all components; mixed trends are

observed.

16

ConclusionCommon components have fewer severe failures over time

and amount of change also decreases.

Variation components show no uniform pattern of decrease in failures.

Even high-reuse variation components continue to change fairly rapidly.

Study suggests there may be no simple answer as to whether reliability tends to improve as product line matures.

Further investigation is needed on the relationship between failure and change in software product lines.

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011

17

Thank You!

Dept. of Computer Science, Iowa State University, MSR, May 22, 2011