Investigating Individual Software Development: An Evaluation of the Leap Toolkit

(1)

Cam Moore

Collaborative Software Development Laboratory

Communication & Information Sciences

University of Hawaii, Manoa

cmoore@hawaii.edu

http://csdl.ics.hawaii.edu/Research/LEAP/LEAP.html

Investigating Individual Software Development: An

Evaluation of the Leap Toolkit

(2)

Outline

Why Is Quality Software Development Important?

Solutions

Thesis

Evaluation

Contributions & Future Directions

(3)

Problem Every Software developer wishes they got home earlier and spent less of their weekend at work.

Software is often:•Late

-Windows 2000•Over budget

-DIA baggage control•Full of defects

-Therac-25

Problem

Solutions

Leap

Evaluation

Contributions

(4)

Solutions

Hope Based (Try Harder)1960s

Product Based (Testing, Review)1970s

Organization Based (CMM, ISO 9000)1980s

Developer Based (PSP, Leap)1990s

Problem

Solutions

Leap

Evaluation

Contributions

(5)

Problems with traditional solutions

Many software process improvement initiatives suffer from one or more of the following problems:

•Heavy weight development constraints

•Organization-level analysis and improvement

•Measurement Dysfunction

•Manual Data Collection

Problem

Solutions

Leap

Evaluation

Contributions

(6)

Supporting Software Developer Improvement

with LEAP LEAP is our design philosophy. All LEAP tools must satisfy four major criteria:

•Light weight

•Empirical

•Anti-Measurement dysfunctional and

•Portable

The Leap toolkit is a reference implementation

Problem

Solutions

Leap

Evaluation

Contributions

(7)

Leap Toolkit Started August 1997 Over 25 public releases

As of 17 Apr. 2000, the size of the Leap Toolkit is:•44KLOC of Java, •2156 Methods, •311 Classes and, •14 Packages

It is available for download from http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html

Problem

Solutions

Leap

Evaluation

Contributions

(8)

Leap Toolkit

(9)

General Thesis

LEAP provides a more accurate and effective way for developers to collect and analyze their software engineering data than methods designed for manual enactment.

Problem

Solutions

Leap

Evaluation

Contributions

(10)

Evaluation To evaluate this thesis I broke it into 3 claims:

I. Leap is able to prevent important classes of data error as identified in Disney98.

II. Leap reduces the level of collection stage errors by reducing the overhead of data collection.

III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP.

Problem

Solutions

Leap

Evaluation

Contributions

(11)

I: Preventing data error I. Leap is able to prevent important classes of data error as identified in Disney98.

Disney’s research discovered the following problems with manual PSP data:

• Students had a 5% error rate in their data

• These errors were significant-Time estimation-Incorrect Yield

Even 95% accuracy is not good enough

Problem

Solutions

Leap

Evaluation

Contributions

(12)

I: Preventing data error The design and implementation of the Leap toolkit address the data error categories:Error Category Leap solution

Calculation error(46%)

Automatedcalculations

Transfer error(20%)

MVCarchitecture

Blank fields(18%)

Default values<Undefined>

Problem

Solutions

Leap

Evaluation

Contributions

(13)

I: Preventing data error Data error categories (cont):

Error Category Leap solution

Entry error (9%) Default valuesPopup menus

Impossible values(6%)

Error checking

Sequence error(1%)

Not an errorProblem

Solutions

Leap

Evaluation

Contributions

(14)

II: Reducing collection error

II. Leap reduces the level of collection stage errors by reducing the overhead of data collection.

Case Study:•16 Students used Leap while developing 8 Java programs in ICS 613

•Data collection:-4 surveys conducted (# responses)-Interviews (11 conducted)-Students’ Leap data (16 data sets)

Problem

Solutions

Leap

Evaluation

Contributions

(15)

II: Reducing collection error

The students were asked to collect three types of data:

•Defects-Students used many different collection methods

•Size-LOCC used for collecting size data

•Time-Two different time collection tools

Problem

Solutions

Leap

Evaluation

Contributions

(16)

II: Reducing collection error

First method of time collection in LeapNaia

Defined Values Default Values Current Time Automatic

Duration

Problem

Solutions

Leap

Evaluation

Contributions

(17)

II: Reducing collection error

Second method of time collection in Leap

Io

Defined Values Current Time One click

operation

AutomaticDuration

Problem

Solutions

Leap

Evaluation

Contributions

(18)

II: Reducing collection error

Using Naia for data entry makes data entry harder

•Students using Naia said data collection was neutral to hard

•Students using Io said data collection was easy to neutral

Problem

Solutions

Leap

Evaluation

Contributions

(19)

II: Reducing collection error

Types of time collection errors•Missing time•Added time•Rounding

Reported use of Naia has high correlation to rounded timesPercentage of times that were a multiple of 5

minutes

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

16 14 7 3 8 1 4 15 2 13 11 10 6 9 12 5

Student

Percentage of times that were a multiple of an hour

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

15 7 14 3 6 8 12 4 13 11 10 1 5 16 2 9

Student

(20)

III: Empirical Investigation III. Leap implements data collection and analyses that enable sophisticated analysis not available in manual methods such as PSP.

Leap allowed me to evaluate 14 different quantitative time estimation techniques.

A

BCActual Size

Planned Size

Avera

ge

Lin

ear

Exp

on

en

tial LOC

Methods

PSP (A, B, or C)

Problem

Solutions

Leap

Evaluation

ContributionsStudent’s

(21)

III: Empirical Investigation I used Leap to calculate the prediction error for each estimation method.•Prediction Error = | estimate - actual |

•Compared methods for each student

•Compared methods for the whole class

Problem

Solutions

Leap

Evaluation

Contributions

actual

(22)

III: Empirical Investigation Individual Students’ Results:

3 of the 16 students had significantly better estimation methods•2 students, their own estimation methods were best (F = 3.93, p < 0.001), (F = 2.25, p = 0.019)

•1 student, exponential regression using actual LOC was best (F = 2.34, p = 0.014)

•5 students, own estimates best

Problem

Solutions

Leap

Evaluation

Contributions

(23)

III: Empirical Investigation Class’ Results:

The students’ own estimates were the most accurate (F = 3.75, p <0.001)

Ignoring the students’ estimates, the Exponential Regression using actual methods was the best (F = 2.40, p = 0.005)

The PSP’s time estimation method was 6th

Problem

Solutions

Leap

Evaluation

Contributions

(24)

Contributions The Leap toolkit•An effective tool for software developer improvement-Reduces overhead-Flexible-Portable

•A novel software engineering tool for:-Education-Research

Problem

Solutions

Leap

Evaluation

Contributions

(25)

Contributions The results of the estimation experiment•Insight into better software estimation techniques

Experience may lead to over-confidenceEstimation Accuracy

-40.00%

-30.00%

-20.00%

-10.00%

0.00%

10.00%

1 2 3 4 5 6

Project

Err

or

% Less Exp

More ExpProblem

Solutions

Leap

Evaluation

Contributions

(26)

Future Directions Replicated studies with more subjects including industrial software developers

Industry adoption of Leap Toolkit

More support for data analysis in Leap Toolkit

Further study into effort estimation

Personal Agents that “observe” the developer

Problem

Solutions

Leap

Evaluation

Contributions

(27)

(28)

Results: Case Study Estimation accuracy unaffected by rounding error•No significant difference between the estimation accuracies of the class (F = 1.71, p = 0.07)

•No significant difference between Naia and Io users (F = 2.70, p = 0.10)

Estimation accuracy

-30.00%

-25.00%

-20.00%

-15.00%

-10.00%

-5.00%

0.00%

5.00%

1 2 3 4 5 6

Project

Err

or

% Naia ave

Io ave

(29)

Results: Case Study There was very little evidence of measurement dysfunction

•Grades not based upon estimation accuracy

•Students did not compare their Leap data

•Most students felt little pressure to make their data look “good”-Most said they felt less pressure-Two with increasing remained very low

(30)

Results: Case Study Students showed traditional software developer improvements

•Increased Productivity

•Improved size estimation skills

•Improved time estimation skills

(31)

Increased Productivity

LOC•EqualSlight improvement

Methods•Equal

Productivity LOC/hr.

0

20

40

60

80

100

120

0 2 4 6 8 10

Project

LO

C/h

r.

Productivity Methods/hr

0.00

2.00

4.00

6.00

8.00

10.00

12.00

0 2 4 6 8 10

Project

Met

ho

ds/

hr

(32)

Productivity Size of methods increased•Projects changed from Command line to GUI

•Last few project were more complicated

Problem

Solutions

Leap

Evaluation

Contributions

LOC/Method

0.00

10.00

20.00

30.00

40.00

50.00

0 2 4 6 8 10

Project

LO

C

(33)

Improved Size Estimation

LOC•Steady Underestimation

Methods•Improvement•BetterDesigns?

Size Estimation Accuracy (LOC)

-100.00%

-50.00%

0.00%

50.00%

100.00%

Project

Est

imat

ion

Err

or

Size Estimation Accuracy (Methods)

-100.00%

-50.00%

0.00%

50.00%

100.00%

150.00%

Project

Est

imat

ion

Err

or

(34)

Improved Time Estimation Class’ time estimation accuracy•Under estimated•Range consistent

Time Estimation Accuracy

-80.00%

-60.00%

-40.00%

-20.00%

0.00%

20.00%

40.00%

60.00%

80.00%

Project

Est

imat

ion

Err

or