Understanding the Human Estimator Gary D. Boetticher [email protected] Univ. of Houston - Clear Lake, Houston, TX, USA .

Understanding the Human Estimator

Gary D. Boetticher [email protected]. of Houston - Clear Lake, Houston, TX, USA

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Nazim Lokhandwala [email protected]. of Houston - Clear Lake, Houston, TX, USA

James C. Helm [email protected]. of Houston - Clear Lake, Houston, TX, USA

Introduction

Chaos Chronicles [Standish03]

300 billion dollars

250,000 new projects

1.2 million dollars per project

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Boehm’s 4X

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Types of Estimation [Jorgenson04]

63 - 86% Human-Based

7 - 16% Algorithmic and Machine

Learners

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Research Focus

Number of Papers On Software Estimation in IEEE [Jorgenson02]

Human-Based Estimation (17%)

Other (83%)

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Statement of Problem

How do human demographics affect human-based estimation?

Can predictive models be constructed using human demographics?

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Investigation Procedure

Collect demographics from participants

Request participants to estimate software components

Build models (Estimates vs. Actuals)

Su

rvey

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Which Demographics?

Basic Demographics

Academic Background

Work Experience

Domain Experience

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

The Survey

http://nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Competitive Procurement Software

Buyer Admin

Buyer1

Buyern

...

Buyer Software

DistributionServer

Supplier1

Supplier2

Suppliern

:

SupplierSoftware

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Sample Estimation Screenshots

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Survey Results Screenshots

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Data Collection

Invitations

Filtered Incomplete Records

122 Final Records

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Participant Educational Background

Most of the

participants hold

Bachelors or

Masters Degrees

Mean MaximumStandard Deviation

Computer Science

Undergrad Courses 8.8525 70 11.6326

Grad Courses 2.4262 15 3.2293

Hardware

Undergrad Courses 3.5246 64 8.0209

Grad Courses 0.5000 10 1.3252

Management Information Systems

Undergrad Courses 0.7705 12 1.5892

Grad Courses 0.4918 9 1.3742

Project Management

Undergrad Courses 0.2951 4 0.6886

Grad Courses 0.8115 6 1.1806

Software Engineering

Undergrad Courses 0.9180 7 1.2958

Grad Courses 2.1557 21 3.1202

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Participant Work Experience

 Mean Maximum

Standard Deviation(Years) (Years)

Years of Experience As      

Hardware Project Manager 0.6557 15 1.9251

Software Project Manager 1.3443 10 2.0811

No of Projects estimated      

Hardware Projects 0.8279 20 2.6307

Software Projects 2.9508 28 4.4848

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Participant Domain Experience

2.2512200.7274 Process Industry

1.3818100.6209 Procurement and Billing    

Domain Experience

Standard Deviation

Maximum(Years)

Mean

(Years)

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Data Preparation

INPUT= 69% zeros…Needs Consolidation

Courses, Workshops, Conferences, Programming Exp.

45 attributed reduced to 14 attributes Highest Degree Achieved…Need Transformation

9 4 2 0 0 0 1 0 0 0 9 1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 9 0 0 2 0 0 0 0 0 0 3 0 7 4 2 4 0 0

0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 5 0 5 2 0

0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 2 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 1

0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 0 1 1 1 0 2 0 0 1 0 0 1 0 0 0 4 0 0 0 0 5 1 0

OUTPUT=MRE=Abs (Total Actual – Total Est.)/(Total

Actual)

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Build Models

Linear Regression (Excel)

Non-Linear Regression (DataFit)

Genetic Programming (GDB_GP)

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

GP Configuration

3 Settings 1000 Chromosomes 50 Generations 512 Chromosomes 128 Generations 1000 Chromosomes 128 Generations

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

20Trialseach

Results: All Demographic Factors

1.87E-15 3.45E-17T-test

0.88470.55920.1550Mean

Non-Linear Regression

Genetic Programming

Linear Regression

1.6470

0.8847

Non-Linear Regression

Std. Error

R Squared

1.38754.4580

0.9174 0.1550

Genetic Programming

Linear Regression

Best Values of R Squared with Min. Std. Error

T-Test between Average R Square Values

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Results: Educational Factors

0.0486 2.74E-13T-test

0.21360.19730.0373Mean

Non-Linear Regression

Genetic Programming

Linear Regression

4.1667

0.2136

Non-Linear Regression

Std. Error

R Squared

3.97384.6101

0.27840.0373

Genetic Programming

Linear Regression

Best Values of R Squared with Min. Std. Error

T-Test between Average R Square Values

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Results: Work Experience

1.54E-11 2.73E-19T-test

0.36980.55640.0596Mean

Non-Linear Regression

Genetic Programming

Linear Regression

4.0644

0.3698

Non-Linear Regression

Std. Error

R Squared

2.28554.5169

0.75720.0596

Genetic Programming

Linear Regression

Best Values of R Squared with Min. Std. Error

T-Test between Average R Square Values

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Results: Domain Experience

4.55E-16 3.27E-23T-test

0.32600.54050.0243Mean

Non-Linear Regression

Genetic Programming

Linear Regression

3.9091

0.3260

Non-Linear Regression

Std. Error

R Squared

2.92834.5425

0.59110.0243

Genetic Programming

Linear Regression

Best Values of R Squared with Min. Std. Error

T-Test between Average R Square Values

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Summary of All Experiments

R Square Values

Linear Regression

Best Case Genetic Prog.

Avg. Case Genetic Prog.

Non-Linear Regression

All Factors 0.1550 0.9174 0.5592 0.8847

Education Only 0.0373 0.2784 0.1973 0.2136

Work Experience Only 0.0596 0.7572 0.5564 0.3698

Domain Experience Only

0.0243 0.5911 0.5405 0.3260

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Best Equation: All Factors. r2 = 0.9174((Log (TechGradCourses + (TechGradCourses ^ ((Log TotWShops)/(Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (((ProcIndExp + (Log (Sin MgmtGradCourses)))/(Sin SWPMExp)) + (Sin ((Cos (TechGradCourses ^ ((ProcIndExp + (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Sin SWPMExp)))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Cos (TechGradCourses ^ ((Log SWProjEstExp) / (((Log (ProcIndExp + (Log (TechGradCourses ^ ((Log SWProjEstExp) / (Log SWProjEstExp)))))) - 3) / (ProcIndExp + (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (TechGradCourses ^ (Log SWProjEstExp))))) / (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos (Log (Log (Log SWProjEstExp)))))))))))))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) + ((Log SWProjEstExp) / (Log SWProjEstExp)))))) / (Log (Log (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp))))))))))))))))))))))) / (TechGradCourses ^ (Log SWProjEstExp)))))) / (((Log ((((Log TotLangExp) / (Log SWProjEstExp)) / (Log SWProjEstExp)) / (Sin SWPMExp))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))) - 3) / (TechGradCourses ^ (Log SWProjEstExp)))))))))) + (((((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + ((TechGradCourses ^ (TechGradCourses ^ (Cos (TechGradCourses ^ ((ProcIndExp + (Log (Log (TechGradCourses ^ (TechGradCourses ^ (Cos (Log (Log (TechGradCourses ^ (Cos ((((Log SWProjEstExp) / ((ProcIndExp + (Log (TechGradCourses ^ (Log (TechGradCourses + (Cos (Log (Log (TechGradCourses ^ (Cos (((((Log SWProjEstExp) / (TechGradCourses ^ (Log SWProjEstExp))) / ((ProcIndExp + (Log (Sin MgmtGradCourses))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / ((Log SWProjEstExp) / (Log SWProjEstExp)))) / (Sin SWPMExp)) / (Sin SWPMExp)))))))))))) / (TechGradCourses ^ (Log SWProjEstExp))))))) / (Sin SWPMExp))))))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (TechGradCourses ^ (Log SWProjEstExp))) / (Sin SWPMExp)))

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Too Much of a Good Thing?

Conclusions

Viability of a human-based est. model Model assessment

Non-linear GP Impact on Human Based Estimation

1) All Factors

2) Domain Experience Work Experience

3) Education

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

Future Directions

Equation Optimizer for GP Collect More Data

Further analysis without consolidation Detailed Effect of Educational Factors

Use other statistical indicators Build other models

Hybrid (Non-linear and GP) Classifiers

Impact of process on estimation

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software Engineering (PROMISE) Workshop

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software

Engineering (PROMISE) Workshop

Questions?

http://nas.cl.uh.edu/boetticher/publications.html The 2nd International Predictor Models in Software

Engineering (PROMISE) Workshop

Thank You !