Empirical Software Engineering Models · CSMR 2008 PuLSE With SPL approach, productivity has tripled # of quality problems has been reduced to 1/5 ... Handbook capturing existing

Empirical Software Engineering Models:Can they become the Equivalent of Physical Laws in Traditional Engineering

Dieter Rombach

TU Kaiserslautern & Fraunhofer IESE

Kaiserslautern, Germany

∏=

××=17

1i

i

bEMSizeaCost

Dieter Rombach

Symposium in Honor of Barry W Boehm

Slide 2

Beijing, China, 26-27 April 2011

THANKS to BARRY BOEHM

• For having been and continuing to be- One of the key visionaries in our field

� Cost prediction (COCOMO)

� Risk mitigation (Risk based SD models)

� Lag-time based increase of defect cost

� Etc.

- A great mentor to me personally

� Met him first in 1986

� Supported my NSF Investigator award nomination in 1990

• For having influenced

- The establishment and successful operation of

the Fraunhofer Institute for Experimental

Software Engineering (IESE) since 1996

- Ongoing work towards a scientific basis for software engineering

∏=

××=17

1i

i

bEMSizeaCost

Dieter Rombach


Slide 3


AGENDA

• Motivation

• Traditional vs. Software Engineering

• “Laws” in Software Engineering

- Empirical Models

- Laws

• State-of-the-Practice / State-of-the-Art

• Towards a Software Engineering Discipline

• Future Challenges

Dieter Rombach


Slide 4


MOTIVATION

Why do we need a scientific basis in software engineering based

on some form of laws?

•Engineering requires scaleability based on law-enforced regularities

•In traditional engineering

- Physical laws require regularities

- Regularities enable scaleability of engineering

•In software engineering

- Cognitive laws are badly encoded (principles instead of laws)

- Regularities are not applied widely (scaleability problems exist)

Dieter Rombach


Slide 5


TRADITIONAL vs. SOFTWARE ENGINEERING

What is the nature of such laws? Why do we often ignore them in

software engineering?

•Physical laws

- Precise

- Absolute

- Non circumventable

- Immediately experiencable (in lab)

Dieter Rombach


Slide 6


Physical Laws as Base for Regularity

Example: Crosstalk in microelectronics wiring

• Small distances between signal wires results in capacitive coupling (crosstalk)

• Predictability and minimization of crosstalk mandatory

�Regular interconnect fabrics with supply shielding between signal wires

S

S SV V S

G

S

SV

G

VS

S SV V S

G

S

SV

G

V

V= Vdd supply

G = Gnd supply

S = SignalLayer 2

Layer 1

Dieter Rombach


Slide 7


Regularity as Base for higher scalability, energy efficiency and lower NRE costs

• Regularity increases scalability, e.g. memory design, multi-core architectures

• Regularity reduces the number of components that need to be designed and verified at all levels of abstraction � reduces NRE costs

• Regularity eases parallelism � improves power efficiency

1Gbit DRAM Intels 48 core architecture

Dieter Rombach


Slide 8


TRADITIONAL vs. SOFTWARE ENGINEERING

What is the nature of such laws? Why do we often ignore them in

software engineering?

•Physical laws

- Precise

- Absolute

- Non circumventable

- Immediately experiencable (in lab)

•Cognitive laws (e.g., system complexity has an impact on dev. effort)

- Imprecise (uncertainty) � Walker Royce

- Relative (context) � Barry Boehm

- Believed to be circumventable (at least for some time)

- Experiencable with a time delay only

Dieter Rombach


Slide 9


Software Engineering (Scaling w/o regularities?)

Dieter Rombach


Slide 10


LAWS in SOFTWARE ENGINEERING

What is the nature of cognitive software engineering laws?

•(A) Empirical predictive models of the following kind:

Problem/ Rqmts

Units

SW-System/Product

t

Q/P/T (A1) == f (A2,Context)<var>

f

Dieter Rombach


Slide 11



• Testing technique T identifies

- 85% of all defects (+/-5%) iff (Code.complexity <= C0 & Exp = high)

- 65% of all defects (+/-10%) iff (Code.complexity > C0 & Exp = high)

- 50% of all defects (+/-10%) iff (Code.complexity > C0 & Exp < high)

- Etc.

Dieter Rombach


Slide 12


Overview Slide 12


time

T == f0 (P)

T == f1 (P, C1)

T == f2 (P, C2)

Development Process

Size of Systems

Size of systems &

Exp. of developers

150%

50%

8%

T_actual – T_planned

Dieter Rombach


Slide 13



What is the nature of cognitive software engineering laws?

•(B) Qualitatively stable empirical predictive models

- Product � product (e.g., design complexity � code defect density)

- Product � process (e.g., design complexity � testing effort)

- Process � product (e.g., inspection defect removal rate � system reliability)

- Process � process (e.g., effort distribution across phases)

Problem/ Rqmts

Units

SW-System/Product

t

σ

Dieter Rombach


Slide 14



• Observations (characterizing � predictive)

- One (or more) study (studies) in one context

- No qualitative/quantitative stability guaranteed

• Laws (predictive)

- A representative set of studies for some context

- Qualitative stability, quantitative variability

• Theories

- A representative set of studies for some context

(all relevant independent variables captured)

- Qualitative stability, quantitative stability (small variability)

Dieter Rombach


Slide 15


STATE-of-the-PRACTICE / STATE-of-the-ART

What is the state of practice?

•Empirical models are used by few (but increasing numbers of)

companies only

• But leading companies do

(see IEEE/SEI Process Achievement Award)

• High correlation with business success (Infosys: SPA Award 2009)

•Naive tendency to reuse models from other environments

(e.g., COCOMO without customization)

Dieter Rombach


Slide 16



What is the state of the art?

•There exist more models than we typically recognize

- mostly in terms of context-specific empirical observations

- rarely in terms of generalized “laws”

•Fraunhofer Institute for Experimental Software Engineering (IESE) maintains an Experience Base

Dieter Rombach


Slide 17


Business Areas Divisions

Executive DirectorProf. Dr. D. Rombach

Scientific DirectorProf. Dr. P. Liggesmeyer

Deputy DirectorProf. Dr. F. Bomarius

Product Sectors, e.g.R. Kalmar

Service Sectors,e.g.M. Ochs

Embedded Systems (ES) Dr. M. Trapp

Process Management (PM) Dr. J. Heidrich

Information Systems (IS) Dr. J. Dörr

ES Development Dr. M. Becker

ES Quality Assurance Dr. R. Eschbach

Meas’t, Pred & Empiricism Dr. J. Heidrich

IS Development Dr. Marcus Trapp

IS Quality Assurance Dr. J. Dörr

Process Compl. & Imp. R. Van Lengen

Slide 17

Automotive & Transport

Automation & Plant Eng

Medical Systems

Information Systems

eGovernment

Health / Energy Mgt.

Fraunhofer IESE

Open Systems with QoS guarantees

Quantitative Management with

GQM+Strategies

Go Mobile

Dieter Rombach


Slide 18


18

ES: Overview of Empirical Results

Method Result Publications

AcES �35% reduction of implementation and

testing effort at same quality level

� ICSR 2008

AcES/RATE,

SAVE

�60% less time needed for architectural

analysis if architectures are visualized

appropriately

� EMSE 2008

SAVE-

Life

�60% fewer architecture violations if

developers are getting live feedback on

their architectural compliance

� PhD Knodel 2010

AcES �Architecture-compliant implementation

reduces development effort by 50%

� PhD Knodel 2010

Dieter Rombach


Slide 19


19

ES: Overview of Empirical Results


PuLSE �Strategic reuse program increases reuse

level by 50%

�Architectural divergences decreased

from 17% to 1%

� ArQuE 02.09

�CSMR 2008

PuLSE �With SPL approach, productivity has

tripled

�# of quality problems has been reduced

to 1/5

� Ricoh 2010

PuLSE-EM �27% less effort on average for

configuration management in a product

line

� IWPSE-EVOL 2009

20

MPI: Overview of Empirical Results


CoBRA Cost

Estimation

�Best-in-class estimation error between 5

and 20% initially

� ICSE 1998 & 2006

�METRICS 2002

�PhD Trendowicz 2008

HyDEEP

Defect

Prediction

�Defect prediction accuracy ~30%

(normally 40-150%)

�Effort for estimation model < 1.5 PD /

expert

� ICSE 2010

� Journal ESE 2010

Specula PM

Dashboards

�Earlier detection of plan deviations: 25-

83%

�More complete detection of plan

deviations: 17-63% more

�Overall effort (including training and data

collection): 10%

�ESEM 2007 & 2008

�PROFES 2008

�PhD Heidrich 2008

21

MPI: Overview of Empirical Results


Defect Flow

Models &

Inspections

�More reliable defect classification:

Kappa 0.65-079 (substantial)

�Detect the defects more locally, e.g. 72% to

100% of analysis defects are detected in the

origination phase, etc.

�Substantial rework reductions up to 90%

�METRICS 2005

�METRIKON 2007

�EuroMICRO 2009

Aggrega-tion

of Empirical

Studies

�Current (unsystematic) summaries often lead to

wrong conclusions

�PBR: 50% of assumptions have proven to be

wrong; 50% could be phrased more

accurately

�Complexity models: 25% of assumptions

have proven to be wrong

�ESEM 2009

�METRIKON 2010

22

IS: Overview of Empirical Results


Satisfy (IESE

NFR Method

part)

�Method leads to almost 100% measurable

NFRs

�Method resulted in ROI values between 2

and 17!

�Method resulted in more complete NFRs (up

to 622% more relevant NFRs identified)

�PhD Dörr, 2010

�RE’ 05

�…

GoMobile

Product (new)

�Half-day usability walkthroughs of iPhone

business apps (performed by two usability

experts) can bring up more than 40 usability

issues, with more than 25% of them being

critical for the acceptance of the app.

�Not yet published

Dieter Rombach


Slide 23



• There exist more models than we typically recognize



• Fraunhofer Institute for Experimental Software Engineering (IESE) maintains an Experience Base

• There exists a large body of empirical facts in our community at large

- e.g., Top Ten (Barry Boehm)

- e.g., Endres/Rombach, Addison, 2003

� inspections

� design principles

Dieter Rombach


Slide 24

Beijing, China, 26-27 April 2011Fraunhofer IESE Series

Editorial Board includes- Dieter Rombach

-Peter Liggesmeyer

-Dave Parnas- Ian Sommerville - Marv Zelkowitz

Handbook capturing existingbody of knowledge

Students can learnabout existing body of knowledge

Practitioners can avoid negliganceof due dilligance

Dieter Rombach


Slide 25


Requirements• Requirements deficiencies are the prime source of project failures (L1)

- Source: Robert Glass [Glas98] et al

- Most defects (> 50%) stem from requirements

- Requirements defects (if not removed quickly) trigger follow-up defects in later activities

Possible solutions:

- early inspections

- formal specs & validation early on

- other forms of prototyping & validation early on

- reuse of requirements docs from similar projects

- etc.

• Defects are most frequent during requirements and design activities and are more

expensive the later they are removed (L2)

- Source: Barry Boehm [Boeh 75] et al

- >80% of defects are caused up-stream (req, design)

- Removal delay is expensive (e.g., factor 10 per phase delay)

It would be nice

to have effect measures

for these possible solutions!

Dieter Rombach


Slide 26


Inspections

• Inspections significantly increase productivity, quality and project stability (L17)

- Source: Mike Fagan [Faga76, Faga86]

- Early defect detection increases quality (no follow-up defects, testing of clean code at the end � quality certification)

- Early defect detection increases productivity (less rework, lower cost per defect)

- Early defect detection increases project stability (better plannable due to fewer rework exceptions)

• Effectiveness of inspections is rather independentof its organizational form (i.e. process), but depends on reading technique (L18)

- Source: Ross Jeffery [Jeff90]

• Perspective-based inspections are highly effective and efficient (L19)

- Source: Victor Basili [Bas96c, Shull00]]

- Best suited for non-formal documents

2.4 Existing Laws

Dieter Rombach


Slide 27



• There exist more models than we typically recognize



• Fraunhofer Institute for Experimental Software Engineering (IESE) maintains an Experience Base

• There exists a large body of empirical facts in our community at large

(e.g., Endres/Rombach, Addison, 2003)

- inspections

- design principles

• More studies need to be done

- repeat (with variation)

- generalize

and results need to be organized and maintained !!!

Dieter Rombach


Slide 28


TOWARDS A SOFTWARE ENGINEERING DISCIPLINE

How do we have to structure our discipline of SE in order to include

the development & usage of empirical models?

•View “empirical evaluation” as an integral part of our discipline

Dieter Rombach


Slide 29



Requires

•Measurement

•Empirical studies

•Model Building

•Model based project

management & governance

NotationsConstruction

Technology

Process

Technology

Empirical Evaluation

Software Engineering

Dieter Rombach


Slide 30






•Methodological basis exists

Dieter Rombach


Slide 31


• Methodological foundation for empirical studies exists and includes approaches for

� relating goals to measurements (GQM, GQM+Strategies)

� piggy-bagging empirical studies on real projects (QIP)

� organizing empirical observations for reuse (EF)

� specific activities such as experimental design, data analysis, interpretation

Methodological Basis for Empirical Studies

Dieter Rombach


Slide 32







•Science vs. engineering

Dieter Rombach


Slide 33


Structure of Traditional Engineering Disciplines

• Engineering practices

• Models

• Pysical Laws & (Continuous) Mathematics

Electrical / Mech Engineering

Physics Mathematics

Models

Dieter Rombach


Slide 34


Possible Structure of a Software Engineering Discipline

Electrical/Mech

Engineering

Physics

Software

Engineering

Computer

ScienceMathematics

Systems Engineering

Phys.Models Emp. Models

Dieter Rombach


Slide 35








•Empirical studies in science & engineering

Dieter Rombach


Slide 36


Possible Structure of a Software Engineering Discipline

Electrical/Mech

Engineering

Physics

Software

Engineering

Computer

ScienceMathematics

Systems EngineeringEmpirical Studies?

Empirical Models?

Experiments

Case Studies

Projects

Org‘s / DisciplinePhys.Models Emp. Models

Dieter Rombach


Slide 37








•Empirical studies in science & engineering

•Towards a theory

Dieter Rombach


Slide 38



• Basic Models

• Model aggregation (PhD Marcus Ciolkowski)

- Identical results for identical C reduce risk (var)

- Identical results for non-identical C may increase risk (var)

� Consider splitting into 2 model lines (2 partial

functions)! � Barry Boehm

- Different results suggest new HC

Q/P/T (A1) == f (A2,C,HC)<var>

Dieter Rombach


Slide 39



How does the discipline benefit?

•Research

- We can produce challengeable results like other scientists!

Dieter Rombach


Slide 40


Studies (IESE Research)

Problem Stmt ( SoP)

with Improvement Hyp.

Solution Stmt ( SoR)

with Improvement Hyp.

Method

or Tool

Research

Technical Solution

Emp. Testing of

solution hypothesis

(typically: experiments)

Emp. Testing of

Problem hypothesis

(typically: case studies)

?

?

Dieter Rombach


Slide 41


Studies (IESE Research) – An Example

Reduction of Cycle Time

(minus 20%)

Inspections of req. (PBR)

(>80% of req defects det)

Research

Technical Solution

(PBR for reqs)

Controlled experiment

(compare with CBR,

vary exp, type of sw)

Case study in industry

(user accept, overall effect)

Dieter Rombach


Slide 42



How does the discipline benefit?

•Research

- We can produce challengeable results like other scientists!

•Education

- We can teach basic laws (!only qualitatively true!)!

•Practice

- We can expect due diligence compliance!

Dieter Rombach


Slide 43


• Challengeable Software Engineering research results MUST include empirical evidence!

• Studies must be

- Repeated (to increase trust)

- Varied (to enlarge scope)

• Individual empirical evidence (observations) must be aggregated to produce “laws” (and eventually theories)

- Panels of experts decide on qualification of empirical models as laws!

FUTURE CHALLENGES (More empirical evidence �� laws)

Dieter Rombach


Slide 44


• Existing Bodies of Evidence are limited

- Research Organizations (like IESE’s Experience Base)

� Focus on experiences wrt. new notations, methods & processes

- Development Organizations

� Focus on good practice notations, methods & processes for development

• Fraunhofer IESE wants to support the establishment of a living Community Experience Base

- Community at large (different domains), e.g.

� Springer may host Experience Base

� IESE is responsible for content management

� Individual areas will be chaired by community experts

- Collaborators are welcome!

FUTURE CHALLENGES (Building a Body of Knowledge)

Dieter Rombach


Slide 45


Congratulations on your 75th Birthday! Thanks for all your contributions! All the best for your future!

Empirical Software Engineering Models · CSMR 2008 PuLSE With SPL approach, productivity has tripled # of quality problems has been reduced to 1/5 ... Handbook capturing existing

Documents