Software Systems as Cities: a Controlled Experiment

REVEAL @ Faculty of InformaticsUniversity of Lugano

Switzerland

Richard Wettel, Michele LanzaPLEIAD @ DCC

University of ChileChile

Romain Robbes

Software Systems as Cities:A Controlled Experiment

Software Systems as Cities

City Metaphor

VISSOFT 2007

City Metaphor

class building

package district

VISSOFT 2007

City Metaphor

class building

package district

VISSOFT 2007

City Metaphor

class building

package district

nesting level color

VISSOFT 2007

City Metaphor

class building

package district

number of methods (NOM) height

number of attributes (NOA) base size

number of lines of code (LOC) color

nesting level color

VISSOFT 2007

Program ComprehensionArgoUMLArgoUMLLOC 136,325

ICPC 2007

Program Comprehension

skyscraperFacadeMDRImplFacadeMDRImpl

NOA 3NOM 349LOC 3,413

office buildingCPPParserCPPParser

NOA 85NOM 204LOC 9,111

parking lotJavaTokenTypesJavaTokenTypes

NOA 173NOM 0LOC 0

house PropPanelEventPropPanelEvent

NOA 2NOM 3LOC 37

Program Comprehension

ICPC 2007

Design Quality Assessment

ArgoUML classesArgoUML classesArgoUML classes

brain class 8

god class 30

god + brain 6

data class 17

unaffected 1,715SoftVis 2008

disharmony map

System Evolution Analysis

time traveling

WCRE 2008


ArgoUML8 major releases6 years

time traveling

WCRE 2008

time


ArgoUML8 major releases6 years

time traveling

WCRE 2008

time

http://codecity.inf.usi.ch

implemented in Smalltalk

ICSE 2008 tool demo

usefulIs it ?

A Controlled Experiment

Design

State of the art?technical report 2010

Design desiderata1 Avoid comparing using a technique against not using it.

2 Involve participants from the industry.

3 Provide a not-so-short tutorial of the experimental tool to the participants.

4 Avoid, whenever possible, giving the tutorial right before the experiment.

5 Use the tutorial to cover both the research behind the approach and the tool.

6 Find a set of relevant tasks.

7 Choose real object systems that are relevant for the tasks.

8 Include more than one object system in the design.

9 Provide the same data to all participants.

10 Limit the amount of time allowed for solving each task.

11 Provide all the details needed to make the experiment replicable.

12 Report results on individual tasks.

13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.

14 Take into account the possible wide range of experience level of the participants.

Design desiderata1 Avoid comparing using a technique against not using it.

2 Involve participants from the industry.

3 Provide a not-so-short tutorial of the experimental tool to the participants.

4 Avoid, whenever possible, giving the tutorial right before the experiment.

5 Use the tutorial to cover both the research behind the approach and the tool.

6 Find a set of relevant tasks.

7 Choose real object systems that are relevant for the tasks.

8 Include more than one object system in the design.

9 Provide the same data to all participants.

10 Limit the amount of time allowed for solving each task.

11 Provide all the details needed to make the experiment replicable.

12 Report results on individual tasks.

13 Include tasks on which the expected result is not always to the advantage of the tool being evaluated.

14 Take into account the possible wide range of experience level of the participants.

Finding a baseline

1. program comprehension

2. design quality assessment

3. system evolution analysis

Finding a baseline




Finding a baseline




Finding a baseline



Tasks

A1 Identity the convention used in the system to organize unit tests.

A2.1&A2.2

What is the spread of term T in the name of the classes, their attributes and methods?

A3 Evaluate the change impact of class C, in terms of intensity and dispersion.

A4.1 Find the three classes with the highest number of methods.

A4.2Find the three classes with the highest average number of lines of code per method.

Tasksprogram comprehension 6


A2.1&A2.2





Tasksprogram comprehension

design quality assessment 4

6

B1.1 Identify the package with the highest percentage of god classes.

B1.2 Identify the god class with the largest number of methods.

B2.1Identify the dominant (affecting the highest number of classes) class-level design problem.

B2.2 Write an overview of the class-level design problems in the system.


A2.1&A2.2





Tasksprogram comprehension

design quality assessment 4

6





5

TasksA1 Identity the convention used in the system to organize unit tests.

A2.1&A2.2









quantitative

qualitative 1

98

Main research questions


Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?1


Does the use of CodeCity increase the correctness of the solutions to program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?

Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?2

1

Variables of the experiment


dependentcorrectnesscorrectnesscorrectnesscorrectnesscorrectness

dependentcompletion timecompletion timecompletion timecompletion timecompletion time

independenttool CodeCityCodeCityCodeCityCodeCity

independenttool Eclipse + ExcelEclipse + ExcelEclipse + ExcelEclipse + Excelindependentobject system sizeobject system sizeobject system sizeobject system size mediumindependentobject system sizeobject system sizeobject system sizeobject system size large

controlledexperience levelexperience levelexperience level beginnerbeginner

controlledexperience levelexperience levelexperience level advancedadvancedcontrolledbackgroundbackground academiaacademiaacademiacontrolledbackgroundbackground industryindustryindustry






















FindBugs

1,320 classes

93,310 LOC

Azureus

4,656 classes

454,387 LOC















The experiment’s design

between-subjectsrandomized-block

The experiment’s design

Tool

CodeCityT1 large

SizeTool

CodeCity

T2 mediumSizeTool

Ecl+ExclT3 large

SizeTool

Ecl+Excl

T4 medium

Size


The experiment’s designbackground academiaacademia industryindustry

experience beginner advanced beginner advanced

B1 B2 B3 B4

Tool

CodeCityT1 large

SizeTool

CodeCity

T2 mediumSizeTool

Ecl+ExclT3 large

SizeTool

Ecl+Excl

T4 medium

Size


The experiment’s designbackground academiaacademia industryindustry

experience beginner advanced beginner advanced

B1 B2 B3 B4

Tool

CodeCityT1 large

SizeTool

CodeCity

T2 mediumSizeTool

Ecl+ExclT3 large

SizeTool

Ecl+Excl

T4 medium

Size


Execution

Experimental runs

Experimental runs

timeday 1

training session(1 hour)

Experimental runs

e1

c1

timeday 1


experiment session(2 hours)

Experimental runs

e1

c1

e2

c2

timeday 1 day 2



Experimental runs

e1

c1

e2

c2

e3

timeday 1 day 2 day 3



Experimental runs

e1

c1

e2

c2

e3

c4

timeday 1 day 2 day 3 day 4



Testing the waters20092009

November December

18 24 25 2 9

Lugano111

31

11

Timeline of the experiment20092009 2010201020102010

November December January February ... April

18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14

Bern

Bologna

Antwerp

Lugano111

31

11

21

1 1 13

11 1

61

56

46

1

Timeline of the experiment20092009 2010201020102010

November December January February ... April

18 24 25 2 9 21 28 5 8 14 28 18 22 24 25 14

Bern

Bologna

Antwerp

Lugano111

31

11

21

1 1 13

11 1

61

56

46

1

remotesessions

remotesession

Treatments and subjects

academiaacademia industry

beginner advanced advanced

CodeCity

large 2 2 6 10CodeCity

medium 3 2 7 12

Ecl+Excl

large 2 3 3 8Ecl+Excl

medium 2 5 4 11

9 12 20 41

Collecting raw data

Collecting raw data

solution

Collecting raw datacompletion time

Controlling time

Controlling time

common time

Controlling time

info on subjects

common time

Name (Task): Remaining time

Assessing correctness1 T2: Findbugs, analyzed with CodeCity

A1

Dispersed. [1pt]

A2.1

Localized [0.5pts]in package edu.umd.cs.findbugs.detect [0.5pts].

A2.2

Dispersedin the following (max. 5) packages [0.2pts for each]:

• edu.umd.cs.findbugs

• edu.umd.cs.findbugs.anttask

• edu.umd.cs.findbugs.ba

• edu.umd.cs.findbugs.ba.deref

• edu.umd.cs.findbugs.ba.jsr305

• edu.umd.cs.findbugs.ba.npe

• edu.umd.cs.findbugs.ba.vna

• edu.umd.cs.findbugs.bcel

• edu.umd.cs.findbugs.classfile

• edu.umd.cs.findbugs.classfile.analysis

• edu.umd.cs.findbugs.classfile.engine

• edu.umd.cs.findbugs.classfile.impl

• edu.umd.cs.findbugs.cloud

• edu.umd.cs.findbugs.cloud.db

• edu.umd.cs.findbugs.detect

• edu.umd.cs.findbugs.gui

• edu.umd.cs.findbugs.gui2

• edu.umd.cs.findbugs.jaif

• edu.umd.cs.findbugs.model

• edu.umd.cs.findbugs.visitclass

• edu.umd.cs.findbugs.workflow

A3: Impact Analysis

Multiple locations.There are 40/41 [0.5pts] classesdefined in the following 3 packages [1/6pts for each]:




A4.1

The 3 classes with the highest number of methods are [ 13pts each correctly placed and 16pts each misplaced]:

1. class AbstractFrameModelingVisitordefined in package edu.umd.cs.findbugs.bacontains 195 methods;

2. class MainFramedefined in package edu.umd.cs.findbugs.gui2contains 119 methods;

3. class BugInstancedefined in package edu.umd.cs.findbugscontains 118 methodsorclass TypeFrameModelingVisitordefined in package edu.umd.cs.findbugs.ba.typecontains 118 methods.

A4.2

The 3 classes with the highest average number of lines of code per method are [ 13pts each correctly placed and 16pts each

misplaced]:

1. class DefaultNullnessAnnotationsdefined in package edu.umd.cs.findbugs.bahas an average of 124 lines of code per method;

2. class DBCloud.PopulateBugsdefined in package edu.umd.cs.findbugs.cloud.dbhas an average of 114.5 lines of code per method;

3. class BytecodeScannerdefined in package edu.umd.cs.findbugs.bahas an average of 80.75 lines of code per method.

B1.1

The package with the highest percentage of god classes in the system isedu.umd.cs.findbugs.ba.deref [0.8pts]which contains 1 [0.1pts] god classesout of a total of 3 [0.1pts] classes.

B1.2

The god class containing the largest number of methods in the system isclass MainFrame [0.8pts]defined in package edu.umd.cs.findbugs.gui2 [0.1pts]which contains 119 [0.1pts] methods.

B2.1

The dominant class-level design problem isDataClass [0.5pts]which affects a number of 67 [0.5pts] classes.

oracles

Assessing correctness1 T2: Findbugs, analyzed with CodeCity

A1

Dispersed. [1pt]

A2.1

Localized [0.5pts]in package edu.umd.cs.findbugs.detect [0.5pts].

A2.2

Dispersedin the following (max. 5) packages [0.2pts for each]:


• edu.umd.cs.findbugs.anttask

• edu.umd.cs.findbugs.ba

• edu.umd.cs.findbugs.ba.deref

• edu.umd.cs.findbugs.ba.jsr305

• edu.umd.cs.findbugs.ba.npe

• edu.umd.cs.findbugs.ba.vna


• edu.umd.cs.findbugs.classfile

• edu.umd.cs.findbugs.classfile.analysis

• edu.umd.cs.findbugs.classfile.engine

• edu.umd.cs.findbugs.classfile.impl

• edu.umd.cs.findbugs.cloud

• edu.umd.cs.findbugs.cloud.db


• edu.umd.cs.findbugs.gui

• edu.umd.cs.findbugs.gui2

• edu.umd.cs.findbugs.jaif

• edu.umd.cs.findbugs.model

• edu.umd.cs.findbugs.visitclass

• edu.umd.cs.findbugs.workflow

A3: Impact Analysis

Multiple locations.There are 40/41 [0.5pts] classesdefined in the following 3 packages [1/6pts for each]:




A4.1

The 3 classes with the highest number of methods are [ 13pts each correctly placed and 16pts each misplaced]:

1. class AbstractFrameModelingVisitordefined in package edu.umd.cs.findbugs.bacontains 195 methods;

2. class MainFramedefined in package edu.umd.cs.findbugs.gui2contains 119 methods;

3. class BugInstancedefined in package edu.umd.cs.findbugscontains 118 methodsorclass TypeFrameModelingVisitordefined in package edu.umd.cs.findbugs.ba.typecontains 118 methods.

A4.2

The 3 classes with the highest average number of lines of code per method are [ 13pts each correctly placed and 16pts each

misplaced]:

1. class DefaultNullnessAnnotationsdefined in package edu.umd.cs.findbugs.bahas an average of 124 lines of code per method;

2. class DBCloud.PopulateBugsdefined in package edu.umd.cs.findbugs.cloud.dbhas an average of 114.5 lines of code per method;

3. class BytecodeScannerdefined in package edu.umd.cs.findbugs.bahas an average of 80.75 lines of code per method.

B1.1

The package with the highest percentage of god classes in the system isedu.umd.cs.findbugs.ba.deref [0.8pts]which contains 1 [0.1pts] god classesout of a total of 3 [0.1pts] classes.

B1.2

The god class containing the largest number of methods in the system isclass MainFrame [0.8pts]defined in package edu.umd.cs.findbugs.gui2 [0.1pts]which contains 119 [0.1pts] methods.

B2.1

The dominant class-level design problem isDataClass [0.5pts]which affects a number of 67 [0.5pts] classes.

oracles

blinding

Results

Statistical test

two-way analysis of variance (ANOVA)95% confidence interval

0

3

4

5

6

7

8

medium large

2

1

Correctness


Ecl+ExclCodeCity

0

3

4

5

6

7

8

medium large

2

1

Correctness


24.26%more correct with CodeCity

large effect size (d=0.89)

Ecl+ExclCodeCity

Correctness

24.26%more correct with CodeCity


large effect size (d=0.89)

2

3

4

5

6

7

8

Ecl+ExclCodeCity

medium large

Completion time

Does the use of CodeCity reduce the time needed to solve program comprehension tasks, compared to non-visual exploration tools, regardless of the object system size?

Ecl+ExclCodeCity

0

30

40

50

60

medium large

20

10

Completion time


12.01%less time with CodeCity

moderate effect size (d=0.63)

Ecl+ExclCodeCity

0

30

40

50

60

medium large

20

10

Completion time

12.01%less time with CodeCity


moderate effect size (d=0.63)

Ecl+ExclCodeCity

20

30

40

50

60

medium large

after the first round

CodeCity

vs

Ecl+Excl+24% correctness-12% completion time

Replicability193

A.5 Data

Code

Treatment

Blocking Criteria

Number ToolSystem size Background Experience

IA011 CodeCity

largeindustry

advanced

IA021 CodeCity

largeindustry

advanced

IA031 CodeCity

largeindustry

advanced

IA041 CodeCity

largeindustry

advanced

AB011 CodeCity

largeacademy

beginner

AB021 CodeCity

largeacademy

beginner

IA051 CodeCity

largeindustry

advanced

AA011 CodeCity

largeacademy

advanced

AA021 CodeCity

largeacademy

advanced

IA061 CodeCity

largeindustry

advanced

IA072 CodeCity

mediumindustry

advanced

IA082 CodeCity

mediumindustry

advanced

IA092 CodeCity

mediumindustry

advanced

AB032 CodeCity

mediumacademy

beginner

IA102 CodeCity

mediumindustry

advanced

IA112 CodeCity

mediumindustry

advanced

IA122 CodeCity

mediumindustry

advanced

AB042 CodeCity

mediumacademy

beginner

AA032 CodeCity

mediumacademy

advanced

AB052 CodeCity

mediumacademy

beginner

AA042 CodeCity

mediumacademy

advanced

IA132 CodeCity

mediumindustry

advanced

IA143 Ecl+Excl large

industryadvanced

AB063 Ecl+Excl large

academybeginner

AB073 Ecl+Excl large

academybeginner

AA053 Ecl+Excl large

academyadvanced


academyadvanced


academyadvanced


industryadvanced


industryadvanced

IA014 Ecl+Excl medium

industryadvanced


industryadvanced


industryadvanced

AB084 Ecl+Excl medium

academybeginner

AB094 Ecl+Excl medium

academybeginner

AA104 Ecl+Excl medium

academyadvanced


academyadvanced


academyadvanced


academyadvanced


academyadvanced


industryadvanced

Table A.2. The assignment of the subjects to treatments and blocks

174

A.4 Task Solution OraclesA.4.1 T1: Azureus, analyzed with CodeCityA1

EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser

[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correct

answer, the answer is completely wrong.A2.1

Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:• com.aelitis.azureus.core• com.aelitis.azureus.core.content• com.aelitis.azureus.core.download• com.aelitis.azureus.core.impl• com.aelitis.azureus.core.lws• com.aelitis.azureus.core.peermanager.peerdb• com.aelitis.azureus.core.stats

• com.aelitis.azureus.core.torrent• com.aelitis.azureus.plugins.net.buddy• com.aelitis.azureus.plugins.net.buddy.swt• com.aelitis.azureus.plugins.net.buddy.tracker• com.aelitis.azureus.plugins.removerules• com.aelitis.azureus.plugins.sharing.hoster• com.aelitis.azureus.plugins.startstoprules.defaultplugin• com.aelitis.azureus.plugins.tracker.dht

• com.aelitis.azureus.plugins.tracker.local• com.aelitis.azureus.plugins.tracker.peerauth• com.aelitis.azureus.ui.swt.content.columns• com.aelitis.azureus.ui.swt.shells.main• com.aelitis.azureus.util

174

A.4 Task Solution OraclesA.4.1 T1: Azureus, analyzed with CodeCityA1

EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser

[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.

A2.1

Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:• com.aelitis.azureus.core• com.aelitis.azureus.core.content• com.aelitis.azureus.core.download• com.aelitis.azureus.core.impl• com.aelitis.azureus.core.lws• com.aelitis.azureus.core.peermanager.peerdb• com.aelitis.azureus.core.stats• com.aelitis.azureus.core.torrent• com.aelitis.azureus.plugins.net.buddy• com.aelitis.azureus.plugins.net.buddy.swt• com.aelitis.azureus.plugins.net.buddy.tracker• com.aelitis.azureus.plugins.removerules• com.aelitis.azureus.plugins.sharing.hoster• com.aelitis.azureus.plugins.startstoprules.defaultplugin• com.aelitis.azureus.plugins.tracker.dht

• com.aelitis.azureus.plugins.tracker.local• com.aelitis.azureus.plugins.tracker.peerauth• com.aelitis.azureus.ui.swt.content.columns• com.aelitis.azureus.ui.swt.shells.main• com.aelitis.azureus.util

174 A.4 Task Solution Oracles

A.4.1 T1: Azureus, analyzed with CodeCityA1

EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.

A2.1

Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:

• com.aelitis.azureus.core

• com.aelitis.azureus.core.content

• com.aelitis.azureus.core.download

• com.aelitis.azureus.core.impl

• com.aelitis.azureus.core.lws

• com.aelitis.azureus.core.peermanager.peerdb

• com.aelitis.azureus.core.stats

• com.aelitis.azureus.core.torrent

• com.aelitis.azureus.plugins.net.buddy

• com.aelitis.azureus.plugins.net.buddy.swt

• com.aelitis.azureus.plugins.net.buddy.tracker

• com.aelitis.azureus.plugins.removerules

• com.aelitis.azureus.plugins.sharing.hoster

• com.aelitis.azureus.plugins.startstoprules.defaultplugin

• com.aelitis.azureus.plugins.tracker.dht

• com.aelitis.azureus.plugins.tracker.local

• com.aelitis.azureus.plugins.tracker.peerauth

• com.aelitis.azureus.ui.swt.content.columns

• com.aelitis.azureus.ui.swt.shells.main

• com.aelitis.azureus.util

Introduction

The aim of this experiment is to compare tool efficiency in supporting software

practitioners analyzing medium to large-scale software systems.You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.You are given maximum 100 minutes for solving 10 tasks (10 minutes per task). You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once

after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order

to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices

check the most appropriate answer and provide additional information, if

requested.

The experiment is concluded with a short debriefing questionnaire.

Thank you for participating in this experiment!! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes

CodeCity Experiment

Participant:

T1

Introduction

The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.You are given maximum 100 minutes for solving 10 tasks (10 minutes per task). You are asked:

• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once after completing all the tasks;• to announce the experimenter that you are moving on to another task, in order to reset your 10-minutes-per-task allocated timer;• not to return to earlier tasks because it affects the timing;• for each task, to fill in the required information. In the case of multiple choices check the most appropriate answer and provide additional information, if requested.


Thank you for participating in this experiment!! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes

CodeCity Experiment

Participant:

T1

Introduction

The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.

You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.

You are given maximum 100 minutes for solving 10 tasks (10 minutes per task).

You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once



check the most appropriate answer and provide additional information, if requested.


Thank you for participating in this experiment!

! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes

CodeCity Experiment

Participant:

T1

Introduction

The aim of this experiment is to compare tool efficiency in supporting software practitioners analyzing medium to large-scale software systems.

You will use CodeCity to analyze Azureus, a BitTorrent client written in Java.

You are given maximum 100 minutes for solving 10 tasks (10 minutes per task).

You are asked:• not to consult any other participant during the experiment;• to perform the tasks in the specified order;• to write down the current time each time before starting to read a task and once



check the most appropriate answer and provide additional information, if requested.


Thank you for participating in this experiment!

! ! ! ! Richard Wettel, Michele Lanza, Romain Robbes

CodeCity Experiment

Participant:

T1

Pre-experimentquestionnaire Handouts

174 A.4 Task Solution Oracles

A.4.1 T1: Azureus, analyzed with CodeCity

A1

EitherThere are no unit tests in the system [1pt],orCentralized in a single package hierarchy whose root is in org.gudy.azureus2.ui.console.multiuser[1pt]. Since there is only one test class (i.e., TestUserManager), if they don’t give the full correctanswer, the answer is completely wrong.

A2.1

Dispersed [0pts otherwise]in the following (max. 5) packages [0.2pts for each]:

• com.aelitis.azureus.core

• com.aelitis.azureus.core.content

• com.aelitis.azureus.core.download

• com.aelitis.azureus.core.impl

• com.aelitis.azureus.core.lws

• com.aelitis.azureus.core.peermanager.peerdb

• com.aelitis.azureus.core.stats

• com.aelitis.azureus.core.torrent

• com.aelitis.azureus.plugins.net.buddy

• com.aelitis.azureus.plugins.net.buddy.swt

• com.aelitis.azureus.plugins.net.buddy.tracker

• com.aelitis.azureus.plugins.removerules

• com.aelitis.azureus.plugins.sharing.hoster

• com.aelitis.azureus.plugins.startstoprules.defaultplugin

• com.aelitis.azureus.plugins.tracker.dht

• com.aelitis.azureus.plugins.tracker.local

• com.aelitis.azureus.plugins.tracker.peerauth

• com.aelitis.azureus.ui.swt.content.columns

• com.aelitis.azureus.ui.swt.shells.main

• com.aelitis.azureus.util

Oracles Data

196

A.5Data

Code

Diffi

culty

Leve

l Per

Task

Tim

ePr

essu

re

A1

A2.1

A2.2

A3

A4.1

A4.2

B1.1

B1.2

B2.1

B2.2

IA01

diffi

cult

inte

rmed

iate

inte

rmed

iate

simpl

e

simpl

e

impo

ssib

le

diffi

cult

trivi

al

trivi

al

inte

rmed

iate

fair

amou

nt

IA02

fair

amou

nt

IA03

diffi

cult

inte

rmed

iate

inte

rmed

iate

diffi

cult

inte

rmed

iate

diffi

cult

simpl

e

simpl

e

simpl

e

impo

ssib

le

fair

amou

nt

IA04

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

impo

ssib

le

diffi

cult

simpl

e

trivi

al

diffi

cult

too

muc

h

AB01

trivi

al

trivi

al

trivi

al

simpl

e

simpl

e

inte

rmed

iate

diffi

cult

simpl

e

simpl

e

inte

rmed

iate

very

little

AB02

trivi

al

trivi

al

trivi

al

trivi

al

simpl

e

diffi

cult

impo

ssib

le

simpl

e

trivi

al

simpl

e

fair

amou

nt

IA05

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

impo

ssib

le

diffi

cult

diffi

cult

trivi

al

simpl

e

not s

om

uch

AA01

inte

rmed

iate

trivi

al

simpl

e

inte

rmed

iate

trivi

al

impo

ssib

le

impo

ssib

le

trivi

al

simpl

e

trivi

al

very

little

AA02

trivi

al

trivi

al

trivi

al

simpl

e

simpl

e

impo

ssib

le

diffi

cult

simpl

e

trivi

al

inte

rmed

iate

not s

om

uch

IA06

inte

rmed

iate

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

simpl

e

simpl

e

diffi

cult

fair

amou

nt

IA07

diffi

cult

inte

rmed

iate

inte

rmed

iate

simpl

e

simpl

e

diffi

cult

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

not s

om

uch

IA08

trivi

al

trivi

al

simpl

e

inte

rmed

iate

inte

rmed

iate

impo

ssib

le

diffi

cult

inte

rmed

iate

simpl

e

inte

rmed

iate

very

little

IA09

diffi

cult

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

diffi

cult

inte

rmed

iate

simpl

e

trivi

al

diffi

cult

not s

om

uch

AB03

simpl

e

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

simpl

e

simpl

e

inte

rmed

iate

very

little

IA10

simpl

e

simpl

e

simpl

e

simpl

e

diffi

cult

impo

ssib

le

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

diffi

cult

not s

om

uch

IA11

simpl

e

simpl

e

simpl

e

diffi

cult

inte

rmed

iate

diffi

cult

diffi

cult

simpl

e

simpl

e

diffi

cult

fair

amou

nt

IA12

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

diffi

cult

trivi

al

simpl

e

simpl

e

diffi

cult

none

AB04

trivi

al

trivi

al

trivi

al

simpl

e

inte

rmed

iate

diffi

cult

trivi

al

trivi

al

simpl

e

diffi

cult

very

little

AA03

trivi

al

trivi

al

trivi

al

diffi

cult

trivi

al

impo

ssib

le

diffi

cult

trivi

al

trivi

al

simpl

e

not s

om

uch

AB05

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

impo

ssib

le

diffi

cult

inte

rmed

iate

trivi

al

simpl

e

not s

om

uch

AA04

inte

rmed

iate

inte

rmed

iate

diffi

cult

diffi

cult

inte

rmed

iate

impo

ssib

le

inte

rmed

iate

inte

rmed

iate

diffi

cult

impo

ssib

le

fair

amou

nt

IA13

trivi

al

simpl

e

simpl

e

simpl

e

inte

rmed

iate

diffi

cult

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

diffi

cult

none

IA14

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

fair

amou

nt

AB06

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

diffi

cult

impo

ssib

le

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

fair

amou

nt

AB07

inte

rmed

iate

simpl

e

simpl

e

impo

ssib

le

trivi

al

diffi

cult

simpl

e

inte

rmed

iate

inte

rmed

iate

diffi

cult

too

muc

h

AA05

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

diffi

cult

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

fair

amou

nt

AA06

inte

rmed

iate

simpl

e

simpl

e

diffi

cult

trivi

al

inte

rmed

iate

simpl

e

simpl

e

inte

rmed

iate

diffi

cult

very

little

AA07

trivi

al

inte

rmed

iate

inte

rmed

iate

trivi

al

trivi

al

simpl

e

inte

rmed

iate

trivi

al

simpl

e

inte

rmed

iate

none

IA15

diffi

cult

inte

rmed

iate

inte

rmed

iate

diffi

cult

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

impo

ssib

le

impo

ssib

le

fair

amou

nt

IA16

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

impo

ssib

le

simpl

e

simpl

e

inte

rmed

iate

simpl

e

inte

rmed

iate

diffi

cult

fair

amou

nt

IA01

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

diffi

cult

trivi

al

simpl

e

impo

ssib

le

trivi

al

simpl

e

diffi

cult

not s

om

uch

IA18

inte

rmed

iate

simpl

e

inte

rmed

iate

diffi

cult

simpl

e

simpl

e

diffi

cult

inte

rmed

iate

simpl

e

diffi

cult

fair

amou

nt

IA19

simpl

e

simpl

e

inte

rmed

iate

simpl

e

trivi

al

simpl

e

simpl

e

simpl

e

inte

rmed

iate

impo

ssib

le

fair

amou

nt

AB08

inte

rmed

iate

inte

rmed

iate

simpl

e

inte

rmed

iate

trivi

al

trivi

al

diffi

cult

trivi

al

simpl

e

inte

rmed

iate

not s

om

uch

AB09

trivi

al

simpl

e

simpl

e

impo

ssib

le

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

simpl

e

diffi

cult

fair

amou

nt

AA10

simpl

e

inte

rmed

iate

inte

rmed

iate

inte

rmed

iate

trivi

al

trivi

al

simpl

e

simpl

e

inte

rmed

iate

impo

ssib

le

very

little

AA11

diffi

cult

simpl

e

inte

rmed

iate

inte

rmed

iate

simpl

e

simpl

e

simpl

e

inte

rmed

iate

inte

rmed

iate

impo

ssib

le

not s

om

uch

AA12

simpl

e

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

simpl

e

simpl

e

inte

rmed

iate

diffi

cult

not s

om

uch

AA13

simpl

e

inte

rmed

iate

inte

rmed

iate

diffi

cult

simpl

e

simpl

e

simpl

e

simpl

e

simpl

e

inte

rmed

iate

not s

om

uch

AA14

trivi

al

trivi

al

trivi

al

impo

ssib

le

trivi

al

trivi

al

diffi

cult

trivi

al

diffi

cult

impo

ssib

le

not s

om

uch

IA20

inte

rmed

iate

simpl

e

simpl

e

inte

rmed

iate

trivi

al

trivi

al

trivi

al

trivi

al

trivi

al

diffi

cult

not s

om

uch

Tabl

eA.

5.Th

esu

bjec

ts’pe

rcei

ved

time

pres

sure

and

task

diffi

culty

192

A.5Data

Code

Age

Job

Posit

ion

Expe

rienc

e Leve

l

Numbe

r ofYe

ars

OOP

Java

Eclip

se

Rev.E

ng.

OOP

Java

Eclip

seRe

v.Eng

.

IA01

30

Develo

per

know

ledge

able

adva

nced

know

ledge

able

begin

ner

7–10

7–10

4–6

1–3

IA02

34

Develo

per

adva

nced

adva

nced

know

ledge

able

know

ledge

able

7–10

4–6

1–3

4–6

IA03

42

CTO, D

evelo

per

expe

rt

know

ledge

able

begin

ner

know

ledge

able

>10

1–3

1–3

>10

IA04

37

Develo

per

adva

nced

adva

nced

know

ledge

able

begin

ner

7–10

7–10

4–6

1–3

AB01

21

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

4–6

4–6

4–6

<1

AB02

21

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

1–3

1–3

1–3

<1

IA05

29

Consu

ltant

, Ph.D

. Stu

dent

expe

rt

begin

ner

begin

ner

know

ledge

able

7–10

7–10

4–6

4–6

AA01

26

Ph.D

. Stu

dent

expe

rt

expe

rt

know

ledge

able

begin

ner

>10

>10

1–3

<1

AA02

26

Ph.D

. Stu

dent

expe

rt

adva

nced

know

ledge

able

know

ledge

able

4–6

1–3

1–3

1–3

IA06

35

Head of

IT

expe

rt

expe

rt

adva

nced

adva

nced

>10

>10

4–6

7–10

IA07

27

Softw

are En

ginee

r

know

ledge

able

know

ledge

able

begin

ner

know

ledge

able

7–10

7–10

1–3

4–6

IA08

25

Softw

are En

ginee

r

know

ledge

able

adva

nced

know

ledge

able

begin

ner

4–6

4–6

1–3

<1

IA09

32

Develo

pmen

t Lead

er,Re

sear

cher

adva

nced

begin

ner

none

adva

nced

7–10

7–10

<1

4–6

AB03

28

Stud

ent

know

ledge

able

know

ledge

able

begin

ner

begin

ner

4–6

1–3

1–3

1–3

IA10

39

Proje

ctM

anag

er

expe

rt

adva

nced

know

ledge

able

know

ledge

able

>10

7–10

7–10

4–6

IA11

38

Consu

ltant

, Sys

temM

anag

er/A

nalys

t

know

ledge

able

begin

ner

begin

ner

adva

nced

7–10

7–10

1–3

7–10

IA12

34

Senio

r Java

Archite

ct

expe

rt

expe

rt

adva

nced

adva

nced

>10

>10

>10

>10

AB04

22

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

know

ledge

able

4–6

1–3

1–3

<1

AA03

22

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

7–10

4–6

4–6

<1

AB05

22

Mas

terSt

uden

t

adva

nced

adva

nced

know

ledge

able

begin

ner

4–6

1–3

1–3

1–3

AA04

29

Ph.D

. Stu

dent

adva

nced

adva

nced

know

ledge

able

begin

ner

4–6

4–6

1–3

<1

IA13

32

Consu

ltant

expe

rt

know

ledge

able

know

ledge

able

expe

rt

7–10

4–6

1–3

7–10

IA14

31

Softw

are Arch

itect

adva

nced

know

ledge

able

know

ledge

able

begin

ner

7–10

7–10

1–3

1–3

AB06

23

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

4–6

1–3

1–3

<1

AB07

23

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

4–6

1–3

1–3

<1

AA05

30

Ph.D

. Stu

dent

adva

nced

adva

nced

adva

nced

know

ledge

able

7–10

7–10

7–10

4–6

AA06

26

Ph.D

. Stu

dent

expe

rt

know

ledge

able

adva

nced

expe

rt

7–10

7–10

4–6

4–6

AA07

30

Ph.D

. Stu

dent

adva

nced

adva

nced

know

ledge

able

know

ledge

able

7–10

7–10

1–3

1–3

IA15

40

Proje

ctM

anag

er

expe

rt

expe

rt

adva

nced

adva

nced

>10

>10

7–10

4–6

IA16

39

Softw

are Arch

itect

adva

nced

adva

nced

know

ledge

able

know

ledge

able

4–6

4–6

4–6

1–3

IA01

30

Develo

per

know

ledge

able

adva

nced

know

ledge

able

begin

ner

7–10

7–10

4–6

1–3

IA17

27

Softw

are En

ginee

r

know

ledge

able

adva

nced

know

ledge

able

begin

ner

4–6

4–6

4–6

<1

IA19

39

Consu

ltant

, Pro

ject M

anag

er,Arch

itect

expe

rt

expe

rt

know

ledge

able

adva

nced

>10

7–10

7–10

4–6

AB08

21

Mas

terSt

uden

t

adva

nced

adva

nced

adva

nced

begin

ner

1–3

1–3

1–3

<1

AB09

23

Ph.D

. Stu

dent

adva

nced

adva

nced

adva

nced

know

ledge

able

4–6

1–3

1–3

1–3

AA10

24

Ph.D

. Stu

dent

adva

nced

adva

nced

adva

nced

adva

nced

4–6

4–6

4–6

1–3

AA11

23

Ph.D

. Stu

dent

adva

nced

adva

nced

adva

nced

know

ledge

able

4–6

4–6

4–6

1–3

AA12

52

Profe

ssor

expe

rt

adva

nced

know

ledge

able

adva

nced

>10

>10

4–6

>10

AA13

28

Ph.D

. Stu

dent

adva

nced

adva

nced

know

ledge

able

know

ledge

able

4–6

4–6

4–6

1–3

AA14

24

Mas

terSt

uden

t

expe

rt

expe

rt

expe

rt

know

ledge

able

4–6

4–6

4–6

1–3

IA20

36

Develo

per

adva

nced

expe

rt

adva

nced

begin

ner

>10

7–10

7–10

1–3

Table

A.1.Th

e subje

cts’ p

erson

alinf

ormati

on, c

luster

edby

treatm

ent c

ombin

ation

s

195

A.5 Data

Code

Completion Time Per Task TotalCompl. Time

A1 A2.1 A2.2 A3 A4.1 A4.2 B1.1 B1.2 B2.1 (excl. A4.2)

IA01 8.80 4.32 5.38 10.00 7.28 4.42 9.85 1.90 1.02 52.97 48.55

IA02 2.42 6.08 1.75 8.58 5.33 10.00 5.25 2.92 2.33 44.67 34.67

IA03 6.25 6.92 2.42 9.43 5.62 9.37 2.55 2.95 1.83 47.33 37.97

IA04 9.08 6.00 4.00 6.08 7.92 10.00 10.00 1.33 1.08 55.50 45.50

AB01 1.33 4.83 3.92 4.75 6.25 10.00 6.92 2.25 2.58 42.83 32.83

AB02 10.00 5.58 1.67 4.25 8.08 10.00 9.42 2.58 1.67 53.25 43.25

IA05 6.33 3.67 1.33 4.75 3.50 10.00 3.75 1.58 2.17 37.08 27.08

AA01 9.67 4.42 2.08 8.75 3.17 10.00 5.17 1.33 1.42 46.00 36.00

AA02 8.33 7.17 2.50 10.00 5.00 9.75 8.33 3.25 1.33 55.67 45.92

IA06 10.00 5.92 4.42 10.00 4.17 10.00 7.33 1.33 1.50 54.67 44.67

IA07 7.25 5.67 6.25 6.33 4.58 10.00 2.42 1.42 2.33 46.25 36.25

IA08 2.67 2.25 2.95 4.55 3.75 10.00 3.33 4.00 1.17 34.67 24.67

IA09 10.00 3.67 3.00 10.00 5.25 7.83 3.67 2.50 1.42 47.33 39.50

AB03 6.33 2.00 9.33 5.00 5.00 8.50 7.17 1.50 1.25 46.08 37.58

IA10 3.25 3.67 7.33 6.83 3.83 7.75 3.08 1.75 8.08 45.58 37.83

IA11 3.83 2.40 5.92 7.17 4.00 6.92 7.42 2.83 1.75 42.23 35.32

IA12 3.75 2.67 4.17 5.58 5.17 10.00 5.75 2.00 1.25 40.33 30.33

AB04 2.67 3.92 2.58 3.67 5.58 9.75 2.00 0.50 3.83 34.50 24.75

AA03 2.50 3.17 3.75 10.00 3.17 6.33 4.08 1.25 2.58 36.83 30.50

AB05 5.50 4.67 4.33 5.58 5.83 10.00 9.58 1.83 1.67 49.00 39.00

AA04 7.08 3.67 5.17 6.33 3.50 7.83 6.00 1.25 2.83 43.67 35.83

IA13 3.00 4.67 2.33 4.75 3.25 10.00 4.83 1.67 2.08 36.58 26.58

IA14 6.67 9.00 2.42 10.00 4.50 5.83 5.33 1.83 4.17 49.75 43.92

AB06 5.67 5.75 2.05 6.95 10.00 10.00 10.00 3.85 2.00 56.27 46.27

AB07 8.75 9.33 4.67 10.00 6.83 10.00 10.00 3.67 2.67 65.92 55.92

AA05 7.00 6.08 3.75 8.42 5.25 4.42 10.00 3.17 6.67 54.75 50.33

AA06 6.83 2.00 3.67 6.58 3.25 6.75 5.33 1.58 1.58 37.58 30.83

AA07 3.67 3.67 2.50 2.17 2.92 5.17 2.75 3.33 1.83 28.00 22.83

IA15 9.75 8.83 5.08 10.00 4.33 10.00 5.08 1.75 7.83 62.67 52.67

IA16 10.00 7.42 4.33 9.50 4.00 6.50 8.50 2.00 4.50 56.75 50.25

IA01 2.55 3.90 4.38 9.20 4.03 4.30 9.92 2.10 2.93 43.32 39.02

IA18 5.28 5.13 4.58 9.82 4.72 3.98 9.77 4.13 3.12 50.53 46.55

IA19 3.33 3.83 4.50 3.50 3.83 5.08 6.58 1.92 5.58 38.17 33.08

AB08 6.08 1.08 10.00 8.83 3.50 5.92 9.42 3.58 1.92 50.33 44.42

AB09 5.83 4.83 4.33 10.00 3.42 6.00 2.00 1.92 1.50 39.83 33.83

AA10 3.17 6.08 5.08 7.33 8.00 3.33 4.83 1.17 2.33 41.33 38.00

AA11 6.17 4.08 6.08 3.83 3.50 5.17 3.67 1.67 2.92 37.08 31.92

AA12 6.75 4.75 3.92 4.75 5.25 3.33 4.42 1.42 3.00 37.58 34.25

AA13 7.00 7.00 10.00 10.00 5.92 5.67 6.33 1.42 5.42 58.75 53.08

AA14 6.33 1.50 3.33 10.00 2.83 3.25 9.17 2.42 4.50 43.33 40.08

IA20 6.83 5.58 4.75 8.00 2.33 3.42 2.42 1.42 1.33 36.08 32.67

Table A.4. The subjects’ task completion time, in minutes

194 A.5 Data

CodeCorrectness Per Task

TotalCorrectness

A1 A2.1 A2.2 A3 A4.1 A4.2 B1.1 B1.2 B2.1 (excl. A4.2)IA01 0.00 1.00 1.00 1.00 1.00 0.00 0.00 1.00 1.00 6.00 6.00IA02 1.00 0.80 0.50 1.00 1.00 0.00 0.00 1.00 1.00 6.30 6.30IA03 0.00 0.00 1.00 0.00 1.00 0.00 0.00 1.00 0.00 3.00 3.00IA04 0.00 1.00 0.00 1.00 1.00 0.00 0.00 1.00 1.00 5.00 5.00AB01 0.00 1.00 1.00 0.80 1.00 0.00 0.00 1.00 1.00 5.80 5.80AB02 0.00 1.00 1.00 0.70 1.00 0.00 0.00 0.00 1.00 4.70 4.70IA05 0.00 1.00 1.00 0.20 1.00 0.00 0.00 1.00 1.00 5.20 5.20AA01 0.00 0.80 1.00 0.40 1.00 0.00 0.00 1.00 0.00 4.20 4.20AA02 0.00 1.00 1.00 0.00 1.00 0.00 1.00 1.00 1.00 6.00 6.00IA06 0.00 1.00 0.00 0.30 1.00 0.00 0.00 1.00 1.00 4.30 4.30IA07 1.00 1.00 1.00 0.17 1.00 0.00 0.00 1.00 1.00 6.17 6.17IA08 1.00 0.50 1.00 0.83 1.00 0.00 1.00 1.00 1.00 7.33 7.33IA09 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00AB03 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00IA10 1.00 1.00 1.00 0.17 1.00 0.00 0.00 1.00 1.00 6.17 6.17IA11 1.00 1.00 1.00 0.00 1.00 0.16 0.00 0.00 1.00 5.16 5.00IA12 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 8.00 8.00AB04 1.00 1.00 1.00 0.67 1.00 0.00 0.00 1.00 1.00 6.67 6.67AA03 1.00 1.00 0.80 0.33 1.00 0.16 0.00 1.00 1.00 6.29 6.13AB05 1.00 1.00 1.00 1.00 1.00 0.00 0.00 1.00 1.00 7.00 7.00AA04 1.00 1.00 1.00 0.50 1.00 0.00 0.00 1.00 1.00 6.50 6.50IA13 1.00 1.00 1.00 0.83 1.00 1.00 0.00 0.00 1.00 6.83 5.83IA14 0.00 0.80 0.33 0.30 1.00 0.67 0.00 1.00 1.00 5.10 4.43AB06 1.00 1.00 1.00 0.00 0.00 0.00 0.00 1.00 0.00 4.00 4.00AB07 0.00 0.60 0.67 0.00 0.00 0.00 0.00 0.00 1.00 2.27 2.27AA05 1.00 1.00 1.00 0.00 1.00 1.00 0.00 1.00 1.00 7.00 6.00AA06 0.00 1.00 0.00 0.00 1.00 1.00 0.00 0.00 1.00 4.00 3.00AA07 1.00 1.00 0.00 0.00 1.00 1.00 0.00 1.00 0.00 5.00 4.00IA15 1.00 0.80 0.00 0.00 1.00 1.00 0.00 1.00 0.00 4.80 3.80IA16 0.00 0.00 0.67 0.00 1.00 0.67 0.00 1.00 1.00 4.34 3.67IA01 1.00 1.00 1.00 0.50 1.00 1.00 0.00 1.00 1.00 7.50 6.50IA18 1.00 1.00 1.00 0.50 1.00 1.00 0.00 1.00 1.00 7.50 6.50IA19 0.00 1.00 0.60 0.00 1.00 1.00 0.00 1.00 0.00 4.60 3.60AB08 0.00 1.00 0.40 0.38 1.00 1.00 1.00 1.00 1.00 6.78 5.78AB09 1.00 1.00 0.60 0.50 1.00 1.00 0.00 1.00 1.00 7.10 6.10AA10 1.00 1.00 0.80 0.38 1.00 1.00 0.00 1.00 1.00 7.18 6.18AA11 1.00 1.00 0.80 0.00 1.00 1.00 0.00 1.00 1.00 6.80 5.80AA12 0.00 0.50 0.00 0.00 1.00 1.00 0.00 1.00 1.00 4.50 3.50AA13 0.00 1.00 0.00 0.25 1.00 1.00 0.00 1.00 1.00 5.25 4.25AA14 1.00 1.00 0.60 0.00 1.00 1.00 0.90 1.00 1.00 7.50 6.50IA20 0.00 1.00 1.00 0.38 1.00 1.00 0.00 1.00 1.00 6.38 5.38

Table A.3. The correctness of the subjects’ solutions to the tasks

technical report 2010

Conclusions

Summary

Summarylist ofdesigndesiderata


6 months4 locations41 subjects


6 months4 locations41 subjects

-12.01%

+24.26%correctness

completiontime

Take-away messageVisualizing software systems as

cities is a viable alternative to the state of the practice approaches for software exploration.

Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and

We thank

Radu Marinescu, Alberto Bacchelli, Mircea Lungu, Lile Hattori, Andrea Reber, Kristina Gulordava, Francesco Rigotti, Vitezslav Humpa, Luca Della Toffola, Marcin Nowak, Aida Hota, Alessio Böckmann, Mattia Filippi, Fabio Parentela, Enrico Baraldi, Giacomo Benvenuti, Andrea Bolognesi, Marco Vit, Marco Brizi, Alberto Brandolini, Renzo Borgatti, Michele Finelli, Yuri Valentini, Fausto Fabbioni, Lorenzo Bragaglia, Riccardo Contri, Fabrizio Pinto, Paolo Gruppioni, Pasquale Granato, Andrea Chiodoni, Mario Fusco, Andrea Polci, Gianfranco Tognana, Glenn Van Loon, Tom Pauwaert, Yorik Gerlo, Serge Demeyer, Quinten Soetens, Bart Smets, Dieter De Hen, Ahmed Lamkanfi, Wim Van Eynde, Nick Baetens, Matthias De Cock, Tim Molderez, Oscar Nierstrasz, Tudor Gîrba, Adrian Lienhard, Fabrizio Perin, Erwann Wernli, Lukas Renggli, Niko Schwarz, David Röthlisberger, Jorge Ressia, Camillo Bruni, Toon Verwaest, Adrian Kuhn, and

We thank

None of the participants has been harmed during the experiment.

http://creativecommons.org/licenses/by/3.0/

Richard Wettelhttp://www.inf.usi.ch/phd/wettel



http://www.inf.unisi.ch/phd/wettel

http://www.inf.unisi.ch/phd/wettel

Software Systems as Cities: a Controlled Experiment

Technology

individual tasks

set of relevant tasks

experimental tool

experiment replicable

tool demo

short tutorial

design desiderata

real object systems