Software Estimation Challenge 2021 - COSMIC Sizing

Software Estimation Challenge 2021Seabed Observatory Control System (SOCS)

Pieter Jansen, Michail Bozoudis, Jacek Pachocki, Mercedes Ripoll, Thomas Backes

9 November 2021

Brussels Sprouts

Agenda

1. COSMIC Function Points1. Non-functional requirements (NFR)2. Estimating approach3. COSMIC Function Point total count4. Standardized approach to account for incompleteness of the requirements

2. Software effort estimate approach1. Knowledge Base and Work Breakdown Structure (WBS)2. Sizing method for non-functional requirements3. Project parameters & assumptions4. Total COSMIC Function Points and development effort

3. Software cost & schedule output

4. Benchmarking

5. Environmental costs estimation (project CO2e footprint)

6. Extra charts & reports

| 2

1. COSMIC Function PointsCounting process overview

Raw requirements

Requirement ID

Req. Title

Description

Details

Enriched requirements

Requirement ID

Sub-requirements (token)

Work Package

Functional Process

Triggering Event

Functional User

Data Group

Functional count

Entries

Exits

Reads

Writes

Requirement ID

Sub-requirements (token)

We started from the provided list of raw requirements (both functional

and non functional).

We broken down & enriched the requirements with COSMIC elements

before we started counting

We associated entries, exits, reads and writes events with each sub-

requirements

| 3

COSMIC Function PointsScope and components

WorkPackage

User Interfaces

Operation Modes

Robot Guidance

Observation and Operation Monitoring

Observing Program Management

Robot Observatory

Control Room

Software boundary

SOCS software

Persistent storage

Entry

Exit

Read Write

Sensors & detectors (10)• Camera• Water current sensor• Environmental sensor• Robot position detectors (speed, heading, depth)• Control unit heat detector• Control unit fan• Control unit vibrations detector / accelerometer• Support actuators and EPH

Instruments (6)• ComCam• RC1, RC2, RC3, RC4, RC5

Humans (4)• Science and external users of the Observatory• Science operations staff• Marine Observatory engineering & technical

operations staff• Marine Observatory Managers

Functional users

External SW and libraries• Real time computers• Servers• Consoles• Switches• Fiber Optics• Disks

Data groups

science observations (incl. scheduling) data

maintenance data

work procedures and workflows data

issues log data

engineering, configuration and calibration data

real-time execution / operation data

environmental data

telemetry data

user provided data

quality data

Work Packages

| 4

COSMIC Function PointsExtract from the CFP table (functional requirements)

REQ WorkPackage FunctionalProcess TriggeringEvent FunctionalUser DataGroup Entry Exit Read Write

REQ-UI-1 User Interfaces Create & edit observing blocks on-the-fly need to define an observation Operators / Scientists / Observers science observations (incl. scheduling) data 8 8 2 2

REQ-UI-1 User Interfaces Reduce and analyze data in real-time need to reduce and analyse data in real-time Operators / Scientists / Observers all data groups 4 4 10 10

REQ-UI-1 User Interfaces Online observatory data reduction pipeline and tools

REQ-UI-1 User Interfaces Observe preparation, including configuring & calibrating instruments need to observe configuration and calibration of instrumentsOperators / Scientists / Observers engineering, configuration and calibration data 4 4

REQ-UI-1 User Interfaces Observe execution, onboard or remote locations need to observe execution Operators / Scientists / Observers real-time execution / operation data 4 4

REQ-UI-1 User InterfacesObserve real time situation awareness and environmental monitoring of

underwater conditionsneed to have situation awareness Operators / Scientists / Observers environmental data 4 4

REQ-UI-1 User InterfacesObserve real time situation awareness and environmental monitoring of

underwater conditionsneed to have situation awareness Operators / Scientists / Observers real-time execution / operation data 4 4

REQ-UI-1 User Interfaces Schedule time observations (Classical observing mode) need to schedule time observations Operators / Scientists / Observers science observations (incl. scheduling) data 1 1 1

REQ-UI-2 User Interfaces Perform engineering activities on instruments need to perform engineering activities on instrumentsOperators / Scientists / Observers engineering, configuration and calibration data 1 1

REQ-UI-2 User Interfaces Perform engineering activities on instruments initiation of engineering activity Instruments engineering, configuration and calibration data 6 6 1 1

REQ-UI-2 User Interfaces Perform maintenance (predictive, preventive, condition-based) need to perform maintenance Operators / Scientists / Observers maintenance data 1 1

REQ-UI-2 User Interfaces Perform maintenance (predictive, preventive, condition-based) initiation of a maintenance activity Instruments maintenance data 6 6 1 1

REQ-UI-2 User Interfaces Perform maintenance (predictive, preventive, condition-based) initiation of a maintenance activity Sensors and detectors maintenance data 10 10 1 1

REQ-UI-2 User Interfaces Perform maintenance (reactive) sensor or detector failure Sensors and detectors real-time execution / operation data 10 1 1

REQ-UI-2 User Interfaces Perform maintenance (reactive) instrument failure Instruments real-time execution / operation data 6 1 1

REQ-UI-2 User Interfaces Perform maintenance (reactive) failure indication Operators / Scientists / Observers maintenance data 4 4 1 1

REQ-UI-2 User Interfaces Perform sequencing need to perform sequencing on instruments Operators / Scientists / Observers engineering, configuration and calibration data 1 1

REQ-UI-2 User Interfaces Perform sequencing initiation of sequencing Instruments engineering, configuration and calibration data 6 6 1 1

REQ-UI-2 User Interfaces Control individual degrees of freedom of instruments need to control individual degrees of freedom of instrumentsOperators / Scientists / Observers engineering, configuration and calibration data 1 1

REQ-UI-2 User Interfaces Control individual degrees of freedom of instruments initiation of controlling degrees of freedom Instruments engineering, configuration and calibration data 6 6 1 1

REQ-UI-2 User Interfaces Calibrate subsystems, detectors, devices, etc. need to calibrate subsystems, detectors, devices, etc. Operators / Scientists / Observers engineering, configuration and calibration data 1 1

REQ-UI-2 User Interfaces Calibrate subsystems, detectors, devices, etc. initiation of calibration Instruments engineering, configuration and calibration data 6 6 1 1

REQ-UI-2 User Interfaces Calibrate subsystems, detectors, devices, etc. initiation of calibration Sensors and detectors engineering, configuration and calibration data 10 10 1 1

REQ-UI-2 User Interfaces Diagnose subsystems, detectors, devices, etc.

REQ-UI-2 User Interfaces Log work status, issues, solutions, etc. need to log work status, issues, solutions Operators / Scientists / Observers issues log data 4 4 1 1

REQ-UI-2 User Interfaces View work procedures need to view work procedures Operators / Scientists / Observers work procedures and workflows data 4 4 1

REQ-UI-2 User Interfaces Support science operation user interfaces

REQ-UI-2 User Interfaces Observe execution activities and workflow

REQ-UI-2 User Interfaces Sequence instruments

REQ-UI-2 User Interfaces Monitor instrument health need to monitor instrument health Instruments maintenance data 6 1

REQ-UI-2 User Interfaces Monitor instrument health need to monitor instrument health Operators / Scientists / Observers maintenance data 4 4 1

REQ-UI-2 User Interfaces Configure, calibrate, and operate instrument on science level

REQ-UI-2 User Interfaces Configure, calibrate, and operate instrument on science level

REQ-UI-2 User Interfaces Create science observation sequences need to sequence science observations Operators / Scientists / Observers science observations (incl. scheduling) data 4 4 1 1

REQ-UI-2 User Interfaces Log science operations need to log science operations Operators / Scientists / Observers science observations (incl. scheduling) data 4 4 1 1

REQ-UI-2 User Interfaces Assess and monitor quality need to assess and monitor quality Operators / Scientists / Observers quality data 4 4 1

| 5

COSMIC Function PointsRe-qualification from Non-Functional Requirement (NFR) to Functional User Requirement (FUR)

• Our Team identified a set of functional requirements within the provided list of non-functional requirements.

• We assumed that the following NFR’s will evolve into FUR’s as project progresses, but not yet flagged as FUR, given what we can read, this is mentioned in the requirements instruction. Presence of key words indicate for us a FUR and not a NFR (e.g. store).

• The size of the below requirements (flagged as NFR) was measured with the COSMIC method.

REQ-G3 # Configuration Capability The SOCS shall provide the capability to define and modify a set of configuration parameters of any software component.

REQ-G4 # Telemetry Capability The SOCS shall provide the capability to sample, display and correlate any signal with the resolution and frequency required to characterize the behavior of the signal.

REQ-G5 # Logging Capability The SOCS shall provide the capability to detect, store and notify the occurrence of relevant operation events.

REQ-G6a # Alarm Capability The SOCS shall provide the capability to detect, store, visualize, notify and manage alarm conditions for any critical parameters that affect function or performance.

REQ-G6b # Provide System-Wide Timestamping Capability The SOCS shall provide a system-wide, consistent, time stamp capability.

REQ-G7 # Telemetry Correlation The SOCS shall provide tools to correlate telemetry information with time-stamped data from controlled subsystems.

REQ-G8 # Telemetry Information Assembly The SOCS shall provide tools to assemble telemetry information from subsystems, specified by user, into multicolumn data table.

REQ-G9 # Continuous Monitoring of Subsystem Health The SOCS shall provide continuous performance, status and system health monitoring for any parameter that affects subsystem performance.

REQ-C5 # Provide Emergency Notifications in the Control Room The SOCS shall provide emergency alerts via both audio and visual notifications in the High Visibility Status Display and other visual cues (e.g. dedicated emergency status LED lights) around the room.

REQ-C6 # Provide Software Emergency Stop Capability The SOCS shall provide an emergency stop capability triggered by software.

| 6

COSMIC Function Points Estimating approach

Platform

Application

Acquisition Method

Dev. Method

Dev. Std.

Class

SEER DB(ISBSG)

All projects

Similar Projects

Entry

Exit

Write

Read

• SEER SEM draws data from thousands of completed projects and use cost estimating relationships to estimate cost & schedule.

• The effect each parameter from the knowledge base has on the project was synthesized and built into SEER.

• The main cost driver is the functional size (e.g. COSMIC function points).

| 7

COSMIC Function Points Robot Guidance CFP input into SEER SEM (example)

A. The CFPs per work package (in this example, Robot Guidance) are summed up.

B. CFPs are split between new and pre-existing functions. Assumption of 55% pre-existing because the development Team will implement the 3rd generation Research Control System.

C. The new and pre-existing function points are entered into SEER under the likely scenario.

REQ WorkPackage FunctionalProcess Entry Exit Read Write

REQ-RG-1 Robot Guidance

Provide the capability to fine-correct tracking errors in the seabed robot

main axes & facility instrument rotator(s) (GIR) using position feedback

from the guide sensors

10 10 1 1

REQ-RG-1 Robot Guidance

Provide the capability to fine-correct tracking errors in the seabed robot

main axes & facility instrument rotator(s) (GIR) using position feedback

from the guide sensors

4 4

REQ-RG-2 Robot GuidanceAccept guide signals from instruments and/or the AR (Adaptive Seabed

robot) system6

REQ-RG-2 Robot GuidanceAccept guide signals from instruments and/or the AR (Adaptive Seabed

robot) system10

REQ-RG-3 Robot Guidance

Provide a guided mode for linearly scanning an object at a specified

angle in the focal plane at a fixed rate and angle while continuously

collecting science data

4 4

REQ-RG-3 Robot Guidance

Provide a guided mode for linearly scanning an object at a specified

angle in the focal plane at a fixed rate and angle while continuously

collecting science data

10 1

REQ-RG-4 Robot GuidanceHave scan rates that are user selectable up to the maximum rate of 1

arcsec per second4 4

REQ-RG-4 Robot GuidanceHave scan rates that are user selectable up to the maximum rate of 1

arcsec per second10 10

REQ-RG-5 Robot Guidance Support scan distances up to the travel limits of the guiders

REQ-RG-5 Robot Guidance Support scan distances up to the travel limits of the guiders

TOTAL 58 32 1 2

TOTAL Entries

58

Exits32

Reads1

Writes2

New (45%)

En Ex R W26 14 0 1

Pre-existing (55%)

En Ex R W32 18 1 1

AB C

B

A

C

MS Excel

SEER SEM

| 8

• Our Team leveraged the three point estimating technique from SEER to account for incompleteness of the requirements.

• Based on the information available, the Cost Estimate Classification (AACEI) is assumed to be a Class 3; the widest expected accuracy range for class 3 is -20% (low band) and +30% (high band).

• This accuracy range was applied in SEER:

-20% +30%widest expected

accuracy range AACEI = Association for Advancement of Cost Engineering International

SEER SEM

COSMIC Function Points Standardized approach to account for incompleteness of the requirements

| 9

2. Software effort estimate approachKnowledge Base (KB) and Work Breakdown Structure (WBS)

• The Knowledge Base selection is the same for all work packages (except for the application):

• The Work Breakdown Structure is composed of 7 Work Packages

SEER SEM

SEER SEM

| 10

• Size of non-functional and project requirements can be estimated with SEER knowledge bases, leveraging historical project data from the ISBSG database.

• For each work package, a specific KB and parameters was applied. These leverage the ISBSG database and enable the estimation the non-functional and projects requirements.

FUR

NFR & PR

FURFUR

NFR & PR

Knowledge base & parameters driving the calculation of non-functional & project requirements (ISBSG Database)

COSMIC functions counted (input into SEER SEM)

User Interfaces Operation Modes Robot Guidance Obs. & Op. Monitoring Obs. Program Mgmt. Robot Observatory Control Room

Knowl. Bases

SEER SEM

Software effort estimate approachSizing method for non-functional requirements

| 11

The following default parameters from the SEER Knowledge Base were modified to reflect the context of the projects:

Personnel Experienced and capable development Team (they will develop the 3rd generation Research Control System)

Multiple site development (60% USA, 20% Europe and 20% in Latin America)

NB: Agile methodology is reflected in the knowledge base (as development method)

Software effort estimate approachProject parameters & assumptions

| 12

• A total of 1,272 COSMIC function points were counted, representing 29,118 hours of effort in total to develop the SW (output from SEER SEM), being the average output.

• An IFPUG function point count exercise was performed in parallel, where a total of 1,284 function points were counted, representing 34,986 hours of development effort (output from SEER SEM).

FP hours

WP1 - User Interfaces 601 14,894

WP2 - Operation Modes 63 1,034

WP3 - Robot Guiding 93 2,656

WP4 - Observation and Operation Monitoring 328 6,942

WP5 - Observing Program Management 51 1,124

WP6 - Robot Observatory 128 2,311

WP7 - Control Room 8 157

Total COSMIC 1,272 29,118

Total IFPUG (benchmarking) 1,284 34,986

WP

COSMIC

SEER-SEM

input: FPs

output: labour hours

Software effort estimate approachTotal COSMIC Function Points and development effort

| 13

Comparison between the COSMIC and IFPUG development effort distributions

Note: The distributions were obtained via Monte Carlo simulation with 5k iterations performed by SEER SEM

MS Excel MS Excel

Software effort estimate approachTotal COSMIC Function Points and development effort

| 14

3. Software cost & schedule outputMultiple site development & labour cost mix (US – EUR – Latin America)

• The development Team is decentralized. Labourcost varies greatly amongst the three locations. A weighted average salary (fully burdened) was estimated for a SW developer (baseline).

• The weighted average SW developer salary was allocated against all the profiles involved into the SW development with an allocation key (based on SEER SEM data).

Burden

50% Tax

50% Overhead & G&A

0% Profit (internal Team/research)

%Yearly salary

(SW developer)Monthly salary FX rate

Monthly salary

(EUR)

Fully burdened

monthly salary (EUR)

US 60% 110,638$ 9,220$ 0.87$ 7,992€ 15,985€

Europe 20% 59,493$ 4,958$ 0.87$ 4,298€ 8,595€

Latin America 20% 360,000MXN 30,000MXN 0.04$ 1,261€ 2,521€

Weighted average 5,907€ 11,814€

MS Excel

SEER profile cost allocation (Programmer =

baseline)

Allocation

key (%)

Calculated monthly

rate (EUR)

Calculated monthly

rate (EUR)

Project Manager Monthly Labor Rate 123% 14,551€ 14,551€

Software/Business Analyst Monthly Labor Rate 77% 9,095€ 9,096€

Software Architect/Designer Monthly Labor Rate 88% 10,454€ 10,455€

Programmer Monthly Labor Rate 100% 11,814€ 11,815€

Data Analyst/Architect Monthly Labor Rate 100% 11,814€ 11,815€

Quality Assurance/Tester Monthly Labor Rate 96% 11,353€ 11,353€

Configuration/Release Manager Monthly Labor Rate 97% 11,454€ 11,454€

Quality Control Lead Monthly Labor Rate 97% 11,454€ 11,454€

Maintenance Monthly Labor Rate 97% 11,499€ 11,499€

MS Excel

| 15

Source image: Introduction to Software Cost Estimation (Chapter 23, 24) - ppt video online download (slideplayer.com)

• The total development cost is estimated at MEUR 2.2 (50% probability level).

• The development schedule is estimated at 23 months (50% probability level).

MS Excel

Software cost & schedule outputTotal Software Development cost & Schedule (Monte Carlo output)

| 16

WBS AllocationDEV EFFORT

(hours)

DEV LABOUR

COST (EUR)

SCHEDULE

(months)

- 1.1: User Interfaces 14,606 1,121,457€ 23.03

- 1.2: Operation Modes 1,109 85,124€ 10.04

- 1.3: Robot Guidance 2,806 215,411€ 15.49

- 1.4: Observation and Op. Monitoring 6,880 528,263€ 18.93

- 1.5: Observing Program Management 918 70,479€ 6.99

- 1.6: Robot Observatory 2,281 175,121€ 13.12

- 1.7: Control Room 152 11,675€ 0.98

TOTAL (50% probability) 28,751 2,207,530€ 23.03

MS Excel

4. Benchmarking

• In parallel to the COSMIC Function Point Count exercise, our Team also performed a counter estimate with IFPUG Function Point Count.

• Comparison against the ISBSG project database was performed.

• Analogy with Avionics SW development project was also performed.

COSMIC Function Point estimation (development effort, cost & schedule)

Counter estimate

IFPUG Function Point Counting

Comparison & Analogy

ISBSG Database

Comparison with Avionics SW

development size

Benchmarking

A

B

C

A B

C

| 17

SEER SEM

SEER SEM

Comparison with ISBSG Database –Development Effort & Schedule

Comparison between COSMIC and IFPUG count

Outlier?

Benchmarking

| 18

• Analogy between the Seabed Observatory Control System (SOCS) project vs. an avionics software project was conducted

• Extract from the COSMIC Functional Size Measurement Method Version 4.0.1 Guideline for Sizing Real-time Software: • A large component of the software of a very

complex real-time avionics system was measured using the COSMIC method. The total size of the requirements (held in a modelling tool) for the component was over 8000 CFP.

• The size of the Seabed Observatory Control System (SOCS) represents approx. 16% of the avionics software component.

8000 CFP

16 %

1,272 CFP

Benchmarking

| 19

5. Environmental costs (project CO2e footprint)Estimation of the mass emissions originating from the development effort

• The development Team will work with laptops to develop the SW. This will generate 1.4 tons of kgCO2e. 55 trees need to be planted to offset the CO2e emissions.

• Following assumptions were used to estimate the kgCO2e emissions & underlying cost:• Average number of staff (8, based on staffing profile)

• Working days of 8 hours (time in use) and 2h of daily standby time

• Energy usage per laptop of 100 w/h (use) & 33 w/h (in standby)

• eGRID CO2e factor and cost of electricity based on US location

Number of trees to plant to

offset CO2e emission:

55

| 20

The end, thank you

Brussels Sprouts

6. Extra reports & charts (back up)

SEER SEM SEER SEM

SEER SEM SEER SEM

| 22