Measures

1Software Process Management

Introduction

Measurement and Metrics

Objectives

The aim of this topic is to introduce the concepts of measurement and metrics as well as the practical skills necessary to define and count LOC and FP, the basis for many software metrics.

You will: – Develop an understanding of why is it important to

measure the process of software engineering and the products it produces

– Develop practical skills in size measurement and basic function point analysis.


Introduction


Introduction

Anything that you need to quantify can be measured in some way that is superior to not measuring it at all.

Tom Gilb

When you can measure what you are speaking about and can express it in numbers, you know something about it. But when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind.

Lord Kelvin

If software development is to be viewed as an engineering discipline, it requires a measurement component that allows us to better understand, evaluate, predict and control the software process and product.

Victor Basili, University of Maryland


Introduction


Why Measure?

We measure for a number of reasons. These include:• To characterize - to gain an understanding of products,

processes,... • To evaluate - to determine status with respect to plans. • To predict - so that we may plan • To Improve - rational use of quantitative information to

identify problems and strategies to remove them


Introduction


What to Measure

• Process

Measure the efficacy of processes. What works, what doesn't.

• Project

Assess the status of projects. Track risk. Identify problem areas. Adjust work flow.

• Product

Measure predefined product attributes (generally related to ISO9126 Software Characteristics)


Introduction


The Goal Question Metrics (GQM) Approach

The GQM approach is based on the idea that in order to measure in a meaningful way we must measure that which will help us to assess and meet our organisational goals.– A Goal is defined for an object, for a variety of reasons, with respect

to various models of quality, from various points of view, relative to a particular environment. The objects of measurement are products, processes and resources.

– Questions are used to characterize the way the assessment/achievement of a specific goal is going to be performed based on some characterizing model. Questions try to characterize the object of measurement with respect to a selected quality issue and to determine its quality from the selected viewpoint. Questions must be answerable in a quantitative manner.

– Metrics are associated with every question in order to answer it in a quantitative way.


Introduction


The Goal Question Metrics (GQM) Approach

Example:– Goal: To improve the outcome of design inspections from the quality

managers point of view.– Question: What is the current yield of design inspections?– Metric: inspection yield = nr defects found / est. total defects– Question: What is the current inspection rate?– Metric: individual inspection rate, inspection period, overall

inspection rate– Question: Is design inspection yield improving?– Metric: current design inspection yield / baseline design inspection

yield * 100– ...


Introduction


Six Sigma Approach

What Is Six Sigma?

Six Sigma is a rigorous and disciplined methodology that uses data and statistical analysis to measure and improve a company's operational performance by identifying and eliminating "defects" in manufacturing and service-related processes.

Commonly defined as 3.4 defects per million opportunities, Six Sigma can be defined and understood at three distinct levels:

metric, methodology and philosophy...


Introduction


Six Sigma Approach

In Six Sigma measurement programs focus on collecting data that will help you to improve your processes in order to better satisfy your customers. It is a common sense notion that something you do in working on a product (some part of the process) will have an effect in terms of the customer's perception of the quality of that product, and that by identifying problems with the process and making subsequent process improvements will improve the customer's perception of the product. The measurments that you should be making are, therefore, the ones that will help you to assess and improve processes with the goal of satisfying the needs of the custmer(s).


Introduction


Six Sigma Approach

This approach bases itself on the operational statement: you must state concisely what it is that you are going to improve (or achieve): "what's wrong with what".

It must be written in simple language using terminology that has the same meaning to everyone who is going to read it.

An operational statement will be stated in terms of the effect of what it is that you are improving.

It should aim to identify the where, when, what and how big of the problem.


Introduction


Six Sigma Approach

For example:

the statement "the customers aren't happy" is of very little use in making them happy.

However, by repeatedly asking why and by analyzing any available data we might arrive at a concise statement such as "The XYZ module is unavailable 10% of the time".

This is a much more useful metric against which we could base the assessment of process changes.

To get to a statement like this requires measurement of availability.


Introduction


Six Sigma ApproachIn order to assess process improvement we must collect data over time.

– This should begin with establishing baseline performance. A baseline is the average of historical data over a specified period of time. In the example above, Module XYZ is unavailable 10% of the time is baseline a statement of the baseline performance.

– Then we must then establish a goal. This goal is stated as a measure of improvement over a period of time in terms of the baseline data. The Six Sigma rate of improvement goal is often quoted as being a 10X improvement over a period of 2 years. If we take this improvement goal and apply it to the example above we would establish the 10X/2years improvement goal as Module XYZ is unavailable 1% of the time. Or stated more usefully, "Module XYZ is available 99% of the time".

So summarizing the example, we may have the following:Operational Statement: "Module XYZ is periodically unavailable"Historical Data: Availability metrics calculated for module XYZ over the past 6 months indicate that it is available 90% of the time.Baseline performance: Module XYZ availability = 90%10x/2years improvement goal: Module XYZ availability = 99%

In order to verify our progress toward the goal we must track actual progress planning and performing periodical reviews of this performance


Introduction


Process Metrics

Process metrics primarily focus on organizational performance and quality achieved as a consequence of a repeatable or managed process. This characterization and evaluation of a process for achieving performance and quality outcomes is known as Quality Assurance.

Process metrics include metrics relating to:– statistical SQA data – defect categorization & analysis – defect removal efficiency (DRE) DRE = E/(E+D)

E number of errors before delivery (or before a particular phase) D number of defects after delivery

– defect propagation from phase to phase – reuse data


Introduction


Project Metrics

Software project metrics are used to adapt workflow and technical activities.

They are primarily used in estimation, monitoring and control of workflow within a project.– Effort/time per SE task – Defects detected per review hour – Scheduled vs. actual milestone dates – Changes (number) and their characteristics – Distribution of effort on SE tasks


Introduction


Product Metrics

product metrics generally focus on the quality of deliverables. They include:– measures of analysis model – complexity of the design

• internal algorithmic complexity • architectural complexity • data flow complexity

– code measures (e.g., Halstead) – Measures of maintainability – defect metrics


Introduction


Measurement Program

A measurement program is an effective means for controlling the software process performance (i.e., the actual results a project achieves by using its defined process) and guiding improvements in the software engineering processes.It has to based on a model in which a comprehensive measurement framework may be constructed, analyzed, and modified as the organization’s goal change.In order to define such a program, it is presented the underlying life cycle thatapplies to the use of software metrics.


Introduction


Metrics Life Cycle

Planning: to identify the scope of the measurement program through– Metric selection

Identify the measures to support organization objectives. The use of Goal/Question/Metric paradigm ensures that all the strategic goals of the company are measured.

– Metric specification

Define the operational measurements procedures: selected metrics have to be provided with specific meanings so that these can be uniformly collected and interpreted


Introduction


Metrics Life Cycle

Implementing: data are collected in accordance with the operational measurement procedures, validated, and analyzed. – Data collection

apparently the most easy, but also the most important.

To be effective it must be automated.– Data validation

one of the easiest and most cost effective way to validate data is simply to observe the data collection process and make sure that data are collected in accordance with the defined procedure (to ensure at least standardization)


Introduction


Metrics Life Cycle

– Data analysis • to evaluate the status of the project with respect to

plans• to improve the software process

In process analysis allows to evaluate whether the project is drifting off track, so that they can be brought back under control by suitable actions.A posteriori analysis permits to controls the trend of the key measures of the whole company, and to refine the baselines to compare against, so that it can be judged whether or not the improvement actions are working as intended and what the side effects may be.


Introduction


Metrics Life Cycle


Introduction


Metrics Life Cycle

Improving: periodically, the entire measurement program is to be reviewed to ensure that it helps the software projects to control their processes, and the whole organization to improve. Metrics can be revised, redefined following again the metric life-cycle.

In addition, in case there exist metrics with dubious values or not used, they can even be retired.


Introduction


Metric Objective

The main purposes of measuring are:

• measuring the Productivity of a project

• measuring and evaluating the Quality of the software product

• controlling a project.

in the SEI CMM Level 3 the management and control of size/complexity, reusable software components, effort and costs is required.

These are also the goals to be used in the GQM approach.


Introduction


Productivity

Establish the relationship between the output and the input of a project

Problems:

• establishing a measure unit for the output

• recognizing all the project/product/management characteristics affecting the expended effort

• measuring the other factors affecting the effort, as rework due to change and reuse


Introduction


Quality

In order to measure the quality of a software product aspects such as :• The defects density and rate• The removal efficiency• The number of bad fixes

have to be considered.


Introduction


Controlling the project

As a support for the Project manager to provide information for performing as much precise estimate as possible and for supporting technical and managerial choices.

problems to be resolved controlling a project are related both to prediction and evaluation of• effort • scheduling• stability • defect


Introduction


Primary Data

The Primary Data are the lower level of the information to be collected and documented when prescribed, in order to allow the calculation analysis helpful to monitor the process and evaluate the products.

These are:• Size• Effort• Defects• Duration (estimated and actual)• Stability (Requirements and Staff Changes)


Introduction


Size

Furthermore the size represents the starting point for the whole metrics system.

It is involved in both assessment and predictive metrics;• for assessment, it is used to measure an artifact or a

system, or also to normalize other metrics• for prediction it serves to give a concrete mean to

express the provisions: the strong connection between Product size and the effort needed to develop the Software Product, the size allows to derive accurate estimates of the effort, cost, and schedule of the project


Introduction


Size

The product size can be expressed in different unit of measurement, according to the adopted estimating/ measuring technique.

As a Backfiring method exists for determining the correspondence between number of Function Points and number of LOC so it ispossible to establish a correspondence factor between Object-Oriented Function Points and Function Points


Introduction


Size – Arpad method

Measures performed at the beginning of the Concept Exploration can only provide indicative information (low confidence level) on the target product size and can be gathered only with indirect measures; the calculation is based on the estimate of the effort; the obtained measure can have a low confidence level, This method is based on the computation of a quantity corresponding to the effort needed to the specification of the system. This calculation considers factors as • number of people to be interviewed, • number of days required to the interviews, • number input and output screens, reports, menus and so on.

With a very simple step, it is possible to obtain a first, indicative estimate of the global effort needed for the complete development of the Software Product (under certain hypothesis.)Considering a reference Productivity Parameter the estimated global effort allows to derive a first estimate of the size.


Introduction


Functionality Oriented Measures (Function Points)

A direct estimate of the size is possible at the end of the Concept Exploration, using the Function Point Analysis method based upon the artifacts produced, i.e. System Requirements and System Architecture.

The Function Point Analysis method gives a way of sizing software through the analysis of the implemented functionality of a system from the user‘s point of view, considering the application as a black box.


Introduction


Functionality Oriented Measures (Function Points)

It consists, simplifying, of two main steps:• to evaluate the unadjusted Function Points (UFP) count,

basing on the number of user functions and on its complexity classification

• to evaluate the adjusted Function Points (FP) count, by scoring fourteen general system characteristics.

These steps allow the calculation of the product size keeping into account also its complexity, i.e. the degree of complicatedness of the product owing to the structure of the aggregated components and related relationships.


Introduction


Function Points comp.

STEP1

Count the number of external inputs, external outputs, external inquiries, internal logic files, and external interface files required. The IFPUG provides the following definitions for these:

• External Input (EI): An EI is an elementary process in which data crosses the boundary from outside to inside. This data may come from a data input screen or another application. The data may be used to maintain one or more internal logical files. The data can be either control information or business information. If the data is control information it does not have to update an internal logical file.

• External Output (EO): An EO is an elementary process in which derived data passes across the boundary from inside to outside. Additionally, an EO may update an ILF. The data creates reports or output files sent to other applications. These reports and files are created from one or more internal logical files and external interface file.

• External Inquiry (EQ): An EQ is an elementary process with both input and output components that result in data retrieval from one or more internal logical files and external interface files. The input process does not update any Internal Logical Files, and the output side does not contain derived data.

• Internal Logic File (ILF): AN ILF is a user identifiable group of logically related data that resides entirely within the applications boundary and is maintained through external inputs.

• External Interface File (EIF): An EIF is a user identifiable group of logically related data that is used for reference purposes only. The data resides entirely outside the application and is maintained by another application. The external interface file is an internal logical file for another application.


Introduction



STEP2 Rate each component (EI,EO,EQ,ILF,EIF) as low, average, or high.

For transactions (EI’s, EO’s, EQ’s) the ranking is based upon the number of files updated or referenced (FTR’s) and the number of data element types (DET’s).

For both ILF’s and EIF’s files the ranking is based upon Record Element Types (RET’s) and Data Element Types (DET’s).

• A record element type (RET) is a user recognizable subgroup of data elements within an ILF or EIF.

• A data element type (DAT) is a unique user recognizable, non-recursive, field.

• File Type Referenced (FTR): A FTR is a file type referenced by a transaction. An FTR must also be an internal logical file or external interface file.


Introduction



STEP3 • Multiply each count by the numerical rating shown for (low,average,high) to determine the

rated value. • Sum the rated values in each row (EI,EO,EQ,ILF,EIF) giving a total value for each type of

component . • Sum the totals in the rightmost column for each component to give Total Number of

Unadjusted Function Points (UAF).


Introduction



STEP4 Next we calculate a value adjustment factor (VAF) based on 14 general system characteristics (GSC's) that rate the general functionality of the application being counted.

Each characteristic has associated descriptions that help determine the degrees of influence of the characteristics. The degrees of influence range on a scale of zero to five, from no influence to strong influence.


Introduction



STEP5

Calculate the Value Adjusted Function point (VAF) using the IFPUG value adjustment equation with the 14 weightings allocated to each General System Characteristic (GSC):

VAF = 0.65 + 0.01*Sum(GSCi)

for all i in [1..14], where 0.65 and 0.01 are empirically derived constants.

That is, VAF is calculated by summing all of the answers to the 14 questions (weighted 0-5), multiplying this sum by the factor 0.01 and then adding 0.65.

STEP 6

The final Function Point Count is obtained by multiplying the VAF times the Unadjusted Function Point (UAF).

FP = UAF * VAF (function points)


Introduction


Product size measures at the Design phase

As the design describes directly the selected solution of the problem, it contains precise information and Function Points method can be usefully applied at this level.

At this level, where the use of reuse and COTS are well defined and delimited, detailed estimates are possible as only those classes to be developed are included in the count.

So the used method could be:• Determination of LOC from class/method analysis,

backfired to FP• OOFP calculation converted to FP


Introduction


OOFP

The Object-Oriented Function Points method uses an object-oriented specification, focusing on object, attributes, and operations. Applying this method, the boundary can be moved to surround individual classes; in this way not only delivered functionality are measured, but also the size and the complexity of the application.

When dealing with OO Programming, these steps have to be ensued • Look at each cluster of object classes as a system and count

Function Points• Analyze the Object Diagrams and count:

• Data Items (Internal Data Items, External Data Items)• Service Requests (Incoming, Outgoing, Status Inquires)

• Forget value adjustment factors• Report unadjusted Function Points


Introduction


Product oriented measures (LOC)

At the beginning of the construction/module test phase, an estimation of the product size, in terms of Lines of Code, has to be performed. • by means of the backfire method, which derives the LOC

from the counted FP • or using corporate data


Introduction


What’s a LOC

The most general approach to size measurement is to count the number of text lines in a source program. In doing this, we typically ignore blank lines and lines with only comments. All other text lines are counted. This approach has the advantage of being simple and easy to automate.

This LOC counting approach has the disadvantage of being sensitive to formatting. Those programmers who write very open code will get more LOC for the same program than would their peers who used more condensed formats.

Even comments influence this counting, you can strip comments, but commented code is much more maintainable than uncommented one.

In doing this, you should also establish the practice of putting a logical LOC on each physical line of the source program.


Introduction


Statements Count (logical LOC)

Counts of logical statements attempt to characterize size in terms of the number of software instructions, irrespective of their relationship to the physical formats in which they appear

Either the operational definition of LOC and statement is to be provided through tables that explicitly identifies the values for each attribute that is to be included in or excluded from our statement counts.

Examples of such table are from

Robert E. Park “Software Size Measurement: A Framework for Counting Source Statements” CMU/SEI-92-TR-020


Introduction


Size versus development time

• LOC are precise, machine countable, environment specific, and they can reflect the mix of reused, modified, and new code.

• The principal disadvantage is that LOC are hard to visualize early in a project.

• the size and development time for the C++ programs I have developed are highly correlated.

• This means that once I have estimated the size of a new program, I can readily calculate the time it will take me to develop it.


Introduction



But ..• What about the use of other languages (other than C+

+)?• What about other development groups (the experience

of developers is important) ?• What about maintenance (there is not just

development)? • What about the quality of requirements, architecture,

design, ….. ?

Then ….


Introduction



This can be true for just a company, – steady (without great turn-over)– with only a particular kind of software developed

(unless it develops metrics for every kind of sw)– just in one language (unless having metrics for all

languages)– Which collects historical data to track its productivity

that will be used for forecasting new products costs


Introduction


Backfiring

A correspondence between the number of LOC and Function Points was researched by Capers Jones, resulting in a technique called BackfiringThis technique makes available a table that, for each different language, provides the number of LOCs corresponding to a Function Points.But• the margin of error in converting LOC data into Function

Points or back is high• the Backfiring conversion factors have to be derived by

studying a large number of products, belonging to different domains


Introduction


Productivity

The productivity is generally measured as the number of working hours needed for the production of a single unit.Simple? Yes, but ….• Many productivity indicators exist in literature; these are generally obtained

as mean values of a set of different projects, which were developed by people with different skills, working in different development environments.

• Many factors which influence these indicators exist. They are not present in every organization at the same time, neither standardizable.

Each organization must collect its own data and use them for the productivity evaluation, concentrating only on factors allowed by the available data.The level to which these factors are present differ from one organization to another, therefore it is important to collect and adopt a company’s own data in different periods. The best way to handle the productivity function is to base it on a family of factors.


Introduction


Productivity

Productivity is affected by:• Project characteristic: describing people and processes

involved in the development effort.• Management characteristic: describing the way how a

project is managed.• Product characteristic: reflecting the nature of the product

itself.


Introduction


Productivity

Project characteristic

Personnel: the topic of personnel includes most of the “human elements” in a project.The personnel characteristic are listed below:

– Education: the level of degree– Experience: the numbers of years of relevant experience in software engineering, or

with– specific software application and technologies or in the organization.– Expert Assistance: it refers to a person (or group of person) external to a project team

who– has knowledge and experience in the application being developed by the project team.– Training: external or internal.– Size: size refers to the numbers of Direct and Support staff involved in the project.– Turnover.

Software Development Environment (SDE). The SDE is the combination of tools, techniques,

and administration used during the development.


Introduction


Productivity

Management characteristicIt describes “how” the project is managed. Management characteristic information is recorded in four main categories:

• User Participation. Record the level of participation by the user or their representative on the project. It should be considered during the project characterization.

• Stability of Product Requirement. It characterize the extend that the requirement remained constant throughout development. It should be measured by the Requirements Change Distribution

• Constraining Management Factors. It specify the management or administrative factors that limited the project, e.g. fixed cost, fixed staff size, fixed functionality, fixed quality and reliability, fixed schedule, limited accessibility to development system, limited accessibility to target system.

• Not Directly Productive Activities These are some activities, that although necessary, do not produce a measurable and tangible artifact, like Project Management and Configuration Management. It is possible to collect the values of the effort spent for these activities for each project and these data will be considered, at the end of the project, for the productivity evaluation.


Introduction


Productivity

Product characteristic:

Criticality: this can be:– Timing Critical: some products that must work in environments where the real-time

behavior, user response, or throughput is critical.– Memory Critical: products that must be fit into a limited amount of memory.– Quality/Reliability Critical: some products that must meet very stringent quality or

reliability criteria.

Degree of Innovation: the technological risk of the project.

Complexity: there are three types of complexity: – data, – programming, – organizational (the last refers to the difficulty in coordinating and communicating with

all parties on the project teams).

Reuse: Some project can use already existing components (with reuse activity) or can developin order to produce reusable components (for reuse activity).


Introduction


Effort

Number of persons per unit time (most common measure for planning and controlling the project progress)

Related to • work performed during each planned activity; • work expended in not directly productive activities (Project management,

Configuration Management, etc.)

The collection of effort data expended on the various activities involves the gathering of the effort spent by each person on those activities, by means of proper forms

Tracking the effort data and comparing them against a reference effort distribution provides a good means for the in-process assessment of the project trend.Effort data are also one of the input data used for the calculation of the Productivity


Introduction


Defect

In general a defect is defined as a product anomaly. Examples include omissions and imperfections found during early life cycle phases and faults contained in software sufficiently mature for test or operationWe distinguish between• testing defects• Inspection defects

if not fixed, would cause one or more of the following to occur:• a defect condition in a later inspection phase,• a defect condition during testing,• a field defect,• nonconformance to requirements and specification,• nonconformance to established standards such as performance,

national language translation, and usability.


Introduction


Defect

Defects are injected into the product or intermediate deliverables of the product at various phases.

In particular, for the development phases before testing, the development activities themselves are subject to defect injection, and the reviews or inspections at the end of the phase activities are the key vehicle for defect removal


Introduction


Defects

Possible measures on defects:Defects number: To evaluate the number of detected defects of the software product

Defect Rate: to evaluate the number of detected defects of the software product per activity with respect to its size.

Defect Density: To evaluate the number of detected defects of the software product with respect to its size.

Removal Efficiency: To evaluate the cumulative percent of the previously injected errors of the Software Product that have been removed prior to delivery


Introduction


Duration

The Duration of an activity is given by the difference between the start and the end date of the activity (elapsed time).

Considerations :• It is possible to distinguish various levels of scheduling: a general level,

which concerns the whole project, the level of Concept Exploration and Iteration and a level which concerns a single activity.

• Each time a Plan is revised, the duration of the activities can be re-estimated; all these estimates have to be collected.

• The Estimated Duration of the activities provides information on the quality of the process, when compared with the actual values.

• The collection of all estimations, compiled in the various planning activities, constitutes the complete history of the scheduling, allowing comparison of the various estimations with the actual values and between them.

• The comparison between the actual and the planned duration values allows to calculate the Schedule Adherence and helps to understand and evaluate the quality of the adopted process.


Introduction


Duration

Possible metric: schedule adherence

to provide information on the progress of the project within a particular activity


Introduction


Stability

Requirements Changes

The initial set of requirements of a project can be changed in three possible ways:• by modifying a requirement of the set• by adding a requirement to the set• by deleting a requirement from the set.

Note that a modified Requirement maintains the same identifier and does not alter the total Requirements number.


Introduction


Stability

Staff ChangesThese data indicate the number of the staffs lost and of the staffs that are added in order to substitute the lost staffs or for achieving the estimated staff number.

• The number of these staff should be recorded per level, and tracked, if there are changes, with a suggested period of a month.

• By tracking the staff addition and losses, it is possible to record the actual staff for each time period and compare it with the estimates.

Metric: staff turnover to provide information on the percentage of staff losses

Note that: the staff changes allow the calculation of the ‘turn-over’ affecting the staff stability: the loss of one or more people, even though substituted keeping unchanged the number of staff of a certain level, can affect in significant manner the productivity,

Measures

Documents

sigma approach

software process

sigma measurement programs

current inspection rate

inspection period

software metrics

object of measurement

individual inspection