Iwsm2014 defect density measurements using cosmic (thomas fehlmann)

1

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Defect Density Measurements Using COSMIC

Thomas M. Fehlmann, ZürichEberhard Kranich, DuisburgEuro Project Office AGE: [email protected] H: www.e-p-o.com

mailto:[email protected]

http://www.e-p-o.com/

2

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Dr. Thomas Fehlmann

1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD

Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG

1981: Dr. Math. ETHZ 1991: Six Sigma for Software Black Belt 1999: Euro Project Office AG, Zürich 2001: Akao Price 2001 for original contributions to QFD 2003: SwissICT Expert for Software Metrics, ICTscope.ch 2004: Member of the Board QFD Institute Deutschland – QFD

Architect 2007: CMMI for Software – Level 4 & 5 2011: Net Promoter® Certified Associate 2012: Member of the DASMA Board 2013: Vice-President ISBSG

Dr. Thomas Fehlmann

3

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Working at T-Systems International GmbH in Bonn, Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board

Eberhard KranichEberhard Kranich

Mathematics and Computer Science Emphasis on Mathematical Statistics Mathematical Optimization Theory of Polynomial Complexity of Algorithms Worked at T-Systems International GmbH in Bonn,

Germany Six Sigma Black Belt for Software Development Software Quality Assurance Manager Member of the DASMA Board

4

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

What is a Defect?

Defect = Behavior impacting expected or required functionality of software How many bugs? By counting the

size of defect repositories? By number of entries???

5

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Software Testing as a Game

Tester sees selected sequences in the UML sequence diagram

Tester can “walk” the data movements when planning or executing tests Functionality becomes visible to

the agile team Defects impacting functionality

become visible to testers

FunctionalProcess

OtherApplication

SomeDevice

8.// Move some data

9.// Move some data

10.// Move some data


OtherDevice

6

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Functionality, Defect Size, and Defect Density

What happens if data movements have defects?

Testers mark the data movement where a defect has been detected

Same Metric: ISO/IEC 19761 COSMIC

FunctionalProcess

OtherApplication

SomeDevice

8.// Move some data

Move some data



OtherDevice

Functional Size Number of Data Movements needed to implement all FUR

Test Size Number of Data Movements executed in Tests

Test Story Collection of Test Cases aiming at certain FURs

Defect Count Number of Data Movements affected by some defect detected in a test story

7

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Defects Density Prediction?

Now he counts the defects! And counts and adjusts test size By ISO/IEC 19761 COSMIC

FunctionalProcess

OtherApplication

SomeDevice

8.// Move some data

9.// Move some data



OtherDevice

How does he knowthat he found

all the defects?

8

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Research Questions

What is Defect Density? Defects per KDLOC?

What is Test Coverage? Code lines executed by some test case?

WHAT IF YOU DON’T HAVE ANY CODE?WHAT IF

USING SERVICES?

9

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

SW Testing and SW Metrics

Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!

10

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

SW Testing and SW Metrics

Counting practices for defect counting are undocumented “Number of Defects Found” per Stages / with Tests / etc. How do you count “Number of Defects”?

Is it simply the number of entries in a defect repository? How can you avoid double reporting? Or make sure two defects are reported twice and not in a single report?

A successor to the “Defect Measurement Manual” published by UKSMA in October 2000 is under review: “Defect Measurement and Analysis Handbook” By European cooperation Important enhancement for ISBSG’s Data Collection!

Revie

w C

om

ments

11

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Learn & Remember

Software Development the Six Sigma Way

Use COSMICISO/IEC 19761

Benchmark!

Count Defects basedon Reference Model

12

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

As a … [Functional User] I want to … [get something done] Such that …[quality characteristic] So that … [value or benefit]

1) Q001 Propose Standard Destinations User of public transportation be able to store my preferred destinations

they are valid for the Ticket Shop

I no longer have to pay fees when catched without tickets

2) Q002 Find Nearest Boarding Station User of public transportation locate nearest station with GPS that's being served right now I immediately can see whether it's right

3) Q003 Process Payment Provider of transportation services

give user access to their preferred payment options

all payments are traceable in Ticket Shop

they can manage spending

4) Q004 Issue Ticket User of public transportation get a valid ticket with settings from Ticket Shop I no longer have to pay fees when catched without tickets

5) Q005 Show Ticket Ticket controller see the validated ticket I can check validity period and travel range

I don't need to go into a dispute with a client

Functional User Requirements

Example: The Ticket Apps

Customer’s Voice: Give me a ticket subito!

Without much ado and questionsif I simply want to go home

13

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

The Ticket Apps

12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP

App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service Local Time

1.// Add New Destination

Traveler

2.// Check whether Destination Exists

3.// Collect Matching Destinations

4.// Show Matching Destinations

5.// Select Exact Destination

6.// Record Selected Destinations

7.// Ask App for a Ticket

8.// Propose Destinations

9.// Proposed Destinations

10.// Select Destination

Prepare

11.// GPS Coordinates

12.// Date & Time

13.// Ask GIS

14.// Nearest Boarding Station

15.// Search Connection

16.// Next Connection

17.// Request Ticket

18.// Send Pricing Proposal

19.// Proposed Price

20.// Confirm Price

21.// Confirm Transaction

22.// Issue Ticket

23.// Show Ticket

24.// Record Selected Destination

14

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

The Ticket Apps – Objects of Interest

Description Type

Wants to buy a ticket to ride Device User

Store a list of destinations that are valid for "getting home" Functional Process

List of standard stations eligible as travel destination Persistent Data

Finds nearest origin station Other Application

Buy the ticket needed for that route Functional Process

External ticket shop able to identify the App’s user Other Application

Finds connections between stations at a given time & day Other Application

Provides GPS coordinates where phone is located Device User

Synchronized time service located in smartphone Device User

App User

Home Destinations

Ticket Purchase

Ticket Shop

Phone's GPS

Timetable Service

Saved Stations

GIS Application

Local Time

15

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

The Ticket Apps – Prepare Destinations

App User Ticket Shop


Prepare





Home Destinations Saved Stations

6.// Record Selected Destination

Preparation: Store standard destination stations on Smartphone After checking with the Ticket Shop whether they exist or which stop to

select Involves Ticket Shop and local data ‘Standard Stations”

16

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

The Ticket App in the IFPUG Model

Boundary IFP=51

EI

EOEQ

ILF

EIF

EQ1 / 2

Select DestinationEI

1 / 3

Enter New Destination

EI2 / 2

Request Ticket

EI2 / 1

Confirm Purchase

EIF2 / 12

Ticket Shop

EIF1 / 3

GIS Application

EIF1 / 5

Timetable Service

EIF1 / 1

Local Time

EO2 / 2

Boarding StationEO3 / 3

Mobile TicketEO1 / 2

Show Matching Destinations

ILF1 / 2

Home Destinations

Less complicated than UML Sequence Diagrams Suits business better Better boundary

identification between different layers

17

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Prepare Destination in the IFPUG Model

It is not obvious what happens Two elementary process

seem unrelated Nevertheless, they update

an ILF [Uh, they forgot the

elementary process for deleting obsolete home destinations!]

Boundary IFP=19

EI

EOEQ

ILF

EIF

ILF1 / 2

Home Destinations

EO1 / 2

Show Matching Destinations

EI1 / 3

Enter New Destination

EIF2 / 12

Ticket Shop

18

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Buy Ticket in the IFPUG Model

Elementary processes stay The ILF ‘Home

Destinations’ becomes an EIF

Adding the two counts yields

IFPUG is useless for functional sizing when developing Apps or counting defects

IFPUG 4.3 is great for identifying system boundaries splitting apps!

Boundary IFP=42

EI

EOEQ

ILF

EIF

EIF2 / 12

Ticket Shop

EIF1 / 3

GIS Application

EIF1 / 5

Timetable Service

EIF1 / 1

Local Time

EIF1 / 2

Home Destinations

EI2 / 2

Request Ticket

EI1 / 1

Confirm Purchase

EO2 / 2

Boarding StationEO2 / 3

Mobile Ticket

EQ1 / 2

Select Destination

19

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Requirements for a Defect Measurement Reference Model

It must be additive The total size must be the sum of the components’ sizes.

It must be understandable Support UML Sequence Diagrams Works well with UML Use Cases Also with Business Process Model & Notation (BPMN) 2.0

It must fit into agile delivery Buglione-Trudel Matrix for managing agile projects Story Cards for Sprints

Story Card for Prepare Destinations Test isReady

Draft isReady

ReviewDone

Final-ized

Appro-ved

Func-tional

R001-01: Prepare Destinations6 x reworked

RefactoringCount:

5

0

0

Business Impact:

Functional Size:

Story Points:

App User Prepare Destinations Standard Stations Ticket Shop







The user prepares his ticket app by entering standarddestinations. The ticket shop is involved for resolvingstation names matches and avoid spelling errors

20

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Learn & Remember



Benchmark!


21

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

What is a Test?

A Software Test has Several Test Stories

• Weighted by Customer’s Priority for the Test Story, reflecting the value for the customer

Each Test Story has many Test Cases• For various kind of test data

A Test Size attribute • Number of data movements executed by test cases

A Test Coverage attribute • Percentage of data movements covered with test cases

An Outcome• Passed or Failed

– Passed: All responses according expectations– Failed: at least one test case didn’t yield the expected response

22

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Sample Test Stories

Eight Test Stories for the Ticket Apps Each Test Story has several Test Cases Each Test Case has defined Test Data The Expected Response is known as per Test Case

Test Story Case 1 Test Data Expected Response Case 2 Test Data Expected Response

CT-A Prepare CT-A.1 Find Nearest Station CT-A.1.1 Enter GPS exactly Returns correct station CT-A.1.2 Enter GPS nearby Returns nearest station

CT-A.2 Served Stations only CT-A.2.1 Select time of service Returns next available connection CT-A.2.2 Select time out of service Return next best served station

CT-A.3 Enter New Destination CT-A.3.1 Enter valid station name Destination stored CT-A.3.2 Enter invalid station name Destination rejected

CT-B Ticketing CT-B.1 Select Destination CT-B.1.1 Station of origin known Asks for destination CT-B.1.2 No destination selected Stops without contacting Ticket Shop

CT-B.2 Get Ticket CT-B.2.1 Both stations known Asks to confirm price CT-B.2.2 Boarding station undefined Doesn't ask for destination

CT-B.3 Price Calculation CT-B.3.1 Both stations known Presents price of next actual connection CT-B.3.2 Both stations known Display intermediate change stations

CT-B.4 Issue Ticket CT-B.4.1 Issue successful, ticket shown All credentials visible CT-B.4.2 Ticket Shop blocked Explanation given

CT-B.5 Payment Tests CT-B.5.1 Credit limit exceeded Returns exceed notice CT-B.5.2 Payment service out of order Returns warning

23

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

What is a Test Case?

A Test Cases has Entry Data (“Test Data”)

• Explaining the environment for the test case• Typically valid, invalid, borderline data• Normal and disturbed communication services

A known Expected Response• The response of the system is known in advance

A known sequence of data movements executed• Defining Test Coverage• Each Test Case has a Size

24

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

The sequence of data movements can be visualized in the sequence diagram per test case

By double-clicking therespectivedata movements

Find Route GIS Application Phone's GPS


9.// Date & Time

10.// Ask GIS


Local Time

What is a Test Case?

Test Story No. 1

User StoriesCT-A.1 Find Nearest Station Q001: Propose Standard Destinations Q002: Find Nearest Boarding Station Q003: Process Payment Q004: Issue Ticket Q005: Show Ticket Expected ResponseCT-A.1.1 Enter GPS exactly E012 E012,X009 Returns correct station

CT-A.1.2 Enter GPS nearby E012,X009 Returns nearest station

CT-A.1.3 Enter malformatted GPS E006 Returns no station

CT-A.1.4 Enter far away GPS position E006,E007,X004,E008 E012,X009 Returns nearest station plus warning

CT-A.1.5 GPS not working E005,E006 E006 Returns warning

Test Story Contribution (CFP): 2 6 0 1 6 Test Size

Test Case Measurementsfor Test Story CT-A.1

25

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Defect Reporting

When a defect has been identified in a test case, it can be recorded by double-clicking the suspect sequence of data movements

User StoriesCT-B.1 Select Destination Q001: Propose Standard Destinations

CT-B.1.1 Station of origin known R001,X001,E005

CT-B.1.2 No destination selected R001,X001

CT-B.1.3 Valid boarding & destination W001,E005,R001,X001CT-B.1.4 List of destinations X003,R001,W001

Test Story Contribution (CFP): 12

Test Case Measurementsfor Test Story CT-B.1

Test Story No. 4

Expected Response CFP Name Label Description Name Label Asks for destination 6

Stops without contacting Ticket Shop 2

Returns next available connection 15All readable and visible 3 #003 Multiple References If a station contains more than one transport

mode, e.g., bus and train, both are recorded as separate destinations

W001 Record Selected Destinations

Test Size 26 1

Defects Observed Data Movements Affected

Defect Count

26

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

12 Entry (E) + 9 eXit (X) + 1 Read (R) + 2 Write (W) = 24 CFP

App User Home Destinations Saved Stations Ticket Shop Ticket Purchase Phone's GPS GIS Application Timetable Service


Traveler






7.// Ask App for a Ticket

8.// Propose Destinations

9.// Proposed Destinations

10.// Select Destination

Prepare


12.// Date & Time

13.// Ask GIS


Extract Report

Select Data Movements forTest Case CT-B.1.4: List of destinationswhen executed in view of Q001: Propose Standard DestinationsThis Test Case identifies Fault #003: Multiple Referencesaffecting Data Movement 6.// Record Selected Destinations

Finish

Record Defects

27

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Count Defects

You can count One Defect

Per Data Movement

Identified by One Test Story

And never more…

©2012-2014 Rhafiel

28

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Compare

This means The more test stories you have, the more defects you can uncover Test size is equally important as functional size Defect density depends from test size

Test Benchmarking

Test Status Summary

Total CFP: 24Defects Pending for Removal: 3 Test Size in CFP: 185

Defects Found in Total: 3 Test Intensity in CFP: 7.7Defect Density: 13% Test Coverage: 79%

29

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Test Stories

Goa

l Tes

t Cov

erag

e

Find

Nea

rest

Sta

tion

Serv

ed S

tatio

ns o

nly

Ente

r New

Des

tinat

ion

Sele

ct D

estin

atio

n

Get

Tic

ket

Pric

e C

alcu

latio

n

Issu

e Ti

cket

Paym

ent T

ests

Ach

ieve

d C

over

age

CT-

A.1

CT-

A.2

CT-

A.3

CT-

B.1

CT-

B.2

CT-

B.3

CT-

B.4

CT-

B.5

Q001 Propose Standard Destinations 0.52 2 9 12 8 9 7 0.55

Q002 Find Nearest Boarding Station 0.45 6 9 7 10 3 5 4 0.45

Q003 Process Payment 0.62 3 4 4 4 10 7 16 0.59

Q004 Issue Ticket 0.24 1 3 1 3 6 2 3 0.23

Q005 Show Ticket 0.29 6 3 2 1 9 3 3 0.31

Ideal Profile for Test Stories 0.18 0.20 0.26 0.40 0.38 0.51 0.36 0.40 Convergence Gap

0.18 0.2 0.2 0.4 0.4 0.5 0.4 0.4 0.060.10 Convergence Range

0.20 Convergence Limit

Test StoriesDeployment Combinator

User Stories

Toggle Measured Controls

Measuring Test Coverage

AchievedResponse

Expected Response Number of

data movementsexecuted

30

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Learn & Remember



Benchmark!


31

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Conclusions

You need more than one model when measuring modern software ISO/IEC 19761 COSMIC is most appropriate for measuring defects Other methods might be suitable for cost estimations

However, in these days, what drives cost more than faulty software?

Stop the nonsense with counting defects in repositories Include defect density goals in contracts Do independent auditing as long as you don’t trust your measurement

equipment

Start Measuring Defects. Now.

32

CustomerOrientation

Lean Six Sigma

AgileProcesses

ProjectEstimations

TransferFunctions

Questions?

Iwsm2014 defect density measurements using cosmic (thomas fehlmann)

Software

defect size

defect counting

customer orientation

defect repository

functionality of software

software metrics

agile team defects

undocumented number