Test Estimation in Practice

TQ PM Tutorial

10/14/2014 1:00:00 PM

"Test Estimation in Practice"

Presented by:

Rob Sabourin

AmiBug.com

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Rob Sabourin AmiBug.com

Rob Sabourin, P. Eng., has more than thirty years of management experience leading teams of software development professionals. A well-respected member of the software engineering community, Rob has managed, trained, mentored, and coached hundreds of top professionals in the field. He frequently speaks at conferences and writes on software engineering, SQA, testing, management, and internationalization. Rob wrote I am a Bug!, the popular software testing children's book; works as an adjunct professor of software engineering at McGill University; and serves as the principle consultant (and president/janitor) of AmiBug.Com, Inc. Contact Rob at Contact Rob at [email protected].

IntroductionTEST Estimation IN PRACTICE

Course timingCourse timing

Electronic

devices

Electronic

devices

Smoking

Smoking

MealsMeals

Facilities

Facilities

Breaks

Breaks

Administrivia

6© 2012 SQE Training V3.0

Introductions

Name

CompanyNameBusiness

EnvironmentHardwareSoftware

ExperienceTestingManagement

Personal Course Objective(s)

Name

CompanyNameBusiness

EnvironmentHardwareSoftware

ExperienceTestingManagement

Personal Course Objective(s)


Course Agenda

1. Estimation2. Estimation Techniques


1estimation

Estimation

Estimate: – A tentative evaluation or rough calculation – A preliminary calculation of the cost of a

project – A judgment based upon one’s impressions;

opinion —The American Heritage Dictionary

It is very difficult to make a vigorous, plausible, and job-risking estimate that is derived by no quantitative method, supported by little data and certified chiefly by hunches of the managers.

— Fred Brooks10© 2012 SQE Training V3.0

The best estimatesThe best estimates

Represent the collective wisdom of practitioners (testers, developers, users, etc.) and have their buy-in

Provide specific, detailed catalogs of the costs, resources, tasks, and people involved

Present, for each activity estimated, the most likely cost, effort, and duration

Estimation: the creation of an approximate target for costs and completion dates Estimation: the creation of an approximate target for costs and completion dates

Test Estimation


Factors that can influence cost, effort, and duration include:Factors that can influence cost, effort, and duration include:

Required level of quality of the system

Size of the system to be tested

Historical data

Process factors (process maturity, etc.)

Material factors (tools, data, etc.)

People factors (skills, experience, managers, etc.)

Test Estimation (cont.)


Test Estimation (cont.)

• Delivery of estimates should include justification

• Negotiation and re-work of estimates is normal

• Final estimates represent a balance of organizational and project goals in the areas of quality, schedule, budget, and features


How Good an Estimator Are You?Low Estimate High Estimate

___________ to ____________

.

.

.

© Steve McConnell, Software Estimation, Microsoft Press, 2006

Surface temperature of the sun

Latitude of ShanghaI

Area of the Asian continent

Birth year of Alexander the Great

Total value of US currency in 2004

Total volume of the Great Lakes

Worldwide box office receipts of Titanic

Total length of Pacific Ocean coastline

Heaviest blue whale ever recorded

# of books published in US since 1776


How Good an Estimator Are You?• Surface temperature of the sun 10,0000 F / 6,000 C

• Latitude of ShanghaI 31° 10′ N

• Area of the Asian continent 17,212,000 sq mi

• Birth year of Alexander the Great 356 BC

• Total value of US currency in 2004 $719.9 Billion

• Total volume of the Great Lakes 5,500 cu mi

• Worldwide box office receipts of Titanic $1.835 billion

• Total length of Pacific Ocean coastline 84,300 miles

• Heaviest blue whale ever recorded 380,000 pounds

• Number of books published in US since 1776 22 million


How Good Is Our Industry (at Estimating)?

• Tata: 62% of projects fail to meet schedule

49% have budget overruns• Moiokken and Jorgensen: 30-40%

overruns


Class Discussion

Why is estimating not done well?

Your top five reasons:

1) Too many variables____________________

2) ____________________________________

5) ____________________________________

6) ____________________________________

5) ____________________________________


Why Estimates Are Inaccurate ― Part I

• Lack of estimating experience• Lack of historical data on which to base estimates• Lack of systematic estimation process, sound techniques, or

models suited to the project• Failure to include essential activities and products within the

scope of the estimates• Unrealistic expectations or assumptions• Failure to recognize and address the uncertainty inherent in

project estimates

Practical Software Measurement

Addison-Wesley, 2001


Why Estimates Are Inaccurate ― Part II

• Lack of education and training• Confusing the target with the estimate• Hope-based planning• Inability to communicate and support estimates• Incomplete, changing, and creeping requirements• Quality surprises (test and re-fix)

—adapted from Linda M. Laird

The Limitations of Estimation


Bohem’s Cone of Uncertainty


NHC Track Forecast Cone


“Testing” Track Forecast Cone

Best ca

se

Expected case

Worst case

T i m e

Tas

ks(or why it is important to constantly re-estimate)


What would have to happen to deliver this in four weeks?What would have to happen to deliver this in four weeks?

What should the estimate have been?What should the estimate have been?

The Fantasy Factor

Weeks

Today

0 1 2 3 4 5 6 7 8 9

1st 3rd2nd


Time

Time

SizeSize

Quality

Quality

Resources

Resources

Estimation

1, 2, 3, or 4 Variables + Many Modifiers:

If it’s not variable, then it’s fixed.


Time vs. Resources

=


2Estimation Techniques

Test Estimation Techniques ― Examples

• Intuition and guess• Work-breakdown-structures• Three-point estimates• Company standards and norms• % of project effort or staffing• Industry averages and predictive models (e.g., FP, TPA )• Team estimation sessions

– Wideband Delphi– Story point sizing– Poker estimation– T-shirt sizing


What do intuition and guess imply?What do intuition and guess imply?

No formal estimation process

Based on the experience of the team member(s) usually without the benefit of recorded metrics

Intuition and Guess


When would we use this technique?When would we use this technique?

Experienced staff members

Little or no recorded metrics available

Great accuracy is not required

Same type of project we’ve done in the past

Or at the other extreme, we don’t have a clue (often called a WAG or SWAG)

Intuition and Guess (cont.)


Project LevelProject Level

Phase LevelPhase Level

Task LevelTask Level



------


------

------

------


------





Work Breakdown Structure



When the test team lacks credibility with their estimates

When we are doing an estimate for a project that is dissimilar to projects we have done in the past

When the project is too large to get our “arms around it” (e.g., it is too big to easily see the big picture)

Work Breakdown Structure (cont.)


When would we not use this technique?When would we not use this technique?

When the project has lots of turbulence!



Approach:Approach:

Develop goal

Break goal into deliverables

Break deliverables into activities

Decompose work into small manageable components that are of sufficient detail to allow you to make estimates of time and/or resources

Work Breakdown Structure ― Top Down



Testing for Completeness

• Status of a task is measurable• Start and end events are clearly defined• Further breakdown makes no sense• Each activity has a deliverable• Time or resources for an activity can be estimated• Duration of activity is within acceptable limits• Each activity is independent• Each activity forms a unique package of work that could

be outsourced



• Level of detail guidelines:– No single activity or group of activities should

exceed 80 hours of effort*– No activity or group of activities should exceed

the reporting period (e.g., if the reporting period is weekly, no activity should exceed one week)

– “If it makes sense” rule: The duration of each activity must pass the common sense rule


Three–Point Estimates

• Three-point estimates is an analytical technique using three cost or duration estimates:– Optimistic– Most likely – Pessimistic

• This technique is applied to improve the accuracy of the estimates of cost or duration

• Three-point estimation techniques can be employed with other estimating techniques such as Delphi


% of Project Effort or Staffing

• Based on the premise that there is a predictable correlation between development effort/time and test effort/time

• The preferred/“dictated” method of estimation in some organizations

• Sometimes associated with developer/tester ratios

• One method recognized by the ISTQB


When Would We Use This Technique?When Would We Use This Technique?An organization is producing more or less the same type of product on an ongoing basis

The organization (development and testing) uses consistent practices and has a stable staff

A crude estimate is needed

It is mandated

It works

% of Project Effort or Staffing


Issues:Issues:

We are basing our estimates on development estimates which are probably suspect

Some products that are relatively easy to develop may be difficult to test and vice versa

This technique may not take into account test automation, etc.

This method may not take the stated quality goals into account

Test Estimates Based on Development Estimates


Class DiscussionWhat is the ideal developer/tester ratio?

= ?


McConnell Developer to Tester Ratio

Environment Relative Importance Ratio(Developer:Tester Ratio)

Business Systems 3:1 to 20:1

Shrink Wrap 1:1 to 5:1

Scientific Projects 5:1 to 20:1

Systems Projects 1:1 to 5:1

Safety Critical Software 5:1 to 1:2

Microsoft Windows 2000 1:2

NASA Space Shuttle Flight Control Software 1:10


Test Point Analysis

• A method with the possibility to perform a technology-independent measurement of the test size of an information system on the basis of a function point analysis, and to use this measurement as a basis for a productivity measurement, an estimate of the required resources, and project management

Software Testing: A Guide to the TMap ApproachMartin Pol, Ruud Teunissen, Erik van Veenendaal


Test Point Analysis (cont.)

For more details on TPA,look in this book.


Principles of Test Point Analysis

• TPA only considers measurable quality characteristics that fall within the range of system or acceptance testing

• TPA is (in principle) analyst independent• TPA depends on the availability of function points• Sufficient knowledge of the test team is a precondition• TPA estimates are based on the assumption that, on average,

one complete re-test will be conducted


Team Estimates

• Teams often produce better estimates than individuals (See The Wisdom of Crowds)

• Team estimates can facilitate buy-in and commitment from the entire team

• Almost all estimating methods can be done using teams


The Wisdom of Crowds

Four elements required to form a wise crowd:

• Diversity of opinion• Independence• Decentralization• Aggregation


Test Estimation Sessions ― Wideband Delphi


Test Estimation Sessions ― Wideband Delphi

• Developed by Barry Boehm and John Farquhar in the 1970s*

• A variant of the existing Delphi method but with more interaction

• Popularized in Boehm’s book Software Engineering Economics in 1981

• Similar to the Platinum Poker method used in some agile projects



Estimation for a new product or technology

To develop an “order of magnitude” estimate

Requirements are shaky or incomplete

Multiple disciplines are involved

Wideband Delphi


Test Estimation Sessions — Wideband Delphi

Steps from Barry Boehm’s book:

1. Coordinator presents each expert with a specification and estimation form

2. Experts discuss issues in group meeting3. Experts fill out form anonymously4. Coordinator distributes a summary5. In a group meeting experts discuss variances6. Experts fill out forms again for as many rounds as

necessary


Wideband Delphi Process Flow

Assemble estimates

and assumptions

Estimation meeting

Individual preparation

Planning and kickoff

meeting

Re-estimate

Acceptable

Done

No Yes

Adapted from Karl Wiegers


Sample Estimation Form — Wideband Delphi

Task Estimate #1 Estimate #2 Estimate #3 Estimate #4 Final

Change

Total


Discussion Items:Discussion Items:

What are story points?● Measure of size/complexity, not time● Relative unit of measure

Is there any relation to function points or test points?

Estimating Using Story Points ― Agile





• Story points are used for long term or release planning and tracking

• Point-estimated stories along with team velocity can be used to provide rough release-level scheduling and project progress

• Since story points are relative size indicators, a two-point story is always twice as big as a one-point story


Why Use Story Points?

• Cheaper• Allow us to change our minds as new

information becomes available• Don’t take a lot of time• Foster collaboration• Consistent• Provide credibility• Can use Planning Poker Cards with story points


Planning Poker


Planning Poker

• Based on the Wideband Delphi method espoused by Barry Boehm and others in the 60s and 70s (and Boehm’s work was possibly based on earlier works dating to the 40s)

• Most commonly used in agile software development

• A study published by the IEEE in April 2007 indicated that Planning Poker achieved less optimistic and more accurate estimates than those obtained through mechanical combinations of individual estimates


Planning Poker Rules

1. Form a group of no more than ten estimators and a moderator. The product owner is usually present but cannot estimate

2. Each participant gets a deck of cards. The decks frequently use a doubling sequence (½, 1, 2, 4, 8, 16, 32…) or the Fibonacci sequence (1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144…) or a “modified” Fibonacci sequence such as (0, 1, 2, 3, 5, 8, 13, 20, 40, 100)

3. The moderator reads one user story at a time to the team

4. The product owner answers questions about the story


Planning Poker Rules (cont.)

5. Each estimator privately selects a card from his or her deck representing their estimate of the “size” of the user story

6. When everyone is ready, all of the cards are flipped over at the same time

7. If there is a consensus, the results are recorded, and the team moves on to the next story

8. If the estimates vary widely, the owners of the high and low estimates defend their positions to the rest of the team


Planning Poker Rules (cont.)

9. The group briefly debates the arguments

10. Repeat from step 5 until the estimates converge

11. Repeat for all stories


Planning Poker Rules ― Tips

1. Those who actually could do the work are the ones who vote

2. Managers don’t vote

3. When there is a tie in the voting between two sizes that are consecutive, just pick the larger size and move on

4. Stop implementation discussions before they go too deep

5. Use an “I need a break” card


Planning Poker Rules ― Tips (cont.)

6. Use a timer to limit discussions

7. If consensus cannot be reached by the end of the third round of voting, pick the largest size and move on

8. Have the person creating the user stories meet with QA and Development leads prior to playing poker to answer the most obvious questions

9. Remember the baseline

10.Have fun!


T-Shirt Sizing

An estimation technique, usually used in agile projects, that uses relative sizing based upon a limited number of values (e.g., t-shirt sizes): S, M, L, XL

Steps:– Make S, M, L, XL cards– Stories are read one at a time– Each developer gives each story a t-shirt size– All developers raise their cards simultaneously– Discuss differences– Go back to step 3– Compile the stories into size buckets– Estimate the time to complete a S, M, L,XL


Hadden’s Size/Complexity Technique

Rita HaddenSmall Medium Large

Simple

Moderate

Complex

Complexity

Size


Sample Estimate


Test Objective Size Complexity Effort

1 to 3 1 to 3 Sessions 1 2 3to001 2 1 4 1 1 4 8to002 2 3 16 2 4 8 16to003 2 3 16 3 8 16 32to004 2 1 4to005 3 1 8to006 2 2 8to007 3 2 16Typical Usage Scenarios 2 2 8

total 80 sessions20 days

Siz

e

Complexity

Simple Defect Estimation Models

• Defect Detection Percentage (DDP)• Historical Extrapolation• “Bug Budgets”


Karl Wiegers’s Estimation Safety Tips

• A goal is not an estimate• The estimate you produce should be unrelated

to what you think the requester wants to hear• The correct answer to any request for an

estimate is “Let me get back to you on that”• Avoid giving single point estimates• Incorporate contingency buffers into estimates


Rick Craig’s Tips for Better Estimates

• Do it!• Collect metrics• Remember the “fantasy” factor• Don’t “pad” your estimates*• Don’t spend a ton of time• Estimates don’t have to be perfect

– Estimates are just estimates– They will change/constantly as you re-estimate– Remember planning risks and contingencies– Remember Brooke’s Law

• If the date is fixed, estimate something else• Use tools• Use ranges of value instead of discrete numbers


Robert Sabourin’s Tips for Better Estimates

• Consider estimating the amount of test coverage possible given a fixed amount of testing effort

• Present stakeholders with alternatives (Plan-a,Plan-b,Plan-9)

• Use ranges of values• Highlight business organizational and

technical factors which impact the estimate

• Using used envelopes to present rough estimates


Thank You

• On behalf of SQE Training, we appreciate your attendance at this course

• If you have additional training or consulting needs, please think of us


AppendixInstructor Material

Brainstorming Testing Estimates

• Min Max Missing– Identify participants– Hold kick off meeting– Individuals build list– Logging meeting– Estimate depth of testing– Estimate size in sessions– Identify new, missing, or

redundant ideas


BrainstormingTesting Ideas

• Min Max Missing– PMI Estimation– Given three values

● MIN = BEST CASE● MAX = WORST CASE● EXPECTED = NORMAL CASE● Estimate = (MIN + 4*EXPECTED + MAX)/6


Effort Estimation

ReadingSteve McConnell, Rapid Development: Taming Wild

Software Schedules. Chapter 8


Estimating Software Projects• Duration (on time)

– Schedule time– Delivery

• Quality (on quality)– Defect density– Defect arrival

• Cost (on budget)– Money

● Fixed● Variable

– Opportunity● What else could we be doing?


Estimating Software Projects

• Effort– Work– Manpower

• Staffing– Team definition– Skills, experiences


Estimation Process

McConnell Basic Steps:

1. Estimate size2. Estimate effort3. Estimate schedule


Estimation Process

1 - Estimate Size


Estimation Size

• SLOC– Source lines of code

• FP– Function points– Synthetic measure of software size


Estimation Size

• Software Tools– Calibration base on relevant experience– Combine several methods– Estimator (free!)

● www.construx.com


Estimation Size

• Analogy– Similar past projects– Relative to past experience– Must take into account

● Team● Technology● Complexity● Development process maturity


Estimating Size

• Algorithmic– Function points– Given requirements determine the number and

complexity of● Inputs● Outputs● Inquiries● Logical files● External interfaces

– Calculate and adjust based on “type of project/technology”


Function Points

Low Complexity Medium Complexity High Complexity

Program Characteristic count multiplier count multiplier count multiplier total

Inputs 6 3 2 4 3 6 44

Outputs 7 4 7 5 0 7 63

Inquiries 0 3 2 4 4 6 32

Logical Internal Files 5 7 2 10 3 15 100

External Interface Files 9 5 0 7 2 10 65

Unadjusted Function Point Total 304

Influence Multiplier 1.15

Adjusted Function Point Total 349.6


Function Points ― Influence Factors

1 Data communications How many communication facilities are there to aid in the transfer or exchange of information with the application or system?

2 Distributed data processing How are distributed data and processing functions handled?

3 Performance Did the user require response time or throughput?

4 Heavily used configuration How heavily used is the current hardware platform where the application will be executed?

5 Transaction rate How frequently are transactions executed daily, weekly, monthly, etc.?

6 On-line data entry What percentage of the information is entered on-line?

7 End-user efficiency Was the application designed for end-user efficiency?


Function Points ― Influence Factors

8 On-line update How many ILFs are updated by on-Llne transaction?

9 Complex processing Does the application have extensive logical or mathematical processing?

10 Reusability Was the application developed to meet one or many user’s (users’) needs?

11 Installation ease How difficult is conversion and installation?

12 Operational ease How effective and/or automated are start-up, back up, and recovery procedures?

13 Multiple sites Was the application specifically designed, developed, and supported to be installed at multiple sites for multiple organizations?

14 Facilitate change Was the application specifically designed, developed, and supported to facilitate change?


Function Points ― Influence Factors Score

Score Influence

0 Not present, or no influence

1 Incidental influence

2 Moderate influence

3 Average influence

4 Significant influence

5 Strong influence throughout


Function Points Influence Factors Score

INF = 0.65 + SCORE/100


Function Points ― Input

• Input– Number of screens, forms, dialogues, controls, or

messages through which an end user or another program adds, deletes, or changes data


Function Points ― Output

• Output– Screens, reports, graphs, or messages generated for

use by end users or other programs


Function Points ― Inquiries

• Inquiries– Direct access to data in database


Function Points ― Logical Files

• Logical Files– Major groups of end user data; could be a “file” or

“database table”


Function Points ― External Interfaces

• External Interface Files– Files controlled by other applications with which the

program interacts


Function Points ― Advantages

• Advantages include– Technology independent– Strong relationship to effort– Commonly used

• Disadvantages include– Requires specialized training to

compute


Estimation Process

2 - Estimate Effort


Effort Estimate

• Jones’s First-Order Estimation Practice

• Effort = (FP count) exponent


Jones’s First Order Exponents

Organization Capability

Kind of Software Best Average Worst

Systems 0.43 0.45 0.48

Business 0.41 0.43 0.46

Shrink Wrap 0.39 0.42 0.45


Estimation Process

3 - Estimate Schedule


Estimating Schedule

• Rule of Thumb (McConnell 8-1)

• Schedule = 3.0 * (effort) 1/3


Estimating Schedule

• Software Engineering Past Data– Example McConnell table 8-8, 8-9

● LOC vs. Schedule and Effort● Different project types


Estimation Convergence

Effort Range Schedule Range

Project Phase Optimistic Pessimistic Optimistic Pessimistic

Initial product definition 0.25 4.00 0.60 1.60

Approved product definition 0.50 2.00 0.80 1.25

Requirement specification 0.67 1.50 0.85 1.15

Product design specification 0.80 1.25 0.90 1.10

Detailed design specification 0.90 1.10 0.95 1.05

Product complete 1.00 1.00 1.00 1.00


Initialproductdefinition

Approvedproductdefinition

Requirementsspecification

Productdesignspecification

Detaileddesignspecification

Productcomplete

1.0×

0.25×

4×

2×

0.5×

1.5×

0.67×

1.25×

0.8×1.0×

0.6×

1.6×

1.25×

0.8×

1.15×

0.85×

1.1×

0.9×

Project Cost (effort and size) Project Schedule

Estimate ― Convergence Graph


More Relationships

Code Volume Approx. 100 LOC/FP, varies widely [Pressman, p. 94]

Schedule (FP)0.4Calendar months

Staffing (FP)/100Average – follows Raleigh curve

Effort Schedule * Staffing = (FP1.4)/150


Rules of Thumb


Rules of Thumb (cont.)

• Experience from several projects indicates that the amount of testing required in a project is proportional to the amount of development required

• Ratio depends on

– History– Organization, culture, structure– Projects– Teams


Rules of Thumb (cont.)

• Hadden’s Size/Complexity Technique

Small Medium Large

Simple

Moderate

Complex

Complexity

Size


Developer/Tester Ratios


Sample Estimate Project A


Test Objective Size Complexity Effort

1 to 3 1 to 3 Sessions 1 2 3to001 2 1 4 1 1 4 8to002 2 3 16 2 4 8 16to003 2 3 16 3 8 16 32to004 2 1 4to005 3 1 8to006 2 2 8to007 3 2 16Typical Usage Scenarios 2 2 8


Siz

e

Complexity

Sample Estimate Project B


Test Objective Size Complexity Effort1 to 3 1 to 3 Sessions 1 2 3

to001 1 1 1 1 1 4 8to002 1 1 1 2 4 8 16to003 2 1 4 3 8 16 32to004 2 1 4to005 2 1 4to006 2 1 4to007 1 1 1to008 1 1 1to009 3 2 16to010 2 2 8to011 2 2 8to012 2 2 8to013 2 2 8to014 3 2 16to015 3 2 16to016 2 2 8Typical Usage Scenarios 2 2 8


Complexity

Siz

e

Sample Estimate Project C



to001 3 2 16 1 1 4 8to002 1 1 1 2 4 8 16to003 1 1 1 3 8 16 32to004 1 1 1to005 1 1 1to006 1 1 1to007 1 2 4to008 1 2 4to009 1 1 1to010 1 2 4to011 1 2 4to012 2 2 8to013 2 2 8to014 2 2 8to015 1 1 1to016 1 1 1to017 2 1 2to018 2 2 8to019 3 2 16Typical Usage Scenarios 2 2 8

total 98 sessions24.5 days

Complexity

Siz

e

Sample Estimate Project D



t001 1 1 1 1 1 4 8t002 1 1 1 2 4 8 16t003 1 1 1 3 8 16 32t004 2 1 4t005 2 1 4t006 2 2 8t007 1 1 1t008 1 1 1t009 1 1 1t010 2 1 4t011 2 2 8t012 1 1 1t013 1 2 4t014 1 2 4t015 1 2 4t016 1 1 1t017 2 1 4t018 1 1 1t019 1 2 4t020 2 1 4t021 1 1 1t022 1 1 1t023 2 2 4t024 1 1 1t025 2 1 4t026 1 1 1t027 1 1 1t028 2 1 4t029 1 2 4t030 1 2 4t031 2 1 4t032 1 1 1Scenarios 2 2 8

total 99 sessions24.75 days

Complexity

Siz

e

Test Estimation in Practice

Technology

sqe training v3

software estimation

rob sabourin

contact rob

pm test estimation

introductiontest estimation

estimation techniques8

completion dates estimation