Agile Methods Cost of Quality: Benefits of Testing Early & Often

Agile Methods

Cost of QualityBenefits of Testing Early & Often

Dr. David F. Rico, PMP, ACP, CSM

Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com

LinkedIn: http://www.linkedin.com/in/davidfricoFacebook: http://www.facebook.com/profile.php?id=1540017424

Dave’s Agile Articles: http://davidfrico.com/agile-message.doc

Author Background DoD contractor with 28+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe

2

Published six books & numerous journal articlesAdjunct at George Wash, UMBC, UMUC, ArgosyAgile Program Management & Lean DevelopmentSpecializes in metrics, models, & cost engineeringSix Sigma, CMMI, ISO 9001, DoDAF, & DoD 5000Cloud Computing, SOA, Web Services, FOSS, etc.

Today’s Whirlwind Environment

3

OverrunsAttritionEscalationRunawaysCancellation

GlobalCompetition

DemandingCustomers

OrganizationDownsizing

SystemComplexity

TechnologyChange

VagueRequirements

Work LifeImbalance

InefficiencyHigh O&MLower DoQVulnerableN-M Breach

ReducedIT Budgets

81 MonthCycle Times

RedundantData Centers

Lack ofInteroperability

PoorIT Security

OverburdeningLegacy Systems

ObsoleteTechnology & Skills

Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press. Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.

Traditional Projects

4

Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled

Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.

Size vs. Quality

Def

ect

Den

sity

0.00

3.20

6.40

9.60

12.80

16.00

0 2 6 25 100 400

Lines of Code (Thousands)

Size vs. Productivity

Cod

e P

rodu

ctio

n R

ate

0.00

1.00

2.00

3.00

4.00

5.00

0 2 6 25 100 400

Lines of Code (Thousands)

Size vs. Requirements Growth

Per

cent

age

0%

8%

16%

24%

32%

40%

0 2 6 25 100 400

Lines of Code (Thousands)

Size vs. SuccessP

erce

ntag

e

0%

12%

24%

36%

48%

60%

0 2 6 25 100 400

Lines of Code (Thousands)

Global Project Failures

5Standish Group. (2010). Chaos summary 2010. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.

Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost

16% 53% 31%

27% 33% 40%

26% 46% 28%

28% 49% 23%

34% 51% 15%

29% 53% 18%

35% 46% 19%

32% 44% 24%

33% 41% 26%

0% 20% 40% 60% 80% 100%

1994

1996

1998

2000

2002

2004

2006

2008

2010

Year

Successful Challenged Failed

$0.0

$0.4

$0.7

$1.1

$1.4

$1.8

2002 2003 2004 2005 2006 2007 2008 2009 2010

Trill

ions

(US

Dolla

rs)

Expenditures Failed Investments

Requirements Defects & Waste

6Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.

Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all

Other 7%

Requirements47%

Design28%

Implementation18%

Defects

Always 7%

Often 13%

Sometimes16%

Rarely19%

Never45%

Waste

NetworkComputer

Operating SystemMiddlewareApplications

APIsGUI

How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success

7Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.

Agile Traditional1 2 3 Faster

Early ROI

Lower Costs

Fewer Defects

Manageable Risk

Better Performance

Smaller Attack Surface

Late

No Value

Cost Overruns

Very Poor Quality

Uncontrollable Risk

Slowest Performance

More Security Incidents Seven Wastes1.Rework2.Motion3.Waiting4.Inventory5 .Transportation6.Overprocessing7 .Overproduction

MINIMIZES MAXIMIZES

JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes

Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success

What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process The goal of agile testing is to deliver early and often

8Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

Traditional Testing

Combining source files

Combining software and environment

Combining software and data

Combining software and tests

Combining developers

Agile Testing

Code is frequently checked in

Code is automatically retrieved

Compilation is done automatically

Tests are done automatically

Code reports are generated

Developers get instant feedback

Code is automatically deployed or packaged for delivery

Thousands of TestsContinuously Executed

No More Late BigBang Integration

Agile Testing Model User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur

9Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.

BuildIntegration

Server

VersionControlServer

BuildScripts

UsesWatches

BuildStatus

ProvidesDeveloper A

Developer B

Developer C

CommitsChanges

CommitsChanges

CommitsChanges

Builds

Database

Analysis

Testing

Reporting

Documentation

Deployment

Early, Automated, Fast,Efficient, & Repeatable

Constant ReadinessState & CM Control

Lean, Waste Free, Low WIP,No Deadlocked Test Queues

Rapidly & SuccessfullyDev. Complex Systems

Agile Testing Done Early & Often Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project

10Maeda, M. K. (2009). Agile testing: Early, often, and Smart. Arlington, MA: Cutter Consortium.

Agile Testing Practices Agile testing consists of seven broad practices Includes automated builds, testing, inspections, etc. Also includes reporting, documentation, deployment, etc.

11

Practice

Building

Database

Inspections

Testing

Feedback

Documentation

Deployment

Description

Frequently assembling products and services to ensure delivery readiness

Frequently generating/analyzing database schemas, queries, and forms

Frequently performing automated static analysis of product/service quality

Frequently performing automated dynamic product and service evaluation

Frequently generating automated status reports/messages for all stakeholders

Frequently performing automated technical/customer document generation

Frequently performing automated delivery of products/services to end users

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

Agile Testing Workflow

12

Traditional vs. Agile Cumulative Flow

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Traditional Cumulative Flow Agile Cumulative Flow

Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP and lead times

Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

Agile Testing Costs & Benefits

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.

Most agile testing tools are “free” open source A build server is no more than a commodity PC 10x more efficient/effective than traditional testing

13

Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes

14Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Agile Cost of Quality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often

15Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Agile Testing Statistics Fewer builds leave in higher bug counts A high number of builds eliminates the defects Goal is to have as many, early builds as possible

16Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.

Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with proper attention

17Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

Wide Impact Tuning

Fast builds – less changes – more green

Parallelization of test runs

ClearCase to subversion

Pre-installing as much as possible

Removal of randomness

Compilation in memory

Installation starting parallel with system build

Focused Impact Tuning

More memory and CPUs

Parallelize builds

Replace 3rd party test libraries

Reduce/remove timeouts in tests

Select different tests

Refactor code & components

Tune the network & software

Tune the database

Agile Cost & Benefit Analysis Costs based on avg. productivity and quality Productivity ranged from 4.7 to 5.9 LOC an hour Costs were $588,202 and benefits were $3,930,631

18Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation. Ft. Lauderdale, FL: J. Ross Publishing.

d1 = [ln(Benefits Costs) + (Rate + 0.5 Risk2) Years] Risk Years, d2 = d1 Risk Years

5

1i

Benefits of Agile Methods Analysis of 23 agile vs. 7,500 traditional projects Agile projects are 54% better than traditional ones Agile has lower costs (61%) and fewer defects (93%)

Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.

Project Cost in Millions $

0.75

1.50

2.25

3.00

2.8

1.1

Before Agile

After Agile

61%LowerCost

Total Staffing

18

11

Before Agile

After Agile

39%LessStaff

5

10

15

20

Delivery Time in Months

5

10

15

20

18

13.5

Before Agile

After Agile

24%Faster

Cumulative Defects

625

1250

1875

2500

2270

381

Before Agile

After Agile

93%Less

Defects

19

Agile vs. Traditional Success Traditional projects succeed at 50% industry avg. Traditional projects are challenged 20% more often Agile projects succeed 3x more and fail 3x less often

Standish Group. (2012). Chaos manifesto. Boston, MA: Author.

20

Agile Traditional

Success42%

Failed9%

Challenged49%

Success14%

Failed29%

Challenged57%

Hoque, F., et al. (2007). Business technology convergence. The role of business technology convergence in innovation and adaptability and its effect on financial performance. Stamford, CT: BTM Institute. 21

Study of 15 agile vs. non-agile Fortune 500 firms Based on models to measure organizational agility Agile firms out perform non agile firms by up to 36%

Benefits of Organizational Agility

Agile Recap Agile methods DON’T mean deliver it now & fix it later Lightweight, yet disciplined approach to development Reduced cost, risk, & waste while improving quality

22Rico, D. F. (2012). What’s really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdfRico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdfRico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf

What How ResultFlexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process

Customer Involve customers early and often throughout development Early feedback

Prioritize Identify highest-priority, value-adding business needs Focus resources

Descope Descope complex programs by an order of magnitude Simplify problem

Decompose Divide the remaining scope into smaller batches Manageable pieces

Iterate Implement pieces one at a time over long periods of time Diffuse risk

Leanness Architect and design the system one iteration at a time JIT waste-free design

Swarm Implement each component in small cross-functional teams Knowledge transfer

Collaborate Use frequent informal communications as often as possible Efficient data transfer

Test Early Incrementally test each component as it is developed Early verification

Test Often Perform system-level regression testing every few minutes Early validation

Adapt Frequently identify optimal process and product solutions Improve performance

Conclusion

23

Agility is the evolution of management thought Confluence of traditional and non-traditional ideas Improve performance by over an order of magnitude

“The world of traditional methods belongs to yesterday”“Don’t waste your time using traditional methods on 21st century projects”

Agile methods are …

Systems development approachesNew product development approachesExpertly designed to be fast and efficientIntentionally lean and free of waste (muda) Systematic highly-disciplined approachesCapable of producing high quality systemsRight-sized, just-enough, and just-in-time tools

Scalable to large, complex mission-critical systems Designed to maximize business value for customers

Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.

Books on ROI of SW Methods Guides to software methods for business leaders Communicates business value of software methods Rosetta stones to unlocking ROI of software methods

http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)

24