Software Quality Costs

7/27/2019 Software Quality Costs

1/12

4 SQP VOL. 15, NO. 2/ 2013, ASQ

In January, ASQ Quality Press publishedPrinciples of Quality Costs: Financial Measures for Strategic Implementation of Quality Management, Fourth Edition,Douglas C. Wood,editor. The section on cost of software qualityexcerpted here contains contributions from anumber of authors who have appeared over theyears inSoftware Quality Professional.

This summary treatment provides an overviewand context for applying the well-establishedcost-of-quality approach to software devel-opment management. By providing (oftensurprising) data on rework and other failure-related costs, the method supports moreobjective decision making on investments inquality assurance and process improvement.

Key wOrdSCost of quality models, economics, maturitylevels, project management, software metrics,software process improvement

CMMI is registered in the U.S. Patent andTrademark Office by Carnegie Mellon University.

S O F T w A r e Q U A L I T y M A N A G e M e N T

Soft a

QualitCostsEditEd by taz daughtrEy

INTrOdUCTIONSo tware is signi cant in that it is both a product and a meanso delivering a product or service. So tware delivers the mostimportant product, namely in ormation. So tware capturesand connects in ormation to link customers to products andservices and localities to global resources. So tware also actsas a control mechanism or systems, transactions, individualprojects, and enterprisewide activities.

So tware consists o the computer programs, associated pro-cedures, documentation, and data pertaining to the operation o a computer system. As a purely intellectual product, it is amongthe most labor-intensive, complex, and errorprone technologiesin human history. Di erentiated rom the hardware, the physicalmachines on which so tware works, there are many types o so tware products and systems in the world today. These areo ten characterized either by business domains ( or example,in ormation systems) or by relationship to other so tware layeredbetween the user and the computing machinery ( or example,programs embedded in microprocessors to create intelligentdevices and appliances).

The deVeLOPMeNT ANdeCONOMICS OF SOFTwAreTo highlight its distinctive characteristics, it is help ul tocontrast the development or engineering o so tware with thato manu actured products.

So tware does not consist o physical components, soit does not align with the Weibull bathtub curve.So tware does not wear out; its obsolescence isrefected in the currency o its design and its interactionwith other so tware.


2/12

Software Quality Costs

www.asq.org 5

So tware is predominantly a customizedcreation, which is not replicated but enhancedor optimized with successive versions.

So tware is an intellectual, rather than a

physical, product; its development is subjectto human and logical constraints rather thanphysical laws.

One cannot assume that a so tware speci ca-tion is stable. Changing o requirements isexpected behavior in so tware development.

Productivity levels vary widely (more so inindividuals than in teams).

Product de ects are the result o poor in ormation,limited in ormation, human misunderstandings,and human mistakes but not de cient materials.

Manu actured goods are valued or their ea-tures; so tware is valued or its interactive

unctionality and non unctional attributes. The economics o so tware quality hinges on

the process o understanding requirements. This process, more so than con ormance,

is commonly responsible or the value o aso tware product.

The costs o the manu acturing phase o so t-ware production are insigni cant. The bulk o so tware development costs are in the require-ments elicitation and speci cation, design,implementation, and testing disciplines.

Statistics cannot be applied to replicationsbecause so tware products are usuallyone o a kind.

The cost o ownership is accounteddi erently because so tware assetsare not capitalized and depreciated.

whAT IS SOFTwAreQUALITy? Although there is no single comprehensiveand complete standard de nition o so twarequality, descriptions and terms are oundin sources such as consensus national andinternational standards (ISO/IEC 9126,ISO/IEC 90003, IEEE So tware EngineeringStandards) and various books on the subject(Figure 1).

ISO/IEC 9126 is a standard that identi es andcaptures quality attributes or computer so tware. Itde nes the major so tware quality attributes, distin-

guishing these as unctionality, reliability, usability,e ciency, maintainability, and portability (Figure 2).Since so tware is used to support, enable, or auto-

mate pro essional services, it is reasonable to applythe service quality models when evaluating the valueand e ectiveness o a so tware solution when used ina service context (Table 1).

To summarize, it is important to rein orce the basicprinciples. The ollowing are de nitions o so twarequality rom several high-level traditional qualityperspectives:

Level of satisfaction. The degree to which acustomer or user perceives that a so twareproduct meets his or her composite needsand expectations

2 0 1 3

, A S Q

ProductRevision ProductTransition

Product Operation

PortabilityReusabilityInteroperability

MaintainabilityFlexibilityTestability

UsabilityReliability Integrity

Correctness Ef ciency

FigurE 1 Software quality attributes

2 0 1 3

, A S Q

ISO/IEC9126

Are the required functionsavailable in the software?

How reliable isthe software?

How easy is it

to transfer thesoftware to anotherenvironment?

How easy is itto modify the

software?

Maintain-ability

Functionality

Pro tability Reliability

UsabilityIs the software

easy to use?Ef ciency

How ef cient isthe software?

FigurE 2 ISO/IEC 9126 software quality attributes


3/12


6 SQP VOL. 15, NO. 2/ 2013, ASQ

Quality costs can be compared to benchmarksand norms.

Deeper analysis can lead to actions to improvethe competitive situation.

The bottom-line e ect o improvement actionscan be measured.

The economic tradeo s involved with so tware

quality become visible, thus leading to betterdecision making.

APPLyING COST OF QUALITyPrINCIPLeS TO SOFTwAreTable 2 lists the our categories typical when applyingcosts o so tware quality. Most prevention costs arein the development cycle, except or organizationwidecosts such as process de nition and metrics collectionand analysis.

Product value. The value o a so twareproduct relative to its various stake-holders and the competition

Key attributes. The extent to which aso tware product possesses a desiredcombination o properties

Freedom from defects. The degreeto which a so tware product workscorrectly in target user environments,

ree rom operational faws Process quality. In relation to the

development process by which theproduct is produced, the extent towhich people do the right things inan e ective way

why IS The COST OFSOFTwAre QUALITy (COSQ)IMPOrTANT NOw?I improving business success through so tware quality

is a corporate goal (Figure 3), then we need answersto a ew simple questions that are not o ten askedin todays so tware development situations. Thesequestions are:

How much does so tware quality cost? What are the bene ts o good quality so tware? How good is our so tware quality?

Once the answers to these questions are known: Quality costs can be compared to overall so tware

production costs and so tware product sales.

tabLE 1 Service quality model applied to software

2 0 1 3

, A S Q

Se v ce Q l d mens on Ex mples relev nce o Sof w e

Image , expec tations Appropriate to sof tware use, a lignmentto service model

Product look and feel, replication of subjectmatter expertise

Reliability Complete transactions, fulfillment of commitments, success on first attempts

Confirmation numbers after transactions,traceability path, audit records, logs

Responsiveness Prompt completion of steps, online oravailable assistance

Usability, online help, frequently asked questions(FAQ) section

Assurance Instill confidence, make software usersfeel safe

Certification, security

Empathy Personalized customer care, retention of customer details

Customization, independent configuration

Tangibles Visually appealing, convenient 24/7 uptime and availability

2 0 1 3

, A S Q

Improve business successthrough better quality software

Questions

Measures

Business goal #1

How muchdoes softwarequality cost?

CoSQ and

related ratios

What are thebene ts of quality software?

How goodis our softwarequality?

Problems

Defectiveness

Pro tReputation

Customersatisfaction

Marketshare

FigurE 3 Cost of software quality (CoSQ) in context


4/12


www.asq.org 7

process improvement initiative, with each increase inmaturity level typically requiring as much as two ormore years o concerted e ort.

There has been longstanding speculation aboutthe nature and extent o reductions in quality costsresulting rom organizational matur ity improvements(Knox 1993). Knox developed a theoretical model orthe cost o so tware quality (Figure 5). Other studieso quality improvement (Jones 2011) have reported a

variety o measures without allowing direct calculationo cost o quality categories. For example, Table 3

COST OF SOFTwAreQUALITy ANd CMMICapability Maturity Model Integration (CMMI) is a pro-cess improvement approach developed by the So twareEngineering Institute (see www.cmmiinstitute.com.).Originally developed as the Capability Maturity Model

or So tware, the ull set o industry best practicesnow includes CMMI models speci c to acquisition,

development, and services.Key practices include causal analysis, con guration

management, data collection, proj-ect management, quality assurance,requirements engineering, riskmanagement, and veri cation andvalidation. Implementation o thesepractices has led to demonstrableorganizational improvements inmany measures o per ormancesuch as schedule and cost predict-

ability, product and service quality,productivity, customer satis action,and return on investment.

The basic CMMI ramework isbuilt around the idea o levels o maturity, each o which can beobtained only a ter success ullyimplementing all key practices o the previous level. The ve-levelmodel (Figure 4) represents a road-map or targeting and sustaining a

tabLE 2 Typical costs of quality for software

2 0 1 3

, A S Q

Cos a e desc p on t p c l cos s

Cost of control orconformance

Preventioncosts

Defect avoidance; qualitybasis definition; project andprocess-oriented interventions

Efforts to define quality and set quality goals, standards, andthresholds; quality trade-off analysis; definition of releasecriteria for acceptance testing and related quality standards;training; process metric creation and planning; formal inspection

Appraisalcosts

Defect detection; discoveryof product nonconformance;finding the level of nonconformance

Quality control gating processes, contract or proposal reviews,quality audits, go/no-go decisions, quality assurance of subcontractors, inspections, static/dynamic analysis, testing,walk-through, desk-checking

Cost of failureof control ornonconformance

Internalfailurecosts

Pre-release defect or anomalycorrection prior to delivery tothe customer

Recode, retest, re-review, redocument, requirements rework,design rework

Externalfailurecosts

Post-release defect or anomalycorrection and related costsafter delivery to the customer

Warranty support, resolution of complaints, reimbursementdamage paid to customer, domino effect on other projects,damage to reputation or enterprise, added marketing to correctreputation problems, penalties

2 0 1 3

, A S Q

Continuous process management Defect prevention process Continuous technology management

Quantitative goals for projects and process

Track goals by metrics and statistical analysis Reuse among the different projects

Organizational standard process established Standard process owned by the organization Project-speci c tailoring

Disciplined project management Process owned by project manager Process varies from project to project

Process unde ned; ad-hoc methods Success depends on few specialists Costs, quality, and deadline not always met

Quality

Risk

Characteristics: Bene ts:Level:

5

Optimizing

4

QuantitativelyManaged

3

De ned

2

Managed

1

Initial

FigurE 4 Basic CMMI framework


5/12


8 SQP VOL. 15, NO. 2/ 2013, ASQ

shows de ect injection and removal rates.(The economics are obscured becausewhat Jones calls e ciency values areactually e ectiveness percentages, and

thus without cost considerations.) A ew repor ted case studies, however,have con rmed suspicions about qualitycost bene ts rom such process improve-ments. The classic rst sets o data werereported in the early and mid-1990s romRaytheon Electronics Systems Group(RES) (Dion 1993; Haley 1996).

Starting at CMMI Level 1, RES intro-duced a so tware process improvementprogram in August 1988. Using the results

o tracking 15 projects, they achieved CMMILevel 3 practices in a little over three years(see Figure 6).

In the Level 1 stage, RESs total CoSQ fuctuatedbetween 55 percent and 67 percent o total project costs,and by the time it reached Level 3 process maturity,their total CoSQ had dropped to approximately 40percent. By 1996, this organizations total CoSQ wasapproximately 15 percent o total project costs andthe rework due to both internal and external noncon-

ormances had been reduced to less than 5 percent.

A third source o CoSQ data is a Price Waterhousestudy (Price Waterhouse 1998) that analyzed the costsand bene ts o so tware quality standards rom asurvey o 19 United Kingdom (UK) so twaresuppliers. The study estimated the cost o con ormance (prevention and appraisalcosts) to be 23 percent to 34 percent o development e ort. The study also estimatednoncon ormance ( ailure) costs at 15 percento development e ort or a total CoSQ o 38percent to 49 percent o development e ort.

It must be noted that this study excluded thecosts o unit testing and rework because thesuppliers could not separate these costs. Withincreases in the estimates to account orthis exclusion, CoSQ in a so tware organiza-tion with a quality system can range rom40 percent to 55 percent o developmentcosts, with a ratio o con ormance costs tononcon ormance costs ranging rom 1.5 to 2.

As compared to the RES graph, thesegures generally agree with a period late in

2 0 1 3

, A S Q

C o s t a s a p e r c e n

t o

f d e v e

l o p m e n

t

SEI CMM Level

60

50

40

30

20

10

0

Prevention

Appraisal

Internalfailure

Externalfailure

TCoSQ

1 2 3 4 5

FigurE 5 Knoxs theoretical model for cost of software quality (Knox 1993)

tabLE 3 Software quality and the SEICapability Maturity Model (CMMI)for projects of 5,000 functionpoints in size (Jones 2011)

2 0 1 3

, A S Q

CMMi leveldefec

po en l pef nc on po n

defecemov l

eff c enc

del ve eefec s pe

f nc on po n

SEI CMMI 1 5.50 73% 1.49

SEI CMMI 2 4.00 90% 0.40

SEI CMMI 3 3.00 95% 0.15

SEI CMMI 4 2.50 97% 0.08

SEI CMMI 5 2.25 98% 0.05

2 0 1 3

, A S Q

70

60

5040

30

20

10

01987 1988 1989 1990 1991 1992 1993 1994 1995 1996

Start of CMM Level 1 CMM Level 3

Rework Cost of conformance(appraisal and prevention)

Appraisal Prevention

FigurE 6 Cost of software quality for 15 projects atRES (Haley 1996)


6/12


www.asq.org 9

Provide cost data or motivational purposesby demonstrating the relationship o employeee orts to the bottom line

Provide a basis or budgeting the quality

operation Compare and identi y the most cost-e ective

process improvements Provide a measure or comparing the success

o projects Identi y quality improvement candidates

eLeMeNTS OF A COSQ eFFOrTThe overall process o measuring and using cost o quality in ormation speci cally or so tware process

improvement is shown in Figure 7. One key input tothe process is de nition o a project work breakdownstructure (WBS). The total process involves gathering

1990 when RES was approaching CMMI Level 3: RESsCoSQ was about 45 percent o development, and itsratio o con ormance to noncon ormance costs wasabout 2.0 (con ormance ~30/noncon ormance ~15).

Based on the data in Figure 6, we can expectCoSQ, with the present state o so tware engineeringpractice, to range rom 20 percent to 70 percent o development costs. Even accounting or the marginbetween production costs and sales, CoSQ appears tobe roughly twice manu acturing CoQ.

More recently, Gibson, Goldenson, and Kost (2006)have cataloged measured decreases in internal ailurecosts or maturity levels 1 through 4 (see Table 4).

The same report also related that an anonymousRaytheon level 3 site had reduced its costs o rework

by 42 percent over several years. Similarly, a SiemensIn ormation Systems organization reduced cost o poor quality rom more than 45 percent to less than30 percent over a three-year period as they movedtoward level 5.

A. Q. Liu (2007) shared results rom 191 projectsconducted as the Motorola So tware Group ChinaCenter upgraded to CMMI rom 2000 to 2006. Due toimproved appraisal processes, particularly product peerreview and so tware test processes, the organization wasable to reduce its overall cost o quality by more thanone-third rom its pre-CMMI baseline and noted thatthe cost o poor quality (presumably ailure costs)remained at less than 5 percent during the same period.

POTeNTIALBeNeFITS OFUSING COSQThe work at RES demon-strates one o the bene ts o

measuring and using CoSQ, justi cation or quality ini-tiatives. Examples o returnon investment (ROI) inquality improvement initia-tives speak most clearly tomanagers responsible ormaintaining a pro table orga-nization. However, CoSQ canbe used or a number o otherbene ts. It can:

tabLE 4 Rework as percentage of totalproject costs versus CMMI level(Gibson, Goldenson, and Kost 2006)

2 0 1 3

, A S Q

Level Pe cen e

1 41

2 183 11

4 6

2 0 1 3

, A S QExit criteria

CoQ project report approvedEntry criteria

Project plan approved

TasksInputs Outputs

Project WBS

CoQ measurement process

Typical CoQ list of tasks

Weighting rules

CoQ data model

CoQ report

Findings

Recommendations

1. Identify CoQ tasks, using project planand WBS

2. Record CoQ effort by category3. Apply weighting rules4. Compute the CoQ5. Develop ndings and recommendations6. Review ndings and recommendations

with management7. Produce nal CoQ project report8. Archive report

MeasuresEffort (staff-hours)

FigurE 7 Top-level modified ETVX description of the process tomeasure the CoSQ (Laporte et al. 2012)


7/12


10 SQP VOL. 15, NO. 2/ 2013, ASQ

tabLE 5 Typical cost items according to the proposed CoSQ model (examples) (Galin 2004)

2 0 1 3

, A S Q

bo n eo n on

espons le fo

q l cos

Cl ss of cos s of sof w e

q l

t p c l q l cos ems fo e c s cl ss

Softwaredevelopmentand SQA teams

Control costs Prevention costs: Investments in development of and updating of procedures and work instructions Investments in development and regular operation costs of software configuration

management system Investments in development and regular operation of software quality metrics Instruction of new employees in those SQA subjects and procedures required by their position Certification of employees for positions that require special certification Internal and external quality audit

Appraisal costs: Performance of various reviews Performance of unit, integration, and software system tests

Quality assurance of subcontractors and other external participantsSoftwaredevelopmentteams

Failure of control costs

Internal failure costs: Redesign or corrections subsequent to design review and test findings Reprogramming or correcting programs in response to test findings Repeated design review and retesting (regression tests)

External failure costs: Resolution of customer complaints during the warranty period Correction of software defects detected during regular operation Damages paid to customer in case of a severe software failure Reimbursement of customers purchase costs, in case of total dissatisfaction Insurance premium against customer damage claims in case of severe software failure Hidden external failure costs, that is, reduction of sales as a result of damaged reputation,

increased investments in sales promotion under-pricing of tender bidding to counter theeffects of past software external failures

Management Control costs Managerial appraisal and control costs Costs of conduct of contract reviews Costs for preparation of project and quality plans and their periodic updating Costs for performance of regular progress control


Internal managerial failure costs: Unplanned development costs resulting from underestimation of resources for submitted

proposals Domino effect: Damages to other projects planned to be performed by team members

involved in delayed projects due to extra costs for recruitment of replacement team members*


External managerial failure costs: Damages paid to customer as compensation for late project completion resulting from an

unrealistic schedule presented in proposal Damages paid to customer as compensation for late project completion resulting from

failure to recruit sufficient and appropriate team members Domino effect: Damages for delayed completion paid to clients of other projects planned to

be performed by team members involved in delayed projects* Hidden external failure costs, that is, reduction of sales as a result of damaged reputation,

increased investments in sales promotion under-pricing of tender bidding to counter theeffects of significant past delayed completion of projects due to managerial failures inappraisal and/or progress control tasks

*Note: These damages should be considered managerial failure costs belonging to the original project, whose scheduling problemsinterfered with the progress of other projects. Should they materialize, one can expect the domino effect to obstruct the progressof several other projects and induce considerable hidden external failure costs.


8/12


www.asq.org 11

data rom sources in accounting and quality.Subsequent steps may include collection o data rom new quality metrics, data analysis,and presentation o results.

AccountingGetting quality cost data assumes that costshave been accounted or using task andexpense elements that can be summed intothe our major categories o quality costs. Many so tware organizations track costs ina manner amenable to quality costing, butmany others do not. In the latter case, apreliminary step o de ining and installingsuch a chart o accounts is required. A sample o such a chart o quality costs isprovided in Table 5. The quality elements ina so tware organizations chart o accountsmust be tailored to re lect its so twareprocess. To realize the ull bene it o CoSQ,it must also allow or the addition o processimprovement tasks.

In best cases, quality costs can be takendirectly rom departmental accountingreports. In other cases, it may be necessaryto resort to basic accounting and engineeringrecords such as schedules, time reports,de ect reports, and purchasing records. Inworst cases, one may all back on interviewswith members o the so tware organization inorder to construct estimates o each qualitycost element. A controlled scienti ic studyis unlikely, and incomplete data can su icein beginning a so tware cost bene it analy-sis. With time and greater understanding,incomplete data can be illed in.

One o the pit alls o a CoSQ programhas to do with controversial cost elements.Usually the question is about which costsare normal operating costs and which arequality costs. An example would be thecost o producing a project managementplan. Although this plan is produced or thesake o managing a projects expenses andschedule, it also infuences product andprocess quality. It is help ul to keep in mindthe ollowing points:

tabLE 6 Sample CoSQ chart (Houston 1999)

2 0 1 3

, A S Q

1 P even on Cos s

1.1 Requirements1.1.1 Marketing research for customer/user quality needs1.1.2 Customer/user quality surveys1.1.3 Product quality risk analysis1.1.4 Prototyping for customer review1.1.5 User requirements/specification reviews/inspections

1.2 Project1.2.1 Project quality planning1.2.2 Project process validation1.2.3 Quality assessment of development platform and tools1.2.4 Platform and tools development for quality1.2.5 Developer quality training1.2.6 Quality metrics data collection1.2.7 Design for quality: software component reuse1.2.8 Formal inspections/peer reviews1.2.9 Project configuration management1.2.10 Project change management1.2.11 Supplier capability assessment

1.3 Reuse library1.3.1 Salaries1.3.2 Expenses1.3.3 Training1.3.4 Platform and tools

1.4 Configuration management administration1.4.1 Salaries1.4.2 Expenses

1.4.3 Training1.4.4 Platform and tools

1.5 SQA administration1.5.1 SQA salaries1.5.2 SQA expenses1.5.3 Software process and standards definition and publication1.5.4 Metrology: data maintenance, analysis, and reporting1.5.5 SQA program planning1.5.6 SQA performance reporting1.5.7 SQA education/training1.5.8 Process improvement1.5.9 SQA process compliance audits

2 app s l Cos s2.1 Supplied product testing2.2 Project appraisal costs

2.2.1 Verification and validation activities2.2.2 Testing: planning, platforms, setup, test data generation,

test execution and logging, reporting, test data evaluation2.2.3 Product quality audits

2.3 External appraisals2.3.1 Process maturity evaluation2.3.2 Field performance trials2.3.3 Special product evaluations


9/12


12 SQP VOL. 15, NO. 2/ 2013, ASQ

So tware metrics are used to make timely, data-driven decisions, track progress, and evaluate the

impact o changes to products, processes, andresources. Because so tware is not tangible, metricscan be either predicted or calculated and o tenrepresent subjective impressions as objective ordinalvalues. Metrics are e ective when linked to goals andobjectives, and they serve as evidence respondingto the questions surrounding the completion o particular goals. For example, i an organization hasan explicit goal o having no open critical de ects onrelease, an appropriate metric used to evaluate goal

ul llment would be the quantity o open critical

Arguments over controversial elementshave been known to sabotage cost o quality programs.

The largest quality costs are those that

are most easily discerned (reviews,so tware quality assurance, testing, andrework). There ore, it is o ten sa e toexclude controversial elements withoutunduly a ecting the total CoSQ.

Consistency throughout a CoSQ pro-gram is more important than thoroughinclusion o quality costs because con-sistency allows or clear identi cationo improvements as well as candidates

or improvement.

Concerns may also arise as to how qualitycosts should be categorized. Again, consis-tency is important. For example, in Table 6the costs associated with ormal inspections(peer reviews) are treated as preventioncosts rather than as appraisal costs. Thisis a matter o interpretation, depending onwhen a work product is considered ready orappraisal. In manu acturing, inspections areconducted on pieces a ter they are produced;in so tware production, inspections may beincorporated into the production processper the peer review key process area or theSEI CMMI Level 3. For documentation, thismeans that a document is not complete untilit has undergone a peer review and has beenrevised. The same is true or code, especiallywhen code inspections precede unit testing;clearly an appraisal activity.

Qualit M t ics Coll ctionThe CoQ has been used primarily in a undamentalapproach to quality, that is ocusing on de ect rates(manu acturing) or service problem reports (serviceindustries) rather than broader approaches thatwould take into account actors such as usability,testability, maintainability, and so orth. The unda-mental approach has the advantages o straight orwardmeasurement and ease o understanding. It also allowscomparison o dissimilar products. Most so twareproducers take a undamental approach to quality.(See Figures 8 and 9, and Table 5 or examples.)

3 in e n l F l e Cos s

3.1 Product design defect costs3.1.1 Causal analysis and reporting3.1.2 Design corrective action3.1.3 Rework and retest due to design corrective action3.1.4 Work products wasted due to design changes

3.2 Purchased product defect cost3.2.1 Defect analysis cost3.2.2 Cost of obtaining product fix3.2.3 Cost of defect work-arounds3.2.4 Rework

3.3 Implementation defect costs3.3.1 Defect measurement and reporting3.3.2 Defect fixing3.3.3 Causal analysis and reporting3.3.4 Project process corrective action3.3.5 Fix inspection3.3.6 Retest and integration

4 Ex e n l F l e Cos s

4.1 Technical support for responding to defect complaints4.2 Product returned due to defect4.3 Maintenance and release due to defects4.4 Defect notification costs4.5 Upgrade due to defect4.6 Service agreement claims (warranty expense reports)4.7 Litigation costs and liability claims (insurance and legal reports)4.8 Penalties (product contract reports)4.9 Costs to maintain customer/user goodwill due to dissatisfaction

(sales reports)4.10 Lost sales/market share due to quality problems (field salesperson

reports)

So ces of q l cos :Ordinarily, quality cost data for the majorityof categories would be obtained from salary and expense reports. Exceptionsin the external failure category are shown here in parentheses.

tabLE 6 Sample CoSQ chart (continued)

2 0 1 3

, A S Q


10/12


www.asq.org 13

de ects relative to total de ects ata particular development milestone(beta test milestone).

Table 7 includes survey data col-

lected in graduate so tware engineeringcourses (Laporte et al. 2012) rom engi-neers and managers working in a widerange o application domains. Most o the industrial data were collected intwo large multinational enterprises:one involved in the transportationsector and the other in the aerospacesector. The numbers in parenthesesindicate the number o individualsresponding to each survey.

A system or collecting both de ectmetrics and the ailure portion o qual-ity costs would provide data havingthe added advantage o aiding in rootcause analysis and o identi ying themost costly de ects.

In order to avoid reporting con-tradictory or irrelevant comparisonsbetween the di erent types o so t-ware and programming methods ( orexample, comparing nuclear-based

medical so tware with a telephonegaming app), the total cost o owner-ship (TCO) is a good place to start ormeasuring CoSQ improvements.

TCO permits the cost o qualityinitiatives to link back to lean and agilemethods or so tware development andmake use o a value stream in so twareengineering to distinguish value-addedactivities (that is, development based onapproved requirements) rom nonvalue-added activities ( or example, ndingand fxing de ects, realignment o miscon gured or incorrectly labeledmodules or components). Examples o wastes that would infate TCO withoutadding value include:

Rework to address missing, incor-rect, or extra code

Unassigned development backlogwaiting or ownership 2 0

1 3

, A S Q

P ojec c e co esCodes for charging time to project activities are 10 characters: TTPPPPANNQTT = Two characters for process type, for example SW for softwarePPPP = Four characters that designate a projectANN = Three characters that designate the activity in the WBS, such as A01Q = One character that designates the category relative to quality costs.

The value of Q is one of the following:D Initial development of work productsP Defect prevention activitiesT Initial test of work productR Product reworkM Miscellaneous/other

Sof w e WbS c v es/CoSQ mo f e sThe following chart indicates what types of categories with regard to quality costs

may be used for each WBS activity. For example, in requirements development,initial development of requirements document would be D, validation of requirements with a customer would be P, reworking requirements with acustomer would be P, and reworking a requirements document would be R.

D P T R M

A01 Project management planning

A02 Requirements development/validation

A03 Product concept development

A04 Requirements analysis

A05 Functional concept development

A06 Define architecture

A07 Test-plan development

A08 Software configuration management (SCM) planning

A09 Detailed functional specification

A10 Detailed design, code, debug

A11 Unit testing

A12 Integration testing

A13 System test scripts

A14 System test specification

A15 User information development

A16 User evaluation

A17 Final test

A18 Beta test

A19 Project plan maintenance and updates

A20 Project tracking and metrics

A21 Software quality assurance

A22 Subcontract management

A23 SCM builds

Other SCM functions

A24 Requirements and change management

A25 Establish/maintain development environment

A26 Miscellaneous activities

FigurE 8 A software project charge scheme using an activity-based WBS with CoSQ modifiers (Houston 1999)


11/12


14 SQP VOL. 15, NO. 2/ 2013, ASQ

Excessive eatures o low or negligible value Unused documentation Concurrent assignments or requent switching Handover between resources Delays or approvals, decisions, and resources

( or example, waiting or code completebe ore beginning quality assurance activities)

CoSQ Anal sis an d namicsPlotting CoSQ against a quality measure (such as totalcost o ownership or product de ects) provides the CoSQ curve and, plotted against time, reveals trends in anorganizations quality processes. This addresses mosto the goals o quality costing: justi cation, motivation,budgeting, and process improvement cost e ectiveness.Pareto analysis, based on the principle that quality costs

are localized (80 percent o quality costs are incurredby 20 percent o the quality noncon ormances), can beused to identi y candidates or process improvement. When CoSQ is categorized by product and processsources, typically one or two sources will be shown toincur much higher costs than the others.

In analyzing CoSQ data, the dynamics o the modelmust be taken into account. The ollowing are someo the actors that a ect CoSQ over time:

Advances in so tware technology, or examplethe prevalence o graphical user inter aces,create new demands on so tware producers.

Growing user sophistication creates demandsor increased unctionality and better support

(documentation, training, technical services). Better applications and systems set new stan-

dards or per ormance and reliability.

2 0 1 3

, A S Q

Cost of software quality

Managements quality costs

Control costsPrevention costs

Appraisal costs

Internal failure costs

External failure costs

Development and software qualityassurance units quality costs


Control costs


Managerial appraisaland control costs

Internal managerialfailure costs

External managerialfailure costs

FigurE 9 The proposed inclusive CoSQ modeltargeted at software development projects (Galin 2004)

tabLE 7 Cost of software quality data (Laporte et al. 2012)

2 0 1 3

, A S Q

S e aame c nEn nee s

(19)

S e aame c nM n e s

(5)

S e bE ope nEn nee s

(13)

S e CE ope nEn nee s

(14)

S e dE ope nEn nee s

(9)

Co sea

2008(8)

Co seb

2008(14)

Co seC

2009(11)

Co sed

2010(8)

Co seE

2011(15)

Co seF

2012(10)

Cost of performance 41% 44% 34% 31% 34% 29% 43% 45% 45% 34% 40%

Cost of rework 30% 26% 23% 41% 34% 28% 29% 30% 25% 32% 31%

Cost of appraisal 18% 14% 32% 21% 26% 24% 18% 14% 20% 27% 20%

Cost of prevention 11% 16% 11% 8% 7% 14% 10% 11% 10% 8% 9%

Quality defects per1,000 lines of code 71 8 23 35 17 403 19 48 35 60 55


12/12


www.asq.org 15

value o quality initiatives and or identi ying qualityinitiative candidates. CoSQ o ers the same promise

or the so tware industry, but could be used more.Initial uses o CoSQ indicate that it represents a very

large percentage o development costs60 percentand higher or organizations that are unaware o improvement opportunities. CoSQ use can, how-ever, demonstrate signi cant cost savingssuch asRaytheons our old reduction in rework or so twareorganizations willing to undertake quality improve-ment initiatives. Perhaps more importantly, the useo CoSQ enables an understanding o the economictradeo s that accompany activities and expendituresmade or improving the quality o delivered so tware.

reFereNCeSDion, R.1993. Process improvement and the corporate balance sheet.IEEE Software 10, no. 4:28-35.

Galin, D. 2004. Toward an inclusive model for costs of software quality.Software Quality Professional 6, no. 4:25-31.

Gibson, D., D. Goldenson, and K. Kost. 2006.Performance results of CMMI-based process improvement (CMU/SEI-2006-TR-004). Software EngineeringInstitute. Carnegie Mellon University. Available at www.sei.cmu.edu/library/abstracts/reports/06tr004.cfm.

Haley, T.J. 1996. Software process improvement at Raytheon.IEEE Software 13, no. 6:33-41.

Houston, D. 1999. Cost of software quality: Justifying software process

improvement to managers.Software Quality Professional 1, no. 2:8-16.Jones, C. 2011. Software quality and software costs.Software Quality Professional 13, no. 3:24-30.

Knox, S.T. 1993. Modeling the cost of software quality.Digital Technical Journal 5, no. 4:9-16.

Laporte, C., N. Berrhouma, M. Doucet, and E. Palza-Vargas. 2012. Measuringthe cost of software quality of a large software project as BombardierTransportation: A case study.Software Quality Professional 14, no. 3:14-31.

Liu, A.Q. 2007. Motorola software groups China center: Value added by CMMI.Software Technology News 10, no. 1:19-23. Available at journal.thecsiac.com/issue/41/79.

Price Waterhouse Management Consultants.1988. Software quality standards:The costs and benefits. A Review for the Department of Trade and Industry.Price Waterhouse Management Consultants. London, England.

Better so tware production technology, orexample in testing and con guration manage-ment, supports higher quality goals.

Reduced cycle time and time to market

increase competition in the marketplace andmay be accompanied by changes in actual orperceived quality.

These actors may have the e ect o shi ting the pointo diminishing returns over time, usually to the right. Forexample, either better so tware production technologyor growing user sophistication can e ect a shi t in theoptimum CoSQ toward a higher level o quality.

CoSQ P s ntationRelationships that have the greatest impact on man-agement are:

Quality costs relative to the total cost o ownership or so tware

Quality costs as a percent o sales Quality costs compared to pro t Quality costs compared to the magnitude o

the current problem

In addition to these, showing CoSQ as a percent o total development costs is appropriate to so tware or

several reasons. First, sales and pro t may not have adirect relationship to the actual cost o a so tware productsince so tware pricing is o ten dictated by market orces.Second, all but a small percentage o so tware develop-ment costs can be measured in labor hours, so the costscan be readily shown in either hours or dollars. Third,the state o the art in so tware development is such thatcomparing quality costs to development costs illustratesthe magnitude o the current challenge.

CONCLUSIONSCoQ is a proven technique in manu acturing andservice industries, both or communicating the

Software Quality Costs

Documents