8/2/2019 Ibm Smarter Quality Management
1/13
8/2/2019 Ibm Smarter Quality Management
2/13
Smarter Quality Management: The Fast Track to Competitive Advantage
1
PA
GE
Organizations that create and deliver
softwarewhether fortheir own IT
operations, for the packaged applications
market,or as the core of their nal product,
as in the systems spacemust grapple not
only with todays tough economic climate,
but also with increased complexity in their
processes and supply chains. Many factors
serve to complicate software delivery,
but competition lies at the heart of this
complexity.
Here are a few examples. In the products
arena, customers demand more from thesoftware components designed for thelatest
hardware, often with requirements that
change rapidly, even as software projects
are underway. Keeping track of changes
while meeting aggressive (and unaltered!)
deadlines is difcult, ifnot impossible. In
the IT space, more businesses are focused
ontheir operational software for capturing
and providing value to their customers andlines of businesses. E-commerce websites
compete to improve customer relations
and simplify online business; businesses
that create highly optimized supply chains
supporting a fast, efcient ecosystem of
partners quickly rise in the market place.
What does this competitive environment
mean for businesses seeking to deliver
high-quality products and services?
Certainly, effective quality management
creates opportunities to deliver key business
benets, such as improved market share,
higher customer satisfaction, and increased
brand equity. But top quality in the completed
product cannot serve as the single guiding
principle by which products are produced
and delivered. Time to market is also key;
costs and risk factors must also be part ofthe balancing act. Get these things wrong,
and you may face unsustainable costs,
missed windows of opportunity, unhappy
customers, even a massive recall or the
complete failure of a system at a critical
moment. Get these things right, and you
can achieve a positive operational return
on investment from efciencies gained in
development activities.
One of the biggest challenges related to
quality management is how to invest
intelligently to minimize risk, given economic
constraints. However, guring out a) how to
relate quality to business outcome and b)
what constitutes the right level of quality for
individual products is not always clear.
This paper introduces a practical approach
to quality management (QM) that helps
reduce time to market without sacricing
quality in the outcome. The underlying
concepts presented here will be familiar
to software project managers, especially
those with QM experience, but I will explain
some fundamentals as we go along toensure all readers seeking these benets
can understand the essential processes
involved.
The nature ofsoftware
developmentHeres one way to understand the soft in
software: it is relatively easy to change. But for
software designed for the commercial space,
where the competitive pressures described
above govern a software projects success
or failure, the softfeature happens to be
one of its riskiest attributes. Thats because
software projects are seldom designed andmanufactured as in traditional engineering
projectsa bridge, for example. While a
bridge is engineered through traditional
planning and architecture based on the laws
8/2/2019 Ibm Smarter Quality Management
3/13
Smarter Quality Management: The Fast Track to Competitive Advantage
2
PA
GE
of physics, then produced according to
an organized plan with a division of labor,
software is, at its essence, simply information.
Its development typically resembles a more
creative process than one bound by the
laws of nature. Walker Royce, IBM Software
Group/Rationals chief software economist,
compares software production to movie
production: a collaboration involving a team
of craftsmen and emerging from the naturally
creative process of artistic yet technical
people.
Over the past two decades, this unique
feature of software has been understoodand embraced by iterative development
practitioners, who now tend to develop
software in stages called iterations. Each
iteration delivers a working, functional
version of the software under development,
so it can be reviewed, tested, and
vetted by stakeholders and other teams
seeking adherence to the original project
requirements. This allows project managersto make smaller, incremental course
corrections during the project life cycle
thus ensuring the nal deliverable is close
to expectationsas opposed to having
separate teams work according to a plan,
assemble various components near project
end, and discover major failures due to
integration or deployment complexities.
For testing teams, the iterative development
process integrates quality management
across the states of the project work ow,
as opposed to relegating test activity to
the end of the project. I will describe the
role of iterative development-based quality
management more fully in the next section.
Qualitymanagement in
the softwaredevelopment life
cycle
What is the role of the testing, or QM, team
during the iterative life cycle? What do they
test for, and how do they know what is
supposed to change from one iteration to
the next?
As noted above, traditional software testing
only occurs late in the life cycle, after
multiple coding teams have spent much
time and effort to deliver their components
toward the complete project. Because these
traditionally managed projects proceed
according to strictly described requirement
sets, and various component teams focuson their portions alone, it is up to the
testers to discover the discontinuities and
malfunctions as these components are
assembledthen its the testers who must
deliver the bad news that much rework has
to be done inorder to get the project back
on track.
Iterative software development techniques
improve on that scenario by introducing
test teams to the process much sooner. A
relatively modest, rst iteration may only
address 15 percentof the full set of project
requirements, but as a functional module
of working code, the completed iteration
can be demonstrated, and tested. So any
defects discovered by test teams at this
early stage have a proportionally small
impact on the larger development team, whomake the xes, then proceed to the next
iteration where more of the requirements
can be incorporated into the working version
(iteration) of the software.
8/2/2019 Ibm Smarter Quality Management
4/13
Smarter Quality Management: The Fast Track to Competitive Advantage
3
PA
GE
The number of iterations required by any
software project depends on many factors, of
course, such as the complexities of the teams
supply chain, the complexity of the software
underdevelopment, the physical locationof team members (are they geographically
distributed, perhaps internationally? or are
they co-located under a single roof?), and
the competitive demands that determine
optimum time to market. During any software
delivery process, When do we release? is
a key question withno simple answer. You
must consider project-specic variables,
such as the cost of delays, the opportunityvalue of early delivery, marketplace quality
expectations, and the costs associated with
defects. Ultimately, the delivery strategy
will be based on the actual or perceived
importance of each variable.
The optimal time
to release
The optimal time to release is when the total
risk exposure isminimal, typically around
the time where the risk associated with
competitive threats starts to outweigh the
risk reduction associated with further quality
improvements, as illustrated in Figure 1.
The best time to release varies widely
based on your software delivery strategy
and your target market. Software delivery
for both IT (internal business systems) and
smart products (products using embeddedsoftware, including system-of-systems
design) is typically dominated by one of two
motivators, depending on the organizations
target market: time to market (schedule
driven), or quality impact (quality driven).
Schedule-driven delivery implies deliver
on time, regardless of other factors and
is often used in industries where Time tomarket is king. Consumer electronics is
one good example, as well as automotive,
segments of the medical industry, and other
markets, where product teams try to gain a
rst mover advantage over their competitors,
(almost) regardless of the risk associated
with inadequate quality. It should also be
noted that schedule-driven delivery is not
limited to the systems space (i.e., the many
embedded software devices industries).
Many IT development teams use schedule-
driven delivery when trying to enhance their
end user experience and increase market
share, taking away from their competitors,
often risking quality in the process.
Figure 2 represents the risks associated
with schedule-driven software delivery. The
green line represents the risks associatedwith the delivery of your product being
reduced over time. The red lines represent
the risk of competitors stealing your market
away increasing over time, as well as risks
associated with opportunity costs.
The intersection is the point in time where
the sum of both lines, i.e. the total risk, is the
lowest. As seen in Figure 2, this point movesto the left as the environment you are in is
more competitive in nature. (Notice that the
intersection point is moving up as well.)
High opportunity cost;
Strong competitionLowestOverallRiskExposure
Manycritical
defects
Low opportunity cost;
Weak competition
Few
minor
defects
Time
RiskExposure
Quality risk ( = Probability of defects x loss due to defects)
Competition risk ( = Probability of competitors x size of loss to competition)
Total risk ( = Sum of all risks)
Figure 1: Minimal risk exposure is when oppor tunity cost and competitivethreats outweigh risk reduction related to quality improvements.
8/2/2019 Ibm Smarter Quality Management
5/13
Smarter Quality Management: The Fast Track to Competitive Advantage
4
PA
GE
Quality issues are often magnied in a
schedule-driven life cycle, given that
software contractors frequently get paid
on a time and materials basis, regardless
of the quality of software they deliver. In
many cases, you may even end up paying
extra for the delivery team to x their own
defects, so the potential costs of defects
to the end user can add up quickly. And
heres an interesting statistic: According to
the Carnegie Mellon Software Engineering
Institute, Data indicate that 60 - 80 percent
of the cost of software development is in
rework.
Quality-driven deliverycan also be costly
but for different reasons. As shown in Figure3, the more critical any defects might be
regarding quality, the longer it takes to get
to an optimum release point.
The release timing for this approach is
governed by achieving the right quality,
moving the optimal time to release further to
the rightbut how do you dene that? Zero
defects is practically impossible to achieve,given that there is no way to determine how
many defects still exist in a piece of code or
the probability of detecting those defects in
use. A target based ondefects xed might
be more realisticbut its still impossibleto
know the number of remaining defects in the
product.
Risk-driven delivery implies delivering
your software when the risk is minimal.
But in practice, we always need to release
earlyearlier than we can. Which typically
implies increasing the risk, right? At least
this is a commonly held view, but is it always
the case?
Within the risk-driven model, the optimal
release time is when risks are sufciently
reduced (not completely eliminated) and
time to market has not been wasted. Inother words, some time is needed to reduce
the most signicant risks, but the company
cannot afford to address every known
risk because the opportunity to beat the
competition is eeting.
So the question is, how can we get to this
point sooner? How do we compress the
release date from the optimal intersection(shown as a blue circle in Figure 4) to a point
earlier in time?
Time
Risk
Exposure
Time to Market is king!Example: Consumer Electronics
High opportunity cost;Strong competition
Figure 2: The blue dots show possible release points, with points of minimalrisk moving forward as competition intensifies (red lines).
Time
Risk
Exposure
Quality is king!Example: Safety Critical applications
Criticality ofDefects
Figure 3: The more critical the implications of defects are, the more time it
takes to get to the lowest risk point where release is possible.
8/2/2019 Ibm Smarter Quality Management
6/13
Smarter Quality Management: The Fast Track to Competitive Advantage
5
PA
GE
We cannot simply cut the time requirement,
because as we move left on the green line,
the risk goes higher. But what if we could
compress the curve described by the green
linepush it down, so to speak? Then we
could not only deliver sooner but lower the
overall risk as well. The intersection point will
move down and to the left. This improved
scenario is shown in Figure 4.
Risk-driven delivery offers a practical
improvement over these two extremes (i.e.
schedule driven vs. quality driven) because
it more cost-effectively balances quality
versus time-to-market considerations. A
risk-driven strategy is a renement of a
quality-driven approach that optimizes risk
exposure against development cost and
time.
For the remainder of this discussion, we
will assume that the software delivery life
cycle is based on a risk-driven approach.
We will explore how to bend the green
curve shown in Figure 4 downward and to
the left, for reduced time to market without
compromising the risk prole.
Understandingquality
management: Its
more than simply
testing
If a faster reduction in risk is the goal,
how do you achieve it? The answer is not
through testing, which is focused simply
on discovering defects. In traditional testingpractices, testing is considered a late stage
activity, squeezed between an often-late
development hand off date and an immovable
ship date. Not only does this practice fail to
yield the benets of incremental, iterative
development techniques explained earlier;
it also minimizes, or at best reduces, the
amount of time spent on quality assurance,
and makes xes all the more difcult unlessyoure willing to compromise the release
date.
As noted earlier, iterative development
techniques greatly improve this situation by
having functional units tested incrementally,
in stages, throughout the life cycle, rather
than leaving the testing phase until project
completion.
And quality management takes this
improvement a step further2.. Quality
management, which is the implementation
of best practices to proactively reduce risk
throughout the whole life cycle, is a risk
reduction mechanism in its own right. By
choosing quality management practices
with the potential to deliver a positive ROI
within a relative short amount of time, you
can justify risk reduction measures from not
only a quality stand point but also a nancial
standpoint.
Time
OptimalTime to Release
Time toMarket
Figure 4: To deliver early, at an improved quality, reduce your risk at an earlierpoint in the li fe cycle.
8/2/2019 Ibm Smarter Quality Management
7/13
8/2/2019 Ibm Smarter Quality Management
8/13
Smarter Quality Management: The Fast Track to Competitive Advantage
7
PA
GE
coverage, one that maps results to testing
activities. The essential principle is this:
prioritize testing according to risk. As
changes occur to working code during the
life cycle, test teams may choose to test
for the most critical requirements rst, and
maybe later test for all critical requirements
remaining (for example, in another iteration).
In other instances, they may choose to run
only the subset of test cases that provide
them with the highest coverage. This
approach is illustrated in Figure 5.
Although you cannot completely remove
risk from a development program, you
can measure it and manage it by taking
the appropriate mitigation actions. This
approach helps ensurethat you are optimally
using your nite testing resources (your
people) to reduce risk as rapidly as possible
in the development cycle. By focusing testing
effort on high-value test cases, from either
the point of view of coverage or contribution
to business value, you essentially prioritize
according to riskand you worry less about
test cases that pertain to noncritical issues.
But does creating the traceability links
require an expensive investment? No, andhere is the basic math. Consider a hypo-thetical medium-size project with some5,000 requirements and 10,000 test cases.Assuming it takes 20 minutes to locate and
link the appropriate test artifacts for each
requirement, it would take approximately10 person-months to create the traceabilitybetween requirements and test cases. You
could potentially reduce this time to oneto two minutes per requirementand atotal of just 10 to 20 person-daysusinga dedicated quality management solutionwith support for capturing traceability linksbetween requirements and test cases. At anominal rate of US$50 per hour, this singlechange corresponds to a potential saving of
around US$75,000. And needless to say, inreal life, the numbers are much bigger.
There are other best practices that contributeto improved quality at a reduced cost. Weare not going to cover them all in this paper,
but here are two for you to consider.
Improved collaboration between theQA team and other stakeholders: From
talking to customers, we learned thaton average, a tester spends only about60 percent of the time performing actual
testing, test planning, or test reporting.The other 40 percent is related toactivities that are collaborative in nature,such as clearing up the requirementswith domain experts or businessanalysts, or exchanging emailsand phonecalls with members of the
development team. This gets worse indistributed organizations. If you couldtrack and manage the collaboration,
it will not only reduce your riskassociated with lack of communicationsand misunderstandings, but also reducethe time for collaborative tasks by 20 -50 percent.
Automated reporting: Creating a report,especially one that goes to high level
management, requires data collectionfrom many sources, sometimes from
teams that are in different time zones,and then formatting this data
appropriately. If you could automate thisactivity, your team will probably useit more often and take the appropriate
Most critical Requirements
All critical Requirements
Low contribution
High Requirements coverageNo. of Requirements
Test Suites
Figure 5:After a change, re-executing all the tests is safe, but expensive andoften unrealistic. By focusing instead on test suites that are the most relevant
to the iteration that is being tested, test teams make more efficient use of their
time and reduce redundancies along with high testing costs.
8/2/2019 Ibm Smarter Quality Management
9/13
Smarter Quality Management: The Fast Track to Competitive Advantage
8
PA
GE
decisions in real time,thus reducingyour risk. How much would this saveyou?
These are some of the quality management
best practices, each of which contributes torisk reduction and therefore increased qualityand reduced cost. Now, lets consider theoverall impact of quality management on thedefects density and the cost of xing them.
The overall
business impact ofquality management
Quality management best practices centeron quality synchronization points acrossthe whole development life cycle. We havediscussed the benets of traceability torequirements, and we have briey notedcollaboration between stakeholders and the
QA team, as well as automated reporting.Other best practices include: allowing qualityprofessionals to contribute to the team effortfrom the very beginning of a project; theintegration of practitioners doing the testing
as part of quality management; and the useof consolidated quality dashboards.
Together, these quality management bestpractices can benetthe overall business in
measurable ways. Using CMMI
3
as aproxyfor the maturity of the development process,Figure 6 shows that the overall businessimpact of quality management is quitecompelling.
Lets assume an organization at CMMI level 2,with 1000 defectsdetected during functionaltesting. Figure 6 shows that on average,without QM practices, about 30 percent ofthe defectsare being detected in functional
testing (the left, blue bar), and therefore thetotal number of defects is 3300. However, byapplying QM practices, the defect detectionrate increases to 58 percent (the right, greenbar), therefore detecting 1914 (58 percent of
3300), or 914 more defects.
As xing defects during User AcceptanceTesting (UAT) is aboutseven times moreexpensive than during unit/integration test,and assuming a x cost of US$120 perdefect during unit/integration test, xing 914defects in UAT is already increasing the costby over half a million dollars!
And this does not even take into accountthe reduction in the number of defects thatresult from applying QM practices in therst place, which makes the savings evenmore signicant. This also does not takeinto account less tangible savings, suchas increased quality, customer retention,and other implications of quality as adifferentiating asset.
As most software development teams are
around CMMI levels 2 or 3, the benets andthe savings described above apply to most
of the industry. But as development teamsbecome more mature, apply QM practices,and move up to CMMI levels 4 and 5, the focusshifts into less obviousbut for some, evenmore importantbenets, such as reductionin the number of defects that are introducedin the rst place, measured improvementsaround planning and execution of quality
related activities, customer retention, andleveraging quality as a differentiating asset.
Impact of Quality Management on Process Efficiency
15%
30%
60%
75%
85%
32%
58%
76%
85% 87%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5
CMMI Levels W/O QM W QM
20% 40% 40% 40% 10%QM Impact:
Figure 6: Graphing percentage of defects detected (Y axis) against an
organizations software development maturity level (X axis).
8/2/2019 Ibm Smarter Quality Management
10/13
Smarter Quality Management: The Fast Track to Competitive Advantage
9
PA
GE
Better quality +lower cost
= improved
competitiveness
In this paper, I have described several
improvements to methods used by softwareteams in the design, testing, and deployment
of software for systems or IT. Teams mayuse these quality management methods to
deploy that software more quickly, whilemitigating quality-related risks throughoutthe life cycle. The multidisciplined practiceof quality management is breaking thefunctional and organizational silos thatare so common in todays companies. Itencourages an analytical process thatsclosely integrated with the development life
cycle.
Analyzing the market and best practices
shows that business outcomes can beoptimized, and that smart improvements
within the realm of proven best practices for
requirements management and traceability,collaborative test planning and automatedreportinga combination of disciplines
that denes quality managementcan helpaddress the need for increased innovation
with more competitive products and servicesto your customers.
The IBM Rational organization is readyto demonstrate these techniques to you.With straightforward adjustments to yourinvestments, deployment practices, and
tooling, we can help you realize thesebenets within a time frame that best suitsyour businesss needs.
We look forward to working with you!
For more
information
To learn more about the IBM Rational qualitymanagement offerings, please contact
your IBM marketing representative or IBMBusiness Partner, or visit the followingwebsite:
ibm.com/software/awdtools/rqm/
Additionally, nancing solutions from IBMGlobal Financing can enable effective cashmanagement, protection from technology
obsolescence, improved total cost of
ownership and return on investment. Also,our Global Asset Recovery Services helpaddress environmental concerns with new,
more energy-efcient solutions. For moreinformation on IBM Global Financing, visit:ibm.com/fnancing
About the AuthorMoshe Cohen is the Market Manager for IBMRational quality management offerings. In hiscurrent role, he works closely with customers,including managers and practitioners, todrive IBM Rational quality managementofferings in both the IT and embedded
systems spaces. Prior to this, he was withTelelogic, dening and driving its Model
Driven Testing solutions. He has extensivehands-on experience in the specication,development, and testing of C3I medical and
telecom applications, including technologyadoptions and driving process improvement
programs. He received his EE and M.Sc inmathematics and computer sciences, bothwith honors, from Beer-Sheva University inIsrael.
http://www-01.ibm.com/software/rational/products/rqm/http://www-03.ibm.com/financing/ww/http://www-03.ibm.com/financing/ww/http://www-01.ibm.com/software/rational/products/rqm/8/2/2019 Ibm Smarter Quality Management
11/13
Smarter Quality Management: The Fast Track to Competitive Advantage
10
PA
GE
Copyright IBM Corporation 2011 IBM Corporation
Software Group Route 100 Somers, NY 10589 U.S.A.
Produced in the United States of America June 2011
All Rights Reserved
IBM, the IBM logo, Rational and ibm.com are
trademarks or registered trademarks of the
International Business Machines Corporation in
the United States, other countries, or both. If these
and other IBM trademarked terms are marked
on their rst occurrence in this information with atrademark symbol ( or ), these symbols indicate
U.S. registered or common law trademarks owned by
IBM at the time this information was published. Such
trademarks may also be registered or common law
trademarks in other countries. A current list of IBM
trademarks is available on the web at Copyright and
trademark informationatibm.com/legal/copytrade.
shtml
Other company, product, or service names may be
trademarks or servicemarks of others.
The information contained in this document is
provided for informational purposes only and
provided as is without warranty of any kind,
express or implied. In addition, this information is
based on IBMs current product plans and strategy,
which are subject to change by IBM without notice.
Without limiting the foregoing, all statements
regarding IBM future direction or intent are subject to
change or withdrawal without notice and represent
goals and objectives only. Nothing contained in this
documentation is intended to, nor shall have the
effect of, creating any warranties or representations
from IBM (or its suppliers or licensors), or altering
the terms and conditions of the applicable license
agreement governing the use of IBM software.
1See the Carnegie Mellon Software Engineering
Institutes CEO andfounders message at http://
www.sei.cmu.edu/about/message/
2For a more complete discussion of quality
management practices, downloadthe paper Value-
driven quality management for complex systems:
Sixstrategies for reducing cost and risk at http://
www14.software.ibm.com/webapp/iwm/web/
s ignup.do?source=swg-rt l -spsm-wp&S_
PKG=wp_RQM_VALUEDRVN_071510
3Capability Maturity Model Integration. CMMI is a
staged approach to process improvement that denes
incremental levels of maturity in software engineering
organizations. For more information see the Software
Engineering Institute (Carnegie Melon University)
website athttp://www.sei.cmu.edu/cmmi/
Please Recycle
http://www.ibm.com/legal/us/en/copytrade.shtmlhttp://www.ibm.com/legal/us/en/copytrade.shtmlhttp://www.ibm.com/legal/us/en/copytrade.shtmlhttp://www.sei.cmu.edu/about/message/http://www.sei.cmu.edu/about/message/http://www.sei.cmu.edu/about/message/https://www14.software.ibm.com/webapp/iwm/web/signup.do?source=swg-rtl-spsm-wp&S_PKG=wp_RQM_VALUEDRVN_071510https://www14.software.ibm.com/webapp/iwm/web/signup.do?source=swg-rtl-spsm-wp&S_PKG=wp_RQM_VALUEDRVN_071510https://www14.software.ibm.com/webapp/iwm/web/signup.do?source=swg-rtl-spsm-wp&S_PKG=wp_RQM_VALUEDRVN_071510http://www.sei.cmu.edu/cmmi/http://www.sei.cmu.edu/cmmi/http://www.sei.cmu.edu/cmmi/http://www.sei.cmu.edu/cmmi/https://www14.software.ibm.com/webapp/iwm/web/signup.do?source=swg-rtl-spsm-wp&S_PKG=wp_RQM_VALUEDRVN_071510http://www.sei.cmu.edu/about/message/http://www.ibm.com/legal/us/en/copytrade.shtml8/2/2019 Ibm Smarter Quality Management
12/13
05 08 November 2012 | RAI Amsterdam
TheEuroSTARConerence
attracts testers rom
around theglobewhocome tolearn, network&
discuss all things testing.
Attendees experience
intensive tutorials, track
sessions & inspirational
keynote speakers, & can
visitEuropes largest
Sotware Testing
Exhibition. Return to
theofce inspired,
motivated, andull onew ideas!
www.eurostarconerences.com
8/2/2019 Ibm Smarter Quality Management
13/13
w w w . e u r o s t a r c o n f e r e n c e s . c o m
Join the EuroSTAR CommunityAccess the latest testing news! Our Community strives to provide testprofessionals with resources that prove benecial to their day-to-day roles.These resources include catalogues of free on-demand webinars, ebooks,
videos, presentations from past conferences and much more...
Follow us on Twitter @esconfs
Remember to use our hash tag #esconfs when tweetingabout EuroSTAR 2012!
Become a fan of EuroSTAR on Facebook
Join our LinkedIn Group
Contribute to the EuroSTAR Blog
Check out our free Webinar Archive
Download our latest ebooks
The EuroSTAR BlogWritten by Testers for Testers
http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/http://www.eurostarconferences.com/