Risk Based Testing

IntroductionSince there isnt enough time andresources to test everything, what should Itest? Almost every software developmenteffort has to answer this question. Many, ifnot most, efforts leave the answer tochance. Inspections and reviews are oftencut short or even eliminated due to timepressures. It is left to the developers todecide what gets unit-tested and how. Asetof developers or independent testersdecide which functions get tested duringthe function test. The system testers decidewhat and how they will test.

What drives the decision making process?Left to chance, the typical influences onwhat gets tested are:

1) The path of least resistance otherwisestated as, "What is the minimallyrequired testing that will satisfy man-agement and not blow my schedule?"

2) Hot topics or "What is the latest crazein testing and what is managementbeing measured on this month?"

3) Out of sight, out of mind-"If I don'tthink about all those things that need tobe tested I don't have to worry aboutthem."

4) I'll test the parts I understand or that

seem interesting to me.5) I'll test the areas where I have been

burnt in the past.6) Tests and tools or "I'll test whatever is

covered by test cases and tools that arealready prepared and available and easyto run."

7) Time-based testing, "I'll make sure Ikeep busy testing during the allottedtest time."

Any and all of these test influences maylead to testing that finds problems and allcan be reported in such a way that we con-vince ourselves that we have "tested" theproduct. As with any endeavor, if youdon't know where you are going, youprobably won't get there. On the otherhand, if we have no particular destinationin mind we will probably satisfied withwherever we end up. That is often the casewith software testing. Often times, themain goal of testing is to convince our-selves that the product is ready to ship.Most any set of activities mixed with theright persuasive words can accomplishthis.

Risk-Based TestingIf you don't want to leave your product's

outcome to chance, there is a more disci-plined approach that can help. Risk-basedtesting helps identify which parts of theproduct have the most influence on thebusiness. It strategizes a plan to maximizethe effectiveness of the testing. It makesvisible what is and is not being tested andwhat the potential risk is for each impor-tant aspect of the product. In short, it bet-ter enables a project team to makeinformed decisions.

Risk-based testing does not impose anyspecific testing methodologies or tech-niques. It is merely a guide to help decideand make visible what gets tested, howmuch it gets tested, when it gets tested, andwhat the associated risk is. It will help youunderstand what impacts any last minutechanges to the test schedule may have onyour project risk.

Risk-based testing doesn't need to be com-plex or burdensome. For the most part, itis practical, intuitive and straightforward.It is not intended for everyone. The lessdisciplined and mature processes will getmuch less good out of it than projects thatemploy and track specific test techniques

September/December 2001 http://www.testinginstitute.com Journal of Software Testing Professionals 1

Risk-BasedTesting

Feature

At a Glance:

Based on Dr. Ingrid B. Ottevangersrisk-based testing strategy withmodifications and additions to makeit work in our local test environment.

by Kelly Whitmill


and stages. It is a poor choice if you reallydon't want to make your strengths andweaknesses visible. For those that want toget the most out of their testing and arewilling to employ a little discipline, hereare the steps that will help you accomplishthat:

1. Identify each important component ofthe system.

2. Determine relative importance of eachcomponent.

3. Determine the risk assessment foreach component.

4. Identify quality characteristics for theproduct.

5. Determine relative importance ofquality characteristics for each compo-nent.

6. Identify list of potential test activities.7. Determine testing depth by test activi-

ties for each component and qualitycharacteristic.

8. Indicate test techniques used by eachtest activity.

9. Assign responsible parties/owners foreach test activity.

10. Indicate expected "risk after mitigation".

11. Redetermine "risk after mitigation" forany in-process changes to tests andschedules.

No individual should be doing this processalone. Input is required from the customer,test, development, marketing, manage-ment, and all other parties involved in thedevelopment and use of the system. It maybe necessary for an individual to puttogether some initial information whichcan then be used by the group as a startingpoint for discussions, negotiations, and awork in progress that can be refined to itsfinal form.

Identify Each ImportantComponent of the SystemSubdivide the system into manageablecomponents. The components mayinclude subsystems, line items, functions,collections of functions, options, or attrib-utes or any grouping that you want tomake visible and manage.

Though there is complete flexibility onhow you subdivide the system, it is impor-tant that you give it careful thought andconsideration. Some questions to help youcome up with a good workable set of com-ponents are:

Is the list of components complete? Arethere any important parts of the system notrepresented by a component? Make surethe entire product/system is representedincluding both new and existing compo-nents.

Does everything in the component haverelatively equal risk? For example, if thecomponent is a command with manyoptions and attributes, some of which areused a lot and would result in high damageif there were errors, while others are usedrather infrequently and have relativelyminor damage if there is an error, youmight want to divide the command up intotwo or more components.

Example: 1) CommandA with high useoptions and attributes, 2) CommandAwithnormal frequency use options and attrib-utes, and 3) CommandA with low fre-quency use options and attributes.

Does the component represent the rightlevel? If you get too detailed in specifyingcomponents you will waste time micro-managing each minute aspect of the sys-tem. If your components are too broad youmay hide some areas that are vulnerable torisk. In general, you may want a moredetailed breakout of new components,components that have a history of prob-lems, and components that have a high rel-ative importance to the system.

Keep in mind that you can do differentlevels of risk-based testing. You may wantto do an initial assessment looking only atmajor subsystems and later further refinethe components for a more detailed riskassessment and strategy.

Table 1 is an example of a subset of com-ponents from an OS/390 Print Server.

Determine the RelativeImportance of EachComponentAssign each component a relative impor-tance. It is sufficient to categorize eachcomponent as Critical, High, Medium,Low, or No Significant Importance. If thenumber of components is small it may beuseful to assign the relative importance inpercentages. If you use percentages, thesum of all components should be 100 per-cent. This is not intended to be a detailed,

Table 1: Example of a subset of components from an OS/390 Print Server

Printing from Windows (Port Monitor)Printing from remote TCP/IPPrinting from Windows (SMB)Printing from Windows (IPP)lp command (high frequency job attributes)lp command (medium frequency job attributes)lp command (low frequencey job attributes)cancel commandlpstat commandPostScript to Modca TransformPCL to Modca TransformIPP ServerIPP Client

8 Journal of Software Testing Professionals http://www.testinginstitute.com September/December 2001

scientific, and algorithmic procedure. It isnot important to come up with precise per-centages; rather, it is intended to provide ageneral picture of the relative importanceof the components that make up the sys-tem. This forces the group to start makingchoices as to what parts of the system aremost important and thus require the mostfocus.

It may be useful to do a sanity check onthe results. If component Ais designated at50% and component B is designated at5%, is component A really 10 times moreimportant than component B? If not, per-haps some adjustment is necessary. Aftergoing through each component andassigning a relative importance, look at thesystem as a whole and see if the percent-ages make sense. (See Table 2.)

Determine the RiskAssessment for EachComponentThe goal is to be objective in assessingrisk. There are sound concepts, proce-dures, and some basic formulas to helpaccomplish this. However, it is veryimportant to keep in mind that the goal isto obtain an objective assessment of thegeneral risk of a component. There is noalgorithm to come up with a precise risk.

Any attempt to produce such an algorithmprobably does more harm than good. Suchalgorithms tend to lead to self-deception,over-analysis, and misplaced effort.

Therefore, even though we may use somemathematical formulas to achieve anobjective risk assessment, it is still not analgorithmic and precise process.

For our purposes, we are defining risk asthe chance that a failure will occur com-bined with the damage of such a failurewhen it does occur.

Risk = chance of failure x damage

The chance of failure is composed of twoelements, the chance of error combinedwith the frequency of use.

Chance of failure = chance of error xfrequency of use

The chance of error is an area where wecan lend objectivity to what is otherwise avery subjective practice. There are knownfactors that tend to influence the chance of

Table 2: Example of Relative Importance of Components

Component ImportancePrinting from Windows (Port Monitor) 20Printing from Remote TCP/IP 5Printing from Windows (SMB) 20Printing from Windows (IPP) 15lp command (high frequency job attributes) 5lp command (medium frequency job attributes) 5lp command (low frequency job attributes) 1cancel command 4lpstat command 5PostScript to Modca Transform 5PCL to Modca Transform 5IPP Server 5IPP Client 5Total 100

Table 3: Chance of Error

5 New or changed complex function5 Completely new function5 Late add function5 Functions that were realized under extreme time pressure5 Functions in which many defects were found earlier (e.g. Previous releases

or during earlier reviews)1 Changed or rewritten function1 (Especially frequently) adjusted functions1 Functions for which certain tools or techniques were employed for the first

time1 Functions which were transferred from one developer to another1 Functions which had to be optimized more frequently than on average1 Functions not tested within the last releases1 Functions with many interfaces1 Inexperienced developers1 Insufficient involvement of users1 Insufficient quality assurance during development1 Insufficient quality of low-level tests1 New development tools and development environment1 Large development teams1 Development team with sub-optimal communication (e.g., owing to geo-

graphical spread or personal cause)

errors in code. By understanding howmany of those factors come to play for thecode in question we can objectively deter-mine the chance of error. Not every factorwill influence the chance of error equally.For each factor, we assign it a value. Thehigher the value the more likely it is tocause an error. The following table showsa list of factors that cause errors. Yourexperience may show that there are otherfactors that need to be added to the tablefor your project. The values assigned arenot precise. They just show that some fac-tors are much more likely than others tocause problems. You may want to adjustthe values assigned based on the history ofyour project and development teams/envi-ronments. (See Table 3 on previous page.)

Determine the chance of error by sum-ming the values of each of the factors thatapply for the specified component.Determine the frequency by categorizingif the component as high use, medium use,or low use.

We assign the following values to thefrequency of use:

Low = 0.5 Medium = 1 High = 2

The reasoning behind these numbers isthat low frequency reduces the chance oferror. Since frequency is used as a multi-plier in determining the chance of failure,a value of .5 will reduce the chance of fail-ure. A medium value of 1 will neitherincrease nor decrease the chance of fail-

ure. A high value of 2 will increase thechance of failure.

Damage is a basic estimate of Low,Medium or High damage that would occurif an error does happen.

To summarize, the basic formula/guide forassessing risk is :

Risk = change of failure x damageChance of failure = chance of errorx frequency of use

Where: chance of error = the sum of all the

error factors frequency of use: Low = .5, Medium = 1,

High = 2 damage = low, medium, or high

Since chance of error and frequency of useare both numeric, we can plug those num-bers into the formula and derive a result.Since the damage estimate is not numericand since we don't want to give the illusionof an algorithmic process when there real-ly isn't one, we will use tables to assess therisk. First, we convert the chance of failureto a low, medium, or high categorization.We could have just categorized chance offailure originally into one of these cate-gories but that would have eliminated theobjectivity of the process. Achance of fail-ure 5 = a high chance of failure.

The reasoning behind these thresholds isthat any time you have multiple chance oferror factors without some mitigating fac-tor like low frequency use, the chance offailure cannot be low. Therefore, anychance of error combined with frequencythat is >2 is considered to be at least amedium chance of failure. If you were tohave a major chance of error factor (factorwith an assigned value of 5 combined withany other chance of error factor, it isassumed that the chance of failure is highunless there is a low frequency of use rate.Also, even if there were no major chanceof error factors, if you had six or moreerror factors involved, then obviously thechance of failure is high.

Now with a categorization of the chanceof failure and damage we can make a riskassessment. (See Table 4.)

It is important to keep all the informationfor determining risk visible. Two compo-nents may each have a high risk but onemay have a much higher chance of failurethan the other. Keeping this informationvisible makes it apparent that the compo-nent with a higher chance of error mayrequire more effort to reduce the risk thanother components with a similar riskassessment but a lower chance of error.(See Table 5.)

Identify QualityCharacteristicsWhat does quality mean to this system,project, product, and organization? For


Table 4: Risk Assessment

Damage = Low Damage = Medium Damage = High

Chance of Failure = Low Risk = Low Risk = Low Risk = Medium

Chance of Failure = Medium Risk = Low Risk = Medium Risk = High

Chance of Failure = High Risk = Medium Risk = High Risk = High

almost all systems, functionality and capa-bility are required quality characteristics.For a Web-based application, perform-ance, reliability and availability may bekey quality characteristics. Regardless ofwhat they are for your system, they needto be identified. Some typical quality char-acteristics include:

Functionality / Capability Suitability / Customer Expectations

( Does it do what the customer wants?) Usability (How easy is it to use?) Reliability (Consistent results, absence

of abends, hangs, etc.) Availability Performance Installability Serviceability (Easy to find and

diagnose problems) Migration Standards Compliance Security Maintainability (Easy to apply

fixes, updates, etc.) Portability Testability Reusability

Identify the set of quality characteristicsthat relate to your product.

Assign Importance ofQuality Characteristics forEach ComponentNot all quality characteristics apply toevery component of the system. Somequality characteristics are more importantto some components than others. For eachcomponent you need to decide whichquality characteristics apply and deter-mine their importance. For example, thegraphical user interface for a productmight list "usability" as its most important

quality characteristic while some Webapplications might list "performance" asone of the top quality characteristics. It isimportant to determine some sort of rela-tive importance. Though you could useany meaningful means of noting the rela-tive importance, a percentage for eachquality characteristic is very descriptive.The total percentage should add up to 100.The following table provides an examplefor some components. (See Table 6.)

Identify List of PotentialTest ActivitiesIdentify all of the potential test activitiesthat may apply to the system. It is impor-tant to list the activity for each major teststage even if the system does not use thattest. For example, you might includeRequirements Inspection, Design Inspec-tion, and Code Inspection even if thesetests are not used by this project. Part ofthe benefit of this risk-based strategy is tomake our risks and decisions visible. Byincluding these test activities, even if notused, it shows that a conscious decisionwas made not to use these test activities.Including test activities for all major teststages also makes it clearly visible whatsome of the key risk mitigation opportuni-


Table 5: Example: of Assessing Risk for each Component

Printing from Windows (Port Monitor) HPrinting from remote TCP/IP LPrinting from Windows (SMB) HPrinting from Windows (IPP) Hlp command (high frequency job attributes) Llp command (medium frequency job attributes) Llp command (low frequencey job attributes) Lcancel command Llpstat command LPostScript to Modca Transform MPCL to Modca Transform MIPP Server MIPP Client M

Table 6: An Example of Some Components

(F)unctionality (U)sability (R)eliability (S)uitability (P)erformance

Printing fromwindows (Port 40 20 20 20Monitor)

Printing from Remote 40 30 30TCP/IP

Printing from Windows 40 30 30(SMB)

Printing from Windows 40 30 30(IPP)

lp command (high frequency job attributes) 50 40 10

ties are as risk is assessed for the project.The test activities are the risk mitigationfactors to reduce the chance of error.

The fact that a test activity is not used isn'tnecessarily a negative thing. It just showsthat the project management chose toaddress the risk using a different means.

Some typical test activities include: Requirements Inspection Requirements Review Design Inspection Design Review Code Inspection Code Review Unit Test Integration Test Function Test

Functional Acceptance Test System Test Installation Test Production / Customer Acceptance Test Beta Test

Various organizations will have differentnames for the testing they do. Use what-ever names are meaningful for your organ-ization. However, do not confuse testactivities with test techniques. For exam-ple, white box testing, black box testing,boundary testing, path testing, and cover-age testing are techniques used by unit andfunction testing. They are not the level weare looking for when listing our test activ-ities. (See Table 7.)

Assign Depth of Testing forQuality CharacteristicsPrevious steps in the process have identi-fied the components of the system, whichquality characteristics are important foreach component, and the potential testactivities available for the testing the sys-tem. Next, you need to specify the testimportance for each component. Decidewhich testing activities will be used foreach component and what depth of testingwill be accomplished by each test activity.Keep it simple and avoid misleadingdetails by using a simple notation of +, ++,+++ to indicate test depth. With + indicat-ing a less important test with less depth onup to +++ which is the most important,and an in-depth test.


Table 7: What Test Depth Means for Various Test Activities

Activity + ++ +++

Review/Inspection One person does a A complete formalA formal inspection is held where there aredesk check of another inspection or review is multiple participants, each participant ispersons code or design not being done but there given a checklist with specific things to to look for any possible is obviously significant- check for, guidelines for coverage pace errors ly more in-depth testing (number of pages reviewed per hour) are

than described for +. followed, are perhaps there is a moderator to facilitate the inspection/review.

Unit Test Ad hoc testing without The test is not formal Test plans are used. Coverage requirementsa written plan and no as is described for +++ are in place and tools are used to verify andmeasurements but the test effort is measure the coverage.

significantly more thanis described for +

Function Test Ad hoc testing without Anything that does not Test plans are written and followed.a written plan and no meet the requirements Coverage is understood, complete, planned, measurements for +++ and is signifi- and measured.

cantly more than +

System Test Ad hoc testing, obser- Anything that does not Test plans are written and followed, tests vation testing, testing meet the requirements are complete and are not shortened to without a written plan for +++ and is signifi- accommodate schedule, tests are measured.and little or no measure- cantly more than +ments

The following table gives some examplesand guidelines for determining test depth.This table is not meant to be conclusiveand definitive but rather it is meant to con-vey the general idea of what test depthmeans for the various test activities.

In this example (See Table 8), the follow-ing abbreviations are used for quality char-acteristics.

F = functionality U = usability R = reliability S = suitability P = performance

The test activities importance and testdepth should correlate to the importanceof the quality characteristics for each com-ponent.

Identify Test TechniquesFor Each Test ActivityIdentify the test techniques to be used byeach test activity. This will make it moreobvious if there is redundancy betweentest activities as well as give a clearer pic-ture of just what the test expectations andstrategy are for each component. It helpsgive an understanding of the extent of for-mality, an indication of the skills andresources that may be needed, and if thetest techniques really fit the test impor-tance and depth required.

An example of one component with testtechniques specified. (See Table 9.)

Assign Responsibility ForEach Test ActivityThe responsibility for each test activity,and perhaps each test technique, should beassigned to an individual, department, ororganization. This clearly communicatesexpectations and reduces the chance ofthings falling through the cracks whichalso reduces risk. Assigning responsibilityalso establishes accountability and pro-vides important contact information.


Printing from Windows (Port Monitor)

Printing from remote TCP/IP

Printing from Windows(SMB)

Printing from Windows (IPP)

lp command (high frequency job attributes)

lp command (medium frequency job attributes)

lp command (low frequency job attributes)

cancel command

lpstat command

PostScript to Modca Transform

PCL to ModcaTransform

IPP Server

IPP Client

FunctionTest

F+++

F+

F++

F++

F+++

F++

F+

F+++

F+++

F+++

F+++

F++

F++

System Test

R++S++P+

F+R+++P+

F+R++P+

F+R++

R+P+

R+P+

R++

R++

F++R++P++S++

F++R++P++S++

Table 8: Example of Test Activity and Test Depth.

The table below shows the depth of testing for only Function Test and System Test.

An example of assigning responsibility fortest activities.

Though it is not necessary for the risk-based testing strategy/process, you mayfind it useful to attach dates to the testactivities. (See Table 10.)

Indicate Expected RiskAfter MitigationOnce the risk for a component has beendetermined, don't change it. However,the test activities should reduce the chanceof an error occurring after the product isreleased. It is useful to understand how thetesting has changed the overall risk expec-tations. We can determine how we haveaffected the chance of an error occurring,redo the process of assessing the risk withthe new chance of error, and record theresults with the rest of the risk strategyinformation. Rather than change the origi-nal risk assessment, we will store a newpiece of information called "risk after mit-igation."

A study by Capers Jones regarding theeffectiveness of test stages in removingdefects provides some useful insight intohow to more objectively assess how wellthe test activities/stages remove defects or,in other words, reduce the chance of error.The study gathered data from over 8000projects and provides a much more reli-able indicator of test effectiveness than theusual guess or "gut feel" approach.

The Capers Jones Study provides the fol-lowing information: (See Table 11.)

For our purposes we will assume that atesting stage implies a complete or in-depth test. Since not every test is in-depthwe will extrapolate and say that each set ofthree "+" marks for a component is equiv-alent to a stage. For example, the compo-nent shown below has a total of eight "+"marks. Based on our assumptions we aresaying that this is equivalent to 2 and 2/3of the Capers Jones test stages. (Table 12.)

Table 10: Assigning Resposibility for Test Activities

Function Test System Test Beta TestPrinting from F +++ R ++ (System Test Dept.)Windows (G.A. Johnson) Automated Load Test(Port Monitor) Coverage Test Automated Endurance

Black Box S ++(Will Baker)Equivalence Automated Operational Class Test Profile Test

Ad Hoc Test P + (System Test Dept.)General observations while running other tests

Numberof Testing Stages

1 Testing Stage2 Testing Stages3 Testing Stages4 Testing Stages5 Testing Stages6 Testing Stages7 Testing Stages8 Testing Stages9 Testing Stages10 Testing Stages11 Testing Stages12 Testing Stages13 Testing Stages14 Testing Stages15 Testing Stages16 Testing Stages17 Testing Stages18 Testing Stages

Percent of Effort Devoted to Testing

10%15%20%25%30%33%36%39%42%45%48%52%55%58%61%64%67%70%

Cumulative Defect Removal Efficiency

50%60%70%75%80%85%87%90%92%94%96%98%99%99.9%99.99%99.999%99.9999%99.99999%

Table 11: Capes Jones Study


Table 9: Example of Test Techniques Specified

Function Test System Test Beta TestPrinting from F +++ R ++Windows Coverage Test Automated Load Test(Port Monitor) Black Box Automated Endurance

Equivalence S ++Class Test Automated Operational

Ad Hoc Test Profile TestBoundaries Test P +

General observations while running other tests

Since working with fractions can beinconvenient we will further extrapolateand produce a table of defect removal effi-ciency based on a per "+" mark basis. Thisextrapolation is based on breaking eachstage into three parts and proportioning thecumulative defect removal efficiencyaccording to the trends shown in theCapers Jones Study. (See Table 13.)This table tells us that we should reduceour chance of error for "Printing fromWindows (Port Monitor)" by 67% since ithas 8 "+" marks. Thus, if our originalchance of error for this component was 12our new chance of error would be (1-.67)

x 12 = ~4. Since chance of failure =chance of error x frequency of use and thiscomponent is a high use component, thechance of failure would be (4 x 2) = 8.Even though the chance of failure wouldhave dropped from 24 to 8, this still falls inthe category of a high chance of failure. If

we wanted to reduce the risk sufficientlyto get out of the high category for chanceof failure we would need to provide addi-tional testing. The adjusted chance of errorand risk after mitigation should be record-ed where it can be visible, but do notchange the original chance of error or riskassessment.

Redetermine "Risk AfterMitigation" for any In-Process Changes in TestActivities or SchedulesAny time you add to or take away from thetest activities you correspondingly dimin-ish or add to your chance of error and riskafter mitigation. You can use the processmentioned in the above section "IndicateExpected Risk After Mitigation" to ratherquickly make an objective assessment ofthe potential impact of any test activitychanges. If the project has to eliminatesome testing in order to meet schedulesthis process will show what additional riskyou should expect. If the project decides toadd additional testing the same processshould show how much the risk shoulddiminish.

Risk-Based TestingStrategy and aComprehensive Test PlanThe risk-based testing strategy is a com-pact format of a comprehensive test plan.If you include the dates of the test activi-ties, almost everything required in a com-prehensive test plan is included. You mayhave to provide a small amount of addi-tional information based on your organiza-tions needs and preferences. The compactformat of the risk-based strategy may be amore convenient working version of the


Number of + Marks

12

(1 Complete Testing Stage) 345

(2 Complete Testing Stages) 678

(3 CompleteTesting Stages) 91011



(6 Complete Testing Stages) 18(7 Complete Testing Stages) 21(8 Complete Testing Stages) 24(9 Complete Testing Stages) 27(10 Complete Testing Stages) 30(11 Complete Testing Stages) 33(12 Complete Testing Stages) 36

Cumulative Defect Removal Efficiency

253750545760646770727475777980828485879092949698

Table 13: Defect Removal Efficiency

Table 12: Example of our Assumptions

Function Test System Test Beta TestPrinting from F +++ R ++windows (Port S ++Monitor) P +

comprehensive test plan and could replacethe comprehensive test plan entirely.

Tools To Create And TrackRisk-Based StrategyInformationThough there are numerous ways torecord the information gathered in risk-based testing, it is assumed that a spread-sheet is entirely adequate and convenient.The expandable table format is a perfectfit. The spreadsheet format works finewhen the assessment calculations are donemanually but falls short in usability andconvenience when trying to automate andscript the assessment calculations.

A database with forms as an interfaceseems to simplify the process and hidesmuch of the complexity and confusionthat can accompany the tables andprocesses associated with calculating therisk of a component.

ConclusionRisk-based testing is not an algorithmicprocess that definitively calculates risk. Itis a strategy that introduces some objectiv-ity and visibility to the risk assessmentprocess. It is a tool to provide help in mak-ing decisions rather than a process whichdictates decisions. It guides a project inunderstanding the components that makeup a system, the associated risks of eachcomponent, all the test activities, and howall the parts relate to each other and influ-ence the overall risk of the product as it isput into production.

It makes all of these elements visibleenabling conscious choices to be maderather operating in the out-of-sight out-of-mind mentality.

AcknowledgmentsThis paper is basically an adaptation of apaper presented at Stareast 2000: "A Risk-Based Test Strategy", Dr. Ingrid B.Ottevanger, presented at Stareast 2000,May 2000. In addition, much of the con-tent is a result of a group effort of the IBMBoulder Printing Division Software andBeta Test Department. Without the enthu-siasm, input, and guidance of this group,this document would not exist.

The Capers Jones February 1998 study oftesting stages and their relationship todefect removal is also key to making thisprocess a valuable, in-process, decisionenabling strategy.

About the AuthorKelly Whitmill is employed in thePrinting Systems Division of IBM inBoulder, CO.


San Antonio, TX April 15-19, 2002Boston, MA May, 2002Washington, DC June 24-28, 2002Chicago, IL July 15-19, 2002San Francisco, CA August, 2002

Featuring 15 in-depth one-day coursestaught by leading industry experts in

Software Testing & Quality.

All courses count towards the

Certification of Software Test Professionals.

For additional details and registrationvisit www.testinginstitute.com

The International SoftwareTest Professionals Week

Risk Based Testing

Documents

ill test

timebased testing

minimallyrequired testing

riskbased testing doesnt

test schedule

test cases

casewith software testing

tested duringthe function