Agile High Speed Testing

8/14/2019 Agile High Speed Testing

http://slidepdf.com/reader/full/agile-high-speed-testing 1/112

Agile & High Speed

Software Testing Techniques

Bob Galen

President & Principal Consultant,

RGCG, LLC – Leading you down the path of agility…www.rgalen.com [email protected]

http://www.rgalen.com/

mailto:[email protected]





Fall / Winter 2008 v2Copyright © 2009 RGalen Consulting Group,

LLC 2

Instructor Bio

Bob Galen

25 years of experience developing and testing software systems

15 years of leadership experience at manager & director levels

RGCG, LLC formed in 2000 and located in Cary, NC

Regular speaker at Development, PM, and SQA conferences

Author of Software Endgames and articles in Better Software,

Software Testing & Performance, and QAI journal

Experienced XP Coach, Certified ScrumMaster, and Agilemethods enthusiast

More info at www.rgalen.com/about.html

http://www.rgalen.com/about.html

http://www.rgalen.com/about.html




LLC 3

Basic Protocol

Ask questions, stay engaged, challenge the ideas

It’s our class Class runs from 8:30am – 4:30pm

10 minute breaks about every hour, lunch around 12 for an hour

We may start early / leave early on 2’nd day Please arrive & return from breaks on time

Turn off misc. devices

Please fill out course evaluation and send me personalfeedback as well – [email protected]






LLC 4

Primary Course Goal

Provide a survey of the latest thinking on fast (time

constrained) testing methods

Including an emphasis on Risk-Based Testing (focus) and

stakeholder management (connection & inclusion)




LLC 5

Introduction

I once interviewed for a SQA manager position. Theinterview team was composed of 5 development

managers and 1 director. Oddly enough, there were notesters. Each of them asked me the same hypothetical question…

What would you do if you’d planned on having 6 weeks for application testing. Then development needed more time to complete its work,so your time was reduced to 2 weeks. How would you assure the

same level of quality for the application?

This class is intended to address this (perhaps all too common ) situation.




LLC 6

Introduction

What would you say in this situation?

More importantly, if you indeed had only 2 weeks, what

would you test?

How would you define and achieve excellence in your

testing strategy?

How would you communicate and engage stakeholders

in the risk?




LLC 7

Introduction

Disclaimer You need to keep in mind that there are no Silver Bullets

in software development projects AND this course

promises none

Sometimes lemons are only lemons – no matter how hard you squeeze

What I do promise is real world examples of how toapproach time & resource constrained testing projects

However, you’ll need to bring your courage, integrity, creativity,

and hard work to the process




LLC 8

Course Outline

Introduction

1. Context Based Testing

2. Just-in-Time Testing

3. Lean in Testing

4. Exploratory Testing

5. Risk-Based Testing

6. Pareto-Based Testing

7. AllPairs Technique &Tool




LLC 9

Section 1

Context–Based Testing Outline:

Introduction to the 4 “Schools” of testing

The “context driven” mindset, learning how to adjust

Exploring the dimensions of your context




LLC 10

Context–Based

“Roots” Thoughts originated by Cem Kaner,

James Bach, and Bret Pettichord

Michael Bolton, Mike Kelly, and Rob

Sabourin are other recent

contributors

Pettichord characterized 4 “schools”

in a 2003 presentation

Lessons Learned in Software Testing: A Context-Driven Approach

– published in 2001

http://www.amazon.com/exec/obidos/ASIN/0471081124/satisinc




LLC 11

Context–Based

Core Philosophy There are NO

“Testing Best Practices”

There are only

Good Practices that are Applied in a Specific

Context

In other words – One Size Does Not Fit All Situations




LLC 12

Context–Based

7 Basic Principles of the Context–Driven School1. The value of any practice depends on its context.

2. There are good practices in context, but there are no best

practices.3. People, working together, are the most important part of any

project's context.

4. Projects unfold over time in ways that are often not predictable.

5. The product is a solution. If the problem isn't solved, the productdoesn't work.

6. Good software testing is a challenging intellectual process.

7. Only through judgment and skill, exercised cooperatively

throughout the entire project, are we able to do the right things atthe right times to effectively test our products.

http://www.context-driven-testing.com/






LLC 13

Context–Based

4 SchoolsAnalytic School

Sees testing as rigorous and

technical with many proponents in academia

Exemplar: Code Coverage

Quality School

Emphasizes process,

policing developers and acting as a gatekeeper

Exemplar: The Gatekeeper

Standard (Factory ) SchoolSees testing as a way to measure progress with emphasis on cost and repeatable standards

Exemplar: Requirements Traceability

Context-Driven SchoolEmphasizes people, setting out to find the bugs that will be most important to stakeholders

Exemplar: Exploratory Testing




LLC 14

Context–Based

4 Schools: Risk-Based FocusAnalytic School

Operational profilesCalculate reliability

Quality School

Uncover project risksProve project is not adhering to

process, not repeatable

Standard School

Feature risk assessmentsSpecific schedule risk

Context-Driven School

Develop understanding of risksDevelop tests targeting

identified risks




LLC 15

Context–Based

Context Explored… Dimensions of context

Let’s explore some of the factors that influence testing

project context…

Politics

External customer pressure

Business conditions

Project schedule and status

Methodology

Test team skill set & capabilities Progress to-date

Product technologies

Cross-team capabilities

Distributed team

Overall team size

Development team capabilities

Requirement clarity Level of change

Expectations

History




LLC 16

Context–Based

Context Explored… Plans are road-maps, not straight-jackets

Context is dynamic, changing daily

Part of establishing context is goal setting & focus

It’s about the people – testers and their skills!

Top priority contexts Testing school connections

Time (within the schedule, spent & remaining)

What’s already been done & not done Setting and resetting expectations

Remember: one size never fits all!




LLC 17

One More School – March 2007

Agile School In March of 2007, Pettichord revisited the 4 Schools and

added another…

Key question: Based on customer conversation &

acceptance, is the story complete?

Exemplar: TDD, Unit testing, automated tests

Authors: Bob Martin, Kent Beck, Brian Marick




LLC 18

5 Schools

Evolution

Analytic

Standard

Quality

Context-Driven Agile




LLC 19

Context–Based

Wrap-up What school(s) do your test team(s) align with?

If you were to adopt a context–based approach, whatwould be some of the benefits?

Some of the issues or impediments?

How would you communicate, explain and defend your

context decisions back home – Within your teams?

Within your projects?

To your stakeholders?




LLC 20

Section 2

Just-in-Time Testing Module Outline:

Introduction to Rob Sabourin & JIT

Exploring test ideas – mine, define, prioritize, plan,

chunk, track. JIT Triage & Workflow

Scenario testing

Working with Development




LLC 21

JIT Testing

Foundations Rob Sabourin (www.amibug.com) has defined a set of

related practices for Just-in-Time (JIT) testing. None

are novel or unique, it’s the combination that’simportant.

Central JIT Themes Ruthless Triage

Collaborate

Test Early and Always, 7x24

Adapt – Daily

Chunk Work

Work in Parallel

http://www.amibug.com/

http://www.amazon.com/gp/reader/0968577407/ref=sib_dp_pt/104-1625313-7535114#reader-link





LLC 22

JIT Testing

Foundations

Begin with the End in Mind (Covey’s 7 Habits)

Declare goals How do we know when we’re finished?

Phased exit criteria, release criteria, hand-off criteria – be clear &be concise

What is the overriding purpose of the product?

How do we support it – well? Define testing practices

Not so much process, as rules of the road for this engagement

Bug flows




LLC 23

JIT Testing

Test Ideas Tests are designed as part of a collaborative, note

based, brainstorming process

The idea is to get all test ideas on the table as a team or group

Think of a test idea as:

A succinct thought A singular test

Connecting to central (project ) goals & (application ) themes

Small, low fidelity artifacts; written on a card or post-it note

Requiring high thought, but low (documentation) effort and overallinvestment

Brainstorm, list, sort, organize, shuffle




LLC 24

JIT Testing

Test Idea Groupings From testing ideas build a set of Test Objectives (TO)

Each can be assigned to a tester as work

Each can include all, part of, or multiple testing ideas

Decide how to organize the ideas (work)

Assign testers to TO’s (considering experience & skill,

availability & impact) Work the test ideas / TO’s into sessions or “chunks”

90 – 120 minutes

Related sets of test ideas

Finest granularity of work planning, scheduling & reporting

Chunk execution planned “in the moment” and not well inadvance




LLC 25

JIT Testing

Test Ideas Where to find them?

Creative approaches: lateral thinking, mind maps, and action

verbs

Investigation: historical reviews, oracle interviews, metrics

Bug taxonomies

Requirements: functional, non-functional, use cases, implicit

Usage Scenarios

Functional analysis: CRUD, boundary analysis, prototypes Failure Modes

Domain or Product Quality Factors

In the moment – experience, exploration & discovery




LLC 26

JIT Testing

Test Idea Brainstorming Gather group of testers and stakeholders

Walk in with “product awareness”

Kick-off the meeting

Everyone build their own lists of ideas

Converge ideas – remove overlaps, fill gaps

Prioritize

Estimate size of the ideas Look to balance granularity of the ideas – roughly same size.

Combine / breakdown as necessary Fit into a 90-120 minute sessions or chunks

Discuss workflow – optimum scheduling and work

assignments




LLC 27

JIT Testing

Lifecycle of a Test Idea

Pops into existence

Discussed, clarified, tuned

Prioritized►►►

Integrated into a testingobjective (TO)

Run

Feedback, change, re-

clarified, re-tuned

Example Priority

1. Test now

2. Test before (time)

3. Test before (release)

4. Nice to have5. May be of interest in future

releases

6. Not of interest in current form

7. Will never be of interest




LLC 28

JIT Testing

Triage Consider it the “daily meeting” for JIT targeting

Review the current business & technical context

Review testing context: testing progress, bugs, discovery andnew test ideas

Work sign-up or assignments Best tester (skill), most appropriate (context SME), most available

(time)

Rework the chunking and workflow

Test idea reprioritization New ideas, new chunks, highest priority focused

Should we skip tests (ideas, whole chunks or scenarios)




LLC 29

JIT Testing

Scenario Based Testing Develop a series of typical usage scenarios for each

type of user

Based on functional descriptions – use cases, requirements,

stories, or observation. Could use story boarding

In parallel with other activity

Use the chunking methods to elaborate scenarios Selection of execution team based on experience

Scenarios cover operational threads across sets of

features

Usually only a few critical scenarios, 5-10-20

Pareto applies here, seeking the critical 20%




LLC 30

JIT Testing

Status Reporting Lightweight, reactive, daily in real-time. Synonymous

with the Agile notions of information radiators and daily

stand-up meetings Report test status at the “chunk” level

Elaboration

Planned / executed Passed / failed; confidence levels

Bug trending Open, closed

Fixed, Remaining to verify, Regressions

Provide broad & deep visibility to ALL Invite ALL interested parties, stakeholders, ultra visibility




LLC 31

JIT Testing

Working with Development Influence the development team in the design phases

Can you control the application via API; below GUI layer?

Error & status logging completeness and availability?

Configuration flexibility (setups, queried)?

Influence any specific testing “hooks” for data gathering &

measurement?

Configuration Management

Integration w/smoke testing

Baseline hand-offs for testing; revert easily to previous release Entry criteria; loose, agile, not as an impediment

Development and testing are on separate servers &

environments




LLC 32

JIT Testing

Wrap-up What are the key JIT practices?

Do you do any of this now?

What works – well?

What doesn’t?

How does JIT relate to your understanding of agile

teams?

Is JIT a test team only function?

Where are the collaborations?




LLC 33

JIT Testing

Morning Workshop ( 30 minutes to 1 Hour ) Gather together into teams of 3-6, pick a leader who will facilitate the

brainstorming

Go to www.google.com Your task is to create a set of interesting test ideas for the Google

search engine. Pick your primary feature set or area and start

brainstorming. Your deliverables should include:

A set of at least 50 varied test ideas

Prioritize them according to impact to the feature area you

selected (High, Medium, Low)

Try and estimate how long each idea would take to execute (in10 minute increments)

De-brief the workshop as a group

http://www.satisfice.com/





LLC 34

Section 3

Lean Software Development – Testing ImplicationsModule Outline:

Overview of Lean Software Development principles

Connect LSD towards testing activity Acceptance is important – release criteria & smoke

tests

Adopting a Just in Time and Just Enough mindsettowards application delivery

Testing only when its ready






LLC 36

7 Core Principles of

Lean Software Development

1. Eliminate Waste – don’t do things that don’t add value

to the system – features, architecture, documentation,

bugs. Also process steps – hand-offs, sign-offs,

decision delays, long prioritization steps, etc.

Implications for Testing

Only document relevant tests; depth based on impact

Keep bug list short, relevant and focused

Create verbal, but well understood hand-offs

All testing artifacts need to be actively used; just enough content




LLC 37



2. Build Quality In – Amplify Learning : put processes in

place that enable learning leading to adaptation andimprovements. Build it right the first time.

Implications to Testing

Become active early and often; collaboration with development Observe, learn, and adjust with the evolution of the application –

don’t assume static models

Become a team quality champion, serving as an example by

providing data, insights and techniques Don’t get stuck in approaches; always add value




LLC 38



3. Create Knowledge – Decide as Late as Possible: illustrates the notion that waiting until you know more is

better than the typical delivery, change and rework

cycle. Decrease speculation; replace with knowing.


Become active early and often; collaboration with development

Continuously observe, learn, and adjust with the evolution of theapplication – don’t become too dependent (stuck) in your plans

Become intimate with the customer; understand the true value

drivers for the release




LLC 39



4. Defer Commitment – Deliver as Fast As Possible :

delivery in this context includes not only the product, butall related artifacts. Deliver with high quality too! Speedcan totally disrupt your competition.


Test whenever possible; take incremental releases

Test what works; skip what doesn’t

Provide real-time (daily) constructive feedback for development

Apply risk-based testing techniques

Automate as much as possible; as soon as possible; to reducecycle-time




LLC 40



5. Deliver Fast – Empower the Team : self-directing teams

are a central force within agile and lean practices andthinking. If you want to go fast, you need engaged,thinking people who can be trusted to make good

decisions and help each other out.


Become active early and often; collaboration with development;

truly become an active partner in “paired” development & testing Apply risk-based testing techniques

Automate as much as possible; as soon as possible; to reducecycle-time




LLC 41



6. Respect People – Build in Integrity : the resulting

system is sensible, useful, easy to maintain and adjust,being put together in a professional manor. Keys aregetting the right people.


Be proud of your profession; continuously learn new ways andapproaches for you domain

Remain Open Minded! Become a business partner; become intimate with how customers

USE your applications

Apply scenario based testing




LLC 42



7. Optimize the Whole – See The Whole : don’t allow

strength in particular areas to guide the entire systemsevolution. Think at a “systems” level. Don’t fall into thesub-optimization trap.

Implications for Testing

Apply scenario based testing; carefully consider the broadrequirements for the application

Maintain a broad view to overall product quality; factor in your business dynamics; become a champion for the customer

Continuously increase the capabilities of your team acrosstechnology, testing, and business domains




LLC 43


7 Wastes in SoftwareManufacturing Testing Software

Development

1. In-ProcessInventory

2. Over Production

3. Extra

Processing

4. Transportation

5. Motion

6. Waiting7. Defects

1. Evolving large test plans& scripts

2. 100% coverage goals

3. Loss of context or SME

4. SDLC driven test cycles

5. Parallel testing projects

6. Development schedules7. Defects

1. Partially DoneWork

2. Extra features

(Gold Plating)

3. Relearning

4. Handoffs

5. Task Switching

6. Delays7. Defects




LLC 44

Lean Testing Practices

Wrap-up, Let’s Brainstorm1. Smoke Testing

2. Release Criteria

3. Small releases

4. Collaboration w/customer on Value!

5. Pair w/Development

6. Influence Quality – Design, TDD

7. Limit Waste – Just enough, Just-in-Time

8. Risk Based Testing9. Limit multitasking

10. Impact driven work assignments

11. Automate wherever possible

12. Test constantly

13. Inspect & Adapt

What practices have I missed?

How do we practically use Lean?

Impact?

Hi, Med, Low




LLC 45

Section 4

Exploratory Testing or ETModule Outline:

Overview & history of Exploratory Testing

Role of heuristics (guidelines, hints) in the process

Developing overall goals and session themes The exploratory session – boundaries & examples

When & where to use ET – come up with some

common scenarios that illustrate the “sweet spots” of ET




LLC 46

Exploratory Testing

History The technique was coined by Cem Kaner in his book –

Testing Computer Software

James Bach (www.satisfice.com) has been expandingthe definition & practice of the technique and isconsidered the father Exploratory Testing in Practice

His brother Jon Bach has focused on managing ETfocused projects, so has expertise on the managementside

Wrongly confused with ad-hoc testing. Contrastingcharacteristics include: Trainability, repeatability, focused or tasked, agile uses, and

experience centered






LLC 47

Exploratory Testing

History The technique is the primary

method used for testing withinthe context-based school.

Key proponents who areforwarding the craft include►►►

Variations include JIT, Lean,Rapid Software Testing, etc.

In fact, James Bach’s ETcourse is entitled RapidSoftware Testing

James & Jon Bach

Scott Barber

Michael Bolton

Elisabeth Hendrickson

Mike Kelly

Jonathan Kohl

James Lindsey Rob Sabourin




LLC 48

pure scriptedfreestyle exploratory

chartersvague scripts

fragmentary

test cases

(scenarios) roles

To know where a test falls on this scale, ask

yourself: “to what extent am I in control of the test, and from where did the idea originate?”

Exploratory Testing

Scripted vs. Exploratory Continuum




LLC 49

Some Exploration Skills and Tactics

Modeling

ResourcingQuestioning

Recording

Reporting

Exploratory testing is a mindset using this skillset.

Chartering

ObservingManipulating

Collaboration

Generating/Elaborating

Overproduction/Abandonment

Abandonment/Recovery

Refocusing

Alternating

Branching/Backtracking

Conjecturing

“MR.Q COMC GOARABC R&R?”

Exploratory Testing

Heuristics as a testing device




LLC 50

Exploratory Testing

Skills & Tactics Modeling

Composing, describing, and working with mental models of the

things you are exploring. Identifying relevant dimensions,variables, and dynamics. A good mental model may manifestitself as having a “feel” for the product; intuitively grasping how itworks.

Resourcing Obtaining tools and information to support your effort. Exploring

sources of such tools and information. Getting people to help you.

Questioning Identifying missing information, conceiving of questions, and

asking questions in a way that elicits the information that youseek.




LLC 51

Exploratory Testing

Skills & Tactics Chartering

Making your own decisions about what you will work on and how

you will work. Understanding your client’s needs, the problemsyou must solve, and assuring that your work is on target.

Observing

Gathering empirical data about the object of your study; collectingdifferent kinds of data, or data about different aspects of the

object. Designing experiments and establishing lab procedures.

Manipulating

Making and managing contact with the object of your study;

configuring and interacting with it.




LLC 52

Exploratory Testing

Skills & Tactics Collaboration

Working and thinking with another person on the same problem;

group problem-solving.

Generating / Elaborating

Working quickly in a manner good enough for the circumstances.

Revisiting the solution later to extend, refine, refactor, or correctit.

Overproduction / Abandonment

Producing many different speculative ideas and makingspeculative experiments, more than you probably need, then

abandoning what doesn’t work. Examples are brainstorming, trial

and error, genetic algorithms, free market dynamics.

l




LLC 53

Exploratory Testing

Skills & Tactics Abandonment / Recovery

Abandoning ideas and materials in such a way as to facilitate

their recovery, should they need to be revisited. Maintaining arepository of old ideas.

Refocusing

Managing the scope and depth of your attention. Looking atdifferent things, looking for different things, in different ways.

Alternating

Switching among or contrasting different activities or perspectivesso as to create or relieve productive tension and make faster

progress.

l T i




LLC 54

Exploratory Testing

Skills & Tactics Branching / Backtracking

Allowing yourself to be productively distracted from one course of

action in order to explore an unanticipated new idea. Identifyingopportunities and pursuing them without losing track of the

process.

Conjecturing Considering possibilities and probabilities. Considering multiple,

incompatible explanations that account for the same facts.

E l T i




LLC 55

Exploratory Testing

Skills & Tactics Recording

Preserving information about your process, progress, and

findings. Taking notes.

Reporting

Making a credible, professional report of your work to your clients

in oral and written form.

E l T i




LLC 56

Exploratory Testing

Analogies Job Interview

Warfare Operations

Newspaper Reporter

Detective

Others?

Bounty Hunter

20 Questions

Psychologist

Going to a conference

Lewis & Clark

E l T i




LLC 57

Exploratory Testing

Session Strategy Exploratory Testing proceeds in a series of

interconnected sessions that are focused on a specific

testing project (application)

Planning the project encompasses establishing a set of

time boxed session charters

Establishing roles and focus areas for the sessions or

groups of sessions

Establishing the session execution dynamics

Starting, Stopping, Re-Chartering, Reporting

Reporting progress to stakeholders & re-establishing the

overall test strategy / charter

E l T i




LLC 58

Exploratory Testing

Session Dynamics Sessions are focused ET events

They are limited in duration (60-120 minutes)

They have a session charter, goal, or focus

The results of the session are captured in a log

Testing path logged, findings & bugs reported, repeatable steps

Sessions are de-briefed (retrospective) with re-chartering asrequired for subsequent sessions

A day of testing is composed of multiple sessions

Often there is a sense of collaboration in the sessions Paired testers; Paired w/developers

Co-located in the same room; lab area

Shared / common data & environment

E l T i




LLC 59

Exploratory Testing

Roles & Feature Areas Roles, as a…

Power user

Specific clients /configurations

User communities

Administrator

Feature Areas of focus… Installation

Compatibility

Database integrity 3’rd party add-ins

Configuration & setup

Usability

Tester, testability

Compliance user Process owner

Business user

Performance

Load

Online help and docs Security

Interoperability

Beta

Exploratory Testing




LLC 60

Observations

(To the degree you think they are

relevant to stakeholders)

• feature model

• text from log files

• text from dialogs

Conjectures

(Inferences based on

experiences)

• test ideas

• questions

• product and project issues• concerns

• risks

Project information

(Independent of observer)

• charter

• test actions• config info

• build details

• tools used

Exploratory Testing

What to Write While Exploring…

E l t T ti




LLC 61

Exploratory Testing

Sweet Spots

Any extremely time constrained testing situation

Anytime you have a lot of ambiguity (stability, feature

operation, etc.)

When you’re blessed with lots of solid domain

experience, SME breadth

Smoke testing; Does it work?

Acceptance testing; Did I get what I expected?

Beta testing; Will we embarrass ourselves? Agile testing – daily explorations: What works? What

doesn’t? Progress? Feedback for development…

E l t T ti




LLC 62

Exploratory Testing Supporting Tools

Philosophically Exploratory Testing is not a tool basedactivity, it’s a human experience based one. So tool

requirements are minimized. That being said…

The following can be useful –

Any tools that allow you to capture screen state information – ex:Spector

Quick, UI interaction tools – ex: Perlclip

Fast logging / scripting tools; ex: Log-Watch

Web based DTS Wiki’s

TestExplorer

E l t T ti




LLC 63

Exploratory Testing Wrap-up

Let’s compare & contrast JIT and ET Where are the similarities?

Do you see differences?

How would an entire ET test project work? Planning? Execution?

Reporting? Adjustment?

Do any of you use the technique now? What are the

sweet spots? Where are the challenges? Suggestions on how to use it?

Read - Context-Driven Yahoo Group discussion !

E pl r t r T ti




LLC 64

Exploratory Testing Afternoon Workshop ( 30 minutes to 1 Hour )

Gather together into the same teams from the morning exercise

Using the same testing focus and test ideas, your task is to

strategize session based exploratory testing and execute a few short(10 minute) sessions

Using your test ideas from the morning session, create a set of interrelated session charters to plan & focus your testing efforts.Your deliverables should include:

The test idea estimates from the morning should be reevaluatedand adjusted as necessary

Combine related test ideas into a set of at least 5-10 sessioncharters that should cover 10 minute time boxed sessions.

Every team member should execute 1-3 sessions and log their “travels”


S ti 5




LLC 65

Section 5

Risk–Based Testing Module Outline:

Realizing you can’t cover it all!

Product decomposition – making a feature map

Risk planning – collaborative analysis with stakeholders

of types of testing, timing of testing, pervasiveness of

testing

Testing phases and activities – selecting the right

places to “focus”

Other management considerations

Ri k B d T ti B k d




LLC 66

Risk–Based Testing Background

It starts with the realization that you can’t test everything – ever!

100% coverage being a long held myth in software development

There are essentially 5 steps in most of the models1. Decompose the application under test into areas of focus

2. Analyze the risk associated with individual areas – technical,quality, business, schedule

3. Assign a risk level to each component4. Plan test execution, based on your SDLC, to maximize riskcoverage

5. Reassess risk at the end of each testing cycle

Ri k B d T tin B k r nd




LLC 67

Risk–Based Testing Background

Risk–Based Testing is effectively a risk mitigation

technique

Not a prevention technique

It’s about trade-offs

Human and physical resources

Ratio’s between Producers (Developers) and Consumers

(Testers)

Time Rework (retesting & verification)

Quality – Coverage vs. Delivery

Visibility into the trade-offs

Risk Based Testing




LLC 68

Risk–Based Testing Decomposition

Application-to-testing decomposition is the first actionyou face. Breaking down the application and its

associated testing artifacts. I prefer a 3 tiered model for test decomposition:

1. Suites: A collection of test cases, focused on testing a relatedset of functions, features or requirements.

2. Cases: A collection of steps, focused towards testing a specificfunction, feature or requirement; It should have a succinct goal

3. Steps: Individual test steps executing the test case

Suites should reflect not only functional, but non-functional requirements.

Risk Based Testing




LLC 69

Risk–Based Testing Decomposition

For risk analysis, we’ll operate at the Suite level

For each Suite, you should determine: Approximate # of cases per suite

Complexity of the Suite (Very, Average, Simple)

Time to design tests for the suite Time to setup & teardown the environment for the suite

Time to run tests for the suite & handle “typical” defects #’s

These are high level estimates, based on historywhenever possible, and will serve as a sizing indicator when we’re prioritizing and scheduling suites

Risk Based Testing




LLC 70

Risk–Based Testing Risk Prioritization

In traditionalprioritization

(Probability: 0-3) x

(Impact: 0-3) = Risk

Level

Calculate a risk level for

each suite

Rank order them Decide how to cover

each via test plans &

schedules

A testing version, more dimensions:

Business Importance

Probability of Instability Overall Complexity

Coverage & Timing

First release (cycle)

Middle cycles

Last release(s) (2 cycles)

Pre-Production Release

Instead of a static view, a continuum

view across different stakeholder

perspectives



Risk Based Testing




LLC 72


Agile – Planning Poker, Risk Planning Workshop

Include stakeholders and interested parties – BA, Architects,

Developers, Testers, PM’s, Stakeholders, Management, etc. Have the nearest SME / feature owner / BA overview a Test

Suite (feature set or requirement set)

Ask questions

Constituents vote via cards on their views to –

Business Importance, Probability of Instability, Overall Complexity,

Coverage & Timing

High & Low participants discuss the “why” behind their views Re-vote until you converge as much as possible

Repeat…

Risk Based Testing




LLC 73


Bottom & Top 20% exercise A typical outcome of all prioritization efforts is that everything (or

+80%) is rated as High While this may be true from individual perspectives, it doesn’t

help the testing team make good choices

At the end of the workshop, have everyone “vote” on

their Top 20% and Bottom 20% using dot voting. Give each of your primary constituents (Development, Business,

Testing) different colors

This will help normalize the feedback while providing goodinsight across perspectives

Remember: all “ties” are broken by the testing team!

Risk–Based Testing




LLC 74

Risk–Based Testing Risk Scheduling & Tracking

Once you have your overall risk assessment and

cyclical feedback, you need to create a plan & schedule

that reflects the tempo and cyclical testing requirementsof your SDLC

Iterative or agile methodologies require more testingcycles

They also increase the complexity of your planning to sensibly

handle rework (re-testing, regression, integration, and repair verifications)

Ensure you don’t over-test, by testing too soon, then too often





LLC 75

Risk–Based Testing Worksheet Example

Download sample worksheet from

http://www.rgalen.com/t_files/AgileAndHSpeed-1dayMaterialsV2.zip

is a zip file link for class supporting documents… This is simply a sample worksheet that I’ve used for risk-based

testing.

It tries to capture:

Complexity

Design, Setup, and Execution time at a suite level

Handle different iterations in a risk–based manor:

High – person days as is

Med – person days / 2; Low – person days / 4

Consider it a high-level, risk-based planning tool for testing projects

(time, focus, & resources – to achieve a balance )

Risk Based Testing






LLC 76


Test Suite – Execution Plan

Test Suites # o f T e

s t C a

s

A v e

r a g e S

i z e

A v e

r a g e

C o m

p l e x i t y

O

p e n i n

g -

P r i o r

i t y

M

i d d l e G

a m e

P r i o r

i t y

E n d

G a m

e

P r i o r

i t y

D e s

i g n T i m e

S e t u

p T i m e

E x e

c u t i o

n T i

I n

i t i a l R e l e a

s

C 1

C 2

C 3

F i n a

l R e l e a

s e

Accept - Smoke 25 S M High High Med 4 1 2 3 2 2 1 2

Func - Database Meta-data Integrity 45 M H Low Med High 5 3 3 4 2 2 3 3

Func - Mware, Business rules 75 L H Low High High 10 2 5 3 5 5 5 5

Func - Real-time data 30 M M High Low High 5 1 2 3 1 1 2 2

Func - Intelligent Searching 45 L M High High High 5 5 3 8 3 3 3 3

Func - Area 3 25 S T Med Med Med 5 1 2 2 1 1 1 2Func - Area 4 40 S T Med Med Med 5 1 2 2 1 1 1 2

Func - Area 5 45 S T Med Med Med 5 1 2 2 1 1 1 2

Func - Common UI Features 150 S T Med Med High 15 2 10 7 5 5 10 10

Comp - Operating systems 30 S T Low Low High 2 3 3 4 1 1 3 3

Comp - Browsers & databases 130 S M Low Low High 3 10 5 11 1 1 5 5

Perf - 5 sources, 5 user scenarios 25 L H Low Med High 15 3 5 4 3 3 5 5Defect Verifications N/A N/A N/A Low High Low 5 1 5 2 5 5 1 5

Regression N/A N/A N/A High Low High 0 1 15 16 4 4 15 15

Automation N/A N/A N/A Low Low Low 10 1 5 2 1 1 1 5

Total Test Cases 665.0 Totals: 94 36 69

Average / time per test case 0.30 Total Time 74 35 35 58 69

Test team members 3.5 Team/Person Days 21 10 10 16 20





LLC 77

Risk Based Testing Your Context

A big part of your risk–based approach has to be

grounded in your context as it relates to

Team ratio’s

Equipment investments & capacity

Methodology

Product domain & any regulatory or compliance requirements Testing process and activities

Automation strategies

No hard lines; always trade-offs





LLC 78

Model Test Setup Test Planning Test Execution TestAutomation

Waterfall Large scale, early

on, dedicatedequipment

Traditional

System Test view

Single pass w/

limited rework

Executed but

rarelydeveloped till

the next release

RUP Enterprise scale,

early on, often

shared equipment

Incremental Test

view

Iterative passes,

moderate - heavy

rework

Executed but

rarely

developed till

the next release

Agile Small scale, often

shared

environmentsuntil later

iterations

TDD model,

planned within

developmentiterations

Within

development

iteration – unit &acceptance

focused

Automated unit,

smoke, and

acceptancetests; minimal

regression

Risk Based Testing Methodology Implications for Testing





LLC 79

Risk Based Testing Wrap-up

What are some of the other factors that influence your projects’ risk?

Is automation a risk mitigation technique?

Stakeholders need to be connected. So far in thecourse… Any lessons in how to effectively do that? Silver Bullets?

Lessons from your own experience?

Risk planning is all about risk collaboration and sharedresponsibility. Agree? Do you have experiences or techniques to share?





LLC 80

Risk Based Testing Methods Comparison

Types ofContext-Based

Testing

Risk Based ExploratoryTesting

JIT

Management &Tracking level

N/A N/A Test Usage

Scenarios

Test Suites Charter-driven

sessions

Ideas “chunked”

into TestObjectives

Team

Collaboration &Execution

Test Cases Test Ideas,

heuristics-driven

testing

Test Ideas

Test Steps N/A N/A

Section 6




LLC 81

Section 6

Pareto-Based Testing Module Outline:

History of Pareto

Charting

Using Pareto as a risk-based localization technique Monitoring risk migration

Pareto




LLC 82

ParetoPrinciple

Italian economist Vilfredo Pareto observed that -

For many phenomena, 80% of the consequences stem

from 20% of the causes

When analyzing personal wealth distribution in Italy.

Also known as the 80-20 rule, the law of the vital few, and the

principle of factor sparsity

Joseph Duran brought the principle forward as a potential qualitymanagement technique

In probability theory referenced as a Pareto distribution

Pareto Principle




LLC 83

Pareto Principle“Thinking” Examples

In a Toyota Prius warehouse –

20% of the component boxes take up 80% of the space

20% of the components make up 80% of the overall vehicle cost

In software applications –

20% of the application code produces 80% of the defects

20% of the developers produce 80% of the defects

20% of the test cases (ideas) find 80% of the defects

20% of the test cases (ideas) take 80% of your time to design &test

20% of the product will be used by 80% of the customers

20% of the requirements will meet 80% of the need

Pareto Principle




LLC 84

o c p“Thinking” Examples

Leads to the notion of defect clustering. Many haveobserved that software bugs will cluster in specific

modules, classes, components, etc.

Think in terms of stable or well made components versus

error-prone, unstable, and fragile components. Whichones should receive most of your attention? Do theareas remain constant?

Often, complexity plays a large part in the clustering.Either solution (true ) complexity OR gold-plating(favored ) complexity.

Open Defects per Functional Area




LLC 85

p p Trending – Pareto (80:20 Rule) Chart

Sample Pareto Chart

30

25

15

10 10

530

5570

8090

100

0

5

10

15

20

25

30

35

UI Mware Parsing SOAP Reports Help

D e f e c t s

0

20

40

60

80

100

120

# Bugs

Cum %





LLC 86

p p“Rolling” Pareto Chart


0

5

10

15

20

25

30

Jan 1-15 Jan 16-31 Feb 1-14 Feb 15-28 Mar 1-15 Mar 16-30Project weeks

# o

f D e f e c t s

Install & Config Internal files Dbase Reporting

R-time analysis Off-line analysis GUI Help & docs

Pareto Principal




LLC 87

pStep 1 – Application Partitioning

The first major challenge to Pareto-Based risk analysis ismeaningfully partitioning your application. Here are some

guidelines – Along architectural boundaries – horizontally and/or vertically

Along design boundaries

At interface points – (API, SOA points, 3’rd party product

integrations, external data acquisition points)

Always do this in conjunction with the development team

The partitioned areas need to be balanced – inapproximate size & complexity

Shoot for 5-12 meaningful areas for tracking

Pareto Principal




LLC 88

pStep 2 – Defect Tracking Setup

Modify your DTS to support specific applicationcomponent areas

During triage, effectively identify and assign defectrepairs and enhancements to component areas

Early on, testers will need development help to clearly identifyroot component areas (about 20% of the time)

If you have historical defect data (w/o partitioning), you

can run an application analysis workshop to partitiondata (post release) for future predictions

It does require discipline and a little extra effort…

Pareto Principal




LLC 89

p Application Analysis Workshop

Sometimes you don’t have the time to start Paretotracking before starting a project, so reflectively analyze

Pareto for future planning –

Decompose your application or a sub-component of it if pressedfor time

Gather defects surfaced

Gather your team (developers, testers)

Discuss locale for each bug and create distribution

Off-line create your curves and publish insights for the “next”release

Can also help fine-tune decomposition areas and train the testteam in defect localization

Pareto Principal




LLC 90

pStep 3 – Observations & Adjustments

Project trending at a component level Look for migration of risk and make adjustments

Look for stabilization or regressions (risk) Identify high risk & low risk component areas at a project level

Map component rates to overall project goals

Trend open & high priority defects at a component level

Track or predict project “done”ness at a component level

Weekly samples of 20% component focus areas –

looking for risk migration Sample weekly, then adjust focus across your testing cycles or

iterations

Pareto Principal




LLC 91

p Tools

Excel can be used to display Pareto like charts, with the

cumulative percent trend needing to be simulated

There are other packages available that will properly

calculate & display Pareto Charts for you. Keeping in

mind that it’s a Six Sigma tool, many are associated withsupporting it.

Section 7




LLC 92

All-Pairs Technique & ToolModule Outline:

Introduce the All-Pairs testing technique

Explore scenarios where All-Pairs technique is

particularly helpful

Explore a hands-on session where the class takes an

example project and produces All Pairs test case

definitions

All-Pairs Testing




LLC 93

g

All-Pairs testing is a method of handling large scalecombinatorial testing problems

Also referred to as Pairwise, Orthogonal Arrays, andCombinatorial Method

Instead of attempting to test all combinations, often avery intimidating figure, it identifies all pairs of variables

that need to be tested in tandem – to achieve reasonablyhigh coverage.

Three primary references include –

Lee Copeland – A Practitioners Guide to Software Test Design James Bach – Open Source, AllPairs implementation

Bernie Berger – Efficient Testing with All-Pairs 2003 StarEastpaper

All-Pairs Testing




LLC 94

gInteroperability Testing

One sweet spot area for All-Pairs testing is interoperability.

Something that faces web application testers every day.

In this example, we want to examine browser compatibility acrossthis specific set of system software levels – focusing on the browser

Considering all combinations, there are (4 x 7 x 4 x 2) or 224

possible test cases for the example.

Client OS Browser App Server Server OS

Win NT IE 5.5 WebSphere Win NT

Win 98 IE 6.0 WebLogic Linux

Win 2000 IE 6.5 Apache

Win XP IE 7.0 IIS

FireFox 1.0

FireFox 2.0

Opera 9.1

All-Pairs Testing




LLC 95

gExample

In All-Pairs test design we are concerned with

Variables of a system

Possible values that variables could take

Then we generate a list of test cases that represent the

pairing of variables (all pairs) as the most interesting setof test cases to approach in your design

All-Pairs Testing TEST CASES

case Client OS Browser App Server Server OS pairings

1 Wi NT IE W b h Wi NT 6




LLC 96

Example

Using ALLPAIRS on theprevious example, wewould identify 28 test cases

as an alternative to the 224for absolute coverage.

We’d then use this output

as guidance whendesigning our test cases.

Note the ‘~’ indicates a don’t care for this variable

1 Win NT IE 5.5 Websphere Win NT 6

2 Win 98 IE 5.5 WebLogic Linux 6

3 Win NT IE 6.0 WebLogic Win NT 5

4 Win 98 IE 6.0 Websphere Linux 5

5 Win NT IE 6.5 Apache Linux 6

6 Win 98 IE 6.5 IIS Win NT 6

7 Win 2000 IE 7.0 Apache Win NT 6

8 Win XP IE 7.0 IIS Linux 6

9 Win 2000 FireFox 1.0 Websphere Linux 5

10 Win XP FireFox 1.0 WebLogic Win NT 511 Win NT FireFox 2.0 IIS Linux 4

12 Win 98 FireFox 2.0 Apache Win NT 4

13 Win 2000 Opera 9.1 WebLogic Linux 4

14 Win XP Opera 9.1 Websphere Win NT 4

15 Win 2000 IE 5.5 IIS ~Win NT 3

16 Win XP IE 5.5 Apache ~Linux 317 Win 2000 IE 6.0 Apache ~Win NT 2

18 Win XP IE 6.0 IIS ~Linux 2

19 Win 2000 IE 6.5 Websphere ~Linux 2

20 Win XP IE 6.5 WebLogic ~Win NT 2

21 Win NT IE 7.0 Websphere ~Win NT 2

22 Win 98 IE 7.0 WebLogic ~Linux 223 Win NT FireFox 1.0 Apache ~Linux 2

24 Win 98 FireFox 1.0 IIS ~Win NT 2

25 Win 2000 FireFox 2.0 WebLogic ~Win NT 2

26 Win XP FireFox 2.0 Websphere ~Linux 2

27 Win NT Opera 9.1 IIS ~Win NT 2

28 Win 98 Opera 9.1 Apache ~Linux 2

All-Pairs Testing




LLC 97

Intent

Defects

The hope of All-Pairs testing is that by running from 1-20% of

your test cases you’ll find 70% - 85% of your overall defects

Coverage

By way of example (Cohen) a set of 300 randomly selected testcases provided 67% statement coverage and 58% decision

coverage for an application. While 200 All-Pairs derived test

cases provided 92% statement and 85% decision coverage.

Important tests can be missed. Use sound judgment

when creating tests and add as required

All-Pairs Testing




LLC 98

Intent

All-Pairs is simply a tool in your test design arsenal.

Don’t use it alone or blindly!

You won’t find all of your bugs exclusively using this tool!

Often the strategy is to use All-Pairs to establish your

baseline set of test cases

Then analyze other business critical combinations and add risk-

based tests as appropriate

All-Pairs Testing




LLC 99

Brainstorming Value Proposition

What are some

testing area

opportunities for All-Pairs?

What are not?

UI type input / output variation testing

(functional)

Cross-platform (interoperability) testing Anything with high numbers of variables

Scenario based testing, with path

(variable) variation

Performance testing, and most other

non-functional testing Exploration

Using it solely to derive your test cases

All-Pairs Testing




LLC 100

Fails when…

A few cautions from James Bach & Patrick J. Schroeder in

paper –

Pairwise Testing: A Best Practice That Isn’t

You don’t select the right values to test with

When you don’t have a good enough oracle

When highly probable combinations get too littleattention

When you don’t know how the variables interact

All-Pairs Testing




LLC 101

Microsoft - PICT

Pairwise Independent Combinatorial Testing tool

Features Model Driven w/allowance for sub-models of variable pairings

Allows for specification of constraints and conditional constraints

Allows for parameters and aliasing Supports variable weighting and seeding for reuse of previous

testing cases and “fixed” tests

Source not available, simply a command line drivenutility Link may be found at http://pairwise.org

All-Pairs Testing

http://pairwise.org/





LLC 102

PICT Example w/Sub Groups

PICT Model

PLATFORM: x86, ia64, amd64

CPUS: Single, Dual, QuadRAM: 128MB, 1GB, 4GB,64GB

HDD: SCSI, IDE

OS: NT4, Win2K, WinXP,Win2K3

IE: 4.0, 5.0, 5.5, 6.0APP: SQLServer, Exchange,

Office

{ PLATFORM, CPUS, RAM, HDD }

@ 3{ OS, IE } @ 2

$|

| order = 2 (defined by /o)|

+-------------------------------------+-----------------------------+| | || order = 3 | order = 2 || | |

{ PLATFORM, CPUS, RAM, HDD } { OS, IE } APP

Subgroups are useful for grouping sets of variables with strong

relationships

All-Pairs Testing




LLC 103

“Extra Credit” Workshop

Go to www.satisfice.com

Download ALLPAIRS to your laptop & unzip the tool

Gather together into “pairs” of 2-3 – no pun intended Read the included ALLPAIRS manual (it’s a short read )

Using all the information you’ve been exposed to so far:

1. Setup a spreadsheet and run ALLPAIRS for the example in theslides

2. Setup a spreadsheet and run ALLPAIRS for Bernie’s Mortgage

example

3. Within your pair, discuss the tool and modify one of the examplesin a significant way (or create your own). Re-run ALLPAIRS.


Class Wrap-up






LLC 104

Questions?

Thank you!

References & Backup




LLC 105





LLC 106

Test Workflow

Initial TestingCycle

InitialReleas

e

Test Execution –Iterations or

Cycles

Subsequent

– IterativeReleases

Final

Release

Test PreparationRisk

BasedTests &

Coverage

Risk

BasedTests &

Coverage

Core Testing References




LLC 107

Beck, Kent, “Test-Driven Development: By Example”, Addision Wesley, (2003) Black, Rex, “Managing the Testing Process – 2’nd Edition”, Wiley, (2002) Black, Rex, “Critical Testing Processes: Plan, Prepare, Perform, Perfect”, Addison Wesley,

(2004)

Copeland, Lee, A Practitioner’s Guide to Software Test Design”, Arctech House, (2004) Crispin, Lisa and House, Tip, “Testing Extreme Programming”, Addison Wesley, (2002) Dustin, Elfriede, “Effective Software Testing: 50 Specific Ways to Improve Your Testing”,

Addison Wesley, (2003) Galen, Bob “Software Endgames – Controlling Mastering the Software Project Endgame”,

Dorset House Publishing, (late 2003 – early 2004)

Kaner, Cem, Bach, James, and Pettichord, Bret, “Lessons Learned in Software Testing – AContext Driven Approach”, Wiley, (2002) Kaner, Cem, Falk, Jack, and Nguyen, Hung Quoc, “Testing Computer Software”, Wiley,

(1999) Larman, Craig, “Agile & Iterative Development – A Manager’s Guide”, Addison Wesley,

(2004) Mugridge, Rick and Cunningham, Ward, FIT for Developing Software: Framework for

Integrated Tests”, Prentice Hall, (2005) Petschenik, Nathan, “System Testing with anAttitude: An Approach That Nurtures Front-Loaded Software Quality”, Dorset House, (2005)

Poppendieck, Mary & Tom, “Lean Software Development – An Agile Toolkit”, AddisonWesley, (2003)

Poppendieck, Mary & Tom, “Implementing Lean Software Development – From Concept toCash”, Addison Wesley, (2006)

Core Web References




LLC 108

Context-Based Testing 4 Schools paper – http://www.io.com/~wazmo/papers/four_schools.pdf (Updated in

March 2007 for Factory School name change and to add the Agile School)

Portal - http://www.compendiumdev.co.uk/page.php?title=context_driven_testing Bret Pettichord’s site – http://www.pettichord.com/


Just-in-Time (JIT) Testing Rob Sabourin has established the practice set called JIT. More on Rob at

www.amibug.com

He’s presented the techniques at Star (www.sqe.com) and ST&P (www.stpcon.com)conference. Look up information on past programs

JIT techniques in ST&P article - http://www.stpmag.com/issues/stp-2005-06.pdf

http://www.io.com/~wazmo/papers/four_schools.pdf

http://www.compendiumdev.co.uk/page.php?title=context_driven_testing

http://www.pettichord.com/



http://www.sqe.com/

http://www.stpcon.com/

http://www.stpmag.com/issues/stp-2005-06.pdf

http://www.stpmag.com/issues/stp-2005-06.pdf

http://www.stpcon.com/

http://www.sqe.com/



http://www.pettichord.com/

http://www.compendiumdev.co.uk/page.php?title=context_driven_testing

http://www.io.com/~wazmo/papers/four_schools.pdf



Core Web References




LLC 110

Jon Bach & Quardev whitepapers - http://www.quardev.com/whitepapers.html

http://www.informit.com/articles/article.asp?p=405514&seqNum=5&rl=1

James makes available his RST slides and reference materials at –

http://www.satisfice.com/rst.pdf http://www.satisfice.com/rst-appendices.pdf

Risk-Based Testing References Florida Institute of Technology On-line course -

http://www.testingeducation.org/BBST/BBSTRisk-BasedTesting.html

Solid research paper - http://www.amland.no/WordDocuments/EuroSTAR99Paper.doc

http://www.csr.ncl.ac.uk/FELIX_Web/1A.R-BT%201.pdf

http://home.c2i.net/schaefer/testing/risktest.doc

Johanna Rothman’s Dev-to-Tester Ratio article –http://www.jrothman.com/Papers/ItDepends.html

Planning Poker - http://www.planningpoker.com/

Core Web References

http://www.quardev.com/whitepapers.html


http://www.satisfice.com/rst.pdf

http://www.satisfice.com/rst-appendices.pdf


http://www.amland.no/WordDocuments/EuroSTAR99Paper.doc



http://www.jrothman.com/Papers/ItDepends.html

http://www.planningpoker.com/

http://www.planningpoker.com/

http://www.jrothman.com/Papers/ItDepends.html



http://www.amland.no/WordDocuments/EuroSTAR99Paper.doc


http://www.satisfice.com/rst-appendices.pdf

http://www.satisfice.com/rst.pdf


http://www.quardev.com/whitepapers.html




LLC 111

Pareto Principal http://en.wikipedia.org/wiki/Pareto_principle Research paper - http://cmriindia.nic.in/PROCEEDINGS%20OF%20ACIT-

%202006/SESSION%201A/S%20Samaddar%20&%20S%20Bhattacharya.doc

All-Pairs Wikipedia - http://en.wikipedia.org/wiki/All-pairs_testing James Bach tool download - http://www.satisfice.com/tools.shtml Bernie Berger’s 2003 StarEast paper source -

http://www.stickyminds.com/getfile.asp?ot=XML&id=6488&fn=XDD6488filelistfilename1.pdf James discussing the Microsoft PICT (free) All-Pairs generation tool –

http://www.satisfice.com/blog/archives/53 http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-

8c6f76cc014b/pict33.msi

http://pairwise.org PICT Overview - http://ripper.eeginc.com/mercury/1134/wed/37633_Merrell_330.pdf Constraints? http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

Pairwise.org - http://www.pairwise.org/default.html Michael Bolton paper - http://www.developsense.com/testing/PairwiseTesting.html

Contact Info

http://en.wikipedia.org/wiki/Pareto_principle

http://cmriindia.nic.in/PROCEEDINGS%20OF%20ACIT-%202006/SESSION%201A/S%20Samaddar%20&%20S%20Bhattacharya.doc




http://en.wikipedia.org/wiki/All-pairs_testing

http://www.satisfice.com/tools.shtml

http://www.stickyminds.com/getfile.asp?ot=XML&id=6488&fn=XDD6488filelistfilename1.pdf



http://www.satisfice.com/blog/archives/53

http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi



http://ripper.eeginc.com/mercury/1134/wed/37633_Merrell_330.pdf

http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

http://www.pairwise.org/default.html



http://www.developsense.com/testing/PairwiseTesting.html

http://www.developsense.com/testing/PairwiseTesting.html



http://ripper.eeginc.com/mercury/1134/wed/37633_Merrell_330.pdf




http://www.satisfice.com/blog/archives/53



http://www.satisfice.com/tools.shtml

http://en.wikipedia.org/wiki/All-pairs_testing



http://en.wikipedia.org/wiki/Pareto_principle



Robert Galen

RGalen Consulting Group, L.L.C.

PO Box 865, Cary, NC 27512

919-272-0719www.rgalen.com

[email protected]

Software Endgames: Eliminating Defects,Controlling Change, and the Countdown to On-Time Delivery published by Dorset House

in Spring 2005. www.rgalen.com for order info, misc. related presentations, and papers.





