AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada [email protected].

AmiBug.Com, Inc.

April 10, 2023 © Robert Sabourin, 2012 Slide 1

Just-In-Time TestingWorkshop

Robert Sabourin

AmiBug.Com, Inc.

Montreal, [email protected]

AmiBug.Com, Inc.


Overview

WelcomeSome PhilosophyBe Pre[paredTesting IdeasTest Triage

AmiBug.Com, Inc.


Just In Time Testing

• Robert Sabourin , Software Evangelist

• President• AmiBug.Com Inc.• Montreal, Quebec,

Canada• [email protected]• www.amibugshare.com

AmiBug.Com, Inc.


Just-In-Time Testing

Welcome

AmiBug.Com, Inc.



• Pain points?– What hurts?– How Much?

AmiBug.Com, Inc.



Just In Time For ________?

AmiBug.Com, Inc.



Some Philosophy

AmiBug.Com, Inc.


Fundamental Question

• How do you know when you are finished?

AmiBug.Com, Inc.


Crosby on Quality

• “Quality is defined as conformance to requirements”

• “Quality is not a measure of GOODNESS”– Phil B. Crosby, Quality is Free

AmiBug.Com, Inc.


“Quality is fitness for use”

Joseph Juran

Quality Control Handbook

AmiBug.Com, Inc.


Gerald M. Weinberg

“Quality is value to some person”

Exploring Requirements Quality

Before Design Dorset House

AmiBug.Com, Inc.


Edsger W. Dijkstra

• “Program testing can be used to show the presence of bugs, but never to show their absence”

AmiBug.Com, Inc.

Quiz

• Application screensare selected with three controls:(a) has 5 options

(b) has 6 options

(c) has 2 options

• How many screens can a user choose?


AmiBug.Com, Inc.

Quiz

• Total Combinations = 6 x 5 x 2 = 60

• To exercise each combination once a total of 60 tests would be required.


AmiBug.Com, Inc.

• How many tests would be required to exercise all possible screens in every possible order?.

April 10, 2023 © Robert Sabourin, 2012

Slide 15

Quiz

AmiBug.Com, Inc.

• To exercise all screens in every possible order would require

60! Test cases

60! = 60 x 59 x 58 x ... 3 x 2 x 1

60! ≈ 8.32 x 10**81


Pop Quiz

Slide 16

AmiBug.Com, Inc.

Pop Quiz

From 7.0 × 10**79

To 1.5 × 10**82


How many atoms are in the observable universe?

Slide 17

AmiBug.Com, Inc.


Control Flow Testing

• Model flow– Create control flow

diagram– Find basis paths

• Minimal set of transactions• Exercise at least once

– Each step– Each decision

AmiBug.Com, Inc.



– Minimal basis paths

• N – number of nodes

• E – number of edges

• P – number of basis paths

• P = E – N + 2

• McCabe Cyclomatic Complexity

B

A

C D

F G

E

H

I

N=9E=10

P=10-9+2P=3

AmiBug.Com, Inc.



– Finding basis paths:

1. Start with a typical baseline2. Flip first decision keep rest as

similar as possible3. Continue flipping decisions on

baseline4. After all decisions on baseline

have been flipped continue on next path

5. Stop when all paths have been exhausted

B

A

C D

F G

E

H

I

N=9E=10

P=10-9+2P=3

AmiBug.Com, Inc.



AmiBug.Com, Inc.


AmiBug.Com, Inc.

ST

Di

M

R

N

Q

P

L A

B

C

D

E

FG

I

J

K

O

H

S

T

U

V W

X

ZY

ACAD

AEAF AA

AG AH AJ ABAMAL

AIAK

AN

AO APAQ


AmiBug.Com, Inc.


Control Flow Diagram to Identify Basis PathsBasis Paths

p01 Start Dispatch L M N P Q Rp02 Start Dispatch X Y AA AB Rp03 Start Dispatch AC AF AJ AI AK Rp04 Start Dispatch AD AE AF AJ AI AK Rp05 Start Dispatch O P Q Rp06 Start Dispatch K P Q Rp07 Start Dispatch I J C D H P Q Rp08 Start Dispatch A B C D H P Q Rp09 Start Dispatch L M N P S T Q Rp10 Start Dispatch L M N P S T U Rp11 Start Dispatch L M N P S T U V W Rp12 Start Dispatch X Z AA AB Rp13 Start Dispatch X Z AA AM Rp14 Start Dispatch X Z AA AM AN AP AQ Rp15 Start Dispatch X Z AA AM AN AO AQ Rp16 Start Dispatch AC AF AG Rp17 Start Dispatch AC AF AH Rp18 Start Dispatch AC AF AJ AL AM Rp19 Start Dispatch AC AF AJ AL Rp20 Start Dispatch I J C D E F Rp21 Start Dispatch I J C D E G Rp22 Start Dispatch A B R

Nodes 45Edges 65Complexity 22 E-N+2

AmiBug.Com, Inc.


Purpose of Testing

• Common definition:– To find bugs before our customers do!

• Broader definition:– The role of testing is to provide objective input to

facilitate business decisions!– Keeps stakeholders aware of all issues or

concerns that relate to shipping a product!

AmiBug.Com, Inc.


Bug Defined

• To make our job more fun, whenever we have a concern with software, we call it a “bug”.

AmiBug.Com, Inc.


• It’s all about people! (and the occasional bug too)


AmiBug.Com, Inc.



Context Drivers

AmiBug.Com, Inc.


Context Drivers - BTO

• Business– Value– To whom?– Why?

• Technology– Solutions

• Organization– Corporate Structure– Team Structure– Roles and Responsibilities

AmiBug.Com, Inc.


AmiBug.Com, Inc.


Context Listeners

• Find Sources• Monitor Drivers• Anticipate Change• React

AmiBug.Com, Inc.



Get Ready, Get Set,Cause here it comes

AmiBug.Com, Inc.


Turbulence

(.wav)


AmiBug.Com, Inc.


Unprepared


AmiBug.Com, Inc.



Sharpen Testing Skills• Thinker• Detective• Reporter• Diplomat• Negotiator• Cheer Leader• Pragmatist

AmiBug.Com, Inc.


Philosophy

We have precious little time to run tests!

We must always be prepared!

AmiBug.Com, Inc.


Time

AmiBug.Com, Inc.



Test Triage

AmiBug.Com, Inc.


"No! Try not, Do. Or do not.

There is no try."

Yoda

Pla

n to

sup

port

cha

nge

AmiBug.Com, Inc.


Testing Ideas

• Collect all testing ideas you can find!– List– Sort– Organize– Shuffle

Pla

n to

sup

port

cha

nge

AmiBug.Com, Inc.


Testing Ideas

• How to find them?– Does system do what it is suppose to do?– Does the system do things it is not supposed to?– How can the system break?– How does the system react to it’s environment?– What characteristics must the system have?– Why have similar systems failed?– How have previous projects failed?

Pla

n to

sup

port

cha

nge

AmiBug.Com, Inc.


Testing Ideas

• Collect testing ideas• From testing ideas build a

series of testing objectives– Each can be assigned as

work to testers– Each can include all, part of,

or multiple testing ideas

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• I often use Index Cards– Unique id– One testing idea per card– Colour indicates source– Shuffled and reviewed– Organized and reorganized– Sorted, grouped, prioritized and collected

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Test Idea Sources

• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Investigative approaches– We become truffle

snorting pigs and try to find useful information in all evidence we discover

– We can even get good ideas from out of date sources

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Capabilities– Use cases– Functional

requirements– Documented

requirements– Implicit requirements

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Failure Modes– What can break?– Reaction to invalid input?– How does software

behave in constrained environment?

• Memory• Disk Space• Network Bandwidth• CPU capacity• Shared resources

– Stress, Load, Volume

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Ada

ptab

ility

Acc

essi

bilit

y

Aud

itabi

lity

Ava

ilabi

lity

Con

tinui

ty

Dep

enda

bilit

y

Exp

anda

bilit

y

Fun

ctio

nalit

y

Inte

grity

Inte

rope

rabi

lity

Mai

ntai

nabi

lity

Ope

rabi

lity

Por

tabi

lity

Rel

iabi

lity

Re-

usab

ility

Sca

labi

lity

Sec

urity

Ser

vice

abili

ty

Test

abili

ty

Usa

bilit

y

Application service providerAutomatic content generatorCustomized accessDatabase accessDeliveryDocument accessFile sharingInformationalInteractiveTransaction orientedUser-provided contentWorkflow oriented

High FocusMedium Focus

Low Focus

Quality Factors ImportanceDifferent Application Types

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Usage Scenarios– Identify classes of

users– Identify how users

will use system– Describe scenarios– Use Story board or

similar approaches– Identify variations

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Creative approaches – Action verbs– Mind Maps– Soap Operas– Lateral Thinking

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


State Models

power up

idleinserting

coins

userchoose

makecoffee

serviceneeded

coin insertedreset button

coin return

right amountenteredcoin return

button pushed

no cupsOR no coffeeOR sensor jam

cup removed

Testing Ideas

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Data– Flow– Structure– Create– Update– Change

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Environment– Hardware– Software– Operating systems– Locales– Browsers– Plug-ins– Co-dependent software

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• White Box– Design– Internal structure– Code

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


Testing Ideas

• Bug taxonomies– Collections of possible bugs– Appendix A of Testing

Computer Software, Kaner, Falk, Nguyen

– Boris Biezer Taxonomy Otto Vinter manages

– Shopping cart taxonomy Giri Vijayaraghavan

Cap

ture

tes

ting

idea

s

AmiBug.Com, Inc.


AmiBug.Com, Inc.


Which test?

•Impact estimation–For each test idea guesstimate:

• benefit of implementation• consequence of implementation• benefit for not implementing• consequence of not implementing

–How credible is the information?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Which test?

Test Idea Rejection – What If?–If the cost/benefit does not make business sense then consider implementing:• part of the test, could that lead to part of the

benefit at a more reasonable cost?• more than the stated test, would that generate

more benefit?• a different test than the stated idea, could that

generate more benefit for less cost?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Test Triage Meeting– Review Context

• Business• Technical• Organizational

– New Information• Test results• Bug results• New testing ideas

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

• Allocate Testing Assignments to Testers– Make sure testers know context– Best thing to test– Best person to test it– Best people to explore it– Best lead– Are subject matter experts required

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Test Triage

Life of a test ideaa. Comes into existence

b. Clarified

c. Prioritizeda. Test Now (before further testing)

b. Test before shipping

c. Nice to have

d. May be of interest in some future release

e. Not of interest in current form

f. Will never be of interest

d. Integrate into a testing objective

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Which test is next?

• Questions– Given state of project, state of business, state of

technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?

– How much effort are we willing to spend continuing to test this project?

– Can we ship yet?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Deciding what not to test?

• Time pressure

– Should we skip a test?– If test failed could system still be of

value to some stakeholder?– If test was skipped could important

bugs have been otherwise found?

Tria

ge t

estin

g id

eas

AmiBug.Com, Inc.


Guidelines and Decisions

• To each stakeholder– risk of failure– consequence of failure– value of success– how much certainty do we have – is it a wild guess or an absolute

truth?

Get

Sta

rted

Rig

ht

AmiBug.Com, Inc.


Bottom Line

• My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!

Get

Sta

rted

Rig

ht • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?

AmiBug.Com, Inc.


Thank You

• Questions?

AmiBug.Com, Inc. November 10, 2013© Robert Sabourin, 2012Slide 1 Just-In-Time Testing Workshop Robert Sabourin AmiBug.Com, Inc. Montreal, Canada [email protected].

Documents

decision slide

philosophy slide

welcome slide

free slide

absence slide

time testing slide

pop quiz slide

control flow testing