Software Testing - a necessary evil

Software Testing - a necessary evil

SQNZ April 2002

Michael Osborne

[email protected]

Does this guy know what he’s talking about?

• Testing is about bugs

• This guy knows about bugs

• In 25+ years of development he’s created thousands of them

• Now, I pay other people to do this

Simply

• Quidquid latine dictum sit, altum sonatur.

• Whatever is said in Latin sounds profound.

• Sedulously eschew obfuscatory hyperverbosity and prolixity.

• Let’s all keep it as simple as possible.• If you don’t understand - interrupt!

From - How To Write Unmaintainable Code http://mindprod.com/unmain.html

What’s going on here?

• Why this presentation?

• enquire vs. inform• tomorrow vs. yesterday• quality vs “quality”• continuous improvement• how to get and give good testing bang for

buck

From - Joel on Software http://www.joelonsoftware.com

What we will cover

• The Principles– Quality– Software Quality– Quality & Testing– The Necessity– The Evil

• Techniques– Types of testing– Testing 2002

• If time permits– we’ll do some testing

Why it’s necessary

• Analysts & developers are imperfect• Capers Jones - bug input is fps to power of 1.2• there is no perfect prevention - yet• you can’t ship (successfully) software that

doesn’t work• it’s a process measure• in a highly sophisticated environment

– it goes from primarily being a QC tool– to a primarily being process measure

Why it’s evil

• You can’t test quality in• You can’t test it all• You can’t prove it works• It’s costly and inefficient• The point is to prove it doesn’t work• The more problems found the better

Testing Explained

• Testing is about finding bugs

Which part don’t you understand?

• If you’re not going to find any bugs why are you doing it?

“A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.”

– Testing Computer Software by Kaner, Falk, Nguyen

It’s an Economics Problem

• Cost of Quality =Cost of Conformance +Cost of Non-Conformance

• Testing adds to Cost of Conformance• It must directly reduce Cost of Non-

Conformance or else it is waste• Testing to do anything other than find bugs is a

WOMBAT– a Waste of Money Brains and Time

How it works - roughly

0

20

40

60

80

100

120

CoCCoNCCoQ

When CoNC is high...

0

200

400

600

800

1000

1200

CoCCoNCCoQ

•It’s appropriate to raise the CoC

High “quality” software

• It is possible to develop very high quality software– An example– NASA space shuttle on-board flight software– Approx. 3 million lines of code– <1 error per 10k LOC after release

– Cost: the order of $1000 / LOC

The “Q” word

• quality vs. “quality”• it’s important we’re clear about this

– it will show the necessity for Software Testing– it will expose some evils of Software Testing

• So, what is quality?• And, what is “quality”?

What is software development?

• Putting bugs into software– ie. hiding bugs in software

• Finding the bugs• Taking them out again• Putting (hiding) new / different bugs in when

you take others out• Finding those bugs• Taking them out again• And so on ...

What is better software development?

• Putting fewer bugs into software– exposing bugs in software

• Finding the important bugs faster• Taking important bugs out again

– assessing the associated risk

• Putting fewer new / different bugs in when you take others out

• And so on …

• Today’s focus is on Software Testing

What is better software testing?

• What is quality software testing?– Quality is …

• conformance to requirements

– Note: test cases are not test requirements

• Finding the important bugs faster?

• which leads to the critical question -

When is testing done?

• Typically - – when the testing schedule ends– when all the test cases have been exercised– when all the bugs have been found and fixed

• If you don’t have a set of testing requirements

• How do you answer the question?

The Zero Defects Approach

• Quality = conformance to requirements• Test exhaustively• Find all defects• Fix all defects• Result - quality software

• a quick digression into Zero Defects– a personal view

Direction vs. Destination

• Current state - some level of defects– Desired state

• more defects• same level of defects• fewer defects

• An asymptote

• Note “six sigma” is not zero• “Zero defects” is a pointer and an accelerator

– use it that way

Process ImprovementD

efe

ct R

ate

s

Uh oh - there’s more than one requirement!!

• Quality = conformance to requirements• There are generally three sets of requirements

– Functionality– Schedule– Quality

• The bad news– they compete

The Good-Enough Software Triangle

• Conformance to all requirements

ScheduleDelivery Date

QualityAbsence of Defects

FunctionalityFeatures

Software Testing - A Wicked Problem

• A “mess” is a complex problem with a “simple” solution

• A “wicked problem” is a “simple” problem with a complex solution– usually because of stakeholder differences

• Each project (testing problem) potentially has a different solution

So?

• How do we do it?

Software Testing 2002

• Context-driven testing– all at www.satisfice.com

• Exploratory testing– heuristics– short bursts– focus on results– purposeful wandering– you must think

Testing Types

• white box - glass box, structural

• path testing - coverage• black box• top-down, bottom-up• unit• function• system• integration• regression

• performance• alpha• beta• compatability• quicky• port • usability• exploratory• hallway• others ...

Bug removal strategy

Impact

Probability

Find & Fix Immediately

Ignore

Later TestPhase

Middle Phase

Ignore FrequentNiggles- ?

FrequentAnnoyances- Remove

??

Urgent

Important

Usability Testing

• Graphical User Interfaces coupled with complex processes create usability issues– what’s clear and obvious to designer/programmer

is meaningless to users

• Doesn’t require a lot of people – 5 to 7 is adequate

• Observe, and listen– understand how they see things– why they do what they do

• Is your interface clear and consistent?

Why usability is critical

• It doesn’t matter if it processes everything right– if they don’t like it (can’t/won’t use it, it failed)

• Prototype, prototype, prototype• Consistent vs. clever

– at all times

• Users will be forgiving with a highly usable system– ask Microsoft

Hallway Testing

• A simple form of usability testing• Open up development to interact more with

users• Approach a person (in your organisation) at

random– invite for 5-10 minute testing session– see what they do with your software from no

background– is it understandable, usable

The evil necessity

• It’s all about finding bugs• A good test is a test that finds a bug• Be clear about what represents quality

– situation by situation

• Establish the context– and be guided by it

• Good software testing is a challenging intellectual process.

• Go for it!