Software Testing #IRL [email protected]. Agenda Why Software Testing Software Testing Principles – Exhaustive testing is impossible – Early Testing.

Software Testing #[email protected]

Agenda

• Why Software Testing• Software Testing Principles

– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy

• Summary• Recommended reading

Other professions in Computer Science

Project Lead Developer Architect

Technical Writer Customer IT Ops

How Other professions in Computer Science sees test

Project Lead Developer Architect

Technical Writer Customer IT Ops

Agenda

• Why Software Testing• Software Testing Principles

– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy

• Summary• Recommended reading

Exhaustive Testing is Impossible

What you Think you are Testing

Developer Forgot «this.»

Off by One

Type Casting

Previous Values

Date and Time

Obey Thermodynamics – it’s also a LAW!

• 1 year of power of sun is enough energy to flip bits to count to all combinations in a 187 bit number.

• 187 bit < 24 byte.

Google: “schneier dog crypteto”http://www.schneier.com/blog/archives/2009/09/the_doghouse_cr.html

Early Testing: V-Model

Architecture

Requirements

Specification

Design

Code

Acceptance Test

System Test

Integration Test

Unit Test

Early Testing: W-Model

Architecture

Requirements

Specification

Design

Code

Acceptance Test

System Test

Integration Test

Unit Test

Design Acceptance Test

Design System Test

Design Integration Test

Design Unit Test

Cost of Defect

Introduced\Detected Reqs Arch Construction System test Production

Requirements 1X 3X 5-10X 10X 10-100X

Architecture - 1X 10X 15X 25-100X

Construction - - 1X 10X 10-25X

http://www.abeacha.com/NIST_press_release_bugs_cost.htm

Quality Attributes: ISO/IEC 9126-1:2001

• Functionality - satisfies needs.– Suitability– Accuracy– Interoperability– Security– Functionality Compliance

• Reliability - maintains operation– Maturity– Fault Tolerance– Recoverability– Reliability Compliance

• Usability - effort needed for use– Understandability– Learnability– Operability– Attractiveness– Usability Compliance

• Efficiency - performance given resources– Time Behaviour– Resource Utilisation– Efficiency Compliance

• Maintainability - make modifications– Analyzability– Changeability– Stability– Testability– Maintainability Compliance

• Portability - change environment– Adaptability– Installability– Co-Existence– Replaceability– Portability Compliance

Types of Testing

Acceptance Test

System Test

Integration Test

Unit Test

Functional Efficiency Interopability

X

X

X

X

X

X

X

Several Layers

Acceptance Test

System Test

Integration Test

Unit Test

Functional Efficiency Security

X

X

X

X

X

X

X

System Int. Test X

Testing shows the presence of bugs, but can not show that there are no defects

• 1986: Programming Pearls, binarySearch() bevist og testet i et kapittel

• 1997: Implementert i Sun java.util.Arrays.binarySearch()

• 2006: Feil funnet og fikset i Java-biblioteket

http://googleresearch.blogspot.it/2006/06/extra-extra-read-all-about-it-nearly.html

Defect Clustering

http://www.slideshare.net/andreas.zeller/myths-in-software-engineering

Bugs Found with Bug Seeding

http://www.slideshare.net/andreas.zeller/seeding-bugs-to-find-bugs-presentation

Pesticide Paradox

Age oftest

Bugs found per dayby a test

Regression Testing

Acceptance Test

System Test

Integration Test

Unit Test

Functional Efficiency Security

X

X

X

X

X

X

X

System Int. Test X

X

Leve

ls

Types

Testing is Context Dependent

Coverage

Cost

0% 50% 100%

Cost oftesting

80-85%?Cost ofdefects

Absence of Errors does not mean the System is OK

Acceptance Test

System Test

Integration Test

Unit Test95%

Performance/Stability

Agenda

• Why Software Testing• Software Testing Principles

– Exhaustive testing is impossible– Early Testing– Testing shows the Presence of Bugs– Defect Clustering– The Pesticide Paradox– Testing is Context Dependent – Absence of errors fallacy

• Summary• Recommended reading

TL;DL

• Testing #FTW #YOLO

• You can’t test everything!• Testing is Context Dependent!

• Get involved early!

• Testing doesn’t prove software is without bugs• Absence of errors does not prove the system is ok

• Defects cluster• Old tests find fewer bugs

Books you should read to learn more about…

People and testing Creative Testing Coding

Software Engineering Testing in Trondheim Fun Web

Software Testing #[email protected]

Copyrights

• Ant Hill: RukaKuusamo.com• Money bag: http://www.flickr.com/photos/68751915@N05/ CC AT SA• Book: http://www.flickr.com/photos/pat00139/ CC AT• Cables: http://www.flickr.com/photos/alq666/ CC AT SA• Bokhylle: Sehoon Lee Phote

http://design-milk.com/squaring-movable-bookshelf-by-sehoon-lee/• Hender: CC AT NM NC http://www.flickr.com/photos/johnkay/• Angel: CC AT NC http://www.flickr.com/photos/jens-h-friedrich• Guarantee: CC AT http://www.flickr.com/photos/vectorportal• Road: CC AT SA http://www.flickr.com/photos/huangjiahui/