Experiences in Software Testing (lecture slides)

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Experiences in

Software Testing

Prof. Dr. Dagmar Monett Díaz Computer Science Dept.

Faculty of Cooperative Studies

Berlin School of Economics and Law

[email protected]

Europe Week, 3rd – 7th March 2014

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 2

The waterfall Software Development

Life-Cycle in theory and practice

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 3 ProgrammingGeeks at Google+

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 4

Agenda

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 5

Agenda

Where does the major content come from?

Why is Software Testing of utter importance?

What is Software Testing?

Testing principles and methods

General testing guidelines

What to do with software errors that cause failures?

Test-Driven Development

Current trends in Software Testing

Further reading, sources of inspiration, and more…

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 6

©

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Software Engineering

Ian Sommerville

9th Edition, 792 pp.

Addison-Wesley, March 2011

ISBN-13: 978-0-137-03515-1

What I also use in my lectures at the HWR…

7

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Softwaretechnik.

Mit Fallbeispielen aus

realen

Entwicklungsprojekten

(Software Engineering.

With case studies from real-world

development projects)

Thomas Grechenig, Mario

Bernhart, Roland Breiteneder,

Karin Kappel

In German, 688 pp.

Pearson Studium, 2010

ISBN-13: 978-3-86894-007-7

What I also use in my lectures at the HWR…

8

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Further reading

I. Sommerville (2011): “Chapter 8: Software Testing”.

With material for instructors available at

http://www.softwareengineering-9.com/

L. Crispin, T. House (2012): “Testing extreme programming”

R. Patton (2005): “Software Testing”

ISTQB Exam Certification website available at

http://istqbexamcertification.com/

9

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 10

Why is Software Testing of utter

importance?

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

When I searched the Web on January 5th 2013…

11

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

When I searched the Web on January 5th 2013…

12

I tried again about 30 seconds later and could

continue searching the Web…

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 13

But what about…

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 14 http://news.bbc.co.uk/2/hi/3772077.stm

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 15

What is Software Testing?

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Definition according to IEEE 610.12, 1990

„Software testing is a formal process carried out by

a specialised testing team in which a software unit,

several integrated software units or an entire

software package are examined by running the

programs on a computer. All the associated tests are

performed according to approved test procedures on

approved test cases.“

16

Interesting issues:

- Software testing as formal process

- Testing by a specialised team

- Approved test procedures and test cases

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 17

Testing principles

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 18

Testing principles (ISTQB)

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 19

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 20

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 21

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 22

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

Principle 4: Defect clustering

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 23

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

Principle 4: Defect clustering

Principle 5: Pesticide paradox

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 24

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

Principle 4: Defect clustering

Principle 5: Pesticide paradox

Principle 6: Testing is context dependent

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 25

Testing principles (ISTQB)

Principle 1: Testing shows presence of defects

Principle 2: Exhaustive testing is impossible

Principle 3: Early testing

Principle 4: Defect clustering

Principle 5: Pesticide paradox

Principle 6: Testing is context dependent

Principle 7: Absence-of-errors fallacy

ISTQB: International Software Testing Qualifications Board

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 26

Active learning exercise:

“Mention some concrete examples of

testing principles’ application!”

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 27

Software testing methods

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

- Functional testing

(Black-box testing)

- Structural testing

(White-box testing)

- Combinations of both

(Grey-box testing)

28

Software testing methods

Images © Master isolated images and Serge Bertasius Photography @ http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 29

Examples of Black-box testing

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 30

Partition testing

Also: “Equivalence class partitioning”

Input data and output results often fall into different

classes where all members of a class are related.

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 31

Partition testing

Also: “Equivalence class partitioning”

Input data and output results often fall into different

classes where all members of a class are related.

Each of these classes is an equivalence partition

or domain where the program behaves in an

equivalent way for each class member.

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 32

Partition testing

Also: “Equivalence class partitioning”

Input data and output results often fall into different

classes where all members of a class are related.

Each of these classes is an equivalence partition

or domain where the program behaves in an

equivalent way for each class member.

Test cases should be chosen from each partition.

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 33

Example:

In a system requirements documentation it is specified that

only product prices starting from 0,01 € and up to 500 € must

be entered by the user.

It can be assumed that products with a price equal to 100 €

will be properly validated and processed.

Furthermore, it can be assumed that all prices between 0,01 €

and 500 € are also correct.

Thus, it is sufficient to test only one arbitrary value from each

equivalence class in order to identify possible improper

behaviour of the system.

Partition testing

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 34

It would be sufficient to consider three partitions:

And then to check only one (arbitrary) representative

of each equivalence class in a test case.

Partition testing

Example (cont.):

invalid partition 1 invalid partition 2 valid partition

x 0 0 < x 500 x > 500

0 500

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 35

Boundary-value analysis

Special case of the partitioning testing

Boundary values are common locations for errors

that result in software failures

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 36

Boundary-value analysis

Special case of the partitioning testing

Boundary values are common locations for errors

that result in software failures

Possible boundary values for the previous example:

invalid partition 1 invalid partition 2 valid partition

0 500

0

0,01

500

500,01

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 37

Boundary-value analysis

Or even better…

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 38

Boundary-value analysis

Or even better…

invalid

partition 1

valid

partition

0 500

0

0,01

500

500,01

215

215

215+1

216

216

216+1

231

231

231+1

invalid partition 2

232

232

232+1

… …

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 39

Examples of White-box testing

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 40

Structure-based methods (i)

Function coverage

Statement coverage

Decision (or branch) coverage

Condition (or predicate) coverage

Multiple condition coverage

…and many other code coverage metrics that are

used in structural software testing!

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 41

Structure-based methods (ii)

int rst(int r, int s, int t){ if (r > 0 || s > 0) { while (r != s) { while (r > s) { r = r - s; }

r = s; }

} else { r = t; } return r; }

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 42

Structure-based methods (iii)

int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 }

r = s; //n5 }

} else { r = t; } //n6 return r; //n7 } //n8

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 43

Structure-based methods (iv)

n0

n1

n6 n2

n7

n8

n3

n4 n5

Control flow graph

int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 }

r = s; //n5 }

} else { r = t; } //n6 return r; //n7 } //n8

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 44

Structure-based methods (v)

Test case: {r = 1, s = 2, t = 4}

n0

n1

n6 n2

n7

n8

n3

n4 n5

Control flow graph

int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 }

r = s; //n5 }

} else { r = t; } //n6 return r; //n7 } //n8

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 45

Structure-based methods (vi)

int rst(int r, int s, int t){ //n0 if (r > 0 || s > 0) { //n1 while (r != s) { //n2 while (r > s) { //n3 r = r - s; //n4 }

r = s; //n5 }

} else { r = t; } //n6 return r; //n7 } //n8

Test case: {r = 1, s = 2, t = 4}

n0

n1

n6 n2

n7

n8

n3

n4 n5

Control flow graph

78% Statement coverage

63% Decision coverage

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 46

General testing guidelines

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

General testing guidelines

Choose inputs that force the system to generate all

error messages

Design inputs that cause input buffers to overflow

Repeat the same input or series of inputs numerous

times

Force invalid outputs to be generated

Force computation results to be too large or too

small.

47

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Test ideas and heuristics

“Test Heuristics Cheat Sheet: Data Type Attacks & Web

Tests”

© Quality Tree Software, Inc. (2006)

Available at http://testobsessed.com/wp-

content/uploads/2011/04/testheuristicscheatsheetv1.pdf

“How To Generate Test Ideas (A checklist). Everything you

need to trigger thoughts, discussions and actions in the

projects you are working on”

© Smart Bear and Ministry of Testing (2013)

Available at http://www.ministryoftesting.com/2013/12/checklist

-how-to-generate-test-ideas/

48

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 49

Active learning exercise:

“Which is then the main goal of

software testing?”

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Press the Bug key!

“For when you're having a bad day at work

and you can't find any bugs”

Andy Glover (@cartoontester)

http://cartoontester.blogspot.de/2012/11/press-bug-key.html

50

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 51

What to do with software errors

that cause failures?

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 53

Who should fix the failures?

How does the tester report them?

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 54

Managing software errors

report

Tester

re-test

rate

assign

correct

deliver

terminate

Test lead Developer Change management

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 55

How should tests be conducted?

Which goals and conditions are

important?

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 56

Testing process (i)

What should be considered when conducting

tests:

(1) Criteria for the success or failure of a test run.

(2) Criteria to start and finish a test run.

(3) Criteria to interrupt or continue a test.

(4) Documentation of results and communication

channels.

(5) Testing methods and depth.

(6) Definition of the test environment.

(cont.)

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 57

Testing process (ii)

What should be considered when conducting

tests:

(7) Definition of the test basis (information and

stakeholders needed).

(8) Problems found and responsibilities for their

elimination.

(9) Objects to be tested.

(10) Properties to be tested.

(11) Properties that should not be tested.

(12) Economic objectives of the test.

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 58

Test-Driven Development (TDD)

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Test-driven development

Approach to program development in which you

inter-leave testing and code development.

TDD was introduced as part of agile methods such

as Extreme Programming.

Tests are written before code and ‘passing’ the tests

is the critical driver of development.

59

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Test-driven development

You develop code incrementally, along with a test

for that increment. You don’t move on to the next

increment until the code that you have developed

passes its test.

Identify new

functionality

Write test Run test

Implement

Functionality

and refactor

fail

pass

60

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 61

All code is guilty…

Image © http://www.printfection.com/codesmack/All-Code-Is-Guilty/_s_58243

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 62

Current trends in Software Testing

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

World Quality Report

• >1500 Respondents: Chief financial officers (CFO), chief information officers (CIO), IT directors, and

quality assurance (QA) directors worldwide (>25 countries)

• Detailed telephone interviews undertaken during April-May 2012 and April-May 2013

4th Edition 2012-2013

5th Edition 2013-2014

Available at

http://www.capgemini.com/insights-and-resources/

by-publication/world-quality-report-2012--2013/

Available at

http://www.capgemini.com/thought-leadership/

world-quality-report-2013-14

63

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Change in testing budgets Figure 1, pp. 13 (2012), Gapgemini, Sogeti and HP

64

2012-13

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Greatest increase in spend

65

Figure 3, pp. 16 (2012), Gapgemini, Sogeti and HP

2012-13

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Increase in mobile testing Figure 10, pp. 25 (2013), Gapgemini, Sogeti and HP

66

2013-14

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Challenges to mobile testing Figure 7, pp. 25 (2012), Gapgemini, Sogeti and HP

67

2012-13

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Challenges to mobile testing Figure 11, pp. 27 (2013), Gapgemini, Sogeti and HP

68

2013-14

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Germany – Top trends pp. 64 (2012), Gapgemini, Sogeti and HP

69

„There is a significant increase in move towards

development of a Testing Centre of Excellence

(TCOE).“

„A cautious migration of applications towards the

cloud has become evident.“

„More German organizations are testing mobile

applications and devices than in any other country

surveyed.“ 2012-13

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

UK – Top trends pp. 72 (2012), Gapgemini, Sogeti and HP

70

„19% of UK organizations currently test mobile

applications or devices.“

„More than 83% of organizations intend to use

Testing as a Service (TaaS) offered by a third

party.“

„Mobile application and device testing is a growing

challenge.“

2012-13

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 71

Homework:

“Search for testing tools and use

them in your current course

projects!”

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 72

To take away…

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 73

Fehlermanagement (i)

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 74

Assessment

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Question

75

(Taken from the public Practice Exam © ISTQB)

Which of the following statements BEST describes one

of the seven key principles of software testing?

(A) Automated tests are better than manual tests for

avoiding the Exhaustive Testing.

(B) Exhaustive testing is, with sufficient effort and tool

support, feasible for all software.

(C) It is normally impossible to test all input / output

combinations for a software system.

(D) The purpose of testing is to demonstrate the absence

of defects.

https://docs.google.com/forms/d/1r338sJJzg_DwpWTouEiEeD7KUBI0VD6k8AdbNI7LcfU/viewform

PLEASE ANSWER AT:

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 76

Last active learning exercise

Image © renjith krishnan at http://www.freedigitalphotos.net/

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 77

The content

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 78

References

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Others…

79

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

Further reading and sites…

ISTQB - International Software Testing Qualifications Board

http://www.istqb.org/

Testing Experience magazine

http://www.testingexperience.com/

Agile Testing Days 2014, “Europe’s greatest testing event”,

Potsdam, Germany

http://www.agiletestingdays.com/call_for_proposals.php

IEEE International Conference on Software Testing,

Verification and Validation (ICST’14), Cleveland, Ohio

https://sites.google.com/site/icst2014/

80

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield

With special thanks to…

81

D. Monett – Europe Week 2014, University of Hertfordshire, Hatfield 82

Slides of the talk per request:

[email protected]

Prof. Dr. Dagmar Monett Díaz

monettdiaz

@dmonett

http://monettdiaz.com