0 All Slides

22/08/2011

1

Software testingNguyen Thanh Binh

IT Faculty, Danang University of TechnologyEmail: [email protected]

Website: www.dut.edu.vn/itf/ntb

2

Prerequisites No testing experience needed Some familiarity with the development

phases of a project would be helpful. Some mathematics would help

3

References1. Paul Jorgensen, Software Testing-A Craftsman's Approach, CRC

Press, 1995.2. Spyos Xanthakis, Pascal Rgnier, Constantin Karapoulios, Le test des

logiciels, Hermes Science, 2000.3. Hung Q. Nguyen and al., Testing application on the Web, John Wiley &

Sons, 2004.4. Ilene Burnstein, Practical Software Testing, Springer, 2003.5. Glenford J. Myers, The art of software testing, Wiley, 2004.6. Cem Kaner, Jack Falk, Hung Q. Nguyen, Testing Computer Software,

2nd Edition, John Wiley & Sons, 1999.7. Boris Beizer, Software Testing Techniques, International Thomson

Computer Press, Second Edition, 1990.8. Neil Bitzenhofer, Software Testing and Verification, Course, MSSE, 2008.9. Paul Ammann and Jeff Offutt, Introduction to Software Testing,

Cambridge University Press, Cambridge, UK, ISBN 0-52188-038-1, 2008.

10. Mauro Pezz, Michal Young, Software Testing and Analysis: Process, Principles, and Techniques, John Wiley & Sons.

22/08/2011

2

4

Course description Cover both theoretical and practical aspects of

testing software The student will participate in the entire range of test

activities: Analyze a requirements document for test conditions Write a test plan Design, create and execute test cases using various

testing approaches Record defects Write a test report

5

Contents Session 1: Introductory lecture

Introductions and expectations Course overview Contents

6

Contents Session 2: Introduction to Software Testing

Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing

22/08/2011

3

7

Contents Session 3: Requirements analysis

Software Development Life Cycle (SDLC) Software Development stage

Requirements Testing and requirements

Learn to think like a tester Some examples

8

Contents Session 4

Exercise 1: Examining requirements

9

Contents Session 5: Structural Testing

White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code Testing State Behavior

22/08/2011

4

10

Contents Session 6: Static Testing

Reviews and the test process Types of review Static analysis

11

Contents Session 7: Functional Testing

Introduction to functional testing Functional testing techniques

Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

12

Contents Session 8: Test Documentation

Test Plan The need for test plans The structure of test plans A Test Plan Template A Test Plan example Testing on a large project

Test Cases Test Case Design Test Case Examples

22/08/2011

5

13

Contents Session 9

Exercise 2: Writing a test plan and test cases

14

Contents Session 10: Levels of Testing

Levels of Testing Integration Testing System Testing

Additional System Test Categories

15

Contents Session 11: Defect Reports/Test Reports

Handling Defects Bug Tracking System Test Reports Examples

22/08/2011

6

16

Contents Session 12: Test Automation and Tools

Test Automation Test tools

17

Why test? List of 107 software failures that should have

been caught by testinghttp://www.cs.tau.ac.il/~nachumd/verify/horror.html

One vital consideration from Myers bookThe Art of Software Testing

Mars Climate Orbiter Mars Polar Lander

18

Software Errors1. The Mars Climate Orbiter crashed in September 1999 because of a

"silly mistake": wrong units in a program. 2. The 1988 shooting down of the Airbus 320 by the USS Vincennes

was attributed to the cryptic and misleading output displayed by the tracking software.

3. Death resulted from inadequate testing of the London Ambulance Service software.

4. Several 1985-7 deaths of cancer patients were due to overdoses of radiation resulting from a race condition between concurrent tasks in the Therac-25 software.

5. Errors in medical software have caused deaths. Details in B.W. Boehm, "Software and its Impact: A Quantitative Assessment," Datamation, 19(5), 48-59(1973).

6. An Airbus A320 crashes at an air show.7. A China Airlines Airbus Industrie A300 crashes on April 26, 1994

killing 264. Recommendations include software modifications.

22/08/2011

7

19

Software Errors The Explosion of the Ariane 5

On June 4, 1996 an unmanned Ariane 5 rocket launched by the European Space Agency exploded just forty seconds after its lift-off from Kourou, French Guiana. The rocket was on its first voyage, after a decade of development costing $7 billion. The destroyed rocket and its cargo were valued at $500 million. A board of inquiry investigated the causes of the explosion and in two weeks issued a report. It turned out that the cause of the failure was a software error in the inertial reference system. Specifically a 64 bit floating point number relating to the horizontal velocity of the rocket with respect to the platform was converted to a 16 bit signed integer. The number was larger than 32,767, the largest integer storeable in a 16 bit signed integer, and thus the conversion failed.

20

Software Errors The Patriot Missile Failure

On February 25, 1991, during the Gulf War, an American Patriot Missile battery in Dharan, Saudi Arabia, failed to track and intercept an incoming Iraqi Scud missile. The Scud struck an American Army barracks, killing 28 soldiers and injuring around 100 other people. A report of the General Accounting office, GAO/IMTEC-92-26, entitled Patriot Missile Defense: Software Problem Led to System Failure at Dhahran, Saudi Arabia reported on the cause of the failure. It turns out that the cause was an inaccurate calculation of the time since boot due to computer arithmetic errors. Specifically, the time in tenths of second as measured by the system's internal clock was multiplied by 1/10 to produce the time in seconds. This calculation was performed using a 24 bit fixed point register. In particular, the value 1/10, which has a non-terminating binary expansion, was chopped at 24 bits after the radix point. The small chopping error, when multiplied by the large number giving the time in tenths of a second, led to a significant error. Indeed, the Patriot battery had been up around 100 hours, and an easy calculation shows that the resulting time error due to the magnified chopping error was about 0.34 seconds. A Scud travels at about 1,676 meters per second, and so travels more than half a kilometer in this time. This was far enough that the incoming Scud was outside the "range gate" that the Patriot tracked. Ironically, the fact that the bad time calculation had been improved in some parts of the code, but not all, contributed to the problem, since it meant that the inaccuracies did not cancel.

21

Software Errors Y2K

Spent some billions dollars

22/08/2011

8

22

Chapter 1 of the textbook A Perspective on Testing Basic Definitions

Error a mistake in design, coding, requirements, even testing

Fault the representation of the error Failure what happens when the fault executes

23

A Perspective on Testing More Definitions

Testing the process of finding errors and of validating the program/system

Test Case a test case has Inputs Steps Outputs Expected results

Process Test plan, Write test cases Run the test cases Evaluate results

24

A Perspective on Testing Test Cases

Test Cases will be discussed in detail in Session 7 and throughout the course.

Testing entails establishing the environment, providing the inputs (running the test case), observing outputs, and comparing to expected outputs.

Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!

22/08/2011

9

25

A Perspective on Testing Identifying Test Cases

Functional Testing The program is a function that maps input values to

output values The only information used is the software specification In our Venn diagram, the Functional Test Cases are a

subset of S Further elaborated on in Part II Math background: Chapter 3 We will discuss in Session 6

26


Structural Testing Uses the information inside the black box the

actual implementation In our Venn diagram, the Structural Test Cases are a

subset of P. Further elaborated on in Part III of the text Math background: Chapter 4 We will discuss this in Session 4 Main method: Test coverage metrics

27


Comparing the two (Functional vs Structural) We will discuss this in Sessions 4 and 6If all specified behaviors have not been implemented,

structural test cases will never be able to recognize this.

Conversely, if the program implements behaviors that have not been specified, this will never be revealed by functional test cases.

22/08/2011

10

28

A Perspective on Testing Levels of Testing

Again, this will be covered in detail in Session 2.

RequirementsSpecification

PreliminaryDesign

DetailedDesign

Coding

Unit Testing

IntegrationTesting

SystemTesting

29

Testing a ProgramA program that we want to test reads in 3 integer values these 3

values are interpreted as the lengths of the sides of a triangle.The program prints a message that states whether the triangle is

Equilateral (all 3 sides equal)Isosceles (exactly 2 of the 3 sides are equal)Scalene (all 3 sides are of a different length)

On a sheet of paper, write specific sets of test data that you feel would adequately test this program.

Introduction to software testing

22/08/2011

11

31

Contents Definitions, Principles, Axioms Stages of testing Perspectives on Software Testing

32

Definitions, Principles, Axioms

33

Definitions Testing Verification Validation Error/Defect Black box testing White box testing

22/08/2011

12

34

Definitions of Testing (1)

The process of executing a program (or part of a program) with the intention of finding errors (Myers, Humphrey)

The purpose of testing is to find errors. Testing is the process of trying to discover every conceivable fault or weakness in a work product (Myers, Kit)

The process of searching for errors (Kaner)

35

Definitions of Testing (2) Testing the process of finding errors and of

validating the program/system (Jorgensen) The purpose of software testing is to find

errors and to get them fixed (Bitzenhofer) The purpose of software testing is to reduce

risk (E&M)

36

What Testing is Not Testing is not just another phase of the

project Testing is not just finding defects Testing is not debugging Testing is not a final exam

22/08/2011

13

37

Verification and Validation: Myers Verification: An attempt to find errors by

executing a program in a test or simulated environment

Validation: An attempt to find errors by executing a program in a real environment

38

Verification and Validation: IEEE

Verification: The process of evaluating a system or component to determine whether the productssatisfy the conditions imposed

Validation: The process of evaluating a system or componentto determine whether it satisfies specified requirements.

39

Verification and Validation: Kaner Verification: Checking a program against the

most closely related design documents or specifications (Design Verification Testing)

Validation: Checking the program against the published user or system requirements (System Validation Testing)

22/08/2011

14

40

Software Fault Terminology Error a mistake in design, coding,

requirements, even testing

Fault the representation of the error

Failure what happens when the fault executes

41

Software Fault Terminology: Examples Error

A wrong loop index Programmer-to-device pointer error

Fault the representation of the error A loop executes one too many times A programmed parameter gets written to the wrong address

Failure what happens when the fault executes A table overflows; memory gets overwritten The physician only thinks he/she changed that heart parameter

42

Black and White Box Black box testing

Designed without knowledge of the programs internal structure and design

Based on functional requirements Also called Functional Testing

White box testing Examines the internal design of the program Requires detailed knowledge of its structure Also called Structural Testing

22/08/2011

15

43

Six Essentials of Software Testing (1)1.The quality of the test process determines the

success of the test effort.2.Prevent defect migration by using early life-

cycle testing techniques.3.The time for software testing tools is now.

Adapted from Software Testing in the Real World, Edward Kit; Addison-Wesley, 1995

44

Six Essentials of Software Testing (2)4.A real person must take responsibility for

improving the testing process.5.Testing is a professional discipline requiring

trained, skilled people.6.Cultivate a positive team attitude of creative

destruction.Adapted from Software Testing in the Real World, Edward Kit; Addison-

Wesley, 1995

45

Some Principles and Axioms of Testing (1) Program testing can be used to show the

presence of bugs, but never their absence. (Dijkstra, 1969)

One of the most difficult problems in testing is knowing when to stop.

It is impossible to test your own program.

22/08/2011

16

46

Some Principles and Axioms of Testing (2) As the number of detected defects in a piece

of software increases, the probability of the existence of more undetected defects also increases.

Testing is an extremely creative and intellectually challenging task (Myers again)

Exhaustive testing is impossible.

47

Stages of testing

48

The Test Process Requirements phase Planning phase Design and development phase Execution phase Reporting phase

22/08/2011

17

49

Software Test Execution Stages Unit Test Design Verification Test (DVT)

Interface, Integration, Exit/Formal System Validation Test (SVT)

Main, Regression, Acceptance Customer Acceptance Test

50

Unit Test Performed by the developers White-box testing Should be done with each release to testing Results should be communicated to the Test

Group

51

Typical Entry Criteria (Unit Test) Module compiles successfully (no errors,

approved warnings) Unit Test Plan complete Unit Test Cases complete Tools are available

22/08/2011

18

52

Typical Unit Test Activities Source level debugging if it wont compile Design and review the Unit Test cases Track test defects (maybe) Measure Unit Test coverage Make sure it builds

53

Typical Exit Criteria (Unit Test) All tests attempted 99% of the tests executed successfully 85% statement coverage Test cases have been stored under

Configuration Management

54

Software Test Stages Unit Test Design Verification Test

Interface, Integration, Exit/Formal Test Phase System Validation Test


22/08/2011

19

55

Design Verification Test (DVT) Performed by the test organization Black box testing Verification of engineering requirements Verification of the software design

56

Three Components of DVT Interface Testing Integration Testing DVT Exit/Formal Test Phase

57

Typical Entry Criteria (DVT) Unit Test complete (for all code being tested) Engineering requirements and program

design are complete The code is under Configuration

Management A current build is available that contains all

necessary integrated code Interface and Integration test cases have

been written

22/08/2011

20

58

Typical DVT activities Design, write and review the Test Plan Write the DVT Test Cases Execute Interface/Integration test cases Track defects to the code and to the tests

59

Interface Testing How does a user interact with the program? What input variables are required? What outputs are produced? Extremely important for e-business

applications

60

Integration Testing Integration between software components Integration with the front-end GUIs Integration between a server and its clients Integration between software and hardware Integration between Web services

22/08/2011

21

61

DVT Exit or Formal Test Phase This is a requirement for System Test Probably involves running a suite of

Regression Tests Defect requirements of some sort must be

present The whole point: Are we ready for System

Test?

62

Typical Exit Criteria (DVT) All test cases executed 98% of the test cases executed successfully High severity defects have been fixed and

verified

63


Interface, Integration, Exit/Formal System Validation Test


22/08/2011

22

64

System Validation Test (SVT) Performed by the test organization Validation of customer requirements Three components

Main Test Regression Test SVT Acceptance Test

65

Typical Entry Criteria (SVT) DVT Exit completes successfully Test Plan has been updated and frozen All test cases have been finalized No high severity problems are open Code is frozen The install works

66

SVT Main Test Verify the functional requirements of the

product from the Customers point of view Includes usability, reliability, performance,

and installation Also has its own Entry and Exit criteria Use Cases form a big part of this testing Marketing issues are addressed here too

22/08/2011

23

67

SVT Regression Test Verify that all defects identified during SVT

have been addressed; i.e. all tests run OK

68

System Acceptance Test Verify final manufacturing build Verify all installation scenarios Verify delivery of all media

69

Typical Exit Criteria (SVT) All test cases attempted All test cases executed successfully, OR

every defect that has been submitted has been addressed

No high severity defects are open Test Cases are under Configuration

Management Final Test Report has been written

22/08/2011

24

70


Interface, Integration, Exit System Validation Test

Main, Regression, Exit Customer Acceptance Test

71

Customer Acceptance Testing Usually performed at the customer site Must be well-written and simple, but

meaningful May be dictated by the customer

72

Perspectives on Software Testing

22/08/2011

25

73

The Objectives and Limits of Testing (1) Limits

You cannot test a program completely Even if you do find the last bug, youll never know

it It takes more time than you have to test less than

youd like You will run out of time before you run out of test

cases

74

The Objectives and Limits of Testing (2) Limits

You cannot test every path You cannot test every valid input You cannot test every invalid input You cannot prove a program correct (because it

isnt!)

75

The Objectives and Limits of Testing (3) Objectives

The purpose of testing is to find problems The purpose of finding problems is to get them

corrected The purpose of testing is to reduce identifiable

risks

22/08/2011

26

76

Tests and Test Cases (1) Test a collection of Test Cases Test Case a test case has inputs, steps,

and outputs, as well as expected results Process

write the Test Plan write test cases run the test cases evaluate results

77

Tests and Test Cases (2) We will discuss Test Cases in detail in Session 7

and throughout the course. Testing entails establishing the environment,

providing the inputs (running the test case), observing outputs, and comparing to expected outputs. (a paraphrase)

Test Cases are developed, reviewed, used, managed, and saved and hopefully reused!

78

What is Quality? Quality from the Producers viewpoint: Does the

product meet the requirements? Quality from the Customers viewpoint: Fitness for

use (reliability, usability, portability, etc.) Quality (one strongly held view): The measure of a

products quality is the satisfaction of the customers, not whether it meets a specification

22/08/2011

27

79

Quality Assurance vs Quality Control Quality Assurance is the activity that

establishes and evaluates the processes that produce products QA improves the product by improving the

process Quality Control is the activity that evaluates

the product produced QC improves the product by finding defects

80

A Generic Testing Approach (SPRAE) Specification

A written statement of expected software behavior Premeditation

Written test plans, test scripts, test data, test schedules Repeatability

Same results every time a result of a mature process Accountability

Test logs; analysis and interpretation of results Economy

The cost effectiveness of testing

81

The Testers Toolkit Static testing Structural testing methods Functional testing methods Non-functional techniques

22/08/2011

28

Requirements analysis

83

Contents Software Development Life Cycle (SDLC) Software Development stage

Requirements Testing and requirements

Learn to think like a tester Some examples Writing test requirements

84

Software Development Life Cycle (SDLC)

22/08/2011

29

85


SDLC is a disciplined and systematic approach that divides the software development process into various phases, such as requirement, design, and coding

The phase-wise software development process helps you track schedule, cost, and quality of the software projects

86


There are six phases of SDLC Feasibility analysis phase includes the analysis of project

requirement. Requirement analysis and specification phase includes gathering,

analyzing, validating, and specifying requirements. Design phase includes translation of requirements specified in the

SRS into a logical structure that can be implemented in a programming language.

Coding phase includes implementation of design specified in the design document into executable programming language code.

Testing phase includes detection of errors in the software. Maintenance phase includes implementation of changes and new

requirements in the software at the customer location.

87

SDLC models Are a tailored form of SDLC phases. Provide a basis for categorizing and

controlling the various activities required to develop and maintain the software system

Typical types of SDLC models are Linear model Iterative model

22/08/2011

30

88

Linear Models Linear models are suitable for projects where

all the requirements are identified and well understood before the design of the software begins.

The two types of linear models are: Waterfall model Prototyping model

89

Waterfall model Describes the software development process in a

linear sequential flow. Defines the software development process in seven

phases: Conception Initiation Analysis Design Construction Integration and testing Implementation and maintenance

90

Waterfall model

22/08/2011

31

91

Prototyping model Is a sample implementation of the system

that shows limited and main functional capabilities of the proposed system.

Helps the customer determine how the features will function in the final software.

92

Prototyping model

93

Iterative model Iterative model is used when the

requirements for the software are likely to evolve throughout the development process.

The various types of iterative models are Spiral model Component-based development model Unified process

22/08/2011

32

94

Spiral model Spiral model

Includes the iterative nature of the prototyping model and the linear nature of the waterfall model.

Is ideal for developing software that are released in various versions.

The six phases of spiral model are: Customer communication Planning Risk analysis Engineering Construction and release Customer evaluation

95

Spiral model

96

Component-Based Development Model In a component-based development model:

Components are reused and combined with other components in the same or other computers in a distributed network to form an application.

All related modules that form a component are tested to ensure that they work together

22/08/2011

33

97

Advantages: Linear Models Requirements are in the hands of the testers

early The project is better designed, with better

focus Late changes to requirements or design are

limited Test tasks of scheduling, planning, etc., are

made easier

98

Advantages: Iterative Models A shippable product is available early Progress (Where are we?) is much easier to

track Many opportunities to reappraise

requirements or rethink the design Easy to drop features

99

Disadvantages: Linear Models Big decisions must be made early It is unknown what will be difficult to

implement Changing a specification affects everyone Tremendous pressure on testers to prove

that a product is NOT ready for release

22/08/2011

34

100

Disadvantages: Iterative Models Product design may become a bottleneck Products often ship with half the features missing Big temptation for developers to keep writing new

code rather than fix whats broken Lack of early planning may cause problems when

later features are added How does Marketing market the product? Constant re-running of the same tests Potentially longer product cycles

101

Effects on Testing: Linear Models Requirements must be well-reviewed early Test plan can be written early When System Testing does start, you are on

the critical path

102

Effects on Testing: Iterative Models Testing starts as soon as the first level of

functionality is reached Test plan is written as you go Requests for design changes can be more

easily made

22/08/2011

35

103

Software Development stageRequirements

104

Software Development Stages Project/Product Planning Requirements and Design Coding and Documentation Testing and fixing bugs Maintenance and Enhancement

105

Why Projects are CancelledProject Impaired Factors % of Responses

1. Incomplete Requirements 13.1%2. Lack of User Involvement 12.4%3. Lack of Resources 10.6%4. Unrealistic Expectations 9.9%5. Lack of Executive Support 9.3%6. Changing Requirements & Specifications 8.7%7. Lack of Planning 8.1%8. Didn't Need It Any Longer 7.5%9. Lack of IT Management 6.2%10. Technology Illiteracy 4.3%Other 9.9%

22/08/2011

36

106

Why Projects are ChallengedProject Challenged Factors % of Responses

1. Lack of User Input 12.8%2. Incomplete Requirements & Specifications 12.3%3. Changing Requirements & Specifications 11.8%4. Lack of Executive Support 7.5%5. Technology Incompetence 7.0%6. Lack of Resources 6.4%7. Unrealistic Expectations 5.9%8. Unclear Objectives 5.3%9. Unrealistic Time Frames 4.3%10. New Technology 3.7%Other 23.0%

107

Why Projects SucceedProject Success Factors % of Responses

1. User Involvement 15.9%2. Executive Management Support 13.9%3. Clear Statement of Requirements 13.0%4. Proper Planning 9.6%5. Realistic Expectations 8.2%6. Smaller Project Milestones 7.7%7. Competent Staff 7.2%8. Ownership 5.3%9. Clear Vision & Objectives 2.9%10. Hard-Working, Focused Staff 2.4%Other 13.9%

108

Software Requirements According to software engineering theory,

software requirements should contain a finite list of behaviors and features, and each requirement should be written to be verifiable.

Given a finite list of requirements and a set of completion criteria, requirements-based testing becomes a feasible process.

22/08/2011

37

109

Typical Requirements (FURPS) Functionality does it do what I need? Usability is it easy to do what I need? Reliability can it do what I need when I

need it? Performance does it do what I need in a

reasonable amount of time? Supportability what if is DOESNT do what I

need?

110

Functionality Requirements Correctness

Extent to which a program satisfies its specifications and meets customer expectations

Features/Capabilities Generality Security

Extent to which access to software or data by unauthorized persons can be controlled

111

Usability Requirements Ease of use

Measures the effort required to learn, operate, prepare input for, and interpret the output of, a program

Intuitive Human factors Consistency Documentation

22/08/2011

38

112

Reliability Requirements Frequency and severity of failure - MTBF Recoverability Predictability Accuracy

113

Performance Requirements Speed Efficiency Resource consumption Throughput Response time

114

Supportability Requirements Testability Adaptability Maintainability

Effort required to locate and fix errors Serviceability

Effort required to modify/update a program Installability Portability

22/08/2011

39

115

Contents

Testing and requirementsLearn to think like a tester

116

Testing Requirements Usually done by review Are they complete? Are they reasonable? Can they be tested?

117

Requirements Testing Testing of the requirements Determine all testable requirements Determine all non-testable requirements Write and execute the tests

22/08/2011

40

118

Reviewing Software Requirements Learn to recognize what has not been

specified (limits, consequences) Learn to recognize what is vague or

ambiguous (sometimes, faster) As a tester:

Make sure testers are included in the reviews! Push back to get requirements clarified Always be thinking And just how am I going to

test this?

119

Reviewing the Design Helps determine how to test Shows what to look for Do you need to read the code?

120

Some examples

Some requirements statements Some examples of Test Cases

22/08/2011

41

121

Requirements Example 1 After a high temperature is detected, an

alarm must be raised quickly

122

Testable Requirements: Ex. 1 If the threshold temperature is reached, an

alarm is triggered. As long as the temperature remains below

the threshold temperature, no alarm is triggered.

If the temperature dips below the threshold (having gone above it), the alarm does not cease.

123

Requirements Example 2 Novice users should be able to learn the

interface with little training

When thinking about a test for this, ask yourself: What would be the criterion for saying the test passed? failed?

22/08/2011

42

124

Requirements Example 3There was a hidden requirement in this

feature change:The menu option 6. Utilities | 1. Security will

now display a new submenu rather than a Security dialog box. The submenu will have 3 items - 1. CSM, 2. CIS, and 3. Options.

125

Requirements Example 4How many testable requirements are here?The Access Profile choices for 2. CIS and 3.

Options will be grayed out unless those options have been purchased and installed.

126

Testable Requirements for Ex. 4

The Access Profile choice for CIS is available if and only if the CIS feature has been installed.

The Access Profile choice for Options is available if and only if the Options feature has been installed.

22/08/2011

43

127

Some Test Cases for Example 41. If the CIS option is installed, Access Profile

choices will be available for CIS2. If the Options option is installed, Access

Profile choices will be available for Options3. If the CIS option is not installed, Access

Profile choices for CIS will not be available 4. If the Options option is not installed,

Access Profile choices for Options will not be available

128

Requirements Example 5

An upgraded security system was being installed with enhanced password protection. Here is one requirement:

Existing passwords which do not conform to the new restrictions will be reset to the User ID, and a list of those users whose passwords were reset will be generated.

129

Requirements Example 6Untestable requirementNumber of invalid attempts before a user is

suspended: This count is reset when a successful logon is completed for the user. The default is that the User will never be suspended. The valid range is from 0 (never) to 10 attempts.

22/08/2011

44

130

Testable Requirements for Example 6Requirement: The number of invalid attempts before suspension

may be set to 2Test Case 1: After 2 bad attempts to log on, the Users USERID is

suspended.Test Case 2: A bad attempt, followed by a good attempt, can then

be followed by 2 bad attempts again before suspension.

The same holds for 1, 3-10

Requirement: The number of invalid attempts before suspension cannot be set to 11

Test Case 3: Attempt to set invalid attempts to 11, or verify it cant be done.

131

Requirements Example 7Vagueness in different disguises:The Advanced Color Module will improve the

print quality of the module.The Advanced Color Module firmware will

increase the number of color shades per panel from 4 - 9 shades to 25 - 30 shades.

The Advanced Color Module will reduce the tiling effects associated with previous modules.

132

Requirements Example 8A lot of test cases from just one sentence:

All new and modified screens will provide online help.

22/08/2011

45

133

Requirements Example 9 The user interface must be easy to use. In case of failure, the system must be easy to

recover and must suffer minimum loss of important data.

134

Lessons Learned Watch for words like always, never, every, etc.

They are difficult to verify. Words like must, should, shall, will, has to,

etc. signal potential requirements. Watch for hidden or unstated requirements. Watch for vague requirements: high, quickly,

better, faster, should be able to. For negative testing, ask What happens if the test

case fails? Ask yourself the question: How will I know if the

requirement is not met?

Structural testing

22/08/2011

46

136

Contents White box testing / Structural testing Graph Theory Control flow criteria Data flow criteria Graph Coverage for Source Code Testing State Behavior

137

White box testing / Structural testing

138

What is white-box software testing? What is white-box software testing?

Basic idea is to test a program based on the structure of a program

Pseudonyms: Glass Box, Clear Box, Crystal Box, Structural Testing

To distinguish from functional (requirements-based, black-box testing)

What do you need for white-box testing? A white-box testing model and test criteria A white-box test design and generation method Program source code

22/08/2011

47

139

White-Box Testing Objectives The major objective of white-box testing is to focus

on internal program structure, and discover all internal program errors Errors made by developers

The major testing focuses Program structures

Program statements and branches Various kinds of program paths

Program internal logic and data structures Program internal behaviors and states

140

Traditional White-Based Software Testing Methods Test Model: control flow graph

Test case design Various white-box testing methods generate test cases based on

a given control flow graph for a program

The goal is to Guarantee that all independent paths within a module have been

exercised at least once Exercise all logical decisions one their true and false sides Execute all loops at their boundaries and within their operational

bounds Exercise internal data structures to assure their validity Exercise all data define and use paths

141

No guarantees Executing all control flow elements does not

guarantee finding all faults Execution of a faulty statement may not always result in a

failure The state may not be corrupted when the statement is

executed with some data values Corrupt state may not propagate through execution to

eventually lead to failure What is the value of structural coverage?

Increases confidence in thoroughness of testing Removes some obvious inadequacies

22/08/2011

48

142

Graph theory

143

Definition of a Graph A set N of nodes, N is not empty

A set N0 of initial nodes, N0 is not empty

A set Nf of final nodes, Nf is not empty

A set E of edges, each edge from one node to another ( ni , nj ), i is predecessor, j is successor

144

Three Example Graphs0

21

3

N0 = { 0 }Nf = { 3 }

0

21

3

N0 = { }Nf = { 3 }

9

0

43

7

1

5

8

2

6

N0 = { 0, 1, 2 }Nf = { 7, 8, 9 }

Not aNot avalidvalidgraphgraph

22/08/2011

49

145

Paths in Graphs Path : A sequence of nodes [n1, n2, , nM]

Each pair of nodes is an edge Length : The number of edges

A single node is a path of length 0 Subpath : A subsequence of nodes in p is a subpath of p Reach (n) : Subgraph that can be reached from n

97 8

0 1 2

43 5 6

Paths

[ 0, 3, 7 ][ 1, 4, 8, 5, 1 ][ 2, 6, 9 ]

Reach (0) = { 0, 3, 4, 7, 8, 5, 1, 9 }Reach ({0, 2}) = GReach([2,6]) = {2, 6, 9}

146

Test Paths and SESEs Test Path : A path that starts at an initial node and ends at a final

node Test paths represent execution of test cases

Some test paths can be executed by many tests Some test paths cannot be executed by any tests

SESE graphs : All test paths start at a single node and end at another node Single-entry, single-exit N0 and Nf have exactly one node

0

2

1

63

5

4Double-diamond graph

Four test paths[ 0, 1, 3, 4, 6 ][ 0, 1, 3, 5, 6 ][ 0, 2, 3, 4, 6 ][ 0, 2, 3, 5, 6 ]

147

Tests and Test Paths path (t) : The test path executed by test t path (T) : The set of test paths executed by the set

of tests T

Each test executes one and only one test path A location in a graph (node or edge) can be reached

from another location if there is a sequence of edges from the first location to the second Syntactic reach : A subpath exists in the graph Semantic reach : A test exists that can execute that subpath

22/08/2011

50

148

Testing and Covering Graphs We use graphs in testing as follows :

Developing a model of the software as a graph Requiring tests to visit or tour specific sets of nodes, edges or subpaths

Test Requirements (TR) : Describe properties of test paths Test Criterion : Rules that define test requirements Satisfaction : Given a set TR of test requirements for a

criterion C, a set of tests T satisfies C on a graph if and only iffor every test requirement in TR, there is a test path in path(T)that meets the test requirement tr

Control flow (Structural) Coverage Criteria : Defined on a graph just in terms of nodes and edges

Data Flow Coverage Criteria : Requires a graph to be annotated with references to variables

149

Control flow test criteria

150

Node and Edge Coverage The first (and simplest) two criteria require that each node

and edge in a graph be executed

NodeNode CoverageCoverage (NC)(NC) :: TestTest setset TT satisfiessatisfies nodenode coveragecoverage onongraphgraph GG ifif onlyonly ifif forfor everyevery syntacticallysyntactically reachablereachable nodenode nninin NN,, therethere isis somesome pathpath pp inin path(T)path(T) suchsuch thatthat pp visitsvisits nn..

NodeNode CoverageCoverage (NC)(NC) :: TRTR containscontains eacheach reachablereachable nodenode ininGG..

We can abbreviate it in terms of the set of test requirements

22/08/2011

51

151

Node and Edge Coverage Edge coverage is slightly stronger than node coverage

EdgeEdge CoverageCoverage (EC)(EC) :: TRTR containscontains eacheach reachablereachable pathpath ofoflengthlength upup toto 11,, inclusive,inclusive, inin GG..

The length up to 1 allows for graphs with one node and no edges

NC and EC are only different when there is an edge and another subpath between a pair of nodes (as in an if-else statement)

Node Coverage : TR = { 0, 1, 2 }Test Path = [ 0, 1, 2 ]

Edge Coverage : TR = { (0,1), (0, 2), (1, 2) }Test Paths = [ 0, 1, 2 ]

[ 0, 2 ]

1

2

0

152

Covering Multiple Edges Edge-pair coverage requires pairs of edges, or subpaths of

length 2EdgeEdge--PairPair CoverageCoverage (EPC)(EPC) :: TRTR containscontains eacheach reachablereachablepathpath ofof lengthlength upup toto 22,, inclusive,inclusive, inin GG..

The length up to 2 is used to include graphs that have less than 2 edges

CompleteComplete PathPath CoverageCoverage (CPC)(CPC) :: TRTR containscontains allall pathspaths ininGG..

SpecifiedSpecified PathPath CoverageCoverage (SPC)(SPC) :: TRTR containscontains aa setset SS ofoftesttest paths,paths, wherewhere SS isis suppliedsupplied asas aa parameterparameter..

The logical extension is to require all paths

Unfortunately, this is impossible if the graph has a loop, so a weak compromise is to make the tester decide which paths:

153

Structural Coverage ExampleNode Coverage

TR = { 0, 1, 2, 3, 4, 5, 6 }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ]

6

0

2

1

3 4

Edge CoverageTR = { (0,1), (0,2), (1,2), (2,3), (2,4), (3,6), (4,5), (4,6), (5,4) }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 2, 4, 5, 4, 6 ]

Edge-Pair CoverageTR = { [0,1,2], [0,2,3], [0,2,4], [1,2,3], [1,2,4], [2,3,6],

[2,4,5], [2,4,6], [4,5,4], [5,4,5], [5,4,6] }Test Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 2, 3, 6 ]

[ 0, 2, 4, 5, 4, 5, 4, 6 ]

Complete Path CoverageTest Paths: [ 0, 1, 2, 3, 6 ] [ 0, 1, 2, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 6 ] [ 0, 1, 2, 4, 5, 4, 5, 4, 5, 4, 6 ]

5

22/08/2011

52

154

Loops in Graphs If a graph contains a loop, it has an infinite number of

paths

Thus, CPC is not feasible

SPC is not satisfactory because the results are subjective and vary with the tester

Attempts to deal with loops: 1970s : Execute cycles once ([4, 5, 4] in previous example, informal) 1980s : Execute each loop, exactly once (formalized) 1990s : Execute loops 0 times, once, more than once (informal description) 2000s : Prime paths

155

Simple Paths and Prime Paths Simple Path : A path from node ni to nj is simple if no node

appears more than once, except possibly the first and last nodes are the same No internal loops Includes all other subpaths A loop is a simple path

Prime Path : A simple path that does not appear as a proper subpath of any other simple path

Simple Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],[ 2, 3, 0, 1 ], [ 0, 1, 3 ], [ 0, 2, 3 ], [ 1, 3, 0 ], [ 2, 3, 0 ],[ 3, 0, 1 ], [3, 0, 2 ], [ 0, 1], [ 0, 2 ], [ 1, 3 ], [ 2, 3 ], [ 3, 0 ], [0], [1], [2], [3]

Prime Paths : [ 0, 1, 3, 0 ], [ 0, 2, 3, 0], [ 1, 3, 0, 1 ],[ 2, 3, 0, 2 ], [ 3, 0, 1, 3 ], [ 3, 0, 2, 3 ], [ 1, 3, 0, 2 ],[ 2, 3, 0, 1 ]

1 2

0

3

156

Prime Path Coverage A simple, elegant and finite criterion that requires loops to be

executed as well as skipped

PrimePrime PathPath CoverageCoverage (PPC)(PPC) :: TRTR containscontains eacheach primeprime pathpath inin GG..

Will tour all paths of length 0, 1, That is, it subsumes node, edge, and edge-pair coverage

22/08/2011

53

157

Round Trips Round-Trip Path : A prime path that starts and ends at the

same node

SimpleSimple RoundRound TripTrip CoverageCoverage (SRTC)(SRTC) :: TRTR containscontains atat leastleastoneone roundround--triptrip pathpath forfor eacheach reachablereachable nodenode inin GG thatthat beginsbeginsandand endsends aa roundround--triptrip pathpath..

CompleteComplete RoundRound TripTrip CoverageCoverage (CRTC)(CRTC) :: TRTR containscontains allallroundround--triptrip pathspaths forfor eacheach reachablereachable nodenode inin GG..

These criteria omit nodes and edges that are not in round trips That is, they do not subsume edge-pair, edge, or node coverage

158

Prime Path Example The previous example has 38 simple paths Only nine prime paths

Prime Paths[ 0, 1, 2, 3, 6 ][ 0, 1, 2, 4, 5 ][ 0, 1, 2, 4, 6 ][ 0, 2, 3, 6 ][ 0, 2, 4, 5][ 0, 2, 4, 6 ]

[ 5, 4, 6 ][ 4, 5, 4 ][ 5, 4, 5 ]

Execute loop once

Execute loop more than once

5

0

2

1

3 4

6

Execute loop 0 times

159

Simple & Prime Path Example

5

0

2

1

3 4

6

Len 0[0][1][2][3][4][5][6] !

! means path terminates

Len 1[0, 1][0, 2][1, 2][2, 3][2, 4][3, 6] ![4, 6] ![4, 5][5, 4]

Len 2[0, 1, 2][0, 2, 3][0, 2, 4][1, 2, 3][1, 2, 4][2, 3, 6] ![2, 4, 6] ![2, 4, 5] ![4, 5, 4] *[5, 4, 6] ![5, 4, 5] * * means path

cycles

Len 3[0, 1, 2, 3][0, 1, 2, 4][0, 2, 3, 6] ![0, 2, 4, 6] ![0, 2, 4, 5] ![1, 2, 3, 6] ![1, 2, 4, 5] ![1, 2, 4, 6] !

Len 4[0, 1, 2, 3, 6] ![0, 1, 2, 4, 6] ![0, 1, 2, 4, 5] !

Prime Paths

Simple paths

22/08/2011

54

160

Data flow test criteria

161

Data Flow CriteriaGoal: Try to ensure that values are computed and used correctly Definition (def) : A location where a value for a variable is stored

into memory Use : A location where a variables value is accessed def (n) or def (e) : The set of variables that are defined by node n or

edge e use (n) or use (e) : The set of variables that are used by node n or

edge e

0

2

1

63

5

4X = 42

Z = X-8

Z = X*2 Defs: def (0) = {X}def (4) = {Z}def (5) = {Z}

Uses: use (4) = {X}use (5) = {X}

162

DU Pairs and DU Paths DU pair : A pair of locations (li, lj) such that a

variable v is defined at li and used at lj Def-clear : A path from li to lj is def-clear with

respect to variable v if v is not given another value on any of the nodes or edges in the path

Reach : If there is a def-clear path from li to lj with respect to v, the def of v at li reaches the use at lj

du-path : A simple subpath that is def-clear with respect to v from a def of v to a use of v

du (ni, nj, v) the set of du-paths from ni to nj du (ni, v) the set of du-paths that start at ni

22/08/2011

55

163

Data Flow Test Criteria Three criteria

Use every def Get to every use Follow all du-paths

164

Data Flow Test Criteria

AllAll--defsdefs coveragecoverage (ADC)(ADC) :: ForFor eacheach setset ofof dudu--pathspaths SS == dudu ((nn,,vv),), TRTR containscontains atat leastleast oneone pathpath dd inin SS..

AllAll--usesuses coveragecoverage (AUC)(AUC) :: ForFor eacheach setset ofof dudu--pathspaths toto usesuses SS ==dudu ((nnii,, nnjj,, vv),), TRTR containscontains atat leastleast oneone pathpath dd inin SS..

AllAll--dudu--pathspaths coveragecoverage (ADUPC)(ADUPC) :: ForFor eacheach setset SS == dudu ((ni,ni, njnj,,vv),), TRTR containscontains everyevery pathpath dd inin SS..

Then we make sure that every def reaches all possible uses

Finally, we cover all the paths between defs and uses

First, we make sure every def reaches a use

165

Data Flow Testing Example

0

2

1

63

5

4X = 42

Z = X-8

Z = X*2

All-defs for X

[ 0, 1, 3, 4 ]All-uses for X

[ 0, 1, 3, 4 ][ 0, 1, 3, 5 ]

All-du-paths for X

[ 0, 1, 3, 4 ][ 0, 2, 3, 4 ][ 0, 1, 3, 5 ][ 0, 2, 3, 5 ]

22/08/2011

56

166

Graph Coverage Criteria Subsumption

Simple Round Trip Coverage

SRTCNode CoverageNC

Edge Coverage

EC

Edge-Pair Coverage

EPC

Prime Path Coverage

PPC

Complete Path Coverage

CPC

Complete Round Trip Coverage

CRTC

All-DU-Paths Coverage

ADUP

All-uses Coverage

AUC

All-defs Coverage

ADC

167

Graph Coverage for Source Code

168

Graph Coverage for Source Code The most common application of graph criteria is to

program source Graph : Usually the control flow graph (CFG) Node coverage : Execute every statement Edge coverage : Execute every branch Loops : Looping structures such as for loops, while

loops, etc. Data flow coverage : Augment the CFG

defs are statements that assign values to variables uses are statements that use variables

22/08/2011

57

169

Control Flow Graphs A CFG models all executions of a method by describing control

structures Nodes : statements or sequences of statements (basic blocks) Edges : Transfers of control Basic Block : A sequence of statements such that if the first

statement is executed, all statements will be (no branches)

CFGs are sometimes annotated with extra information branch predicates defs uses

Rules for translating statements into graphs

170

CFG : The if Statementif (x < y){

y = 0;x = x + 1;

}else{

x = y;}

4

1

2 3

x >= yx < y

x = yy = 0

x = x + 1

if (x < y){

y = 0;x = x + 1;

}3

1

2 x >= yx < y

y = 0x = x + 1

171

CFG : The if-Return Statement

if (x < y){

return;}print (x);return;

3

1

2 x >= yx < y

return

print (x)return

No edge from node 2 to 3.The return nodes must be distinct.

22/08/2011

58

172

Loops Loops require extra nodes to be added

Nodes that do not represent statements or basic blocks

173

CFG : while and for Loopsx = 0;while (x < y){

y = f (x, y);x = x + 1;

}

1x = 0

43y =f(x,y)x = x + 1

x >= yx < y

for (x = 0; x < y; x++){

y = f (x, y);}

1

x = x + 1

2

3 5

x >= yx < y

y = f (x, y)

4

2dummy node

x = 0implicitly

initializes loop

implicitly increments loop

174

CFG : The case (switch) Structure

read ( c) ;switch ( c ){

case N:y = 25;break;

case Y:y = 50;break;

default:y = 0;break;

}print (y);

5

1 read ( c );c == N

y = 0;break;

2 43

c == Y default

y = 50;break;

y = 25;break;

print (y);

22/08/2011

59

175

Example Control Flowpublic static void computeStats (int [ ] numbers){

int length = numbers.length;double med, var, sd, mean, sum, varsum;

sum = 0;for (int i = 0; i < length; i++){

sum += numbers [ i ];} med = numbers [ length / 2 ];mean = sum / (double) length;varsum = 0;for (int i = 0; i < length; i++){

varsum = varsum + ((numbers [ I ] - mean) * (numbers [ I ] - mean));}var = varsum / ( length - 1.0 );sd = Math.sqrt ( var );System.out.println ("length: " + length);System.out.println ("mean: " + mean);System.out.println ("median: " + med);System.out.println ("variance: " + var);System.out.println ("standard deviation: " + sd);

}

176

Control Flow Graph for Statspublic static void computeStats (int [ ] numbers){





}

i = 0

i >= length

i < lengthi++

i >= lengthi < length

i = 0

1

2

3

54

6

87

Control Flow TRs and Test Paths PPC

1

2

3

54

6

87

TRA. [ 3, 4, 3 ]B. [ 4, 3, 4 ]C. [ 7, 6, 7 ]D. [ 7, 6, 8 ]E. [ 6, 7, 6 ]F. [ 1, 2, 3, 4 ]G. [ 4, 3, 5, 6, 7 ]H. [ 4, 3, 5, 6, 8 ]I. [ 1, 2, 3, 5, 6, 7 ]J. [ 1, 2, 3, 5, 6, 8 ]

Test Pathsi. [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]ii. [ 1, 2, 3, 4, 3, 4, 3,

5, 6, 7, 6, 7, 6, 8 ]iii. [ 1, 2, 3, 4, 3, 5, 6, 8 ]iv. [ 1, 2, 3, 5, 6, 7, 6, 8 ]v. [ 1, 2, 3, 5, 6, 8 ]

Prime Path Coverage

22/08/2011

60

178

Data Flow Coverage for Source def : a location where a value is stored into memory

x appears on the left side of an assignment (x = 44;) x is an actual parameter in a call and the method changes its value x is a formal parameter of a method (implicit def when method

starts) x is an input to a program

use : a location where variables value is accessed x appears on the right side of an assignment x appears in a conditional test x is an actual parameter to a method x is an output of the program x is an output of a method in a return statement

If a def and a use appear on the same node, then it is only a DU-pair if the def occurs after the use and the node is in a loop

179

Example Data Flowpublic static void computeStats (int [ ] numbers){





}

180

Control Flow Graph for Stats

8

1

2

4

3

5

6

7

( numbers )sum = 0length = numbers.length

i = 0

i >= length

i < length

sum += numbers [ i ]i++

med = numbers [ length / 2 ]mean = sum / (double) length;varsum = 0i = 0

i >= length

i < length

varsum =

i++

var = varsum / ( length - 1.0 )sd = Math.sqrt ( var )print (length, mean, med, var, sd)

22/08/2011

61

181

CFG for Stats With Defs & Uses

8

1

2

4

3

5

6

7

def (1) = { numbers, sum, length }

def (2) = { i }

def (5) = { med, mean, varsum, i }use (5) = { numbers, length, sum }

use (3, 5) = { i, length }use (3, 4) = { i, length }

def (4) = { sum, i }use (4) = { sum, numbers, i }

use (6, 8) = { i, length }use (6, 7) = { i, length }

def (7) = { varsum, i }use (7) = { varsum, numbers, i, mean }

182

Defs and Uses Tables for StatsNode Def Use

1 { numbers, sum, length }

2 { i }34 { sum, i } { numbers, i, sum }5 { med, mean,

varsum, i }{ numbers, length, sum }

67 { varsum, i } { varsum, numbers, i,

mean }8 { var, sd } { varsum, length, var,

mean, med, var, sd }

Edge Use

(1, 2)(2, 3)(3, 4) { i, length }(4, 3)(3, 5) { i, length }(5, 6)(6, 7) { i, length }(7, 6)(6, 8) { i, length }

DU Pairs for Statsvariable DU Pairs

numbers (1, 4) (1, 5) (1, 7)length (1, 5) (1, 8) (1, (3,4)) (1, (3,5)) (1, (6,7)) (1, (6,8))med (5, 8)var (8, 8)sd (8, 8)mean (5, 7) (5, 8)sum (1, 4) (1, 5) (4, 4) (4, 5)varsum (5, 7) (5, 8) (7, 7) (7, 8)i (2, 4) (2, (3,4)) (2, (3,5)) (2, 7) (2, (6,7)) (2, (6,8))

(4, 4) (4, (3,4)) (4, (3,5)) (4, 7) (4, (6,7)) (4, (6,8))(5, 7) (5, (6,7)) (5, (6,8))(7, 7) (7, (6,7)) (7, (6,8))

22/08/2011

62

184

DU Paths for Stats No DuplicatesThere are 38 DU paths for Stats, but only 12 unique

[ 1, 2, 3, 4 ][ 1, 2, 3, 5 ][ 1, 2, 3, 5, 6, 7 ][ 1, 2, 3, 5, 6, 8 ][ 2, 3, 4 ][ 2, 3, 5 ]

[ 4, 3, 4 ][ 4, 3, 5 ][ 5, 6, 7 ][ 5, 6, 8 ][ 7, 6, 7 ][ 7, 6, 8 ]

5 expect a loop not to be entered

5 require at least one iteration of a loop

2 require at least two iteration of a loop

185

Test Cases and Test PathsTest Case : numbers = (44) ; length = 1Test Path : [ 1, 2, 3, 4, 3, 5, 6, 7, 6, 8 ]Additional DU Paths covered[ 1, 2, 3, 4 ] [ 2, 3, 4 ] [ 4, 3, 5 ] [ 5, 6, 7 ] [ 7, 6, 8 ]The five stars that require at least one iteration of a loopTest Case : numbers = (2, 10, 15) ; length = 3Test Path : [ 1, 2, 3, 4, 3, 4, 3, 4, 3, 5, 6, 7, 6, 7, 6, 7, 6, 8 ]DU Paths covered[ 4, 3, 4 ] [ 7, 6, 7 ]The two stars that require at least two iterations of a loopOther DU paths require arrays with length 0 to skip loopsBut the method fails with divide by zero on the statement

mean = sum / (double) length; A fault wasfound

186

Testing State Behavior

22/08/2011

63

187

Testing State Behavior A finite state machine (FSM) is a graph that

describes how software variables are modified during execution

Nodes : States, representing sets of values for key variables

Edges : Transitions, possible changes in the state

Off On

switch up

switch down

188

Finite State Machine Two Variables

Tropical Depressioncirculation = yes

windspeed < 39mph

Tropical Stormcirculation = yes

windspeed = 39..73 mph

Major Hurricanecirculation = yes

windspeed >= 110 mph

Hurricanecirculation = yes

windspeed 74..109 mph

Something Elsecirculation = no

windspeed = 0..38 mph

Other variables may exist but not be part of state

189

Finite State Machines are Common FSMs can accurately model many kinds of software

Embedded and control software (think electronic gadgets) Abstract data types Compilers and operating systems Web applications

Creating FSMs can help find software problems Numerous languages for expressing FSMs

UML statecharts Automata State tables (SCR) Petri nets

Limitations : FSMs are not always practical for programs that have lots of states (for example, GUIs)

22/08/2011

64

190

Annotations on FSMs FSMs can be annotated with different types of actions

Actions on transitions Entry actions to nodes Exit actions on nodes

Actions can express changes to variables or conditions on variables

These slides use the basics: Preconditions (guards) : conditions that must be true for

transitions to be taken Triggering events : changes to variables that cause transitions to

be taken

This is close to the UML Statecharts, but not exactly the same

191

Example Annotations

Closed Open

open elevator door

pre: elevSpeed = 0

trigger: openButton = pressed

192

Covering FSMs Node coverage : execute every state (state coverage) Edge coverage : execute every transition (transition coverage) Edge-pair coverage : execute pairs of transitions (transition-pair)

Data flow: Nodes often do not include defs or uses of variables Defs of variables in triggers are used immediately (the next state) Defs and uses are usually computed for guards, or states are

extended FSMs typically only model a subset of the variables

Generating FSMs is often harder than covering them

22/08/2011

65

193

Deriving FSMs With some projects, a FSM (such as a statechart) was created

during design Tester should check to see if the FSM is still current with respect

to the implementation If not, it is very helpful for the tester to derive the FSM Strategies for deriving FSMs from a program:

1. Combining control flow graphs2. Using the software structure3. Modeling state variables4. Using implicit or explicit specifications

Discussion of these strategies based on a digital watch Class Watch uses class Time

194

Class Watchclass Watch// Constant values for the button (inputs)private static final int NEXT = 0;private static final int UP = 1;private static final int DOWN = 2;// Constant values for the stateprivate static final int TIME = 5;private static final int STOPWATCH = 6;private static final int ALARM = 7;// Primary state variableprivate int mode = TIME;// Three separate times, one for each stateprivate Time watch, stopwatch, alarm;

public Watch () // Constructorpublic void doTransition (int button) // Handles inputspublic String toString () // Converts values

class Time ( inner class )private int hour = 0;private int minute = 0;

public void changeTime (int button)public String toString ()

195

// Takes the appropriate transition when a button is pushed.public void doTransition (int button){

switch ( mode ){

case TIME:if (button == NEXT)

mode = STOPWATCH;else

watch.changeTime (button);break;

case STOPWATCH:if (button == NEXT)

mode = ALARM;else

stopwatch.changeTime (button);break;

case ALARM:if (button == NEXT)

mode = TIME;else

alarm.changeTime (button);break;

default:break;

}} // end doTransition()

// Increases or decreases the time.// Rolls around when necessary.public void changeTime (int button){

if (button == UP){

minute += 1;if (minute >= 60){

minute = 0;hour += 1;if (hour >= 12)

hour = 0;}

}else if (button == DOWN){

minute -= 1;if (minute < 0){

minute = 59;hour -= 1;if (hour

22/08/2011

66

196

1. Combining Control Flow Graphs The first instinct for inexperienced developers is to

draw CFGs and link them together

This is really not a FSM

Several problems Methods must return to correct callsites built-in

nondeterminism Implementation must be available before graph can be built This graph does not scale up

Watch example

197

CFGs for Watch

5 6 7 8

1

2 3 4

11 12 13

9 10

14

doTransition ()1

12

6

8

2

4

10

6

8

2

4

10

changeTime ()

198

2. Using the Software Structure A more experienced programmer may map methods

to states

These are really not states

Problems Subjective different testers get different graphs Requires in-depth knowledge of implementation Detailed design must be present

Watch example

22/08/2011

67

199

Software Structure for Watch

button

inMain

inChangeTimeinDoTransition

button==NEXT / change mode

button==UP or button==DOWN

button==DOWN / change hour, minute

button==UP / change hour, minute

??? Not clear what triggers this

??? Should inChangeTime transit back to inMain???

200

3. Modeling State Variables More mechanical State variables are usually defined early First identify all state variables, then choose

which are relevant In theory, every combination of values for the

state variables defines a different state In practice, we must identify ranges, or sets

of values, that are all in one state Some states may not be feasible

201

State Variables in WatchConstants NEXT, UP, DOWN TIME, STOPWATCH, ALARMNon-constants int mode Time watch, stopwatch, alarmTime class variables int hour int minute

Not relevant, really just values

Merge into the three Time variables

22/08/2011

68

202

State Variables and ValuesThese three ranges actually represent quite a bit of thought and semantic domain knowledge of the program

Relevant State Variables

mode : TIME, STOPWATCH, ALARM watch : 12:00, 12:01..12:59, 01:00..11:59 stopwatch : 12:00, 12:01..12:59, 01:00..11:59 alarm : 12:00, 12:01..12:59, 01:00..11:59

Total 3*3*3*3 = 81 states

But the three watches are independent, so we only care about 3+3+3 = 9 states

(still a messy graph )

203

State Variable Model for Watchmode = TIMEwatch = 12:00

mode = TIMEwatch = 12:01.. 12:59

mode = TIMEwatch = 01:00..12:59

mode = STOPstopw = 12:00

mode = STOPstopw = 12:01.. 12:59

mode = STOPstopw = 01:00..12:59

mode = ALARMalarm = 12:00

mode = ALARMalarm = 12:01.. 12:59

mode = ALARMalarm = 01:00..12:59

nextnext

next nextnext next

nextnext next

nextnext next

next

next

next

next

next

next

nextnext

nextnext next

nextnext

next next

204

NonDeterminism in the State Variable Model Each state has three outgoing transitions on next

This is a form of non-determinism, but it is not reflected in the implementation

Which transition is taken depends on the current state of the other watch

The 81-state model would not have this non-determinism

This situation can also be handled by a hierarchy of FSMs, where each watch is in a separate FSM and they are organized together

22/08/2011

69

205

4. Using Implicit or Explicit Specifications Relies on explicit requirements or formal specifications that

describe software behavior

These could be derived by the tester

These FSMs will sometimes look much like the implementation-based FSM, and sometimes much like the state-variable model For watch, the specification-based FSM looks just like the state-

variable FSM, so is not shown

The disadvantage of FSM testing is that some implementation decisions are not modeled in the FSM

206

Tradeoffs in Applying Graph Coverage Criteria to FSMs There are two advantages

1. Tests can be designed before implementation2. Analyzing FSMs is much easier than analyzing source

There are three disadvantages1. Some implementation decisions are not modeled in the

FSM2. There is some variation in the results because of the

subjective nature of deriving FSMs3. Tests have to mapped to actual inputs to the program

the names that appear in the FSM may not be the same as the names in the program

Static testing

22/08/2011

70

Contents Reviews and the test process Types of review Static analysis

208

Reviews and the test process

209

Static testing Static: Not Dynamic, i.e. you dont execute

the code This is the process of testing some

documentation products

The primary goal of Static Testing is to reduce defects in the software by reducing defects in the documentation from which the software is developed

210

22/08/2011

71

Static techniques Do not execute code

People techniques Specific techniques

Desk checking Developer tests his/her own documentation

Walkthroughs Developer walks the participants through the

documentation Peer Reviews/Inspections

More formal documentation is distributed beforehand

211

Benefits of reviews Development productivity improvement Reduced development timescales Reduced testing time and cost Lifetime cost reductions Reduced fault levels Improved customer relations

212

Reviews are cost-effective 10 times reduction in faults reaching test,

testing cost reduced by 50% to 80% Freedman & Weinberg, Handbook of

Walkthroughs, Inspections & Technical Reviews reduce faults by a factor of 10

Yourdon, Structured Walkthroughs 25% reduction in schedules, remove 80% -

95% of faults at each stage, 28 times reduction in maintenance cost, many others Gilb & Graham, Software Inspection

213

22/08/2011

72

What can be Inspected? Anything written down can be Inspected

policy, strategy, business plans, marketing or advertising material, contracts

system requirements, feasibility studies, acceptance test plans

test plans, test designs, test cases, test results system designs, logical & physical software code user manuals, procedures, training material

214

What can be reviewed? anything which could be Inspected

i.e. anything written down plans, visions, big picture, strategic

directions, ideas project progress

work completed to schedule, etc. Should we develop this marketing options

215

What to review / Inspect?

216

Tests

Tests

Tests

Tests

Requirements

Design

Code

Functions

Integration T

Unit Test

Accept. Test

System Test

22/08/2011

73

Costs of reviews Rough guide: 5%-15% of development effort

half day a week is 10% Effort required for reviews

planning (by leader / moderator) preparation / self-study checking meeting fixing / editing / follow-up recording & analysis of statistics / metrics process improvement (should!)

217

Types of review

218

Types of review of documents Informal Review (undocumented)

widely viewed as useful and cheap (but no one can prove it!). A helpful first step for chaotic organisations.

Technical Review (or peer review) includes peer and technical experts, no management

participation. Normally documented, fault-finding. Can be rather subjective.

Decision-making Review group discusses document and makes a decision

about the content, e.g. how something should be done, go or no-go decision, or technical comments

219

22/08/2011

74

Types of review of documents Walkthrough

author guides the group through a document and his or her thought processes, so all understand the same thing, consensus on changes to make

Inspection formal individual and group checking, using sources

and standards, according to generic and specific rules and checklists, using entry and exit criteria, Leader must be trained & certified, metrics required

220

Reviews in general 1 Objectives / goals

validation & verification against specifications & standards

achieve consensus (excluding Inspection) process improvement (ideal, included in

Inspection)

221

Reviews in general 2 Activities

planning overview / kickoff meeting (Inspection) preparation / individual checking review meeting (not always) follow-up (for some types) metrics recording & analysis (Inspections and

sometimes reviews)

222

22/08/2011

75

Reviews in general 3 Roles and responsibilities

Leader / moderator - plans the review / Inspection, chooses participants, helps & encourages, conducts the meeting, performs follow-up, manages metrics

Author of the document being reviewed / Inspected Reviewers / Inspectors - specialised fault-finding roles for

Inspection Managers - excluded from some types of review, need to

plan project time for review / Inspection Others: e.g. Inspection/ review Co-ordinator

223

Reviews in general 4 Deliverables

Changes (edits) in review product Change requests for source documents

(predecessor documents to product being reviewed / Inspected)

Process improvement suggestions to the review / Inspection process to the development process which produced the

product just reviewed / Inspected Metrics (Inspection and some types of review)

224

Reviews in general 5 Pitfalls (they dont always work!)

Lack of training in the technique (especially Inspection, the most formal)

Lack of or quality of documentation - what is being reviewed / Inspected

Lack of management support Failure to improve processes

225

22/08/2011

76

Inspection is different

the document to be reviewed is given out in advance

typically dozens of pages to review

instructions are "please review this"

226

not just product, sources

chunk or sample

training, roles

Inspection is different

some people have time to look through it and make comments before the meeting (which is difficult to arrange)

the meeting often lasts for hours

227

entry criteria to meeting, may not be worth holding

2 max., often much shorter

The Inspection Process

228

SoftwareDevelopment

Stage

.

.

Planning

Kickoff

IndChk Meet Edit

ChangeRequest

Process Improvement

Entry

Next SoftwareDevelopment

Stage

Exit

22/08/2011

77

Static analysis

229

What can static analysis do? Remember: static techniques do not execute the

code

A form of automated testing check for violations of standards check for things which may be a fault

Descended from compiler technology a compiler statically analyses code, and knows a lot

about it, e.g. variable usage; finds syntax faults static analysis tools extend this knowledge can find unreachable code, undeclared variables,

parameter type mis-matches, uncalled functions & procedures, array bound violations, etc.

230

Data flow analysis

This is the study of program variables variable defined* where a value is stored into it variable used where the stored value is accessed variable is undefined before it is defined or when it

goes out of scope

231*defined should not be confused with declared

x = y + zIF a > b THEN read(S)

x is defined, y and z are used

a and b are used, S is defined

22/08/2011

78

Data flow analysis faults

232

n := 0read (x)n := 1while x > y do

beginread (y)write( n*y)x := x - n

end

Data flow anomaly: n isre-defined without being used

Data flow fault: y is usedbefore it has been defined(first time around the loop)

Control flow analysis Highlights:

nodes not accessible from start node infinite loops multiple entry to loops whether code is well structured, i.e. reducible whether code conforms to a flowchart grammar any jumps to undefined labels any labels not jumped to cyclomatic complexity and other metrics

233

Cyclomatic complexity cyclomatic complexity is a measure of the

complexity of a flow graph (and therefore the code that the flow graph

represents) the more complex the flow graph, the greater

the measure it can most easily be calculated as:

complexity = number of decisions + 1

234

22/08/2011

79

Which flow graph is most complex?

235

1

2 3 5

What is the cyclomatic complexity?

Example of control flow graph

236

Result = 0Right = 0DO WHILE more Questions

IF Answer = Correct THEN Right = Right + 1

ENDIFEND DOResult = (Right / Questions)IF Result > 60% THEN

Print "pass"ELSE

Print "failENDIF

do

if r=r+1

end

init

if

res

pass

fail

end

Pseudo-code:

Other static metrics lines of code (LOC) operands & operators (Halsteads metrics) fan-in & fan-out nesting levels function calls OO metrics: inheritance tree depth, number

of methods, coupling & cohesion symbolic evaluation

237

22/08/2011

80

Limitations and advantages Limitations

cannot distinguish "fail-safe" code from programming faults or anomalies (often creates overload of spurious error messages)

does not execute the code, so not related to operating conditions

Advantages can find faults difficult to "see" gives objective quality assessment of code

238

Summary Reviews help to find faults in development

and test documentation, and should be applied early

Types of review: informal, walkthrough, technical / peer review, Inspection

Static analysis can find faults and give information about code without executing

239

Functional testing

22/08/2011

81

241

Contents Introduction to functional testing Functional testing techniques

Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

242

Introduction to functional testing

243

Functional testing Functional testing: Deriving test cases from

program specifications Functional refers to the source of information used in

test case design, not to what is tested Also known as:

specification-based testing (from specifications) black-box testing (no view of the code)

Functional specification = description of intended program behavior either formal or informal

22/08/2011

82

244

Systematic vs Random Testing Random (uniform):

Pick possible inputs uniformly Avoids designer bias

A real problem: The test designer can make the same logical mistakes and bad assumptions as the program designer (especially if they are the same person)

But treats all inputs as equally valuable Systematic (non-uniform):

Try to select inputs that are especially valuable Usually by choosing representatives of classes that are apt

to fail often or not at all Functional testing is systematic testing

245

Why Not Random? Non-uniform distribution of faults Example: Java class roots applies quadratic

equation

Incomplete implementation logic: Program does not properly handle the case in which b2 - 4ac < 0 and a=0

Random sampling is unlikely to choose a=0.0 and b=0.0

246

Systematic Partition TestingFailure (valuable test case)No failure

Failures are sparse in the space of possible inputs ...

... but dense in some parts of the space

If we systematically test some cases from each part, we will include the dense parts

Functional testing is one way of drawing pink lines to isolate regions with likely failures

The

sp

ace

o

f po

ssib

le in

put v

alu

es

(the

ha

ysta

ck)

22/08/2011

83

247

Functional testing: exploiting the specification Functional testing uses the specification (formal or

informal) to partition the input space E.g., specification of roots program suggests division

between cases with zero, one, and two real roots Test each category, and boundaries between

categories No guarantees, but experience suggests failures often lie

at the boundaries (as in the roots program)

248

Why functional testing? Timely

Often useful in refining specifications and assessing testability before code is written

Effective finds some classes of fault (e.g., missing logic) that can

elude other approaches Widely applicable

to any description of program behavior serving as spec at any level of granularity from module to system testing.

Economical typically less expensive to design and execute than

structural (code-based) test cases

249

Early functional test design Program code is not necessary

Only a description of intended behavior is needed Even incomplete and informal specifications can be used

Although precise, complete specifications lead to better test suites

Early functional test design has side benefits Often reveals ambiguities and inconsistency in spec Useful for assessing testability

And improving test schedule and budget by improving spec Useful explanation of specification

or in the extreme case (as in XP), test cases are the spec

22/08/2011

84

250

Functional versus Structural:Classes of faults

Different testing strategies (functional, structural) are most effective for different classes of faults

Functional testing is best for finding design faults

Structural testing is best for finding programming faults

251

Functional vs structural test: granularity levels Functional test applies at all granularity levels:

Unit (from module interface spec) Integration (from API or subsystem spec) System (from system requirements spec) Regression (from system requirements + bug history)

Structural (code-based) test design applies to relatively small parts of a system: Unit Integration

252

Steps: From specification to test cases 1. Decompose the specification

If the specification is large, break it into independently testable features to be considered in testing

2. Select representatives Representative values of each input, or Representative behaviors of a model

Often simple input/output transformations dont describe a system. We use models in program specification, in program design, and in test design

3. Form test specifications Typically: combinations of input values, or model behaviors

4. Produce and execute actual tests

22/08/2011

85

253

From specification to test cases

254

Simple example: Postal code lookup

Input: ZIP code (5-digit US Postal code)

Output: List of cities What are some

representative values (or classes of value) to test?

255

Example: Representative values

Simple example with one input, one output

Note prevalence of boundary values (0 cities, 6 characters)

and error cases

Correct zip code With 0, 1, or many cities

Malformed zip code Empty; 1-4 characters; 6 characters; very long Non-digit characters Non-character data

22/08/2011

86

256

Functional testing techniques Boundary Value testing Equivalence Class testing Special Value testing Decision Tables

257

Examples

258

The Triangle ProgramIn order for 3 integers a, b, and c to be

the sides of a triangle, we must have c1 a + b > cc2 a + c > bc3 b + c > a

A triangle is: Equilateral if all 3 sides are equal Isosceles if 2 sides are equal Scalene if no two sides are equal

a b

c

22/08/2011

87

259

The Triangle Program The Triangle Program reads in 3 integers and

decides if they form an equilateral triangle, an isosceles triangle, a scalene triangle, or if they dont form a triangle at all

The logic of the program is clear, but complex, due to the relationships between the inputs and the outputs

We assume that each side length is between 1 and 200

260

NextDate Function This program reads in a date in a certain

format and prints out the next days date. For example, an input of 03 31 1998 gives an

output of 04 01 1998. The year is constrained to lie between 1812

and 2012 inclusive

261

The Commission Problem A salesman sells locks, stocks and barrels On a monthly basis, the salesperson reports how

many of each he or she has sold Due to manufacturing constraints, we have:

1 #locks 701 #stocks 801 #barrels 90

Goal: Determine total sales and the commission

22/08/2011

88

262

The Commission Problem The parts sell for the following:

Locks $45 eachStocks $30 eachBarrels $25 each

Commission is computed on sales:10% on sales up to and including $1000; 15% on the next $800; and 20% on any amount over $1800

This program has calculation logic to test inputs and outputs are straightforward

263

Boundary Value testing

264

Boundary/Limits Testing Test values, sizes or quantities near the design

limits Value limits Length limits Volume limits Null strings vs Empty strings

Errors tend to occur near the extreme values of inputs

Robustness: How does the software react when boundaries are exceeded?

22/08/2011

89

265

Single Fault AssumptionThe Single Fault assumption from reliability

states:Failures are rarely the result of the

simultaneous occurrence of two (or more) faults.

266

Extensions of Boundary Testing Boundary value testing

Stay within the limits Robustness testing

Exceed the limits Worst-case testing

Like boundary value testing above, but discard Single Fault Assumption

Robust worst-case testing Like robustness testing above, but discard Single Fault

Assumption

267

Boundary Value TestingThe basic idea of boundary value analysis is to

use input variable values at their minimum, just above the minimum, at a nominal value, just below the maximum, and at the maximum

22/08/2011

90

268

Input Boundary Value TestingInput Boundary Value Testinga b

Test cases for a variable x, where a x b

Experience shows that errors occur more frequently for extreme values of a variable

x

x(min) x(min+) x(nom) x(max -) x(max)

269

Input Boundary Value Testing Input Boundary Value Testing ((2 2 Variables)Variables)

Test cases for variables x1 and x2, wherea x1 b and c x2 d

As in reliability theory, two variables rarelyassume their extreme values simultaneously.

a b

c

d

x2

x1

270

Robustness Testing An extension of Boundary Value testing:

Include values below the minimum value and above the maximum value

This is a form of what we call negative testing -- how does the program handle errors in the input?

This type of testing may not be possible with some strongly-typed languages, GUIs with fixed palette values or drop-down lists, etc.

22/08/2011

91

271

Robustness TestingRobustn

0 All Slides

Documents

test documentation test

software testing techniques

practical software testing

testing computer software

testing application

testing experience

software testing definitions

art of software testing