Fault based testing

FAULT BASED TESTING

INTRODUCTION

o It is necessary to test an object-oriented system at a variety of different levels

o The goal is to uncover errors that may occur as classes collaborate with one another and subsystems communicate across architectural layers

• Testing begins "in the small" on methods within a class and on collaboration between classes

• As class integration occurs, use-based testing and fault-based testing are applied

• Finally, use cases are used to uncover errors during the software validation phase

o Conventional test case design is driven by an input-process-output view of software

oObject-oriented testing focuses on designing appropriate sequences of methods to exercise the states of a class

TEST OBJECTIVES

oUntil now, we have not been very precise in our use of the notion of an 'error'.

o In order to appreciate the following discussion, it is important to make a careful distinction between the notions error, fault, and failure.

An error is a human action that produces an incorrect result. The consequence of an error is software containing a fault.

Error

A fault thus is the manifestation of an error. If encountered, a fault may result in a failure

Fault

OBJECTIVE

Design tests that have a high likelihood

of uncovering plausible faults

Begins with the analysis model

- The tester looks for plausible faults (i.e., aspects

of the implementation of the system that may result

in defects)- To determine whether

these faults exist, test cases are designed to exercise the

design or code

If the analysis and design models can provide insight

into what is likely to go wrong, then fault-based

testing can find a significant number of errors

FAULT BASED TESTING

• In fault-based testing strategies, we do not directly consider the artifact being tested when assessing the test adequacy. We only take into account the test set. Fault-based techniques are aimed at finding a test set with a high ability to detect faults.

• We will discuss two fault-based testing techniques: error seeding and mutation testing

ERROR SEEDING AND MUTATION TESTING

Error Seeding Technique:

No mutants are present here.

Here source code is tested within itself.

Errors are introduced directly

Test cases which detect errors are used for

testing.

It is less efficient error testing technique.

It requires less time. 

Mutation Technique:

Mutants are developed for testing.

Here mutants are combined, compared for testing to find error introduced.

Special techniques are used to introduce errors.

Here, test cases which killmutants are used for testing.

It is more efficient than error seeding.

It is more time consuming

Error seeding and mutation testing are both error-oriented techniques and are generally, applicable to all levels of testing.

MUTATION TECHNIQUE

In Mutation Testing, we mutate (change) certain statements in the source code and check if the test cases are able to find the errors. It is a type of white box testing which is mainly used for unit testing. The changes in mutant program are kept extremely small, so it does not affect the overall objective of the program.

GOAL OF MUTATION TESTING

The goal of Mutation Testing is to assess the quality of the test cases which should be robust enough to fail mutant code. This method is also called as Fault based testing strategy as it involves creating fault in the program

HOW TO EXECUTE MUTATION TESTING?

Following are the steps to execute mutation testing:

Step 1: Faults are introduced into the source code of the program by creating many versions called mutants. Each mutant should contain a single fault, and the goal is to cause the mutant version to fail which demonstrates the effectiveness of the test cases.

Step 2: Test cases are applied to the original program and also to the mutant program. A test case should be adequate, and it is tweaked to detect faults in a program.

Step 3: Compare the results of original and mutant program.

Step 4: If the original program and mutant programs generate the same output, then that the mutant is killed by the test case. Hence the test case is good enough to detect the change between the original and the mutant program.

Step 5: If the original program and mutant program generate different output, Mutant is kept alive. In such cases , more effective test cases need to be created that  kill all mutants.

HOW TO CREATE MUTANT PROGRAMS?

Original Program

If (x>y)

Print "Hello"

Else

Print "Hi"

Mutant Program

If(x)

Print "Hello"

Else

Print "Hi"

A mutation is nothing but a single syntactic change that is made to the program statement. Each mutant program should differ from the original program by one mutation.

WHAT TO CHANGE IN A MUTANT PROGRAM?

MUTATION SCORE

• The mutation score is defined as the percentage of killed mutants with the total number of mutants.

• Mutation Score = (Killed Mutants / Total number of Mutants) * 100

Test cases are mutation adequate if the score is 100%. Experimental results have shown that mutation testing is an effective approach for the measuring the adequacy of the test cases. But, the main drawback is that the high cost of generating the mutants and executing each test case against that mutant program.

FAULT INJECTION

FAULT INJECTION IS A SOFTWARE TESTING TECHNIQUE BY INTRODUCING FAULTS INTO THE CODE FOR IMPROVING THE COVERAGE AND USUALLY USED WITH STRESS TESTING FOR ROBUSTNESS OF THE DEVELOPED SOFTWARE.

WHAT IS FAULT?

A fault is the cause of an error. An incorrect step, process, or data definition in a computer program which causes the program to perform in an unintended or unanticipated manner. A software fault lies in software, a hardware fault lies in hardware.

SecurityRecoverability

Performance

Look & Feel

Functional & Logical

Business Logic

FAULT INJECTION METHODS:

• It is a fault injection technique where source code is modified to inject simulated faults into a system.

Compile Time Injections

• It makes use of software trigger to inject a fault into a software system during run time.

• The Trigger can be of two types, Time Based triggers and Interrupt Based Triggers.

Run-Time Injections