CS 451 Software Engineering Winter 2009yfcai/CS451/slides/TestingOverview.pdf · The integration process is preformed in five steps: 1. The main control module is used as a test driver,

1 Drexel University

CS 451 Software Engineering

Winter 2009

1

Yuanfang Cai

Room 104, University Crossings

215.895.0298

[email protected]

Drexel University

Testing Process

Drexel University

Testing Testing only reveals the presence of defects Does not identify nature and location of defects Identifying & removing the defect => role of

debugging and rework Preparing test cases, performing testing, defects

identification & removal all consume effort Overall testing becomes very expensive : 30-

50% development cost

Drexel University

Levels of Testing The code contains requirement defects, design

defects, and coding defects Nature of defects is different for different injection

stages One type of testing will be unable to detect the

different types of defects Different levels of testing are used to uncover

these defects

Drexel University

User needs Acceptance testing

Requirement specification

System testing

Design

code

Integration testing

Unit testing

Levels of Testing

Drexel University

Unit Testing Different modules tested separately Focus: defects injected during coding Essentially a code verification technique, covered

in previous chapter UT is closely associated with coding Frequently the programmer does UT; coding

phase sometimes called “coding and unit testing”

Drexel University

Integration Testing Focuses on interaction of modules in a

subsystem Unit tested modules combined to form

subsystems Test cases to “exercise” the interaction of

modules in different ways May be skipped if the system is not too large

Drexel University

System Testing Entire software system is tested Focus: does the software implement the

requirements? Validation exercise for the system with

respect to the requirements Generally the final testing stage before the

software is delivered May be done by independent people Defects removed by developers Most time consuming test phase

Drexel University

Acceptance Testing Focus: Does the software satisfy user needs? Generally done by end users/customer in

customer environment, with real data Only after successful AT software is deployed Any defects found,are removed by developers Acceptance test plan is based on the acceptance

test criteria in the SRS

Drexel University

Other forms of testing Performance testing

tools needed to “measure” performance Stress testing

load the system to peak, load generation tools needed Regression testing

test that previous functionality works alright important when changes are made Previous test records are needed for comparisons Prioritization of testcases needed when complete test

suite cannot be executed for a change Smoke testing

Nightly build

Drexel University

Defect logging and tracking A large software may have thousands of defects,

found by many different people Often person who fixes (usually the coder) is

different from who finds Due to large scope, reporting and fixing of

defects cannot be done informally Defects found are usually logged in a defect

tracking system and then tracked to closure Defect logging and tracking is one of the best

practices in industry

Drexel University

Defect logging… A defect in a software project has a life cycle of

its own, like Found by someone, sometime and logged along with

info about it (submitted) Job of fixing is assigned; person debugs and then fixes

(fixed) The manager or the submitter verifies that the defect is

indeed fixed (closed) More elaborate life cycles possible

Drexel University

Validation and Verification Validation: a different set of activities that

ensures that the software that has been built is traceable to customer requirements. Are we building the right product?

Verification: the set of activities the ensures

that software correctly implements a specific function. Are we building the product right?

15 Drexel University

Organizing Testing Software testing may be viewed as a spiral process:


Organizing Testing Software Testing Steps


Software Testing Strategies How do you know when you are done testing?

There is no definitive answer to this question.

Many possible answers: You are never done testing. You are done testing when you run out of money or

time. Depends upon the certainty of failure you desire.

17


Strategic Issues Specify product requirements in a quantifiable

manner long before testing commences State testing objectives explicitly.

test coverage, mean time to failure, cost to find and fix defects, remaining defect density or frequency of occurrence

Understand the users of the software and develop a profile for each user category.

Build robust software that is designed to test itself.

18

Drexel University

Unit Testing


Testing Strategies Unit Testing – focuses verification effort on the smallest

unit of software design. (The component or module) Considerations: The module interface is tested to ensure that information

properly flows into and out of the program unit under test. All independent paths through the control structure are

exercised to ensure that all statements in a module have been executed at least once.

Boundary conditions are tested to ensure that the module operates properly.

Error handling paths are tested.

20


Testing Strategies Unit Testing

Procedure Unit testing is

normally considered as an adjunct to the coding step.

Design of tests can be done before coding begins (Agile programming)


Testing Strategies Unit Testing Procedure

Since each component is not a stand alone program, a bit of work must be done to perform the test.

Create a driver, usually a main program, that accepts test case data and passes it to the component to be tested. It also may print relevant results.

Stubs replace modules that may not be coded and are subordinate to the component to be tested. A stub may do minimum data manipulation , provides verification of entry, and returns control to the module undergoing testing.

Unit testing is simpler when modules are highly cohesive.

22

Drexel University

Junit Example Java Code to Test

public class Math { static public int add(int a, int b) { return a + b; } }

Drexel University

Junit example JUNIT test case

import junit.framework.*; public class TestMath extends TestCase { public void testAdd() {

int num1 = 3; int num2 = 2; int total = 5; int sum = 0; sum = Math.add(num1, num2); assertEquals(sum, total); } }

Drexel University

Integration Testing

Drexel University

Integration Testing Why is unit testing not enough?

Data can be lost across an interface. One module can have an inadvertent effect on another. Global data structures can present problems Individually acceptable imprecision may become magnified.

Integration testing is a systematic technique for constructing the software architecture while at the same time conducting tests to uncover errors associated with interfacing.

The objective is to take unit tested components and build a program structure that has been dictated by design.

Drexel University

Integration Testing There is a tendency to attempt non-incremental

integration – construct the whole program via a “big-bang” approach.

Thus to combine all components at once and test it as an entire program.

Top Down Integration is an incremental approach to construction of the software architecture. Modules are integrated by moving downward through the control hierarchy, beginning with the main control module.

Drexel University

Integration Testing -- Top Down Modules subordinate to the main control module are

incorporated into the structure in either a depth first or breath first manner.

Drexel University

Integration Testing -- Top Down The integration process is preformed in five steps: 1. The main control module is used as a test driver, and

stubs are substituted for all components directly subordinate to the main control module.

2. Depending on the integration approach selected (i.e. depth or breath first), subordinate stubs are replaced one at a time with actual components.

3. Tests are conducted as each component is integrated. 4. On completion of each set of tests, another stub is

replaced with the real component. 5. Regression testing may be conducted to ensure that

new errors have not been introduced.

Drexel University

Bottom Up Integration Begins construction and testing with atomic modules.

Since all subordinates are always available, there is no need for stubs.

Bottom up integration follows the following steps: 1. Low-level components are combined into clusters that perform a

specific software subfunction. 2. A driver (control program for testing) is written to coordinate test

cases input and output. 3. The cluster is tested. 4. Drivers are removed and clusters are combined moving upward in

the program structure.

One major disadvantage of bottom up integration is that the program as an entity does not exist until the last module is added.

Drexel University

Bottom Up Integration

Drexel University

Regression Testing

Drexel University

Regression Testing Each time a new module is added, the software changes. These changes may cause problems with functions that

previously worked. Therefore, we must re-execute some subset of tests that

were already conducted to ensure that changes have not propagated unintended side effects.

Any time you test, if successful you will find errors. When those errors are corrected, software changes. Therefore, regression testing must be performed in order to ensure that those changes did not propagate unintended side effects.

Regression tests may be conducted manually – I do not recommend this.

Drexel University

Testing in the OO Context

Drexel University

Unit Testing in the OO Context With OO the concept of the unit changes. Because of the dependence of sub classes, unit testing

is a bit more complicated. In the OO context, unit testing is basically Class testing.

Cannot simply test methods in a super class and assume they are still valid in each subclass

Subclasses may re-define important attributes or subordinate methods

Must fully test all methods of all classes

Drexel University

Integration Testing in the OO Context

Unlike non OO programming, OO programming does not have an hierarchical control structure.

Integrating one object at a time is also difficult because of the direct and indirect interactions of the components that make up the class.

There are two basic strategies: Thread-based Use-based

Drexel University

Test Strategies for OO Software Thread-based testing integrates the set of

classes required to respond to one input or event for the system.

Each thread is integrated and tested individually.

Regression testing is applied to ensure no side effects occur.

Good for GUIs, other event-driven multithreaded software

Drexel University

Test Strategies for OO Software Use-based testing begins the construction of

system by testing those classes that use very few server classes.

After independent classes are tested, the next layer of classes, called dependent classes are tested.

This continues until the entire system is constructed.

AKA: Bottom-up testing with objects

Drexel University

Test Strategies for OO Software Drivers can be used to test operations at the

lowest level and for testing groups of classes. A driver may be used to replace the user

interface. Stubs may be used where collaboration

between classes is required but one or more of the collaborating classes is not fully implemented.

Drexel University

Test Strategies for OO Software Cluster testing is one step in integration testing

of OO software. Here, a cluster of collaborating classes is

exercised by designing test cases that attempt to uncover errors in the collaborations.

Drexel University

Validation Testing Validation Testing Validation testing begins at the culmination of

integrations testing. Testing focuses on the user-visible actions and

user-recognizable output from the system. Validation can be defined as: Validation succeeds when software functions in a

manner that can be reasonably expected by the customer.

Software validation is achieved through a series of tests that demonstrate conformity with software requirements.

Drexel University

Test Strategies for Publicly-used Software What to do if your product does not have one

or a few customers? Most software product builders use a process

called alpha and beta testing to uncover errors that only the end-user seems able to find.

The alpha test is conducted at the developer’s site by end-users.

The beta test is conducted at the end-user sites. Beta tests usually do not have the developer present.

Drexel University

Recovery Testing Recovery Testing – most computer-based

systems must recover from faults and resume processing within a prespecified time.

Recovery testing is a system test that forces the software to fail in a variety of ways and verifies that recovery is properly preformed.

Drexel University

Security Testing Any computer system that manages sensitive

information or causes actions that can improperly harm or benefit individuals is a target for improper or illegal penetration.

Lot’s of security issues. In your code In libraries and frameworks In database access In OS layers

Security testing finds faults with the protection mechanisms built into the system that may allow improper access.

Drexel University

Security Testing During security testing, the tester plays the role(s) of

individual who desires to penetrate the system. Penetration testing requires high skill and the ability to

think outside the box Best to get an independent testing group that

specializes in penetration testing Usually it’s not the software, but people issues.

People often give out secure information. No system can truly protect against this 100%.

Drexel University

Stress Testing So far all testing assumed normal program functions

and performance.

Stress tests are designed to confront programs with abnormal quantity, frequency, or volume.

Multiple interrupts per second Increased data rate Test max memory Excessive disk hunting

Drexel University

Performance Testing Designed to test the run-time performance of software

within the context of an integrated system.

Should be performed at all steps of the testing process.

Not until system elements are fully integrated that performance of a system can really be ascertained.

Often requires hardware and software instrumentation to measure resource utilization.

Drexel University

Summary

Validation and Verification

Different Types of Testing Context purposes

CS 451 Software Engineering Winter 2009yfcai/CS451/slides/TestingOverview.pdf · The integration process is preformed in five steps: 1. The main control module is used as a test driver,

Documents