Chap 5 Selecting and Installing Software Testing Tools

Selecting and Selecting and Installing Installing Software Software

Testing ToolsTesting Tools- Aarti Gokhale.- Aarti Gokhale.

Testing Tools – The Hammers Testing Tools – The Hammers of Testingof Testing

• New methods of testing have been developed to improve testing effectiveness and testing efficiency.

• Testing, like program development,– Generates a lot of information– Necessitates numerous computer

executions– Requires co-ordination and

communication between testers

• Testing tools can ease the burden of test production (e.g. producing the test data, test scripts) , test execution, general information handling and communication.

• Testing techniques are few in number and broad in scope; that means a technique can be used in a wider range of situations.

• Testing tools are, however, numerous and narrow in scope; that means each tool provides different capabilities; but accomplish only a very specific aspect of whole of the testing objectives.

• Testing tools enhance the effectiveness and efficiency of testing. Thus, they can be compared to using a hammer to drive-in a nail. Without the hammer, this would not be possible.

• However, that is only one aspect of testing tools. Most important is the selection of proper testing tool or set of tools, to test in a given situation. Selection of proper testing tool affects the effectiveness and efficiency of testing, as well.

• Thus, if a testing tool does not match the testing purpose well, then instead of helping in the testing, the testing tool can be counter productive. It may not help reducing the testing effort or speeding it, but it may add more work to testing or may not be able to detect satisfactory number of defects. This also can have serious impact on waste of time and over spending.

• E.g. for a very large nail to be driven into a wood, a relatively smaller hammer would increase efforts and chances of accidents, and causing frustrations rather than any benefits. Also, otherwise, for a small size nail, one must not use a heavy hammer.

• The testing tools can make desirable difference and efficiency achieved depends on selecting the proper tool. Hence, selection of tools is a very important decision.

• Many of the testing tools have not been in use because of the following reasons:-

– Some of them are highly specialized (simulation)

– The cost of use is very high (symbolic execution)

• The disciplined manual testing techniques have been most useful. They can be used without too much of capital investment. However, they require firm commitment and consistently disciplined testing practices.

• Careful planning, clearly stated testing objectives, precisely defined tools, good management, organized record keeping, and strong commitment are necessary for successful testing.

Selecting and Using the Test Selecting and Using the Test Tools Tools

There are four steps as follows:-• Step 1. Matching the tool to its use• Step 2. Selecting a tool appropriate

to the life cycle phase• Step 3. Matching the tool to the skill

level of the tester• Step 4. Selecting an affordable tool

Step 1. Matching the tool to its use• A tool is to accomplish a task. Therefore,

more suitable the testing tool to the meet the goal of the testing task, the more efficient the testing process can be.

• A wrong testing tool not only decreases the efficiency, but there is no assurance that the testing objectives will be achieved.

• The testing technique should be selected prior to the tool.

• Matrix test tool Vs test techniques can be used to narrow the choice of more appropriate tools.

• Following testing tools are useful in all the types of testing use: Checklist, Error Guessing, and Inspections.

• For meeting the Error Handling purpose of testing, the following testing techniques are useful: Acceptance Test Criteria, Boundary Value Analysis, Cause Effect Graphing, Checklist, Correctness Proof, Data Dictionary, Desk Checking, Error Guessing, Exhaustive Testing, Flowcharts, Inspection, Instrumentation, Integrated Test Facility, System Logs, Test Data, Test Data Generator

• For testing purpose of Controls, following testing tools can be used:- Cause-Effect Graphing, Checklist, Code Comparison, Confirmation/Examination, Correctness Proof, Desk Checking, Error Guessing, Exhaustive Checking, Fact Finding, Flowchart, Instrumentation, Integrated Test Facility, Risk Matrix, Test Data, Test Data Generator, walkthroughs.

• For testing Compliance, following techniques are useful: Acceptance Test Criteria, Checklist, Compiler based Analysis, Confirmation/Examination, Correctness Proof, Design Reviews, Error Guessing, Fact Finding, Inspections, Peer Review, System Logs.

Step 2. Selecting a tool appropriate to the life cycle phase

• The testing type varies with the phase in which the testing occurs. Just as the methods change, so do the tools. So, it becomes necessary that the testing tool selected is appropriate for the life cycle phase in which it is used.

• As the life cycle phases progress, the tools shift from manual methods of testing to more automated methods.

• In fact, some of the most productive testing (testing that detects and removes significantly potential defects entered into the system and those which may be very costly later) can occur during the early phases of the life cycle using manual tools.

• During the Requirements Phase, the following testing tools can be used:- Checklist, Confirmation/Examination, Desk Checking, Error Guessing, Fact Finding, Flowchart, Inspections, Modeling, Peer Review, Risk Matrix, Scoring, Walkthroughs.

• During the Design Phase, the following testing tools can be used:- Cause-Effect Graphing, Checklist, Confirmation/Examination, Correctness Proof, Design-based functional testing, Design Reviews, Desk Checking, Error Guessing, Executable Specs, Fact Finding, Flow Chart, Inspections, Modeling, Peer Review, Risk Matrix, Scoring, Walkthroughs

• During the Programming Phase, the following testing tools can be used:- Boundary Value Analysis, Cause-Effect Graphing, Checklist, Compiler-based Analysis, Coverage-based Metric Testing, Data Flow Analysis, Desk Checking, Error Guessing, Fact Finding, Flow chart, Inspections, Peer Review, Symbolic Execution, Test Data, Tracing, Walkthroughs

• During the Testing Phase, the following testing tools can be used:- Acceptance Test Criteria, Boundary Value Analysis, Checklist, Compiler-based Analysis, Complexity-based Metric testing, Confirmation/Examination, Correctness Proof, Coverage-based metric testing, Data Dictionary, Design-based Functional testing, Disaster Test, Error Guessing, Exhaustive Testing, Fact Finding, Inspections, Instrumentation, Parallel Simulation, Peer Review, Snapshot, System Logs, Test Data, Test Data Generator, Utility Programs, Volume Testing

• During the Installation Phase, the following testing tools can be used:- Acceptance Test Criteria, Checklist, Confirmation/Examination, Error Guessing, Fact Finding, Inspections, Instrumentation, Parallel Simulation, Peer Review, System Logs, Utility Programs, Volume Testing

• During the Maintenance Phase, the following testing tools can be used:- Checklist, Code Comparison, Desk Checking, Disaster Test, Error Guessing, Fact Finding, Inspections, Instrumentation, Integrated Test Facility, Peer Review, SCARF, System Logs, Test Data, Test Data Generator, Tracing, Utility Programs

• The tester skills and overall understanding of the testing tasks is important in selecting the tool.

• User Skills: Requires individuals to have– In-depth knowledge of the application – In-depth knowledge of the business purpose

for which the application is being developed. – General management skills used to achieve

the mission of user area, – Knowledge of identifying and dealing with

user problems.

• Programming Skills: Requires following skills:-Understanding of computer concepts, Flowcharting

– Programming skills in the languages used by the organization

– Debugging and documenting computer programs

• System Skills: Requires the following skills:-– The ability to translate user requirements into

computer system design specifications– Flowcharting, Problem Analysis, Design

Methodologies, Computer Operations, General Business Skills, Error identification, and analysis in automated applications and project management.

– Normally posses good programming skills.

• Technical Skills: These are as follows:-

– Understands a highly technical specialty

– Can exhibit a reasonable performance at work at that specialty

– E.g. systems programming, database administration, statistics, accounting and operating software packages

Step 4. Selecting an affordable tool:• Affordability of a tools deals with its

cost and expected time of completion of testing using that tool. Even the most desirable, but very costly and/or time-consuming tool, may not be recommended.

• Some times one may have to buy special hardware, software etc. for using a tool. This may add to costs.

• The prices may vary depending upon the features provided, reliability of the tools and above all vendor policy.

• The costs could be classified as High, Medium and Low, based on how much do they vary for the same set/ range of features.

• A tool may be time consuming, if it requires specific training, and/or it requires additional steps of actions from a tester or it has any unavoidable manual interventions involved, etc. Also, if the tool documentation is not of adequate quality, the tester may take longer to complete a testing task.

Appointing Managers for Appointing Managers for Testing ToolsTesting Tools

• The objectives are as follows:-– Assign someone accountability to oversee

the tool usage.– Create a source of competency about how to

use the tool– Provide a training ground for future

managers

• Managing a tool involves budgeting, planning, training, and related managerial responsibilities.

• There are three steps in appointing a tool manager, as follows:-– Step-1: Tool Manager Selection– Step-2: Assign the Tool Manager Duties– Step-3: Limiting the Tool Manager’s

Tenure

Step 1: Tool Manager Selection• Tool Manager Selection should look for

the following skills in the candidate: Organization, Training abilities, Tool proficiency, Managerial Qualities such as planning, organizing, directing, controlling, etc.

• In addition to the tool manager, an assistant tool manager should also be named for each tool.

• This individual will not have any direct managerial responsibilities, but will serve as a backup for the tool manager.

Step-2: Assign the Tool Manager Duties• The Tool Managerial Duties are as

follows:-– Assisting colleagues in the use of tools

• The tool manager should be available to assist other staff members in the use of the tool.

• This is normally done using a concept called ‘hot-line’.

– Tool training• The initial tool training normally comes from

the vendor.• However, additional tool training is the

responsibility of the tool manager.

– Tool Vendor Contact• The tool manager would be the official contact

for the tool vendor.• Questions from staff regarding the use of the

tool that can only be answered by the vendor should be funneled through the tool manager to the vendor.

– Annual Tool Plan• The tool manager should develop an annual

tool plan complete with planned usage, schedule, and resources needed to effectively utilize the tool.

– Installing Tool Upgrade • As vendors issue new versions of the tool,

the tool manager is responsible for ensuring that those upgrades are properly incorporated and that the involved parties are made aware and trained, if necessary.

– Preparing Annual tool Report• At the end of year, or planning period, the

tool manager should prepare for IT management an overview of the use of the tool during the year.

• This will require the tool manager to maintain statistics on tool usage, problems, costs, upgrades and so forth.

– Determining timing of tool replacements• The tool manager, being responsible for a

specific software tool, should also be responsible for determining when the tool is no longer effective or when better tools can be acquired to replace it.

Step-3: Limiting the Tool Manager’s Tenure• It is recommended that an individual

serve two years as a manager for a specific tool.

• Bringing in a new tool manager every two years tends to revitalize the use of that tool in the organization.

• The tool manager can be transferred to manage another tool.

• In instances where tools are highly specialized, very complex, it may be desirable to keep an individual manager for longer than a two- year period.