Test execution may_04_2006

Test ExecutionTest Execution& Reporting& Reporting

AgendaAgenda

General Test procedureGeneral Test procedure Executing tests Executing tests

Set up systems and software Set up systems and software

Execute test cases Execute test cases Verifying and analyzing results Verifying and analyzing results ReportingReporting

General Test ProcedureGeneral Test Procedure

A Single test normally includes three steps:A Single test normally includes three steps: PlanningPlanning Execution Execution EvaluationEvaluation

Tester Plan testingExecute testing

Evaluate testing

General Test ProcedureGeneral Test Procedure

Plan testing

Execute testing

Evaluate testing

If I click OK, without input into text field, the application will probably crash.

Let’s see if it does

It sure did!

PlanningPlanning It is the first step to determine what should be tested. Also It is the first step to determine what should be tested. Also

referred as test case design.referred as test case design.

Design the test cases (inputs and outputs) used to test the Design the test cases (inputs and outputs) used to test the system:system:

Identify the test case (Identify the test case (What has to be testedWhat has to be tested).).

Specify the test activities( Specify the test activities( How to proceedHow to proceed).).

Indicate how test results should be defined (Indicate how test results should be defined (Correct outcomeCorrect outcome).).

ExecutionExecution

It’s Show Time!It’s Show Time!

Execution of a test is a three step Execution of a test is a three step

Process:Process: Preparing the Test InfrastructurePreparing the Test Infrastructure Conduct Test.Conduct Test. Verify Results and Report Results.Verify Results and Report Results.

Prepare Test InfrastructurePrepare Test Infrastructure

The prepare Test Infrastructure task The prepare Test Infrastructure task ensures that all the components such as ensures that all the components such as technical resources, human resources, technical resources, human resources, configuration, data, and test design configuration, data, and test design specification are in place for successful specification are in place for successful execution of the test.execution of the test.

The key to setting up the testing The key to setting up the testing environment is to establish a set of environment is to establish a set of standards that everyone in the project can standards that everyone in the project can adhere to. adhere to.

Prepare Test InfrastructurePrepare Test Infrastructure

Establish a set of standards that the Establish a set of standards that the organization follows.organization follows.

The key to setting up the testing The key to setting up the testing environment is to establish a set of environment is to establish a set of standards that everyone in the standards that everyone in the project can adhere to. project can adhere to.

Prepare Prepare Test InfrastructureTest Infrastructure

Components of the testing Components of the testing environment include:environment include:

Supporting software

An inventory of hardware

The code under test

A workbook or execution logs


Supporting softwareSupporting software

Includes operating systems, middleware, Includes operating systems, middleware, system utilities, application, patches and so system utilities, application, patches and so on.on.

Supporting software is stored on CD or Supporting software is stored on CD or available on a shared file system.available on a shared file system.

All supporting software will need to be All supporting software will need to be installed in the environment in the exact installed in the environment in the exact sequence required by the components.sequence required by the components.


Supporting softwareSupporting software

Detailed installation steps will need to be provided Detailed installation steps will need to be provided to you as part of the implementation of the test to you as part of the implementation of the test cases.cases.

When installing multiple supporting software When installing multiple supporting software elements, the order of installation is critical to the elements, the order of installation is critical to the proper functioning of the environment. proper functioning of the environment.


An inventory of hardware:An inventory of hardware:

Properly setting up the test-execution environment Properly setting up the test-execution environment requires to have access the inventory of hardware requires to have access the inventory of hardware that is needed, along with any special instructions that is needed, along with any special instructions for configuring the hardware.for configuring the hardware.

It is important to ensure that you execute the test It is important to ensure that you execute the test case in the exact hardware configuration identified case in the exact hardware configuration identified by the test cases.by the test cases.

Tester must confirm that the hardware meets all Tester must confirm that the hardware meets all requirements and specifications such as those for requirements and specifications such as those for memory, disk space, processor speeds and so on.memory, disk space, processor speeds and so on.


Code under test:Code under test:

Code under test, also referred to as a driver or a Code under test, also referred to as a driver or a build is the packaged new code that is ready to be build is the packaged new code that is ready to be installed and tested.installed and tested.

Properly preparing the environment for test-case Properly preparing the environment for test-case execution also requires that you have access to the execution also requires that you have access to the code under test.code under test.

Code is stored in a well-known place with naming Code is stored in a well-known place with naming conventions that indicate the exact driver or build conventions that indicate the exact driver or build levels.levels.

Detailed procedures need to be in place to enable Detailed procedures need to be in place to enable you to install the code accurately and efficiently.you to install the code accurately and efficiently.


Workbook or execution logs:Workbook or execution logs:

Workbooks and execution logs are used to record Workbooks and execution logs are used to record the test-execution activity.the test-execution activity.

Workbooks are used when manually running tests, Workbooks are used when manually running tests, while execution logs are used to capture execution while execution logs are used to capture execution of tests run programmatically. of tests run programmatically.

Store the log files in a well-known place with Store the log files in a well-known place with naming conventions to indicate the date captured naming conventions to indicate the date captured and the build level being tested.and the build level being tested.

Conduct TestConduct Test

Execute Test caseExecute Test case The purpose of the task is to The purpose of the task is to

implement the test execution implement the test execution plan, produce test results, and plan, produce test results, and compare between the expected compare between the expected test results and actual test test results and actual test results.results.

After set up the environment, After set up the environment, ready to execute the tests.ready to execute the tests.

Execute each test exactly as Execute each test exactly as defined in the test-case defined in the test-case implementation.implementation.

Execute Test case – The processExecute Test case – The process

Software to be tested

Test Case

Output

Verifier

Test CaseGeneration

Testing processTesting process

Design testcases

Prepare testdata

Run programwith test data

Compare resultsto test cases

Testcases

Testdata

Testresults

Testreports

InputInput

Actual OutputActual Output

Comparison Process

Comparison Process

Test case : Specifies• Inputs + pre-test state

of the software• Expected results (outputs and state)

Test case : Specifies• Inputs + pre-test state

of the software• Expected results (outputs and state)

Testing policesTesting polices

Only exhaustive testing can show a program Only exhaustive testing can show a program is free from defects. However, exhaustive is free from defects. However, exhaustive testing is impossible,testing is impossible,

Testing policies define the approach to be Testing policies define the approach to be used in selecting system tests:used in selecting system tests:• All functions accessed through menus All functions accessed through menus

should be tested.should be tested.• Where user input is required, all functions Where user input is required, all functions

must be tested with correct and incorrect must be tested with correct and incorrect input.input.

Testing guidelinesTesting guidelines

Testing guidelines are hints for the testing Testing guidelines are hints for the testing team to help them choose tests that will team to help them choose tests that will reveal defects in the systemreveal defects in the system

• Choose inputs that force the system to Choose inputs that force the system to generate all error messages.generate all error messages.

• Design inputs that cause buffers to overflow.Design inputs that cause buffers to overflow.• Repeat the same input or input series several Repeat the same input or input series several

times.times.• Force invalid outputs to be generated.Force invalid outputs to be generated.• Force computation results to be too large or Force computation results to be too large or

too small.too small.

Verify the existence of the DefectVerify the existence of the Defect

Verify that the element under test, Verify that the element under test, the execution environment and the the execution environment and the test case-implementation are test case-implementation are consistent with the test case designconsistent with the test case design

Verify and confirm the discrepancy Verify and confirm the discrepancy between the expected test results between the expected test results and actual test results.and actual test results.

Verify and Analyze ResultsVerify and Analyze Results

Analyze the DefectAnalyze the Defect The purpose of the task is to analyze The purpose of the task is to analyze

and resolve variances encountered and resolve variances encountered during test execution. during test execution.

Each variance is analyzed to Each variance is analyzed to determine the cause. determine the cause.

Analysis may indicate that the cause Analysis may indicate that the cause of the variance is a defect, or that a of the variance is a defect, or that a change request is required.change request is required.


Analyze the DefectAnalyze the Defect Investigate further and Investigate further and

gather as much gather as much information as possible information as possible about the symptoms. about the symptoms.

Analyze failures by varying Analyze failures by varying conditions, options and conditions, options and settings to gauge the settings to gauge the severity of the defect severity of the defect


Analyze failures by varying conditions, options and settingsAnalyze failures by varying conditions, options and settings

Change the Change the conditionsconditions under which an error occurred by under which an error occurred by changing what you do. As an example, if the problem seems changing what you do. As an example, if the problem seems related to entering data very quickly, try slowing your rate of related to entering data very quickly, try slowing your rate of input. input.

Change the Change the options and settingsoptions and settings of the program under of the program under test. As an example, if the problem seems related to desktop test. As an example, if the problem seems related to desktop screen-resolution settings, try varying the settings. screen-resolution settings, try varying the settings.

Change the Change the software and hardware environmentsoftware and hardware environment. For . For example, if the defect seems related to a low-end hardware example, if the defect seems related to a low-end hardware configuration, try using the lowest configuration specified in configuration, try using the lowest configuration specified in the requirements. the requirements.

Change the Change the datadata used by the program. As an example, if the used by the program. As an example, if the defect seems related to strings exceeding a certain number defect seems related to strings exceeding a certain number of characters in length, create a second set of test data to of characters in length, create a second set of test data to check the hypothesis. check the hypothesis.


Documenting your Documenting your execution resultsexecution results

Test logs to record Test logs to record test execution test execution

Create a test Create a test incident report for incident report for problems problems

ReportingReporting

Run Test

Test Logs

ERROR REPORT

TEST CASES

Verify & Analyze Result

Bug Tracking

Effective defect Effective defect reportsreports

Convey the proper Convey the proper message, gets the message, gets the job done and job done and simplifies the simplifies the process for process for everyone. everyone.

Can Pig Ride Can Pig Ride technique to write technique to write effective incident effective incident reports.reports.

ReportingReporting

C Condense: Say it clearly but briefly

AAccurate: Is it a user error or set-up problem?

System problem? Application problem? Test-case problem?

N Neutral: Just the facts; no zingers, no humor, no emotion, no sarcasm

P Precise: Explicitly, what is the problem?

I Isolate: What was done to isolate the problem?

G Generalize: What was done to understand how general the problem is?

R Re-create: What is essential to re-create this problem?

I Impact: What is the impact if the bug were to surface in a customer environment

D Debug: What will the developer need to debug this problem?

E Evidence: What will prove the existence of the error?

Test-Defect report Test-Defect report The IEEE Std. for Software Test Documentation for The IEEE Std. for Software Test Documentation for

Test Defect template offers the following fields. Test Defect template offers the following fields.

ReportingReporting

• Incident Summary report IdentifierSpecify the unique identifier assigned to this test-defect

report.

•Incident SummarySummarize the incident. Identify the test items involved,

indicating their version and revision level. Supply references to the appropriate test-procedure specification, test-case specification and test log.

Incident DescriptionProvide a description of the incident that includes the following items:

Inputs Expected results Actual results Date and time Procedure steps Environment Attempts to repeat Testers Observers

Related activities and observationsInclude any of these that may help to

isolate the correct the cause of the incident.

Test-Defect report Test-Defect report The IEEE Std. for Software Test Documentation for The IEEE Std. for Software Test Documentation for

Test Defect template offers the following fields. Test Defect template offers the following fields.

ReportingReporting

ResolveResolve True defects - referred to the Defect True defects - referred to the Defect

Management process. Management process. Rated for severity and a priority Rated for severity and a priority Change Requests - referred to the Change Change Requests - referred to the Change

Management processManagement processPrioritizes the revision of requirements, Prioritizes the revision of requirements, the specification, design and construction the specification, design and construction of each item. of each item. Changed items reenter the test process Changed items reenter the test process

ReportingReporting

Test execution may_04_2006

Education

test results

test casesexecute test

single test

test designconfiguration

test infrastructurepreparing

test activitiesspecify

test infrastructure

test case design