Validating and Improving Test-Case Effectiveness Author: Yuri Chernak Presenter: Lam, Man Tat.

Validating and Improving Validating and Improving Test-Case EffectivenessTest-Case Effectiveness

Author: Yuri ChernakAuthor: Yuri Chernak

Presenter: Lam, Man TatPresenter: Lam, Man Tat

AgendaAgenda

Some definition that need to be Some definition that need to be understandunderstand

How verification and validation is doneHow verification and validation is done

Overview of how in process validation of Overview of how in process validation of test cases is donetest cases is done

Case-studyCase-study

ConclusionConclusion

Functional testingFunctional testing

By focusing on a system's functionalityBy focusing on a system's functionality

Looking for as many defects as possibleLooking for as many defects as possible

Test plan documents and test case Test plan documents and test case specifications are important deliverablesspecifications are important deliverables

Test CasesTest Cases

Critical for effective testingCritical for effective testing

Testers use test-case specifications does Testers use test-case specifications does not guarantee that systems are sufficiently not guarantee that systems are sufficiently testedtested

VerificationVerification

First verify test case specifications at the First verify test case specifications at the end of the test design phaseend of the test design phase

Reviews or inspections – evaluate test Reviews or inspections – evaluate test case specifications for their correctness case specifications for their correctness and completenessand completeness

Test cases that passed verification could Test cases that passed verification could have weak failure-detecting abilityhave weak failure-detecting ability

ValidationValidation

Can process as soon as testers have Can process as soon as testers have executed all test casesexecuted all test cases

Determines whether the test cases were Determines whether the test cases were sufficiently effective in finding defectssufficiently effective in finding defects

If test cases effectiveness has not proved If test cases effectiveness has not proved satisfactory, it is not too late to analyze the satisfactory, it is not too late to analyze the reasons and correct the test processreasons and correct the test process

Test case effectiveness metric introducedTest case effectiveness metric introduced

Test case effectiveness metricTest case effectiveness metric

Mainly use for online client server systemsMainly use for online client server systems

Certain number of defects are always Certain number of defects are always found as a side effectfound as a side effect

Assume that the more defects test cases Assume that the more defects test cases find, the more effective they arefind, the more effective they are

Define as the ratio of defects found by test Define as the ratio of defects found by test cases (Ntc) to the total number of defects cases (Ntc) to the total number of defects (Ntot) reportes during the test cycle(Ntot) reportes during the test cycle

Test case effectiveness metric Test case effectiveness metric Cont’Cont’

TCE = Ntc / Ntot *100%TCE = Ntc / Ntot *100%

TCE metric evaluates test cases from the test TCE metric evaluates test cases from the test cycle perspective, which provides in process cycle perspective, which provides in process feedback to the project team on how well a test feedback to the project team on how well a test suite has worked for testerssuite has worked for testers

Compare the actual TCE for a given project with Compare the actual TCE for a given project with a baseline value (such as previous successful a baseline value (such as previous successful project, author experience suggests 75 %)project, author experience suggests 75 %)

Test case effectiveness metric Test case effectiveness metric Cont’Cont’

If the TCE value is at baseline or above, If the TCE value is at baseline or above, we can conclude that the test cases have we can conclude that the test cases have been sufficiently effective in test cyclebeen sufficiently effective in test cycle

If the TCE value falls below the baseline, If the TCE value falls below the baseline, the higher is the risk of user dissatisfactionthe higher is the risk of user dissatisfaction

Improving test case effectivenessImproving test case effectiveness

If in process validation finds test case If in process validation finds test case effectiveness to be less than acceptable, effectiveness to be less than acceptable, the project team should analyze the cause the project team should analyze the cause and identify areas that need to be improveand identify areas that need to be improve

Author’s proposed improvement Author’s proposed improvement framework was based on IBM’s “Test case framework was based on IBM’s “Test case escape”escape”

IBM’s “Test case escape”IBM’s “Test case escape”

Define as “product defects that a particular Define as “product defects that a particular test failed to find, but which were found in test failed to find, but which were found in a later test, or by a customer [in a later test, or by a customer [in production]production]

Author’s proposed improvement Author’s proposed improvement frameworkframework

Define as software defects that a given Define as software defects that a given suite of test cases failed to find but that suite of test cases failed to find but that were found as a side effect in the same were found as a side effect in the same test cycletest cycle

Rather than compare the Ntot with the Rather than compare the Ntot with the whole processwhole process

Author’s proposed improvement Author’s proposed improvement frameworkframework

1- Understand and document the test 1- Understand and document the test process used by the project teamprocess used by the project team

2- Make assumptions about factors that 2- Make assumptions about factors that affect test casesaffect test cases

3- Gather defect data3- Gather defect data

4- Identify main factors4- Identify main factors

5- Implement corrective actions5- Implement corrective actions

Understand, document the test Understand, document the test processprocess

Test planning – definition of the scope, Test planning – definition of the scope, objectives, and approach to testingobjectives, and approach to testing

Test design – the design of the test casesTest design – the design of the test cases

Test preparation and execution – Test preparation and execution – preparation of the test environment preparation of the test environment executing test cases and finding defectsexecuting test cases and finding defects

Test evaluation and improvement – Test evaluation and improvement – analyzing the results of testinganalyzing the results of testing

Make assumptionMake assumption

Once understands and documents the test Once understands and documents the test process, the project team should analyze process, the project team should analyze each phase and identify factors that can each phase and identify factors that can affect test case effectivenessaffect test case effectiveness

Lets look at some exampleLets look at some example

Example of making assumptionExample of making assumption

Test planning – if the functional Test planning – if the functional specifications do not completely define specifications do not completely define functional features, the test plan document functional features, the test plan document will not be complete either. So, the test will not be complete either. So, the test cases will not completely cover a system’s cases will not completely cover a system’s functionalityfunctionality

Example of making assumption 2Example of making assumption 2

Test design – a common example could Test design – a common example could be lack of negative test cases (such as be lack of negative test cases (such as abnormal conditions by using either invalid abnormal conditions by using either invalid data input or the wrong user actiondata input or the wrong user action

Test design – test case specifications Test design – test case specifications could be simply incorrectcould be simply incorrect

Factors affecting test case Factors affecting test case effectivenesseffectiveness

Gather defect data and perform Gather defect data and perform analysisanalysis

Defect tracking systemDefect tracking system

Able to identify which defects were found Able to identify which defects were found as result of test cases and which were as result of test cases and which were found as a side effectfound as a side effect

Once selected, defects should be Once selected, defects should be classified according to one of the factors classified according to one of the factors based on the analysisbased on the analysis

Test Case escape classification Test Case escape classification logiclogic

Gather defect data and perform Gather defect data and perform analysis cont’analysis cont’

Analysis is used to evaluate each test Analysis is used to evaluate each test case and understand why the test suite case and understand why the test suite missed the corresponding defect during missed the corresponding defect during the test executionthe test execution

Example: A lack of negative test cases in Example: A lack of negative test cases in is a common cause of missed defects. is a common cause of missed defects. Can you see which category it will be Can you see which category it will be under?under?

Identify the main factorsIdentify the main factors

At this point, all test case escapes have At this point, all test case escapes have been classified according to their been classified according to their respective causesrespective causes

Build a “Pareto chart” which displays Build a “Pareto chart” which displays frequency bars in descending order, frequency bars in descending order, convenient for analyzing (will be an convenient for analyzing (will be an example later)example later)

Implement corrective actionsImplement corrective actions

After identification of the main causes of After identification of the main causes of test case escapes, the project team should test case escapes, the project team should implement corrective actions and repeat implement corrective actions and repeat the test executionthe test execution

Example of corrective actionsExample of corrective actions

Rework functonal specifications and test Rework functonal specifications and test case specificationcase specification

Use a traceability matrix to ensure Use a traceability matrix to ensure complete coverage of business rulescomplete coverage of business rules

Use checklists to design test case Use checklists to design test case specificationspecification

Implement training of testers on test Implement training of testers on test executionexecution

Case studyCase study

Banking application intended for external Banking application intended for external clientsclients

Project team consisted 10 developers and Project team consisted 10 developers and 3 testers3 testers

183 defects reported183 defects reported

71 found by test case escapes71 found by test case escapes

112 found by test case execution112 found by test case execution

Case study cont’Case study cont’

TCE = 61%TCE = 61%

Lower than the acceptable level of 75%Lower than the acceptable level of 75%

Project team performed the test process Project team performed the test process improvement according to the frameworkimprovement according to the framework

Pareto chart created for analysisPareto chart created for analysis

Example of Pareto chartExample of Pareto chart

Case study cont’Case study cont’

Analysis of the distribution showed Analysis of the distribution showed incomplete test design and incomplete incomplete test design and incomplete functional specification to be the main functional specification to be the main factors causing missed defects.factors causing missed defects.

Project team correcting and completing Project team correcting and completing the test design and functional specificationthe test design and functional specification

Case study cont’Case study cont’

As a result, 48 additional defects found As a result, 48 additional defects found before releasing the productbefore releasing the product

By the end of the second month the By the end of the second month the number of production, only 23 defects was number of production, only 23 defects was foundfound

Indicated sufficient effectiveness of test Indicated sufficient effectiveness of test progress, user should be satisfied with the progress, user should be satisfied with the system qualitysystem quality

ConclusionConclusion

This technique intended to give project This technique intended to give project teams better visibility into test process teams better visibility into test process effectiveness effectiveness beforebefore their system are their system are released into productionreleased into production

Help evaluate each test case and Help evaluate each test case and understand why the test suite missed the understand why the test suite missed the corresponding defect during test executioncorresponding defect during test execution

ConclusionConclusion

Help testers revise and improve the test Help testers revise and improve the test suitesuite

In other word, it helps testers find In other word, it helps testers find additional defects and deliver a better additional defects and deliver a better software productsoftware product

If my presentation did not If my presentation did not put you into sleep and you put you into sleep and you have a question… have a question…

Now its time!!!Now its time!!!