Test suite minimization

Test Suite Minimization

A Concept Analysis Inspired Greedy Algorithm for Test Suite

Minimization

http://llvm.org/pubs/2005-09-PASTE-GreedySuiteMinimization.pdf

Motivation

• Large softwares typically have large number of tests and their complete runs take several hours

• For minimal changes a complex software undergoes daily, a lot of these tests are only testing what’s already tested

• These tests are run in nightly release process, for pre-check in requirements and check-ins consuming time, energy and dollars

Problem

• How do we know which tests to run given a software change?– Test coverage information for each tests can

answer this question

• But we can do better. Do we need to run all the tests according to the test coverage?– But this problem is hard. NP-hard.

Problem in an examplepublic boolean isWeekend(int day) { if(day > 5) { return false; //R1 } else { return true; //R2 }}

Test/Req R1 R2 R3

T1 X X

T2 X

T3 X X

//T1public void testIsWeekend(int ) {

AssertFalse(isWeekend(Friday.index))//5AssertTrue(isWeekend(Saturday.index))//6

}//T2public void testIsBefore() {

AssertTrue(isBefore(Monday.index, Friday.index));

}//T3 public void testMonday() { AssertFalse(isWeekend(Monday.index))//5

AssertTrue(isBefore(Monday.index, Friday.index));

}

public boolean isBefore(int day1, int day2) { return day1 < day2 //R3}

Problem in an example• Let us consider two functions:

– IsWeekend: If days are numbered from Monday to Sunday, every day numbered less than 5 is a weekday, otherwise it’s a weekend. These are marked as requirements R1 and R2 respectively

– IsBefore checks if day1 comes before day2 and is our requirement R3

• Now also consider 3 test cases– T1 tests if Friday and Saturday is a Weekend day– T2 tests if Monday comes before Friday– T3 tests if Monday is weekend and comes before Friday.

• Let us also create a table to map which tests which of the requirements. Now from the table we can see that if I run T1 and T2, the T3 is redundant. If Friday is a weekday and Monday comes before Friday then Monday has to be a weekday

Problem Definition

• Given a test-requirement matrix where– A test tests all the requirements listed against it– A requirement may be satisfied by any test listed

against it

• Find the minimum number of tests which can test all the requirements

Outline

• Problem is NP Hard – Reducible from Set Cover

• Heuristics used to solve this problem– Greedy– HGS Algorithm

• Delayed Greedy Algorithm using Concept Analysis (this algorithm)

NP Hard

If there are n different things, how many subsets of n can I create?

2n

Therefore, if there are n tests, how many tests should I run to have enough coverage?

Convention

• In the following slides, circles denote requirements and the rectangles denote tests– Rectangles span multiple circles. This implies that a

test can test multiple requirements

• A requirement can be completely tested by any test in which it appears.

• Red rectangles represent selected tests

Greedy Algorithm

• Assuming the given requirements and tests, Greedy algorithm tries to pick the smallest set of test cases

• To achieve this, Greedy algorithm choses a test case which covers the maximum number of requirements. It then removes the covered requirements and test from consideration

• The algorithm ends when no requirements are left

Greedy

Choose the set with max uncovered element

Greedy

Choose the set with max uncovered elements

Greedy

Choose the set with max uncovered elements

Greedy

The final solution of Greedy algorithm

Optimal Solution

However, the optimal solution is given below

Problem with Greedy?

• The algorithm missed the optimal solution. Why?

• Choice about picking the first set was made too early!

HGS Algorithm

• Choose requirements which are covered by k tests cases starting with k = 1

• Pick the test case which covers the most requirements. If more than one tests qualify then pick randomly

• Remove the requirements by the picked test. Continue with the rest

HGS AlgorithmPick requirements which are tested by k tests. k = 1

HGS AlgorithmPick requirements which are tested by k tests. k = 1

HGS AlgorithmPick requirements which are tested by k tests. k = 2

HGS AlgorithmPick requirements which are tested by k tests. k = 2

HGS AlgorithmPick nodes which are present in k subsets starting with k = 3. Pick test T’ arbitrarily among T and T’

T

T’

HGS AlgorithmPick nodes which are present in k subsets starting with k = 3

HGS AlgorithmNow find the remaining requirement (shown as red circle). Pick a test randomly which satisfies it

HGS AlgorithmPick nodes which are present in k subsets starting with k = 3. Picked a test randomly among the three

HGS AlgorithmThis makes one of the earlier chosen tests (dark blue border) redundant

Problem with HGS?

Implications could not be derived while making choices

Problems with other Heuristics

• Choice about picking the first set was made too early

• Implications could not be derived

How Concept Analysis works?

Find and eliminate all the dependencies before choosing subsets

How Concept Analysis works?

t3 tests requirements r2 and r5. While t5 tests r5 only. Therefore if t3 is run, t5 can be ignored

How Concept Analysis works?

If r4 is tested, then r1 is tested also. Similarly, r6 and r3

How Concept Analysis works?

Remove r1, r3 and t5

How Concept Analysis works?

Similarly, t1 can be removed in favor of t3

How Concept Analysis works?

Now apply Greedy minimum set cover algorithm on that rest to find the minimum number of tests to be run to test all the requirements

How Concept Analysis works?

• Minimization of test-requirement matrix to remove redundancies

• Delayed sub-set selection to avoid picking sub-optimal tests

• Delayed Greedy algorithm performs at least as well as the Greedy algorithm

Concept Analysis - Details

• Lattice• Concepts• Lattice of Concepts• Labels• Strong concept• Algorithm

Latticehttps://en.wikipedia.org/wiki/Lattice_(order). For example show below is a lattice of subsets of a set {a,b,c}

Concept• Consider an ordered pair (t, r) where t is the set of tests

and r is a set of requirements

• Such a set is called maximal grouping if t(or r) is the maximal set of tests(or requirements) related to all requirements(or tests) in r(or t)– i.e. t = ∩iri where ri r and r∈ i is a collection of tests from

reqruiements-test table.– i.e. r = ∩iti where ti t and t∈ i is a collection of tests from

requirements-test table.

• Then this maximal grouping (t, r) is called Concept

Lattice of Concepts

Lattice of Concepts• The lattice is ordered from top to bottom for tests i.e. at

each level from top to bottom, number of tests in the concept increases

• And from bottom to top for requirements i.e. at each level from bottom to top, number of requirements in the concept increases

• Not all the points of the concept lattice have a maximal requirement set and therefore, not all of them will be found on the CA lattice e.g. {t5}

Labels

• Lattice points are labeled with test case (requirement) name if it is the smallest (according to the partial order) lattice point in which it appears e.g– t3 appears on c4 and t5

appears on c8. – Similarly, r1 appears on c5

and r4 appears on c1

Strongest concept

• Those concepts which are immediately above the bottom– e.g. c1, c2, c3, c4

are strongest concept of this lattice

Object implicationPick the overlapping test cases (e.g. t3, t5) and remove the one which is a subset of the other. From the lattice, these are labeled concepts ci and cj where ci < cj

Attribute ImplicationPick the requirements which overlap and remove the one which is a superset of the other. From the lattice, these are labeled concepts ci and cj where ci < cj

Object Reduction - 2

Owner Reduction

Empty Table

Pick the strongest concepts from the reduced lattice

Delayed Greedy Algorithm

• Repeat until table is empty– Repeat until table changes• Object Implication• Attribute Implication• Owner Reduction and choose strongest concepts

– One step of Greedy

Experiments

• No such requirement-test matrix exists for most softwares. How can this approach be tested?

• Requirements: Branches and def-use pairs (using LLVM)

• Tests: Identify tests which hit these requirements

Summary

• Minimization of test-requirement matrix to remove redundancies

• Delayed sub-set selection to avoid picking sub-optimal tests

• Greedy Algorithm on the remaining table

• Never performs worse than other approximation algorithms.

• Minimize the number of tests to run for a given set of requirements

• The problem is NP hard. However, calculation needs to be done only once.

• Given a minimized requirement-test matrix, changes to a requirement will need only limited tests to be run

Problem Solution – Delayed Greedy Algo