Generating pairwise combinatorial test set using artificial parameters and values

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Generating Pairwise Combinatorial Test Set Using Artificial Parameters and

Values

Mohammed I. Younis School of Electrical and Electronic Engineering

Universiti Sains Malaysia Engineering Campus

14300 Nibong Tebal, Penang, Malaysia

[email protected]

Kamal Z. Zamli School of Electrical and Electronic Engineering

Universiti Sains Malaysia Engineering Campus

14300 Nibong Tebal, Penang, Malaysia

[email protected]

Nor Ashidi Mat Isa School of Electrical and Electronic Engineering

Universiti Sains Malaysia Engineering Campus

14300 Nibong Tebal, Penang, Malaysia

[email protected]

Abstract In order to meet market demands for quality

software products, software engineers are increasingly under pressure to test more lines of codes. To maintain acceptable test coverage, software engineers need to consider a significantly large number of test set. Many combinations of possible input parameters, hardware/software environments, and system conditions need to be tested and verified against for conformance. Often, this results into combinatorial explosion problem (i.e. too many test data set too consider). Earlier work suggests that pairwise sampling strategy based on parameter interactions of variables can be effective. This paper discusses an efficient pairwise strategy, termed RA and ORA, that can systematically minimize the pairwise test set generated from higher order test parameters to lower order ones. In doing so, this paper demonstrates and compares the results against existing strategies including IRPS, IPO, GA, ACA, Jenny and All Pairs.

1. Introduction Nowadays, we are increasingly dependent on software to assist as well as facilitate our daily chores. In fact, whenever possible, most hardware implementation is now being replaced by the software counterpart. From the washing machine controllers, mobile phone applications to the sophisticated airplane control systems, the growing dependent on software can be attributed to a number of factors. Unlike hardware, software does not wear out. Thus, the use of software

can also help to control maintenance costs. Additionally, software is also malleable and can be easily changed as the need arises.

Covering as much as 40 to 50 percent of the total software development costs, testing can be considered one of the most important activities for software validation and verification. Lack of testing can lead to disastrous consequences including loss of data, fortunes and even lives. Many combinations of possible input parameters, hardware/software environments, and system conditions need to be tested and verified against for conformance based on the system’s specification. Often, this results into combinatorial explosion problem.

Combinatorial explosion problem [4][5][6][7][8] poses one of the biggest challenges in modern computer science due to the fact that it often kills traditional approaches to analysis, verification, monitoring and control. A number of techniques have been explored in the past to address this NP complete problem. Undoubtedly, parallel testing (e.g. [12],[13],[14]) can be employed to reduce the time required for performing the tests. Nevertheless, as software and hardware are getting more complex than ever, parallel testing approach becomes immensely expensive due to the need for faster and higher capability processors along state-of-the-art computer hardware. Apart from parallel testing, random testing could also be another option. However, random testing [e.g. [2]) tends to dwell on unfair distribution of the test set.

Earlier wok suggests that pairwise sampling strategy

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

(i.e. based on two-way parameter interaction) can be effective to systematically reduce the test data set to some manageable combinations (yet with the capability to detect from 60 to 80 percent of the faults

[4][5]). Here, any two combinations of parameter values are to be covered by at least one test to ensure acceptable test coverage. Buil

ding and complementing earlier work, the paper proposes two efficient pairwise strategies, called RA and ORA, that can systematically minimize pairwise test set generated from higher order test parameters to lower order ones. Thus, RA and ORA, can be used in conjunction with existing pairwise strategies (e.g. IRPS [6], AETG and its variations [5], IPO [4][8], GA [9], ACA [9] and All Pairs [10]).

This paper is organized as follows. Section 2 discusses some related work. Section 3 introduces the concept of adding the artificial parameters along with a step-by-step example using the proposed strategy. Section 4 highlights comparison of both RA, and ORA with existing strategies. Finally, section 6 gives the conclusion and suggestion for future work.

2. Related Work

As discuss earlier, many different sampling

strategies exist in the literature. In general, existing strategies can be categorized into two categories based on the dominant approaches, that is, algebraic approaches or computational approaches.

Algebraic approaches construct test sets using pre-defined rules. Most algebraic approaches compute test sets directly by a mathematical function [4]. Thus, the computations involved in algebraic approaches are typically lightweight, and in some cases, algebraic approaches can produce the most optimal test sets.

In a nut shell, algebraic approaches are often based on the extensions of the mathematical methods for constructing orthogonal arrays (OA) [1], and covering arrays (CA) [15]. Some variations of the algebraic approach also exploit recursion in order to permit the construction of larger test sets from smaller ones.

Unlike algebraic approaches, computational approaches often rely on the generation of the all pair combinations. Based on all pair combinations, the computational approaches iteratively search the combinations space to generate the required test case until all pairs have been covered. Nevertheless, in the case where the number of pairs to be considered is significantly large, adopting computational approaches can be expensive due to the need to consider explicit enumeration from all the combination space. Some of these strategies are AETG (Automatic Efficient Test Generator) [5], GA (Genetic Algorithm) and ACA (Ant Colony Algorithm) [9], Jenny [11], In Parameter Order (IPO) strategy [4] [8], hill climbing and

simulated annealing (SA) [7], and the Intersection of Residues Pair Set Strategy (IRPS) [6]. 3. Illustrative Examples

Based on empirical results, we noted that higher

number of parameters and values give better size for the generated set. For example, in IRPS strategy the size for 11 11-valued parameters is 121 while the size for 10 10-valued parameters is 149, and the execution time for the first is less than the later. Our contribution in this paper is to generate 10 10-valued parameters pairwise test set from 11 11-valued parameters pairwise set. Here, we expect that the size of the generated test set be at most 121 or something less than it. As will be shown later, by adding artificial parameters and values, we can obtain smaller test set size. To illustrate this issue, we have considered the IRPS strategy for 11 11-valued parameters. The obtained test set is given in Table 1. Apart from IRPS, it should be noted here that our strategy discussed here could be equally applicable to other strategies as well.

Our first step is to remove the artificial parameter (i.e. the last columns), and replace the artificial variables (i.e. all values that equal to 10) by don’t care (-). Using this straightforward optimization , note that T11 contains all don’t care, so the resulted test size equal to 120 test cases only (i.e. as given in Table 2). It should be noted that all parameters and variables have balance coverage. We called this RA strategy.

Finally, our strategy (we call it optimized RA or ORA) is to optimize the resulted test sets by removing the test cases that contains don’t care values. This is done iteratively one-pair-at-a-time by replacing don’t care values with suitable values and removing repetitions if they exists. For example, referring to Table 2, T120 (i.e. -b9c8d7e6f5g4h3i2j1) can be removed. If T120 is removed, then 9*8/2=36 pairs are also removed, namely; (b9, c8), (b9, d7), (b9, e6), (b9, f5), (b9, g4), (b9, h3), (b9, i2), (b9, j1), (c8, d7), (c8, e6), (c8, f5), (c8, g4), (c8, h3), (c8, i2), (c8, j1), (d7, e6), (d7, f5), (d7, g4), (d7, h3), (d7, i2), (d7, j1), (e6, f5), (e6, g4), (e6, h3), (e6, i2), (e6, j1), (f5, g4), (f5, h3), (f5, i2), (f5, j1), (g4, h3), (g4, i2), (g4, j1), (h3, i2), (h3, j1), and (i2, j1). These pairs are distributed to test cases: T19, T69, T49, T39, T99, T29, T89, T79, T26, T17, T91, T73, T82, T44, T64, T77, T34, T15, T101, T58, T66, T45, T31, T63, T13, T38, T62, T97, T103,

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

T51, T67, T41, T25, T86, T105, and T92 respectively. By the same token, T12 can also be removed (i.e.

a1b2c3d4e5f6g7h8i9-) resulting into the removal of the following pairs: (a1, b2), (a1, c3), (a1, d4), (a1, e5), (a1, f6), (a1, g7), (a1, h8), (a1, i9), (b2, c3), (b2, d4), (b2, e5), (b2, f6), (b2, g7), (b2, h8), (b2, i9), (c3, d4), (c3, e5), (c3, f6), (c3,g7), (c3,h8), (c3, i9), (d4, e5), (d4, f6), (d4, g7), (d4, h8), (d4, i9), (e5, f6), (e5, g7), (e5, h8), (e5, i9), (f6, g7), (f6, h8), (f6, i9), (g7, h8), (g7, i9), and (h8, i9). These pairs are distributed to test cases: T100, T111, T109, T56 , T98 , T78, T76 , T54 , T93 , T53 , T113 , T22 , T70 , T80 , T40 , T55 , T106 , T68 , T28 , T46 , T102 , T74 , T104 , T112 , T61 , T117 , T81 , T94 , T42 , T88 , T52 , T114 , T85 , T33 , T57, and T27 respectively. The resulting test set after optimization is given in Table 3.

4. Result and Discussion

As discussed earlier, our RA and ORA strategy

starts with existing pairwise test set. This test set could be generated from any of the existing strategies. The advantage using RA and ORA strategy is the fact that the computation will be minimal compared to generating the pairwise set completely from scratch. In this case, lower order pairwise test set can be generated by the addition of the artificial parameters and values from known higher order pairwise test set. Both RA and ORA guaranty the coverage of pairwise interaction with minimal test set. Which save efforts and execution time for the generation of the test case as well as running the test.

Concerning the performance, Table 4 compares our result obtained in both RA, ORA techniques with some available published results and tools. It is clear that both RA, ORA give minimal or near minimal solutions and outperformed IRPS [6], IPO [4][8], Jenny [11], GA and ACA [9], as well as All Pairs [10] in both the size of generated test set and the execution time. Also, ORA outperformed RA in the size of generated test case but required more time of execution due to the optimization process.

5. Conclusion

In this paper we propose two strategies, namely RA,

ORA, for building test set from higher order test set that contains more parameters and values. Both ORA and RA are performs well, and give minimal number of the test set than other published result and available tools. As a continuation of this work, we plan to investigate other optimization technique that will give

optimal results and run under the grid environment.

6. References

[1] Bush, K.A., Orthogonal Array of Index Unity, Annals of Mathematical Statistics, Vol. 23 (1952), pp. 426-434.

[2] Ammann, P.E. and Offutt, A.J., Using Formal Methods to Derive Test Frames in Category-Partition Testing. In Proc. of the 9th Annual Conference on Computer Assurance (COMPASS’94), IEEE CS Press, June 1994, pp. 69-80.

[3] Kuhn, D.R., Wallace, D.R., Gallo, A.M., Software Fault Interactions and Implications for Software Testing., IEEE Transaction on Software Engineering, Vol. 30, No. 6, IEEE Computer Society (June 2004), pp. 418-421.

[4] Lei, Y., Kacker, R., Kuhn, D.R., Okun, V., Lawrence, J., IPOG: A General Strategy for T-Way Software Testing. In Proc. of the 14th Annual IEEE Intl. Conf. and Workshops on the Engineering of Computer-Based Systems, Tucson, AZ, IEEE Computer Society Press (March 2007) pp. 549-556.

[5] Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C., The AETG system: An Approach to Testing Based on Combinatorial Design, IEEE Transaction on Software Engineering, 23 (7), (1997), pp. 437-443.

[6] Younis, M.I., Zamli, K.Z., and Isa, N.A.M., “IRPS - An Efficient Test Data Generation Strategy for Pairwise Testing”, accepted for publication in KES 2008 (forthcoming).

[7] Cohen, M.B., Designing Test Suites for Software Interaction Testing, PhD Thesis, The University of Auckland (2004).

[8] Lei, Y., Tai, K.C., In-Parameter-Order: A Test Generating Strategy for Pairwise Testing, IEEE Transaction on Software Engineering, 28 (1), (2002), pp. 1-3.

[9] Shiba, T., Tsuchiya, T., Kikuno, T.: Using Artificial Life Techniques to Generate Test Cases for Combinatorial Testing. In Proc. of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04), Hong Kong, China, (September 2004), pp.72-77.

[10] URL: http://www.satisfice.com. [11] URL: http://burtleburtle.net/bob/math/jenny.html. [12] Zamli, K.Z., Mat Isa, N.A., Klaib, M.F.J., and Azizan,

S.N. Designing a Combinatorial Java Unit Testing Tool. In Proc. of the IASTED Intl. Conference on Advances in Computer Science and Technology (ACST 2007), Phuket, Thailand, April 2007.

[13] Zamli, K.Z., Mat Isa, N.A., Klaib, M.F.J, Azizan, S.N., A Tool for Automated Test Data Generation (and Execution) Based on Combinatorial Approach, International Journal of Software Engineering and Its Applications, July 2007, pp. 19-34.

[14] Zamli. K.Z. and Mat Isa, N.A., JTst: An Automated Unit Testing Tool for Java Program, American Journal of Applied Science, Vol 5, No 2, Sept 2007, pp. 77-82.

[15] Zekaaoui, L., Mixed Covering Arrays on Graphs and Tabu Search Algorithm, MSc Thesis, Ottawa-Carleton

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Institute for Computer Science, Univ of Ottawa, September 2006.

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.

978-1-4244-2328-6/08/$25.00 © 2008 IEEE

Authorized licensed use limited to: UNIVERSITI SAINS MALAYSIA. Downloaded on April 15, 2009 at 05:16 from IEEE Xplore. Restrictions apply.