Stareast2008

Copyright © 2008 Testcover.com, LLC. All rights reserved.

Testcover.com

Pairwise Testing Comes of Pairwise Testing Comes of AgeAge

George B. Sherwood, Ph.D.

Thursday May 8, 1:30 p.m.T14 Pairwise Testing


Testcover.com Pairwise Testing Comes of Pairwise Testing Comes of

AgeAge1920

Design of Experiments1940 Orthogonal Arrays

OAs for Manufacturing1960

Covering Arrays1980 OAs for Software

OATS, CATS, AETG Pairwise testing2000 … …

Many more tools Much more testing2020


Testcover.com

ÉÉvariste Galoisvariste Galois• French mathematician, studied

abstract algebra & group theory• Radical Republican during the reign

of Louis Philippe• Short life with tragic ending:• 1829

• Father committed suicide

• Failed entry to École Polytechnique

• 1830• Expelled from École

Normale for criticizing director

• 1831• Imprisoned for wearing

illegal uniform • 1832

• Rejected by Stéphanie-Felice du Motel

• Shot in duel at age 20


Testcover.com

2121stst Century Testing Century Testing• Increasing dependency on software,

networks and distributed applications

• Increasing expectation and need for quality

• Facts of life:• More features and complexity• Shorter development cycles• More reuse of components


Testcover.com

Why Pairwise Testing?Why Pairwise Testing?• Statistical coverage to find faults? Improved quality• Small number of test cases? Reduced cost


Testcover.comPairwise Testing ExamplePairwise Testing Example

• Test 3 new applications on a networked computer

• 4 test configuration factors with 3 values

• 34=81 configurations in total – too many

• 3 configurations cover all values individually – but only a few interactions

Op System Connection Browser ApplicationXP Dial-up Int Explorer App1OS X DSL Netscape App2Linux LAN Firefox App3


Testcover.com

Pairwise TestingPairwise Testing• Includes all pairs of interactions

between test factors• Provides a small set of test cases –

a practical alternative to testing all combinations

• Started almost 20 years ago• Increasingly popular way to

improve quality and reduce cost


Testcover.com

Galois Field ExampleGalois Field Example• Modulo 3 arithmetic – GF(3)• Usual + or x, divide by 3 & take

remainder

• Use GF(3) tables to make orthogonal array

+ 0 1 2 x 0 1 20 0 1 2 0 0 0 01 1 2 0 1 0 1 22 2 0 1 2 0 2 1


Testcover.com

Orthogonal Array OA(9;3Orthogonal Array OA(9;333))• Replace each element in the x table

with the corresponding column from the + table

+ 0 1 2 x 0 1 20 0 1 2 0 0 0 01 1 2 0 1 1 12 2 0 1 2 2 2

1 0 1 21 2 02 0 1

2 0 2 11 0 22 1 0


Testcover.com

Orthogonal Array OA(9;3Orthogonal Array OA(9;344))• Add one column which pairs a

different value with every 3 rows

0 0 0 00 1 1 10 2 2 21 0 1 21 1 2 01 2 0 12 0 2 12 1 0 22 2 1 0


Testcover.com Networked Computer Networked Computer

ExampleExample• 4 test configuration factors with 3

values• Replace numbers in OA(9;34) template

with test factor valuesOp System Connection Browser Application0 XP 0 Dial-up 0 Int Explorer 0 App11 OS X 1 DSL 1 Netscape 1 App22 Linux 2 LAN 2 Firefox 2 App3



ExampleExample• Only 9 configurations are needed to

cover all 54 pairs of values using OA(9;34)

Op System Connection Browser ApplicationXP Dial-up Int Explorer App 1XP DSL Netscape App 2XP LAN Firefox App 3

OS X Dial-up Netscape App 3OS X DSL Firefox App 1OS X LAN Int Explorer App 2Linux Dial-up Firefox App 2Linux DSL Int Explorer App 3Linux LAN Netscape App 1


Testcover.com Design of Design of

ExperimentsExperiments• Ronald A. Fisher analyzed

data from crop yield experiments• Developed new statistical methods to study experimental data

• Pioneered the mathematical principles of experimental design


Testcover.com

Orthogonal ArraysOrthogonal Arrays

• Orthogonal Array designs are used in agricultural & manufacturing experiments, clinical trials, market research…

• Results can show effects of factors and how they interact

• Calyampudi Radhakrishna Rao introduced Orthogonal Arrays


Testcover.com OAs for OAs for

ManufacturingManufacturing• Genichi Taguchi adapted

Fisher’s experimental design to improve quality of manufactured goods• Used OAs to select test parameters in creative ways

• Worldwide influence in improving manufactured quality


Testcover.com

OAs for Software TestOAs for Software Test• Keizo Tatsumi, Robert Mandl & others

applied these ideas to software testing

• Taguchi visited Bell Labs and collaborated with Madhav Phadke

• Orthogonal Array Test System (OATS) offered statistical coverage where exhaustive testing was not possible


Testcover.com

StarLAN and StarGROUPStarLAN and StarGROUP• Client installation test

• 101 combinations of PCs and OS versions

• 15 network cards• Tried to use OATS to reduce

number of test configurations• OATS could not handle our

complex configuration constraints correctly

• Generated impossible tests, as in previous example



ExampleExample• Two of the 9 configurations from

OA(9;34) are invalid

Op System Connection Browser ApplicationXP Dial-up Int Explorer App 1XP DSL Netscape App 2XP LAN Firefox App 3

OS X Dial-up Netscape App 3OS X DSL Firefox App 1OS X LAN Int Explorer App 2Linux Dial-up Firefox App 2Linux DSL Int Explorer App 3Linux LAN Netscape App 1


Testcover.com

What to do?What to do?• Skip the bad cases? Then for each case

there is no coverage for 5 good pairs, along with the invalid one

• Replace values ‘Linux’, ‘OS X’ or ‘Int Explorer’ to avoid bad pairs? Still miss good pairs

• Solution: Computer search for small number of cases to cover all allowed pairs.



ExampleExample• 11 configurations cover 52 allowed pairs

Op System Connection Browser ApplicationXP Dial-up Netscape App 1XP DSL Firefox App 2

OS X DSL Netscape App 3OS X LAN Firefox App 1Linux Dial-up Firefox App 3Linux LAN Netscape App 2XP LAN Int Explorer App 3

OS X Dial-up Firefox App 2Linux DSL Netscape App 1XP Dial-up Int Explorer App 2XP DSL Int Explorer App 1


Testcover.com Constrained Array Test Constrained Array Test

SystemSystem• Search among allowed test cases

to handle constraints• Find next “best” test case to

minimize uncovered pairs• Secondary result: fewer test cases

than OATS


Testcover.com

Types of ArraysTypes of Arrays

Orthogonal Arrays Covering Arrays

All pairs in equal numbers at least onceTest cases a few fewer typically

Fixed values OA(18;37) CA(14;37)Mixed values OA(12;3124) CA(6;3124)


Testcover.com

Early ToolsEarly Tools

Orthogonal ArrayTest System (OATS)

Constrained ArrayTest System (CATS)

Automatic EfficientTest Generator (AETG)

Bell Labs Bell Labs Bellcore (Telcordia)Internal tool Internal tool Commercial service

OA templateswith dependencies

Greedy search forCAs with constraints

Greedy search forCAs with constraints


Testcover.com Pairwise Selection of Input Pairwise Selection of Input

DataData• Identify equivalence classes

• Analyze functional requirements, use cases• Partition factor value combinations for equivalent

processing and expected results• Generate test cases for different classes

separately• All pairs from each class go to same resulting state• Example: Don’t mix error combinations with normal

combinations


Testcover.com Uncle Katzendogg’s Uncle Katzendogg’s

Guaranteed Long-term Guaranteed Long-term Weather ForecastsWeather Forecasts


Testcover.com

Weather Forecast ExampleWeather Forecast Example• 4 test factors: Month, Day, Year, Zip code• 3 sets of test cases leading to different

states:1. Valid dates with valid zip codes

? Forecast returned

1. Invalid dates with valid zip codes? Date error returned

1. Valid dates with invalid zip codes? Zip code error returned


Testcover.com Weather Forecast Valid Input Weather Forecast Valid Input

Factor ValuesFactor Values+ all good dates + feb last day

Month: jan feb mar apr may jun jul aug sep oct nov dec

Month: feb

Day: 1 10 Day: 28Year: 2008 2009 2010 Year: 2009 2010

Zip code: 07722 32819 Zip code: 07722 32819+ long month last day + leap day

Month: jan mar may jul aug oct dec

Month: feb

Day: 31 Day: 29Year: 2008 2009 2010 Year: 2008

Zip code: 07722 32819 Zip code: 07722 32819+ short month last day

Month: apr jun sep novDay: 30

Year: 2008 2009 2010Zip code: 07722 32819


Testcover.com Weather Forecast Valid

Input Test CasesMonth Day Year Zip code Month Day Year Zip code Month Day Year Zip codejan 1 2008 07722 jun 1 2009 32819 nov 1 2010 32819jan 10 2009 32819 jul 1 2008 07722 dec 1 2009 32819feb 1 2010 32819 jul 10 2009 32819 jul 31 2010 32819mar 10 2010 07722 aug 10 2008 32819 jan 31 2010 32819aug 31 2009 07722 sep 10 2008 32819 mar 31 2008 07722oct 31 2008 32819 dec 10 2008 32819 feb 28 2010 32819nov 30 2009 32819 dec 31 2010 07722 feb 10 2008 32819jun 30 2008 07722 sep 30 2009 07722 may 10 2009 07722feb 28 2009 07722 apr 10 2009 32819 may 1 2010 32819mar 1 2009 32819 jun 10 2010 07722 sep 1 2009 32819may 10 2008 32819 aug 1 2010 32819 sep 10 2010 07722may 31 2010 07722 oct 1 2008 07722 oct 10 2010 32819apr 30 2010 32819 oct 10 2009 32819 nov 10 2009 07722feb 29 2008 07722 nov 10 2008 32819 feb 29 2008 32819apr 1 2008 07722


Testcover.com Weather Forecast Invalid Weather Forecast Invalid

Date Factor ValuesDate Factor Values+ too long long month + feb too long, regular year

Month: jan mar may jul aug oct dec

Month: feb

Day: 32 Day: 29Year: 2008 Year: 2010

Zip code: 07722 Zip code: 07722+ too long short month + feb too long, leap year

Month: apr jun sep nov Month: febDay: 31 Day: 30

Year: 2009 Year: 2008Zip code: 32819 Zip code: 32819


Testcover.com Weather Forecast Invalid Weather Forecast Invalid

Date Test CasesDate Test CasesMonth Day Year Zip Codejan 32 2008 07722apr 31 2009 32819feb 29 2010 07722feb 30 2008 32819mar 32 2008 07722may 32 2008 07722jul 32 2008 07722aug 32 2008 07722oct 32 2008 07722dec 32 2008 07722jun 31 2009 32819sep 31 2009 32819nov 31 2009 32819


Testcover.com Weather Forecast Invalid Zip Weather Forecast Invalid Zip

CodeCodeTest casesFactor values

+ bad zip code Month Day Year Zip codeMonth: feb may feb 1 2009 00001Day: 1 10 may 10 2010 00001Year: 2009 2010 may 10 2009 99999Zip code: 00001 99999 may 1 2010 99999

feb 10 2010 99999


Testcover.com

Pairwise Testing TodayPairwise Testing Today• Test demands continue to grow• Pairwise testing is a popular

response for effective coverage with a limited number of test cases

• Improved training is needed, e.g. to handle constraints, equivalence classes

• Careless use has led to controversy


Testcover.com

Available Tools

1. CATS (Constrained Array Test System) *)

[Sherwood] Bell Labs.

2. OATS (Orthogonal Array Test System) *)

[Phadke] AT&T

3. AETG Telecordia Web-based, commercial

4. IPO (PairTest) *) [Tai/Lei]

5. TConfig [Williams] Java-applet

6. TCG (Test Case Generator) *) NASA

7. AllPairs Satisfice Perl script, free, GPL

8. Pro-Test SigmaZone GUI, commercial

9. CTS (Combinatorial Test Services) IBM Free for non-commercial use

10. Jenny [Jenkins]

Command-line, free, public-domain

11. ReduceArray2 STSC, U.S. Air Force Spreadsheet-based, free

12. TestCover Testcover.com Web-based, commercial

Pairwise.org Tools List – part Pairwise.org Tools List – part 11


Testcover.com

13. DDA *) [Colburn/Cohen/Turban]

14. Test Vector Generator GUI, free

15. OA1 k sharp technology

16. CTE-XL Daimler Chrystler GUI, free

17. AllPairs [McDowell] Command-line, free

18. Intelligent Test Case Handler (replaces CTS)

IBM Free for non-commercial use

19. CaseMaker Díaz & Hilterscheid GUI, commercial

20. PICT Microsoft Command-line, free

21. rdExpert Phadke Associates, Inc.

22. OATSGen *) Motorola

23. SmartTest

Smartware Technologies Inc.

GUI, commercial

24. EXACT *) [Yan/Zhang]

25. AllPairs MetaCommunications Free new

*) Not known to be available publicly

Pairwise.org Tools List – part Pairwise.org Tools List – part 22


Testcover.com

Bach & SchroederBach & Schroeder• “Pairwise Testing: A Best Practice that Isn’t”

• Useful criticism of careless pairwise testing• MS Word options example – 31212

Pairs coveredTest cases Pairwise Random

10 100% 88%23 100%


Testcover.com

Pairwise Testing EfficiencyPairwise Testing Efficiency• Pairwise generators are more efficient

than random test case selection• Significant improvements in pairwise

efficiency are possible• Practitioners need pairwise testing to

meet modern testing challenges


Testcover.com

How do Searches Work?How do Searches Work?

Uncovered pairs

Factor values

?

Test cases

• Go to the space of all possible test cases• Look for small number of test cases (short

path) to minimize the uncovered pairs


Testcover.com

Search VariationsSearch Variations• Several algorithms for different

tradeoffs• Test practitioner goals:

• Few test cases• Quick responses• Practicality – mixed values, constraints

• Researcher goals:• Fewer test cases• More factors• New methods, better solutions


Testcover.comFinding the Minimal Finding the Minimal

Set of Test Cases is Set of Test Cases is HardHard ?

• Mixed covering array constructions can complement searches using less processing

• Typically searches are CPU bound

• Improvement diminishes with processing time


Testcover.com Change Mixed CA Values Change Mixed CA Values

Without SearchingWithout Searching• Example:

Change OA(9;34) to CA(12;413122)• Reduce the number of values in

two of the factors• Increase the number of values in

one factor


Testcover.com

OA(9;3OA(9;344) to CA(9;3) to CA(9;3222222) ) • Replace all the 2s in any 2 factors

with unassigned (*) values0 0 0 00 1 1 10 2 * *1 0 1 *1 1 * 01 2 0 12 0 * 12 1 0 *2 2 1 0

0 0 0 00 1 1 10 2 2 21 0 1 21 1 2 01 2 0 12 0 2 12 1 0 22 2 1 0


Testcover.com

CA(9;3CA(9;3222222) to CA(12;4) to CA(12;41133112222))• Add 3 rows with 3s to

one of the factors with 3 values

• For each of the other factors, pair its values with the 3s in the new rows

0 0 0 00 1 1 10 2 * *1 0 1 *1 1 * 01 2 0 12 0 * 12 1 0 *2 2 1 03 0 0 03 1 1 13 2 * *


Testcover.com More CA Factors from More CA Factors from

Products and SearchesProducts and Searches• Product sequence:

• OA(9;34), CA(15;315), CA(21;357), …• Nurmela’s tabu search:

• CA(15;320)• Products using tabu search result:

• CA(15;320), CA(21;374), CA(27;3282), …• Combined methods give better

results


Testcover.comOptimal Mixed CAs & More Optimal Mixed CAs & More

FactorsFactors• Add n test cases to OA(25;56) to get optimal

array• Each factor of 5 values can be replaced by the

indicated factors without additional test cases

• Example: Replace 3 factors in CA(35;7155) to get CA(35;715248360)

n Optimal Array Replacement Factors0 OA(25;56) 35 26,435

5 CA(30;6155) 312 287,516

10 CA(35;7155) 44 360 21,410,864

15 CA(40;8155) 44 390 219,969,152

20 CA(45;9155) 55 416 3324 2320,932,800

25 CA(50;10155) 55 416 3810 24,653,525,600


Testcover.com

Mixed Covering ArraysMixed Covering Arrays• Compared with OA templates

• More choices for numbers of values• Fewer test cases typically

• Compared with greedy search• Faster responses (less processing)• Fewer test cases when CAs are

optimal• Search needed for constraints


Testcover.com

Future Pairwise TrendsFuture Pairwise Trends• Platforms:

• Faster, distributed processing• Tool Software:

• Smarter searches, less processing• Fewer test cases

• Improved test efficiency• Faster responses

• What-if test designs, test changes• Use with agile, incremental cycles

Stareast2008

Technology

example test

test factors

allowed test cases

fewer test cases

sets of test cases

test factor values

test parameters

invalid date test cases