73 Tuning Parameters In Heuristics By Using Design Of Experiments Methods Arif Arin; Ghaith Rabadi; Resit Unal Department of Engineering Management and Systems Engineering, Old Dominion University [email protected]Abstract. With the growing complexity of today's large scale problems, it has become more difficult to find optimal solutions by using exact mathematical methods. The need to find near-optimal solutions in an acceptable time frame requires heuristic approaches. In many cases, however, most heuristics have several parameters that need to be "tuned" before they can reach good results. The problem then turns into "finding best parameter setting" for the heuristics to solve the problems efficiently and timely. One- Factor-At-a-Time (OFAT) approach for parameter tuning neglects the interactions between parameters. Design of Experiments (DOE) tools can be instead employed to tune the parameters more effectively. In this paper, we seek the best parameter setting for a Genetic Algorithm (GA) to solve the single machine total weighted tardiness problem in which n jobs must be scheduled on a single machine without preemption, and the objective is to minimize the total weighted tardiness. Benchmark instances for the problem are available in the literature. To fine tune the GA parameters in the most efficient way, we compare multiple DOE models including 2-level (2 k ) full factorial design, orthogonal array design, central composite design, D-optimal design and signal-to-noise (SIN) ratios. In each DOE method, a mathematical model is created using regression analysis, and solved to obtain the best parameter setting. After verification runs using the tuned parameter setting, the preliminary results for optimal solutions of multiple instances were found efficiently. 1. INTRODUCTION One of the most important effects of the improving modern sciences and technologies is to enable us understand and model real life problems realistically and in more details. The natural outcome of this fact is the rapid increase of dimensions and complexity of the problems. With the growing complexity of today's large scale problems, it has become more difficult to find optimal solutions by using only exact mathematical methods. Due to the concern of efficiency in terms of the solution quality, the need to find near-optimal solutions in an acceptable time frame requires using heuristic approaches. Heuristics are quite new approaches in the field of combinatorial optimization. A heuristic can be defined as "a generic algorithmic template that can be used for finding high quality solutions of hard combinatorial optimization problems" [1]. Heuristic approaches have already proved themselves in many large scale optimization problems by offering near-optimal solutions where there is no optimal solution found by other approaches. In many cases, however, most heuristics have several parameters that need to be "tuned" before they can reach good results. The accepted values of the parameters to be employed in the heuristics have considerably significant impact on both solution process and the solution itself. To obtain the best reSUlts, the problem then turns into "finding the best parameter setting" for the heuristics to solve the problems efficiently and timely, which becomes an optimization problem by itself. There are various methods used to find the best parameter setting in the literature. One-Factor-At- a-Time (OFAT) approach for parameter tuning is one of them; however, it neglects the interactions between the parameters that might change the whole solution process and quality of solution. Particularly, in terms of the interactions, Design of Experiments (DOE) methods are promising approaches and can be easily employed to tune the parameters more effectively. In this paper, we seek the best parameter setting for a genetic algorithm to solve the single machine total weighted tardiness problem in which n jobs must be scheduled on a single machine without preemption, and the objective is to minimize the total weighted tardiness. Benchmark instances for the single machine total weighted tardiness problem are available in the literature. 2. DESIGN OF EXPERIMENTS (DOE) To fine tune the genetic algorithm parameters in the most efficient way, we compare mUltiple DOE tools including 2-level (2 k ) full factorial design, orthogonal array design, central composite design, D-optimal design and signal-to-noise (SIN) ratios method. In each DOE method, a mathematical model is created using regression analysis, and solved to obtain the best parameter setting. After verification runs for other benchmark instances by using the tuned parameter setting, https://ntrs.nasa.gov/search.jsp?R=20100012849 2020-05-20T12:16:56+00:00Z
6
Embed
Tuning Parameters In Heuristics By Using Design Of Experiments Methods · 2016-06-09 · 73 Tuning Parameters In Heuristics By Using Design Of Experiments Methods ArifArin; Ghaith
This document is posted to help you gain knowledge. Please leave a comment to let me know what you think about it! Share it to your friends and learn new things together.
Transcript
73
Tuning Parameters In Heuristics By Using Design OfExperiments Methods
Arif Arin; Ghaith Rabadi; Resit Unal
Department of Engineering Management and Systems Engineering, Old Dominion [email protected]
Abstract. With the growing complexity of today's large scale problems, it has become more difficult tofind optimal solutions by using exact mathematical methods. The need to find near-optimal solutions in anacceptable time frame requires heuristic approaches. In many cases, however, most heuristics haveseveral parameters that need to be "tuned" before they can reach good results. The problem then turnsinto "finding best parameter setting" for the heuristics to solve the problems efficiently and timely. OneFactor-At-a-Time (OFAT) approach for parameter tuning neglects the interactions between parameters.Design of Experiments (DOE) tools can be instead employed to tune the parameters more effectively. Inthis paper, we seek the best parameter setting for a Genetic Algorithm (GA) to solve the single machinetotal weighted tardiness problem in which n jobs must be scheduled on a single machine withoutpreemption, and the objective is to minimize the total weighted tardiness. Benchmark instances for theproblem are available in the literature. To fine tune the GA parameters in the most efficient way, wecompare multiple DOE models including 2-level (2k
) full factorial design, orthogonal array design, centralcomposite design, D-optimal design and signal-to-noise (SIN) ratios. In each DOE method, amathematical model is created using regression analysis, and solved to obtain the best parameter setting.After verification runs using the tuned parameter setting, the preliminary results for optimal solutions ofmultiple instances were found efficiently.
1. INTRODUCTIONOne of the most important effects of the improvingmodern sciences and technologies is to enable usunderstand and model real life problemsrealistically and in more details. The naturaloutcome of this fact is the rapid increase ofdimensions and complexity of the problems. Withthe growing complexity of today's large scaleproblems, it has become more difficult to findoptimal solutions by using only exactmathematical methods. Due to the concern ofefficiency in terms of the solution quality, the needto find near-optimal solutions in an acceptabletime frame requires using heuristic approaches.
Heuristics are quite new approaches in the field ofcombinatorial optimization. A heuristic can bedefined as "a generic algorithmic template thatcan be used for finding high quality solutions ofhard combinatorial optimization problems" [1].Heuristic approaches have already provedthemselves in many large scale optimizationproblems by offering near-optimal solutions wherethere is no optimal solution found by otherapproaches. In many cases, however, mostheuristics have several parameters that need tobe "tuned" before they can reach good results.The accepted values of the parameters to beemployed in the heuristics have considerablysignificant impact on both solution process andthe solution itself. To obtain the best reSUlts, theproblem then turns into "finding the bestparameter setting" for the heuristics to solve the
problems efficiently and timely, which becomes anoptimization problem by itself.
There are various methods used to find the bestparameter setting in the literature. One-Factor-Ata-Time (OFAT) approach for parameter tuning isone of them; however, it neglects the interactionsbetween the parameters that might change thewhole solution process and quality of solution.Particularly, in terms of the interactions, Design ofExperiments (DOE) methods are promisingapproaches and can be easily employed to tunethe parameters more effectively.
In this paper, we seek the best parameter settingfor a genetic algorithm to solve the single machinetotal weighted tardiness problem in which n jobsmust be scheduled on a single machine withoutpreemption, and the objective is to minimize thetotal weighted tardiness. Benchmark instances forthe single machine total weighted tardinessproblem are available in the literature.
2. DESIGN OF EXPERIMENTS (DOE)
To fine tune the genetic algorithm parameters inthe most efficient way, we compare mUltiple DOEtools including 2-level (2k
) full factorial design,orthogonal array design, central compositedesign, D-optimal design and signal-to-noise(SIN) ratios method. In each DOE method, amathematical model is created using regressionanalysis, and solved to obtain the best parametersetting. After verification runs for other benchmarkinstances by using the tuned parameter setting,
DOE methods presented will be compared interms of their solution qualities.
The single machine total weighted tardinessproblem is used in this paper as a difficult problemto demonstrate the use of DOE for setting theoptimization Genetic Algorithm (GA) parameters.In this problem, n jobs must be scheduled on asingle machine where each job j has a givenprocessing time Pj and a due date dj . Thetardiness Tj is defined as max (0, Crdj) where Cj isthe job's completion time - a decision variablethat is based on the job sequence. The objectivefunction then becomes to minimize LJ=l wjTj.This is a well known problem to which benchmarkproblems are available. In seeking best parametersetting for the GA, we will be using a MS-excelAdd-in called Evolver from Palisade [6].
We first implemented the problem in Excelspreadsheet, and used the first instance of 40-jobbenchmark problem to compare different DOEmethods that are discussed below. The upper andlower levels for the GA parameters are given inthe Table 1.
The GA stopping criteria are to run for 10 minutesor to stop whenever the percent deviation of thesolution from the optimal solution/best solutionfound so far becomes O. In the following sections,we discuss and compare five DOE methods tosee which method performs best.
2.1. 2-Level (2k) Full Factorial Design
2-Level (2k) full factorial design is the one of the
most widely used DOE tools. In 2k full factorialdesign, k is the number of factors. After the lowerand upper levels of the factors are determined, allcombinations of these factor levels are studiedsimultaneously. In order to analyze the design,each factor should be linearly independent, whichmeans the covariance of the factors should beequal to zero. The covariance is a measure oflinear relationship between two random variables[5], and can be calculated by using the followingequation where E(x) stands for the expected valueofx.
Cov(x,y) = E(x,y) - E(x)E(y)
To calculate the covariance of the design, atransformation is needed from the lower andupper levels to (-1) and (+1), respectively. Afterthese substitutions, because E(x,y) = 0, E(x) = 0,and E(y) = 0, Cov(x,y) is equal to zero. Inorthogonal designs, the covariance is alwaysequal to zero.
The 2k full factorial design is generated by usingYates algorithm. According to this algorithm; forthe first factor, a column of (-1) and (+1) is writtendown with the signs alternating each time. For thesecond factor, the signs alternate in pairs, for thethird factor they alternate in triple, and so on. Tocreate the interactions columns, the levels of theeach factor forming the interactions are simplymultiplied.
In an experimental design, the number ofexperiments (rows) must at least be equal to thetotal degrees of freedom (DF) required for thestudy, as shown in Table 2.
Table 2: DF for 2k full factorial design with k=3
Factors/Interactions DFOverall Mean 1A,B,C 3 (2-1)AB, AC, BC 3(2-1)(2-1)ABC 1(2-1 )(2-1 )(2-1)
Total 8
One drawback of 2k full factorial design is rapidincrease of the number of experiments whileincreasing the number of the factors (25=32,28=256, i O=1024). In 1940's, Fisher showed thatmeaningful results can be obtained by conductinga selected fraction of full factorial design which iscalled fractional factorial design, 2k
-p
, where pstands for the fraction portion.
Since there are 3 factors (k=3) in our problem,23=8 experiments are needed to run for 2-level fullfactorial design in Table 3.
In each experiment, the factors, or parameters,are set and run according to the design. After thesolutions Y obtained from the experiments areanalyzed by implementing regression analysis,the mathematical model is derived. However,because R2 value of the model is 1.00, the termthat has minimum effect (AB) is removed, andafter running the regression analysis again, thefollowing model with R2 = 0.96 is obtained.
When this model is solved by employing ExcelSolver to minimize Y, the parameter setting isfound by using 2k full factorial design as
75
"Crossover = 0.01, Mutation = 0.2, Population =30".
2.2. Orthogonal Array DesignThe fact that effects of 3 or higher interactionstend to be insignificant, and therefore may beignored, bring us to the fractional factorial designtype named orthogonal array (OA) design whereonly main factors and 2-factor interactions areconsidered. A typical OA tabulation is in the formof La(bC
), where a is the number of experiments, bis the number of levels, and c is the number ofcolumns. Taguchi has formulated 18 standard OAdesigns [7], however they can also be modified byusing various methods. To select the appropriateOA, first, number of factors and levels for eachfactor, and 2-factor interactions to be estimatedmust be defined. After calculating the OF, the OAwith the closest number of the experiments to OFis selected. Interaction tables, or linear graphsdeveloped by Taguchi are then utilized to followthe confounding pattern.
The OF of our problem for OA is 7 due to theabsence of 3-factor interactions. The mostappropriate OA for 3 factors, 2 levels and 7experiments is La(27
Because there are only 3 factors in the problem,all 2-factor interactions are included. As younotice, the 2k full factorial and OA designs withk=3 are about the same. The reason is that thenumber of factors is quite small, and increasingthis number will clearly bring out the advantagesof OA designs in terms of the number ofexperiments needed to study.
After implementing regression analysis for the OAdesign, the same mathematical model with 2k fullfactorial design is derived, except for the ABCterm. This model has R2 value of 0.65. As in 2k fullfactorial design, Excel Solver gives the samesolution set for A, B, and C, respectively, namely,the parameter setting for the OA design is again"Crossover = 0.01, Mutation = 0.2, Population =30".
2.3. Central Composite Design
In 2k full factorial and OA designs it is assumedthat the relationship between the 2-level factors is
linear. It is possible to increase the number oflevels to 3 to capture the nonlinearity, however, itwould be a bit controversial and none of the rulesfor the 2-levels would apply in those designs.Also, this would not be the best candidate forcontinuous factors like parameters used inheuristics. A better approach to cope with thenonlinearity and continuous factors could beResponse Surface Method using the CentralComposite Design (CCO) developed by Box &Wilson in 1950's [4].
CCO is a first-order design augmented byadditional points that allow the estimation of thesecond-order mathematical model. CCO consistsof a full factorial or fractional factorial design (2k or2k
•P), a center point (a row of zero's), and two
points on axes for each factor at a distance a fromthe design center which result 2k+2k+1 or2k
-P+2k+1experiments in total. The distance a iscalculated as (number of experiments in fractionalportion)1/4. It is possible to choose a = +1, whichis then called face-centered design.
In our problem, 23 = 8 experiments for thefractional portion, 2(3) = 6 experiments for axialportion, and 1 experiment for center portion, total15 experiments are needed. The distance a isequal to (8)1/4 == 1.4. To be able to set theparameters for each experiment, the levels of theparameters must be coded for the values (-1.4, -1,0,1,1.4). The complete CCO with k = 3 is shownin Table 5.
The solution set produced by Excel Solver is backcoded to their real values, and the parametersetting found by CCO is "Crossover = 0.218,Mutation = 0.193, Population = 100".
76
2.4. D-Optimal Design
CCD is quite an efficient design especially due toadding the second-order nonlinearity; however, insome cases it may not be enough to understandthe relationships between factors. And also, thenumber of experiences must be kept to anabsolute minimum. If a design has an absoluteminimum number of experiments, such design iscalled "saturated design". The minimum numberof experiments can be calculated as (n+1)(n+2)/2where n is number of factors. Besides theseadvantages, if some experiments are infeasible,saturated designs can be still used by extractingthese experiments from the design.
As some of the interesting features of saturateddesigns, unlike the previous DOE methods, theyare not orthogonal and there are no degrees offreedom to test the accuracy of the model.
Saturated designs are constructed by applied 0optimality criterion. The following equation is theestimator of simple linear regression:
y= bo + L>iX;
where bo is the intercept, bj are the slopes. If thisequation is written in matrix form, we have:
y= XB+c.The set of design B can be estimated in thefollowing form by applying the Least SquareRegression method.
B= (XTXr1XTy
A statistical measure of accuracy of B is thevariance-covariance matrix:
V(B) = 0-2(XTXr'
where a2 is the variance of the error. V(B) is afunction of (XTXr' and to increase the accuracy,(XTXr' should be minimized. Statistically,minimizing (XTXr' is equal to maximizing thedeterminant of (XTX). "0" in the term of D-optimalcomes from the first letter of the word"determinant". There are some heuristics [2], andsoftware [3] to come up with a design thatmaximizes the determinant of (XTX). To obtainmore accurate results, D-optimal designs can beaugmented by adding more experiments.
The absolute minimum of experiments for ourproblem is 10 [=(3+1)(3+2)/2], and the D-optimaldesign displayed in Table 6 is created byaugmenting the design by 2 experiments.
Like CCD, the levels of the parameters must becoded for the values (-1, 0, 1). With the help ofregression analysis, the following mathematicalmodel is acquired:
Y = 92.48 - 0.63A + 2.62B + 8.37C - 6.38AB - 2
9.13AC + 15.63BC + 19.86A2 + 23.11 B2- 18.83C
After the solution set given by Excel Solver isback coded to their real values, and the parametersetting found by D-Optimal is "Crossover = 0.420,Mutation = 0.148, Population = 30".
Table 6: D-Optimal Design with k = 3
A B C AB AC BC A;/' B' C'1 -1 -1 -1 1 1 1 1 1 12 -1 -1 1 1 -1 -1 1 1 13 -1 a a a 0 a 1 a a4 -1 1 -1 -1 1 -1 1 1 15 -1 1 1 -1 -1 1 1 1 16 a -1 a a a a a 1 a7 a a 1 a a a a a 18 1 -1 -1 -1 -1 1 1 1 19 1 -1 1 -1 1 -1 1 1 1
10 1 1 -1 1 -1 -1 1 1 111 1 1 a 1 a a 1 1 a12 1 1 1 1 1 1 1 1 1
2.5. Signal-To-Noise (SIN) Ratio
DOE methods until this section are only based onone instance of our problem, and do not considerany information of other instances. The method ofsignal-to-noise (SIN) ratio can be defined as aperformance measure that takes the mean andthe variability into account, and give the ability touse information of other instances in seeking thebest parameter setting. It involves two types offactors: control factors and noise factors. Noisefactors cause variability which leads to loss ofquality. There are three kinds of noise; outernoise, inner noise, and between product noise, orhere can be defined as "between instance noise"is the main reason in applying SIN ratio method inour problem.
Generally, data analysis using SIN ratio (11) can beperformed to achieve three types of purposes:smaller-the-better, larger-the-better and nominalthe-best. Since our target is to minimize the totalweighted tardiness for the single machine, theappropriate type of '1 is smaller-the-better. Tominimize the sensitivity to noise factors, wemaximize 11 which is calculated by the followingequation [4].
1] = -lOloglo (y2 + 0-2)
In addition to first instance, fourth and ninthinstance are randomly selected as different"products". Unlike in other methods, instead ?fOA, D-optimal design in Table 6 is used In
creating the experiments for each instancebecause of its advantages, and 11 is calculated asthe outcome for each experiment. Threereplications of D-optimal design for threeinstances increase the total number ofexperiments by 36 (=3x12).
77
Table 7: Parameter settings of DOE methods
To be able to compare the solutions for differentinstances, the percent deviation of the solutionfrom the optimal solution/best known solution isused instead of the real outcomes of theexperiments.
Because 2k full factorial and orthogonal arraydesigns give same parameter settings for 3factors, their common results share the first threecolumns.
DOE Crossover Mutation PopulationType Prob.(A) Prob. (B) Size (C)
After applying five DOE methods to find the bestparameter setting for the single machine totalweighted tardiness problem, the findings aresummarized in Table 7. To test which method ismost effective with this problem, these parametersettings are used in solving the first 20 instancesfor both 40-job problems in Table 8 and 50-jobproblems in Table 9 respectively [8].
Table 8: Comparison of Parameter settings for 40-job problem
After applying the steps of D-optimal design foreach instance, the regression analysis is run for toobtain the following mathematical model:
After back coding the findings in Excel Solver totheir real values, the parameter setting found byD-Optimal are "Crossover = 0.465, Mutation =0.157, Population =30".
Orthogonal Array & 2' Central Composite D-Optimal SIN Ratios
Inst Full Factorial Designs Design Design Design%Dev Iteration Time YoDev Iteration Time %Dev Iteration Time %Dev Iteration Time
According to data from the 40-job and 50-jobproblems, the SIN ratios and D-optimal designsseem to be the best two methods of the five DOEmethods. While SIN ratios design could reachoptimum solutions/best known in 8 instances for40-job and 6 instances 50-job problems, 0optimal design could obtain them in 7 instancesfor the 40-job problems, and 6 instances for the50-job problems. In terms of average percentagedeviation, the number of iteration and running
time, they are also better than the other threemethods. We might accept that SIN ratios designis slightly better than D-optimal design, but itneeds three times more experiments than 0optimal design. Even though all DOE methods arecompleted based on the first instance of 40-jobproblem, the parameter settings found in theseprocesses produce very close results to the 50-jobproblems which gives an idea about therobustness of the parameter settings.
78
Table 9: Comparison of Parameter settings for 50-job problem
Orthogonal Array & 2K Central Composite D-Optimal SIN Ratios
nst. Full Factorial Designs Design Design Design%Devlteration Time %Dev Iteration Time VoDev. Iteration Time %Dev. Iteration Time
DOE offers a practical way to tune the heuristicparameters. Because the number of parameters,or factors, is not the same for all heuristics, it isimportant to select the right DOE method. Table10 shows how fast the number of experimentsincreases for a small amount of increase in thenumber of factors with three levels. Otherimportant issues' include selecting the number oflevels, values of the levels, the type ofrelationships between factors, and the cost ofrunning of an experiment.
It should be noted that the same parametersetting produces different solutions for differentinstances although all instances are created fromthe same distributions. For the total weightedtardiness problem, the most effective methodsturned out to be the D-Optimal and SIN RatiosDesign, with the D-Optimal design requiring lessruns.
This paper presented a structured framework onusing DOE to tune optimization algorithmparameters. The weighted tardiness schedulingproblem was used as a vehicle to demonstrate the
approach. The same approach can be applied toother problems.
REFERENCES
1. Birattari, M. (2009) "Tuning Metaheuristics: AMachine Learning Perspective", Studies inComputational Intelligence, Volume 197,Springer-Verlag Berlin Heidelberg.
2. Box, J.M., Drapper, N.R. (1974) "On minimumpoint second order designs", Technometrics, 16,613-616.
3. JMP, Design of Experiments Guide (2007),SAS Institute Inc, Cary, North Carolina.
4. Montgomery, D.C. (2001) "Design and Analysisof Experiments", Fifth Edition, John Wiley & Sons,Inc., USA.
5. Montgomery, D.C., Runger G.C. (2003)"Applied Statistics Applied Statistics forEngineers", Third Edition, John Wiley & Sons,Inc., USA.
6. Palisade Corporation (2008), "Evolver, TheGenetic Algorithm Solver for Microsoft Excel",Version 5.0, Ithaca, NY USA.
7. Phadke, M;S. (1989) "Quality EngineeringUsing Robust Design", AT&T Bell Laboratories,PTR Prentice-Hall, Inc., New Jersey.