Equal Angle Distribution of Polling Directions in Direct-Search Methods

The purpose of this paper is twofold: first, to introduce deterministic strategies for directional direct-search methods, including new instances of the mesh adaptive direct-search (Mads) and the generating set search (Gss) class of algorithms, which utilize a nice distribution of Poll directions when compared to other strategies, and second, to introduce variants of each algorithm which utilize a minimal positive basis at each step.The strategies base their Poll directions on the use of the QR decomposition to obtain an orthogonal set of directions or on using the equal angular directions from a regular simplex centered at the origin with vertices on the unit sphere. Test results are presented on a set of smooth, nonsmooth, unconstrained, and constrained problems that give comparisons between the various implementations of these directional direct-search methods.


Introduction
In [1], Vicente and Custódio introduced a framework for directional direct-search methods (DSM) encompassing both the Mads [2] class of algorithms (using an underlying mesh and simple decrease) and the Gss [3] class of algorithms (using sufficient decrease under a forcing function ) for black-box optimization problems of the form min ∈Ω  () , where the objective function  : Ω ⊆ R  → R ⋃{∞} is nonsmooth, possibly discontinuous and Ω is the set of feasible points.The algorithms under this framework are shown to have first order convergence results in the Rockafellar sense [4] for discontinuous functions.The framework is shown in Algorithm 1.
While both algorithms fall under this framework and are shown to have the same convergence results, there are tradeoffs for either choice.The Mads class of algorithms not only requires a bit more complexity in choosing polling directions, but also requires simple decrease in order to accept an iteration as successful.Under the given framework, any instance of Gss requires sufficient decrease using a forcing function that must be chosen under unknown criteria.
The sufficient decrease condition, however, can be implemented in such a way that it is mostly negligible and the method for choosing polling directions is simpler because of the absence of a mesh.
Each iteration of this DSM framework is characterized by two steps, a Search and a Poll.The Poll step may be opportunistic (the Poll stops if a successful point has been found) and is governed by a Poll size parameter   (Mads denotes this Δ   ) such that lim inf  → ∞   = 0 under certain assumptions.These assumptions include the requirement that at each iteration either the set of Search and Poll points lie on an integer mesh (Mads) or in order for the iteration to be a success the algorithm must find a point that satisfies sufficient decrease for some forcing function  : (0, ∞) → (0, ∞) that is continuous, nondecreasing with ()/ → 0, when  ↓ 0 as shown in Algorithm 1.The use of  in the figure indicates the use of either a forcing function  (sufficient decrease) or the zero function (simple decrease).
In [2], Audet and Dennis introduced the mesh adaptive direct-search (Mads) class of algorithms for (1), where the objective function  either has no available derivative information or the derivative information is unreliable and Ω is the set of feasible points determined by black-box constraint functions.Under certain assumptions, Mads is (0) Initialization  0 ∈ Ω,  0 > 0, 0 <  1 ≤  The directional direct-search method of [1].
shown to converge to both first-order [2] and second-order [5] stationary points in the Clarke sense [6], as well as the previously mentioned first-order stationary points in the Rockafellar sense.
Mads was introduced as a means of extending the Gps [7] class of algorithms by allowing for an infinite number of polling directions when exploring the space of variables.The first instance of Mads called LtMads [2] relied on a random lower triangular matrix to generate the set of Poll directions and relied on probabilistic arguments to obtain convergence results.This instance was later shown to have undesirably large angles between Poll directions at some iterations.To correct these deficiencies, a second instance of Mads called OrthoMads [8] was introduced that is deterministic and uses a maximal orthogonal basis at each iteration for the set of polling directions.
Each iteration of Mads is characterized by two steps, a Search and a local Poll, performed on a mesh whose fineness approaches zero in the limit.At each iteration the mesh   is defined by where   is the set of all evaluated points at the start of iteration , Δ   is the mesh size parameter at iteration , and  ∈ R ×  is a matrix with   directions in R  .The Poll step is executed if the Search step fails to find a lower objective function value than the current best solution.The Poll step evaluates mesh points adjacent to the current best solution in directions that form a positive spanning set for R  .The Poll step can be characterized by the following definition taken from [2].Definition 1.At iteration , the set of Poll trial points for the Mads algorithm is defined to be where   is a positive spanning set such that 0 ∉   and for each  ∈   , (i)  can be written as a nonnegative integer combination of the directions of , (ii) the distance in ℓ ∞ -norm from the Poll center   to a Poll trial point is bounded above by the Poll size parameter Δ   , and (iii) limits (as defined by Coope and Price [9]) of the normalized sets   are positive spanning sets.
In [2] it is observed that the Poll and mesh size parameters produced by a Mads instance satisfy lim inf OrthoMads [8] uses the Halton sequence [10] and the scaled Householder transformation,  = ‖‖ 2 (  − 2VV  ) with  ∈ R  and V = /‖‖, to construct an orthogonal basis for R  at each iteration.It defines where ℓ  is the mesh index, with ℓ 0 = 0 at the initial iteration.OrthoMads has been shown to behave well in practice, but recently a new variant of Mads, called QrMads [11], was introduced to counter the tendency of OrthoMads to generate a nonuniform set of polling directions.QrMads augments a single direction from a dense sequence on S −1 with a matrix of full rank and then uses the QR decomposition to construct an orthogonal basis that is then rounded to the current mesh.It defines where ℓ  satisfies and is chosen so that after rounding the basis does not become degenerate.
Although QrMads was shown to have a more uniform set of polling directions than OrthoMads, the version tested in [11] is not deterministic.This is due to the fact that the augmented matrix with full rank was generated at random.Consequently, one outcome of this paper is to introduce a deterministic version of QrMads by deterministically generating a matrix with full rank.
It has been observed, in some cases, that reducing the number of function evaluations at every iteration from a maximal positive spanning set to a minimal positive spanning set can improve the performance of these types of algorithms (see, e.g., [12][13][14]).For this reason, this paper introduces instances of Mads and Gss, called EadMads and EadGss, that have versions using both a maximal positive basis (2 polling directions) and a minimal positive basis ( + 1 polling directions).This is accomplished by generating either an orthogonal basis or a regular simplex at each iteration.In the case of the maximal variant, the algorithmic framework for EadMads is exactly the same as QrMads and the new name supersedes the old.The new name was chosen to represent the use of a set of directions, for each Poll step, that have an "equal angle distribution." By this it is meant that within the positive basis, the angle between any two directions (except a direction and its opposite, when applicable) is constant.
Section 2 gives the methods for constructing maximal and minimal polling sets by finding either an orthogonal set of directions or the vertices of a regular simplex centered at the origin on the unit sphere.Section 3 gives the method used by EadMads for constructing a polling set on the current mesh consisting of "nearly orthogonal" directions or a "nearly regular" simplex.Section 4 outlines the EadMads algorithm and shows that it is a valid Mads instance that shares the theoretical convergence results of this class of algorithms.Section 5 outlines the EadGss instance of Gss which is essentially the same as EadMads except that it requires sufficient decrease instead of restricting the Search and Poll points to a mesh.Finally, in Section 6, a method for making both EadMads and EadGss deterministic is detailed and numerical results comparing each against OrthoMads, QrMads, and one another on a set of test problems taken from the optimization literature are given.

Constructing a Basis
At each iteration of the Mads algorithm (and DSM framework) a positive spanning set   is required.In this section, we discuss a method for taking a single direction from a sequence that is dense on the unit sphere and forming either an orthogonal basis or the set of vertices of a regular simplex on the unit sphere centered at the origin that includes the given direction.We can then take the negatives of all the directions from the orthogonal basis to form a maximal positive spanning set or take the vertices of the regular simplex to form a minimal positive spanning set.Since one of the directions from the dense set is included in each positive spanning set, we will then have ⋃ ∞ =1   as dense in the unit sphere.

Orthogonal Basis.
To generate an orthogonal basis from a given direction   from a dense set, we employ the QR decomposition of an  ×  matrix   of full rank.This matrix   can be created by augmenting   with any matrix of full rank (e.g., the identity or a rotation matrix) and taking the first  columns that form a basis.The QR decomposition then generates unique orthogonal matrix   and upper triangular matrix   such that     =   .As this method will generate the same basis as the Gram-Schmidt process, the first direction will correspond to   .

Regular Simplex.
To generate a regular simplex,   , centered at the origin with vertices on the unit sphere from a given direction   from a dense set, we again start from an  ×  matrix   of full rank, created in the same manner as above, and use the following algorithm.
Let   ∈ R × , with   ∈ S −1 ,  = 1, . . ., , where   is the th column of   and {  }  =1 is linearly independent.We construct a regular simplex,   , with vertices V 1 , . . ., V +1 on the unit sphere by using the fact that V  ⋅ V  = −(1/) for any two vertices of the simplex.This can be done as follows.
(iii) Set Proposition 2. V +1 is the unique solution to the following equation: Due to rounding issues, this algorithm has the same unfortunate computational problems as the Gram-Schmidt process and may give an answer that does not closely approximate a regular simplex, possibly even containing complex directions.To ensure that a valid regular simplex is generated, an orthogonal matrix can be used in place of an arbitrary matrix of full rank.This can be accomplished by finding the QR decomposition of   and then applying the algorithm to the orthogonal matrix   .In the following proof, we will simply assume that   is already orthogonal.
Proof.To find the solution to (10) for each , we employ the method of Lagrange multipliers.First we define the Lagrangian function as follows: We then solve the system of the following equations: First, we multiply ( 12) by V  and use ( 14) to obtain We can then solve for  +1 to obtain From (12) and by induction, we can see that for each , V  ∈ span{ 1 , . . .,   }.Consequently,  +1 ⋅ V  = 0,  = 1, . . ., .We can then reduce (16) to Using , we solve Now we can substitute ( 18) into ( 17) and simplify to obtain Next, we substitute ( 18) and ( 19) into ( 12) to obtain Then we take the norm of each side and use 13), and the fact that ‖ +1 ‖ = 1 to obtain the quadratic equation We then get  1 = ±1/ √ ( −  + 1) 2 − ( −  + 1)/.
Note, the above computation shows that when orthogonal directions are used, problem (10) has a unique solution.This is reflected by the fact that the above quadratic equation has two real solutions, one corresponding to the closest point to  +1 within the constrained region and the other corresponding to the farthest point.In order to ensure that the process for generating a regular simplex is deterministic, it is important to note that this unique closest point exists and is found.We can also observe that each new vertex V +1 of the regular simplex is linearly independent to the set {V  }  =1 , showing that the first  vertices of the simplex form a basis for R  .

Rounding a Basis to the Mesh
Since each iteration of the Mads class of algorithms requires a positive spanning set to lie on a mesh, it is necessary to take any basis from Section 2 and obtain a close approximation to that basis lying on the given mesh.To obtain a "nearly orthogonal" integer basis or a "nearly regular" simplex with integer vertices we take each column vector   of   or the first  columns V  of   , then project onto the hypercube centered at the origin with integer side length 2 ⋅ 2 |ℓ  |+2ℓ  , and then round each component to the nearest integer as follows: where round(⋅) refers to the usual rounding operation applied to each component of the vector.We then generate the new basis   with column vectors   .When   is formed from to get a minimal positive spanning set.Note, for small values of 2 |ℓ  | , it is possible that the new set of vectors {  : 1 ≤  ≤ } will not form a basis.Consequently, we add the constant 2ℓ  to the exponent in order to ensure that   will be nonsingular.In [11], it was shown that when   is formed from   and ℓ  satisfies (7), the resulting set of directions will form a basis.
To find the constant ℓ  when   is obtained from   , we use the fact that the distance from a matrix  to the nearest singular matrix  (in Frobenius norm) is bounded below by the smallest singular value of ; that is, if  is singular, then ‖ − ‖ ≥   ().Thus we need the singular values of the matrix comprised of the first  columns of   .Proposition 3 gives this information.(Note: in [13] the matrices  and    shown below are studied in a similar context.)Proposition 3. Let  ∈ R × be a matrix with columns corresponding to any  vertices of a regular simplex centered at the the origin with vertices on S −1 .Then the singular values of  are  1 = √( + 1)/ and  2 = √( + 1)/, . . .,   = 1/√.
Proof.We need to find the eigenvalues of   .We use the fact that the vertices, V 1 , . . ., V +1 , of a regular simplex centered at the origin on S −1 have the property to obtain Then observe that showing that    has eigenvalue 1/.Next, observe that showing that ( + 1)/ is an eigenvalue with eigenvector Similarly, are also eigenvectors with eigenvalue ( + 1)/.Since these vectors are linearly independent, they form a basis for the corresponding eigenspace, thus accounting for all  eigenvalues.We finish by taking the square roots to obtain the singular values of .
Next, we need the fact that for any two  ×  matrices  and ,   () ≥   ()  (), where   denotes the smallest singular value (follows from Theorem 3.1 of [21]).Then letting  be the matrix with columns we obtain   () =   () ≥   ()  () ≥ 2 |ℓ  |+2ℓ  (1/√).So for  +  to be nonsingular, we need where  represents the perturbation matrix resulting from rounding.Thus we require for all values of ℓ  .Then letting ℓ  = 0, we obtain the following requirement for the constant ℓ  determined only by the dimension : Finally, we end with the following proposition from [11] which will be useful for showing EadMads as a valid instance of Mads and for establishing convergence results.regular" simplex with the length of each column in ℓ ∞ -norm bounded by Δ   /Δ   .In the case of the orthogonal basis,   is completed to a maximal positive basis composed of the 2 directions given by   = [  −  ].In the case of the regular simplex,   is completed to a minimal positive basis composed of the  + 1 directions given by   = [  − ∑  =1   ].This then forces the trial point   + Δ       to be exactly 2 −ℓ  from the Poll center in ℓ ∞ -norm (except possibly in the case of the direction − ∑  =1   ).The following proposition from [11] will be useful to show that EadMads has the same convergence properties as in [2].(23), is dense on S −1 .

Proposition 5. The set of normalized directions {𝑑
Proposition 6 was proven in [11] when EadMads uses 2 polling directions (  is generated from   ) and so only the proof for the  + 1 version (  is generated from   ) is included.

Proposition 6. EadMads is a valid Mads instance with the convergence properties from [2] (and [1]).
Proof.We need to show that the Poll directions for EadMads satisfy the following five properties from [2,22].
(i) Each   is a positive spanning set: this follows from the use of the constant ℓ  to ensure that the round procedure on the first  columns of   results in a basis.
( iterations is dense on the unit sphere: this follows from the strategy chosen for updating   so that the entire sequence {  } ∞ =1 corresponds to the sequence of failed iterates, by Proposition 5, and (4) which ensures the existence of large |ℓ  |.
(v) Limits (as defined by [9]) of convergent subsequences of the normalized sets   = {/‖‖ :  ∈   } are positive spanning sets: for this property we need to show det(  ) > , for all , for some constant  > 0.
The result will then follow as in [22].
First, note that the set S = { ∈ R × : columns of  correspond to  vertices of a regular simplex centered at the origin on the unit sphere} is bounded since S ⊂ { ∈ R × : ‖‖ = √} ⊂ (0, √ + ), where ‖ ⋅ ‖ denotes the Frobenius norm on R × and (0, √ + ) is the ball of radius √ +  centered around the matrix of zeros.Also, the set   = { : ‖ − ‖ ≤  for any  ∈ S} is bounded since any such  would satisfy ‖‖ ≤ √ +  and so   ⊂ (0, √ +  + ).Now, since the determinant function is  1 Finally, note that where ℓ  is defined above.We then let and conclude that det (  ) > , for all .

The EadGss Instance of a Directional Direct-Search Method
At each iteration, the EadGss instance of Gss uses a positive spanning set whose directions have an equal angle distribution either by using the QR decomposition to generate an orthogonal basis or by generating a regular simplex centered at the origin.The integer   and the update rules are chosen so that there will be a subsequence of unsuccessful iterations with   → 0 and such that the directions used in this subsequence will correspond to the entire sequence {  } ∞ =1 , where {  } ∞ =1 is dense on S −1 .The Poll size parameter is given by   = 2 −ℓ  .The update rules for ℓ  are as in Algorithm 3, which describes the EadGss algorithm.The Poll directions   are determined by using any sequence {  } ∞ =1 that is dense on the unit sphere (or the hypercube [−1, 1]  ), the index ℓ  , a sequence of × matrices of full rank, and the index   .At each iteration, a direction from the dense sequence    is augmented with a matrix to obtain a basis and then either QR decomposition or the algorithm from Section 2.2 (along with the QR decomposition) is used to find a basis matrix with the direction from the first column corresponding to    .The resulting matrix will then be either orthogonal,   , or correspond to  vertices of a regular simplex,   , with the length of each column having ℓ 2 -norm one.In the case of the orthogonal basis,   is completed to a maximal positive basis composed of 2 directions,   = [  −  ].In the case of the regular simplex, the first  columns of   are completed to a minimal positive basis composed of +1 directions by finding the final vertex of the simplex, where {V  }  =1 are the columns of   .
It should be noted that this algorithm is very similar to that for EadMads (Algorithm 2), with the exception that EadGss uses sufficient decrease at each iteration instead of confining the Search and Poll steps to a mesh.Consequently, there are only two main differences.The first is that in the final step of the construction of   , EadMads rounds and EadGss does not (hence no need for separate   for the 2 and  + 1 variants).The second difference is in the determination of a successful iterate; EadGss uses a forcing function to determine sufficient decrease and EadMads uses simple decrease.

Numerical Tests
In this section, numerical test results are given to compare various instances of Mads and Gss amongst one another.Each test was performed using a MATLAB implementation.The tests were performed on 130 problems taken from the optimization literature, with dimensions between nine and 60.The test problems were chosen to satisfy three categories.The first category consists of 33 smooth functions, the second category consists of 60 nonsmooth functions, and the third category consists of 37 constrained test problems, as listed in Table 1.Since many of the problems are variably dimensioned versions of one another, no generalization was used more than three times.The distribution of problems by dimension is given in Table 2.
The choices for Δ   and Δ   for OrthoMads correspond to (5), for the 2 versions of EadMads they correspond to ( 6) and (7), and for the +1 version of EadMads they correspond to (33) and (31).For each version of EadGss   = 2 −ℓ  .The mesh index ℓ  is allowed to be negative for all instances of both algorithms and for each instance of Mads the initial mesh size Δ  0 is set to one.The stopping criteria are satisfied when the number of function evaluations reach 3,000.For each implementation, a Poll step was required at every iteration and an opportunistic strategy was used (the Poll stops if a successful point has been found).For each test, no search was performed and constraints are handled using the extreme barrier method (() = +∞ if  ∉ Ω).
In order to create and test deterministic versions of both EadMads and EadGss, the Sobol sequence [23] is used in place of either the Halton sequence or the sequence of partition centers used for QrMads.This is done for two reasons.Although the Halton sequence is deterministic, the sequence of partition centers is not.In higher dimensions, however, it can be observed that the Halton sequence exhibits poor uniformity.Sobol sequences, on the other hand, are both deterministic and remain uniform.For each test, the Sobol sequence was generated using the sobolset command in MATLAB and used in the given order.
At each iteration    is taken to be the  ℎ  Sobol direction.This is then augmented with the next 2 − 1 Sobol directions, taken in order, and the identity matrix.This is done in order to ensure that a matrix of full rank is created.From this matrix, a new matrix is formed from the first  columns that form a basis for R  and then the QR decomposition is applied to this new matrix.When a minimal positive basis is required, the resulting orthogonal matrix is then input into the simplex algorithm of Section 2.2.For EadMads, these directions are then rounded to the current mesh.In place of the mesh, EadGss uses a sufficient decrease requirement implemented with the forcing function  =  2 .To compare the various instances of each algorithm, the results from the tests were used to generate data profiles, as outlined in [24].For each test function,   is defined to be the smallest value found for  by any of the instances of either Mads or Gss after 3,000 function evaluations.Then after each evaluation, the algorithm is declared a success if the smallest known value for  thus far satisfies the test  () ≤   + 0.001 ( ( 0 ) −   ) . (41) The number of successes over all the test functions is then recorded as a percent of success.The results for each instance of Mads or Gss are then plotted as the number of function evaluations against the percent of problems to obtain the data profile.
The first test performed was to compare both the nondeterministic and deterministic instances of EadMads with 2 polling directions against OrthoMads.The data profiles for the deterministic instance of EadMads, five runs of the nondeterministic instance of EadMads (formerly called QrMads), and OrthoMads are given in Figure 1.A data profile is given for each instance using the set of all test functions, the set of smooth test functions only, the set of nonsmooth test functions only, and the set of constrained test functions only.In this figure, we can see that the deterministic instance of EadMads performs very well compared to the other two instances when considering the set of all test functions or just the set of nonsmooth functions.When just considering the set of smooth or constrained test functions, the deterministic instance of EadMads performs roughly as well as the nondeterministic instance and better than OrthoMads.
Figure 2 illustrates the distribution and the density of directions for all three Mads instances.Each instance was run on the Rosenbrock function with dimensions  = 10 and 20.In each case, there are roughly 500 directions shown and each direction is projected onto the plane defined by the first two coordinates.The resulting plots are typical of what can be expected regardless of the choices of the two coordinates.In this figure, we can see that the directions for both the deterministic and nondeterministic instances of EadMads have relatively uniform distributions with little clustering.The directions for OrthoMads, on the other hand tend to cluster within specific regions.While the directions for both instances of EadMads are very similar, with the unit sphere being covered nicely, the test results from this section indicate that the deterministic instance is expected to perform as well as or better than the nondeterministic instance (QrMads).Note 1.In Figures 2, 4, and 6, the EadMads and EadGss directions become more restricted towards the center as the dimension increases.This behavior is a reflection of the concentration of measure phenomenon and the fact that the observable diameter of S −1 converges to zero as  → ∞ (for more on this topic see [25]).Intuitively, one can imagine that as n increases, the contribution to the norm of a vector from the first two elements gets smaller.
The next test performed was to compare the deterministic instance of EadGss with 2 polling directions to a nondeterministic instance of Gss with 2 polling directions using random directions in place of the Sobol sequence (this is essentially the algorithm tested in [1]).The data profiles for the deterministic instance of EadGss and five runs of the nondeterministic instance are given in Figure 3.A data profile is given for each instance using the set of all test functions, the set of smooth test functions only, the set of nonsmooth test functions only, and the set of constrained test functions only.From this figure, we can observe that the deterministic instance performed better on the sets containing all the test functions, the nonsmooth test functions, and the constrained test functions.Both the deterministic and nondeterministic instances show similar performances on the set of smooth test functions.
Figure 4 illustrates the distribution and the density of directions for both instances of Gss.Each instance was run on the Rosenbrock function with dimensions  = 10 and 20.Similar to what is observed in Figure 2, we can see that the directions from these two instances of EadGss have relatively uniform distributions with little clustering.
The last test performed was to compare the deterministic instances of EadMads and EadGss with both  + 1 and 2 polling directions.The data profiles for each instance of the two algorithms are given in Figure 5.A data profile is given for each instance using the set of all test functions, the set of smooth test functions only, the set of nonsmooth test functions only, and the set of constrained test functions only.From this figure, we can observe that the  + 1 instances of both EadMads and EadGss perform better than the 2 instances on both the set of all test problems and the set of smooth test problems.We can see a lot of parity amongst the four instances on the set of nonsmooth functions with the two  + 1 instances doing a bit better as the number of function evaluations approaches 3,000.The results from the set of constrained test functions are a little more mixed with the  + 1 instance of EadMads performing very well overall and the  + 1 instance of EadGss performing well as the number of function evaluations approaches 3,000.Figure 6 illustrates the distribution and the density of directions for the  + 1, deterministic instances of EadMads and EadGss.Each instance was run on the Rosenbrock function with dimensions  = 10 and 20.In this figure, we can observe that the overall distribution of directions for the  + 1 instances of EadMads and EadGss is very similar to one another and while they do exhibit some clustering, they still have relatively uniform distributions overall.

Conclusions
When choosing a direct-search method, often two desirable attributes have a deterministic algorithm and the ability to use a minimal positive spanning set.This paper has outlined strategies for making both the Mads and Gss algorithms deterministic, ensuring that the end results are repeatable, and outlined a method for using a set of  + 1 directions generated from a regular simplex.These strategies were devised in a way that modifications are readily available without losing convergence results.For example, one could easily substitute any deterministic sequence that is dense on the unit sphere into the framework presented in this paper.Consequently, an area of future inquiry could include studying the behavior of these algorithms using a variety of different sequences.
It was not the direct intention of this paper to make a comparison of the Mads algorithm and Gss algorithm using sufficient decrease.Although these two algorithms are similar, they each have advantages and disadvantages.The advantage to the Mads class of algorithms is that it only requires simple decrease, with the disadvantage being that it has the added complexity of requiring Poll and Search points to lie on a mesh at each iteration.Alternately, Gss has the advantage of eliminating the mesh requirement but is burdened by needing a sufficient decrease requirement before accepting a new best point.One can imagine that the advantages of the two algorithms may be more desirable for different real-world applications.Consequently, an area of future study could include a more in-depth comparison of these two algorithms.
In this paper, constraints are handled using the extreme barrier method.Recently, however, there have been suggestions for better ways to handle constraints, such as the progressive barrier method [26].Another area of future study would include analyzing the methods from this paper in conjunction with the progressive barrier method, as well as other extensions of the Mads algorithm.

Figure 1 :
Figure 1: Data profiles for three instances of Mads, including five runs of QrMads, on 130 test functions.

Figure 2 :
Figure 2: First two coordinates of the normalized Poll directions for three variants of the Mads algorithm on the -dimensional Rosenbrock function with  = 10 and 20.

Figure 3 :
Figure 3: Data profiles for two variants of EadGss, including five runs of EadGss with random directions, on 130 test functions.

Figure 4 :
Figure 4: First two coordinates of the normalized Poll directions for two variants of EadGss on the -dimensional Rosenbrock function with  = 10 and 20.

Figure 5 :
Figure 5: Data profiles for four instances of EadMads on 130 test functions.

Figure 6 :
Figure 6: First two coordinates of the normalized Poll directions for the variants of EadGss and EadMads using the Sobol sequence and  + 1 polling directions on the -dimensional Rosenbrock function with  = 10 and 20.

Table 1 :
List of test problems where S: smooth, N: nonsmooth, and C: constrained.

Table 2 :
Distribution of test problems used for the numerical results.