Efficient Simulation Budget Allocation for Ranking the Top m Designs

We consider the problem of ranking the top m designs out of k alternatives. Using the optimal computing budget allocation framework, we formulate this problem as that of maximizing the probability of correctly ranking the top m designs subject to the constraint of a fixed limited simulation budget. We derive the convergence rate of the false ranking probability based on the large deviation theory. The asymptotically optimal allocation rule is obtained by maximizing this convergence rate function. To implement the simulation budget allocation rule, we suggest a heuristic sequential algorithm.Numerical experiments are conducted to compare the effectiveness of the proposed simulation budget allocation rule. The numerical results indicate that the proposed asymptotically optimal allocation rule performs the best comparing with other allocation rules.


Introduction
Discrete event system (DES) simulation has been widely used for analyzing and evaluating complex systems since the assumptions for deriving an analytical solution are rarely satisfied in real situation.While DES simulation has been successful in solving many practical problems in a variety of areas such as supply chain systems, healthcare systems, and manufacturing systems, the concerns on the efficiency have never ended [1].To obtain a statistically significant value, a large number of simulation replications are needed for each design.The performance of each design is then estimated by its sample mean.The ultimate accuracy of this estimator cannot be improved faster than (1/ √ ), where  is the number of simulation replications.
Ordinal optimization (OO) which aims to obtain a good estimate through ordinal comparisons although the estimated value is still poor emerges as a way to improve the simulation efficiency [2].If the goal of the simulation experiment is to identify the good designs instead of finding the accurate estimate of the true performance value, which is true in many real applications, OO can reduce the number of simulation replications significantly [3,4].Intuitively, a larger portion of the total simulation replications should be allocated to those designs that are critical in identifying the best design in order to achieve a high probability of correct selection.Based on this idea, an optimal computing budget allocation (OCBA) framework has been proposed to enhance the simulation efficiency further [5,6].OCBA focuses on the efficiency of simulation by intelligently allocating further replications based on both the mean and the variance.In parallel with OCBA, two other well-known ranking and selection procedures frequently used in simulation are the indifference zone (IZ) procedure and value information procedure (VIP).The IZ procedure focuses on finding a feasible way to guarantee that the prespecified probability of correct selection can be achieved [7].The VIP uses the Bayesian posterior distribution to describe the evidence of correct selection and allocates further replications based on maximizing the value information [8,9].The three popular procedures of finding the best design are compared in [10].Extensions of the research in OCBA and ranking and selection include subset selection [11][12][13], selecting the Pareto set for multiple objective functions [14], selecting the best design subject to stochastic constraints [15,16], and complete ranking [17].A detailed summary of the existing work in OCBA can be found in [18].
To the best of our knowledge, no previous research has considered the simulation budget allocation for ranking the top  designs out of  alternatives.The developed top  ranking procedure is most useful when the designs for comparison have multiple dimensions of performance measurements with some qualitative criteria such as environmental consideration and political feasibility.Top  ranking procedure can help the decision makers to identify the ranking of the top  designs based on the quantitative performance measurement.Final decision can be made by incorporating other qualitative performance measurements.Hence, the top  ranking gives decision makers a more flexible and people oriented way to support the decision making process.In addition, the top  ranking procedure can also be integrated into some evolutionary search algorithms, where the ranking information of the top solutions is needed in order to determine the search direction of next iteration.For example, the ranking of the top candidate solutions may be needed in each iteration of the genetic algorithm since better candidate solutions are given higher probabilities to reproduce in genetic algorithms.
In this paper, we consider the problem of ranking the top  designs out of total  alternatives, where the performance value of each design can only be estimated with noise via simulation.The objective of this paper is to determine how to allocate the simulation replications among the  designs such that the probability of correctly ranking the top  designs can be maximized.The organization of this paper is as follows.Section 2 provides the mathematical formulation of the top  ranking problem using OCBA framework.In Section 3, we derive the convergence rate function of the false ranking probability.Section 4 derives the asymptotically optimal allocation rule using large deviation theory.A sequential allocation algorithm is proposed in Section 5. Section 6 conducts numerical experiments to demonstrate the effectiveness of the proposed simulation budget allocation rule.Finally, we conclude our paper in Section 7.

Problem Formulation
Consider the problem of ranking the top  designs out of  alternatives.The performance of each design can only be estimated with noise via simulation.The mean performance is used as the ranking criterion.In order to have a steady mean performance value, a large number of simulation replications are needed because of the randomness of individual samples.Given that a total of  simulation replications are available, the objective is to find the best allocation of the total  replications to the  designs in order to maximize the probability of correctly ranking the top  designs.
Given that  1 < ⋅ ⋅ ⋅ <   < ⋅ ⋅ ⋅ <   , the ranking of the top  designs is correctly identified if   (  ) ≤  +1 ( +1 ) for all  = 1, . . .,  − 1 and if   (  ) ≤   (  ) for all  =  + 1, . . ., .Mathematically, we can write the probability of correctly ranking the top  designs as follows: Hence, an optimal computing budget allocation problem can be formulated by maximizing the probability of correctly ranking the top  designs: Maximizing the probability of correctly ranking the top  designs is equivalent to minimizing the false ranking probability of the top  designs.Using asymptotical analysis, this is also equivalent to maximizing the convergence rate at which the false ranking probability goes to zero.In this paper, we use large deviation theory to derive this convergence rate function and reformulate the optimization model (2) by maximizing the convergence rate function.The assumptions needed in this paper are stated as follows.
Assumption 1.The performance of every design is independently simulated.
The independence of each design ensures that the samples ( 1 , . . .,  ,   ) for each  = 1, . . .,  generated are independent.Thus, the results we obtained will not be affected by the correlations among different designs.

Rate Function of the False Ranking Probability
We now derive the probability of false ranking other than the correct ranking probability, followed by the corresponding derivation of its large deviation principle.Recall that the probability of correctly ranking the top  designs is defined in (1).The probability of falsely ranking the top  designs is simply its complement; that is, where (⋅)  represents the complement of (⋅).(FR  ) has a lower bound, and an upper bound ub = ( − 1) × lb such that, assuming the limit exists, lim Theorem 3 formally states that the limit exists and the overall convergence rate function is the minimum rate function of each probability.The convergence rate can be understood as the speed at which false ranking probability goes to zero.
Then, it can be concluded that lim Now we are in the position to derive the assumed function   (⋅).Define   = (  (  ),  +1 ( +1 )).The cumulant moment generating function of   can be written as Under Assumption 2, lim By the Gärtner-Ellis Theorem [20], {  ,  = 1, 2, . ..} satisfies large deviation principle with good rate function which can be expressed as follows: Hence, from large deviation principle, lim Similarly, lim Therefore, the convergence rate function of the false ranking probability can be expressed as follows:

Asymptotically Optimal Allocation
The objective is to maximize the probability of correctly ranking the top  designs.This can be achieved by minimizing the false ranking probability.It is also equivalent to maximizing the convergence rate of (FR  ) subject to ∑  =1   = 1 and   ≥ 0, for all  = 1, . . ., .The optimization model in (1) can be reexpressed as max min { min By [19],     () +  +1  +1 () is a strictly increasing concave function.The infimum of concave functions is also concave.Likewise, the minimum of concave functions is a concave function too.Define (  ,  +1 ) = arg inf  (    () +  +1  +1 ()).As shown in [19] =  +1 ( (  ,  +1 )) . ( The result can similarly be applied to     () +     ().Therefore, the optimization model ( 16) is a concave maximization problem and it can be reexpressed as follows: Since model ( 18) is strictly concave and the functions of  are continuous, a unique optimal solution must exist and the Karush-Kuhn-Tucker (KKT) conditions are necessary and sufficient for global optimality.
From the KKT conditions on problem (18) ) is the optimal solution to (18) if and only if it is also an optimal solution to (19).
Proof.We assume that a point satisfying the KKT condition of ( 18) is also feasible to (19).We first prove the forward and backward assertions.We then prove that the assumption that a point satisfying the KKT condition of ( 18) is also feasible to (19) is indeed correct.
Suppose  * is the optimal solution to (18).Since the feasible region of ( 19) is a subset of that of (18), if the optimal solution to ( 18) is feasible to (19), it must be optimal to (19).Since the KKT conditions are necessary and sufficient for optimality in (18),  * must satisfy the KKT conditions of (18).Hence,  * is feasible to (19).Therefore, if a point satisfies the KKT condition in (18), it must be optimal to (19).
Suppose the optimal solution to ( 18) is  * and the optimal solution to ( 19) is α * , and  * ̸ = α * .Since the KKT conditions are necessary and sufficient condition to (18), thus,  * must satisfy the KKT conditions.Furthermore, the objective function of ( 19) is the same as that of (18), and the feasible region of ( 19) is a subset of that of (18).Therefore,  * must be infeasible to (19).However, we assumed that a point satisfying the KKT conditions of (18) must be feasible to (19).We have thus reached a contradiction.So we must have  * = α * .
Since the problem ( 18) is a concave optimization problem, the first order condition is also the optimality condition.

Sequential Allocation
Based on the results from Theorem 4, we can compute the simulation budget allocation rule using (28) for design performances with any arbitrary distributions or using (29) for normally distributed performances if the parameters of the design performance distribution are given.However, no information on the design performance distribution is known before simulation experiments are conducted.To overcome this dilemma, we suggest a heuristic sequential allocation algorithm in order to implement the allocation rule.The algorithm for sequential allocation is summarized in Algorithm 1.
Define  to be the iteration number and define    ,  = 1, . . .,  to be the total number of simulation replications that have been allocated to design  up to iteration . is the total number of simulation replications available.Δ is the number of incremental simulation replications for each iteration. 0 is the initial number of simulation runs for each design.
As the simulation continues, design  will be ranked number  for all  ≤ .The ranking of the top  designs may change from iteration to iteration although it will converge to the true ranking when the total computating budget goes to infinity.When the ranking of the top  designs changes, the budget allocation in the loop will be applied immediately.Therefore, the actual proportion of budget for every system will converge to the optimal proportion when the number of iterations is sufficiently large.
Furthermore, we need to take note of  0 , the initial number of replications for every design. 0 cannot be too small because the estimation of the rate function can be poor especially when the variance of the performance is large.On the other hand, if  0 is too large, some designs will be allocated excessively compared with their optimal allocation numbers.When the total budget is very limited, designs that need more replications may suffer from large  0 and this would eventually affect the simulation results.Other than the initial number of replications, the incremental budget Δ is also important in the implementation process.Large Δ results in wasting of budget, while small Δ will lead to expensive computation in the loop.

Numerical Experiments
In this section, we test the proposed simulation budget allocation rule for ranking the top  designs by comparing it with different allocation rules: equal allocation which simulates each design equally and the OCBA- procedure [12].Although OCBA- only considers the selection of the top  designs and do not aim to identify the ranking of the top  designs, we can use it here for benchmarking purpose.In all the experiments below, the performance of each design is assumed to follow the normal distribution.Therefore, the optimal allocation rule can be obtained by solving (29).
The assumption of normal distribution is generally held in simulation experiments since the output is obtained from an average performance or batch means, so that the central limit theorem effect holds.

Computing Budget Allocation Rules
Equal Allocation.The simulation replications are allocated equally to each design; that is,   = 1/,  = 1, . . ., .This is the simplest allocation rule and it can serve as a benchmark for other allocation procedures.
Top  Ranking (OCBA-Rm).The simulation budget allocation is derived using (29) and implemented using the sequential allocation algorithm proposed in Section 5.The allocation rule is solved by using the solver "fminimax" in Matlab.

Numerical Results for Different Allocation Procedures.
To compare the performance of the procedures, we carried out numerical experiments for the different allocation procedures discussed above.In comparing the procedures, the effectiveness of the procedures is measured by the probability of correctly ranking the top  designs ((CR  )) which is estimated by the fraction of the times that the procedure successfully identifies the correct ranking of the top  designs out of 10,000 independent simulation runs.Each of the procedures simulates each of the  designs for  0 = 20 replications initially as recommended in [1,12].The simulation budget is increased by Δ = 40 for each iteration.Table 1 summarizes the mean and variance for the three experiments that will be conducted in this section.In this experiment, the total number of designs is 20, that is, k = 20 and the objective is to identify the ranking of the top 5 designs, that is,  = 5.
Figure 1 shows the numerical comparison for the three allocation rules, where (a) is for the equal spacing scenario, (b) is for the equal variance scenario, and (c) is for the increasing spacing but decreasing variance scenario.
The experiment results show that the proposed simulation budget allocation rule OCBA-Rm performs the best in all three experiments.It is also interesting to note that OCBA-, which performs significantly better than EA when the objective is selecting the top  designs, fares much worse than EA in all three experiments when the objective is to identify the ranking of the top  designs.This is because OCBA- only focuses on distinguishing the set of the top  designs and the set of the nontop  designs without considering the ranking within the top  designs.Based on the numerical results, we can see that it is important to derive the OCBA-Rm when the ranking of the top  alternatives is needed.By using the proposed OCBA-Rm allocation rule, significant number of simulation budgets can be saved compared with EA and OCBA-.

Conclusion
In this paper, we study the problem of simulation budget allocation of ranking the top  designs out of  alternatives.
Based on the large deviation theory, we have derived the asymptotically optimal allocation rule to maximize the probability of correctly ranking the top  designs.In addition, a heuristic sequential allocation algorithm is suggested to implement the simulation budget allocation rule.Numerical experiments are conducted to compare the effectiveness of the proposed simulation budget allocation with some existing allocation procedures.The contribution of this paper is twofold.From a ranking and selection perspective, we offer a heuristic for ranking the top  designs out of  alternatives, where our empirical studies show that it can be more efficient than the existing methods.From the computing budget allocation perspective, our heuristic illustrates how the previous OCBA method for identifying the single best design can be modified to identify the ranking of the top  designs.In particular, we derive the asymptotically optimal allocation using large deviation theory, which allows us to remove the assumption that the performance of each design must be normally distributed.

Figure 1 :
Figure 1: Probability of correctly ranking the top  designs under three different allocation rules.

Table 1 :
Parameters for the numerical experiments.