A New Approach to Estimate the Critical Constant of Selection Procedures

A solution to the ranking and selection problem of determining a subset of size m containing at least c of the v best from k normal distributions has been developed. The best distributions are those having, for example, i the smallest means, or ii the smallest variances. This paper reviews various applicable algorithms and supplies the operating constants needed to apply these solutions. The constants are computed using a histogram approximation algorithm and Monte Carlo integration.


Introduction
Discrete-event simulation has been widely used to compare alternative system designs or operating policies.When evaluating k alternative system designs, we select one or more systems as the best and control the probability that the selected systems really are the best.This goal is achieved by using a class of ranking and selection R&S procedures in simulation see 1 for a detailed description .Chen 2, 3 considered a general version of this problem to select a subset of size m containing at least c of the v best of k normally distributed systems with the lth smallest θ i l mean or variance , where θ i 1 ≤ θ i 2 ≤ • • • ≤ θ i k represent proposed sampling solutions.Moreover, in practice, if the difference between θ i v and θ i v 1 is very small, we might not care if we mistakenly choose system i v 1 , whose expected response is θ i v 1 .The "practically significant" difference d * a positive real number between a desired and a satisfactory system is called the indifference zone in the statistical literature, and it represents the smallest difference about which we care.
If the absolute difference is θ i v 1 − θ i v < d * or the relative difference is θ i v 1 /θ i v < d * , we say that the systems are in the indifference zone for correct selection.Note that the absolute difference and the relative difference are two completely separate problems.On the other hand, if the absolute difference is θ i v 1 − θ i v ≥ d * or the relative difference is θ i v 1 /θ i v ≥ d * , we say that the systems are in the preference zone for correct selection.
Let P * denote the required minimal probability of correct selection.The goal is to make a correct selection CS with probability at least P * , where 1.1 If P * < P c, v, m, k , the precision requirement is satisfied by choosing the subset at random.The minimal correct selection probability P * and the "indifference" amount d * are specified by the user.If c v m 1, the problem is to choose the best system.When m > c v 1, we are interested in choosing a subset of size m that contains the best.If c v m > 1, we are interested in choosing the m best systems.This paper proposes a new approach to estimate the operating constants needed to apply the solutions in 2, 3 .We first review these procedures in Sections 2 and 3. We then describe the histogram approximation algorithm of Chen and Kelton 4 and the Monte Carlo integration technique used to compute tables of operating constants in Section 4. A brief illustration demonstrating how to apply the tables is provided in Section 5.

Selection Problems with Respect to Means
Consider k independent normal distributions having means μ i and variances σ 2 i , i 1, 2, . . ., k.It is assumed that μ i and σ 2 i are unknown.Selection procedures generally sample certain observations from each alternative at the initial stage and select the systems having the best sample means as the best systems.The question that arises is whether enough observations have been sampled and if not, the number of additional observations that are needed.Hence, at the second stage of the procedure, the required number of observations is generally estimated based on the available sample variances and sample means.
Extending the work of Dudewicz and Dalal 5 and Mahamunulu 6 , Chen 3 proposed a two-stage solution when the parameter of interest θ i is μ i .Chen's procedure is as follows.Let X ij , i 1, 2, . . ., k; j 1, 2, . . ., n i be the jth observation from the ith population.
Randomly sample n 0 observations from each of the k populations.Let X Compute where h is a constant to be described later and z denotes the integer ceiling round-up of the real number z. Randomly sample an additional n i − n 0 observations from the ith population.
We then compute the second-stage sample means: Define the weights and W i2 1 − W i1 , for i 1, 2, . . ., k. Compute the weighted sample means and select the m systems with the smallest X i values.Note that the expression for W i1 was chosen to guarantee that where is the beta distribution with shape parameters a and b.In our case, f and F are, respectively, the pdf and cdf of the t distribution with n 0 − 1 d.f.For selection problems with respect to means, the least favorable configuration Mahamunulu 6 .Let X c be the cth smallest weighted sample mean from X i l for l 1, 2, . . ., v and μ c be its unknown true mean.Let X u be the uth u m − c 1 smallest weighted sample mean from X i l for l v 1, v 2, . . ., k and μ u be its unknown true mean.We can write the probability of correct selection as

2.7
The inequality follows because We equate the right-hand side to P * to solve for h.The value of h is determined by P T c ≤ T u h P * .Let τ T c − T u .Then P τ ≤ h P * .That is, under the LFC, the value of h is the P * quantile of the distribution of τ.
For example, if we are interested in the probability of correctly selecting a subset of size 5 containing 3 of the first 3 best from 10 alternatives, then T c ∼ g 3,3 t c and T u ∼ g 7,3 t u .Furthermore, if the initial sample size is n 0 20, then f and F are, respectively, the pdf and cdf of the t-distribution with 19 d.f.
Let w c u denote the one-tailed P * confidence interval c.i. half-width of μ u − μ c .We conclude that the sample sizes allocated by 2.1 achieve w c u ≤ d * .That is, the indifference amount d * in 2.1 corresponds to the upper bound of the desired c.i. half-width w c u .Hence, under the LFC μ c d * μ u , the sample sizes allocated by 2.1 ensure that

2.9
The last equality follows from the definition of the c.i.Note that if w c u > d * , then P CS < P * .It can be shown that Koenig and Law 8 provide some h values for the case that c v 1 or c v m.This paper supplies a table of the h values with selected c, v, m, and k; where c, v, and m may be different.

Selection Problems with Respect to Variances
Extending the work of Bechhofer and Sobel 9 and Mahamunulu 6 , Chen 2 proposed a single-stage solution when the parameter of interest θ i is σ 2 i .Chen's procedure is as follows.Let n i n 0 for i 1, 2, . . ., k.Thus, we use the notation s 2 i instead of s 2 i n i in the rest of this section.Randomly sample n 0 observations from each of the k populations and compute Select the m systems with the smallest s 2 i .For selection problems with respect to variances, the LFC occurs when The derivation is based on the fact that n 0 − 1 s 2 i /σ 2 i has a χ 2 distribution with n 0 − 1 d.f.Let s 2 c be the cth smallest sample variance from s 2 i l for l 1, 2, . . ., v and σ 2 c be its unknown true variance.Let s 2 u be the uth u m − c 1 smallest sample variance from s 2 i l for l v 1, v 2, . . ., k and σ 2 u be its unknown true variance.Then

3.1
The third equality follows because 6

Advances in Decision Sciences
Let ω and Ω, respectively, denote the pdf and cdf of the χ 2 distribution with

3.3
We can compute P CS values under the LFC given k, m, v, c, d * , and n 0 .Bechhofer and Sobel 9 provide some P CS for the cases that k ≤ 4. We provide additional P CS values for some selected parameters in Section 4.2.
Let γ X c /X u .There exists some 0 ≤ p ≤ 1 such that P γ ≤ d * p.Hence, under the LFC, the value of d * is the p quantile of the distribution of the random variable γ.For example, if we are interested in the probability of correctly selecting a subset of size 5 containing all 3 of the first 3 best from 10 alternatives, then X c ∼ g 3,3 x c and X u ∼ g 7,3 x u , with f and F replaced by ω and Ω, respectively.
If users prefer to specify the indifference amount d * in the absolute form, d a θ, θ 0 θ − θ 0 , instead of the relative form, d r θ, θ 0 θ/θ 0 , when the parameter of interest is a scale parameter, we can transform the absolute indifference amount into the relative indifference, d r θ, θ 0 1 d a θ, θ 0 /θ 0 .Since θ 0 is unknown, the estimator θ 0 needs to be used and d r θ, θ 0 ≈ 1 d * / θ 0 .Moreover, a conservative adjustment can be used.Rank the sample variances such that s Let y q be the q quantile of the χ 2 distribution with n b v − 1 d.f., where 0 < q < 1.We can conservatively set d r σ Conversely, if users prefer to specify the indifference amount in the relative form instead of the absolute form when the parameter of interest is the location parameter, we can set d a θ, θ 0 ≈ d * − 1 θ 0 .

Method of Computation
Analytical solutions to multidimensional integration problems in the previous section are difficult to obtain.Below we show our approaches to find h and P CS .

Computing the Value of h
Recall that under the LFC the value of h is the P * quantile of the distribution of τ.Consequently, we can use any quantile-estimation procedures to estimate the P * quantile of the variable τ given k, m, v, c, and n 0 .In this section, we briefly review quantile estimates and the histogram-approximation procedure of Chen and Kelton 4 .
Let X 1 , X 2 , . . ., X n be a sequence of i.i.d.independent and identically distributed random variables from a continuous cdf F x with pdf f x .Let x p 0 < p < 1 denote the 100pth percentile or the p quantile, which has the property that F x p Pr X ≤ x p p. Thus, x p inf{x : F x ≥ p}.If Y 1 , Y 2 , . . ., Y n are the order statistics corresponding to the X i 's from n independent observations i.e., Y i is the ith smallest of X 1 , X 2 , . . ., X n , then a point estimator for x p based on the order statistics is the sample p quantile: x p y np . 4.1 Chen and Kelton 4 control the precision of quantile estimates by ensuring that the p quantile estimator x p satisfies the following: Using this precision requirement i.e., 4.2 , the required sample size n p for a fixed-samplesize procedure of estimating the p quantile of an i.i.d.sequence is the minimum n p that satisfies where z 1−α 1 /2 is the 1 − α 1 /2 quantile of the standard normal distribution, is the maximum proportional half-width of the c.i., and 1 − α 1 is the confidence level.For example, if the data are independent and we would like to have 95% confidence that the coverage of the 0.9 quantile estimator has no more than 0.0005 deviation from the true but unknown quantile, the required sample size is n p ≥ 1382976 1.960 2 0.9 1 − 0.9 /0.0005 2 .Consequently, we are 97.5% confident that the quantile estimate will cover at least p − 0.0005 for p ≥ 0.9 , with a sample size of 1382976.
The histogram-approximation procedure sets up a series of grid points based on a pilot run.New samples are then stored in the corresponding grids according to their observed value.A histogram is created at the end of the procedure when it has processed the required sample size.The p quantile estimator is obtained by interpolating among grid points.Interested readers can see 4 for the detailed steps of the histogram-approximation procedure.
In the appendix, we show how to generate order statistics random variates without storing and sorting the entire sequence.In order to use this algorithm, we need to be able to perform an inverse transformation of the cdf of the random variable.Unfortunately, the inverse transformation of the cdf of the t-distribution and 2.8 are not available.Nevertheless, numerical methods are available to compute the inverse of the cdf of the tdistribution; see 10 .Hence, the variates T c and T u can be generated efficiently without sorting a series of t-distributed variables.
Table 1 shows the resulting h values for several chosen k, m, v, c, n 0 , and P * .Four significant digits are retained.Negative values indicate that P * can be achieved with a sample size of n 0 1 and are set to 0.

Computing the Probability of Correct Selection P(CS)
Monte Carlo integration can be used to approximately evaluate the integrals.Let hypercube V be the integration volume and hypercube V ⊆ V .Monte Carlo integration picks random uniformly distributed points over some simple domain V , which contains V , checks whether each point is within V , and estimates the area of V as the area of V multiplied by the fraction of points falling within V .Suppose that we pick randomly distributed points X 1 , . . ., X n in d-dimensional volume V to determine the integral of a function f in this volume: where see Press et al. 11 .Note that V f 2 − f /n is a one standard deviation error estimate of the integral and g is a function to be specified depending on the problem at hand.
In our case V is the unit volume and g will be the indicator function of whether a correct selection was made.Let r i be the index of the ith simulation and If the target p 0.9 and n 1000000, then P p ≥ p − 0.0003 ≥ 0.84.We perform simulation experiments to estimate the value of the integrals.Table 2 shows the resulting probability of correct selection with four significant digits for several chosen k, m, v, c, and n 0 .

An Illustration
As a brief illustration of how to use Tables 1 and 2, consider 10 systems with θ i being the expected performance of the ith system, i 1, 2, . . ., 10.It is desired to select 4 systems such used instead of d * in the above equation, we obtain strict equality.Note that under the LFC, d r σ 2 u , σ 2 c d * .

Table 1 :
Values of h for the subset selection procedure.
correct selection was made in simulation r i , If we perform n independent simulation replications and the observed P CS is p, then Note that the number of times that the best design is selected from n simulation runs has a binomial distribution B n, p , where n ≥ 0 is the number of trials and 0 ≤ p ≤ 1 is the success probability.Furthermore, when n is large, B n, p can be approximated by the normal distribution N np, np 1 − p with mean np and variance np 1 − p 7 .Consequently,

Table 2 :
Values of P CS when n 0 20.