Journal of Applied Mathematics

Volume 2013 (2013), Article ID 492421, 20 pages

http://dx.doi.org/10.1155/2013/492421

Review Article

A Review of Ranking Models in Data Envelopment Analysis

F. Hosseinzadeh Lotfi,¹ G. R. Jahanshahloo,¹ M. Khodabakhshi,² M. Rostamy-Malkhlifeh,¹ Z. Moghaddas,¹ and M. Vaez-Ghasemi¹

¹Department of Mathematics, Science and Research Branch, Islamic Azad University, Tehran, Iran

²Department of Mathematics, Faculty of Science, Lorestan University, Khorramabad, Iran

Received 26 January 2013; Accepted 27 May 2013

Academic Editor: Hadi Nasseri

Copyright © 2013 F. Hosseinzadeh Lotfi et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

In the course of improving various abilities of data envelopment analysis (DEA) models, many investigations have been carried out for ranking decision-making units (DMUs). This is an important issue both in theory and practice. There exist a variety of papers which apply different ranking methods to a real data set. Here the ranking methods are divided into seven groups. As each of the existing methods can be viewed from different aspects, it is possible that somewhat these groups have an overlapping with the others. The first group conducts the evaluation by a cross-efficiency matrix where the units are self- and peer-evaluated. In the second one, the ranking units are based on the optimal weights obtained from multiplier model of DEA technique. In the third group, super-efficiency methods are dealt with which are based on the idea of excluding the unit under evaluation and analyzing the changes of frontier. The fourth group involves methods based on benchmarking, which adopts the idea of being a useful target for the inefficient units. The fourth group uses the multivariate statistical techniques, usually applied after conducting the DEA classification. The fifth research area ranks inefficient units through proportional measures of inefficiency. The sixth approach involves multiple-criteria decision methodologies with the DEA technique. In the last group, some different methods of ranking units are mentioned.

1. Introduction

Data envelopment analysis as a mathematical tool was initiated by Farrell [1] and Charnes et al. [2]. They formulated a linear programming problem with which it is possible to evaluate decision-making units (DMUs) with multiple inputs and outputs. Note that in this technique it is not necessary to know the production function. In this technique, an LP problem is solved for each DMU, and the relative efficiency of each unit obtained as a linear combination of corresponding optimal weights. In this problem, weights are free to get their value to show the under evaluation unit in optimistic viewpoint. Those units with optimal objective function equal to one are called “best practice.” These units are located onto the efficient frontier, and those far away from this frontier are called inefficient. As proved in DEA literature, at least one of the units is located onto this frontier. Note that units located unto this frontier can be considered as benchmarks for inefficient units. Based on what Charnes et al. [2] provided, many extensions to DEA Models are presented in the literature. As the example of the most important one Banker et al. [3] can be mentioned. Also, multiplicative and additive models developed in the literature by Charnes et al. [4–6]. In that time Thrall [7] provided a complete comparison of all classic DEA models.

One of the important issues discussed in DEA literature is ranking efficient units since the efficient units obtained in the efficiency score of one cannot be compared with each other on the basis of this criterion any more. Therefore, it seems necessary to provide models for further discrimination among these units. Many papers are presented in the literature review for ranking the efficient units. Note that Adler and Golany [8] used principle component analysis for improving the discrimination of DEA. But this attempt was not sufficient and just made a reduction in number of efficient units not rank units completely. Many papers presented in the literature for ranking efficient units; one of the first papers is Young and Hamer [9]. One important field in ranking is cross-efficiency; to name a few, consider Sexton et al. [10], Rödder and Reucher [11], Örkcü and Bal [12], Wu et al. [13], Jahanshahloo et al. [14], Wang et al. [15], Ramón et al. [16], Guo and Wu [17], Contreras [18], Wu et al. [19], Zerafat Angiz et al. [20], and Washio and Yamada [21].

In the literature there exist other methods based on finding optimal weights in DEA analysis as Jahanshahloo et al. [22], Wang et al. [23], Alirezaee and Afsharian [24], Liu and Hsuan Peng [25], Wang et al. [26], Hatefi and Torabi [27], Hosseinzadeh Lotfi et al. [28], Wang et al. [29], and Ramón et al. [30].

One of the important fields in ranking is super efficiency presented by Andersen and Petersen [31], Mehrabian et al. [32], Tone [33], Jahanshahloo et al. [34], Jahanshahloo et al. [35], Chen and Sherman [36], Amirteimoori et al. [37], Jahanshahloo et al. [38], Li et al. [39], Sadjadi et al. [40], Gholam Abri et al. [41], Jahanshahloo et al. [42], Noura et al. [43], Ashrafi et al. [44], Chen et al. [45], Rezai Balf et al. [46], and Chen et al. [47].

Another important field in ranking is benchmarking methods such as Torgersen et al. [48], Sueyoshi [49], Jahanshahloo et al. [50], Lu and Lo [51], and Chen and Deng [52].

One important field is using statistical tools for ranking units first suggested by Friedman and Sinuany-Stern [53] and Mecit and Alp [54].

One of the significant fields in ranking is unseeing multicriteria decision-making (MCDM) methodologies and DEA analysis. To mention a few, consider Joro et al. [55], Li and Reeves [56], Belton and Stewart [57], Sinuany-Stern et al. [58], Strassert and Prato [59], Chen [60], Jablonsky [61], Wang and Jiang [62], and Hosseinzadeh Lotfi et al. [63].

Also there exist some other ranking methods not much developed and extended in the literature, Seiford and Zhu [64], Jahanshahloo [65], Jahanshahloo et al. [34], Jahanshahloo et al. [66], Jahanshahloo and Afzalinejad [67], Amirteimoori [68], Kao [69], Khodabakhshi and Aryavash [70], and Zerafat Angiz et al. [71].

In addition to the theoretical papers presented in ranking literature there exist a variety of papers which used these new models in applications such as Charnes et al. [72], Cook and Kress [73], Cook et al. [74], Martić and Savić [75], De Leeneer and Pastijn [76], Lins et al. [77], Paralikas and Lygeros [78], Ali and Nakosteen [79], Martin and Roman [80], Raab and Feroz [81], Wang et al. [26], Williams and Van Dyke [82], Jürges and Schneider [83], Giokas and Pentzaropoulos [84], Darvish et al. [85], Lu and Lo [51], Feroz et al. [86], Sadjadi et al. [40], Ramón et al. [30], and Sitarz [87].

There exist some papers which reviewed ranking methods, as Adler et al. [88]. In this paper, most of the ranking methods, specially the new ones described in the literature, are reviewed. Here, the different ranking methods are classified into seven groups after reviewing the basic DEA method in Section 2. In Section 3, the cross-efficiency technique will be discussed. In this method first suggested by Sexton et al. [10], the DMUs are self- and peer-assessed. In Section 4, some of the ranking methods based on optimal weights obtained from DEA models, common set of weights are briefly reviewed. Super-efficiency methods first introduced by Andersen and Petersen [31] will be reviewed in Section 5. The basic idea is based on the idea of leaving out one unit and assessing by the remaining units. Section 6 discusses the evaluation of DMUs through benchmarking, an approach originating in Torgersen et al. [48]. Section 7 will review the papers which use the statistical tools for ranking units first suggested by Friedman and Sinuany-Stern [53] such as canonical correlation analysis and discriminant analysis. Section 8 discusses the ranking of units based on multi-criteria decision-making (MCDM) methodologies and DEA. Section 9 discusses some different ranking methods existing in the DEA literature. Section 10 presents the results of the various methodologies applied to an example.

2. Data Envelopment Analysis

Data envelopment analysis is a mathematical programming technique for performance evaluation of a set of decision-making units.

Let a set consists of homogeneous decision-making units to be evaluated. Assume that each of these units uses inputs to produce outputs . Moreover, and are considered to be nonnegative vectors. We define the set of production possibility as .

When variable returns to scale form of technology is assumed we have and

and when constant returns to scale form of technology is assumed we have and

The two-phase enveloping problem with constant returns to scale form of technology, first provided by Charnes et al. [2], is as follows:

The two-phase enveloping problem with variable returns to scale form of technology, first provided by Banker et al. [3], is as follows:

As regards the above-mentioned problems and due to corresponding feasible region it is evident that .

According to the definition of and , an envelop constructed through units called best practice or efficient. Considering mentioned problems if a is not CCR (BCC) efficient, it is possible to project this DMU onto the CCR (BCC) efficiency frontier considering the following formulas:

The dual model corresponding to the following model is as follows:

For overcoming the problem of zero weights, and variability of weights use of assurance region is suggested by Thompson et al. [89–91].

As mentioned in the literature usually there exist more than one efficient units, and these units cannot be further compared to each other on basis of efficiency scores. Thus, it felt necessary to provide new models for ranking these units. There exist a variety of ranking models in context of data envelopment analysis. In the remaining of this paper we review some of these models.

3. Cross-Efficiency Ranking Techniques

Sexton et al. [10] provided a method for ranking units based on this idea that units are self- and peer-evaluated. For deriving the cross-efficiency of any using weights chosen by , they proposed the following equation:

where are optimal weights obtained from the following model for under assessment

Now received the average cross-efficiency score as ; for further details about this averaging see also Green et al. [92]. Doyle and Green [93] also used cross-efficiency matrix for ranking units. According to this method for ranking DMUs, many investigations have been done as reviewed in Adler et al. [88].

Rödder and Reucher [11] presented a consensual peer-based DEA model for ranking units. As the authors said, this method is generalized twofold. The first is an optimal efficiency improving input allocation; the second aim is the choice of a peer DMU whose corresponding price is acceptable for the other units. Consider

The higher the degree of input variation is, the better the chance to be efficient will be.

Örkcü and Bal [12] provided a goal programming technique to be used in the second stage of the cross-evaluation. Their modified model is as follows:

As the authors noted, there exist alternative optimal solutions.

Wu et al. [13] described the main suffering of cross-efficiency when the ultimate average cross-efficiency utilized for ranking units. For removing this shortcoming they eliminated the assumption of average and utilized the Shannon entropy in order to obtain the weights for ultimate cross-efficiency scores. Jahanshahloo et al. [14] provided a method for selecting symmetric weights to be used in DEA cross-efficacy.

Step 1. Efficiency of DMUs needs to be computed.

Step 2. Choose the solutions, in accordance with the secondary goal for each DMU, as follows:

Step 3. The cross-efficiency for any , using the weights that   has chosen in the previous model, is then used as follows:

Wang et al. [15] provided a cross-efficiency evaluation based on ideal and anti-ideal units for ranking. As the authors mentioned, a DMU could choose a unique set of input and output weights to make its distance from ideal DMU as small as possible, or the distance from anti-ideal DMU as large as possible, or the both. Thus, according to this idea they proposed the following procedure for cross-efficiency evaluation.

Model 1. Minimization of the distance from ideal DMU.

Model 2. Maximization of the distance from anti-ideal DMU.

Model 3. Maximization of the distance between ideal DMU and anti-ideal DMU.

Model 4. Maximization of the relative closeness.

The authors mentioned that the bigger the relative closeness of a DMU is the better performance it will have.

In a paper, Ramón et al. [16] selected the profiles of weights used in cross-efficiency assessment. As the authors said they tried to prevent unrealistic weighting. They have discussed the zero weights as they excluded variables from the evaluation. In the calculation of cross-efficiency scores, they proposed to ignore the profiles of those weights of the unit under evaluation that among their alternate optimal solutions cannot choose nonzero weights. They also proposed the “peer-restricted” cross-efficiency evaluation where the units assessed in a peer evaluation which means profiles of weights of some inefficient units are not considered. Finally, the presented approach extended to derive a common set of weights. Guo and Wu [17] provided a complete ranking of DMUs with undesirable outputs using restriction in DEA. As the author mentioned this model is presented to realize a unique ranking of units by “maximal balanced index” according to the obtained optimal shadow prices

where is the optimal objective function of multiplier model.

Contreras [18] used cross-evaluation for ranking units in DEA methodology. The idea is based upon introducing a model for optimizing the rank position of DMUs

Consider and solve the following model:

As it is obvious, nonuniqueness of optimal weight may occur.

Wu et al. [19] proposed a weight balanced DEA model to reduce differences in weights data and zero weights

Therefore, the cross-efficiency score of is the average of these cross-efficiencies:

where

As it is obvious, nonuniqueness of optimal weight may occur.

Zerafat Angiz et al. [20] introduced a cross-efficiency matrix based on this idea that ranking order is much more significant than individual efficiency score. Thus, they have provided the following procedure.

Step 1. Considering CCR model, calculate the efficiency score of all DMUs and consider as the efficiency score of .

Step 2. Now the cross-efficiency matrix can be constructed by . Note that is used as the diagonal elements of .

Step 3. Convert the cross-efficiency matrix into a cross-ranking matrix as , in which is the ranking order of in column of matrix .

Step 4. Construct the preference matrix as considering matrix where is the number of time that is placed in rank .

Step 5. Construct matrix as in which is calculated by summing the efficiency scores in matrix Z, corresponds to , being placed in rank .

Step 6. Obtain a common set of weight for final ranking of DMUs using the following modified method:

where obtained from Step 5 and is the optimal solution of the following model. Finally, the DMUs are ranked based on their values.

Washio and Yamada [21] discussed that in real cases finding the best ranking is more significant than acquiring the most advantage weight and maximizing the efficiency. Thus they presented a model called rank-based measure (RBM) for evaluating units from different viewpoint. Thus, they suggested a method for acquiring those weight resulted from the best ranking as long as calculating those weight that maximizes the efficiency score. Finally they applied the presented model to the cross-efficiency assessment.

4. Ranking Techniques Based on Finding Optimal Weights in DEA Analysis

Jahanshahloo et al. [22] gave a note on some of the DEA models for complete ranking using common set of weights. They proved that by solving only one problem, it is possible to determine the common set of weights

where is the set of efficient units of the following model:

Note that DMUs can be ranked based on the evaluation of their efficiencies.

Wang et al. [23] provided an aggregating preference ranking. In this paper use of ordered weighted averaging (OWA) operator is proposed for aggregating preference rankings. Let be the relative importance weight given to the jth ranking place and the vote candidate i receives in the jth ranking place. The total score of each candidate is defined as

Alirezaee and Afsharian [24] discussed multiplier model, in which the variables are considered as shadow prices; note that and are total revenue and cost of which are considered in optimization problem. The authors claimed that is the profit restriction for . If is considered to be the efficient production function, then

They mentioned that the connected profit of the DMU is zero when shadow prices are derived from the technology and called this situation as balance situation. As the authors mentioned therefore in the case that is efficient but is inefficient or the efficiency score of both DMUs is the same, and it obtains more negative quantity in balance index, it can be concluded that has a better rank than .

Liu and Hsuan Peng [25] in their paper proposed a method for determining the common set of weights for ranking units. In common weights analysis methodology they provided the following model:

The mentioned ratio form the linear equations.

Wang et al. [26] proposed a paper for ranking decision-making units by imposing a minimum weight restriction in DEA. The authors noted that using data envelopment analysis, it is not possible to distinguish between DEA efficient units. Thus, they presented a method for ranking units using imposing minimum weight restriction for the input-output data. As they mentioned, these weights restrictions are decided by a decision maker (DM) or an assessor as regards the solutions to a series of LP models considered for determining a maximin weight for each efficient DMU.

Hatefi and Torabi [27] proposed a common weight multi criteria decision analysis (MCDA)-data envelopment analysis (DEA) for constructing composite indicators (CIs). As the authors proved the presented model can discriminate between efficient units. The obtained common weights have discriminating power more than those obtained from previous models. Finally they studied the robustness and discriminating power of the proposed method by Spearman's rank correlation coefficient.

Hosseinzadeh Lotfi et al. [28] proposed one DEA ranking model based on applying aggregate units. In doing so artificial units called aggregate units are defined as follows. The aggregate unit is shown by

where . Note that is the set of efficient units.

First, it is tried to maximize the efficiency score of the and then to maximize the efficiency score of the . For resolving the existence of alternative solutions the authors presented an approach comprising simple linear problems to achieve the most appropriate optimal solutions among all alternative optimal solutions. Finally, they proposed the RI index for ranking all efficient DMUs. Let and be the optimal solutions of the multiplier model for assent and . Consider , thus .

Wang et al. [29] presented two nonlinear regression models for deriving common set of weights for fully rank units

Ramón et al. [30] aimed at deriving a common set of weights for ranking units. As the authors mentioned the idea is based upon minimization of the deviations of the common weights from the nonzero weights obtained from DEA. Furthermore, several norms are used for measuring such differences.

5. Super-Efficiency Ranking Techniques

Super efficiency models introduced in DEA technique are based upon the idea of leave one out and assessing this unit trough the remanding units.

Andersen and Petersen [31] introduced a model for ranking efficient units. The proposed model is as follows:

Although this idea is useful for further discriminating efficient units, it has been shown in the literature that it may be infeasible and nonstable. Thrall [7] mentioned the infeasibility of super-efficiency CCR model. Also a condition under which infeasibility occurred in super-efficiency DEA models is mentioned by Zhu [94], Seiford and Zhu [95], and Dulá and Hickman [96].

Hashimoto [97] provided a model based on the idea of one leave out and assurance region for ranking units.

Mehrabian et al. [32] presented a complete ranking for efficiency units in DEA context. As the authors mentioned this model does not have difficulties of A.P. model

With this method it is not possible to rank nonextreme efficient units.

Tone [33] presented super efficiency of SBM model. This model has the advantages of nonradial models, and it is always feasible and stable

Jahanshahloo et al. [34] added some ratio constraints to the multiplier form of A.P. model and introduced a new method for ranking DMUs

is efficient if the optimal objective function of the previous model is greater than or equal one.

Jahanshahloo et al. [35] presented a method for ranking efficient units on basis of the idea of one leave out and norm. As the authors proved this model is always feasible and stable

where .

In their paper Chen and Sherman [36] presented a nonradial super-efficiency method and discussed the advantage of it. They verified that this model is invariant to units of input/output measurement. Let .

Step 1. Solve the following model to find the extreme efficient units in :

Consider as the set of efficient units of .

Step 2. Solve the following model:

Consider as the optimal solution of the previous model and solve the following model for :

According to the obtained optimal solution of the previous model for let , and , .

Step 3. Solve the following model:

Now according to the optimal solution of the previous model, solve the following model for each :

Step 4. Let for and for If ø, then stop otherwise; if ø, let and go to Step 3. Now define as the average RNSE-DEA index

Amirteimoori et al. [37] provided a distance-based approach for ranking efficient units. The presented method is a new method utilized norm. As noted in their paper, this new approach does not have difficulties of other methods

This method cannot rank nonextreme efficient units.

Jahanshahloo et al. [38] presented modified MAJ model for ranking efficiency units in DEA technique

where . It cannot rank nonextreme efficient units.

Li et al. [39] presented a new method for ranking which does not have difficulties of earlier methods. The presented model is always feasible and stable

In this case extreme efficient units cannot be ranked.

In a paper Khodabakhshi [98] addresses super efficiency on improved outputs. He mentioned that as A.P. model may be infeasible under variable returns to scale technology, using the presented model gives a complete ranking when getting an input combination for improving outputs is suitable.

Sadjadi et al. [40] presented a robust super-efficiency DEA for ranking efficient units. They noted that as in most of the times exact data do not exist and the stochastic super-efficiency model presented in their paper incorporates the robust counterpart of super-efficiency DEA

where are input-output data. This model does not rank nonextreme efficient units. It may be unstable and infeasible.

Gholam Abri et al. [41] proposed a model for ranking efficient units. They used representation theory and represented the DMU under assessment as a convex combination of extreme efficient units. As the authors noted it is expected that the performance of is the same as the performance of convex combination of extreme efficient units. Thus, it is possible to represent as follows:

As regards representation theorem, this system has constraints and variables, . If this system has a unique solution we will have , otherwise two models should be considered

Consider and as the optimal solution of the above-mentioned models, respectively. If , this is the same as what has been mentioned previously. If , then mentioned models provide an interval which helps rank units from the worst to the best. Sometimes it will obtain the ranking score with a bounded interval .

Jahanshahloo et al. [42] presented models for ranking efficient units. The presented models are somehow a modification of cross-efficiency model that overcomes the difficulty of alternative optimal weights. The authors in their paper with regard to the changes and also utilizing TOPSIS technique presented a new super-efficient method for ranking units. The presented model is as follows:

where is the efficiency score of using corresponding weights. Also is the efficiency of using optimal weights of . Noura et al. [43] provided a method for ranking efficient units based on this idea that more effective and useful units in society should have better rank.

Step 1. For each efficient unit choose the lower and upper limit for each inputs and outputs. Let be the set of efficient units

Step 2. In accordance with the previous step, here, the utility inputs and outputs are as follows:

Step 3. Consider dimensionless introduced as follows for each efficient unit belonging to :

where is representative of a small and nonzero number used for not dividing by zero. Now consider as follows which shows that more successful will be if the larger value of the is

where .

Ashrafi et al. [44] introduced an enhanced Russell measure of super efficiency for ranking efficient units in DEA. The linear counterpart of the proposed model is as follows:

It cannot rank nonextreme efficient units.

Chen et al. [45] proposed a modified super-efficiency method for ranking units based on simultaneous input-output projection. The presented model overcomes the infeasibility problem

Rezai Balf et al. [46] provide a model for ranking units based on Tchebycheff norm. As proved that this model is always feasible and stable, it seems to have superiority over other models

where

Chen et al. [47] for overcoming the infeasibility problem that occurred in variable returns to scale super-efficiency DEA model according to a directional distance function developed Nerlove-Luenberger (N-L) measure of super-efficiency.

6. Benchmarking Ranking Techniques

Sueyoshi et al. [49] proposes a “benchmark approach” for baseball evaluation. This method is the combination of DEA and (Offensive earned-run average) OERA. As the authors noted, using this method it is possible to select best units and also their ranking orders. They mentioned that using only DEA may result in a shortcoming in assessment as many efficient units can be identified. Thus, the authors used slack-adjusted DEA model and OERA to overcome this difficulty.

Jahanshahloo et al. [50] presented a new model for ranking DMUs based on alteration in reference set. The idea is based on this fact that efficient units can be the target unit for inefficient units

Finally the ranking order for each efficient unit b can be computing by

Note that this method cannot rank nonextreme efficient units.

Lu and Lo [51] provided an interactive benchmark model for ranking units. The idea is based upon considering a fixed unit as a benchmark and calculating the efficiency of other units, pair by pair, to this unit. This procedure continues when all units are accounted for as a benchmark unit. Consider as a unit under evaluation and as a benchmark. Assume

Then using , the efficiency of benchmark can be obtained. Using the following index which indicates the increment in efficiency of a unit by moving from peer appraisal to self-appraisal, it is now possible to rank units. Note that the less the magnitude of this index is the better rank for corresponding unit will be obtained

where and are, respectively, efficiency in BCC model and normalization of of . Chen [52] provided a paper for ranking efficient and inefficient units in DEA. They noted that the evaluation of efficient units is based upon the alterations in efficiency of all inefficient units by omitting in reference set. At first, solve the following model which measures the efficiency of when is eliminated from the reference set

Then, again, solve the previous model, this time for calculating the efficiency score of each inefficient unit when each of efficient units, in turn, is eliminated from the reference set and calculate the efficiency change

Then, calculate the following index called as “MCDE” index for each efficient and inefficient unit:

Now, in accordance with the magnitude of the acquired index it is possible to rank units. Those units with higher score have better ranking order.

7. Ranking Techniques by Multivariate Statistics in the DEA

As described in DEA literature in DEA technique frontier is taken into consideration rather than central tendency considered in regression analysis. DEA technique considers that an envelope encompasses through all the observations as tight as possible and does not try to fit regression planes in center of data. In DEA methodology each unit is considered initially and compared to the efficient frontier, but in regression analysis a procedure is considered in which a single function fits to the data. DEA uses different weights for different units but does not let the units use weights of other units. Canonical correlation analysis, as Friedman and Sinuany-Stern [53] noted, can be used for ranking units. This method is somehow the extension of regression analysis, Adler et al. [88]. The aim in canonical correlation analysis is to find a single vector common weight for the inputs and outputs of all units. Consider as the composite input and output and as corresponding weights, respectively. The presented model by Tatsuoka and Lohnes [99] is as follows:

where , and are respectively defined as the matrices of the sums of squares and sums of products of the variables.

Friedman and Sinuany-Stern [53] while defining the ratio of linear combinations of the inputs and outputs, used canonical correlation analysis. As they noted that scaling ratio of the canonical correlation analysis/DEA is unbounded.

Sinuany-Stern et al. [100] used linear discriminant analysis for ranking units. They defined

is said to be efficient if where is a critical value based on the midpoint of the means of the discriminant function value of the two groups, Morrison [101]. The larger the amount of is the better rank will have.

Friedman and Sinuany-Stern [53] noted that as cross-efficiency/DEA, canonical correlation analysis/DEA and discriminant analysis/DEA, ranking orders may vary from each other, thus it seems necessary to introduce the combined ranking (CO/DEA). Combined ranking, for each unit, considered all the ranks obtained from the above-mentioned rankings. Moreover, statistical tests are; one for goodness of fit between DEA and a specific ranking and the other for testing correlation between variety of ranking orders, Siegel and Castellan [102].

8. Ranking with Multicriteria Decision-Making (MCDM) Methodologies and DEA

Li and Reeves [56] for increasing discrimination power of DEA presented a multiple-objective linear program (MOLP). As the authors mentioned using minimax and minsum efficiency in addition to the standard DEA objective function help to increase discrimination power of DEA.

Strassert and Prato [59] presented the balancing and ranking method which uses a three-step procedure for deriving an overall complete or partial final order of options. In the first step, derive an outranking matrix for all options from the criteria values. Considering this matrix it is possible to show the frequency with which one option is ranked higher than the other options. In the second step, by triangularizing the outranking matrix establish an implicit preordering or provisional ordering of options. The outranking matrix shows the degree to which there is a complete overall order of options. In the third step, based on information given in an advantages-disadvantages table, the provisional ordering is subjected to different screening and balancing operations.

Chen [60] utilized a nonparametric approach, DEA, to estimate and rank the efficiency of association rules with multiple criteria in following steps. Proposed postprocessing approach is as follows.

Step 1. Input data for association rule mining.

Step 2. Mine association rules by using the a priori algorithm with minimum support and minimum confidence.

Step 3. Determine subjective interestingness measures by further considering the domain related knowledge.

Step 4. Calculate the preference scores of association rules discovered in Step 2 by using Cook and Kress’s DEA model.

Step 5. Discriminate the efficient association rules found in Step 3 by using Obata and Ishii’s [103] discriminate model.

Step 6. Select rules for implementation by considering the reference scores generated in Step 5 and domain related knowledge.

Jablonsky [61] presented two original models, super SBM and AHP, for ranking of efficient units in DEA. As the author mentioned these models are based on multiple criteria decision-making techniques-goal programming and analytic hierarchy process. Super SBM model for ranking units is as follows:

Wang and Jiang [62] presented an alternative mixed integer linear programming models in order to identify the most efficient units in DEA technique. As the authors mentioned presented models can make full use of input-output information with no need to specify any assurance regions for input and output weights to avoid zero weights

Hosseinzadeh Lotfi et al. [63] provided an improved three-stage method for ranking alternatives in multiple criteria decision analysis. In the first stage, based on the best and worst weights in the optimistic and pessimistic cases, obtain the rank position of each alternative, respectively. The obtained weight in the first stage is not unique; thus it seems necessary to introduce a secondary goal that is used in the second stage. Finally, in the third stage, the ranks of the alternatives compute in the optimistic or pessimistic case. It is mentionable that the model proposed in the third stage is a multicriteria decision-making (MCDM) model, and it is solved by mixed integer programming. As the authors mentioned and provided in their paper this model can be converted into an LP problem

In the previous model, the rank vector for each alternative is computed by the ideal rank.

9. Some Other Ranking Techniques

Seiford and Zhu [64] presented the context-dependent DEA method for ranking units. Let be the set of decision-making units. Consider in which where is the optimal value of following model:

The previous model with is the original CCR model. Note that DMUs in show the first level efficient frontier. indicates the second level efficient frontier when the first level efficient frontier is omitted. The following algorithm finds these efficient frontiers.

Step 1. Set and evaluate the entire set of DMUs, . In this way the first level efficient DMU, , is identified.

Step 2. Exclude the efficient DMUs from future DEA runs and set . (If stop).

Step 3. Evaluate the new subset of “inefficient” DMUs, , to obtain a new set of efficient DMUs, .

Step 4. Let . Go to Step 2.

When ø, the algorithm stops.

As the authors proved in this way it is possible to rank the DMUs in the first efficient frontier based upon their attractiveness scores and identify the best one.

Jahanshahloo et al. [65] provided a paper for ranking units using gradient line. As the authors mentioned the advantage of this model is stability and robustness

Note that is the gradient of hyperplane which supports on , the obtained PPS by omitting the DMU under assessment in . As the authors proved a unit is efficient iff the optimal objective function of the following model is greater than zero. Consider

Intersection of and efficient surface of is a half line where its equation is . To rank the length of connecting arc with intersection point of line and previous ellipse is calculated in space. This intersection is as follows:

Now, the length of connecting arc to the point corresponding to in plan is calculated as follows: , where

Jahanshahloo et al. [34] also provided a paper with the concept of advantage in data envelopment analysis.

In their paper Jahanshahloo et al. [66] considering Monte Carlo method presented a new method of ranking.

Step 1. Generate a uniformly distributed sequence of on.

Step 2. Random numbers should be classified into pairs like in a way that each number is used just one time.

Step 3. Compute and .

Step 4. Estimate the integral by .

Now consider as an efficient unit measure those units that are dominated . As for dome DMUs this would be unbounded, thus the authors, for each unit, bounded the region. Then, for if , then is in the RED of . Now by using all the hits (the above condition) can be counted, where is the measure of the whole region. Jahanshahloo and Afzalinejad [67] presented a method based on distance of the unit under evaluation to the full inefficient frontier. They presented two models, radian and nonradial

Amirteimoori [68] based on the same idea considered both efficient and antiefficient frontiers for efficiency analysis and ranking units.

Kao [69] mentioned that determining the weights of individual criteria in multiple criteria decision analysis in a way that all alternatives can be compared according to the aggregate performance of all criteria is of great importance. As Kao noted this problem relates to search for alternatives with a shorter and longer distance, respectively, to the ideal and anti-ideal units. He proposed a measure that considered the calculation of the relative position of an alternative between the ideal and anti-ideal for finding an appropriate rankings.

Khodabakhshi and Aryavash [70] presented a method for ranking all units using DEA concept.

First, Compute the minimum and maximum efficiency values of each DMU in regard to this assumption that the sum of efficiency values of all DMUs equals to 1.

Second, determine the rank of each DMU in relation to a combination of its minimum and maximum efficiency values.

Zerafat Angiz et al. [71] proposed a technique in order to aggregate the opinions of experts in voting system. As the authors mentioned the presented method uses fuzzy concept, and it is computationally efficient and can fully rank alternatives. At first, number of votes given to a rank position was grouped to construct fuzzy numbers, and then the artificial ideal alternative introduced. Furthermore, by performing DEA the efficiency measure of alternatives was obtained considering artificial ideal alternative compared by each of the alternatives pair by pair. Thus alternatives are ranked in accordance with their efficiency scores. If this method cannot completely rank alternatives, weight restrictions based on fuzzy concept are imposed into the analysis.

10. Different Applications in Ranking Units

As discussed formerly there exist a variety of ranking methods and applications in theliterature. Nowadays DEA models are widely used in different areas for efficiency evaluation, benchmarking and target setting, ranking entities, and so forth. Ranking units, as one of the important issues in DEA, has been performed in different areas. Jahanshahloo et al. [42] ranked cities in Iran to find the best place for creating a data factory. Hosseinzadeh Lotfi et al. [28] used a new method for ranking in order to find the best place for power plant location. Ali and Nakosteen [79], Amirteimoori et al. [37] and Alirezaee and Afsharian [24], Soltanifar and Hosseinzadeh Lotfi [104], Zerafat Angiz et al. [71], Hosseinzadeh Lotfi et al. [28], Jahanshahloo [50], Chen and Deng [52], and Jablonsky [61] used different ranking methods in banking system. Sadjadi [40] ranked provincial gas companies in Iran. Mehrabian et al. [32], Li et al. [39], Örkcü and Bal [12], and Wu et al. [19] ranked different departments of universities. Jahanshahloo et al. [35], Jahanshahloo and Afzalinejad [67] utilized the presented models in ranking 28 Chinese cities. Jahanshahloo at al. [35] provided an application to burden sharing amongst NATO member nations. Jahanshahloo et al. [14] utilized the presented model for ranking nursery homes. Contreras [18] and Wang et al. [23] used the provided ranking techniques in ranking candidates. Lu and Lo [51] applied their method on an application to financial holding companies. Hosseinzadeh Lotfi et al. [63] consider an empirical example in which voters are asked to rank two out of seven alternatives. Wang and Jiang [62] utilized the provided method in facility layout design in manufacturing systems and performance evaluation of 30 OECD countries. Chen [60] used an example of market basket data in order to illustrate the provided approach.

11. Application

In this section, some of the reviewed models are applied on the example used in Soltanifar and Hosseinzadeh Lotfi [104]. Consider twenty commercial banks of Iran with input-output data tabulated in Table 1 and summarized as follows. Also the results of CCR model are listed in this table. As it can be seen seven units are efficient, DMUS 1,4, 7, 12, 15, 17, and 20. Inputs are staff, computer terminal, and space. Outputs are deposits, loans granted, and charge.

Table 1: Inputs, outputs, and efficiency scores.

CCR efficiency

Consider some of the important ranking methods in literature as follows:

 R.M1: A.P. model [31] is based upon the idea of leave unit evaluation out and measuring the distance of the unit under evaluation from the production possibility set constructed by the remaining DMUs. R.M2: MAJ model [32] presented for ranking efficient which is always stable but might be infeasible in some cases. R.M3: Modified MAJ model [38] overcomes the problem which might occurr in MAJ model. R.M4: A new model based on the idea of alterations in the reference set of the inefficient units [50]. R.M5: A model presented by Li et al. [39] which is a super-efficiency method that does not have the suffering in previous methods. R.M6: Slack-based model [33] is based upon the input and output variables at the same time.  R.M7: SA DEA model [49] overcomes the problem of infeasibility which existed in A.P. model. R.M8: Cross-efficiency [10] is provided based on using weights of each unit under evaluation in optimality for other units. R.M9: A model based on finding common set of weights [25] which determine the common set of weights for DMUs and ranked DMUs based this idea. R.M10: A model based on finding common set of weights [22, 66] for ranking efficient units. R.M11: L1-norm model [34, 35, 65], the idea is based upon the leave-one-out efficient unit and l1-norm which is always feasible and stable. R.M12: -norm model Rezai Balf et al. [46] provided a method with more ability over other existing methods, based on Tchebycheff norm. R.M13: An enhanced Russel measure of super-efficiency model for ranking units [44]. R.M14: A ranking model which considers the distance of unit from the full inefficient frontier [38] R.M15: A modified super-efficiency model [45] which overcomes the infeasibility that may happen in problem. This model is based on simultaneous projection of input output.

In accordance with properties of different ranking models in order to rank efficient units, consider Table 2.

 : Feasibility : Ranking extreme efficient units  : Complexity in computation  : Instability  : Absence of multiple optimal solution  : Dependency to and slacks  : Dependency to the number of efficient and inefficient units.

Table 2: Matrix of properties.

R.M1
—
—

—

—

R.M2

—

R.M3

—

R.M4

—
—

R.M5

—

—

R.M6

—
—

—

R.M7

—
—

—

R.M8

—
—

R.M9

—
—
—

R.M10

—
—
—

R.M11

—

—
—

R.M12

—

—
—

R.M13

—
—

—

R.M14

—
—
—
—

—

R.M15

—
—

—

In Tables 3 and 4 the rank of efficient units considering the above mentioned methods is listed. Note that E.D. shows efficient DMUs (E.D.), and are those explained previously.

Table 3: Ranking orders.

E.D.
R.M1
R.M2
R.M3
R.M4
R.M5
R.M6
R.M7
R.M8

Table 4: Ranking orders.

E.D.
R.M9
R.M10
R.M11
R.M12
R.M13
R.M14
R.M15

12. Conclusion

In this paper, the DEA ranking was reviewed and classified into seven general groups. In the first group, those papers based on a cross-efficiency matrix were reviewed. In this field, DMUs have been evaluated by self- and peer pressure. The second group of papers is based on those papers looking for optimal weights in DEA analysis. The third one is the super-efficiency method. By omitting the under evaluation unit and constructing a new frontier by the remaining units, the unit under evaluation can get an equal or greater score. The fourth group is based on benchmarking idea. In this class, the effect of an efficient unit considered as a target for inefficient units is investigated. This idea is very useful for the managers in decision making. Another class, the fifth one, involves the application of multivariate statistical tools. The sixth section discusses the ranking methods based on multicriteria decision-making (MCDM) methodologies and DEA. The final section includes some various ranking methods presented in the literature. Finally, in an application, the result of some of the above-mentioned ranking methods is presented.

References

M. J. Farrell, “The measurement of productive efficiency,” Journal of the Royal Statistical Society A, vol. 120, pp. 253–281, 1957.

A. Charnes, W. W. Cooper, and E. Rhodes, “Measuring the efficiency of decision making units,” European Journal of Operational Research, vol. 2, no. 6, pp. 429–444, 1978.

R. D. Banker, A. Charnes, and W. W. Cooper, “Some models for estimating technical and scale inefficiencies in data envelopment analysis,” Management Science, vol. 30, no. 9, pp. 1078–1092, 1984.

A. Charnes, W. W. Cooper, L. Seiford, and J. Stutz, “A multiplicative model for efficiency analysis,” Socio-Economic Planning Sciences, vol. 16, no. 5, pp. 223–224, 1982.

A. Charnes, C. T. Clark, W. W. Cooper, and B. Golany, “A developmental study of data envelopment analysis in measuring the efficiency of maintenance units in the U.S. air forces,” Annals of Operations Research, vol. 2, no. 1, pp. 95–112, 1984.

A. Charnes, W. W. Cooper, B. Golany, L. Seiford, and J. Stutz, “Foundations of data envelopment analysis for Pareto-Koopmans efficient empirical production functions,” Journal of Econometrics, vol. 30, no. 1-2, pp. 91–107, 1985.

R. M. Thrall, “Duality, classification and slacks in DEA,” Annals of Operations Research, vol. 66, pp. 109–138, 1996.

N. Adler and B. Golany, “Evaluation of deregulated airline networks using data envelopment analysis combined with principal component analysis with an application to Western Europe,” European Journal of Operational Research, vol. 132, no. 2, pp. 260–273, 2001.

F. W. Young and R. M. Hamer, Multidimensional Scaling, History, Theory and Applications, Lawrence Erlbaum, London, UK, 1987.

T. R. Sexton, R. H. Silkman, and A. J. Hogan, “Data envelopment analysis: critique and extensions,” in Measuring Efficiency: An Assessment of Data Envelopment Analysis, R. H. Silkman, Ed., pp. 73–105, Jossey-Bass, San Francisco, Calif, USA, 1986.

W. Rödder and E. Reucher, “A consensual peer-based DEA-model with optimized cross-efficiencies: input allocation instead of radial reduction,” European Journal of Operational Research, vol. 212, no. 1, pp. 148–154, 2011.

H. H. Örkcü and H. Bal, “Goal programming approaches for data envelopment analysis cross efficiency evaluation,” Applied Mathematics and Computation, vol. 218, no. 2, pp. 346–356, 2011.

J. Wu, J. Sun, L. Liang, and Y. Zha, “Determination of weights for ultimate cross efficiency using Shannon entropy,” Expert Systems with Applications, vol. 38, no. 5, pp. 5162–5165, 2011.

G. R. Jahanshahloo, F. H. Lotfi, Y. Jafari, and R. Maddahi, “Selecting symmetric weights as a secondary goal in DEA cross-efficiency evaluation,” Applied Mathematical Modelling, vol. 35, no. 1, pp. 544–549, 2011.

Y.-M. Wang, K.-S. Chin, and Y. Luo, “Cross-efficiency evaluation based on ideal and anti-ideal decision making units,” Expert Systems with Applications, vol. 38, no. 8, pp. 10312–10319, 2011.

N. Ramón, J. L. Ruiz, and I. Sirvent, “Reducing differences between profiles of weights: a “peer-restricted” cross-efficiency evaluation,” Omega, vol. 39, no. 6, pp. 634–641, 2011.

D. Guo and J. Wu, “A complete ranking of DMUs with undesirable outputs using restrictions in DEA models,” Mathematical and Computer Modelling, 2012.

I. Contreras, “Optimizing the rank position of the DMU as secondary goal in DEA cross-evaluation,” Applied Mathematical Modelling, vol. 36, no. 6, pp. 2642–2648, 2012.

J. Wu, J. Sun, and L. Liang, “Cross efficiency evaluation method based on weight-balanced data envelopment analysis model,” Computers and Industrial Engineering, vol. 63, pp. 513–519, 2012.

M. Zerafat Angiz, A. Mustafa, and M. J. Kamali, “Cross-ranking of decision making units in data envelopment analysis,” Applied Mathematical Modelling, vol. 37, no. 1-2, pp. 398–405, 2013.

S. Washio and S. Yamada, “Evaluation method based on ranking in data envelopment analysis,” Expert Systems with Applications, vol. 40, no. 1, pp. 257–262, 2013.

G. R. Jahanshahloo, A. Memariani, F. H. Lotfi, and H. Z. Rezai, “A note on some of DEA models and finding efficiency and complete ranking using common set of weights,” Applied Mathematics and Computation, vol. 166, no. 2, pp. 265–281, 2005.

Y.-M. Wang, Y. Luo, and Z. Hua, “Aggregating preference rankings using OWA operator weights,” Information Sciences, vol. 177, no. 16, pp. 3356–3363, 2007.

M. R. Alirezaee and M. Afsharian, “A complete ranking of DMUs using restrictions in DEA models,” Applied Mathematics and Computation, vol. 189, no. 2, pp. 1550–1559, 2007.

F.-H. F. Liu and H. Hsuan Peng, “Ranking of units on the DEA frontier with common weights,” Computers and Operations Research, vol. 35, no. 5, pp. 1624–1637, 2008.

Y.-M. Wang, Y. Luo, and L. Liang, “Ranking decision making units by imposing a minimum weight restriction in the data envelopment analysis,” Journal of Computational and Applied Mathematics, vol. 223, no. 1, pp. 469–484, 2009.

S. M. Hatefi and S. A. Torabi, “A common weight MCDA-DEA approach to construct composite indicators,” Ecological Economics, vol. 70, no. 1, pp. 114–120, 2010.

F. Hosseinzadeh Lotfi, A. A. Noora, G. R. Jahanshahloo, and M. Reshadi, “One DEA ranking method based on applying aggregate units,” Expert Systems with Applications, vol. 38, no. 10, pp. 13468–13471, 2011.

Y.-M. Wang, Y. Luo, and Y.-X. Lan, “Common weights for fully ranking decision making units by regression analysis,” Expert Systems with Applications, vol. 38, no. 8, pp. 9122–9128, 2011.

N. Ramón, J. L. Ruiz, and I. Sirvent, “Common sets of weights as summaries of DEA profiles of weights: with an application to the ranking of professional tennis players,” Expert Systems with Applications, vol. 39, no. 5, pp. 4882–4889, 2012.

P. Andersen and N. C. Petersen, “A procedure for ranking efficient units in data envelopment analysis,” Management Science, vol. 39, no. 10, pp. 1261–1294, 1993.

S. Mehrabian, M. R. Alirezaee, and G. R. Jahanshahloo, “A complete efficiency ranking of decision making units in data envelopment analysis,” Computational Optimization and Applications, vol. 14, no. 2, pp. 261–266, 1999.

K. Tone, “A slacks-based measure of super-efficiency in data envelopment analysis,” European Journal of Operational Research, vol. 143, no. 1, pp. 32–41, 2002.

G. R. Jahanshahloo, M. Sanei, and N. Shoja, “Modified ranking models, using the concept of advantage in data envelopment analysis,” Working paper, 2004

G. R. Jahanshahloo, F. Hosseinzadeh Lotfi, N. Shoja, G. Tohidi, and S. Razavyan, “Ranking using -norm in data envelopment analysis,” Applied Mathematics and Computation, vol. 153, no. 1, pp. 215–224, 2004.

Y. Chen and H. D. Sherman, “The benefits of non-radial vs. radial super-efficiency DEA: an application to burden-sharing amongst NATO member nations,” Socio-Economic Planning Sciences, vol. 38, no. 4, pp. 307–320, 2004.

A. Amirteimoori, G. Jahanshahloo, and S. Kordrostami, “Ranking of decision making units in data envelopment analysis: a distance-based approach,” Applied Mathematics and Computation, vol. 171, no. 1, pp. 122–135, 2005.

G. R. Jahanshahloo, L. Pourkarimi, and M. Zarepisheh, “Modified MAJ model for ranking decision making units in data envelopment analysis,” Applied Mathematics and Computation, vol. 174, no. 2, pp. 1054–1059, 2006.

S. Li, G. R. Jahanshahloo, and M. Khodabakhshi, “A super-efficiency model for ranking efficient units in data envelopment analysis,” Applied Mathematics and Computation, vol. 184, no. 2, pp. 638–648, 2007.

S. J. Sadjadi, H. Omrani, S. Abdollahzadeh, M. Alinaghian, and H. Mohammadi, “A robust super-efficiency data envelopment analysis model for ranking of provincial gas companies in Iran,” Expert Systems with Applications, vol. 38, no. 9, pp. 10875–10881, 2011.

A. Gholam Abri, G. R. Jahanshahloo, F. Hosseinzadeh Lotfi, N. Shoja, and M. Fallah Jelodar, “A new method for ranking non-extreme efficient units in data envelopment analysis,” Optimization Letters, vol. 7, no. 2, pp. 309–324, 2011.

G. R. Jahanshahloo, M. Khodabakhshi, F. Hosseinzadeh Lotfi, and M. R. Moazami Goudarzi, “A cross-efficiency model based on super-efficiency for ranking units through the TOPSIS approach and its extension to the interval case,” Mathematical and Computer Modelling, vol. 53, no. 9-10, pp. 1946–1955, 2011.

A. A. Noura, F. Hosseinzadeh Lotfi, G. R. Jahanshahloo, and S. Fanati Rashidi, “Super-efficiency in DEA by effectiveness of each unit in society,” Applied Mathematics Letters, vol. 24, no. 5, pp. 623–626, 2011.

A. Ashrafi, A. B. Jaafar, L. S. Lee, and M. R. A. Bakar, “An enhanced russell measure of super-efficiency for ranking efficient units in data envelopment analysis,” The American Journal of Applied Sciences, vol. 8, no. 1, pp. 92–96, 2011.

J.-X. Chen, M. Deng, and S. Gingras, “A modified super-efficiency measure based on simultaneous input-output projection in data envelopment analysis,” Computers & Operations Research, vol. 38, no. 2, pp. 496–504, 2011.

F. Rezai Balf, H. Zhiani Rezai, G. R. Jahanshahloo, and F. Hosseinzadeh Lotfi, “Ranking efficient DMUs using the Tchebycheff norm,” Applied Mathematical Modelling, vol. 36, no. 1, pp. 46–56, 2012.

Y. Chen, J. Du, and J. Huo, “Super-efficiency based on a modified directional distance function,” Omega, vol. 41, no. 3, pp. 621–625, 2013.

A. M. Torgersen, F. R. Foørsund, and S. A. C. Kittelsen, “Slack-adjusted efficiency measures and ranking of efficient units,” Journal of Productivity Analysis, vol. 7, no. 4, pp. 379–398, 1996.

T. Sueyoshi, “DEA non-parametric ranking test and index measurement: slack-adjusted DEA and an application to Japanese agriculture cooperatives,” Omega, vol. 27, no. 3, pp. 315–326, 1999.

G. R. Jahanshahloo, H. V. Junior, F. H. Lotfi, and D. Akbarian, “A new DEA ranking system based on changing the reference set,” European Journal of Operational Research, vol. 181, no. 1, pp. 331–337, 2007.

W.-M. Lu and S.-F. Lo, “An interactive benchmark model ranking performers: application to financial holding companies,” Mathematical and Computer Modelling, vol. 49, no. 1-2, pp. 172–179, 2009.

J.-X. Chen and M. Deng, “A cross-dependence based ranking system for efficient and inefficient units in DEA,” Expert Systems with Applications, vol. 38, no. 8, pp. 9648–9655, 2011.

L. Friedman and Z. Sinuany-Stern, “Scaling units via the canonical correlation analysis in the DEA context,” European Journal of Operational Research, vol. 100, no. 3, pp. 629–637, 1997.

E. D. Mecit and I. Alp, “A new proposed model of restricted data envelopment analysis by correlation coefficients,” Applied Mathematical Modeling, vol. 3, no. 5, pp. 3407–3425, 2013.

T. Joro, P. Korhonen, and J. Wallenius, “Structural comparison of data envelopment analysis and multiple objective linear programming,” Management Science, vol. 44, no. 7, pp. 962–970, 1998.

X.-B. Li and G. R. Reeves, “Multiple criteria approach to data envelopment analysis,” European Journal of Operational Research, vol. 115, no. 3, pp. 507–517, 1999.

V. Belton and T. J. Stewart, “DEA and MCDA: Competing or complementary approaches?” in Advances in Decision Analysis, N. Meskens and M. Roubens, Eds., Kluwer Academic, Norwell, Mass, USA, 1999.

Z. Sinuany-Stern, A. Mehrez, and Y. Hadad, “An AHP/DEA methodology for ranking decision making units,” International Transactions in Operational Research, vol. 7, no. 2, pp. 109–124, 2000.

G. Strassert and T. Prato, “Selecting farming systems using a new multiple criteria decision model: the balancing and ranking method,” Ecological Economics, vol. 40, no. 2, pp. 269–277, 2002.

M.-C. Chen, “Ranking discovered rules from data mining with multiple criteria by data envelopment analysis,” Expert Systems with Applications, vol. 33, no. 4, pp. 1110–1116, 2007.

J. Jablonsky, “Multicriteria approaches for ranking of efficient units in DEA models,” Central European Journal of Operations Research, vol. 20, no. 3, pp. 435–449, 2012.

Y.-M. Wang and P. Jiang, “Alternative mixed integer linear programming models for identifying the most efficient decision making unit in data envelopment analysis,” Computers and Industrial Engineering, vol. 62, no. 2, pp. 546–553, 2012.

F. Hosseinzadeh Lotfi, M. Rostamy-Malkhalifeh, N. Aghayi, Z. Ghelej Beigi, and K. Gholami, “An improved method for ranking alternatives in multiple criteria decision analysis,” Applied Mathematical Modelling, vol. 37, no. 1-2, pp. 25–33, 2013.

L. M. Seiford and J. Zhu, “Context-dependent data envelopment analysis: measuring attractiveness and progress,” Omega, vol. 31, no. 5, pp. 397–408, 2003.

G. R. Jahanshahloo, M. Sanei, F. Hosseinzadeh Lotfi, and N. Shoja, “Using the gradient line for ranking DMUs in DEA,” Applied Mathematics and Computation, vol. 151, no. 1, pp. 209–219, 2004.

G. R. Jahanshahloo, F. Hosseinzadeh Lotfi, H. Zhiani Rezai, and F. Rezai Balf, “Using Monte Carlo method for ranking efficient DMUs,” Applied Mathematics and Computation, vol. 162, no. 1, pp. 371–379, 2005.

G. R. Jahanshahloo and M. Afzalinejad, “A ranking method based on a full-inefficient frontier,” Applied Mathematical Modelling, vol. 30, no. 3, pp. 248–260, 2006.

A. Amirteimoori, “DEA efficiency analysis: efficient and anti-efficient frontier,” Applied Mathematics and Computation, vol. 186, no. 1, pp. 10–16, 2007.

C. Kao, “Weight determination for consistently ranking alternatives in multiple criteria decision analysis,” Applied Mathematical Modelling, vol. 34, no. 7, pp. 1779–1787, 2010.

M. Khodabakhshi and K. Aryavash, “Ranking all units in data envelopment analysis,” Applied Mathematics Letters, vol. 25, no. 12, pp. 2066–2070, 2012.

M. Zerafat Angiz, A. Tajaddini, A. Mustafa, and M. Jalal Kamali, “Ranking alternatives in a preferential voting system using fuzzy concepts and data envelopment analysis,” Computers and Industrial Engineering, vol. 63, no. 4, pp. 784–790, 2012.

A. Charnes, W. W. Cooper, and S. Li, “Using data envelopment analysis to evaluate efficiency in the economic performance of Chinese cities,” Socio-Economic Planning Sciences, vol. 23, no. 6, pp. 325–344, 1989.

W. D. Cook and M. Kress, “An mth generation model for weak ranging of players in a tournament,” Journal of the Operational Research Society, vol. 41, no. 12, pp. 1111–1119, 1990.

W. D. Cook, J. Doyle, R. Green, and M. Kress, “Ranking players in multiple tournaments,” Computers & Operations Research, vol. 23, no. 9, pp. 869–880, 1996.

M. Martić and G. Savić, “An application of DEA for comparative analysis and ranking of regions in Serbia with regards to social-economic development,” European Journal of Operational Research, vol. 132, no. 2, pp. 343–356, 2001.

I. De Leeneer and H. Pastijn, “Selecting land mine detection strategies by means of outranking MCDM techniques,” European Journal of Operational Research, vol. 139, no. 2, pp. 327–338, 2002.

M. P. E. Lins, E. G. Gomes, J. C. C. B. Soares de Mello, and A. J. R. Soares de Mello, “Olympic ranking based on a zero sum gains DEA model,” European Journal of Operational Research, vol. 148, no. 2, pp. 312–322, 2003.

A. N. Paralikas and A. I. Lygeros, “A multi-criteria and fuzzy logic based methodology for the relative ranking of the fire hazard of chemical substances and installations,” Process Safety and Environmental Protection, vol. 83, no. 2 B, pp. 122–134, 2005.

A. I. Ali and R. Nakosteen, “Ranking industry performance in the US,” Socio-Economic Planning Sciences, vol. 39, no. 1, pp. 11–24, 2005.

J. C. Martin and C. Roman, “A benchmarking analysis of spanish commercial airports. A comparison between SMOP and DEA ranking methods,” Networks and Spatial Economics, vol. 6, no. 2, pp. 111–134, 2006.

R. L. Raab and E. H. Feroz, “A productivity growth accounting approach to the ranking of developing and developed nations,” International Journal of Accounting, vol. 42, no. 4, pp. 396–415, 2007.

R. Williams and N. Van Dyke, “Measuring the international standing of universities with an application to Australian universities,” Higher Education, vol. 53, no. 6, pp. 819–841, 2007.

H. Jürges and K. Schneider, “Fair ranking of teachers,” Empirical Economics, vol. 32, no. 2-3, pp. 411–431, 2007.

D. I. Giokas and G. C. Pentzaropoulos, “Efficiency ranking of the OECD member states in the area of telecommunications: a composite AHP/DEA study,” Telecommunications Policy, vol. 32, no. 9-10, pp. 672–685, 2008.

M. Darvish, M. Yasaei, and A. Saeedi, “Application of the graph theory and matrix methods to contractor ranking,” International Journal of Project Management, vol. 27, no. 6, pp. 610–619, 2009.

E. H. Feroz, R. L. Raab, G. T. Ulleberg, and K. Alsharif, “Global warming and environmental production efficiency ranking of the Kyoto Protocol nations,” Journal of Environmental Management, vol. 90, no. 2, pp. 1178–1183, 2009.

S. Sitarz, “The medal points’ incenter for rankings in sport,” Applied Mathematics Letters, vol. 26, no. 4, pp. 408–412, 2013.

N. Adler, L. Friedman, and Z. Sinuany-Stern, “Review of ranking methods in the data envelopment analysis context,” European Journal of Operational Research, vol. 140, no. 2, pp. 249–265, 2002.

R. G. Thompson, F. D. Singleton, R. M. Thrall, and B. A. Smith, “Comparative site evaluations for locating a high energy physics lab in Texas,” Interfaces, vol. 16, pp. 35–49, 1986.

R. G. Thompson, L. N. Langemeier, C.-T. Lee, E. Lee, and R. M. Thrall, “The role of multiplier bounds in efficiency analysis with application to Kansas farming,” Journal of Econometrics, vol. 46, no. 1-2, pp. 93–108, 1990.

R. G. Thompson, E. Lee, and R. M. Thrall, “DEA/AR-efficiency of U.S. independent oil/gas producers over time,” Computers and Operations Research, vol. 19, no. 5, pp. 377–391, 1992.

R. H. Green, J. R. Doyle, and W. D. Cook, “Preference voting and project ranking using DEA and cross-evaluation,” European Journal of Operational Research, vol. 90, no. 3, pp. 461–472, 1996.

J. Doyle and R. Green, “Efficiency and cross-efficiency in DEA: derivations, meanings and uses,” Journal of the Operational Research Society, vol. 45, no. 5, pp. 567–578, 1994.

J. Zhu, “Robustness of the efficient decision-making units in data envelopment analysis,” European Journal of Operational Research, vol. 90, pp. 451–460, 1996.

L. M. Seiford and J. Zhu, “Infeasibility of super-efficiency data envelopment analysis models,” INFOR Journal, vol. 37, no. 2, pp. 174–187, 1999.

J. H. Dulá and B. L. Hickman, “Effects of excluding the column being scored from the DEA envelopment LP technology matrix,” Journal of the Operational Research Society, vol. 48, no. 10, pp. 1001–1012, 1997.

A. Hashimoto, “A ranked voting system using a DEA/AR exclusion model: a note,” European Journal of Operational Research, vol. 97, no. 3, pp. 600–604, 1997.

M. Khodabakhshi, “A super-efficiency model based on improved outputs in data envelopment analysis,” Applied Mathematics and Computation, vol. 184, no. 2, pp. 695–703, 2007.

M. M. Tatsuoka and P. R. Lohnes, Multivariant Analysis: Techniques For Educational and Psycologial Research, Macmillan Publishing Company, New York, NY, USA, 2nd edition, 1988.

Z. Sinuany-Stern, A. Mehrez, and A. Barboy, “Academic departments efficiency via DEA,” Computers and Operations Research, vol. 21, no. 5, pp. 543–556, 1994.

D. F. Morrison, Multivariate Statistical Methods, McGraw-Hill, New York, NY, USA, 2nd edition, 1976.

S. Siegel and N. J. Castellan, Nonparametric Statistics for the Behavioral Sciences, McGraw-Hill, New York, NY, USA, 1998.

T. Obata and H. Ishii, “A method for discriminating efficient candidates with ranked voting data,” European Journal of Operational Research, vol. 151, no. 1, pp. 233–237, 2003.

M. Soltanifar and F. Hosseinzadeh Lotfi, “The voting analytic hierarchy process method for discriminating among efficient decision making units in data envelopment analysis,” Computers and Industrial Engineering, vol. 60, no. 4, pp. 585–592, 2011.