Research Article A Hybrid Distance-Based Ideal-Seeking Consensus Ranking Model

Ordinal consensus ranking problems have received much attention in the management science literature. A problem arises in situations where a group of k decision makers (DMs) is asked to rank order n alternatives. The question is how to combine the DM rankings into one consensus ranking. Several different approaches have been suggested to aggregate DM responses into a compromise or consensus ranking; however, the similarity of consensus rankings generated by the different algorithms is largely unknown. In this paper, we propose a new hybrid distance-based ideal-seeking consensus ranking model (DCM). The proposed hybrid model combines parts of the two commonly used consensus ranking techniques of Beck and Lin (1983) and Cook and Kress (1985) into an intuitive and computationally simple model. We illustrate our method and then run a Monte Carlo simulation across a range of k and n to compare the similarity of the consensus rankings generated by our method with the best-known method of Borda and Kendall (Kendall 1962) and the two methods proposed by Beck and Lin (1983) and Cook and Kress (1985). DCM and Beck and Lin's method yielded the most similar consensus rankings, whereas the Cook-Kress method and the Borda-Kendall method yielded the least similar consensus rankings.


Introduction
The problem of combining a set of rankings to obtain an overall consensus or compromise ranking representative of the group has been studied by numerous authors and researchers for more than two centuries.Consensus ranking has a strong interdisciplinary nature and is considered in many disciplines like organizational sciences, psychology, public policy administration, marketing research, and management science.Several different approaches have been suggested to aggregate DM responses into a compromise or consensus ranking.Many methods for aggregating individual preferences have been proposed since the early works of Borda [1,2], Black [3], Arrow [4], Goodman and Markowitz [5], Coombs [6], Riker [7], Kemeny and Snell [8], Kendall [9], Inada [10], Davis et al. [11], Bogart [12,13], Bowman and Colantoni [14], Blin and Whinston [15], Keesey [16], and Anscombe [17].
Consensus ranking problems can be categorized into two basic groups-cardinal problems and ordinal problems.Cardinal ranking formulations occur where a decision maker (DM) is able to express the degree of preference of one alternative over another.Cardinal ranking is a form of utility function.In contrast, ordinal rankings do not require a degree of preference.A complete ordinal ranking of n alternatives must be an arrangement of the integers (1,2,...,n).Consensus derivation with cardinal ranking involves the optimization of continuous functions over the entirety of a convex region.However, ordinal problems are more complex because they involve such optimization over a particular discrete region.In this study, we concentrate on ordinal ranking problems.The attractiveness of ordinal representation and formulation is due, in part, to the minimal amount of information required where each DM only expresses a preference of one alternative over another.
The simplest form of consensus derivation from ordinal ranking is the majority rule.Borda [1,2] proposed the "method of marks" to derive a consensus of opinions by determining the average of the ranks assigned by DMs to each alternative with the winning alternative being the one with the lowest average.A similar version of this model was later presented by Kendall [9].Kendall [9] was the first to study the ordinal ranking problem in a statistical framework by approaching the problem as an estimation problem.Kendall's solution-to rank alternatives according to the sums of the ranks-is equivalent to Borda's method of marks.The Borda-Kendall (BAK) technique is the most widely used consensus ranking method in practice because of its computational simplicity (see Cook and Seiford [18], Jensen [19]).However, many authors regard the BAK technique as unstructured and ad hoc.These authors argue that the BAK method does not satisfy the social welfare axioms proposed by Arrow [4].Cook and Seiford [18] further studied the BAK technique and proposed a "minimum variance" method for determining the consensus ranking.Inada [10] also studied the majority rule and showed the conditions under which the majority rule satisfies Arrow's axioms.Bowman and Colantoni [14] and Blin and Whinston [15] presented integer programming models to solve the majority rule problems under transitivity.
A popular method for deriving a consensus is to define a distance function on the set of all rankings and then determine the closest possible ordinal ranking in the minimum distance sense.Kemeny and Snell [8] proposed a distance measure that represented the degree of correlation between a pair of rankings along with a set of axioms similar to those given by Arrow [4].Using their distance measure, Kemeny and Snell [8] proposed the median and mean rankings as acceptable forms of consensus.Bogart [12,13] generalized the Kemeny and Snell [8] theory to a broader group of problems by considering both transitive and intransitive orderings.Cook and Saipe [20] developed a branch and bound algorithm to determine the median of a set of ordinal rankings.Cook and Seiford [21] examined the problem of deriving consensus among a set of ordinal rankings and developed a set of axioms which any "reasonable" distance measure should satisfy.Cook and Kress [22] extended the Kemeny and Snell [8] theory by replacing their binary matrix with a preference intensity matrix representing strength of preference within an ordinal scale.Cook and Kress [23] and Cook et al. [24] further studied the complete preference case using distance functions and developed a network model for deriving the optimal consensus ranking that minimized disagreement among a group of DMs (see Cook and Kress [22]).
Several other researchers have used integer programming and goal programming to solve consensus ranking problems since the early work of Bowman and Colantoni [14] and Blin and Whinston [15].Ali et al. [25] presented an integer programming approach to derive consensus rankings from the distance function.Cook et al. [26] used an extensive simulation experiment to compare the integer-programming approach with a heuristic procedure.Iz and Jelassi [27] used goal programming to measure the individual preferences of group members through an ordinal ranking scheme.González-Pach ón and Romero [28] further developed an interval goal programming model for aggregating incomplete individual patterns of preference in group consensus problems.
Multiple criteria are also commonly used to formulate and solve consensus ranking problems.Cook and Kress [29] proposed a weighted ordinal ranking model in which each of a set of n alternatives was given an ordinal rank on a set of criteria.Their model addressed (1) the importance weight associated with each criterion, (2) the importance of the various positions at which an alternative can be placed, and (3) the precision with which a DM can differentiate among alternatives on any given criterion.Cook et al. [30] extended this concept to the situation where each alternative can be assessed in terms of only a subset of the set of criteria under consideration.Muralidharan et al. [31] have also proposed a consensus ranking model combining ordinal rankings with multiple criteria and importance weights.Valadares Tavares [32] proposed a consensus ranking model in terms of the weights space by avoiding any assumptions about the distance between ranked alternatives.Jensen [19] has suggested that various consensus ranking methods proposed in the literature yield controversial rankings and/or rankings that are vulnerable to considerable dispute.They concluded that an optimal consensus ranking method cannot be dictated to a group of DMs.It is better to understand the strengths and weaknesses of each method and leave the choice of a particular method to the DMs.
In this paper, we propose a new hybrid distance-based ideal-seeking consensus ranking method, that is, the distance-based consensus model (DCM).DCM satisfies the axioms for a "reasonable" distance measure proposed by Arrow [4] and Cook and Seiford [21].We first identify an initial preference matrix showing the number of times each alternative is ranked ahead of each of the other alternatives.Next, we find all possible ideal consensus ranking matrices where the rankings of all n alternatives for all k DMs are identified.Finally, the distance between each ideal matrix and the initial preference matrix is used to find the optimal solution.The optimal solution is the ideal consensus ranking matrix with the minimum distance.The similarity of consensus rankings generated by the different algorithms is largely unknown.We examine the similarity in rankings generated by our method with the best-known method of BAK and two other commonly used techniques proposed by Beck and Lin [33] and Cook and Kress [22].We use a Monte Carlo simulation to examine the extent to which these algorithms yield similar rank ordering across a range of problems with k DMs and n alternatives.
Beck and Lin [33] have developed a procedure for approximating the optimal consensus rankings known as maximize agreement heuristic (MAH).MAH is commonly used in practice because of its simplicity, flexibility, and general performance (see Kengpol and Tuominen [34], Lewis and Butler [35], Tavana et al. [36], Tavana [37,38]).MAH is intended to derive consensus orderings that reflect collective DM agreement.An agreement is defined as a case where alternative i is preferred to alternative j by a given DM and alternative i is ranked above alternative j in the final consensus ranking.A disagreement is a case where alternative i is preferred to alternative j by a DM and alternative j is ranked above alternative i in the final consensus ranking.Under the MAH, alternative i is positioned above alternative j in the final ordering if the difference in total DM agreement and disagreement about the relative orderings of alternatives i and j is positive, and alternative j is positioned above alternative i if this difference is negative.These positioning assignments are iteratively made based on the maximum absolute agreement/disagreement difference of all unassigned alternatives.A detailed explanation of MAH is presented in Appendix A.
Cook and Kress [22] have suggested a more complicated method referred to as the consensus ranking model (CRM) for representing strength of preference within an ordinal scale.In CRM, a DM orders n alternatives in q positions where n ≤ q.The resulting ranking shows the DM order of preference and the relative positioning of the alternatives represents his or her intensity of preference.CRM guarantees common units across DMs if each DM orders the same number of alternatives into the same number of positions.
The remainder of the paper is organized as follows.The next section describes the mathematical details of our model followed by an illustrative example in Section 3 and a description of our study in Section 4. In Section 5, we discuss our results and in Section 6, we present our conclusions.

The model
DCM is a hybrid model that combines the most clear-cut and precise algorithms of MAH (see Beck and Lin [33]) and CRM (see Cook and Kress [22]) into an intuitive and computationally simple model.Like MAH, DCM creates a single preference matrix representing the number of times each alternative is ranked ahead of each of the other alternatives.MAH evaluates each alternative individually, ranking one alternative at a time, rebuilding agreement matrices inconsiderate of any previously ranked alternatives, until all alternatives are ranked.Like CRM, DCM employs the ideal matrix comparison methodology in lieu of the repetitive task of rebuilding agreement matrices in MAH.An ideal matrix is developed for each possible permutation of alternatives.The ideal matrix is an "agreement matrix" that would result if all DMs were in total agreement on the rankings of all alternatives.Distance comparisons (sum of the absolute differences of the actual versus each ideal matrix) are used to find the optimal solution or the "nearest" ideal matrix to the initial preference matrix.
To formulate an algebraic model of DCM, consider a general consensus ranking problem with k DMs and n alternatives.Let us define an initial preference matrix, A = (a i j ), where a i j is the number of times alternative, i is ranked ahead of alternative j.We further define an ideal matrix, C = (c i j ), where the rankings of all n alternatives for all k DMs are identical and c i j = k, when i is ranked ahead of j and c i j = 0, when i is not ranked ahead of j.Note that c ii = 0 for i = 1,2,...,n.
Next, we define a set of properties, definitions, and axioms similar to those proposed by Cook and Kress [22].Consider all the ideal matrices C = (c i j ).Each matrix represents a ranking of the n alternatives by the k DMs.
Property 2.1 (transitivity).For each matrix C, if alternative i is preferred to alternative j and j is preferred to k, then alternative i is preferred to alternative k.Definition 2.2.A ranking C 1 is said to be adjacent to a ranking C 2 if there exist an i and j such that (c 1 i, j = k and c 2 i, j = 0) or (c 1 i, j = 0 and c 2 i, j = k) and (c 1 i, j = c 2 i, j for all other pairs of (i, j)).
Disagreement between two different rankings C i will be measured in terms of a distance function on the set of all the rankings.A set of conditions or axioms which such a distance function should satisfy are as follows.

Axiom 2.4 (metric requirements). For any three rankings
Axiom 2.5 (proportionality).The distance between two adjacent rankings is proportional to the degree of adjacency.
Given these definitions and axioms, the following distance function can be defined between any two rankings C 1 and C 2 : Now, we consider all possible ideal matrices and calculate the distance between each one and the initial preference matrix.We use the distance function defined in (2.1) to calculate the distance between an ideal matrix C and the preference matrix A as follows: (2. 2) The optimal solution is denoted by the ideal consensus ranking matrix that minimizes the distance between C and A.

An illustrative example
The problem of ranking advanced-technology projects by the Shuttle Project Engineering Office at the Kennedy Space Center (KSC) is used to further describe the DCM model developed in this study.Project evaluation is the primary responsibility of the groundsystem working committee, which has seven members.We refer to the seven members as the DMs.Contractors and divisions within KSC routinely submit proposals for evaluation and possible funding.The committee considers the importance of each project relative to the longevity of the space-shuttle program.Each of the seven DMs is asked to provide his or her rankings of the five projects under consideration in this example.Table 3.1 shows the individual rankings for DMs A through G and projects 1 though 5.

Borda-Kendall model (BAK).
Using Borda-Kendall, we calculate the mean value of the ranks for each project over all DMs.For example, in Table 3.1, project 1 was ranked 2nd by DM A, 4th by DM B, 3rd by DM C, 4th by DM D, and 5th by DM E, 1st by DM F, and 3rd by DM G.The mean ranking for project 1 over all DMs is (2 + 4 + 3 + 4 + 5 + 1 + 3)/7 = 3.14.Table 3.2 shows the mean ranked values for all Projects along with the final combined (consensus) rankings.The project with the lowest combined score is most preferred and the project with the highest combined score is lest preferred.The final consensus ranking of the five projects under consideration using BAK is 3-4-1-5-2.

Consensus ranking model (CRM).
Using CRM, we first construct a preference intensity matrix showing the number of times project i is preferred over project j for each DM.CRM evaluates all possible rank order permutations by developing a frequency matrix of total agreement (ideal) for each permutation.There are 120 possible permutations for the five projects under consideration.For each possible permutation, a total agreement frequency matrix is constructed and compared to each DM's preference intensity matrix.Next, for each total agreement frequency matrix, the distance from that matrix to the preference intensity matrix is computed by calculating the sum of the absolute differences between the total agreement frequency matrix and the preference intensity matrix.Table 3.3 shows the preference intensity matrices, the total agreement matrix, absolute differences, and distance sums for the consensus ranking where project 3 is ranked 1st, project 5 is ranked 2nd, project 2 is ranked 3rd, project 1 is ranked 4th, and project 4 is ranked 5th (permutation 3-5-2-1-4).The total agreement matrix shown in Table 3.3, naturally the same for each DM, is the matrix that minimizes the absolute differences across DMs and hence is the matrix for the best of the 120 permutations.In essence, the same process was followed for the other 119 total agreement frequency matrices and the one presented in Table 3.3 is the "best" matrix or stated differently reflects the optimal ranking according to CRM.

Maximize agreement heuristic model (MAH).
MAH uses an agreement matrix to rank each project one-at-a-time.If for project i, P i = 0, implying that no DM prefers project i to any other project, project i is placed at the bottom of the final consensus ranking.However, if for project i, N i = 0, implying that no DM prefers any other alternative over project i, project i is placed at the top of the ranking.Table 3.4 shows the consensus ranking calculations for our example.As shown in Table 3.4(a), there are no zero values in either P or N in this example.Therefore, the difference in total DM agreement and disagreement (P i − N i ) is calculated for each project, and project 2 with the smallest (P i − N i ) is placed at the bottom of the consensus ranking.Since project 2 is assigned a position in the final consensus ranking, it is eliminated from further consideration.The remaining projects form a new matrix and the process is repeated until all projects are ranked.Tables 3.4(b) through 3.4(e) show the remaining iterations.The consensus ranking is 4-3-1-5-2.That is, project 4 is ranked 1st; project 3, 2nd; project 1, 3rd; project 5, 4th; and project 2, 5th.

Hybrid distance-based model (DCM).
Like MAH, DCM creates a single preference matrix representing the number of times each project is ranked ahead of all other projects and like CRM, DCM employs the ideal matrix comparison methodology in lieu of the repetitive task of rebuilding agreement matrices in MAH.The first step in DCM is to develop an initial frequency matrix showing the number of times project i is ranked ahead of project j.Next, we consider an ideal matrix where the rankings of all five projects for all seven DMs are identical.An ideal matrix is developed for each possible permutation of projects.Distance comparisons (sum of the absolute differences of the frequency matrices and the ideal matrices) are used to find the optimal solution or the "nearest" ideal matrix to the frequency matrix.In this example, permutation 4-3-5-1-2 with a total distance of A i 52 is the consensus ranking.Table 3.5 shows the frequency and ideal matrices along with the absolute difference matrix for the consensus ranking solution.

The study
In this study, we used Monte Carlo simulation (see Figure 4.1) to compare the performance of our method (DCM) with BAK (see Kendall [9]), MAH (see Beck and Lin  [33]), and CRM (see Cook and Kress [22]).Our testing platform was a Pentium 4 CPU, 3.33 GHz, with 1.00 GB RAM, running under Microsoft Windows XP.We randomly generated rankings of n alternatives for each of the k DMs.Initial individual rankings for each trial were generated using uniformly distributed random numbers from the Mersenne Twister (see Matsumoto and Nishimura [39]) random number generator.Each of the four consensus algorithms was then used to aggregate the individual rankings into a single consensus ranking of all n items for all k DMs.
For each scenario, the number of alternatives to be ranked, n, varied from 3 to 6.The number of DMs, k, varied from 3 to 7. One thousand repetitions for each n, k combination were conducted.The result of the experiment was twenty separate unique n, k combinations.One thousand repetitions of each unique n, k combination resulted in 20,000 total trials.For each trial, each of the four techniques was used to generate group consensus rankings.Therefore, the analysis generated a total of 80,000 data vectors in the form of group consensus rankings.

The results
For each trial, where each trial begins with the same initial individual DM rankings of the n alternatives, there were four consensus rankings of the alternatives (DCM, BAK, MAH, and CRM).One goal of our Monte Carlo simulation was to examine the extent to which the four algorithms yield similar rank ordering across a range of problems with k DMs and n alternatives.The most commonly used measures of association (similarity) with ordinal rankings are Spearman's rho and Kendall's tau, but these two measures are not identical in magnitude because their underlying logic and computational formulae are quite different.The choice of measures is not a trivial one because Kendall [40] has noted that values of tau and rho are similar at some magnitudes, but differ appreciably at others.Some advantages of tau over rho have long been discussed (see Schaeffer and Levitt [41]).For example, the distribution of tau is normal not only for large values of N (as is rho) but also for very small values (see Kendall [40]).Also, in most instances, rho is a biased estimator, whereas tau provides an unbiased estimate of the true population correlation (see Hays [42]).One study found that, relative to rho, tau provided adequate control aof type I errors and tighter confidence intervals (see Arndt et al. [43]).Despite its advantages relative to rho, Emond and Mason [44] have recently shown that Kendall's tau is a flawed measure of agreement between weak orderings (i.e., when tied rankings are allowed).They therefore presented a new rank correlation coefficient (tau-x) and described its application to consensus ranking problems.Tau-x is an extension of Kendall's tau that handles tied rankings in a different way.It is the unique rank correlation coefficient that is equivalent to the Kemeny-Snell distance metric on the set of all weak orderings of n alternatives.Emond and Mason [44] describe the advantages of tau-x (relative to Kendall's tau) and show that tau-x provides a more mathematically tractable solution because all the ranking information can be summarized in a single combined input matrix.Moreover, tau-x allows researchers to handle consensus ranking problems with weights, ties, and partial inputs.In view of the above, we used tau-x to assess the similarity of consensus rankings generated by the four different approaches.We conducted several analyses to help clarify the results presented in Table 5.1.First, it is noteworthy that across all 120 cells in Table 5.1, the mean similarity (tau-x) of rankings was 0.89 (SD = 0.06).This indicates that the consensus rankings generated by the four algorithms (DCM, CRM, MAH, and BAK) were generally similar.
We examined whether the number of alternatives to be ranked (n) or the number of DMs (k) was related to the similarity of rankings.Across all 120 cells in Table 5.1, the correlation of n with mean similarity (tau-x) of rankings was −0.50 (P < .01).As the number of alternatives to be ranked increased, the similarity of rankings generated by the four algorithms decreased.When there were 3 alternatives to be ranked, the mean similarity among the four algorithms was 0.93.When there were 4 alternatives to be ranked, the mean similarity was 0.91.When there were 5 alternatives to be ranked, the mean similarity was 0.88.When there were 6 alternatives to be ranked, the mean similarity was 0.85.The correlation of k with mean similarity (tau-x) of rankings was −0.24 (P < .01).As the number of decision makers increased, the similarity of rankings generated by the four algorithms tended to decrease.When there were 3 decision makers, the mean similarity among the four algorithms was 0.91.When there were 4 decision makers, the mean similarity was also 0.91.When there were 5 decision makers, the mean similarity was 0.88.When there were 6 decision makers, the mean similarity was 0.89.And when there were 7 decision makers, the mean similarity was 0.87.
Next, to determine whether some pairs of algorithms generated more similar consensus rankings than other pairs, we conducted a one-way analysis of variance (ANOVA) where the dependent variable was the mean similarity (tau-x) of rankings and the independent variable was the six pairs of algorithms (i.e., DCM and MAH, DCM and CRM, DCM and BAK, MAH and CRM, MAH and BAK, and CRM and BAK).Results indicated that there were significant differences in the similarity of rankings generated by different pairs of algorithms (F = 28.52,df = 5,114,P < 0.01).The mean similarity (taux) for each pair of algorithms is listed in Table 5.2.The similarity of rankings generated by different pairs of algorithms ranged from 0.82 to 0.96.DCM and MAH yielded the most similar consensus rankings (tau-x = 0.96), whereas CRM and BAK yielded the least similar consensus rankings (tau-x = 0.82).From the perspective of computational complexity, DCM is much simpler than CRM; it is therefore noteworthy that DCM and CRM yielded quite similar consensus rankings (tau-x = 0.92).
In sum, these results show that, although consensus rankings generated by different algorithms were often similar, differences in rankings among the algorithms were sometimes of sufficient magnitude that they cannot routinely be viewed as interchangeable from a practical perspective.

Conclusions
These results can be interpreted from the perspective of psychometric theory where reliability is often estimated by examining the correlation between two methods of rank ordering individuals (or alternatives).If two methods (e.g., two judges, two "parallel" forms of a test) yield highly similar results, this is viewed as evidence of their reliability.That is, when two judges generate highly similar ratings (or rankings) of a group of individuals or alternatives, their ratings (or rankings) are said to be reliable.Nunnally's [45] classic text notes that the level of "satisfactory" reliability depends on how a measure is used.In the context of basic research, measures with reliability of 0.80 (e.g., the correlation between two "parallel" forms of a test) are useful.In applied settings (e.g., where an exact score on a measure determines whether an applicant will be accepted into a school, program, or organization), higher levels of reliability (above 0.90) are desirable.In this context, it is noteworthy that the mean similarity of rankings (tau-x) generated by DCM and MAH (0.96) and by DCM and CRM (0.92) exceeds the level associated with satisfactory reliability in applied settings.The similarity of rankings generated by other pairs of algorithms exceeds 0.80, thereby indicating that rankings generated by these algorithms are similar but certainly not identical.
Future research should examine how various algorithms perform under different conditions.For example, when there is relatively high agreement among the rankings initially provided by different DMs, consensus controversies are likely to be minimal and a variety of algorithms will lead to similar consensus rankings (Valadares Tavares [32]).Consensus controversies are more likely to arise when there is considerable disagreement among the rankings initially provided by different DMs (especially when these disagreements occur concerning the top or bottom alternatives).In applied settings, consensus controversies are especially likely to arise when there are two or more subsets of DMs, each of whom displays high agreement within the subset but high disagreement with the other subset(s) of DMs.
It would also be useful to understand whether any consensus algorithms closely mirror the consensus decisions reached by interacting groups whose members have first completed individual rankings.That is, individuals sometimes rank alternatives (e.g., job applicants, requests for capital funding) before meeting as a group to discuss their rankings and arrive at a group consensus.In some instances, group members with some characteristics (e.g., high status, assertive) are likely to be more influential than others (e.g., with less status, deferential), and thereby ultimately persuade other group members to accept their rankings.In such cases, the group's consensus ranking will closely correspond to the individual rankings initially generated by some (influential) members but will differ from the individual rankings initially generated by other members.It would be of practical and theoretical value to understand the similarity between consensus rankings arrived at by group discussion and consensus rankings arrived at solely by the application of algorithms such as those examined in this paper.From a practical perspective, if some algorithms yield rankings that closely match the consensus decisions reached by interacting groups, then groups might be more comfortable using those algorithms to combine their individual rankings without investing the sometimes substantial time and energy required to discuss individual rankings and reach consensus.
It is unlikely that any consensus algorithm will be free from criticism in all circumstances (see Jensen [19]).Future research should examine the performance and properties of algorithms under real-world conditions where consensus controversies are likely Madjid Tavana et al. 15 to arise, and explore user reactions to and acceptance of different algorithms.Doing so will enhance the likelihood that consensus algorithms will be adopted by DMs in applied settings.

Appendices
A. Maximize agreement heuristic model (MAH) of Beck and Lin [33] Assume that each one of a group of k DMs has ranked n alternatives.Assuming further that the opinions of the k DMs are to be valued equally, the MAH seeks to arrive at the consensus ranking of the alternatives for the group as a whole.According to Beck and Lin, MAH defines an agreement matrix, A, where each element a i j represents the number of DMs who have preferred Alternative i to Alternative j.Strict preference is important.If a DM is indifferent between i and j, he or she is not counted in a i j .The sum of a i j for each alternative i across all columns represents the positive preference vector, P, where P i = n j=1 a i j ; i = 1,2,3,...,n.Similarly, the sum of a i j for each alternative across all rows represents the negative preference vector, N, where N i = n j=1 a ji ; i = 1,2,3,...,n.If for alternative i, P i = 0, implying that no DM prefers alternative i to any other alternative, alternative i is placed at the bottom (in subsequent iterations, at the next available position at the bottom) of the final consensus ranking.However, if for alternative i, N i = 0, implying that no DM prefers any other alternative over alternative i, alternative i is placed at the top (in subsequent iterations, at the next available position at the top) of the ranking.
When there are no zero values in either P or N, the difference in total DM agreement and disagreement (P i − N i ) is calculated for each alternative, and alternative i with the largest absolute difference |P i − N i | is considered.If (P i − N i ) is positive, alternative i is placed in the next available position at top of the final consensus ranking, and if the difference is negative, alternative i is placed in the next available position at the bottom of the consensus ranking.Any ties are broken arbitrarily.Once an alternative is assigned a position in the final consensus ranking, that alternative is eliminated from further consideration.The remaining alternatives form a new matrix and the process is repeated until all alternatives are ranked.

B. Consensus ranking model of Cook and Kress [22]
Assume that a group of k DMs has ranked n alternatives (v 1 ,...,v n ).Each DM ranks the alternatives.A preference intensity matrix, M = {m i j }, is given for each DM where m i j is the number of times v i is preferred over v j (m i j = 0 if v i is tied with v j ).A consensus ranking matrix, B = {b i j }, is the matrix that minimizes the distance between B and the matrices

Table 5 .
1. Mean similarity (tau-x) of consensus rankings generated by four algorithms for combinations of n alternatives ranked by k decision makers.

Table 5 .
2. Mean similarity (tau-x) for each pair of consensus algorithms.Note that homogeneous subsets (based on post-hoc Tukey HSD tests) share a common superscript.