MPE Mathematical Problems in Engineering 1563-5147 1024-123X Hindawi 10.1155/2018/3695627 3695627 Research Article Complementary Judgment Matrix Method with Imprecise Information for Multicriteria Decision-Making http://orcid.org/0000-0002-0313-5643 Wang Haichao 1 2 http://orcid.org/0000-0001-7882-2918 Lahdelma Risto 1 3 http://orcid.org/0000-0003-4641-4535 Salminen Pekka 4 Hanne Thomas 1 Aalto University Department of Mechanical Engineering Otakaari 4 FIN-02150 Espoo Finland aalto.fi 2 Institute of Building Environment and Facility Engineering School of Civil Engineering Dalian University of Technology Dalian 116024 China dlut.edu.cn 3 Aalto University Department of Mathematics and Systems Analysis Otakaari 4 FIN-02150 Espoo Finland aalto.fi 4 University of Jyväskylä School of Business and Economics P.O. Box 35 FIN-40014 Finland jyu.fi 2018 9102018 2018 12 02 2018 31 08 2018 06 09 2018 9102018 2018 Copyright © 2018 Haichao Wang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The complementary judgment matrix (CJM) method is an MCDA (multicriteria decision aiding) method based on pairwise comparisons. As in AHP, the decision-maker (DM) can specify his/her preferences using pairwise comparisons, both between different criteria and between different alternatives with respect to each criterion. The DM specifies his/her preferences by allocating two nonnegative comparison values so that their sum is 1. We measure and pinpoint possible inconsistency by inconsistency errors. We also compare the consistency of CJM and AHP trough simulation. Because preference judgments are always more or less imprecise or uncertain, we introduce a way to represent the uncertainty through stochastic distributions, and a computational method to treat the uncertainty. As in Stochastic Multicriteria Acceptability Analysis (SMAA), we consider different uncertainty levels: precise comparisons, imprecise comparisons with a stochastic distribution, and missing comparisons between criteria. We compute rank acceptability indices for the alternatives, describing the probability of an alternative to obtain a given rank considering the level of uncertainty and study the influence of the uncertainty on the SMAA-CJM results.

China National Key Research and Development Program-China-Finland Intergovernmental Cooperation in Science and Technology Innovation 2016YFE0114500 Academy of Finland Funding 299186 Dalian University of Technology
1. Introduction

The complementary judgment matrix (CJM) method is an MCDA (multicriteria decision aiding) method based on hierarchical decomposition of the decision criteria into subcriteria, evaluation of preferences using pairwise comparisons, and aggregating the results into an overall evaluation of the alternatives. The earliest publications about the CJM method are by Lin and Xu , Su , and Dong et al. . As in AHP , the criteria form a hierarchical tree-like structure, where the root node is the overall decision problem, and the branches at each node correspond to criteria or subcriteria. The leaf nodes correspond to the different decision alternatives. Figure 1 illustrates a school selection problem as an example.

Hierarchy structure of a school selection problem.

At each node of the hierarchy, the decision-maker (DM) performs pairwise comparisons between each pair of criteria or subcriteria. At the bottom level, the DM is asked to compare each pair of alternatives with respect to each criterion. Thus, the DM evaluates through pairwise comparisons both the relative importance of different criteria, and the performance of each alternative with respect to these criteria.

The CJM method differs from AHP mainly in five aspects: (1) the pairwise comparisons are expressed differently, (2) a different numerical scale is used to represent the verbal preference statements, (3) a computationally simpler and more intelligible procedure is used to aggregate the comparisons, (4) the set of comparisons can be incomplete, and (5) individual inconsistent comparisons can be automatically detected. In the CJM method, the DM assigns to each pair of compared entities (i, j) nonnegative weights aij, and aji that are complementary, i.e., aij + aji = 1. When comparing criteria, the ratios of these weights correspond to trade-off ratios. We should point out that some variants of the CJM method have a different interpretation for aij. For example Wang and Guo  treat crisp aij values as fuzzy membership function values for the expression that ‘i is more important than j’. Such an interpretation is different from our assumption that aij are related to trade-off ratios. Fuzzy techniques have been applied also with the AHP method .

Subjective comparison values are always more or less uncertain or imprecise. In particular in group decision-making, it may be difficult to represent the preferences of multiple DMs by precise comparison values. Fuzzy set theory has been employed to cope with the uncertainty and vagueness involved in conducting the comparisons between components of a decision model [12, 13]. Some previous works have also treated such imprecision either using interval numbers or fuzzy numbers [1, 1417]. Alternatively, uncertain or imprecise information in MCDA can be represented by stochastic variables and Monte-Carlo analysis [18, 19]. Durbach et al.  developed uncertainty modelling techniques in AHP as extension of Stochastic Multicriteria Acceptability Analysis (SMAA). We present in this paper a way to represent such imprecise or uncertain preference information in the CJM method through stochastic distributions and a computational method to treat this information in the analysis. We do this by introducing a new variant of SMAA  that applies the decision model of the CJM method. As in SMAA, we compute various descriptive measures for the problem. SMAA is based on simulating uncertain criteria evaluations and preferences and collecting statistics on the performance of each alternative. The DMs are given rank acceptability indices for each alternative, describing the variety of different preferences that support an alternative for the best rank or any particular rank. This information can be used for classifying the alternatives into more or less acceptable ones and those that are not acceptable at all. Pairwise winning indices describe the probability of one alternative to be more preferred than another. SMAA also computes central weights describing typical trade-off weights between criteria that make an alternative the most preferred one. It is also possible to measure with confidence factors whether the performance of alternatives has been assessed accurately enough for decision-making.

This paper is organized as follows. Section 2 presents the CJM method with some extensions. Section 3 presents the new SMAA-CJM method. Section 4 demonstrates the method using a small example. Section 5 compares the consistency of preference statements expressed using the CJM and AHP scales through simulation. This is followed by discussion and conclusions.

2. The Complementary Judgment Matrix Method 2.1. Expressing Judgments

At each node of the criteria hierarchy (see Figure 1), the DM performs pairwise comparisons between each pair of criteria or subcriteria to express their relative importance. At the bottom level, the DM is asked to compare the relative performance of each pair of alternatives with respect to each (sub-) criterion. The DM expresses the intensity of his/her preference either by choosing among verbal preference statements, or by giving directly the complementary positive weights aij and aji such that(1)aij+aji=1.The ratio of the complementary weights corresponds to trade-off ratios, i.e., (2)aijaji=wiwj,i,j1,,n,where wi and wj are the trade-off weights for the ith and jth criterion, respectively.

Any number of preference levels can be used in CJM, but to allow comparison with AHP, we use nine levels as in AHP. Table 1 presents the verbal preference statements and corresponding CJM weights in per cent. Equal preference is represented by the 50/50 ratio. The strongest preference is represented by the 90/10 ratio. If the DM’s preference falls between the listed statements, the in-between weights can be used. The scale has uniform step size.

Verbal preference statements and comparison values in CJM (aij), AHP (a~ij), and transformed scales between AHP and CJM.

CJM AHP AHP→CJM CJM→AHP
Verbal statement a i j (%) a ~ i j (%)
i is equally important/good as j 50 1 50.0 1
55 2 66.7 1.22
i is a little more important/better than j 60 3 75.0 1.5
65 4 80.0 1.86
i is moderately more important/better than j 70 5 83.3 2.33
75 6 85.7 3
i is much more important/better than j 80 7 87.5 4
85 8 88.9 5.67
i is extremely more important/better than j 90 9 90.0 9

At each node of the hierarchy, the comparison values are organized into a complementary judgment matrix:(3)aij=0.5a12a1na210.5a2nan1an20.5.

2.2. Comparison with AHP

For comparison, the AHP comparison values a~ij are represented in the second column of Table 1. In AHP the comparison values represent trade-off ratios:(4)a~ij=wiwj,i,j1,,n.To allow comparing the CJM scale with the AHP scale, we set the left-hand-side (LHS) of (2) equal to LHS of (4) to obtain aij/aji=a~ij. Then we solve aji=1-aij from (1) and substitute this into the previous expression. This gives the transformation from AHP comparison values into corresponding CJM weights and vice versa:(5)aij=a~ij1+a~ij,i,j1,,n,(6)a~ij==aij1-aij,i,j1,,n.The last two columns in Table 1 show the standard AHP scale transformed into CJM weights and CJM weights transformed into AHP comparison values. Both scales are equivalent for the first and last preference statement. However, the CJM scale has smaller steps between the weaker preference statements than AHP and larger steps between the stronger preference statements. The CJM scale has earlier been applied in AHP by Salo and Hämäläinen , who named it the balanced scale for AHP. Observe that the verbal preference statements carry only ordinal information, while the numerical values try to represent corresponding judgements on cardinal (ratio) scales. In general, no fixed cardinal scale can represent accurately the subjective verbal preference statements of different DMs, because DMs have different interpretations on the verbal statements and their relative intensities.

2.3. Solving the Weights

The complementary judgement matrix [aij] contains more information than necessary to determine the weights uniquely. The redundant information may serve for detecting inaccuracies and possible errors in the expressed preferences. If the judgment matrix is fully consistent, we can find weights that satisfy each equation (7) as equality. The matrix is consistent if (aij/aji)(ajk/akj) = (aik/aki) for each i, j, k. In practice, the matrix may contain some level of inconsistency. In this case we can solve the weights from (7) in the least squares (LSQ) sense.

Different techniques to solve the weights have been presented in literature. Here we present a technique that is a little simpler and computationally more efficient than the eigenvalue method of AHP. The eigenvalue method requires iterative calculation of the eigenvector while the LSQ solution is obtained in closed form. First, we solve aij from (1) and (2) obtaining(7)aij=wiwi+wj,i,j1,,n.Then we multiply (7) by (wi+wj) to obtain the linear system:(8)aij-1wi+aijwj=0,i,j1,,n.In addition, to get a unique solution, we consider the normalization condition for the weights Σwj = 1. To solve the system we solve (arbitrarily) the last weight from the normalization equation:(9)wn=1-j=1n-1wj,and substitute it into (8). This yields a linear equation system with (n-1) variables and n2 equations. The system can be easily reduced. Firstly, the n equations corresponding to i = j hold trivially and they can therefore be omitted. Secondly, due to symmetry, the error in the equation for aij is the complement to that for aji. Therefore it is necessary to consider only the equations corresponding to either the upper or lower triangle of [aij]. The resulting linear equation system with (n-1) variables and (n2-n)/2 constraints is of the following form:(10)Hw^=b,where w^=[w1,,wn-1]. When n = 2, there is only a single equation and the consistent solution (w1=b1/H11) is trivially found. When n ≥ 3, the system is overdetermined and the LSQ solution is(11)w^=HTH-1HTb.Rather than forming the matrix inverse explicitly, system (11) is solved efficiently by forming the Cholesky factorization of the symmetric matrix HTH = LLT where L is a lower triangular matrix (see e.g., Stewart ). Then vector x is solved from the lower triangular system Lx=HTb and after that w^ from the upper triangular system LTw^=x. After solving w^, the last weight wn is computed from (9).

Observe that this method of solving the weights does not require a complete set of comparisons. A sufficient requirement is that the graph formed by pairwise comparisons between entities is connected. This gives great flexibility for the DM in large problems, where comparing every pair of entities would be too laborious.

LSQ solution of the weights is also applicable with multiple DMs who provide their (precise) comparisons independently. All comparisons are then collected into a common linear equation system (8) from which the weights are solved. This approach finds weights that satisfy different preferences in the LSQ sense. Section 3 describes another way to handle the preferences of multiple DMs.

2.4. Evaluating the Scores in the Hierarchy

After the weights have been computed at each node of the criteria hierarchy, a score sk is computed for each alternative k. At the lowest level criteria nodes t, the criterion score for each alternative equals its weight. At the higher level nodes, the score for each alternative is computed as a weighted average of the scores at the lower level. Writing the node identifier as superscript for scores and weights, we have(12)skt=wktwhentatlowestleveljStwjtskjwhentathigherlevel,where S(t) refers to the set of subnodes of node t in the hierarchy. The overall score for each alternative is the score computed at the top node.

2.5. Measuring the Inconsistency

The redundant information provided by pairwise comparisons serves two purposes in the CJM method. Firstly, the weights solved from the overdetermined system (7)-(8) can provide more accurate preference information compared to the case where only a minimal number of comparisons are made. Secondly, large inconsistency may indicate that the DM has expressed his/her preferences incorrectly. The DM is encouraged to revise his/her comparisons if too large inconsistency is detected.

Xu  suggests that the AHP inconsistency ratio (IR) is computed also in the CJM method to detect excess inconsistency. Before this method can be applied, it is necessary to transform the CJM weights into the corresponding reciprocal matrix [a~ij] of AHP using (6). Then a consistency index CI = (λmax-n)/(n-1) is computed, where λmax is the principal eigenvalue and n is the dimension of the reciprocal matrix. Finally, IR = CI/RI, where the random index RI is the average consistency index of a large number of random reciprocal matrices. If IR exceeds 10%, the DM is urged to revise his/her comparisons.

We suggest here a different technique for the CJM method. We simply compute the inconsistency errors in (7) based on expressed aij and weights from the LSQ solution:(13)eij=aij-wiwi+wj,i,j1,,n.Observe that eij = -eji. If the absolute value |eij| is too large for any of the comparisons, the DM should reconsider his/her comparisons. The advantage of the inconsistency errors is that the DM can understand them easily, because they are directly related to his/her comparison values. Another advantage with inconsistency errors is that they can pinpoint comparisons that are most likely incorrect. If only one or a few comparisons are found too inconsistent, it may be sufficient that the DM only reconsiders these. If many comparisons are inconsistent, we suggest that the DM reconsiders all comparisons. The DM can specify a threshold for the inconsistency errors to identify too inconsistent comparisons based on his/her accuracy level when making the comparisons. We suggest ±0.1 as a reasonable threshold, because it corresponds to one step uncertainty on the verbal scale.

3. The SMAA-CJM Method

Some restrictions of the basic CJM method are that it cannot treat imprecise information, and it does not explicitly support combining the preferences of multiple DMs. Some extensions of the CJM method exist for treating imprecise information as intervals  or fuzzy numbers . In this paper, we extend the CJM method by representing the elements of the complementary judgment matrix [aij] as probability distributions.

The DMs can give their pairwise comparisons either as precise values or as intervals. The inconsistency errors are computed for each DM, and if they are too large, the DMs are allowed to revise their comparisons. In case of intervals, we suggest computing the inconsistency errors based on the midpoints of the intervals. We next combine the individual DMs’ pairwise comparisons into intervals [aijmin,aijmax] where aijmin is the minimal value that any DM has expressed and aijmax is the maximal value. The aggregated comparison values are then represented by stochastic variables with a suitable probability distribution in the intervals. The complementary value pairs (aij, aji) are treated as dependent distributions to make their sum 1. Technically, it is possible to use arbitrary distributions. However, in the absence of information about the distribution shape, we apply a uniform distribution in the interval. More complex distributions can be estimated based on preference information provided by a large number of DMs. If the interval is degenerate, i.e., aijmin=aijmax, we use Dirac’s delta function (the unit impulse function) as the distribution.

After representing the aggregated pairwise comparisons by suitable distributions, the performance of each alternative is analysed through stochastic simulation by drawing simultaneously pairwise comparisons from their corresponding distributions and computing the score for each alternative as in the CJM method. A sufficient number of simulation rounds is between 10 000 and 100 000 . During the simulation, statistics is collected about the weights at different nodes of the hierarchy, the overall score of the alternatives, and their ranking. Based on the statistics, the following descriptive measures are computed for evaluating the alternatives.

Average overall score for different alternatives. This generalizes the crisp CJM overall score to consider imprecise comparison values.

Average criterion score for different alternatives. This generalizes the corresponding crisp CJM criterion scores to consider imprecise comparison values.

The rank acceptability index bir measures the variety of different preferences that grant alternative xi rank r. The rank acceptability indices can be used for ranking the alternatives roughly, or for finding compromise alternatives in case no alternative obtains sufficient acceptability for the first rank. Potential compromise alternatives are those with high acceptability for the best ranks. Alternatives that obtain high acceptability for the worst ranks should be avoided .

The first rank acceptability indexbi1 measures the variety of different preferences that make alternative i most preferred. In other words, the acceptability index measures how widely acceptable the alternative is. The acceptability index can be interpreted as the share of people voting for the alternative, assuming that the applied distribution for comparison values represents the voters’ preferences. Zero acceptability means that the alternative is inefficient, i.e., no preferences make it best .

The pairwise winning indexcik is the probability for alternative i to be more preferred than alternative k. This index can be used to exclude alternatives that are dominated by others  and also for forming a stochastic ranking among the alternatives [29, 30].

The central weight vectorwic describes what kinds of weights are favourable for alternative i, i.e., make it most preferred. The central weights can be presented to the DMs in order to help them understand how different weights correspond to different choices with the assumed preference model. The central weights are undefined for inefficient alternatives .

The confidence factor pic is the probability for alternative i to be most preferred when the central weight vector for that alternative is selected. In other words, the confidence factor measures if the performance of the alternative has been assessed accurately enough, so that it can be selected under favourable preferences between criteria .

4. Example

To illustrate the SMAA-CJM method, we consider the AHP problem for evaluating 3 high schools (A, B, C) in terms of 6 criteria (One,…, Six) . First we evaluate the problem using precise comparisons in CJM and compare the results with AHP. Secondly, we evaluate the problem with smaller number of pairwise comparisons in CJM, thirdly by considering the comparisons as imprecise, and fourthly by assuming that comparison information between criteria is missing.

4.1. Precise Comparisons

In the original AHP problem, the preferences were expressed verbally and mapped on the AHP scale (1, 2, …, 9). For CJM comparisons we use the uniform scale (50%, 55%, …, 90%) presented in Table 1. The resulting CJM comparisons for the problem are shown in Table 2 between the criteria and in Table 3 between the alternatives. To omit redundant information, only the upper triangle of each comparison matrix is presented, because the diagonal elements are equal to 0.5 and the lower triangle elements are equal to the complement of the upper triangle.

Pairwise CJM comparisons (%) between criteria (One, …, Six).

Criterion Two Three Four Five Six
One 70 80 70 60 50
Two 60 30 25 25
Three 35 30 30
Four 30 25
Five 50

Pairwise CJM comparisons between alternatives with respect to different criteria.

One B C Two B C
A 40 45 A 50 50
B 60 B 50

Three B C Four B C

A 70 50 A 90 80
B 30 B 30

Five B C Six B C

A 45 50 A 75 65
B 55 B 40

Solving the weights from the precise CJM comparisons gives the criterion scores, average weights and overall scores for alternatives shown in Table 4(a). Alternative A obtains the highest score 0.41 followed by C (0.30) and B (0.29). With precise information, the alternatives (A, B, C) obtain distinct ranks (1, 3, 2) deterministically. This is indicated by the rank acceptability indices bA,1 = bB,3 = bC,2 = 100% and zero for the remaining indices as well as the pairwise winning indices cA,B = cA,C = cC,B = 100%. However, because subjective information from the DM is always uncertain and B and C obtain almost identical overall score, alternatives B and C could be considered equally good in practice.

CJM criterion scores, criterion weights, and overall scores for alternatives

Alt.crit. One Two Three Four Five Six Score
A 0.27 0.33 0.41 0.74 0.31 0.53 0.41
B 0.43 0.33 0.18 0.08 0.38 0.18 0.29
C 0.30 0.33 0.41 0.18 0.31 0.28 0.30
Weights (%) 26.9 8.3 7.8 10.7 21.6 24.7

AHP criterion scores, criterion weights, and overall scores for alternatives

Alt.crit. One Two Three Four Five Six Score
A 0.16 0.33 0.45 0.77 0.25 0.69 0.40
B 0.59 0.33 0.09 0.05 0.50 0.09 0.36
C 0.25 0.33 0.45 0.17 0.25 0.23 0.24
Weights (%) 33.0 4.7 3.3 9.2 22.6 27.3

Table 4(b) shows the corresponding results using standard AHP. We observe that the CJM results are somewhat different from AHP results. Alternative A obtains almost identical overall score and the best rank with both methods. However, alternatives B and C obtain different overall scores and reversed ranks. Also, the criterion scores and the criterion weights are quite different. The differences are mainly due to the different scales used to represent verbal preference statements in CJM and AHP. Transforming the comparisons on the CJM scale by (6) into AHP comparisons (CJM->AHP column in Table 1) and evaluating the model using AHP gives nearly identical results as the CJM method. This is natural, because both the LSQ solution and eigenvalue method give the same weights with consistent comparisons and, as we will see, in this example only small inconsistency is present.

Next we evaluate the consistency of the CJM comparisons in terms of the inconsistency errors (eij) introduced in this paper and in terms of the inconsistency ratio (IR) of AHP. For the comparisons between criteria (Table 2) the maximal inconsistency error -0.14 occurs between criteria Two and Four (by formula (7) with aij=0.30, wTwo=8.3% and wFour=10.7%). All other comparisons are well below the threshold of ±0.1. Therefore, we would suggest the DM to reconsider his/her comparisons between criteria and in particular the comparison between criteria Two and Four. The negative sign of the inconsistency error indicates that the expressed comparison value (30%) is smaller than the consistent value (44%). Instead of the preference statement ‘criterion Four is ‘moderately more important than Two’, a consistent statement would have been between ‘a little more important’ and ‘equally important’.

The IR for the comparisons between criteria is 0.02 (CI = 0.03, RI = 1.25) which is clearly below the suggested threshold 0.1 for sufficient consistency. Because IR is a kind of average measure for inconsistency, it is insensitive to a single inconsistent comparison and fails to detect the clearly inconsistent comparison. For related discussion, see Bana e Costa and Vansnick . Also, the IR does not pinpoint the most likely sources of inconsistency. In the original AHP model the comparisons between criteria were slightly too inconsistent with CI=0.137, RI=1.24 and IR=0.109.

For the comparisons between alternatives (Table 3) all inconsistency errors are clearly below the suggested threshold, with the largest inconsistency error of -0.02 found at criterion One between alternatives A and C. Also the IRs are well below the consistency threshold (IR = 0.004, 0, 0, 0.00014, 0, and 0.0006, correspondingly).

4.2. Smaller Number of Pairwise Comparisons

The disadvantage with performing the full set of pairwise comparisons between each pair of entities (alternatives or criteria) is that the number of comparisons increases quadratically by the number of compared entities. With n compared entities, the number of pairwise comparisons is n(n-1)/2. When the number of compared entities is large, performing the full set comparisons is in practice infeasible due to the large cognitive load on the decision-maker. For example, Saaty and Ozdemir  suggest that the full set of comparisons with more than 7 entities inherently lead to inconsistency. Bozóki et al.  proved the increase of inconsistency empirically and showed that a subset of the comparisons can be used to approximate the results based on the full set of comparisons.

The LSQ method for solving the weights in CJM works also with a subset of pairwise comparisons, provided that the graph formed by pairwise comparisons between entities is connected. This means that for each pair of entities A, B, they are either compared directly, or there exists a path of comparisons connecting A and B via other entities. The minimal sufficient number of comparisons is n-1. Of course, in that case no redundant information is provided, equation system (8) has a unique solution, and the LSQ method is not required.

As a compromise between the maximal and minimal number of comparisons, we suggest (for problems with many entities) comparing each entity systematically only with a small number of other entities. In the following, we suggest two methods for reducing the number of comparisons.

Before making comparisons, the DM should first order the entities according to their importance or preference. Saaty  applied such ordering in an example, although he did not explicitly define the ordering as part of the AHP procedure. Ordering the entities simplifies making the pairwise comparisons, because the mutual order of each pair of entities has already been determined and only the (verbal or numerical) preference statement is required. We believe that this reduces the risk of mistakes in preference statements. Also, individual ordinally inconsistent comparisons are easy to spot immediately from the comparison matrix when it is ordered this way (see Section 5 and Xu et al. ).

Method 1.

After ordering the entities, the DM compares each entity only with the two following entities. The necessary number of comparisons is then 2n-3. For example, with 6 entities, a total of 9 comparisons are required: 1&2, 1&3, 2&3, 2&4, 3&4, 3&5, 4&5, 4&6, 5&6. This method has the advantage that each entity is compared only with entities that are as similar as possible; no more than two places before or after itself. Comparisons between extremely different entities are avoided. This is good because it is difficult to express accurate comparisons between very different entities.

Method 2.

After ordering the entities, the DM compares each entity only with the first and last entity. With 6 entities this method results in comparisons: 1&6, 1&2, 2&6, 1&3, 3&6, 1&4, 4&6, 1&5, 5&6. This method has the advantage that it reduces the DMs cognitive load in the comparisons because during the process he/she becomes ‘more familiar’ with the first and last entities, and at least one of them appears in every comparison.

We should point out that we are not suggesting any particular order in which the subset of comparisons are made. Bozóki et al.  came to the slightly surprising conclusion that the order in which the pairwise comparisons are made has no effect on the consistency. However, the order is not irrelevant, because the order of questions may affect the results through the anchoring bias.

We illustrate smaller sets of comparisons using the school selection problem. After ordering the criteria into importance order, the full set of comparisons is shown in Figure 2. Note that after ordering, all consistent comparisons in the upper triangle should be at least 50%. Also, consistent comparisons should satisfy ai,jai,j+1 along rows and ai+1,jai,j along columns of the CJM. Some small violations of the latter conditions do appear in Figure 2.

Pairwise CJM comparisons (%) between criteria in importance order and subsets 1 & 2.

The comparisons according to the first method appear on the bottom two diagonals and for the second method on the first row and last column of Figure 2. Table 5 shows the criterion weights and overall scores for alternatives using the full set of comparisons and using the subsets by the first and second method for reducing the number of comparisons. Because there is some inconsistency in the comparisons, it is natural that the weights differ depending on which subset of comparisons is included. The differences are quite small, maximally about 3% points. However, the importance order of weights is the same in all three cases. The overall scores for the alternatives are in practice identical using different sets of comparisons, resulting in the same recommendation: Alternative A is best and B and C are in practice equally good.

Criterion weights (%) and overall scores for alternatives using full set of comparisons, and subsets of comparisons.

Criterion weights Overall scores
Comparisons One Two Three Four Five Six A B C
Full set 26.9 8.3 7.8 10.7 21.6 24.7 0.41 0.29 0.30
Subset 1 29.8 6.3 4.9 9.7 22.5 26.7 0.40 0.30 0.29
Subset 2 26.9 8.3 7.8 10.7 21.6 24.7 0.42 0.29 0.29
4.3. Imprecise Comparisons

Next we introduce imprecision to the problem and analyse it using SMAA-CJM. We assume that the uncertainty of each comparison aij in Tables 2 and 3 is ±10% points and use a uniform distribution to represent this uncertainty. Solving the model with imprecision gives almost identical criterion scores, average weights, and overall scores for alternatives as with precise comparisons (Table 4(a)). However, the ranking of the alternatives becomes uncertain, as shown by the rank acceptability indices and pairwise winning indices in Figure 3.

Rank acceptability indices and pairwise winning indices with imprecise comparisons.

Rank acceptability indices

Pairwise winning indices

Alternative A with 99.97% first rank acceptability is in practice the only candidate for the first rank while alternatives B and C obtain only 0.06% and 0.01% acceptability for the first rank. However, the second rank acceptability of B and C is now 42% and 58%, which shows clearly that we cannot be sure about which alternative is the second best one. The same conclusions can be made from the pairwise winning indices cA,B = 99.94%, cA,C = 99.99%, but alternatives B and C win each other with 42% and 58% probability, correspondingly. In this case the pairwise winning indices between B and C are almost identical to their second rank acceptability indices because A obtains almost always the first rank. The central weights for A nearly coincide with the average weights (bottom row of Table 4(a)) resulting into confidence factor pAc= 100%. The confidence factors of B and C are 0.02%, which means that even considering the uncertainty of criteria preference information, these alternatives are in practice inefficient.

4.4. Missing Comparisons between Criteria

We demonstrate next how the SMAA-CJM method can be used when no comparison information among criteria (Table 2) exists. We consider only the pairwise comparisons between alternatives with respect to different criteria (Table 3) and the associated ±10% point imprecision for the comparisons aij between alternatives with respect to each criterion. We represent missing preferences among the criteria by nonnegative normalized weights, wj≥0, Σwj=1, that follow a uniform joint distribution.

The resulting criterion scores are identical with the previous analysis, because we have the same comparison and uncertainty information between alternatives. The uniform weight distribution results into average weights for each criterion equal to 1/6 ≈ 16.7%. The average overall scores for the alternatives A, B, C are 0.43, 0.26, and 0.30, correspondingly.

Figure 4 shows the resulting rank acceptability indices and pairwise winning indices. We observe that the increased uncertainty in the comparisons is reflected as increased uncertainty in the ranking. Now both B and C with 4.7% and 3.5% first rank acceptability could, at least in theory, be the best one under suitable preferences for criteria.

Rank acceptability indices and pairwise winning indices with missing comparisons between criteria.

Rank acceptability indices

Pairwise winning indices: School 1 versus 2

Table 6 shows the central weights and confidence factors for the alternatives. The central weights identify what kind of trade-off weights between criteria make each alternative most preferred. We can see that different alternatives are favoured by dramatically different weights. For example, alternative B would require about 39% of the weight to be placed on criterion One alone. The confidence factor for B is 55%, which means that even with its central weights B will not be the best alternative with certainty. For C, the confidence factor is even lower, only 19%. This means that the criteria measurements are too uncertain to justify choosing C even with its central weights.

Central weights and confidence factors for the alternatives (%).

Alt. One Two Three Four Five Six p c
A 15 16 17 18 16 18 100
B 39 16 8 4 26 7 55
C 19 23 29 4 19 7 19
5. Comparison of CJM and AHP Scales

The most significant differences between CJM and AHP results stem from the different scales used to represent verbal preference statements. Because the verbal preference statements carry only ordinal information and DMs have different interpretation of the intensities of the preference statements, no fixed numerical scale can properly represent the ordinal verbal comparisons. However, the integer scale of AHP from 1 to 9 is particularly problematic, because in many cases it is impossible to express consistent comparisons between three or more entities. For example, if criterion 1 is moderately more important than criterion 2 (a~12=5) and criterion 2 is moderately more important than criterion 3 (a~23=5), a consistent comparison between criteria 1 and 3 is impossible to express using the AHP scale (a~13=25). This problem occurs partly because the AHP scale is too sparse for the weaker preference statements and too dense for stronger statements. A cascade of a few comparisons even with very weak preference values exceeds soon the strongest value. Because the CJM scale is better balanced, i.e., denser for weaker preferences and sparser for stronger preferences, we wanted to test if it performs better than the AHP scale. With the CJM scale, the above example results into CJM comparisons a12=a23=70% which correspond to a~12=a~23=2.33. The consistent comparison a~13=5.44 corresponds to CJM comparison a13=84% which is very close to scale value 85%.

To compare the two scales, we generated a large number of random ordinally consistent comparison matrices for different numbers of criteria. Ordinal consistency means the natural transitivity of preference statements that a consistent and logical DM should follow . The transitivity of preference statements can be expressed as

If A is preferred to B by intensity aAB and B is preferred to C by intensity aBC, then A is preferred to C by intensity aACmax{aAB,aBC}

Random ordinally consistent comparison matrices between n entities and m levels of preference intensity can be generated by generating first a set of random weights w1>w2,…,>wn and thresholds t1<t2<⋯<tm-1 in range [0,1] and then setting each comparison value aij equal to the smallest intensity k such that wi-wj<tk. We note that the simulated weights satisfy the condition of order preservation (COP) with respect to the generated comparisons .

Table 7 shows the average IRs and inconsistency errors for 1000 randomly generated ordinally consistent comparison matrices using the AHP and CJM scales. Using the AHP scale, the IR is on average in the range [0.16,0.18], i.e., clearly above the suggested consistency threshold of 0.1. This means that when the DM is ordinally consistent, the AHP scale comparisons are on average cardinally inconsistent. For the CJM scale the average IR is in range [0.059,0.064], i.e., clearly consistent. When measured by the IR, the CJM scale gives from 2.6 to 2.9 times better consistency than the AHP scale.

Inconsistency ratios and inconsistency errors with AHP and CJM scales.

Inconsistency ratio (IR) Inconsistency error
Number of criteria AHP CJM AHP/CJM ratio AHP CJM AHP/CJM ratio
3 0.18 0.062 2.9 0.090 0.049 1.8
4 0.16 0.059 2.8 0.17 0.098 1.7
5 0.17 0.064 2.6 0.22 0.13 1.6
6 0.17 0.063 2.7 0.25 0.16 1.5
7 0.17 0.061 2.8 0.27 0.19 1.5
8 0.17 0.063 2.7 0.30 0.21 1.4

In terms of the average inconsistency error the results are similar. The inconsistency error for each matrix is the maximal |eij| and Table 7 shows the average for each number of criteria. Using the AHP scale, the inconsistency error is in the range [0.09,0.30] increasing with the number of criteria. This is natural, because with larger number of comparisons, the maximal error is likely to be larger. Except for the 3 criterion case, the inconsistency errors exceed the suggested threshold of 0.1 clearly. Using the CJM scale, the inconsistency errors are in the range[0.049,0.207] exceeding the suggested threshold with 5 or more criteria. Also when measured by the inconsistency error, the CJM scale gives clearly better consistency than the AHP scale: from 1.4 to 1.8 times better.

We conclude that although not perfect, the more balanced CJM scale is clearly better than the AHP scale in its ability to represent the cardinal preferences of an ordinally consistent DM. Similar results were previously obtained by Pöyhönen et al.  who compared the two scales empirically using a group of students.

6. Discussion

The comparison values of CJM have a natural interpretation. Considering only two criteria at a time, the DM can interpret the comparison values as trade-off weights that he/she assigns to the criteria. The DM can express these weights either as a normalized complementary pair (e.g., 0.8, 0.2) or as a pair of nonnegative numbers (e.g., 4, 1) that are normalized to satisfy the complementarity condition. Similarly, when comparing two alternatives with respect to a criterion, the DM is in effect distributing partial value between the two alternatives. Of course, it is also possible to evaluate the performance of alternatives through other techniques and to use pairwise comparisons only for assessing criteria weights. For example, criteria measured on natural scales can be normalized to partial values in range [0,1]. This makes the CJM method conformant with linear value theory.

No fixed cardinal scale can represent precisely the verbal preference statements of different DMs. Instead, each DM could define their own cardinal scale that represents his/her verbal statements. Alternatively, DMs could express their preferences cardinally in the first place. In practice this may be difficult for many DMs.

Another approach for cardinalizing ordinal preference statements is based on ordinal regression, as in UTA , MACBETH , UTAGMS , and GRIP  methods. These methods use verbal preference and indifference statements between pairs of alternatives or criteria to assess constraints on the parameters of an additive value function.

7. Conclusions

We have introduced the SMAA-CJM method for representing uncertain or imprecise information through stochastic distributions in the Complementary Judgment Matrix method and a simulation approach for analysing the resulting model. A particular strength of the method is that it allows flexible modelling of different kinds of imprecision, uncertainty, or even partially missing preference information. This is useful in decision processes, where the information is gradually refined during the process. The method is also suitable for group decision-making problems, where it is difficult for DMs to agree on precise pairwise comparisons. The method allows using distributions that include each DM’s preferences. Alternatively, the weight solution method of CJM can find weights that match different DMs’ preferences as well as possible in the LSQ sense.

We also introduced the inconsistency error as a measure for how consistent each comparison is, i.e., how much each comparison value differs from the consistent value. These measures are easy for the DMs to understand, because they are directly related to their comparison values. Another advantage of inconsistency errors is that they identify comparisons that are most likely incorrect. A reasonable threshold for the inconsistency errors is ±0.1, corresponding to one step uncertainty on the verbal scale. Also this threshold is easy for the DMs to understand.

We conducted simulation experiments using a large number of different sized (3,…,8 criteria) ordinally consistent comparison matrices. The results showed that the balanced comparison scale of CJM results in more consistent results than the standard AHP scale. The consistency was better in terms of both the inconsistency ratio (IR) of AHP and the inconsistency error of SMAA-CJM. An earlier empirical study with students gave similar results.

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper. The funds received did not lead to any conflicts of interest.

Acknowledgments

This work was supported by the China National Key Research and Development Program-China-Finland Intergovernmental Cooperation in Science and Technology Innovation (Funding no. 2016YFE0114500) and Academy of Finland Funding (Grant no. 299186). The authors would also like to acknowledge the ‘Xinghai’ Talent Project of Dalian University of Technology.

Lin J. C. Xu Z. S. A new scale in fuzzy AHP Operations Research and Management Science 1998 7 2 37 40 Su W. H. Study on Multicriteria Comprehensive Evaluation Theories and Methodologies [Ph.D. thesis] 2000 Xiamen, Fujian, China Xiamen University Dong R. M. Wang Q. L. Chen J. D. Discussion on a certain type of judgement matrices Journal of Beijing Institute of Technology 2000 20 4 407 411 MR1780141 Saaty T. L. The Analytic Hierarchy Process 1980 New York, NY, USA McGraw-Hill MR773297 Zbl0587.90002 Wang X.-Q. Guo Q. The priority of fuzzy complementary judgment matrix and its application in procurement tenders for government project Proceedings of the 17th International Conference on Industrial Engineering and Engineering Management (IE and EM '10) October 2010 148 151 2-s2.0-78650607212 Li M. Extension of axiomatic design method for fuzzy linguistic multiple criteria group decision making with incomplete weight information Mathematical Problems in Engineering 2012 2012 17 634326 10.1155/2012/634326 Zbl1264.90192 Wang M.-K. Hwang K. P. Using FAHP methods evaluation and screening of intellectual property rights managers in Taiwan Asia-Pacific Journal of Operational Research 2014 31 6 1 25 10.1142/S0217595914500481 MR3291603 Li Y. Li Y. H. Han Z. J. Li G. Q. Gao K. Wang Z. H. Determination of optimal opening scheme for electromagnetic loop networks based on fuzzy analytic hierarchy process Mathematical Problems in Engineering 2016 2016 11 3965608 10.1155/2016/3965608 Arsovski S. Todorovic G. Lazić Z. Arsovski Z. Ljepava N. Aleksic A. Model for selection of the best location based on fuzzy AHP and Hurwitz methods Mathematical Problems in Engineering 2017 2017 12 2803461 10.1155/2017/2803461 2-s2.0-85031934674 Liang H. Zhang S. Su Y. Evaluating the efficiency of industrialization process in prefabricated residential buildings using a fuzzy multicriteria decision-making method Mathematical Problems in Engineering 2017 2017 12 6078490 10.1155/2017/6078490 Shieh M.-D. Li Y. F. Yang C.-C. Product form design model based on multiobjective optimization and multicriteria decision-making Mathematical Problems in Engineering 2017 2017 15 5187521 10.1155/2017/5187521 Shams F. Mohamed S. Fayek A. R. Improving consistency evaluation in fuzzy multi-attribute pairwise comparison-based decision-making methods Asia-Pacific Journal of Operational Research 2014 31 4 1 22 10.1142/S0217595914500249 MR3249601 Zhou H. Wang J. Li X.-E. Wang J.-Q. Intuitionistic hesitant linguistic sets and their application in multi-criteria decision-making problems Operational Research 2016 16 1 131 160 2-s2.0-84941662117 10.1007/s12351-015-0199-4 Xu Z. S. A practical method for priority of interval number complementary judgement matrix Surveys in Operations Research and Management Science 2001 10 1 16 19 Shi W. Peng X. Li J. Gao Y. A new ranking method in the interval number complementary judgement matrix Proceedings of the 5th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD '08) October 2008 Jinan Shandong, China 147 150 10.1109/FSKD.2008.273 Liu S. Zhang Y. T. Yu D. Y. Consistency improvement and priority of triangular fuzzy number complementary judgment matrix based on niche genetic algorithm Systems Engineering - Theory & Practice 2011 31 3 522 529 Xu Z. S. Research on compatibility and consistency of fuzzy complementary judgement matrices Journal of PLA University of Science & Technology 2002 3 2 94 96 Wudhikarn R. Chakpitak N. Neubert G. Use of an analytic network process and Monte Carlo analysis in new product formula selection decisions Asia-Pacific Journal of Operational Research 2015 32 2 1 27 10.1142/S0217595915500074 MR3326748 He J. Feng C. Hu D. Liang L. A decision model for emergency warehouse location based on a novel stochastic MCDA method: evidence from China Mathematical Problems in Engineering 2017 2017 10 7804781 10.1155/2017/7804781 Durbach I. Lahdelma R. Salminen P. The analytic hierarchy process with stochastic judgements European Journal of Operational Research 2014 238 2 552 559 10.1016/j.ejor.2014.03.045 MR3210947 Zbl1338.90199 2-s2.0-84901650261 Lahdelma R. Hokkanen J. Salminen P. SMAA—stochastic multiobjective acceptability analysis European Journal of Operational Research 1998 106 1 137 143 10.1016/s0377-2217(97)00163-x 2-s2.0-0000936944 Lahdelma R. Salminen P. SMAA-2: stochastic multicriteria acceptability analysis for group decision making Operations Research 2001 49 3 444 454 10.1287/opre.49.3.444.11220 Zbl1163.90552 2-s2.0-0035328540 Lahdelma R. Salminen P. The shape of the utility or value function in stochastic multicriteria acceptability analysis OR Spectrum 2012 34 4 785 802 10.1007/s00291-011-0244-5 MR2982040 Zbl1282.91086 2-s2.0-84866728895 Salo A. A. Hämäläinen R. P. On the measurement of preferences in the analytic hierarchy process Journal of Multi-Criteria Decision Analysis 1998 6 309 319 Stewart G. W. Matrix Algorithms 1998 volume I: Basic Decompositions Philadelphia, PA, USA Society for Industrial and Applied Mathematics 10.1137/1.9781611971408 MR1653546 Xu Z. Two approaches to improving the consistency of complementary judgement matrix Applied Mathematics-A Journal of Chinese Universities Series B 2002 17 2 227 235 10.1007/s11766-002-0049-z MR1904920 Zbl1014.91009 Tervonen T. Lahdelma R. Implementing stochastic multicriteria acceptability analysis European Journal of Operational Research 2007 178 2 500 513 10.1016/j.ejor.2005.12.037 Zbl1107.90026 2-s2.0-33751010777 Leskinen P. Viitanen J. Kangas A. Kangas J. Alternatives to incorporate uncertainty and risk attitude in multicriteria evaluation of forest plans Forest Science 2006 52 3 304 312 2-s2.0-33744966635 Loikkanen O. Lahdelma R. Salminen P. Multicriteria evaluation of sustainable energy solutions for Colosseum Sustainable Cities and Society 2017 35 289 297 2-s2.0-85028076511 10.1016/j.scs.2017.07.019 Kirppu H. Lahdelma R. Salminen P. Multicriteria evaluation of carbon-neutral heat-only production technologies for district heating Applied Thermal Engineering 2018 130 466 476 2-s2.0-85036640838 10.1016/j.applthermaleng.2017.10.161 Hauser D. Tadikamalla P. The analytic hierarchy process in an uncertain environment: a simulation approach European Journal of Operational Research 1996 91 1 27 37 10.1016/0377-2217(95)00002-x Zbl1126.90354 2-s2.0-0002086469 Bana e Costa C. A. Vansnick J.-C. A critical analysis of the eigenvalue method used to derive priorities in AHP European Journal of Operational Research 2008 187 3 1422 1428 10.1016/j.ejor.2006.09.022 MR2378343 Zbl1137.91350 2-s2.0-36849024019 Saaty T. L. Ozdemir M. S. Why the magic number seven plus or minus two Mathematical and Computer Modelling 2003 38 3-4 233 244 10.1016/S0895-7177(03)90083-5 MR2004992 Zbl1106.91312 2-s2.0-0141838925 Bozóki S. Dezső L. Poesz A. Temesi J. Analysis of pairwise comparison matrices: an empirical research Annals of Operations Research 2013 211 1 511 528 10.1007/s10479-013-1328-1 MR3151450 Xu Y. Gupta J. N. D. Wang H. The ordinal consistency of an incomplete reciprocal preference relation Fuzzy Sets and Systems 2014 246 62 77 2-s2.0-84900450981 10.1016/j.fss.2013.08.001 Zbl1314.91103 Pöyhönen M. A. Hämäläinen R. P. Salo A. A. An experiment on the numerical modelling of verbal ratio statements Journal of Multi-Criteria Decision Analysis 1997 6 1 1 10 2-s2.0-79551623331 10.1002/(SICI)1099-1360(199701)6:1<1::AID-MCDA111>3.0.CO;2-W Zbl0892.90001 Jacquet-Lagrèze E. Siskos Y. Assessing a set of additive utility functions for multicriteria decision making: The UTA method European Journal of Operational Research 1982 10 2 151 164 Bana e Costa C. Vansnick J.-C. MACBETH—an interactive path towards the construction of cardinal value functions International Transactions in Operational Research 1994 1 4 489 500 10.1016/0969-6016(94)90010-8 Zbl0857.90004 Greco S. Mousseau V. Słowiński R. Ordinal regression revisited: multiple criteria ranking using a set of additive value functions European Journal of Operational Research 2008 191 2 416 435 10.1016/j.ejor.2007.08.013 MR2435124 Zbl1147.90013 Figueira J. R. Greco S. Słowiński R. Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method European Journal of Operational Research 2009 195 2 460 486 10.1016/j.ejor.2008.02.006 MR2482445