Advances in Decision Sciences

Volume 2013 (2013), Article ID 697072, 9 pages

http://dx.doi.org/10.1155/2013/697072

Research Article

Verbal Decision Analysis: Foundations and Trends

Helen M. Moshkovich and Alexander I. Mechitov

College of Business, University of Montevallo, Station No. 6530, Montevallo, AL 35115, USA

Received 23 April 2013; Accepted 12 September 2013

Academic Editor: David Bulger

Copyright © 2013 Helen M. Moshkovich and Alexander I. Mechitov. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

The primary goal of research in multiple criteria decision analysis is to develop tools to help people make more reasonable decisions. In many cases, the development of such tools requires the combination of knowledge derived from such areas as applied mathematics, cognitive psychology, and organizational behavior. Verbal Decision Analysis (VDA) is an example of such a combination. It is based on valid mathematical principles, takes into account peculiarities of human information processing system, and fits the decision process into existing organizational environments. The basic underpinnings of Verbal Decision Analysis are demonstrated by early VDA methods, such as ZAPROS and ORCLASS. New trends in their later modifications are discussed. Published applications of VDA methods are presented to support the findings.

1. Introduction

A large number of decision making situations require taking into consideration multiple, often conflicting, influencing factors or criteria. A number of different multiple criteria decision aids (MCDA) have been developed to support this class of decision problems. In the majority of multiple criteria tasks, the “objectively” best alternative may often not be defined. As a result, in spite of crucial differences in approaches, many methods dealing with multiple criteria environments are based on subjective information about relative importance of objectives and criteria to the decision maker.

The subjective nature of information in the majority of multiple criteria models makes it a challenge to obtain consistent and accurate results in such tasks. As a result, elicitation of the decision makers’ preferences should take into account peculiarities of human behavior in the decision processes. This is the central goal of Verbal Decision Analysis or VDA [1, 2].

The Verbal Decision Analysis is a framework for designing methods of MCDA by using preferential information from the decision makers in the ordinal form, a type of judgment known to be much more stable and consistent. VDA is based on the same principles as multiattribute utility theory (MAUT) [3] but is oriented on using the verbal form of preference elicitation and on evaluation of alternative decisions without resort to numbers.

Traditional methods of VDA were oriented on problems with a rather large number of alternatives but a relatively small number of criteria. They were designed to elicit a sound preference relationship that could be applied to future sets of real alternatives, while many other methods, for example, outranking methods [4] or AHP [5], tended to compare a given set of alternatives.

New developments in the area of VDA illustrate some changes in the goals and ways of verbal analysis. The paper reviews the main principles of Verbal Decision Analysis, describes classical methods associated with this framework, and then analyzes new trends in the development and application of new methods.

2. Basic Principles of Verbal Decision Analysis (VDA)

Term “Verbal Decision Analysis” was introduced in 1997 by Larichev and Moshkovich [1] though research within the framework had started many years before (see, e.g., [6, 7]). The main idea behind the term is that there is a need for decision aids which allow the decision maker to express his/her evaluations and preferences in a verbal form, and this verbal form should not be transformed into a quantitative form in any arbitrary way.

In the area of MCDA, the decision maker is usually the central person in the decision process and tries to maximize utility (under uncertainty) or value function that depends on the criteria and/or attributes. Verbal Decision Analysis acknowledges that known constraints of human information processing system as well as the psychological validity of input data in decision analysis should be taken account while designing decision aiding methods. VDA is oriented on the construction and application of such methods.

2.1. Use Language for Problem Description That Is Natural to the Decision Maker

This requirement means that if the decision maker takes into account qualitative characteristics of the alternatives (e.g., “car in a good condition”) or uses generalized notions while conducting analysis (e.g., “good credit,” “high risk,” etc.), the decision aids should use them for alternatives’ evaluations. Here comes the notion of “verbal scales.”

2.2. Implement Psychologically Valid Measurements and Preference Elicitation Procedures

Larichev [8] carried out the initial analysis of the literature to differentiate between the elicitation of different types of evaluations from the decision maker. Some types of information lacked corresponding research. Additional types of admissible information elicitation types were introduced later (see, e.g., [9, 10]).

In aggregate, there are four major admissible groups of preferential information elicitation:

(i)rank ordering of criteria importance,(ii)qualitative comparison of attribute values against one, two, or three criteria,(iii)qualitative evaluation of probabilities,(iv)direct classification of an alternative.

2.3. Incorporate Procedures for Consistency Check of Decision Maker’s Preferences

The requirement means that all or part of the elicited information should provide auxiliary preferences which may be used to check for consistency of preferences as well as for verification of the underlying axioms. For example, many MAUT methods assume preferential independence of criteria and/or transitivity of preferences.

2.4. Procedures Should Be “Transparent” to the Decision Maker and Provide Explanations of the Results

The last requirement assumes that the information elicited from the decision maker will be used in an easily understandable way to provide the solution, and as such the result may be easily explained to the decision maker.

Within this framework, the majority of the VDA methods are based on the rules of dominance (Pareto Principle) as a result of ordinal scales and transitivity of preferences. If we try to compare VDA to the existing major groups of approaches to multiple criteria decision analysis [11], we would be able to find mutual features and crucial differences with the majority of them.

Like outranking methods [4], VDA provides outranking relationships among multiple criteria alternatives which leads in the majority of cases to their partial order. At the same time, VDA is designed to elicit a sound preference relationship that can be applied to future cases while outranking methods are intended to compare a given set of alternatives.

On another hand, VDA is based on the same principles as multiattribute utility theory (MAUT) but is oriented on using the verbal form of preference elicitation and on evaluation of alternative decisions without resorting to numbers. As MAUT, classical methods of VDA work in the criterion space to provide universal ordering rules for any set of real alternatives.

As VDA, the Analytical Hierarchical Process (AHP) [5] elicits preference information from the decision makers in an ordinal form but turns these verbal ordinal evaluations into numerical ones for further analysis. As outranking methods, AHP works only with real alternatives, and preferences obtained in the process have to be revised in the case of additional alternatives. Figure 1 shows VDA place among the popular groups of MCDA methods.

(a) Comparison of approaches based on the resulting order and applicability of derived rules 

(b) Comparison of approaches based on the use of heuristics and quantitative versus qualitative form of preference elicitation

Figure 1: VDA among other approaches to MCDA.

3. Classical Methods of VDA

In 1997, three methods were introduced as a VDA toolkit for three major types of decision problems. Method ZAPROS based on [7] is used for rank ordering alternatives, ORCLASS based on [6] is used for an ordinal classification of alternatives, and method PARK based on [12] is used for the selection of the best alternative.

All methods assume that discrete alternatives are evaluated against a set of criteria with finite number of possible verbal values   where  .

In this case, we can form a set of all possible vectors in the space of criteria of the size .

There is also a subset of vectors, describing real alternatives. All methods elicit order of criterion values within each scale thus providing ordinal scales for each criterion.

3.1. Method ZAPROS

Method ZAPROS is intended for problems in which there is a need to rank order a rather large number of alternatives and the set of the alternatives may change while decision rules stay in place. An example of such a problem will be the distribution of limited resources among research projects submitted to a government agency [1]. If there is a system for projects’ evaluation and comparison (ranking), it is possible to apply this system for any set of projects.

Method ZAPROS elicits the decision maker’s preferences through pairwise comparisons of hypothetical alternatives from the set near the so-called reference points—all the best or all the worst possible values. These alternatives differ against only two criteria, and the decision maker has to state preference for one on them or indifference between them. An example of the question to the decision maker in the case of job applicants’ evaluation would be as follows:

“Would you prefer a candidate who is above average in research but average in teaching or a candidate who is average in research but above average in teaching if all the other characteristics for them are at their best level?”

Elicited information is checked for consistency through transitivity of preferences and comparing the same criterion values at two reference points (for criterion independency check). As a result of the complete set of these comparisons, the so-called Joint Ordinal Scale (JOS) in the criterion space is formed. It rank orders different criterion values and is not connected to a specific set of alternatives.

JOS allows partial pairwise comparison of real alternatives (see Figure 2). Part (a) shows comparison of two alternatives based on the dominance due to ordinal criterion scales. Part (b) shows dominance relations between different criterion values due to the Joint Ordinal Scale. These comparisons form the basis for partial rank ordering of real alternatives. For more on the procedure, see [1].

(a) Comparison on the basis of dominance 

(b) Comparison on the basis of JOS

Figure 2: Comparison of alternatives and in ZAPROS.

3.2. Method ORCLASS

Method ORCLASS is used when there is a need to define appropriate class for each real alternative out of classes . The classes are ordinal in the sense that all alternatives in class are preferred to class , and so on. One example ORCLASS application may be classifying submitted papers to the journal where classes are “accept,” “accept with minor changes,” “rewrite and resubmit,” and “reject”. Another one is connected to loan application where each application has to be classified as “good,” “acceptable,” and “poor.”

As ZAPROS, ORCLASS aims at constructing the classification rule in the criterion space and then applying it to any submitted alternative(s). The decision maker is presented with hypothetical alternatives (vectors from ) and classifies them. Due to ordinal scales all vectors from are partially ordered. Figure 3 shows this partial order for the case of three criteria with two estimates for each. One represents the best possible value, while two represents the least preferable value.

Figure 3: Partial order among vectors from based on ordinal scales.

Based on the dominance principle, possible classes for other alternatives from are reduced based on the ordinal nature of classes—dominating alternatives should belong to this or better class, while dominated alternatives should belong to this or less preferable class. For example, let us assume that the task is to distribute vectors in Figure 3 among three ordinal classes , , and where is the most preferable one. If vector (2,1,2) is presented to the decision maker and assigned class , then the possible classes of all vectors dominating it, (1,1,1), (1,1,2), and (2,1,1), are reduced to classes and . Vector (2,2,2) dominated, but (2,1,2) may belong to class or . If presented vector is assigned class , then classification for all dominating vectors will be reduced to one possible class of . Thus, it is not necessary to present these vectors for classification by the decision maker.

The process stops when all vectors from are classified.

The classification process allows for consistency check. If the decision maker assigns class outside of the ones possible for this vector (based on previous classifications), there is a contradiction which may be explained to the decision maker and resolved through some reclassification.

Different heuristics are used to find vectors from whose classifications will maximize the number of indirectly classified ones. The approach proposed in classical ORCLASS is based on calculation of potential “informativeness” of unclassified vectors from . It favors those which define an approximately equal number of vectors by indirect classification in case of any admissible class. For more on the procedure, see [1]. The result of the process is a complete classification of all vectors from .

For easier classification of real alternatives once the classification is built, the notion of class boundaries is introduced. Lower border for each class is presented by least preferable vectors from the class on the basis of dominance, while upper border is formed by the most preferable vectors of the class. These two borders accurately represent each class and may be used to define if another vector belongs or does not belong to the class.

3.3. Method PACOM

Method PACOM, called in Russian publications as PARK and in earlier version as ASTRIDA [12], is oriented on the selection of the best alternative out of their small number. This is the only method out of the three where the process is oriented on the pairwise comparison of the real alternatives based on the compensation principle—disadvantages of one alternative are counterbalanced by disadvantages of the other alternatives.

Let us assume that we have two real alternatives: and . Alternative has better values for the first criteria, while alternative has better values against all other criteria. Comparison of these two alternatives is based on eliciting information about the relative preference for different alternatives’ values and is carried out as the comparison of basic hypothetical alternatives, differing in values upon two criteria or more criteria, but only when one disadvantage of alternative may be “compensated” by one or more advantage of the alternative (see Figure 4).

(a) Real alternatives 

(b) Comparison of hypothetical vectors

(c) Next comparison if is preferred to

Figure 4: Compensation procedures in PACOM.

Information about comparison of basic alternatives may be used to compare real alternatives. It is proposed to form hypothetical alternatives for comparison in an iterative goal-seeking mode, quickly establishing the principal possibility for comparison of two real alternatives.

If alternatives and are left incomparable, it is concluded that it is not possible to compare these alternatives using current set of criteria. The two alternatives are left incomparable, but additional information about the decision maker’s preferences concerning comparison of these alternatives is obtained. The decision maker is asked to mark the minimal changes in values of one of these two alternatives in a way to be able to say that one alternative is more preferable or equal to the other. Alternative obtained in such a way is called an “adjusted alternative” and is stored in a special list of adjusted alternatives.

When all pairs of real alternatives from the initial list are analyzed using the proposed procedures, the general analysis of the results is carried out. As the aim of the analysis is to select the best alternative if two alternatives are compared, the least preferable one is excluded from the list and is not used in further analysis (as it cannot be a candidate for the final choice). The more preferable alternative is then compared with the next alternative, and so on. If there is only one alternative left in the initial list, the problem is solved. If there are incomparable alternatives left in the list, it is concluded that likely there is no satisfactory alternative among those in the initial list to make the real choice. The decision maker is proposed to analyze the list of adjusted alternatives that has been formed in the process of the analysis (evaluating the possibility to obtain a real alternative with such characteristics).

In this part, we described the “classical methods” of VDA. All presented methods are theoretically sound: all approaches assume additive value function though they do not seek it explicitly, elicit preference information in a qualitative form, provide opportunities to check this information for consistency, and apply only easily explainable rules of dominance and transitivity to reach the solution. Proposed methods do not guarantee complete rank ordering of alternatives or selection of the best one. Those working within the VDA framework believe that if it is not possible to select the best alternative based on the qualitative analysis, the solution should not be forced by assigning arbitrary numbers to different evaluations. The VDA approach recommends in this case that the redesign of the problem is necessary through new criteria, modified scales, and/or modified set of alternatives.

4. New Methods within the VDA Framework

During the last decades, quite a number of new approaches within the field appeared, and modifications of the “classical methods” occurred. Figure 5 illustrates the connection between the older and the newer VDA methods. The list includes all the methods the authors were able to come across up to date to the best of their ability. It is likely that they were not able to find all of them due to the wide spread of the resources available now for publication and due to the fact that many publications in this area are mostly in Russian and to a lesser extent in Latin American sources. The search was purposefully limited to publications available in English.

Figure 5: Verbal Decision Analysis methods.

4.1. ZAPROS Based Methods

ZAPROS III [13] as well as STEP-ZAPROS [14] increases Joint Ordinal Scale (JOS) to a Joint Scale for Quality Variations (JSQV) to allow comparison of not only individual values against different criteria (as in Figure 2) but pairs of criterion values as well (see Figure 6). The advancement is achieved through additional questions to the decision maker on the pairwise comparison of hypothetical alternatives from differing in values against two criteria.

Figure 6: Comparison of alternatives and on the basis of JSQV.

In ZAPROS III or STEP-ZAPROS, it is possible to ask the decision maker the questions as follows:

“Would you prefer a candidate who is above average in research but below average in teaching or a candidate who is average in research but average in teaching if all the other characteristics for them are at their best level?”

The main difference between ZAPROS III and STEP-ZAPROS is in the completeness of the process. In ZAPROS III, all the above mentioned hypothetical alternatives should be compared. After that the JSVQ is formed and applied to comparison of real alternatives. As the number of comparisons is large, this approach is mostly appropriate for a relatively small number of criteria and small number of possible criterion values.

STEP-ZAPROS uses additional comparisons only after the JOS is used to compare real alternatives. The method proposes an iterative procedure of identifying a minimal set of comparisons necessary to compare a small number of real alternatives left incomparable on the basis of JOS.

UniCombos [15] is a computerized system which is based on the ideas of ZAPROS, but it has three major differences:

(a)the approach assumes that we need to rank order only a small number of real alternatives;(b)a decision maker can consistently compare alternatives’ values against more than two criteria (in general, they try to limit it to three);(c)the ability of the decision maker to compare complex combinations of alternatives’ values is helped by special ways of visualization of those values.

As STEP-ZAPROS, UniCombos assumes three steps in the process. The first step, as in all VDA methods, is connected with comparison of alternatives’ values on one criterion, resulting in ordinal scales and the dominance rule for pairwise comparison. If the rank ordering is not achieved, the second step is to compare all alternatives’ values against two criteria (the so-called “dyads”). It is the same as constructing JSQV in ZAPROS III, but the comparisons are limited to combinations, present among the real alternatives and necessary to compare real alternatives.

As an interactive system, the UniCombos checks the comparability of the real alternatives after each additional piece of preferential information is obtained. It will stop as soon as all alternatives are compared and/or the best alternative is found. Once all possible pairs of alternatives’ values are compared, but there are still incomparable alternatives in the set, the system will present the decision maker with the so-called “tryads” of alternatives’ values—values different against three criteria (analogous to example in Figure 4(c)).

4.2. ORCLASS Based Methods

Main improvements in ORCLASS approach were oriented on making the process of information elicitation more efficient. As the decision maker has to classify many hypothetical alternatives from , any procedures that can minimize their number is an improvement to the process. Methods CYCLE [16] and DIFCLASS [17] differ from ORCLASS only in how they find hypothetical alternatives from to be presented to the decision maker for classification.

DIFCLASS is applicable only in cases with two decision classes where indirect classification of alternatives is quick and always consistent. CYCLE proposes the construction of “chains” of vectors between vectors and from which are known to belong to different classes. The “chain” is constructed sequentially by changing one criterion value in vector by one level until we obtain criterion values of vector . Then, the most “informative” vector is searched only in the chain, thus essentially lowering the computational complexity of the algorithm. The process is dynamic and searches for the “longest” chain between two vectors. The effectiveness of the approach was compared to the algorithms of monotone function decoding and appeared much more effective for smaller problems and simpler borders while being somewhat less effective in more complex cases.

Subset of alternatives classification (SAC) [18] and classification of real alternatives (CLARA) [19]. Methods are designed for problems with a relatively small number of alternatives needed to be classified only once (not to construct a classification rule in the criterion space). In this case, a modified approach may be used to decrease the number of vectors the decision maker has to classify.

In the SAC method, the principle of evaluating “informativeness” of vectors from is the same as in ORCLASS, but only the indirectly classified real alternatives are taken into account (not all alternatives from the set ). This makes the process less complex. Another difference is that in SAC you can evaluate “relative informativeness” of a vector using variance in the number of indirectly classified alternatives. The variance is dependent on the number of classes and the number of criteria. The recommended values are between 2.2 and 3.5. If is equal to zero, the “informativeness” is calculated the same as in ORCLASS.

Method CLARA is also oriented on classification of real alternatives, but selection of alternatives to be presented to the decision maker for classification is based on method CYCLE. Again only real alternatives are taken into account when constructing and analyzing chains.

4.3. Method SNOD Based on PACOM

As PACOM, scale of normalized and ordinal differences (SNOD) [20] method is oriented on the selection of the best alternative out of their small number. The goal of SNOD is to make the process proposed in PACOM more efficient. This is achieved by some computerized preliminary analysis of potential quality of real alternatives.

First, normalized scales are used to “quantify” criterion values. Then, these values are used for pairwise comparison of real alternatives to define the “potentially best alternative.” All alternatives dominated by this one are eliminated from the analysis. All other alternatives are ordered in accordance to the total scoring difference between the “potentially best alternative” and this one.

This order is used in the dialog with the decision maker in assumption that it will lead to the solution quicker than in the traditional way of PACOM. Though use of quantified scales for alternatives evaluation and/or comparison is outside the framework of VDA, this method does not influence the outcome of the decision process, only its efficiency. At no time, any final decisions are made on the basis of the preliminary analysis. If the best alternative is not found quickly, previously discarded alternatives will be returned into the process, and “normalized and ordinal differences” may be revised.

4.4. Trends in VDA Methods

The analysis of new VDA methods shows several trends in their modifications:

(i)tendency to concentrate on the comparison/classification of real alternatives as opposed to the idea of working in the criterion space for universal rules;(ii)implementation of iterative procedures for preferential information elicitation with the goal of solving just the problem at hand; (iii)gradual increase in the complexity of information elicited from the decision maker and minimization of verification processes.

Figure 7 summarizes these findings. At the same time main principles of VDA are maintained in all modifications: qualitative information elicitation, verification of the information for consistency, and explainable rules of using the information for alternatives’ evaluation and comparisons.

Figure 7: Trends in VDA methods.

5. VDA Applications

VDA has positive features of using psychologically valid preference input, providing checks for input consistency, and implementing mathematically sound rules. VDA was used in a number of applications starting with earlier applications of ZAPROS in R&D planning [1], ORCLASS for job applicants’ selection [21], and PACOM for pipeline selection [22–24].

Tamanini et al. [25] applied ZAPROS III to rank order tools in Alzheimer’s disease diagnosis. In this work, preferences were obtained through questionnaires from experts and postmortem patient diagnosis. The study enabled identification of tests that would more quickly detect patients with Alzheimer’s disease. Ustinovichius et al. [26] used UniCombos to compare construction contracts using seven criteria for three real alternatives. ZAPROS was used in rank ordering real retailer commercialization decisions in Brazil based on discussions and analysis with key managers [27]. Mendes et al. [28] demonstrated use of VDA in the design of mobile television application, applying ZAPROS to the characteristics of prototypes based on user experience and intentions.

The ordinal classification approach was used for R&D planning and journals’ evaluation, as well as for job selection [1, 29]. Yevseyeva et al. [30] applied a SAC like method for neuropsychology patient diagnosis. CLARA was used in several applications concerned with the decision making in the area of construction [19, 31]. Ordinal classification approach was successfully applied in modifying tasks with many criteria into subtasks of smaller size. In criterion hierarchy, scales for higher level criteria presented ordinal decision classes for the combination of lower level criteria. This approach was successfully used in evaluating investment risk in construction projects [31]. They had six final decision classes, with six criteria at the highest level with 3-4 criterion values. Each of these criteria was a combination of 3-4 criteria at the lower level. The problem of this size can be solved only by a hierarchy. Gomes et al. [32] applied ORCLASS to marketing decisions for a small Brazilian business engaged in the distribution of dental products.

6. Conclusion

MCDA is an applied science. The primary goal of research in MCDA is to develop tools to help people make more reasonable decisions. In many cases, the development of such tools requires combination of knowledge derived from such areas as applied mathematics, cognitive psychology, and organizational behavior. Verbal Decision Analysis is an example of such a combination. It is based on valid mathematical principles, takes into account peculiarities of human information processing system, and places the decision process within the organizational environment of the decision making.

This paper has reviewed the basics of Verbal Decision Analysis. They were demonstrated with early VDA methods, such as ZAPROS and ORCLASS, and their later modifications, such as ZAPROS III, UniCombos, CLARA, and others.

The clear differences between “classical” VDA and more recent modification of the methods analyzed in this paper is the tendency to concentrate on problems with relatively small number of real alternatives and to carry out goal oriented solution process rather than constructing possible decision rules in the criterion space. The observed changes may be a reflection of the goal of MCDA researchers to create methods which are easier to apply in the real world. The majority of real life situations include a relatively small number of unique alternatives. The decision situation will change with their different set and as a result will require reelicitation of relevant preference information. Iterative processes based on real alternatives decrease the time and effort the decision maker has to invest into the process and produces quicker results.

There is an active research in further development of VDA methods, both in Russia, the home of VDA, and in the Americas. A number of published applications to real decision problems were discussed here, demonstrating the maturity of VDA. Next important step in the evolution of VDA will be the production of easily available computer systems. Currently, only proprietary research version of such systems exists in different parts of the world.

References

O. I. Larichev and H. M. Moshkovich, Verbal Decision Analysis for Unstructured Problems, Kluwer Academic Press, Boston, Mass, USA, 1997.

H. M. Moshkovich, A. I. Mechitov, and D. L. Olson, “Verbal Decision Analysis,” in State of the Art of Multiple Criteria Decision Analysis, J. Figueira, S. Greco, and M. Ehrgott, Eds., International Series in Operations Research and Management Science, pp. 609–637, Springer, New York, NY, USA, 2005.

R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs, John Wiley & Sons, New York, NY, USA, 1976.

B. Roy, Multicriteria Methodology for Decision Aiding, Kluwer Academic Publisher, Dordchert, The Netherlands, 1996.

T. L. Saaty, The Analytic Hierarchy Process, McGraw-Hill, New York, NY, USA, 1980.

O. I. Larichev and H. M. Moshkovich, “An approach to ordinal classification problems,” International Transactions in Operational Research, vol. 1, no. 3, pp. 375–385, 1994.

O. I. Larichev and H. M. Moshkovich, “ZAPROS-LM—a method and system for ordering multiattribute alternatives,” European Journal of Operational Research, vol. 82, no. 3, pp. 503–521, 1995.

O. I. Larichev, “Cognitive validity in design of decision-aiding techniques,” Journal of Multi-Criteria Decision Analysis, vol. 1, no. 3, pp. 127–138, 1992.

E. M. Furems, O. I. Larichev, G. V. Roizenson, A. V. Lotov, and K. Miettinen, “Human Behavior in a Multi-Criteria Choice Problem with Individual Tasks of Different Difficulties,” International Journal of Information Technology & Decision Making, vol. 2, no. 1, pp. 29–40, 2003.

O. I. Larichev, D. L. Olson, H. M. Moshkovich, and A. J. Mechitov, “Numerical vs cardinal measurements in multiattribute decision making: how exact is enough?” Organizational Behavior and Human Decision Processes, vol. 64, no. 1, pp. 9–21, 1995.

J. Wallenius, J. S. Dyer, P. C. Fishburn, R. E. Steuer, S. Zionts, and K. Deb, “Multiple criteria decision making, multiattribute utility theory: recent accomplishments and what lies ahead,” Management Science, vol. 54, no. 7, pp. 1336–1349, 2008.

D. Berkeley, P. Humphreys, O. I. Larichev, and H. M. Moshkovich, “Modeling and supporting the process of choice between alternatives: the focus of ASTRIDA,” in Environments for Supporting Decision Processes, H. G. Sol J Vecsenyi, Ed., pp. 59–62, North Holland, Amsterdam, The Netherlands, 1990.

O. I. Larichev, “Ranking multicriteria alternatives: the method ZAPROS III,” European Journal of Operational Research, vol. 131, no. 3, pp. 550–558, 2001.

H. M. Moshkovich, A. I. Mechitov, and D. L. Olson, “Ordinal judgments in multiattribute decision analysis,” European Journal of Operational Research, vol. 137, no. 3, pp. 625–641, 2002.

I. Ashikmin and E. Furems, “UniCombos: intelligent decision support system for multi-criteria comparison and choice,” Journal of Multi-Criteria Decision Analysis, vol. 13, no. 2-3, pp. 147–157, 2005.

O. Larichev, A. Asanov, and Y. Naryzhny, “Effectiveness evaluation of expert classification methods,” European Journal of Operational Research, vol. 138, no. 2, pp. 260–273, 2002.

O. Larichev and A. Bolotov, “The DIFKLASS system: construction of complete and noncontradictory expert knowledge bases in problems of differential classification,” Automatic Documentation and Mathematical Linguistics, vol. 30, no. 5, pp. 12–17, 1996.

O. I. Larichev, A. V. Kortnev, and D. Y. Kochin, “Decision support system for classification of a finite set of multicriteria alternatives,” Decision Support Systems, vol. 33, no. 1, pp. 13–21, 2002.

L. Ustinovichius, V. Kutut, D. Kochin, and G. Shevchenko, “Verbal analysis of renovation investment strategy of old town,” in Proceedings of the 25th International Symposium on Automation and Robotics in Construction (ISARC '08), E. K. Zavadskas, A. Kaklauskas, and M. J. Skibniewski, Eds., pp. 397–411, 2008.

l. Ustnovichius and D. Kochin, “Verbal decision analysis methods for determining the efficiency of investment in construction,” Foundations of Civil and Environmental Engineering, vol. 5, pp. 35–46, 2004.

H. M. Moshkovich, A. I. Mechitov, and D. L. Olson, “Rule induction in data mining: effect of ordinal scales,” Expert Systems with Applications, vol. 22, no. 4, pp. 303–311, 2002.

N. E. Flanders, R. V. Brown, Y. Andre'eva, and O. Larichev, “Justifying public decisions in arctic oil and gas development: American and Russian approaches,” Arctic, vol. 51, no. 3, pp. 262–279, 1998.

O. Larichev and R. Brown, “Numerical and verbal decision analysis: comparison on practical cases,” Journal of Multi-Criteria Decision Analysis, vol. 9, pp. 263–273, 2000.

O. I. Larichev, R. Brown, E. Andre'eva, and N. Flanders, “Categorical decision analysis for environmental management: a Siberian gas distributing case,” in Contribution To Decision Making, J. P. Caverni, M. Bar-Hillel, F. H. Barron, and H. Jungermann, Eds., pp. 255–286, North Holland, Amsterdam, The Netherlands.

I. Tamanini, A. K. de Castro, P. R. Pinheiro, and M. C. D. Pinheiro, “Applied neuroimaging to the diagnosis of alzheimer's disease: a multicriteria model,” Communications in Computer and Information Science, vol. 49, pp. 532–541, 2009.

L. Ustinovichius, A. Barvidas, A. Vishnevskaja, and I. V. Ashikhmin, “Multicriteria verbal analysis for the decision of construction problems,” Technological and Economic Development of Economy, vol. 15, no. 2, pp. 326–340, 2009.

G. Rodrigues and E. Cohen, “Optimizing marketing segmentation through an information system using verbal decision analysis in the Chamber of Commerce in Rio de Janeiro (CDLRIO),” in Proceedings of the Marketing & Management Development Conference, vol. 3, no. 1, pp. 432–436, Academy of World Business, 2008.

M. S. Mendes, A. L. Carvalho, E. Furtado, and P. R. Pinheiro, “Towards analyzing alternatives of interaction design based on verbal decision analysis of user experience,” International Journal of Interactive Mobile Technologies, vol. 4, no. 2, pp. 17–231, 2010.

A. I. Mechitov, H. M. Moshkovich, and D. L. Olson, “Problems of decision rule elicitation in a classification task,” Decision Support Systems, vol. 12, no. 2, pp. 115–126, 1994.

I. Yevseyeva, K. Miettinen, and P. Räsänen, “Verbal ordinal classification with multicriteria decision aiding,” European Journal of Operational Research, vol. 185, no. 3, pp. 964–983, 2008.

G. Shevchenko, L. Ustinovichius, and A. Andruškevičius, “Multi-attribute analysis of investments risk alternatives in construction,” Technological and Economic Development of Economy, vol. 14, no. 3, pp. 428–443, 2008.

L. F. A. M. Gomes, H. Moshkovich, and A. Torres, “Marketing decisions in small businesses: how verbal decision analysis can help,” International Journal of Management and Decision Making, vol. 11, no. 1, pp. 19–36, 2010.