Supplier Evaluation Process by Pairwise Comparisons

We propose to assess suppliers by using consistency-driven pairwise comparisons for tangible and intangible criteria. The tangible criteria are simpler to compare (e.g., the price of a service is lower than that of another service with identical characteristics). Intangible criteria are more difficult to assess. The proposed model combines assessments of both types of criteria. The main contribution of this paper is the presentation of an extension framework for the selection of suppliers in a procurement process.The final weights are computed from relative pairwise comparisons. For the needs of the paper, surveys were conducted among Polish managers dealing with cooperation with suppliers in their enterprises.The Polish practice and restricted bidding are discussed, too.


Introduction
The success of an enterprise does not depend only on all the cooperating subjects [1].Efficiency of a given subject as a whole is not its basic source of competitive advantage.It is the efficiency of various types of activities that the enterprise undertakes when delivering its product to the market [2].These actions create a supply chain.The main objective of the supply chain is to provide the maximum value to customers at low costs and high speed [3,4].
An important issue of a supply chain management is the procurement [5].The cost of products and services acquired from external suppliers is significant for most manufacturing firms [6].On average, manufacturers purchases of goods and services amount to 55% of revenues and it is in contrast to labor costs of 6% and overhead expenses of 3% of revenues [7].For high-technology firms, purchased material and services represent up to 80% of total product costs [8,9].
The objective of the procurement process must be the harmonization of internal processes of buyers and suppliers in order to avoid a waste of resources within the supply chain.It may be achieved when a great amount of emphasis is put on establishing and maintaining good relations with suppliers.Optimization in the area of delivery may bring a chance of sizable savings.It is noteworthy that the scale of a company's activity may result in greater losses caused by incorrect purchasing processes.
Reliable suppliers enable manufacturers to reduce inventory level and improve product quality, which is the main reason why concerns about appropriate suppliers are increasing.One of the prime responsibilities of the purchasing function is the evaluation and selection of suppliers.Some researchers even indicate that evaluation and selection of suppliers are critical due to their contribution to supply chain performance [10].
Enterprises build a base of suppliers with whom they cooperate more or less closely for many years.When searching for solutions related to the choice of a new supplier or evaluation of an existing one, people responsible for such decisions take into account a range of criteria.These criteria are very often worked out on the basis of the company's experience.
Carefully selected, competitive suppliers can go a long way in minimizing adverse impacts and, in fact, in enhancing positive impacts on the quality of the output of an organization.The importance of an adequate framework for the selection of suppliers for an organization has been stressed in the literature [11,12].
The supplier selection process requires predetermination of several issues, including the number of suppliers with which the organization wishes to contract for a given product or service [13].It is difficult to determine how many suppliers should work for a given organization.Some authors argue that there is now a trend to reduce the number of suppliers to a manageable level [14,15], even to a single source in extreme cases.
No clear recommendation regarding the number of suppliers is confirmed by the results of the research.Every third company does not follow any general rule.This is understandable.On the one hand, a large number of suppliers ensures lower prices for customers, offers greater safety, and decreases the risk of stopping production.On the other hand, it raises the operating costs of such cooperation (maintenance costs of information systems, controlling, sourcing, negotiation, setting conditions for the cooperation, audits, etc.) [16,17].
In the process of partner selection, several methods are used.Among them, categorical method, weighted-point method, vendor performance matrix approach, vendor profile analysis (VPA), multicriteria decision aid (MCDA), multiple objective programming (MOP) such as goal programming, data envelopment analysis (DEA), and multiattribute utility theory (MAUT).The recent studies about the use of fuzzy theories and their development in the supplier selection problem are particularly interesting [18][19][20].In [21] the authors proposed a new tool for supplier selection based on a combination of the grey system and the uncertainty theory neither of which requires any probability distribution or fuzzy membership function.
The structure of the paper is as follows: the characteristic of bidding process is discussed in Section 2.Then, Section 3 describes supplier evaluation problems and Section 4 presents pairwise comparisons method and review of the other applications and the literature on the pairwise comparisons.Section 5 provides example of a tendering model proposed by the authors.Next, Section 6 contains consistency analysis of tendering model.Section 7 proposes an example of suppliers assessment.The conclusions of research are outlined in Section 8.

Bidding Process
The previously mentioned methods have some limitations because the supplier selection process is mostly based on intuition.There is no theoretical base or consistent method of predicting the best bid.It is not uncommon for the evaluation panel to arrive at a deadlock when a part of the panel favors one solution, because of certain criteria, while the other part insists on another solution since it scores better on different criteria.The decision making process nearly always involves some kind of constituency in modern democratic societies.We have various boards of governors or directors, committees, task groups, city councils, panels of experts, and individuals, each with a specific agenda.Heated discussions and various ways of dispute and argumentation often take place to arrive at certain decisions.
Most constituencies have worked out precise and practical policies for running meetings in an orderly and effective manner.What we lack, however, is a device for drawing solid consistent conclusions and all too often the loudest individual wins!Unfortunately, loudness does not necessarily go along with wisdom.Casual thinking is not efficient in predicting complex outcomes.
The main goal of tendering is the selection of the most suitable supplier from the company point of view.Through a bidding we try to achieve the following: (i) the setting of common input constraints for potential suppliers, (ii) the selection of the best supplier based on tangible and intangible but constant (during the entire bidding process) criteria, which allow us to compare the proposed offers, (iii) a minimization of the influence of informal interests on selection of an offer thanks to the application of a strict selection process.
One of the most instrumental conditions of a fair bidding is the necessity of a precise scoring system of all criteria and their preferences in the tendering documents to be used during the selection process by the selection panel.
A selection panel is obligated to use objective and measurable evaluation criteria during the selection process.The goal of such an action is to have more comparability and fewer intangible or arbitrary decisions.Application of such quantification in the request for proposals (RFPs) to specific bidding components, together with the bidding amount, helps in the selection of the best offer.A recommended practice is the assignment of weights (e.g., factors which are positive points) to each aspect of an offer with respect to the criteria, listed in the bidding documents.As a result, the best offer is the one with the highest total score.
In restricted bidding (as opposed to open bidding), only invited parties may submit a bid.In the first stage of a two-stage bidding process suppliers submit preliminary bids without the total amount.However, the selection panel may request precise specification of some parameters.Successful bidders from the first stage are invited to submit a complete bid for the second stage.Two-stage bidding may be considered a special case of an open bidding.The main differences are as follows: (i) the possibility of submitting two tendering offers by bidders, (ii) the use of two selections instead of one; the first selection to prequalify tenders and the second to find the true winner, (iii) the possibility (and sometimes necessity) of negotiations with bidders, (iv) the possibility of using the negotiation results, to change or restrain crucial constraints of the request for proposals, before they are distributed again to bidders selected during the first stage of bidding.
The two-stage bidding process is recommended in situations: (i) where it is difficult to predict certain parameters such as technical, quality, service, or construction tasks, (ii) where necessity demands negotiating with suppliers or constructors because of the specific character of the supplies and/or construction services, (iii) when the tendering is related to research, assessment, or any other specialized service.Standardization should not only be perceived as a "nuts and bolts" or physical concept (without which new products and spare parts would be far more expensive), but also as an intellectual concept.In the case of "nuts and bolts," it is a concept of the standard thread pitch.Today, we use standards to such a degree that we often forget about other alternatives.

Is
The problem is that we often assume that we have a measure for nearly everything!No one questions the practicality of the measure of length (meter or foot), or weight (kg or pound), as they are used in everyday life.There are, however, situations where it is more useful and natural to use pairwise comparisons.Consider this example of the reliability of two suppliers  and .The reliability of supplier  is evaluated as 4 on the scale of 0 to 5 and  is 2 on the same scale.
There is no standardized unit of reliability; hence, it is easier to say that  is two times more reliable than  and it is what we call "pairwise comparison."Interestingly the use of a standardized measure unit such as a meter stick is also a pairwise comparison.The statement "the length of  is 4 m" is an abbreviation of "by a pairwise comparison of  to one meter we have a factor of 4." Using pairwise comparisons (see, e.g., [22][23][24]) is natural and it does not need to be imprecise.We have become so accustomed to having standards that sometimes we find it difficult to imagine a situation where no standard measures exist.The truth is that there are many such situations.
Measurement of the environment or environmental pollution are good examples of situations where a standard yardstick seems to be missing.For example, it would be hard to use a cubic meter of environment as a standard measure since in one cubic meter of environment there could be millions of ants but only a fraction of an elephant.How could one decide if a fraction of an elephant is less significant than a colony of ants?

Pairwise Comparisons
Casual thinking is partial and fragmentary and is not an effective way to measure intangibles.In the decision making process, many factors must be considered simultaneously and with about the same degree of importance.As such, an approach with more finesse is necessary to obtain a clear and unambiguous conclusion.It has been shown by numerous examples that the pairwise comparisons method can be used to draw final conclusions in a comparatively easy and elegant way.The brilliance of the pairwise comparisons could be reduced to a rule of common sense: consider two objects at a time if you are unable to handle more than that.Llull noticed it in 13th century.
The practical and theoretical virtue of the pairwise comparisons methodology is its simplicity.The goal of pairwise comparisons is to establish the relative preference of two criteria in situations in which it is impractical (or sometimes meaningless) to provide the absolute estimations of the criteria.To this end, an expert (or a team of experts) provides relative comparison coefficients   > 0, which are meant to be a substitute for the quotients   /  of the unknown (or even undefined) absolute values of criteria   ,   > 0. The quotients   /  are also sometimes called relative weights in the literature: where   expresses an expert's relative preference of criteria   , over   .
A pairwise comparisons matrix A is called reciprocal if   = 1/  for every ,  = 1, . . .,  (then automatically   = 1 for every  = 1, . . ., ).Even a stronger condition seems natural.A pairwise comparisons matrix A is called consistent if   =   ⋅   for every , ,  = 1, . . ., .While every consistent matrix is reciprocal, the inverse in general fails.Consistent matrices correspond to the ideal situation, in which there are exact values  1 , . . .,   for criteria.The quotients   =   /  form a consistent matrix.Conversely, the starting point of the pairwise comparisons inference theory states that, for every  ×  consistent matrix A = (  ), there exists positive real numbers  1 , . . .  such that   =   /  for every ,  = 1, . . ., .Such vector  = ( 1 , . . .  ) is unique up to a multiplicative constant.
How can we establish fair weights?Is there a theory to help us?The weighting classification needs to be done on a fair basis for every criterion, which ought to have its share in contributing to the overall judgment.A fair solution is to compare all criteria in pairs using, for example, a small scale from 1 to 3 (as mathematically reasoned, on the basis of the Fulop's constant, in [25]), presented in Table 1.The solution accuracy of not-so-inconsistent matrices depends on the inconsistency (see [26] for details).Smaller scale contributes to decreased inconsistency.One may note that the consistency-driven approach is, in brief, the next step forward in the development of pairwise comparisons.By concentrating on the consistency as the means to improve the precision of judgments it picks up where Analytic Hierarchy Process arrived to.AHP inconsistency index detected the existence of the inconsistency by the eigenvalue-based heuristic and its rather questionable threshold of 10% randomly generated matrices.In [27], a mathematical reasoning (based on two counter examples) was provided to reject the eigenvaluebased inconsistency.
It is not our goal to present the consistency-driven approach here, but only its application to evaluation of proposals.The theoretical foundations of this method are based on [28][29][30], while [31] presents convincing statistical evidence that the pairwise comparisons are contributing to the improvement of accuracy.The pairwise comparisons methodology was introduced by Thurstone, in 1927 (see [30]).Its further extension by the consistency-driven analysis (see the next section as well as [28]) can be employed as a powerful theoretical framework for the evaluation of tenders.
One of the notable pairwise comparisons past applications of national importance is the evaluation of site proposals for nuclear power plants in Holland (rejected by Dutch government, for details, see [22]).The Monte Carlo study presented in [26] discounts a claim of superiority of any particular method of solving the pairwise comparisons matrix.In fact, the accuracy of solutions strongly depend on the consistency of judgments making the consistency-driven approach suitable for applications.
A new definition of inconsistency introduced in [28] is instrumental for the inconsistency analysis.This definition allows us to locate the most inconsistent judgments and is instrumental for extending the hierarchical model of pairwise comparisons by the inconsistency analysis.
Issues related to public bids are documented in [32].Some of the legal aspects of bids are outlined in [33].Multiattributes (multicriteria) evaluation and its relation to the analytic hierarchy process is presented by some authors.None of them, however, presents a comprehensive approach focused only on the bidding process and the evaluation of overall of criteria.Our approach includes all the aspects.

Example of Tendering Model
When the company already has a specific set of suppliers, with whom it intends to cooperate, it should choose the best of the many suppliers, characterized by various features.The vendor selection process must, therefore, be properly designed.Tools that help with the decision on the selection of a partner and thus allow us to assess bids from multiple suppliers and compare them and point to the best ones are necessary.
Supplier evaluation is based on a set of both tangible and intangible criteria.The former are relatively easy to compare (e.g., the price of a product is lower than that of another product with identical characteristics).The latter is more difficult to survey and require detailed considerations.
A practical model of a tendering process needs to be as flexible as possible (see also conclusions).Presenting any model for such process is risky, since some readers may conclude at this point that it is not suitable for him/her.One may always discount any model as irrelevant.However, leaving a reader without any practical application of the presented framework is unacceptable.Therefore, a compromised solution is proposed and Box 1 contains a set of selected criteria most frequently used in tenders.These criteria are based on the indicators in the SCOR documentation [34,35], articles, studies, reports about selections criteria, analyses of companies, and interviews with experts.Thus, our model is a mixture of a lot of approaches to supplier selection.We chose some criteria of supplier selection which were verified by experts in supply chains.The proposed list is not exhaustive and does not pretend to be complete but still fairly representative.It is worthwhile to note that the authority issuing a tender can select arbitrary criteria for each type of bid.They can also scale (or weight) particular criteria, depending on the kind (or extent) of work, the required potential of the contractor, or the necessary level of technology.
The criteria have been divided into five main groups: (i) quality of product or service (qua), (ii) flexibility and adaptability (flex), (iii) organizational potential (pot), (iv) financial standing and payment conditions (fin), (v) experience (exp).
The above-mentioned criteria together form a Supplier Quality Index (SQI).The two criteria: price and delivery time are not mentioned here deliberately.These are the tangible factors, which can be easily compared with each other, but they are usually threshold factors.We could, for example, get a very good deal from a supplier who could deliver what we need now in ten years.For this reason, our model includes factors which are usually labeled other factors.For this reason, our model should be regarded as an extension to the classical model of supplier selection since the computed factor may Tender evaluation criteria Quality of product or service (qua) Reliability and stability of the product or service (reliability) Quality standards (q-standards) Quality guarantees (q-guaranty) Quality control system (q-system) Flexibility and adaptability (flex) Ease of change introduction (e.g., of resources, processes) (change) Readiness to execute orders according to clients terms (ready) Ability to deal with a changing economic environment and crisis situations (a-deal) Ability to produce personalized products or improve existing ones according to customers' requirements (a-personal)  [36]).Moreover, Simon is a winner of Nobel Prize in Economics (1978).Besides, the tangible factors are usually the criteria of the greatest weight in the evaluation process.SQI is, therefore, an important addition to the analysis of suppliers.Obviously, when choosing suppliers, all types of criteria can be considered.Often, they form close relationships.For example, price is inextricably linked to the quality of the product or service.However, immeasurable factors such as payment terms and supplier flexibility have an indirect effect on the price of the product or service.
The criteria presented in Box 1 may not always meet the needs of the company.Therefore, they should be regarded as a starting point to determine its own set of criteria that a specific company will use.This means that this list is not exhaustive and, if required, can be extended or shortened.It shall be noted, however, that the adopted criteria should be known to all the suppliers from the enterprise network being evaluated.
For the needs of the article, surveys were conducted among managers dealing with cooperation with suppliers in their enterprises.The study was divided into two stages.First, the entrepreneurs were interviewed in order to make a list of the most important criteria for selecting suppliers in a supply chain.On this basis, a questionnaire was prepared for the previously mentioned survey.The second stage of the study (with the use of the questionnaire) was aimed at establishing the importance of the individual criteria (by comparing them in pairs) for selecting suppliers in a supply chain.
The survey was conducted in September and October of 2013.A representative sample of 49 entrepreneurs from Poland participated.The respondents included mainly directors, logistics specialists, supply chain managers, and merchants.It is worth to note that according to this formula:   = 0.25 × ( / ) 2 in [37] showing a table with  ; a sample size of 49 observations is sufficient to achieve approximately 85% certainty usually desired for a pilot study.
Every person had to answer 10 questions and, thus, determine 10 ratios:  12 ,  13 , Table 2 demonstrates a matrix with relative comparisons.
A scale of 1 to 3 (and its inverse 1/3 to 1) is used.In the represented case, the highest importance has been assigned to qua because the quality of product and service (e.g., reliability and stability of the product or service) is usually of great importance to customers.When compared to the flexibility and adaptability (flex), qua has been assessed to be 1.5 (a compromised evaluation between 1 and 2 from Table 1).Flexibility and adaptability are a factor related to the ability to deal with a changing environment and to the reaction time to unexpected demand changes.It is assumed that organizational potential (pot) is less important for the customers (comparison of qua to pot is 1.2).This allows one to assess such evaluation criteria as production capacity, logistics opportunities, experience in cooperation, management, and organizational skills.The assessment of the importance of the quality factors against the financial standing and payment conditions (fin) (e.g., cash conversion cycle, financial stability) is set to 1.4 (when compared to qua).The qua factor is only 1.3 times more important than the experience (exp), which is related to the number of customers, recommendation of other customers, and so forth.Flex against pot is assessed to 0.8 which means that pot is 1.2 times more important than flex (see Figure 1).The relationships between flex and fin, as well as flex and exp, are quite similar and equal 1.1 and 0.9, respectively.Pot against fin is set to 1.4 and 1.2 when compared to exp.The last comparison is fin to exp and it equals only 1.1, which means fin is almost equally important as exp.It is worth to note that the maximum of the relative judgments in our research equals only 1.5.This means that there are no significant differences between the presented criteria.No relative judgment is absolute, or even strongly important in comparison to the others.Bold face 1's on the main diagonal are arbitrary values due to the fact that they represent a relative ratio of a criterium against itself.Values below the main diagonal do not need to be entered by the user.They are reciprocal to the corresponding values in the upper triangle (for details see matrix A).
Figure 1 presents data from Table 2 on a bar chart.

Consistency Analysis of Tendering Model and Final Weights for Evaluated Criteria
One of the challenges posed to the pairwise comparisons method is the lack of consistency of the pairwise comparisons matrices which arise in practice (see [38]).Given an  ×  PC matrix A which is not consistent, the theory attempts to provide a consistent  ×  matrix B which differs from matrix A "as little as possible."Let  = ( 1 , . . .,   ) be the eigenvector of A corresponding to ; the largest eigenvalue of A. A statistical experiment (see [26]) shows, however, that the accuracy of weights does not strongly depend on the method.In particular, the geometric means method (see [39]) produced similar results with high accuracy to the ten million cases.There is, however, a strong relationship between accuracy and consistency.This is the main focus of the consistency-driven approach.
In practice, inconsistent judgments are difficult to avoid, when at least three factors are independently compared against each other [27].In this study, consistency was successfully achieved.For example, let us look closely at the ratios of the first three criteria in Table 2: qua (for short ), flex (denoted by ), and pot (referred to as ).The assessment of  against  is 1.5;  against  is assessed as 0.8.The ration of  to  is 1.2.
Let us try to illustrate the inconsistency analysis.From / = 1.5 and / = 0.9, we can infer that / = 1.5 ⋅ 0.9 = 1.35, which is different from our original assessments (see Table 2 where / = 1.3).In fact we do not know which assessment was incorrect.In particular (a frequent case in practice) each original assessment might have been (and usually is) only slightly inaccurate.
For details related to consistency analysis see [28].
Table 3 shows the consistency analysis for all triads (, , ) from the matrix with relative comparisons (see Table 2), where  = /,  = /, and  = /.The maximum value of cf is only 0.2, which means that the respondents' answers are consistent enough based on the former application as well as theory provided in [28].Basically, the heuristic "by one off" and the "satisficing rule" (mentioned in [36]) are used.Simply, the comparisons do not need to be 100% exact, but they should be good enough and "by one off" from the ideally consistent minimal cycle of three comparisons (needed to compare three criteria which is the minimum size of a cycle, two elements do not create a cycle).
By increasing or decreasing PC matrix entries (the most inconsistent triads are easily located by the simple search procedure), we develop a very good orientation quickly.
All computations, including the final weights, are done by the Concluder software.It is freely distributed by Souce-Forge as Concluder with a tutorial posted on YouTube as jConcluder.It is a flexible and powerful evaluation tool for complex systems with the expectations of having more applications in the future as more decisions are made under growing financial stress.
It is not important to address all mathematical aspects of getting the final weights but the eigenvector method (see [28] for details) can be used to obtain results illustrated in Figure 2. As we can see, the product quality and service factor has the highest weight (25%).The second most important criterion is the financial standing and payment conditions (22%).The other three factors are quite similar and they equal 17-18%.
One common concern needs to be addressed: Can we do it?in short, Yes, you can.Using a more precise consistencydriven approach may look complicated at first glance.It may be particularly visible when it comes to making the comparative judgments.How can we start?Is it not just easier to assign points to the list?It is even advisable to start with assigning some points using the so-called by eye common sense method.Having that done, we can easily construct the pairwise comparisons matrix A (see Section 6) by simple division of points for the corresponding criteria.In fact, one may even give up at this point; however, after a careful examination of the matrix, we often may be unpleasantly surprised by our own judgments.We may, for example, discover that certain ratios are surprisingly small, while others are out of common sense limits, since comparing two at a time is easier than by eye estimation.

Example of Suppliers Assessment
The next stage of the supplier selection is their assessment.Having established the criteria, knowing their weights, one can proceed to the analysis of suppliers in the given circumstances.If these criteria and suppliers are abundant, it may be helpful to apply a multicriterion analysis using mathematical methods, such as optimization at the so-called multiplicity of attributes.It involves calculation of the synthetic evaluation with the use of the weighted mean, in the way presented below.With a list of individual ratings criteria:  1 ,  2 , . . .,   with weights  1 ,  2 , . . .,   , where   > 0, the weighted mean can be calculated:  = ∑  =1     / ∑  =1   .The weighted mean is calculated for individual suppliers.Finally, those suppliers for which the mean value is highest are taken into account.
For the needs of the paper the authors took into account a large enterprise which produces clothes and is located in Poland.The aim of the company was to implement a supplier evaluation process that would allow us to choose the best fabric for the production of clothes from among the suppliers (, , , ).Table 4 presents the criteria from Box 1 supplemented with weights (from Figure 2), assessments, and synthetic assessments calculated (assessment multiplied by weight).The grading scale ranges from 1 to 5 (1 being the worst rating and 5 the best).In this case, the best final grade was awarded to supplier  (score 4.31), which, despite lower quality (lower   value) than that of supplier , received better evaluations for most of the other criteria (higher   value) from the recipient.If supplier  improved its financial standing and payment conditions, and in particular the loan possibilities, cash conversion cycle, and capital rotation, the result would be even better and it would reinforce its position among other suppliers of the analyzed company.Supplier  got the worst evaluation (score 3.35).This supplier has the worst product or service quality and organizational potential of all.
The authors are aware that the scheme presented in this section may not seem very complicated.However, the procedures offered by the multicriterion programming should be conceptually simple enough for a person who does not have preparation in mathematics and operations research to use them without difficulty.It is very important, since the evaluation of many criteria (e.g., flexibility and adaptability) cannot take place without the participation of employees, even though most of the steps in this procedure are automated.
A recent application of the presented method for assessment at a national level in Poland is in [40].

Conclusions
This study presents a theoretical framework for building a logical model for evaluation of suppliers.It also proposes how Where R-tot means running total of the weight the final weights may be obtained from relative pairwise comparisons.However, the framework is not a ready cookbook for constructing such models.Models can be constructed in many different ways.A good model should, however, contain both tangible and intangible criteria.Identification of major criteria is one of the essential components of constructing models.Each model strongly depends on the specifics of the given case and its business environment.The presented approach allows the buyers to create their own list of criteria without any limitations.The price and delivery time have not been included in our approach.They are threshold factors and reserved for the final decision by the evaluation panel.In other words, the model provides the assessments of tenders as a vector of weights.These weights are applied to evaluation of each individual proposal.The final product of the evaluation generates a list of tenders with the overall score (received by a vector product of weights and corresponding evaluations for each criteria), plus the price and the delivery time.Frequently, it is an evaluation panel's decision whether or not to award the contract to the best tender according to the overall score.
There are easy cases when the winner's proposed price and delivery time are similar to the next, but in other cases it may be substantially different.Simply, the financial constraints may prohibit consideration of some solutions (e.g., the delivery time may be unacceptable).The consistencydriven pairwise comparisons method can be used by everyone on a personal computer thanks to aforementioned Concluder software.
More research is needed to shortlist all intangible criteria which are used to evaluate suppliers.Currently, it is done by the intuition but it needs to be investigated.The proposed model can process any number of such criteria.
The authors intend to carry out studies on a larger sample of managers who will represent not only Polish companies but also foreign ones.This will allow us to compare the results and highlight potential differences in the supplier evaluation criteria.In addition, it will increase the representativeness of the research and hence the results will be more universal.

Figure 1 :
Figure 1: Relative judgments for the group of criteria on a bar chart.

Figure 2 :
Figure 2: The final weights for the evaluated criteria.

Table 4 :
Sample assignment of numerical values to criteria for supplier evaluation.