Mass Collaboration-Driven Method for Recommending Product Ideas Based on Dempster-Shafer Theory of Evidence

In the mass collaboration mode, there exist a large number of product ideas with low value density and thousands of participants who are differed on their professional backgrounds, knowledge structures, and value orientations. It is impossible for each participant to give a comprehensive evaluation of each idea as that in traditional methods for the reasons as mentioned above. In order to solve this problem, a mass collaboration-driven method for recommending product ideas is proposed based on Dempster-Shafer theory of evidence (DST). Firstly, the method for computing basic probability assignment (BPA) function, which can effectively reflect the facticity of experts’ evaluations, is introduced by discounting belief degrees with weights to extract the evaluation information of product ideas. Then, Dempster’s combination rule is used to combine the derived BPA functions for two times: the first one is to combine the discounted BPA functions on all criteria with respect to a specified expert and the other is to combine the combined BPA functions for all experts with respect to a specified alternative. Finally, the steps of mass collaborationdrivenmethod for recommending product ideas based on theDST are proposed. An illustrative example is provided to demonstrate the applicability of the proposed method.


Introduction
Product idea refers to the production of novel and useful idea to complex, novel, and ill-defined problems [1][2][3].Successful innovation requires high-quality ideas and how to effectively generate novel and useful product ideas continues to be a critical issue for both design scholars and practitioners.
At present, research on product ideas is mainly divided into two processes: one is idea generation, whereby ideas emerge in an associative manner, and the other is idea evaluation, whereby generated ideas are evaluated and screened.From the perspective of idea generation.Chou presented an ideation method for generating new product ideas [4]; Kazmi and Kytola attempted to explore assessment of a company's new product ideas generation potential linked to their industrial teams' diversified capabilities as well as their work potential [5]; Banović et al. used projective and creative research techniques to involve consumers in the process of modification and generation of new product ideas [6]; Shroyer et al. presented a qualitative case study of a professional design team's use of ideas generation with analyses at five emergent timescales [7]; Giller et al. suggested that either unbounded or prohibitive task instructions should be used when crowdsourcing innovative ideas [8]; Kwon et al. proposed a methodology to be served as an essential supporting tool for generating creative ideas that could spark innovation [9]; Mirtalaie et al. presented a systemic framework for product designers in the ideation phase of new product development.From the perspective of product ideas evaluation; Steele et al. developed a measure of ideas evaluation self-efficacy [10,11]; Özaygen and Balagué proposed a methodology to reduce crowd innovation voting bias and to help managers to better select the ideas [12]; Hao et al. explored the neural correlates underlying the effects of ideas evaluation on idea generation in creative thinking [13]; Mayseless et al. proposed an explanatory model of ideas centered upon the key role of the left temporoparietal regions in evaluating and inhibiting ideas [14]; Hoornaert et al. established a new model that can improve the reliability of ideas ranking by means of automated information retrieval methods, linear methods, and nonlinear machine-learning algorithms [15].
With the development of information technology, more and more subjects can participate in the innovation [16], and many enterprises have carried out product innovation in mass collaboration mode and achieved initial results [17,18], as mass collaboration mode has reduced BMW's R & D cycle from 2 years to half a year and made Linux Kernel's software system update continuously in weeks; HUAWEI focuses on "user perception" and has established a close cooperative relationship with more than 30 telecom operators, industry alliances and institutions, vertical industry leaders, and so on and has released a new generation of Easy Macro solutions and a new generation of 3D MIMO solutions.Although the mass collaboration mode has helped many enterprises achieve some results, product innovation based on this mode is still an important issue facing the enterprise at present stage.On the one hand, there are a lot of ideas in the generation stage.Thousands of participants can publish their opinions on product innovation with various Internet platforms, thus generating a large number of ideas with low value density.On the other hand, there are many participants in the idea evaluation stage.The participants not only include the internal members of the enterprises but also include the vast number of current and potential users.Some scholars have done some researches on idea generation and idea evaluation in product innovation, which are of great significance to enrich the research achievements in this field.However, the large number of ideas and participates in the mass collaboration mode leads to the existing achievements which are not suitable for solving the problem of product idea evaluation.The reason is that the participants may come from the relevant subjects such as the internal members and social public, and the knowledge, preference and ability of each participant may be different from each other.It is impossible for each participant to give a comprehensive evaluation of each idea as that in the traditional methods.
In order to solve the above problems, this paper puts forward a mass collaboration-driven method for recommending product ideas based on Dempster-Shafer theory of evidence (DST), on the basis of taking full consideration of the characteristics of the large number of ideas and the large number of participants.Existing methods in this field require all experts to give complete evaluation information on all criteria for each idea, but this is difficult to achieve in the mass collaboration mode.In contrast, considering the incompleteness of expert knowledge, the proposed method allows each expert to evaluate only one or more criteria for an idea in his/her field of expertise.Moreover, it allows each expert to be assigned different weights in each criterion, so as to obtain more reasonable and effective evaluation information.However, the proposed method can only sort the ideas from a static point of view and cannot achieve realtime sorting for the added ideas and evaluation information.
The rest of this paper is organized as follows.In Section 2, we introduce the framework of evaluating product ideas.In Section 3, we propose the method for recommending product ideas based on the DST and analyze the specific steps of the proposed method.In Section 4, we use an example to illustrate the combination process of the proposed method.Section 5 concludes the paper.

Framework of Evaluating Product Ideas
2.1.Generation of Ideas.Ever since the publication of Osborn's (1953) influential book Applied Imagination, many organizations consider group brainstorming as a particularly effective technique for generating large numbers of creative ideas [19].To stimulate creativity, a group idea-generation technique was developed and popularized under the name of brainstorming [20].This technique has been extended to electronic brainstorming, in which group members simultaneously produce as many ideas as possible on computers in a short period of time [21].In view of the advantages of electronic brainstorming that electronic brainstorming groups perform better than face-to-face groups, in both the number of high-quality ideas and the average originality of ideas, we decide to use this method to generate the needed ideas.
In order to get effective product ideas, enterprises should publish innovative requirements on the Internet platform.For example, on the premise of guaranteeing the quality of the product, compared with the extravagant outer packaging, the enterprise can require a simple and clear design in the appearance and packaging of the product.It cannot only reduce the cost of packaging but also attract consumers' attentions.In other words, the enterprise can require new product ideas to be differentiated in terms of product functionality, durability, style design, delivery, installation, customer training, and consulting services, so that new products are distinguished from the same products on the market.
After the enterprise announces the innovation request, the public can be the participants in the mass collaboration mode and all of them can express their opinions and discuss freely on the various interactive platforms.In order to get as much product ideas as possible, enterprises adopt electronic brainstorming method and make full use of all kinds of online platforms to encourage participants to participate in the process of product ideas generation around innovative requirements.The goal of electronic brainstorming is to generate a list of ideas, applying four rules: (a) focus on quantity, (b) withhold criticism, (c) welcome unusual ideas, and (d) combine and improve one's own ideas.Thus, a large number of product ideas have been generated.

Classification of Participants.
Nowadays, many companies rely on online innovation communities to collaborate together with users and integrate external user knowledge within firm boundaries for new product development purposes [22].All people except internal members of the enterprise can be regarded as users, who either have bought a product (current users) or are likely to buy a product (potential users).Numerous studies have shown that positive interaction between users and enterprises is conducive to the transformation of new technologies and the development of new products.In addition, employees, as insiders, have a deeper understanding of the real situation of the enterprise and the goal of product innovation and can participate in the whole process of product innovation.To sum up, according to the differences in value orientation and knowledge structure, this paper divides the participants in the mass collaborative mode into three categories: internal members of the enterprise, current users, and potential users.
(1) Internal members of the enterprise: As the originator and manager of the product ideas, internal members of the enterprise participate in the formation of the ideas in the whole process, which can be divided into managers, technical developers, production managers, and marketers.Managers mainly evaluate an idea from the perspective of economic benefits; technical developers mainly evaluate an idea from the perspective of enterprises' ability to transform existing technologies into product characteristics; production managers mainly evaluate an idea from the perspective of the production parameters (such as environment, medium temperature, pressure, etc.) and the technological design method; marketers mainly evaluate an idea from the perspective of marketing requirements.
(2) Current users: The current users refer to the users of a technology, product, or service, who have used one product.They are the actual experiencers of the product, have a more comprehensive understanding of the advantages and disadvantages of the product, can provide the corresponding information for the problems existing in the product, and can also evaluate the existing ideas.
(3) Potential users: The potential users are those users who have not been used but have the desire to buy some kind of product or service, have the ability to buy, have the right to make a decision, and have a demand for the function provided by the product.They can evaluate an idea from the perspective of whether the product can satisfy their needs.

Evaluation Criteria for Product Ideas.
Early research on idea used the number of creative ideas as the only criterion to measure the quality of idea.With the deepening of idea evaluation, the quantitative evaluation method in the past is no longer applicable.At present, the general view of the research on product idea evaluation criteria is multicriteria comprehensive decision-making.This view holds that idea evaluation depends on multiple criteria and comprehensive decision-making [23].
Plucker et al. proposed the idea that the evaluation of product idea should include two dimensions: novelty and practicality [24].Chan et al. suggested that the customer lifetime value (CLV) calculated can be regarded as the final evaluation criterion used to support idea screening decisions [25].Rietzschel et al. proposed that a good idea should be new and practical and put forward two criteria: novelty and feasibility [26].Diedrich et al. believed that the primary criterion for judging an idea is novelty, followed by usefulness [27].Schwarz and Bodendorf suggested that a good idea should be feasible, novel, profitable, useful, and highly mature [28].Hart et al. pointed out that technical feasibility is the most commonly used evaluation criterion in idea evaluation [29].MacCrimmon and Wagner believed that the criteria of idea decision-making need to take into account five aspects: novelty, nonobvious, relevance, practicality, and thoroughness [30].Carbonell-Foulquié et al. believed that strategic conformity, technical feasibility, customer acceptability, economic performance, and marketing opportunity are 5 major criteria for evaluating an idea [31].Dean et al. pointed out that the essence of an idea is novelty, based on which four first level criteria of evaluation, novelty, feasibility, relevance and particularity, are established, and two level criteria are established under each criterion [32].
The above evaluation criteria are based on many angles and discussed the idea itself, but this is not to be ignored that the fundamental purpose of ideas research and development is to better maintain the enterprise's own development.Therefore, the economic and technological benefits of ideas should also be emphasized.Hietikko proposed a model with two 2-dimensional tables, one for customer and the other for producer [33].The value for customer is divided into two factors: usefulness and economic efficiency, and the table of producer is divided into marketability and productivity.Eling et al. put forward that when evaluating an idea, we should consider the speed of evaluation and whether it can bring benefits to enterprises [34].Lauto and Valentin proposed that the decision time and the technological achievements contained in the idea must be taken into consideration [35].Ferioli et al. further refined the practicability of the idea evaluation criteria into two aspects: technology executable and economic benefits [36].Gutierrez put forward six principles of idea evaluation, which are core competitiveness, market position, module economic value, R & D capital, risk awareness, and organizational goal contribution [37].
On the basis of referring to the relevant literature, we select 4 criteria as the evaluation criteria of product ideas, as shown in Table 1.For examples, internal members of enterprise can evaluate the above 4 criteria according to the principle of whether an idea can bring benefits to enterprises.Current users and potential users can evaluate the above 4 criteria according to the principle of whether an idea meets their needs.Suppose

Evaluation criteria
Meaning of the criteria Novelty(c 1 ) Degree to which an idea is not only rare, but is also ingenious, imaginative, or surprising.Acceptability(c 2 ) Acceptability of the market or user, or the popularity of new products in the market.Usefulness(c 3 ) Value of an idea that can be brought to users and enterprises after implementation.Feasibility(c 4 ) Possibility of product development, the feasibility of existing technology, and the possibility of mass production.

Novelty Acceptability Usefulness Feasibility Internal members
Current users

Potential users
Experts that the evaluation information of alternative a  evaluated by experts E  with respect to criterion c  is expressed as   , .Thus, the framework of idea evaluation can be shown in Figure 1.

The Proposed Method
In this paper, based on the comprehensive consideration of weight, the DST and the Pignistic probability are used for ideas selection and ranking.The DST is an extension of classical probability theory by generalization of the Bayesian theory of subjective probability.Being a mathematical framework for representation of uncertainty, the DST combines the degrees of belief derived from independent items of evidences.It mainly deals with four concepts: frame of discernment, basic probability assignment (BPA), the belief or mass function, and the plausibility [38].Its advantages are mainly manifested in the following three aspects: first, it can handle more uncertainty in real world; second, no prior distribution is needed before the combination of evidence from individual information sources; third, it allows one to specify a degree of ignorance in some situations instead of being forced to be assigned for probabilities [39].The relevant definitions are shown as follows [40][41][42][43].

Information Extraction Method.
In the mass collaboration mode, considering the differences in knowledge structure, cognitive ability, and value orientation, every expert can only evaluate one or several criteria of an idea, which determines that there may be uncertain information about "incomplete known" or "completely unknown" in the evaluation information.Compared with the fact that each expert can evaluate all the criteria in traditional methods, the BPA function, as an expression of uncertain information, allows each expert to evaluate the criteria that he has the ability to evaluate, thereby maximally ensuring the authenticity of the evaluation information [44][45][46].In view of this, we use the BPA function to extract the evaluation information.Definition 1. Suppose the grade of an idea is expressed as   ( = 1, ⋅ ⋅ ⋅ , ), each grade is exclusive, and the finite set of mutually exclusive and exhaustive propositions Θ = { 1 ,  2 , ⋅ ⋅ ⋅ ,   } is the frame of discernment.
Definition 2. Supposing that the set of every subset of Θ is 2 Θ ,  is any subsets of Θ, and m() represents the degree of belief, if the mapping function  : 2 Θ → [0, 1] could fulfill In this paper, let the frame of discernment is Θ = { 1 ,  2 },  1 =Excellent, and  2 =Non-excellent, which represents the answer of "Is the alternative a  excellent or not on criterion c  for experts E  ?".In order to obtain the evaluation information of three types of experts within the same framework of discernment Θ, we assume that the choice of  = 0, which means that the experts E  do not give any evaluation information on the performance of a  on criterion c  .

Weight Determination Method for Three Types of Experts.
Because of the differences in knowledge structure and value orientation, the weights of the three types of experts on a criterion are different [47].Weight is the subjectively relative importance degree of an expert to another with respect to a criterion and usually can be determined by some computing methods such as Analytic Hierarchy Process (AHP) and Analytic Network Process (ANP).In this paper, we suggest using AHP method to determine the weights of three experts on a criterion.Three types of experts are compared two by two, thus the judgment matrix is constructed [48].The relative importance of each expert is obtained through normalization and a series of calculation steps.The weight of experts E  on criterion c  can be expressed as w  ,   ≥ 0, ∑ 3 =1   = 1, ∀.In addition, in order to ensure that the evaluation information is more scientific and reasonable, the weight of each criterion should also be considered.We also suggest using AHP method to determine the weights of each criterion, which can be expressed as w  ,   ≥ 0, ∑ 4 =1   = 1.

Information Combination Method.
In this paper, Dempster's combination rule is used to combine the derived BPA functions for two times.The first time is to combine BPA functions on all criteria with respect to the specified experts E  , so we can obtain the overall evaluation value for alternative a  on all criteria given by experts E  (called individual combination).The second time is to combine the combined BPAs for all experts with respect to a specified alternative a  , so we can obtain the overall evaluation value for alternative a  for all experts (called group combination).Due to the weight of each criterion and the weight of various experts on each criterion being different, in order to ensure the authenticity of evidence sources, we adopt the Shafer's discounting method to discount the obtained BPA function   , , and the discounting method is shown in After making the Shafer's discounting,  , , is integrated with the weight of each criterion and the weight of each type of expert, and then Dempster's combination rule can be used to make companion.Dempster's combination rule can be expressed as with  = ∑   ∩  =0  1 (  ) 2 (  ), where ,  ,, and   are subsets of Θ, ⊕ is the orthogonal sum operator, and K is a normalization constant, called the conflict coefficient of two BPAs Through the first combination, we can get the comprehensive evaluation value of experts E  on all criteria of alternative a  , that is,    .Then Dempster's combination rule is used again to combine the combined BPA functions    .Obviously, the combination process can be represented as    1 ⊕   2 ⊕   3 =   , so we can obtain the overall evaluation value for alternative a  for all experts, expressed as   .

Method
Steps.The steps of mass collaboration-driven method for recommending product ideas based on the DST are as follows.
Step 1. Identify alternative ideas.In the mass collaboration mode, the product ideas are identified by various participants based on product innovation requirements.A set of ideas can be expressed as  = {  |  = 1, ⋅ ⋅ ⋅ , }, and the main purpose of this paper is to rank them.
Step 2. Set decision parameters.Suppose a set of criteria is expressed as  = {  |  = 1, ⋅ ⋅ ⋅ , 4}, a set of three types of experts is expressed as  = {  |  = 1, 2, 3}, a set of grades is expressed as Θ = { 1 ,  2 }, a set of utilities for grades is expressed as  = {0 ≤  1 ≤  2 ≤ 1}, a set of weights of experts' type is expressed as   = {  |   ≥ 0; ∑ 3 =1   = 1, ∀}, and a set of weights of criteria is expressed as Step 3. Select the first idea.Let k=1 and take a 1 as the initial evaluation idea to be evaluated.
Step 4. Expert evaluation.In the mass collaboration mode, three kinds of experts, internal members of the enterprises, Mathematical Problems in Engineering current users, and potential users, can evaluate the various attributes of existing ideas.The evaluation value of alternative a  evaluated by experts E  with respect to criterion c  can expressed as   , = {(, , , ) |  ⊆ Θ,∑ ⊆Θ  , , = 1}, ∀, , .
Step 5. Make individual combination.Taking the specific experts E  as the combination target, based on w  and w  , Shafer's discounting is used to discount   , and get the discounted BPA functions  , , .Then Dempster's combination rule is used to combine  , , and we get the comprehensive evaluation value of experts E  on all criteria of alternative a  , expressed as    .
Step 6. Make group combination.Taking all experts as the combination target, Dempster's combination rule is used to combine    , and we get the overall evaluation value for alternative a  for all experts, expressed as   .
Step 7. Determine probability distribution.The BPA functions   obtained by Step 6 are converted to the form of probability distribution by the Pignistic probability.So we can obtain where is the concept of plausibility, and Step 8. Change the alternative idea.Let k=k+1; if k ≤ K, it means taking  k as the current evaluation idea, and going back to Step 4; if k>K, it indicates that the combination of all the ideas has been completed and going to Step 9.
Step 9. Rank alternatives.By combining grade with probability, we can get the probability distribution of the overall value of a  , which can be expressed as   = {(  ,    ) |  = 1, 2},  = 1, ⋅ ⋅ ⋅ , .According to the rank utility and the expected utility theory, the total utility of a  is calculated as The greater the total utility U  is, the better a  is.Then we can get the rank of all the ideas.

Illustrative Example
HUAWEI Technologies is a private high-tech firm based in Shenzhen, China, focusing on the provision of next generation telecommunications networks.Huawei is committed to providing Internet-protocol (IP)-based fixed & mobile communication (FMC) solutions to ensure that end users are able to experience consistent communication services at anytime, anywhere.HUAWEI's vision is "to enrich life through communication," and its mission is to "to focus on our customers' market challenges and needs by providing excellent communications network solutions and services in order to consistently create maximum value for customers."HUAWEI focuses on "user perception" and has established a close cooperative relationship with more than 30 telecom operators, industry alliances and institutions, vertical industry leaders, and so on.In order to give full play to the wisdom of group users, HUAWEI set up HUAWEI interactive community on the official website to encourage the majority of participants to participate in the platform exchange.It aims at providing a platform for direct communication with users, collecting users' opinions and needs, allowing users to define the next product of HUAWEI, discussing and communicating in circles in the form of a community, providing a rich data document, and allowing users to release their opinions on the platform.As a result, HUAWEI can receive a lot of product ideas, but how to select effective product ideas among a large number of product ideas with low value density is an urgent problem to be solved.This paper takes HUAWEI innovation as the background, and assumes that there are several product ideas, which can be evaluated by three types of experts: internal members of enterprise, current users, and potential users, so as to illustrate the decision making process of the proposed method.In August, 2018, HUAWEI interactive community Managers released a post entitled "What new APP features do you want?Say it, and win M3 panel computer!".All participants can respond to this post, either by presenting new ideas, or by evaluating existing ones.This post is valid for one month.In the meantime, the enterprise received 45 ideas and needed to rank these ideas for easy selection.In this paper, we take 3 ideas as the representative to introduce the ranking process of ideas.So, the set of ideas can be expressed as  = {  |  = 1, 2, 3}, a 1 = Change the theme color of APP according to the color of cell phone shell, a 2 = Home service center makes a manageable-sorted look, and a 3 = Add automatic save draft function.
This paper proposes the following assumptions: (a) there are 300, 500 and 200 individuals in three types of experts, namely, E 1 = Internal members of the enterprise, E 2 =Current users, E 3 = Potential users; (b) there are four criteria c 1 = Novelty, c 2 = Acceptability, and c 3 = Usefulness, c 4 = Feasibility; (c) the weights of the four criteria are assumed to be 0.17, 0.28, 0.26, 0.29; (d) on criterion c 1 , the weights of three types of experts are assumed to be 0.2, 0.4, and 0.4; on criterion c 2 , the weights of three types of experts are assumed to be 0.3, 0.4, and 0.3; On criterion c 3 , the weights of three types of experts are assumed to be 0.2, 0.5, and 0.3; On criterion c 4 , the weights of three types of experts are 0.4, 0.3, and 0.3; (e) the evaluation set of grades can be expressed as Θ = { 1 ,  2 },  1 =Excellent, and  2 =Non-excellent, which represents that each expert is asked by the question "Is the alternative a  excellent or not on criterion c  ?", and the utility of each grade is set as  1 = 1,  2 = 0.In this paper, the evaluation information of the 1000 experts on alternative a  is obtained through simulation, as shown in Table 2.
The numbers in Table 2 indicate the number of individuals in E  who think alternative a  is excellent, nonexcellent, or , .The weights of each criterion and the weights of experts on each criterion are substituted into (3), and the Shafer's discounting is used to discount   , , so the BPA functions  , , discounted by weight can be obtained.The discounted BPA functions are shown in Table 4.
According to Dempster's combination rule, we use (4) to combine the discounted BPA functions  , , .For the first time of combination, we can get the comprehensive evaluation value of experts E  on all criteria of alternative a  , expressed as    .Then, using Dempster's combination rule to combine    again, we can get the overall evaluation value for alternative a  for all experts, expressed as   .By substituting   in (5), we can get the probability distribution of comprehensive performance at all grades of alternative a  (see the 2-3 columns of Table 5).By substituting grade utility  1 = 1,  2 = 0 and probability distribution into (6), we can calculate the total utility U  of alternative a  (see the 4 th column of Table 5,).According to the total utility U  , we can know that the ranking is a 3 > a 1 >a 2 .

Conclusions
In the mass collaboration mode, the large number of ideas and participates leads to the existing methods which are not suitable for solving the problem of product idea evaluation.In this paper, we propose a mass collaboration-driven method for recommending product ideas based on DST.Firstly, the BPA function computing method is introduced by discounting belief degrees with weights to extract the evaluation information of product ideas, which can effectively reflect the facticity of experts' evaluation.Secondly, Dempster's combination rule is used to combine the derived BPA functions for two times: the first one is to combine the discounted BPA functions on all criteria with respect to a specified expert and the other is to combine the combined BPA functions for all experts with respect to a specified alternative.Finally, the steps of mass collaboration-driven method for recommending product ideas based on the DST are proposed.An example is proposed to illustrate the recommendation mechanism by the proposed method.The main contribution of this paper is that the DST is innovatively introduced to solve the problem of recommending product ideas in the mass collaboration mode, in which each participant can expediently give real assessments with his/her experiences and their assessments can be reasonably combined with Dempster's rule.It is worth noting that this paper only proposes the mass collaboration-driven method for recommending product ideas from the static combination perspective but does not consider the dynamic interaction.As a result, studying the dynamic interaction recommendation problem for the product ideas in the mass collaboration mode will be the next research direction.
Evaluation.Each product idea can be regarded as an alternative.Suppose that a set of alternative ideas is expressed as  = {  |  = 1, ⋅ ⋅ ⋅ , }, a set of three types of experts is expressed as  = {  |  = 1, 2, 3}, and a set of criteria is expressed as = {  |  = 1, ⋅ ⋅ ⋅ , 4}.Suppose the number of individuals in the experts E  is I  and    is the s ℎ individual in E  , then the set of individuals in E  can be expressed as   = {   |  = 1, ⋅ ⋅ ⋅ ,   },  = 1, 2, 3.According to the above, E 1 = Internal members of the enterprise, E 2 =Current users, E 3 = Potential users, c 1 = Novelty, c 2 = Acceptability, c 3 = Usefulness, and c 4 = Feasibility.After the generation of the ideas, for each idea, every expert can give an evaluation information on the performance of a certain criterion according to his own knowledge and experience.

Table 1 :
Evaluation criteria of ideas.
is excellent on criterion c  , then   → =  1 ; if    could not give any evaluation information on the performance of a  on criterion c  , then   → = Θ = { 1 ,  2 }.
with respect to criterion c  on alternative a  is   → .For example, if    thinks that the alternative a

Table 2 :
Evaluation information of all experts.

Table 3 :
BPA functions for all experts.

Table 3 .
The evaluation value of alternative a  evaluated by experts E  with respect to criterion c  is shown in Table3, expressed as

Table 5 :
Probability distribution and utility.