MCDM Approach for Assigning Task to the Workers by Selected Features Based on Multiple Criteria in Crowdsourcing

Crowdsourcing in simple words is the outsourcing of a task to an online market to be performed by a diverse group of crowds in order to utilize human intelligence. Due to online labor markets and performing parallel tasks, the crowdsourcing activity is timeand cost-efficient. During crowdsourcing activity, selecting the proper labeled tasks and assigning them to an appropriate worker are a challenge for everyone. A mechanism has been proposed in the current study for assigning the task to the workers. .e proposed mechanism is a multicriteria-based task assignment (MBTA) mechanism for assigning the task to the most suitable worker. .is mechanism uses approaches for weighting the criteria and ranking the workers. .ese MCDMmethods are Criteria Importance .rough Intercriteria Correlation (CRITIC) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). Criteria have been made for the workers based on the identified features in the literature. Weight has been assigned to these selected features/criteria with the help of the CRITIC method. .e TOPSIS method has been used for the evaluation of workers, with the help of which the ranking of workers is performed in order to get the most suitable worker for the selected tasks to be performed. .e proposed work is novel in several ways; for example, the existing methods are mostly based on single criterion or some specific criteria, while this work is based onmultiple criteria including all the important features. Furthermore, it is also identified from the literature that none of the authors used MCDM methods for task assignment in crowdsourcing before this research.


Introduction
e term crowdsourcing refers to the outsourcing of different tasks to a huge amount of people known as crowd in order to utilize collective human intelligence. Crowdsourcing was first defined by J. Howe as the outsourcing of tasks or work to a network of undefined people by means of an open call format. e term crowdsourcing represents an act of an institution by taking a function performed by crowd and then outsourcing it to a network of undefined group of individuals [1]. e process of crowdsourcing contains three main actors, the requester or client who requests the work or task to be performed, the crowd who performs the requested task, and the platforms who act as a broker between the clients and crowd [2]. Crowdsourcing platforms contain paid and unpaid platforms, where in paid platforms crowd performs the tasks as a result of monetary rewards, while in unpaid platforms the volunteer crowd performs the tasks [3]. e use of crowdsourcing is increasing day after day as it is time-and cost-efficient for software development and many other tasks. It has been applied in several domains such as design-based apps, text translation to different languages, and labelling datasets [2]. e term CSE (crowdsourced software engineering) is derived from crowdsourcing. By means of an open call, it recruits online workers globally to perform several software engineering tasks such as requirement elicitation, coding, designing, and testing. It reduces the time to market due to parallelism. CSE rapidly gained interest in both industry and academia [4]. Day after day, crowdsourcing is gaining the attention of the communities. While crowdsourcing the task assignment is a challenging phase for everyone, selecting an appropriate labeled task from the client or the requestor and assigning it to an appropriate worker are a challenging issue in crowdsourcing. During crowdsourcing process, some of the crowds select irrelevant tasks in order to get the rewards but they do not have the potential to perform the task. As a result, the crowd then submit the low-quality result and thus decrease the client trust, which directly affects the crowdsourcing process. So it is shown that the task assignment is an important step to be taken during the crowdsourcing activity [5]. at is why a solution is required in order to fix the task assignment problem. e main contributions of the proposed study are as follows: (i) A mechanism has been presented for addressing the issue regarding the assignment of different tasks during crowdsourcing activity (ii) e proposed work identified different features of workers and selected the important ones for making the criteria for task assignment (iii) e existing task assignment methods are mostly based on single criterion, while this work is based on multiple criteria (iv) Two MCDM methods, CRITIC and TOPSIS, have been used for giving weight to the selected features and evaluation of the workers to rank them for assigning tasks to the appropriate workers e remainder of the article is composed of four main sections. Section 2 describes the existing/related work, Section 3 describes research method, and the conclusion is given in Section 4.

Related Work
In the existing literature, the task assignment models and methods are recommended with the help of different techniques. e competitor's history of participation such as participation frequency and recency as well as winning frequency and recency along with tenure and last performance is derived in order to construct a model [6]. Literature work also proposed the framework for task recommendation for the task-based preference modelling as well as preference-based recommendation of task, the goal of which is to recommend task to crowd [7]. e recommendation algorithm for the personalized task is also advocated by the authors. e authors also suggest approaching the issue regarding the design of the mechanism of the personalized task recommendations. e three task selection methods, heuristic-based method, bound-based method, and active learning method, have also been identified by the literature [8]. e SmartCrowd framework is also proposed by the authors, which focuses on the optimization of crowd to the task assignment in a knowledge-intensive crowdsourcing environment. It focuses on the production of knowledge instead of simple tasks. It also integrates multiple factors of humans such as crowd expertise, the required wage, and the ratio of acceptance in the assignment process [9]. e task offloading was considered as one of the significant areas of research [10].
A bandit formulation for the assignment of tasks in heterogeneous crowdsourcing is also proposed, which is known as bandit-based task assignment. e worker is represented by the arm of the bandit. It mainly focuses on the selection strategy of workers in the heterogeneous crowdsourcing.
e goal is to select the worker that is suitable and good for the task [11]. A dynamic solution is also presented for the assignment problem of tasks in the crowdsourcing platforms. e crowd select approach offers an algorithm for assigning workers to the tasks in a costefficient way and also ensures the accuracy of the tasks. e two main components of the approach are worker selection and their error rate prediction [12]. A learning algorithm has been described by the authors, which groups the task from the history into cluster and then derives from every cluster the features of the worker which optimize the quality of the contribution. en these features are used by the algorithm to select the appropriate worker for the task [13]. To measure the effect of personality on the selection of tasks, the experiment was conducted based on the task characteristics such as type, money, and time. e type of personality of the workers was measured based on the Myers-Briggs Type Indicator. Sixteen different personality types are categorized. en an experiment was performed (four rounds) to determine which worker belongs to which category. e round 1 and round 3 personality types are interested in prize and money instead of complexity. e other two rounds of personality types want the prize and money rather than deadlines [14]. e Skills Improvement Aware Framework is also proposed by the authors to recommend crowd for task in the development process of software and crowdsourcing. A study of the developer performance on the TopCoder platform was also conducted [15]. e assignment problem of tasks is also explored in a budget constraint with different variety of skill levels and different required quality. An algorithm was also designed for the generation of outcomes for many-to-one matching problem with upper bound and lower bound and the skill level of the worker [16]. e sensitive task assignment is proposed by the authors. First of all, sensitive task is partitioned and then assigned to workers. For the avoidance of colluding participants with the help of which they can exchange the data, a three-step task assignment method is proposed, which is known as sensitive task assignment. e steps include collusion estimation, worker selection, and task partitioning [17]. A data-driven learning approach was also proposed by the authors. e supervised learning and reinforcement learning have been combined in the approach to enable the agents to imitate the task allocation strategies, which show good performance [18]. TOP-K-T and TOP-K-W are the two real-time recommendation algorithms proposed by Safran and Che. e first one computes the appropriate task for a worker and the second one computes the appropriate worker for the task [19]. e expertise prediction heuristics have also been proposed by the authors to identify the experts automatically and to filter a nonexpert 2 Scientific Programming during crowdsourcing activity. Based upon the four expertise prediction heuristics, an experiment was performed. ese include demographics of evaluator, reaction time of evaluation, and mechanical reasoning aptitude, and the last one is easy version of the evaluation tasks [20]. e Learning Automata Based Task Assignment (LEATask) is also proposed by the authors, which works upon the worker similarity in their performances. e algorithm has exploration stage and exploitation stage [21]. Different areas can be considered in the task assignment allocation such as energy cooperation and joint information for heterogeneous networks [22]. e batch allocation technique is also proposed for the crowdsourcing tasks with the overlapping skill requirements.
e designed heuristics approaches include corebased batch allocation and layered batch allocation. e experiment is made on the upwork dataset [23]. Two online task assignment mechanisms are also developed by the authors, which will dynamically assign set of tasks to the incoming crowd (worker) with the help of which the worker will gain the maximum expected gain and maximum expected and potential gain. e task is divided into clusters by the authors and then they proposed a Latent Topic Model to describe the structure of topic and expertise of the workers [24]. Based on the deep learning, the existing literature also proposed Tag-Semantic Task Recommendation model. e similarity of the word vectors is computed and then the Semantic Tag Matrix Database has been established based upon the Word2vec deep learning.
en the recommendation model for task is established based upon the Semantic Tags to achieve the recommendation of the tasks in crowdsourcing. e task and worker relevancy is obtained by computing the tags similarity [25]. e Dynamic Utility Task Allocation (DUTA) algorithm was proposed in the paper. For the estimation of the worker's initial value, the attributes are used, which are given by the worker during the time of registration. e developmental capabilities of workers are also calculated by the history of completed task, complexity of tasks, and quality and efficiency of the result. en the matching degree is calculated based upon the weight of crowd skills and posted task requirement [26]. e active time of the worker is also used to get a solution for Multitask Multiworker Allocation. e three factors that the authors consider are the ability of the worker, active time of the worker, and the complexity of task module. e individuals are divided into collaborative groups, and then, for the optimal selection of worker to perform a task, the Hungarian algorithm is used [27]. e automatic detection for improper task is also proposed in the process of crowdsourcing. With the help of analyzing the estimated classifier, a variety of effective measures for the detection of improper task are observed, including words that appeared in task information, reward or money which will be received by the workers after performing the task, and their qualification for performing the task [5]. Various research fields can benefit from using the proposed approach such as IoT underlying heterogeneity [28], investigation of data aggregation in mobile sensor networks for IIoT [29], sharing of resource in heterogeneous vehicular network [30], and many other fields.

Methodology
As the assignment of task during crowdsourcing activity is a challenge for everyone, in the current study, a mechanism is proposed for task assignment method, which is based on multiple criteria. e study proposed a multicriteria-based task assignment (MBTA) mechanism. Two methods have been used in the current study. e CRITIC method has been used for assigning weights to the selected features and the TOPSIS method is used for ranking the workers. e work done in these methods can also be performed manually, but selecting these methods for doing the proposed work can give authenticity and appropriateness to the research work. Performing work manually contains several chances of mistakes, but doing it with predefined and already experimented methods increases the quality of the work; therefore, these two methods have been selected for the proposed study. e details of the study are discussed in the following sections.

Criteria for Task Assignment.
To define the criteria for task assignment, first of all a variety of features have been identified from the existing literature. 33 of the most famous and important features have been identified during literature study. ese features are then analyzed and the most important ones are selected for the development of mechanism for task assignment. e weight has then been assigned to these identified features by the CRIRIC method, which is discussed in the next section. e list of identified features is shown in Table 1. Table 1 shows all the identified features analyzed during systematic literature review. e important features are then selected from the list in order to develop a mechanism.

Case Study.
To complete data collection for making criteria, a case study was performed. In this case study, the issues regarding the assignment of task have been highlighted. All the gaps have been discussed briefly. A comprehensive observation was carried out in order to select the features for criteria from the identified features during literature. e experts were asked different questions in order to select the most important features. ese features were then scaled, ranging from 1 to 10, with the help of experts. A group of experts scaled these features so that important features get more weight among the other features, thus making it easy to rank the crowd who have good qualities at the top. After that, these features have been used for analyzing and making criteria as well as for evaluation of the workers to rank them for assigning a task. As all the features have been identified from the existing literature, for further analysis, experts were asked some questions based on these selected features in order to analyze the importance of these features. e questions the experts were asked are shown in Table 2.
A list of the selected features is shown in Table 3.

Weight of Selected
Features. e features have been analyzed by experts in the relevant field. Scaling was given to each criterion/feature, ranging from 1 to 10, by these experts Scientific Programming 3 in order to get the most important criteria. Weights have been assigned to all these selected features with the help of the CRITIC method. e final weights have been obtained by applying equations (1)-(4), respectively. e final weights have been shown in Table 4 and Figure 1. Table 5 describes the scales of the selected features. e weights for each criterion are shown in Figure 2.

Proposed Mechanism for Task Assignment.
e MBTA mechanism has been proposed, which is based on multiple criteria. is mechanism has been developed based upon two methods. e CRITIC method is used to assign weights to the selected features, and then the TOPSIS method is used for ranking the workers. e details are discussed in the following sections.

CRITIC Approach for Allocating Weights to Features.
CRITIC is a type of correlation method which was first introduced in 1995. It is a multicriteria decision-making approach that is used for assigning weights to features or criteria during research work. During this method, the weight is assigned to the criteria objectively rather than by pairwise comparison or decision-makers judgments [41].
"m" is the number of possible alternatives such as Ai, when i � 1, 2, 3, . . ., m, and "n" is the number of evaluation criteria such as C j for j � 1, 2, 3, . . ., n, in a problem. e following steps are followed in the approach.

Q1
What is the importance of worker history while assigning a task?

Q2
What is the importance of trustworthiness in task assignment?

Q3
How much worker qualification matters during task assignment? Q4 Is the reliability of the worker important for assigning a task? Q5 What is the role of response ratio in assigning a task? Q6 Does skill level matter for task assignment? Q7 Is the quality of task important for clients? Q8 What is the importance of delivery time in crowdsourcing? Q9 What is the role of cost in assigning a task? 4 Scientific Programming

Scientific Programming
In equation (1), X ij shows the performance value of the ith alternative on the jth criterion.
(2) X * ij is the normalized performance value of the ith alternative on the jth criterion.
Step-3. Calculating Standard Deviation and Its Correlation In the third step, the weights of the jth criterion can be found with the following equation: In equation (3), C j is the amount of information contained in the jth criterion. C j is calculated as follows: where σ j is the standard deviation of the jth criterion and rjj′ is the correlation coefficient between the two criteria [41].

Numerical Work of the CRITIC Method.
Weights are assigned to the criteria using the CRITIC method. e determination of this study was to find the top worker based upon the features for the offered task. e workers that will perform the tasks have been used as alternatives such as A1, A2, A3, A4, A5, A6, A7, A8, A9, and A10; and the features have been used as criteria such as worker history (C1), trustworthiness/honesty (C2), worker qualification (C3), reliability (C4), response ratio/quality of response (C5), skill level (C6), quality of task (C7), delivery time (C8), and cheap/cost-effective/cost-efficient (C9). Decision matrix has been established for these 10 workers (alternatives) with respect to defined features/ criteria as shown in Table 6. e results given in Table 6 are obtained by normalizing the decision matrix while applying equation (2). Figure 2 shows the steps followed in this method. Now calculations of the CRITIC method are followed step by step. Table 5 describes the CRITIC method decision matrix. Table 7 shows the CRITIC method normalized decision matrix.
Measure of conflict has been calculated as shown in Table 8.
Standard deviation and its correlation with other criteria have been calculated for criteria weights as shown in Table 9.
For each worker, all the 9 features/criteria have been scaled, ranging from 1 to 10, as shown in Table 4.

TOPSIS Approach for Ranking of Workers.
e TOPSIS approach deals with achieving ideal solutions. is approach has adopted simple computation procedures and thus it is reliable and well established. e selected alternatives in the TOPSIS method should have a minimum distance from positive ideal solution and maximum distance from negative ideal solution [41]. In this work, we will apply the TOPSIS method for ranking the alternatives. In this section, first of all, the TOPSIS method along with its steps and procedure to be followed will be discussed and then how this method has been used in this research will be discussed. e following are the steps used in the TOPSIS method in order to select and rank the best workers among different alternatives: Step-1. Determining Weight and Building Decision Matrix Decision matrix D is constructed in the first step by using multiple criteria and alternatives. For example, for "n" number of alternatives and criteria, the decision matrix can be found as follows: where A 1 , A 2 , A 3 , . . ., A n are variable alternatives and C 1 , C 2 , C 3 , . . ., C n are the criteria.
Step-2. Normalized Decision Matrix As the input data of the decision matrix is originated from several different sources, it has to be converted into a dimensionless matrix by normalization. e comparison between different criteria is done via this dimension matrix. By using formula (6), a normalized decision matrix has been built.   Scientific Programming where i � 1, . . ., m and j � 1, . . ., n. Step

-3. Weighted Normalized Decision Matrix
As it is not necessary that the importances of all attributes will be the same, by multiplying the elements of the normalized decision matrix with random weight number, a weighted normalized decision matrix can be obtained. e weight number for multiplication is given in the following formula: Step- 4

. Finding Ideal Positive and Negative Solutions
In this step, A + denotes positive ideal solution and A − denotes negative ideal solution. ese are demonstrated through the weighted decision matrix.
where J denotes the beneficial attributes and J′ denotes the nonbeneficial attributes.
Step-5. Separation Measures By the following formulas, ideal and nonideal separations are calculated.
Step-6. Finding Relative Closeness It is determined with respect to the ideal solutions by using the following equation: Step-7. Alternatives Ranking By using C i value, the process of ranking is prepared; high C i value shows top rank order of the alternative, which can be labeled as superior in terms of efficiency.
e descending order can be adopted for the comparison of improved performance [41].

Numerical Work of the TOPSIS Method.
In this section, evaluation of the workers and their ranking will be obtained based upon 9 identified features by using the TOPSIS method. e data has been collected from different questionaries answered by several experts in relevant fields. e decision matrix is constructed by the data obtained from the panel of experts.
All the work is done step by step as shown in Figure 3. By using equation (6), normalized decision is obtained. e results are listed in Table 10 along with criteria weights.
Decision matrix based on weighted normalization is obtained by using equation (7). Ideal positive and ideal negative solutions are calculated by using equations (8) and (9) and their values are given in Table 11.
Positive ideal solution and negative ideal solution are used for finding ideal and nonideal separation measures. ese separation measures are calculated by using equations (10) and (11). Ideal separation measures (S + ) for W1, W2, Ranking is done upon the value of P i , and the high value of P i shows the top alternative. After relative closeness has been calculated, the ranking of workers is done based upon the value of P i . In this research, alternative W3 had higher P i value among other alternatives and thus got the first ranking, while W1 had second highest value and got rank 2, and so on. As W3 had higher P i value and was ranked as 1, it was more reliable among all the other workers and was the most appropriate for the selected task to be performed. e details are shown in Table 13.
From Table 13, ranking of the workers is clearly presented, and tasks will be assigned to the most suitable workers according to their ranking. Graphical representation of workers' ranking is shown in Figure 4.

Conclusion
Assigning the task to the most appropriate worker is very important in crowdsourcing because if the task is assigned to an inappropriate worker it affects crowdsourcing activity in several ways such as waste of time, money, and clients trusts. e proposed research presents a mechanism for assigning a task to the worker. is proposed mechanism is based on multiple criteria. Worker features such as worker history, trustworthiness/honesty, worker qualification, reliability, response ratio/quality of response, skill level, quality of task, delivery time, and cheap/cost-effective/costefficient are selected by the identified features. Two MCDM methods, CRITIC and TOPSIS, have been used. Weights have been assigned to these features by the CRITIC method and then evaluation and ranking of the workers have been analyzed by the TOPSIS method in order to assign the task to the most appropriate worker. As the existing task assignment is based on single criterion, the proposed work is novel in terms of assigning workers based on multiple criteria as well as using MCDM methods for current work in crowdsourcing.

Data Availability
No data were used to support this study.