Crowdsourcing in simple words is the outsourcing of a task to an online market to be performed by a diverse group of crowds in order to utilize human intelligence. Due to online labor markets and performing parallel tasks, the crowdsourcing activity is time- and cost-efficient. During crowdsourcing activity, selecting the proper labeled tasks and assigning them to an appropriate worker are a challenge for everyone. A mechanism has been proposed in the current study for assigning the task to the workers. The proposed mechanism is a multicriteria-based task assignment (MBTA) mechanism for assigning the task to the most suitable worker. This mechanism uses approaches for weighting the criteria and ranking the workers. These MCDM methods are Criteria Importance Through Intercriteria Correlation (CRITIC) and Technique for Order Preference by Similarity to Ideal Solution (TOPSIS). Criteria have been made for the workers based on the identified features in the literature. Weight has been assigned to these selected features/criteria with the help of the CRITIC method. The TOPSIS method has been used for the evaluation of workers, with the help of which the ranking of workers is performed in order to get the most suitable worker for the selected tasks to be performed. The proposed work is novel in several ways; for example, the existing methods are mostly based on single criterion or some specific criteria, while this work is based on multiple criteria including all the important features. Furthermore, it is also identified from the literature that none of the authors used MCDM methods for task assignment in crowdsourcing before this research.
The term crowdsourcing refers to the outsourcing of different tasks to a huge amount of people known as crowd in order to utilize collective human intelligence. Crowdsourcing was first defined by J. Howe as the outsourcing of tasks or work to a network of undefined people by means of an open call format. The term crowdsourcing represents an act of an institution by taking a function performed by crowd and then outsourcing it to a network of undefined group of individuals [
The term CSE (crowdsourced software engineering) is derived from crowdsourcing. By means of an open call, it recruits online workers globally to perform several software engineering tasks such as requirement elicitation, coding, designing, and testing. It reduces the time to market due to parallelism. CSE rapidly gained interest in both industry and academia [ A mechanism has been presented for addressing the issue regarding the assignment of different tasks during crowdsourcing activity The proposed work identified different features of workers and selected the important ones for making the criteria for task assignment The existing task assignment methods are mostly based on single criterion, while this work is based on multiple criteria Two MCDM methods, CRITIC and TOPSIS, have been used for giving weight to the selected features and evaluation of the workers to rank them for assigning tasks to the appropriate workers
The remainder of the article is composed of four main sections. Section
In the existing literature, the task assignment models and methods are recommended with the help of different techniques. The competitor’s history of participation such as participation frequency and recency as well as winning frequency and recency along with tenure and last performance is derived in order to construct a model [
A bandit formulation for the assignment of tasks in heterogeneous crowdsourcing is also proposed, which is known as bandit-based task assignment. The worker is represented by the arm of the bandit. It mainly focuses on the selection strategy of workers in the heterogeneous crowdsourcing. The goal is to select the worker that is suitable and good for the task [
The assignment problem of tasks is also explored in a budget constraint with different variety of skill levels and different required quality. An algorithm was also designed for the generation of outcomes for many-to-one matching problem with upper bound and lower bound and the skill level of the worker [
The batch allocation technique is also proposed for the crowdsourcing tasks with the overlapping skill requirements. The designed heuristics approaches include core-based batch allocation and layered batch allocation. The experiment is made on the upwork dataset [
The active time of the worker is also used to get a solution for Multitask Multiworker Allocation. The three factors that the authors consider are the ability of the worker, active time of the worker, and the complexity of task module. The individuals are divided into collaborative groups, and then, for the optimal selection of worker to perform a task, the Hungarian algorithm is used [
As the assignment of task during crowdsourcing activity is a challenge for everyone, in the current study, a mechanism is proposed for task assignment method, which is based on multiple criteria. The study proposed a multicriteria-based task assignment (MBTA) mechanism. Two methods have been used in the current study. The CRITIC method has been used for assigning weights to the selected features and the TOPSIS method is used for ranking the workers. The work done in these methods can also be performed manually, but selecting these methods for doing the proposed work can give authenticity and appropriateness to the research work. Performing work manually contains several chances of mistakes, but doing it with predefined and already experimented methods increases the quality of the work; therefore, these two methods have been selected for the proposed study. The details of the study are discussed in the following sections.
To define the criteria for task assignment, first of all a variety of features have been identified from the existing literature. 33 of the most famous and important features have been identified during literature study. These features are then analyzed and the most important ones are selected for the development of mechanism for task assignment. The weight has then been assigned to these identified features by the CRIRIC method, which is discussed in the next section. The list of identified features is shown in Table
Identified features.
S. no. | Features | Citations |
---|---|---|
1 | Profile management | [ |
2 | Flexibility | [ |
3 | Worker history | [ |
4 | Worker performance history | [ |
5 | Worker task searching history | [ |
6 | Task completion ratio | [ |
7 | Period of time for task | [ |
8 | Participation frequency | [ |
9 | Participation recency | [ |
10 | Winning frequency | [ |
11 | Winning recency | [ |
12 | Tenure | [ |
13 | Reliability | [ |
14 | Worker qualification | [ |
15 | Quality of task | [ |
16 | Knowledge | [ |
17 | Skills/expertise | [ |
18 | Cheap/cost-effective/cost-efficient | [ |
19 | Software worker behavior | [ |
20 | Task similarity | [ |
21 | Delivery time | [ |
22 | Task acceptance ratio | [ |
23 | Accuracy ratio | [ |
24 | Response ratio/quality of response | [ |
25 | Quality of task | [ |
26 | Trustworthy/honesty | [ |
27 | Relevant experience | [ |
28 | Interest | [ |
29 | Reaction time | [ |
30 | Personality type | [ |
31 | Skill level | [ |
32 | Active time | [ |
33 | Development efficiency | [ |
Table 1 shows all the identified features analyzed during systematic literature review. The important features are then selected from the list in order to develop a mechanism.
To complete data collection for making criteria, a case study was performed. In this case study, the issues regarding the assignment of task have been highlighted. All the gaps have been discussed briefly. A comprehensive observation was carried out in order to select the features for criteria from the identified features during literature. The experts were asked different questions in order to select the most important features. These features were then scaled, ranging from 1 to 10, with the help of experts. A group of experts scaled these features so that important features get more weight among the other features, thus making it easy to rank the crowd who have good qualities at the top. After that, these features have been used for analyzing and making criteria as well as for evaluation of the workers to rank them for assigning a task. As all the features have been identified from the existing literature, for further analysis, experts were asked some questions based on these selected features in order to analyze the importance of these features. The questions the experts were asked are shown in Table
Questionnaire.
Q1 | What is the importance of worker history while assigning a task? |
Q2 | What is the importance of trustworthiness in task assignment? |
Q3 | How much worker qualification matters during task assignment? |
Q4 | Is the reliability of the worker important for assigning a task? |
Q5 | What is the role of response ratio in assigning a task? |
Q6 | Does skill level matter for task assignment? |
Q7 | Is the quality of task important for clients? |
Q8 | What is the importance of delivery time in crowdsourcing? |
Q9 | What is the role of cost in assigning a task? |
A list of the selected features is shown in Table
Selected features.
S. no. | Features | Citations |
---|---|---|
1 | Worker history | [ |
2 | Trustworthiness/honesty | [ |
3 | Worker qualification | [ |
4 | Reliability | [ |
5 | Response ratio/quality of response | [ |
6 | Skill level | [ |
7 | Quality of task | [ |
8 | Delivery time | [ |
9 | Cheap/cost-effective/cost-efficient | [ |
The features have been analyzed by experts in the relevant field. Scaling was given to each criterion/feature, ranging from 1 to 10, by these experts in order to get the most important criteria. Weights have been assigned to all these selected features with the help of the CRITIC method. The final weights have been obtained by applying equations (
Measuring standard deviation and its correlation with other criteria for criteria weights.
Sum | Standard deviation | ||
---|---|---|---|
8.888 | 0.332 | 2.953 | 0.114 |
8.879 | 0.313 | 2.778 | 0.107 |
8.286 | 0.389 | 3.226 | 0.125 |
8.690 | 0.314 | 2.731 | 0.105 |
8.768 | 0.359 | 3.149 | 0.122 |
9.070 | 0.281 | 2.549 | 0.098 |
8.830 | 0.359 | 3.172 | 0.122 |
7.810 | 0.334 | 2.611 | 0.101 |
8.087 | 0.337 | 2.728 | 0.105 |
Criteria weights.
Scales of the selected features.
Features | Worker history | Trustworthiness/honesty | Qualification | Reliability | Response ratio/quality of response | Skill level | Quality of task | Delivery time | Cheap/cost-effective/cost-efficient |
---|---|---|---|---|---|---|---|---|---|
Worker 1 | 6 | 7 | 5 | 8 | 4 | 3 | 9 | 7 | 8 |
Worker 2 | 3 | 4 | 8 | 6 | 5 | 7 | 6 | 9 | 7 |
Worker 3 | 9 | 7 | 8 | 5 | 7 | 7 | 6 | 4 | 5 |
Worker 4 | 5 | 7 | 4 | 8 | 9 | 6 | 5 | 8 | 7 |
Worker 5 | 7 | 5 | 3 | 8 | 9 | 6 | 4 | 2 | 6 |
Worker 6 | 4 | 8 | 7 | 5 | 3 | 6 | 9 | 1 | 5 |
Worker 7 | 8 | 5 | 7 | 3 | 2 | 8 | 6 | 2 | 8 |
Worker 8 | 2 | 7 | 7 | 6 | 8 | 9 | 3 | 4 | 5 |
Worker 9 | 7 | 6 | 3 | 1 | 8 | 8 | 9 | 5 | 4 |
Worker 10 | 1 | 7 | 5 | 9 | 6 | 8 | 2 | 4 | 3 |
The weights for each criterion are shown in Figure
The MBTA mechanism has been proposed, which is based on multiple criteria. This mechanism has been developed based upon two methods. The CRITIC method is used to assign weights to the selected features, and then the TOPSIS method is used for ranking the workers. The details are discussed in the following sections.
CRITIC is a type of correlation method which was first introduced in 1995. It is a multicriteria decision-making approach that is used for assigning weights to features or criteria during research work. During this method, the weight is assigned to the criteria objectively rather than by pairwise comparison or decision-makers judgments [
“ Step-1. Building a Decision Matrix A decision matrix “ In equation ( Step-2. Decision Matrix Normalization The process of normalization is done through the following equation: Step-3. Calculating Standard Deviation and Its Correlation In the third step, the weights of the In equation (
Weights are assigned to the criteria using the CRITIC method. The determination of this study was to find the top worker based upon the features for the offered task. The workers that will perform the tasks have been used as alternatives such as
The CRITIC method decision matrix.
Decision matrix | |||||||||
---|---|---|---|---|---|---|---|---|---|
6 | 7 | 5 | 8 | 4 | 3 | 9 | 7 | 8 | |
3 | 4 | 8 | 6 | 5 | 7 | 6 | 9 | 7 | |
9 | 7 | 8 | 5 | 7 | 7 | 6 | 4 | 5 | |
5 | 7 | 4 | 8 | 9 | 6 | 5 | 8 | 7 | |
7 | 5 | 3 | 8 | 9 | 6 | 4 | 2 | 6 | |
4 | 8 | 7 | 5 | 3 | 6 | 9 | 1 | 5 | |
8 | 5 | 7 | 3 | 2 | 8 | 6 | 2 | 8 | |
2 | 7 | 7 | 6 | 8 | 9 | 3 | 4 | 5 | |
7 | 6 | 3 | 1 | 8 | 8 | 9 | 5 | 4 | |
1 | 7 | 5 | 9 | 6 | 8 | 2 | 4 | 3 | |
Best | 9 | 8 | 8 | 9 | 9 | 9 | 9 | 9 | 8 |
Worst | 1 | 4 | 3 | 1 | 2 | 3 | 2 | 1 | 3 |
Steps of the CRITIC method.
Now calculations of the CRITIC method are followed step by step. Table
Table
The CRITIC method normalized decision matrix.
Normalized decision matrix | |||||||||
---|---|---|---|---|---|---|---|---|---|
0.375 | 0.750 | 0.600 | 0.125 | 0.714 | 1.000 | 0.000 | 0.250 | 0.000 | |
0.750 | 0.000 | 0.000 | 0.375 | 0.571 | 0.333 | 0.429 | 0.000 | 0.200 | |
0.000 | 0.750 | 0.000 | 0.500 | 0.286 | 0.333 | 0.429 | 0.625 | 0.600 | |
0.500 | 0.750 | 0.800 | 0.125 | 0.000 | 0.500 | 0.571 | 0.125 | 0.200 | |
0.250 | 0.250 | 1.000 | 0.125 | 0.000 | 0.500 | 0.714 | 0.875 | 0.400 | |
0.625 | 0.000 | 0.200 | 0.500 | 0.857 | 0.500 | 0.000 | 1.000 | 0.600 | |
0.125 | 0.750 | 0.200 | 0.750 | 1.000 | 0.167 | 0.429 | 0.875 | 0.000 | |
0.875 | 0.250 | 0.200 | 0.375 | 0.143 | 0.000 | 0.857 | 0.625 | 0.600 | |
0.250 | 0.500 | 1.000 | 1.000 | 0.143 | 0.167 | 0.000 | 0.500 | 0.800 | |
1.000 | 0.250 | 0.600 | 0.000 | 0.429 | 0.167 | 1.000 | 0.625 | 1.000 | |
Standard deviation | 0.332 | 0.313 | 0.389 | 0.314 | 0.359 | 0.281 | 0.359 | 0.334 | 0.337 |
Measure of conflict has been calculated as shown in Table
Correlation coefficients.
1.000 | −0.655 | −0.116 | −0.462 | −0.020 | −0.238 | 0.452 | −0.206 | 0.357 | |
−0.655 | 1.000 | 0.178 | 0.117 | −0.046 | 0.242 | −0.166 | −0.159 | −0.389 | |
−0.116 | 0.178 | 1.000 | −0.143 | −0.538 | 0.217 | −0.007 | −0.026 | 0.149 | |
−0.237 | 0.117 | −0.143 | 1.000 | 0.213 | −0.424 | −0.529 | 0.241 | 0.073 | |
−0.020 | −0.046 | −0.543 | 0.213 | 1.000 | 0.189 | −0.420 | 0.221 | −0.362 | |
0.000 | 0.242 | 0.217 | −0.424 | 0.189 | 1.000 | −0.503 | −0.266 | −0.523 | |
0.452 | −0.166 | −0.007 | −0.529 | −0.420 | −0.503 | 1.000 | 0.059 | 0.283 | |
−0.206 | −0.159 | −0.026 | 0.241 | 0.221 | −0.266 | 0.059 | 1.000 | 0.325 | |
0.357 | −0.389 | 0.149 | 0.073 | −0.362 | −0.523 | 0.283 | 0.325 | 1.000 |
Standard deviation and its correlation with other criteria have been calculated for criteria weights as shown in Table
Measure of conflict calculations.
0.000 | 1.655 | 1.116 | 1.462 | 1.020 | 1.238 | 0.548 | 1.206 | 0.643 | |
1.655 | 0.000 | 0.822 | 0.883 | 1.046 | 0.758 | 1.166 | 1.159 | 1.389 | |
1.116 | 0.822 | 0.000 | 1.143 | 1.538 | 0.783 | 1.007 | 1.026 | 0.851 | |
1.237 | 0.883 | 1.143 | 0.000 | 0.787 | 1.424 | 1.529 | 0.759 | 0.927 | |
1.020 | 1.046 | 1.543 | 0.787 | 0.000 | 0.811 | 1.420 | 0.779 | 1.362 | |
1.000 | 0.758 | 0.783 | 1.424 | 0.811 | 0.000 | 1.503 | 1.266 | 1.523 | |
0.548 | 1.166 | 1.007 | 1.529 | 1.420 | 1.503 | 0.000 | 0.941 | 0.717 | |
1.206 | 1.159 | 1.026 | 0.759 | 0.779 | 1.266 | 0.941 | 0.000 | 0.675 | |
0.643 | 1.389 | 0.851 | 0.927 | 1.362 | 1.523 | 0.717 | 0.675 | 0.000 |
For each worker, all the 9 features/criteria have been scaled, ranging from 1 to 10, as shown in Table
The TOPSIS approach deals with achieving ideal solutions. This approach has adopted simple computation procedures and thus it is reliable and well established. The selected alternatives in the TOPSIS method should have a minimum distance from positive ideal solution and maximum distance from negative ideal solution [ Decision matrix where As the input data of the decision matrix is originated from several different sources, it has to be converted into a dimensionless matrix by normalization. The comparison between different criteria is done via this dimension matrix. By using formula ( where As it is not necessary that the importances of all attributes will be the same, by multiplying the elements of the normalized decision matrix with random weight number, a weighted normalized decision matrix can be obtained. The weight number for multiplication is given in the following formula: In this step, where By the following formulas, ideal and nonideal separations are calculated. It is determined with respect to the ideal solutions by using the following equation: By using
In this section, evaluation of the workers and their ranking will be obtained based upon 9 identified features by using the TOPSIS method. The data has been collected from different questionaries answered by several experts in relevant fields. The decision matrix is constructed by the data obtained from the panel of experts.
All the work is done step by step as shown in Figure
Steps of the TOPSIS method.
By using equation (
Decision matrix.
Input data | |||||||||
---|---|---|---|---|---|---|---|---|---|
Attributes | Worker history | Trustworthiness/honesty | Qualification | Reliability | Response ratio/quality of response | Skill level | Quality of task | Delivery time | Cheap/cost-effective/cost-efficient |
6 | 7 | 5 | 8 | 4 | 3 | 9 | 7 | 8 | |
3 | 4 | 8 | 6 | 5 | 7 | 6 | 9 | 7 | |
9 | 7 | 8 | 5 | 7 | 7 | 6 | 4 | 5 | |
5 | 7 | 4 | 8 | 9 | 6 | 5 | 8 | 7 | |
7 | 5 | 3 | 8 | 9 | 6 | 4 | 2 | 6 | |
4 | 8 | 7 | 5 | 3 | 6 | 9 | 1 | 5 | |
8 | 5 | 7 | 3 | 2 | 8 | 6 | 2 | 8 | |
2 | 7 | 7 | 6 | 8 | 9 | 3 | 4 | 5 | |
7 | 6 | 3 | 1 | 8 | 8 | 9 | 5 | 4 | |
1 | 7 | 5 | 9 | 6 | 8 | 2 | 4 | 3 |
Decision matrix based on weighted normalization is obtained by using equation (
Normalized decision matrix.
Attributes | Worker history | Trustworthiness/honesty | Qualification | Reliability | Response ratio/quality of response | Skill level | Quality of task | Delivery time | Cheap/cost-effective/cost-efficient |
---|---|---|---|---|---|---|---|---|---|
0.33 | 0.35 | 0.26 | 0.40 | 0.19 | 0.14 | 0.45 | 0.42 | 0.42 | |
0.16 | 0.20 | 0.42 | 0.30 | 0.24 | 0.32 | 0.30 | 0.54 | 0.37 | |
0.49 | 0.35 | 0.42 | 0.25 | 0.34 | 0.32 | 0.30 | 0.24 | 0.26 | |
0.27 | 0.35 | 0.21 | 0.40 | 0.43 | 0.27 | 0.25 | 0.48 | 0.37 | |
0.38 | 0.25 | 0.16 | 0.40 | 0.43 | 0.27 | 0.20 | 0.12 | 0.32 | |
0.22 | 0.39 | 0.37 | 0.25 | 0.14 | 0.27 | 0.45 | 0.06 | 0.26 | |
0.44 | 0.39 | 0.37 | 0.25 | 0.14 | 0.27 | 0.45 | 0.06 | 0.26 | |
0.11 | 0.25 | 0.37 | 0.15 | 0.10 | 0.36 | 0.30 | 0.12 | 0.42 | |
0.38 | 0.35 | 0.37 | 0.30 | 0.39 | 0.41 | 0.15 | 0.24 | 0.26 | |
0.05 | 0.30 | 0.16 | 0.05 | 0.39 | 0.36 | 0.45 | 0.30 | 0.21 | |
Weights | 0.114 | 0.107 | 0.125 | 0.105 | 0.122 | 0.098 | 0.122 | 0.101 | 0.105 |
For alternatives such as
Normalized decision matrix.
Weighted normalized data | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Attributes | Worker history | Trustworthiness/honesty | Qualification | Reliability | Response ratio/quality of response | Skill level | Quality of task | Delivery time | Cheap/cost-effective/cost-efficient | ||
0.037 | 0.037 | 0.033 | 0.042 | 0.023 | 0.013 | 0.055 | 0.042 | 0.044 | 0.050 | 0.078 | |
0.019 | 0.021 | 0.053 | 0.031 | 0.029 | 0.031 | 0.037 | 0.055 | 0.039 | 0.054 | 0.074 | |
0.056 | 0.037 | 0.053 | 0.026 | 0.041 | 0.031 | 0.037 | 0.024 | 0.028 | 0.045 | 0.078 | |
0.031 | 0.037 | 0.026 | 0.042 | 0.053 | 0.027 | 0.030 | 0.049 | 0.039 | 0.080 | 0.080 | |
0.044 | 0.026 | 0.020 | 0.042 | 0.053 | 0.027 | 0.024 | 0.012 | 0.033 | 0.067 | 0.070 | |
0.025 | 0.042 | 0.046 | 0.026 | 0.018 | 0.027 | 0.055 | 0.006 | 0.028 | 0.073 | 0.059 | |
0.050 | 0.042 | 0.046 | 0.026 | 0.018 | 0.027 | 0.055 | 0.006 | 0.028 | 0.066 | 0.071 | |
0.012 | 0.026 | 0.046 | 0.016 | 0.012 | 0.036 | 0.037 | 0.012 | 0.044 | 0.082 | 0.047 | |
0.044 | 0.037 | 0.046 | 0.031 | 0.047 | 0.040 | 0.018 | 0.024 | 0.028 | 0.054 | 0.073 | |
0.006 | 0.032 | 0.020 | 0.005 | 0.047 | 0.036 | 0.055 | 0.030 | 0.022 | 0.078 | 0.061 | |
Positive ideal | 0.056 | 0.042 | 0.053 | 0.042 | 0.053 | 0.040 | 0.055 | 0.055 | 0.044 | ||
Negative ideal | 0.006 | 0.021 | 0.020 | 0.005 | 0.012 | 0.013 | 0.018 | 0.006 | 0.022 |
Positive ideal solution and negative ideal solution are used for finding ideal and nonideal separation measures. These separation measures are calculated by using equations (
Ranking is done upon the value of
Ranking of workers.
Workers | Rank | ||||
---|---|---|---|---|---|
0.050 | 0.078 | 0.128 | 0.6087 | 2 | |
0.054 | 0.074 | 0.129 | 0.5776 | 3 | |
0.045 | 0.078 | 0.123 | 0.6353 | 1 | |
0.080 | 0.080 | 0.159 | 0.5000 | 6 | |
0.067 | 0.070 | 0.137 | 0.5085 | 7 | |
0.073 | 0.059 | 0.132 | 0.4481 | 8 | |
0.066 | 0.071 | 0.137 | 0.5180 | 5 | |
0.082 | 0.047 | 0.129 | 0.3650 | 10 | |
0.054 | 0.073 | 0.127 | 0.5758 | 4 | |
0.078 | 0.061 | 0.140 | 0.4389 | 9 |
From Table
Workers’ ranking.
As the figure shows, workers
Assigning the task to the most appropriate worker is very important in crowdsourcing because if the task is assigned to an inappropriate worker it affects crowdsourcing activity in several ways such as waste of time, money, and clients trusts. The proposed research presents a mechanism for assigning a task to the worker. This proposed mechanism is based on multiple criteria. Worker features such as worker history, trustworthiness/honesty, worker qualification, reliability, response ratio/quality of response, skill level, quality of task, delivery time, and cheap/cost-effective/cost-efficient are selected by the identified features. Two MCDM methods, CRITIC and TOPSIS, have been used. Weights have been assigned to these features by the CRITIC method and then evaluation and ranking of the workers have been analyzed by the TOPSIS method in order to assign the task to the most appropriate worker. As the existing task assignment is based on single criterion, the proposed work is novel in terms of assigning workers based on multiple criteria as well as using MCDM methods for current work in crowdsourcing.
No data were used to support this study.
The authors declare that there are no conflicts of interest.