Paddy Crop and Weed Discrimination: A Multiple Classifier System Approach

Weeds are unwanted plants that grow among crops. +ese weeds can significantly reduce the yield and quality of the farm output. Unfortunately, site-specific weedmanagement is not followed in most of the cases.+at is, instead of treating a field with a specific type of herbicide, the field is treated with a broadcast herbicide application.+is broadcast application of the herbicide has resulted in herbicide-resistant weeds and has many ill effects on the natural environment.+is has prompted many research studies to seek the most effective weed management techniques. One such technique is computer vision-based automatic weed detection and identification. Using this technique, weeds can be detected and identified and a suitable herbicide can be recommended to the farmers.+erefore, it is important for the computer vision technique to successfully identify and classify the crops and weeds from the digital images. +is paper investigates the multiple classifier systems built using support vector machines and random forest classifiers for plant classification in classifying paddy crops and weeds from digital images. Digital images of paddy crops and weeds from the paddy fields were acquired using three different cameras fixed at different heights from the ground. Texture, color, and shape features were extracted from the digital images after background subtraction and used for classification. A simple and newmethod was used as a decision function in the multiple classifier systems. An accuracy of 91.36% was obtained by the multiple classifier systems and was found to outperform single classifier systems.


Introduction
Computer vision often abbreviated as CV is defined as the process of analyzing images and videos automatically to obtain meaningful inference or measurements without human intervention. It is a multidisciplinary field that involves artificial intelligence (AI) and machine learning concepts. e goal of computer vision is to understand the content of digital images and videos [1]. is involves developing methods and techniques which attempt to reproduce the human capability of cognition and recognition. Computer vision applications are being extensively used in the agricultural domain for the discrimination of crops and weeds, plant classification, crop disease identification, and so forth. e automatic weeding machine was proposed in [2], which was based on computer vision techniques [3]. Automation of various agricultural tasks such as disease detection in crops, precise spraying of pesticides, prediction of crop yield, estimation of soil texture, automatic grading of fruits, estimation of crop biomass, management of water balance in the irrigation system, and monitoring plant growth has been done using computer vision techniques [4,5]. In [6], diseased plant leaves were identified using image processing and soft computing techniques. In [7], discrimination of corn crop and weed species was done using the C5 algorithm based on textural and spectral features. Wireless sensor networks and wireless visual sensor networks are contributing to sending the sensed data of the field either in image or in text form to the remote machine where it will be processed and analyzed for some kind of decision-making [8]. In [9], a wireless sensor network was used to predict the occurrence of bud necrosis virus in groundnut crops. In [10], a wireless sensor network was implemented to predict the water requirement in semi-arid regions. Soft computing techniques along with wireless sensor network technology are helping farmers and agronomists to make the right decision at the right time. A precision agriculture model resulted due to the advances in wireless sensor networks and computer vision techniques [11]. is model can help to reorganize the entire farming system with low input, high efficiency, and sustainable farming.
Weeds can be defined as undesirable plants growing with crops. Generally, they are referred to as plants out-of-space. ey compete with the crops for soil nutrients and water. Here, the term competition can be referred to as weed plants nourishing at the cost of rice [12]. is competition usually does not result in the death of the rice crop but definitely leads to reduced yield. e problem caused by weeds and the importance of weed management is not recognized by the general population and sometimes even by agriculturists. One of the main reasons for agriculturists to ignore weeds is high cost involving manual weeding and expensive herbicides. Other than competing with rice plants for soil nutrients, water, and space, they also act as alternate hosts to various pests, which in turn attack crops and destroy them. Weeds are the main reason for heavy yield losses and sometimes are responsible for complete crop failure [13]. It is reported that annually India is incurring a loss of INR 1050 million because of weeds in paddy fields [14]. Classification of weeds in rice fields is usually based on their life cycle, their habitat, and their gross morphological features [15]. ey are broadly categorized as sedges, grass, and broadleaved weed [16], as shown in Figures 1(a)-1(c), respectively [17,18]. e multiple classifier system (MCS) is a way of using many classifiers to make a final decision in a classification process. e ensemble of different classifiers has been used lately in pattern recognition to improve the performance and aims at increasing the accuracy of the single classifier system [19]. e idea is that two or more diverse classifiers when grouped help in negating the errors made by the individual classifiers [20]. ere are two types of decision functions that are commonly used in the design of MCSs. ey are classifier fusion (combination-based) and classifier selection (selection-based) [21,22]. In classifier fusion, the outputs of several classifiers in the MCS or ensemble are fused to obtain the final decision. Some of the classifier fusion techniques are majority voting, sum rule, product rule, and so forth [23]. In classifier selection, one classifier's output is selected from the pool of classifier members. An example of the classification selection method is cross-validation accuracy (CVM). e classifier selection method can be static or dynamic. CVM is a static classifier selection method. In [24], Kuncheva has proposed clustering-based dynamic classifier selection methods.
In [25], an MCS was proposed for the classification of leaves from the digital images based on texture and shape features. Heterogeneous classifiers are made of SVM and neural network classifiers with different texture features such as local binary patterns, histogram features (HoG), and Zernike moments. e dataset was obtained from ImageCLEF. Results show that improvement relative to the classification score was reported in the literature for the ImageCLEF-2011 and ImageCLEF-2012 datasets.
In [26], an SVM-based MCS was developed to recognize diseased wheat leaves. e MCS was designed as a two-level structure.
ree SVM classifiers were trained with color features, texture features, and shape features separately. At level-0, the output of the individual three classifiers was fed as training data to the level-1 classifiers. is way the errors made by level-0 classifiers are compensated and the direct mapping limitation at level-0 was solved.
In [23], a review was made about the different topology of MCSs. Also, different types of decision functions are explained that can be used when implementing the MCS. e paper gives insight for the developers working on MCSs.
In [27], a review is made about MCSs as hybrid systems for information fusion. e paper explains the different types of hybrid systems built using the MCS. In addition, this paper explains different combinatorial functions to fuse the results.
MCSs have been used widely in the field of biometrics. Research favors the solutions where information about the iris, voice, fingerprints as well as features are provided through a combination of information rather than information generated from a single source. MCSs are also used in document analysis, remote sensing data analysis systems, medical decision support systems, and so forth [28]. However, MCSs are less explored in the plant classification domain. In this study, MCSs were designed to classify paddy crops and weeds from digital images. e main contributions of this study are as follows: (i) Creation of a digital image dataset of paddy crops and weeds. e dataset contains around 1500 images taken under natural varying lighting conditions. e images were acquired using Sony Cybershot (DSC-W220), Canon PowerShot SD3500 IS, and Raspberry Pi (RP-v5647) cameras. e dataset also contains annotated images. (ii) Two selection-based MCSs were designed: one with calibrated RF classifier and calibrated SVM classifier called MCS-1; another with uncalibrated RF classifier and uncalibrated SVM called MCS-2. e performance of two MCSs is compared and analyzed. (iii) A new dynamic classifier selection method is proposed.

Dataset Creation.
Two digital cameras (Canon Power-Shot SD3500 IS) and Sony Cybershot (DSC-W220) were used to acquire images from paddy fields around Manipal region of Karnataka state in India. Images were acquired under natural variable lighting conditions with the camera fixed at different heights such as 0.61 m, 0.91 m, and 1.22 m from the ground. e cameras were fixed on a tripod and were facing down towards the ground. Images were also acquired from the Raspberry Pi (RP ov5647) camera installed in the paddy field. e weeds and crops of varying canopy sizes were selected for acquisition to increase the difficulty of identification. e dataset contains paddy plants and weeds from the early growth stage (1-leaf seedling stage) to the flowering stage. ese growth stages are very critical because the effect of weed will be maximum [29]. Both transplanted and direct-seeded rice fields were selected for acquisition. e images were stored in RGB color space in JPG format. e MATLAB R2018a was used to process the images. While working in MATLAB, images were resized to 1296 X 966. e dataset consists of 1517 images. Figure 2(a) shows an image acquired from the Canon camera fixed at a distance of 0.61 m from the ground.  Figure 6(a) shows the image acquired by the Raspberry Pi camera and Figure 6(b) is its corresponding background subtracted image.

Feature Extraction and Feature
Selection. e soil and water background was removed to retain only green vegetation. Connected component algorithm and successive erosion and dilation [30] were used to remove possible overlapping and to extract individual plants for feature extraction as shown in Figures 7 and 8. e images with too heavy overlapping were not considered and left out from classification. Color features were extracted using the methods explained in [31]. Two different types of shape features were extracted, namely, size independent features and Hu's moments using the method as explained in [8]. Texture features were extracted using the Laws' texture masks [32] using the method explained in [33]. Ninety-three features were extracted. From these, seventy-one best-performing features were selected using the feature selection method based on analysis of variance (ANOVA) [34,35]. Table 1 gives information about the number of samples classwise used in the classification task.

Design of Multiple Classifier Systems.
Multiple classifier systems (MCSs) can be built using two or more same types of classifiers or different types of classifiers whose outputs are then fused to determine the final class. e individual classifiers can be trained on the same data, or with different data partitions or data with different feature subsets. If the participating classifiers are trained with different partitions, then techniques such as bagging and boosting can be used [36]. Since the decision support system is built on either homogenous or heterogeneous models, MCSs are the subcategory of hybrid intelligent systems.

Diversity Metric.
MCSs can show improved performance than single classifier systems provided very diverse classifiers be used in the design of MCSs [37]. erefore, before designing the MCS, the diversity metric was calculated for different pairs of classifiers. ere are many diversity metrics such as entropy, double fault, correlation, and Yule's statistic [38]. In this study, Yule's statistic was used to calculate the diversity of two classifiers based on the pairwise diversity metric [39]. If the value of Yule's statistic is nearer to zero, then the classifiers are more diversified. e Yule's statistic Q is given by where a, b, c, and d are given in Table 2.
Steps in finding Yule's Statistic: (1) Take sample data (300) and split into training data and test data (2) Fit the two models on training data. From Table 3, the combination of RF and SVM has the lowest Q i,j . e value of a for this combination was 0.8334. erefore, these two classifiers were selected to design a selection-based MCS. In selection-based MCSs, for each input sample (test), the output of one classifier among many classifiers will be selected dynamically based on a selection mechanism. Since heterogeneous classifiers were used in this study, it is important to evaluate the quality of individual classifiers and use them for a proper classifier selection. erefore, the weights of each class for each classifier was obtained. To do this, the whole dataset was divided into two sets called Train set-1 and Test set in the ratio 80 : 20. Trainset-1 was again divided into two sets in the ratio 80 : 20 as Train set-2 and Weight set. After fitting the classifiers with Train set-2 data, it was tested by Weight set. e weight for each class of each classifier was obtained by dividing the number of Weight set samples belonged to each class that was correctly classified by the classifier by the total number of Weight set samples.
is procedure is summarized in Algorithm 1. For the final class selection, the probability of the sample belonging to each class is taken and summed with the corresponding weight.
e class that has the highest value is selected as the final class. e final class selection by MCS-2 is summarized in Algorithm 2.

Designing of MCS with Calibrated Classifiers
(1) Calibrating SVM and RF Classifiers. ere are many ways of calibrating the classifiers. e most commonly used methods are platt scaling and isotonic regression [40,41].
is type of scaling is used when the distortion in the predicted probabilities is in the sigmoid shape. e outputs of the classification system (scores) are passed through the sigmoid function, where parameters   Class 3 (sedges) 421 Table 2: Reference for pairwise diversity metrics, from [39].
Classifier j is correct Classifier j is incorrect Classifier i is correct a b Classifier i is incorrect c d is fitting is done by gradient descent [42] to reduce where p i � 1/1 + (exp(A × f + B)).
(3) Isotonic Regression. Let f i be the prediction from the classifier and the corresponding actual target be y i , then the isotonic regression can be defined as where m is a nondecreasing function. e isotonic regression will show superior performance when there are large data (beyond 1000). In this study, isotonic regression was used for calibrating the classifiers. Cross-validation is helpful to solve the problem of overfitting. In this research work, 5-fold cross-validation was used. In this, the base estimator (RF and SVM) was fit on the train set of the cross-validation generator, and the test set is used for calibration. e probabilities for each of the folds are then averaged for prediction. Algorithm 3 summarizes the design of the MCS using calibrated classifiers. Table 4 shows the multiclass confusion matrix, and the diagonal element gives you the true positives (TP) of respective classes. Here, we have a four-class confusion matrix. According to [43], to evaluate the performance of the classifier in a multiclass classification case, for each separate class C i , TP i , true negative (TN i ), false positives (FP i ), false negative (FN i ), Accuracy i , Recall i , and Specificity i can be calculated from the counts, count i from each class C i . e performance of the classifier is calculated in two ways, one using macroaveraging and another microaveraging. In the case of macroaveraging, an evaluation parameter is the average of the same parameter, but in the case of microaveraging, cumulative sum of counts to get the cumulative values of TP, TN, FP, and FN is obtained, and then evaluation parameters are calculated. In this study, macroaveraging was used. e calculation of evaluation parameters for multiclass classification is as shown in Table 5 where l is the number of classes.

Evaluation Using Area Under Curve.
e receiver operating characteristics (ROC) curve is used in machine learning to depict the performance of a classifier visually. It is a graph of the false-positive rate (FPR) against the truepositive rate (TPR). e area under the ROC curve commonly referred to as AUC is another measure used to assess the performance of the given classifier [44,45]. It gives us the discriminative ability of the given classifier. at is, it gives us the probability with which the classifier will rank a randomly chosen positive instance. For example, if we get an AUC value as 0.8, then it means that a randomly chosen positive instance has a higher score than for a negative instance 80% of the time. If most of the time classifier cannot clearly distinguish between the groups, we have AUC as 0.5, for a random classifier. For the best classifier, we have AUC as 1. us, this area gives us the predictive accuracy of a classifier model. Higher the value of AUC, the better the model is. Figures 9  and 10 show AUC for MCS-1 and MCS-2. e value of AUC for all the class types in both the MCSs is above 0.95 indicating the excellent performance of the MCSs.
Cohen's kappa [46,47], calculated for different partitions of the train and test data, is shown in the box plot in Figure 11. Tables 6-8 show the classification results obtained by MCS-1, MCS-2, and single classifiers (RF and SVM), respectively. e following are the observations made from this study. International Journal of Agronomy (1) Accuracy of 91.36% is obtained in classifying paddy crops and three types of weeds. Even though it is difficult to compare the result of this study with other related works due to a different field and other conditions, three studies [48][49][50] have been selected for comparison. Even though the method described in [50] achieved better results when compared to this study, the work was carried out on a single image, and we believe that, it cannot be considered as a model for rice and weed discrimination. In [49], the work has been performed to detect the rice plant based on rice ears, while the proposed method works for different crop and weed sizes. e work carried out in [48] achieved a slightly better result than this study, but it is computationally intensive and resource-intensive.

Case Study
e methodology of feature extraction and classification of this research work was applied to the rice and weed dataset created by [48]. Table 9 shows the classification result obtained on this dataset.
e result in Table 9 shows that the feature set extracted has the potential for effective feature vector representation of paddy crop and weed images for the classification process.
(1) Input (2) Train set-1 Train data set with class labels (3) Test-set Test data (4) k Number of classes (5) n Number of classifiers (6) W [n] [k] Weight of each class of each classifier (7) Output (8) Y [ ] Class labels (9) len ⟵ length of Test set (10) Y [ ] ⟵ NULL (11) for i ⟵ 1 to n do (12) Train C i on Train set-1 (13) end for (14) for i ⟵ 1 to len do (15) p Predict probability of classes for sample S i using C1, where S i Є Test-set (16) q Predict probability of classes for sample S i using C2, where S i Є Test-set (17) end for (18) for i ⟵ 1 to len do (19) for j ⟵ 0 to k − 1 do (20) p end if (34) if q[i][j] > max2 then (35) max2 q[i][j] (36) index2 j (37) end if (38) end for (39) if max1 > max2 then (41) Y  International Journal of Agronomy  (11) for i ⟵ 1 to n do (12) Fit calibrated C i using isotonic regression on Train set-1 (13) end for (14) for i ⟵ 1 to len do (15) p Predict probability of classes for sample S i using C1, where S i Є Test-set (16) q Predict probability of classes for sample S i using C2, where S i Є Test-set (17) end for (18) for i ⟵ 1 to len do (19) for j ⟵ 0 to k − 1 do (20) p end if (34) if q[i][j] > max2 then (35) max2 ⟵ q[i][j] (36) index2 ⟵ j (37) end if (38) end for (39) if max1 > max2 then (40) : International Journal of Agronomy      (3) e misclassification of paddies is found to be more in MCS-1 and it is misclassified as grass. is could be attributed to the fact that features of paddies and grass are more or less the same [52,53]. Hence, the chances of the true probability values of paddy class (class 2) matching with grass class (class 1) are largely resulting in misclassification. is could be reduced by increasing the number of samples of grass weeds with different growth sizes. (4) Kappa between two MCSs around 0.929 indicates that two MCSs agree with each other greatly. (5) e extracted features are good enough to discriminate paddy crops and weeds. e method is device-independent. is is the first dataset of paddy crops and three types of weeds to the best knowledge of authors.

Conclusion
In this study, two MCSs were designed and assessed for the classification of paddy crops and weeds from the digital images. e approach was first to create digital images of paddy crops and weeds from paddy fields using digital cameras, and the Raspberry Pi camera was fixed at different heights from the ground to make the method device-independent. e soil and water background was removed. Texture, color, and shape features were extracted. Two selection-based MCSs were designed, one with calibrated random forest and calibrated SVM classifiers called as MCS-1 and another MCS with uncalibrated random forest classifier and uncalibrated SVM classifier called as MCS-2. MCS-1 and MCS-2 outperformed the single classifier systems. In addition, it was found that the MCS designed with calibrated classifiers performed slightly better than the MCS designed with uncalibrated classifiers. e features extraction and classification process proposed in this research work were applied on a publicly available paddy crop and weed dataset and results obtained are very promising. e study showed that the extracted features are good enough to classify paddy crops and weeds. is work could be used to recommend suitable herbicide for a particular type of weed based on the classification results to avoid broadcast application of the herbicides. is could lead to the reduction of herbicide-resistant weeds, contamination of the groundwater, and other ill effects of overuse of the herbicides.
In the next phase of this research, we intend to expand the paddy crop and weed digital image dataset. We intend to design different types of MCSs for the classification of paddy crops and weeds and analyze the performance. In addition, we intend to carry out extended research to develop a general crop and weed discrimination model using a selection based-MCS.
Data Availability e datasets are not made public yet. However, interested readers can get the sample images upon request from the corresponding author through e-mail: mamtha.bc@ manipal.edu.