Mathematical Problems in Engineering
Volume 2018 (2018), Article ID 2814897, 12 pages
https://doi.org/10.1155/2018/2814897

Research Article

Feature Selection and Overlapping Clustering-Based Multilabel Classification Model

Liwen Peng and Yongguo Liu

Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China

Correspondence should be addressed to Yongguo Liu; liuyg_cn@163.com

Received 4 June 2017; Revised 2 November 2017; Accepted 10 December 2017; Published 4 January 2018

Academic Editor: Bogdan Dumitrescu

Copyright © 2018 Liwen Peng and Yongguo Liu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Multilabel classification (MLC) learning, which is widely applied in real-world applications, is a very important problem in machine learning. Some studies show that a clustering-based MLC framework performs effectively compared to a nonclustering framework. In this paper, we explore the clustering-based MLC problem. Multilabel feature selection also plays an important role in classification learning because many redundant and irrelevant features can degrade performance and a good feature selection algorithm can reduce computational complexity and improve classification accuracy. In this study, we consider feature dependence and feature interaction simultaneously, and we propose a multilabel feature selection algorithm as a preprocessing stage before MLC. Typically, existing cluster-based MLC frameworks employ a hard cluster method. In practice, the instances of multilabel datasets are distinguished in a single cluster by such frameworks; however, the overlapping nature of multilabel instances is such that, in real-life applications, instances may not belong to only a single class. Therefore, we propose a MLC model that combines feature selection with an overlapping clustering algorithm. Experimental results demonstrate that various clustering algorithms show different performance for MLC, and the proposed overlapping clustering-based MLC model may be more suitable.

1. Introduction

The multilabel classification (MLC) problem, which is applicable to a wide variety of domains, such as music classification and bioinformatics [1], has received increasing attention. However, situations where single instances are associated with multiple labels remain challenging. Most algorithms treat such MLC tasks as multiple binary classification tasks. However, this approach may not consider potential correlations among features and labels.

A good MLC solution must be effective and efficient; however, a large number of redundant and irrelevant attributes may increase computational costs and the time required to learn and test a multilabel classifier, which reduces classification performance. Feature selection, which is an important technique in data mining and machine learning, has been widely used in classification models to enhance performance. Selecting features before applying classification methods to original datasets has many advantages, such as refining the data, reducing computational costs, and improving classification accuracy [2, 3]. Therefore, we utilise a feature selection algorithm to improve the quality of MLC.

Various feature selection methods have been proposed, for example, statistics, rough set methods, information gain, and mutual information (MI). A wide variety of research has shown that no single feature selection method can handle all situations. Many studies have demonstrated that MI-based feature selection methods are effective and efficient because the MI can handle different types of attributes, does not make any assumptions, and can measure nonlinear relations between variables [4]. Recently, many algorithms to select significant features for MLC have been proposed. However, most of these methods do not consider that a single attribute may affect various labels differently. The concept of interaction information has become more relevant because it can reflect the relevance, redundancy, and complementarity among attributes and labels; thus, it is an effective feature selection method. In this study, we propose an algorithm to improve MLC performance by selecting significant attributes based on interaction information between attributes and labels.

Some studies have shown that clustering-based MLC methods can improve predictive performance and reduce time costs; however, those studies used nonoverlapping clustering methods to handle multilabel datasets. We know that, in MLC, one object may belong to multiple classes; however, algorithms based on nonoverlapping clustering, that is, hard division methods, do not consider such situations. In contrast, overlapping clustering-based methods consider this situation when they handle datasets. Therefore, we propose an overlapping clustering-based MLC (OCBMLC) model.

The remainder of this paper is organised as follows: Section 2 describes related work, Section 3 provides background information, Section 4 describes the proposed multilabel feature selection algorithm and MLC model, Section 5 introduces experimental data, evaluation criteria, and experimental results, and conclusions and suggestions for future work are presented in Section 6.

2. Related Work

Currently, a variety of algorithms have been developed to handle MLC problems [5–15]. In traditional classification methods, each instance has a single label; however, in MLC, an instance can have more than one label. MLC algorithms can be divided into problem transformation methods (PT) and algorithm adaptation methods (AA) [9].

PT methods convert multilabel data to single-label data; thus, a single-label classification method can be used. Label powerset, binary relevance [10], and random ensemble learning with k-label sets [11] are classic PT methods. The AA approaches extend single-label algorithms to process multilabel data directly. BP-MLL [12] and ML-KNN are two popular AA methods. BP-MLL is a widely used MLC backpropagation algorithm. An important characteristic of this algorithm is the introduction of an error function that considers multiple labels. The ML-KNN AA method [13] determines the labels of a new object using the maximum a posteriori principle. The ML-KNN algorithm obtains a label set based on the statistical information of the label sets of the -nearest neighbours of a test instance.

Many studies have proven that redundant and irrelevant features can increase computational costs, reduce performance, and result in overfitting. These problems also exist in MLC. Many feature selection methods have been proposed to handle these problems and improve MLC. Battiti [14] proposed the Mutual Information Feature Selection algorithm, which selects the maximum relevance term, to address these problems. Peng et al. [15] introduced an improvement algorithm called Minimal-Redundancy and Max-Relevance, and Lin et al. [16] proposed a multilabel feature selection algorithm that combines MI with max-dependency and min-redundancy. In addition, over the past few years, unsupervised, clustering, and other technologies have been used to reduce dimensionality. For example, Li et al. [17] proposed a clustering-guided sparse structural learning algorithm that integrates clustering and a sparse structure in a united framework to select the most useful features. They also proposed an algorithm [18] that employs nonnegative spectral clustering and controls the redundancy between features to select significant features. Cai et al. [19] presented the Unified Sparse Subspace Learning (USSL) framework, which employs a dimension reduction technique that incorporates a subspace learning method. The USSL framework has demonstrated good performance. Li et al. [20] proposed the Robust Structured Subspace Learning (RSSL) framework that combines subspace learning theory and features learning. Their experimental results demonstrated that the RSSL framework performed well for image understanding tasks.

Recently, Kommu et al. [21] proposed two methods based on probabilistic theory to solve multilabel learning problems. In the first method, their algorithm uses logistic regression and a nearest neighbour classifier for MLC. Note that Partial Information is used in this approach. In the second method, their algorithm deals with the concept of grouping related labels. Association Rules are also introduced in the second approach. Guo and Li [22] proposed the Improved Conditional Dependency Networks framework for MLC. This method uses label correlations in the training stage and CDNs in the testing stage. Yu et al. [23] used a rough sets approach for MLC that considers the associations between labels. They evaluated the performance of their approach using seven multilabel datasets.

Nasierding et al. [24, 25] presented an effective CBMLC framework that combines a clustering algorithm with an MLC algorithm. Various clustering methods, such as -means, EM, and Sequential Information Bottleneck, are used for training. Note that, with this framework, labels are ignored during training phase. Nasierding et al. [26] compared clustering and nonclustering MLC methods for image and video annotations. Tahir et al. [27] proposed a method that combines a multilabel learning approach with fusion techniques. They used various multilabel learners to select a label set and demonstrated that ensemble techniques can avoid the disadvantages of different learners.

3. Background Theory

3.1. Entropy and Mutual Information

In this section, we introduce the theories of Entropy and MI. Here, we assume that all variables are discrete or data attributes can be discretised using different discrete methods. Shannon’s entropy [28] is the uncertainty measure of a random variable, and it has been widely used in various domains. Here, let be a discrete variable and be the probability density function. Formally, the entropy of is defined as follows:

Assume that and are two random discrete variables. is the joint probability of and . The joint entropy is defined as follows:

If the value of random variable is known and variable is not, the remaining uncertainty of variable can be measured by the conditional entropy defined as follows:

The minimum value of is zero when random variable is statistically dependent on random variable . The maximum conditional entropy value occurs when the two variables are statistically independent.

The relationship between conditional and joint entropy can be defined as follows:

MI is the amount of information shared by two variables and is defined as follows:

Note that the two random variables are statistically independent when is zero. The relation between MI and entropy can be defined as follows:

Let be a random variable . The conditional MI and joint MI represent the information of two variables in the context of a third variable and are defined as follows:

Multi-information, which was introduced by McGill [29], is an extension of two random variables that can handle the interaction among more than two random variables. Mathematically, multi-information is defined as follows:

Multi-information can be positive, negative, or zero [30]. If the multi-information value is zero, the random variables are independent in the context of the third variable. If the value is negative, the variables have redundant information and a positive value indicates that together the random variables can provide more information than each variable taken individually.

3.2. Overlapping Clustering Algorithm

Fuzzy -Means (FCM) algorithms are widely used in fuzzy clustering learning. Fuzzy clustering, which is a type of overlapping clustering, differs from hard clustering. The FCM clustering algorithm assigns data points (examples) to a cluster, and the fuzzy membership of data points indicates the extent to which data points pertain to their clusters [31].

Suppose is a set of vectors for clustering. Vectors represent the attributes of the object . Here, a fuzzy partition matrix is defined as

where .

Note that examples can belong to more than one cluster with different degrees of membership. The object function of the FCM algorithm obtains the minimum value as follows [32]:

where is the membership degree matrix, parameter is the weight exponent that defines the fuzziness of the resulting clusters, and is the Euclidian distance between object and the cluster centre .

The objective function is minimized by updating the partition matrix and cluster centre as follows:

The FCM membership function is defined as follows:

Here, is the membership value of the th object and th cluster, is the number of clusters, and is the cluster centre of the th cluster.

4. Proposed Multilabel Classification Model

4.1. Proposed Multilabel Feature Selection Method

In information theory, is the original feature set, and the subset is the compact feature subset where . The selected subset should maximise the joint information between the subset with compact dimension and the class label .

Such a method is impractical because it is difficult to calculate the high-dimensional probability density function. Therefore, some efficient methods have been proposed to approximate the ideal solution [14–16]. Generally, most multilabel feature selection methods based on MI consider the relevance and redundancy terms. In practice, such methods and their variants calculate the MI between a candidate feature and the selected features subset; however, they do not sufficiently consider interaction information among attributes and class labels, ignore feature cooperation, and allow all features to be competitive.

We know that a candidate feature for multilabel feature selection should have one of the highest MI values for all class labels. This is referred to as the relevance term. Multilabel feature relevance terms have been defined previously, and we use the following definition.

Definition 1. Let denote a candidate feature and be a class label. The relevance term is expressed as follows:

We can obtain two properties according to this definition.

Property 2. If candidate feature and each class label are mutually independent, then the MI of and is minimum.

Property 3. If each class label is determined completely by , then the MI of and is maximum.

According to the above properties, we can use Definition 1 to select relevant candidate features. However, classes combined with previously selected features may produce interaction. Therefore, we should consider the interaction information among the candidate feature, the selected features, and the classes during feature selection. Differing from existing feature selection methods, we consider the interaction information between a feature and a single class and the pairwise interaction between features and all class labels. Our interaction metric is defined as follows:

Here, is the selected features subset, denotes the label set, and denotes the candidate feature.

It is well known that multilabel feature selection attempts to select a set of features with the highest discrimination power for all labels. According to the above discussion, we combine (14) and (15) using the feature interaction maximum of the minimum criteria to propose a new goal function (referred to as max-dependence and interaction (MDI)) for multilabel feature selection. Here, the candidate features are considered to have the highest relevance and beneficial interaction with all class labels. The proposed MDI goal function is expressed as follows:

With this function, the first term is the relevance between the candidate features and all class labels, and the second term focuses on the interaction information among , , and . The proposed goal function can select features with the greatest discrimination power. The pseudocode of the proposed algorithm is as Pseudocode 1.

(Initialisation) Set “initial set of features”;

“empty set’’.

(Computation of the MI with the output class set)

For compute

(Choice of the first feature) Find the feature that

maximizes ; set ; set .

(Greedy selection) Repeat until ;

(selection of the next feature) choose the feature

;

set ; set ;

(output) Output the set with the selected features.

Pseudocode 1

4.2. Proposed Multilabel Classification Model

There are some experimental results that show that CBMLC methods can improve the predictive classification performance and reduce algorithm training time compared to existing popular multilabel methods [24–26]. The results of those models show that the classification performance of clustering-based methods is effective. However, those algorithms were used for nonoverlapping clustering methods, such as EM and -means, prior to MLC. Therefore, the original data will be set into several disjoint data clusters in nonoverlapping methods.

Clustering methods are usually classified into hard clustering and fuzzy clustering. In hard clustering, instances are distinguished in a single cluster. However, due to the overlapping nature of instances, generally, they do not belong to only a single class in real-world applications. This property limits the practical application of hard clustering, especially for MLC.

FCM is an effective classic fuzzy clustering method based on an objective function concept and is widely used in clustering. The FCM approach uses alternating optimisation strategies to solve nonlinear and nonmonitor clustering problems. We know that one instance may own multiple classes in multilabel data, and the FCM algorithm can handle one instance that belongs to more than one cluster simultaneously. This allows the use of a fuzzy clustering method that assigns a single object to several clusters. Therefore, we propose the OCBMLC model in combination with the FCM algorithm to improve performance. Figure 1 shows the basic procedure of the proposed OCBMLC model.

Figure 1: Basic procedure of OCBMLC model.

5. Experiments and Results

5.1. Datasets

In our experiments, we used three public multilabel datasets, that is, the emotions, yeast, and scene datasets. These datasets were taken from the Mulan Library. The emotions dataset contains examples of songs according to people’s emotions [33]. The yeast dataset includes information about genes functions [34], and the scene [35] dataset includes a series of landscape patterns. Table 1 shows the statistics of the three multilabel benchmark datasets.

Table 1: The statistic of the multilabel benchmark datasets.

Dataset
Domain
Instances
Features
labels
Cardinality

Emotions
Music
593
72
6
1.869

Yeast
Biology
2417
103
14
4.237

Scene
Image
2407
294
6
1.074

In Table 1, “Domain” denotes the dataset domain, “Instances” is the number of instances in the dataset, “Features” is the number of attributes, “Labels” is the number of labels in the datasets, and “Cardinality” is the average number of labels associated with each instance.

5.2. Experimental Setting

At the multilabel feature selection stage, in order to calculate MI convenience, we discretise continuous features into 10 bins using an equal-width strategy. The evaluation approaches for MLC differ from traditional single-label classifications. Note that the Hamming loss and micro -measure evaluation criteria are widely used for MLC; thus, we used these criteria in our experiments.

Note that non-OCBMLC models use -means and EM algorithms to cluster original datasets, and OCBMLC model uses the FCM algorithm on the data after dimension reduction. The overlapping and nonoverlapping frameworks both employ ML-KNN as the classifier. The number of clusters in -means, EM, and FCM is all set between 2 to 7. In this study, a cross-validation strategy was used for each combination of algorithm framework and dataset. All experiments used MATLAB 2012 on an Intel Core-i5 2.3 GHz processor with 8 GB memory.

5.3. Evaluation Metrics

The evaluations of an MLC system differ from that of a single-label classification system. Note that some criteria that evaluate the performance of an MLC system have been employed previously [36]. Among such evaluation metrics, we employed Hamming loss and the micro -measure criteria.

Here, let be a set of n test examples and be the predict label set for the test instance . is the ground truth label set for . The Hamming loss indicates the number of erroneous labels to the total number of labels, where a smaller Hamming loss value indicates better classification performance. The Hamming loss value is calculated as follows:

The micro -measure represents the harmonic means between precision and recall, and it is calculated from false positives, false negatives, true positives, and true negatives. The -measure and microaveraging are evaluated as follows:

Here, denotes true positives and denotes false positives, and and are true and false negatives, respectively, for labels after a separate binary evaluation is performed. Note that a greater micro -measure value indicates better classification performance of a multilabel algorithm.

5.4. Results

In this study, we used Hamming loss and the micro -measure as experimental evaluation metrics and employed ML-KNN as the multilabel classifier. Note that, in all cases, we indicate the best results in bold values in Tables 2, 3, 4, 5, 6, 7, 8, and 9.

5.4.1. Comparisons of Feature Selection Methods

To demonstrate the efficacy of the proposed feature selection algorithm, we compared the proposed feature selection method to other MLC models based on clustering using the emotion dataset. We also compared a feature selection method that only considers the dependence between features and classes using the proposed algorithm in which interaction information among features and classes is considered. Here, we refer to the criterion that considers only dependence as the max-dependence criterion, where . This criterion was used to select candidate features.

In this experiment, “DEP_max” represents the features selected by the max-dependence criterion, which ignores interaction information when selecting candidate features, and “MDI” represents features selected by the proposed algorithm, which considers dependence and interaction information among the candidate features and each class simultaneously. Here, we selected the top features for MLC according to the MDI and DEP_max criteria, and we used the average Hamming loss and micro -measure values based on the selected top features subset by comparing the values from the original feature sets.

Table 2 shows the Hamming loss values obtained when we used the features selected according to MDI, DEP_max, and the original features from the emotion dataset, and Table 3 shows the micro -measure values when we employed features selected according to MDI, DEP_max, and the original features. In terms of the feature selection methods, we found that the performance of DEP_max is no better than that of the other models even though we used the original feature subset. However, the MDI performance is better and more stable when the clustering number is 2 to 7. It is likely that features selected by only considering Max-Relevance could generate abundant redundancy, which means that the dependence among those features could be large. Therefore, the proposed feature selection function may be better suited for MLC, and we observed the same from the experimental results.

Table 2: Hamming loss measure for the models based on -means clustering on emotions dataset. is the number of attributes, and is the amount of clusters. is the average value of Hamming loss.

Classification model

Emotions

-means + ML-KNN
2
72
0.2018

EM + ML-KNN
2
72
0.2019

DEP_max + -means + ML-KNN
2
%
0.2021

MDI + -means + ML-KNN
2
%
0.2016

-means + ML-KNN
3
72
0.2021

EM + ML-KNN
3
72
0.2013

DEP_max + -means + ML-KNN
3
%
0.2034

MDI + -means + ML-KNN
3
%
0.1986

-means + ML-KNN
4
72
0.2106

EM + ML-KNN
4
72
0.2030

DEP_max + -means + ML-KNN
4
%
0.2156

MDI + -means + ML-KNN
4
%
0.2024

-means + ML-KNN
5
72
0.2119

EM + ML-KNN
5
72
0.2027

DEP_max + -means + ML-KNN
5
%
0.2168

MDI + -means + ML-KNN
5
%
0.2005

-means + ML-KNN
6
72
0.2089

EM + ML-KNN
6
72
0.2030

DEP_max + -means + ML-KNN
6
%
0.2078

MDI + -means + ML-KNN
6
%
0.2056

-means + ML-KNN
7
72
0.2120

EM + ML-KNN
7
72
0.2064

DEP_max + -means + ML-KNN
7
%
0.2126

MDI + -means + ML-KNN
7
%
0.2030

Table 3: Micro -measure for all models on emotions dataset. is the number of attributes, and is the amount of clusters. is the average value of micro -measure.

Classification model

Emotions

-means + ML-KNN
2
72
0.6538

EM + ML-KNN
2
72
0.6585

DEP_max + -means + ML-KNN
2
%
0.6572

MDI + -means + ML-KNN
2
s%
0.6590

-means + ML-KNN
3
72
0.6562

EM + ML-KNN
3
72
0.6614

DEP_max + -means + ML-KNN
3
%
0.6532

MDI + -means + ML-KNN
3
%
0.6650

-means + ML-KNN
4
72
0.6375

EM + ML-KNN
4
72
0.6523

DEP_max + -means + ML-KNN
4
%
0.6350

MDI + -means + ML-KNN
4
%
0.6635

-means + ML-KNN
5
72
0.6427

EM + ML-KNN
5
72
0.6571

DEP_max + -means + ML-KNN
5
%
0.6430

MDI + -means + ML-KNN
5
%
0.6590

-means + ML-KNN
6
72
0.6411

EM + ML-KNN
6
72
0.6596

DEP_max + -means + ML-KNN
6
%
0.6423

MDI + -means + ML-KNN
6
%
0.6621

-means + ML-KNN
7
72
0.6380

EM + ML-KNN
7
72
0.6549

DEP_max + -means + ML-KNN
7
%
0.6382

MDI + -means + ML-KNN
7
%
0.6576

5.4.2. Comparisons of Multilabel Classification Models

The results obtained by the MLC models with the emotions dataset relative to Hamming loss and the micro -measure are shown in Tables 4 and 5. We selected the top in the selected feature subset as the final feature subset for use with the proposed model. Table 4 demonstrates that the proposed OCBMLC framework achieved the lowest Hamming loss value () with the emotions dataset. Table 5 shows that the proposed framework achieved the highest micro -measure () with the emotions dataset. As shown in Figures 2 and 3, the predictive performance of the proposed model achieved the best results with the emotions dataset when . As shown in Figures 2 and 3, respectively, the Hamming loss demonstrates the minimum value and the micro -measure demonstrates the maximum value when we used the MDI feature selection method to select the top features as the classification attributes subset.

Table 4: Hamming loss measure for all models on emotions dataset. is the number of attributes, and is the amount of clusters.

Classification model

Emotions

-means + ML-KNN
2
72

EM + ML-KNN
2
72

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
72

EM + ML-KNN
3
72

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
72

EM + ML-KNN
4
72

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
72

EM + ML-KNN
5
72

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
72

EM + ML-KNN
6
72

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
72

EM + ML-KNN
7
72

MDI + FCM + ML-KNN
7
20%

30%

40%

Table 5: Micro -measure for all models on emotions dataset. is the number of attributes, and is the amount of clusters.

Classification model

Emotions

-means + ML-KNN
2
72

EM + ML-KNN
2
72

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
72

EM + ML-KNN
3
72

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
72

EM + ML-KNN
4
72

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
72

EM + ML-KNN
5
72

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
72

EM + ML-KNN
6
72

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
72

EM + ML-KNN
7
72

MDI + FCM + ML-KNN
7
20%

30%

40%

Figure 2: Hamming loss for all models and the number of clusters in emotions.

Figure 3: Micro for all models and the number of clusters in emotions.

To demonstrate the classification performance of the proposed model, we also selected the top in the selected feature subset as an experimental feature subset. The Hamming loss and micro -measure results of the MLC model with the yeast dataset are shown in Tables 6 and 7. As shown in Figures 4 and 5, the Hamming loss and micro -measure demonstrate the best results when with 40% of the features selected from the original data attributes. In addition, it was found that the evaluation criterion value of MLC was reduced with an increasing number of clusters.

Table 6: Hamming loss measure for all models on yeast dataset. is the number of attributes, and is the amount of clusters.

Classification model

Yeast

-means + ML-KNN
2
103

EM + ML-KNN
2
103

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
103

EM + ML-KNN
3
103

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
103

EM + ML-KNN
4
103

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
103

EM + ML-KNN
5
103

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
103

EM + ML-KNN
6
103

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
103

EM + ML-KNN
7
103

MDI + FCM + ML-KNN
7
20%

30%

40%

Table 7: Micro -measure for all models on yeast dataset. is the number of attributes, and is the amount of clusters.

Classification model

Yeast

-means + ML-KNN
2
103

EM + ML-KNN
2
103

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
103

EM + ML-KNN
3
103

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
103

EM + ML-KNN
4
103

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
103

EM + ML-KNN
5
103

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
103

EM + ML-KNN
6
103

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
103

EM + ML-KNN
7
103

MDI + FCM + ML-KNN
7
20%

30%

40%

Figure 4: Hamming loss for all models and the number of clusters in yeast.

Figure 5: Micro measure for all models and the number of clusters in yeast.

Tables 8 and 9 show that the OCBMLC model achieved the top predictive performance (Hamming loss = ; micro ) with the scene dataset. Figures 6 and 7 show that the Hamming loss and micro -measure values outperformed the “EM and ML-KNN” and “-means and ML-KNN” models when and 30% of the features of the original data attributes were selected for the scene dataset. Thus, we conclude that the proposed OCBMLC model outperforms the other classification models.

Table 8: Hamming loss measure for all models on scene dataset. is the number of attributes, and is the amount of clusters.

Classification model

Scene

-means + ML-KNN
2
294

EM + ML-KNN
2
294

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
294

EM + ML-KNN
3
294

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
294

EM + ML-KNN
4
294

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
294

EM + ML-KNN
5
294

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
294

EM + ML-KNN
6
294

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
294

EM + ML-KNN
7
294

MDI + FCM + ML-KNN
7
20%

30%

40%

Table 9: Micro -measure for all models on scene dataset. is the number of attributes, and is the amount of clusters.

Classification model

Scene

-means + ML-KNN
2
294

EM + ML-KNN
2
294

MDI + FCM + ML-KNN
2
20%

30%

40%

-means + ML-KNN
3
294

EM + ML-KNN
3
294

MDI + FCM + ML-KNN
3
20%

30%

40%

-means + ML-KNN
4
294

EM + ML-KNN
4
294

MDI + FCM + ML-KNN
4
20%

30%

40%

-means + ML-KNN
5
294

EM + ML-KNN
5
294

MDI + FCM + ML-KNN
5
20%

30%

40%

-means + ML-KNN
6
294

EM + ML-KNN
6
294

MDI + FCM + ML-KNN
6
20%

30%

40%

-means + ML-KNN
7
294

EM + ML-KNN
7
294

MDI + FCM + ML-KNN
7
20%

30%

40%

Figure 6: Hamming loss for all models and the number of clusters in scene.

Figure 7: Micro measure for all models and the number of clusters in scene.

When we selected the top 30% or 40% of features using the proposed feature selection algorithm for MLC, the proposed OCBMLC model achieved the best performance because it can select features with max-dependence in consideration of the classes and interaction among features and each class. Thus, the proposed feature selection algorithm can select features with the best discrimination power. The experimental results prove that the proposed feature selection algorithm directly improves classification performance, and it is almost always better than models that use all total features from the data. The experimental results also show that the model based on overlapping clustering outperforms models based on hard clustering. In a multilabel dataset, one instance may belong to multiple labels; however, hard clustering methods attempt to assign a single instance to a single label. Therefore, such methods may not be suitable for multilabel datasets. In contrast, overlapping clustering methods consider situations where single instances do in fact belong to multiple classes.

6. Conclusion

This paper has proposed an overlapping clustering-based MLC model that includes a feature selection phase for original datasets. We have also proposed a new multilabel feature selection algorithm that can effectively select significant features to improve classification performance. The proposed MLC framework includes an initial overlapping clustering phase. The proposed model considers the fact that multilabel data examples may not be related to a single class but may belong to multiple classes in many cases. Therefore, overlapping clustering may be more suitable for such situations. Experimental results show that the proposed model can increase predictive performance compared to a nonoverlapping clustering framework. In addition, the results demonstrate that feature selection plays an important role in classification. In future, we plan to further explore and develop a better and more robust feature selection method or overlapping clustering algorithm for MLC tasks.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this article.

Acknowledgments

This research was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 60903074 and the National High Technology Research and Development Program of China (863 Program) under Grant 2008AA01Z119.

References

C. Vens, J. Struyf, L. Schietgat, S. Džeroski, and H. Blockeel, “Decision trees for hierarchical multi-label classification,” Machine Learning, vol. 73, no. 2, pp. 185–214, 2008.

J. Lee and D.-W. Kim, “Fast multi-label feature selection based on information-theoretic feature ranking,” Pattern Recognition, vol. 48, no. 9, pp. 2761–2771, 2015.

J. Lee and D. W. Kim, “Memetic feature selection algorithm for multi-label classification,” Information Sciences, vol. 293, pp. 80–96, 2015.

P. E. Meyer, C. Schretter, and G. Bontempi, “Information-theoretic feature selection in microarray data using variable complementarity,” IEEE Journal of Selected Topics in Signal Processing, vol. 2, no. 3, pp. 261–274, 2008.

K. Feng, J. Rong, and R. Sukthankar, “Correlated label propagation with application to multi-label learning,” in Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2006, pp. 1719–1726, USA, June 2006.

G. Tsoumakas, I. Katakis, and I. Vlahavas, “Effective and efficient multilabel classification in domains with large number of labels,” Ecml/pkdd Workshop on Mining Multidimensional Data, 2008.

D. Mena, E. Montañés, J. R. Quevedo, and J. J. del Coz, “An overview of inference methods in probabilistic classifier chains for multilabel classification,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 6, no. 6, pp. 215–230, 2016.

G. Tsoumakas and I. Vlahavas, “Random k-Labelsets: An Ensemble Method for Multilabel Classification,” in Machine Learning: ECML, 2007.

G. Tsoumakas, I. Katakis, and I. Vlahavas, “Data mining and knowledge discovery handbook,” Kybernetes, vol. 33, no. 7, pp. 809–835, 2010.

K. Brinker and E. Hullermeier, “Case-based multi-label ranking,” in Proceeding of the Conference on Artificial Intelligence (IJCAI’07), pp. 702–707, India, 2007.

G. Tsoumakas, I. Katakis, and I. Vlahavas, “Random k-labelsets for multilabel classification,” IEEE Transactions on Knowledge and Data Engineering, vol. 23, no. 7, pp. 1079–1089, 2011.

M.-L. Zhang and Z.-H. Zhou, “Multilabel neural networks with applications to functional genomics and text categorization,” IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 10, pp. 1338–1351, 2006.

M.-L. Zhang and Z.-H. Zhou, “ML-KNN: a lazy learning approach to multi-label learning,” Pattern Recognition, vol. 40, no. 7, pp. 2038–2048, 2007.

R. Battiti, “Using mutual information for selecting features in supervised neural net learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 5, no. 4, pp. 537–550, 1994.

H. Peng, F. Long, and C. Ding, “Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1226–1238, 2005.

Y. Lin, Q. Hu, J. Liu, and J. Duan, “Multi-label feature selection based on max-dependency and min-redundancy,” Neurocomputing, vol. 168, pp. 92–103, 2015.

Z. C. Li, J. Liu, Y. Yang et al., “Clustering-guided sparse structural learning for unsupervised feature selection,” IEEE Transactions on Knowledge & Data Engineering, vol. 26, no. 9, pp. 2138–2150, 2014.

Z. Li and J. Tang, “Unsupervised feature selection via nonnegative spectral analysis and redundancy control,” IEEE Transactions on Image Processing, vol. 24, no. 12, pp. 5343–5355, 2015.

D. Cai, X. He, and J. Han, “Spectral regression: a unified approach for Sparse Subspace Learning,” in Proceedings of the 7th IEEE International Conference on Data Mining (ICDM '07), pp. 73–82, Omaha, Neb, USA, October 2007.

Z. C. Li, J. H. Tang et al., “Robust structured subspace learning for data representation,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 37, no. 10, pp. 2085–2098, 2015.

G. R. Kommu, M. Trupthi, and S. Pabboju, “A novel approach for multi-label classification using probabilistic classifiers,” in Proceedings of the 2014 International Conference on Advances in Engineering and Technology Research, ICAETR 2014, India, August 2014.

T. Guo and G. Li, “Improved Conditional Dependency Networks for Multi-label Classification,” in Proceeding of the International Conference on Measuring Technology & Mechatronics Automation, pp. 561–565, 2015.

Y. Yu, W. Pedrycz, and D. Miao, “Neighborhood rough sets based multi-label classification for automatic image annotation,” International Journal of Approximate Reasoning, vol. 54, no. 9, pp. 1373–1387, 2013.

G. Nasierding, G. Tsoumakas, and A. Z. Kouzani, “Clustering based multi-label classification for image annotation and retrieval,” in Proceedings of the 2009 IEEE International Conference on Systems, Man and Cybernetics, SMC 2009, pp. 4514–4519, USA, October 2009.

G. Nasierding and A. Sajjanhar, “Multi-label classification with clustering for image and text categorization,” in Proceedings of the 2013 6th International Congress on Image and Signal Processing, CISP 2013, pp. 869–874, China, December 2013.

G. Nasierding, Y. Li, and A. Sajjanhar, “Robustness comparison of clustering - Based vs. non-clustering multi-label classifications for image and video annotations,” in Proceedings of the 8th International Congress on Image and Signal Processing, CISP 2015, pp. 691–696, China, October 2015.

M. A. Tahir, J. Kittler, K. Mikolajczyk et al., “Improving multilabel classification performance by using ensemble of multi-label classifiers,” in Proceedings of the International Conference on Multiple Classifier Systems, International Workshop, DBLP, pp. 11–21, 2010.

T. M. Cover and J. A. Thomas, Elements of Information Theory, John Wiley & Sons, New York, NY, USA, 1991.

W. J. McGill, “Multivariate information transmission,” Psychometrika, vol. 19, no. 2, pp. 97–116, 1954.

M. Bennasar, Y. Hicks, and R. Setchi, “Feature selection using Joint Mutual Information Maximisation,” Expert Systems with Applications, vol. 42, no. 22, pp. 8520–8532, 2015.

J. Nayak, B. Naik, and H. Behera, “Fuzzy C-means (FCM) clustering algorithm: a decade review from 2000 to 2014,” in Computational Intelligence in Data Mining—Volume 2, L. C. Jain, H. S. Behera, J. K. Mandal, and D. P. Mohapatra, Eds., vol. 32 of Smart Innovation, Systems and Technologies, pp. 133–149, Springer, 2015.

K. Zou, Z. Wang, and M. Hu, “An new initialization method for fuzzy -means algorithm,” Fuzzy Optimization and Decision Making. A Journal of Modeling and Computation Under Uncertainty, vol. 7, no. 4, pp. 409–416, 2008.

K. Trohidis, G. Tsoumakas, G. Kalliris, and I. Vlahavas, “Multi-label classification of music into emotions,” in Proceedings of the International Symposium on Music Information Retrieval, pp. 325–330, September 2008.

A. Elissee and J. Weston, “A kernel method for multi-labelled classification,” in Proceedings of the International Conference on Neural Information Processing Systems, pp. 681–687, 2001.

M. R. Boutell, J. Luo, X. Shen, and C. M. Brown, “Learning multi-label scene classification,” Pattern Recognition, vol. 37, no. 9, pp. 1757–1771, 2004.

G. Tsoumakas, I. Katakis, and I. Vlahavas, “A Review of Multi-Label Classification Methods,” in Proceedings of the International Conference on the 2nd ADBIS Workshop on Data Mining and Knowledge Discovery, pp. 99–109, 2010.