A Support Vector Classifier Based on Vague Similarity Measure

Support vector machine (SVM) is a popular machine learning method for its high generalizaiton ability. How to find the adaptive kernel function is a key problem to SVM from theory to practical applications.This paper proposes a support vector classifer based on vague sigmoid kernel and its similarity measure. The proposed method uses the characteristic of vague set, and replaces the traditional inner product with vague similaritymeasure between training samples.The experimental results show that the proposed method can reduce the CPU time and maintain the classification accuracy.


Introduction
Support vector machine (SVM) constructs a hyperplane or set of hyperplanes in a high-dimensional feature space, which can be used for classification, regression, or other tasks.It was first introduced by Vapnik [1] for classification and has been applied to many application fields successfully.SVM is based on the structural risk minimization principle, which incorporates capacity control to prevent overfitting and thus is a partial solution to the bias-variance trade-off dilemma.The basic idea of SVM classification is to find such a separating hyperplane that corresponds to the largest possible margin between points of different classes.
How to find the adaptive kernel function is a key problem to SVM from theory to practical applications.There are several kernel functions, for example, the linear kernel, the polynomial kernel, and the RBF kernel.These kernel functions are positive semidefinite (PSD).However, some non-PSD matrices are used in practice.An important one is the sigmoid kernel (  ,   ) = tanh(     + ), which is related to neural networks.It was first pointed out by Vapnik [1] that sigmoid kernel matrix might not be PSD for the certain values of parameters,  and .However, the sigmoid kernel matrix is conditionally positive definite in certain parameters and thus it is valid kernel.
Meanwhile, datasets in real life are usually noisy, and a classifier which is obtained by training with noisy data cannot classify some data samples correctly.So, fuzzy theory is introduced in support vector machines by many researchers in order to solve the above problem.There exist two cases.The first one is fuzzy support vector machine (FSVM) [2][3][4].FSVM takes into account the noisy in the training set and associates a fuzzy membership with every sample which will account for the uncertainty in the class to which it belongs.It uses the membership function to express the membership grade of a sample belonging to positive class or negative class.The second one is to combine fuzzy theory to kernel functions of SVM and propose a fuzzy kernel-based SVM.Fuzzy kernel is apt to construct a robust classifier and process classification and regression of uncertain or fuzzy data.Soria-Olivas et al. [5] propose a fuzzy-based activation function for artificial neural networks.Camps-Valls et al. [6] extend the fuzzy-based activation function and propose a support vector classifier based on fuzzy sigmoid kernel.The fuzzy sigmoid function allows lower computational cost and higher rate of positive eigenvalues of the kernel matrix than those from the standard sigmoid kernel [6].Yang et al. [7] develop a kernel fuzzy c-means clustering-based fuzzy SVM algorithm to deal with the classification problems with outliers or noises.
In fuzzy theories, the vague set theory [8] is one of the methods used to deal with uncertain information and has gradually become more and more popular for handling decision-making problems.Since vague sets can provide more information than fuzzy sets, they are considered superior in mathematical analysis of uncertain information.This paper combines vague sets with sigmoid kernel and proposes a novel support vector classifier based on vague sigmoid kernel.
The rest of this paper is organized as follows: Section 2 reviews the related research and briefly describes the vague set theory and support vector machine.We present a novel support vector classifier based on vague sigmoid kernel and its similarity measure in Section 3. Section 4 presents the experimental results obtained on benchmark data sets and analyzes the performance of the proposed algorithm.Section 5 concludes the paper with some final remarks.

Vague Set and Support Vector Machine
2.1.The Conception of Vague Set.Fuzzy set theory was first proposed by Zadeh [9].It is an important mathematical approach to uncertain and fuzzy data analysis and has successfully been applied in the areas of fuzzy control, fuzzy decision making, and so on.
Introduced by Gau and Buehrer [8], vague set is a generalization of the concept of a fuzzy set.Note that the vague set is the same with the intuitionistic fuzzy set in essence according to some research work [10].The major advantage of vague set over fuzzy set is that the former makes descriptions of the objective world more realistic, practical, and accurate.Presently, many scholars have been interested in the theory and already made further studies.They have been widely applied in medical diagnosis, decision making, pattern recognition, uncertain knowledge acquisition, and so forth [11][12][13][14].
Definition 1 (vague sets [8]).Let  be the universe of discourse,  = { 1 ,  2 , . . .,  −1 ,   }, with a generic element of  denoted by   .A vague set  in  is characterized by a truth-membership function   and a false-membership function   , where When the universe of discourse  is continuous, a vague set  can be written as When the universe of discourse  is discrete, a vague set  can be written as Let   (  ) be an uncertain degree of   in vague set A,   (  ) = 1 −   (  ) −   (  ).  (  ) characterizes the precision of our knowledge about   .If   (  ) is small, our knowledge about   is relatively precise; if it is large, we know correspondingly little.If 1 −   (  ) is equal to   (  ), our knowledge about   is exact, and the theory reverts back to that of fuzzy sets.If both 1 −   (  ) and   (  ) are equal to 1 or 0, our knowledge about   is very exact, and the theory reverts back to that of ordinary sets.
For example, let  be a vague set with truth-membership function   and false-membership function   , respectively.If a vague value is [0.5, 0.8], then according to Definition 1 we can see that   (  ) = 0.5,   (  ) = 1 − 0.8 = 0.2, and   (  ) = 1 − 0.5 − 0.2 = 0.3.It can be interpreted as "assume that the total number of the votes is 10, the votes for a resolution is 5 in favor, 2 against, and 3 abstentions." Obviously, fuzzy set cannot exactly denote and process the type of obscure information.
Many similarity measures have been proposed in the literature for measuring the degree of similarity between vague sets.Chen [15,16] proposed the concept of similarity measures between vague sets and defined its expression   (X,Y) as follows:   (, ) = 1 − |(() − ())/2|, where  and  are two vague values, () =   −   , and () =   −   .It is obvious that the larger the value of   (, ), the more the similarity between the vague values  and .Hung and Yang [11] presented three new similarity measures between intuitionistic fuzzy sets based on Hausdorff distance.Li et al. [17] analyzed and summarized several similarity measures between vague sets.Dou et al. [18] developed a new similarity measure of vague sets and defined the new relative degree of similarity measures to solve the fuzzy shortest path problem.

Support Vector Machine and Its Kernel.
In this section, we briefly review the learning algorithm of support vector machine (SVM) initially proposed in [1].Given a binary classification problem represented by a dataset {( 1 ,  1 ), ( 2 ,  2 ), . . ., (  ,   )}, where   ⊂ R  represents an n-dimensional data sample and   ∈ {+1, −1} represents the class of that data sample, for  = 1, . . ., , the goal of the SVM learning algorithm is to find an optimal hyperplane that separates these data samples into two classes.In order to find a better separation of classes, the data are first transformed into a higher-dimensional feature space by a mapping function .Then, a possible separating hyperplane, which resides in the higher-dimensional feature space, can be represented by  ⋅  () +  = 0. (4) The support vector technique requires the solution of the following optimization problem: subject to the constraints Mathematical Problems in Engineering 3 where the training vectors   are mapped into a higherdimensional space by the function  parameter  is a user-specified positive parameter that controls the trade-off between maximizing the margin and minimizing the training error term.The slack variables   > 0 hold for misclassified samples, and therefore, ∑  =1   can be thought of as a measure of the amount of misclassifications.This quadratic optimization problem can be solved by constructing a Lagrangian representation and transforming it into the following dual problem: subject to the constraints where   is the Lagrangian parameter.Note that the kernel trick (  ,   ) = (  ) ⋅ (  ) is used in the last equality in (7).The Karush-Kuhn-Tucker conditions of SVM are defined by The sample   with the corresponding nonzero   is called a support vector.The optimal value of weight vector  0 is obtained by , where   is the number of support vectors.The optimal value of bias  0 can be computed from the Karush-Kuhn-Tucker conditions (9); namely,  0 =   −  0 ⋅ (  ), for a random support vector sample   .Once the optimal pair ( 0 ,  0 ) is determined, the SVM decision function is then given by where (  ,   ) is called the kernel function as follows: Several typical kernel functions are the linear kernel (  ,   ) =   ⋅  , the polynomial kernel (  ,   ) = (     + )  , and the RBF kernel (  ,   ) = exp(−‖  −   ‖ 2 ).
The kernel functions above must satisfy Mercer condition.Namely, the kernel function matrix is a symmetric and positive semidefinite (PSD) matrix.Nevertheless, some non-PSD matrices are used in SVM in practice [19].The sigmoid kernel (  ,   ) = tanh(     + ) is an available non-PSD kernel function.The sigmoid kernel is also known as the hyperbolic tangent kernel and as the multilayer perceptron (MLP) kernel, which comes from the neural networks field.
It was first pointed out by Vapnik [1] that its kernel matrix might be non-PSD for certain values of the paramenters  and .When the kernel function is non-PSD, (11) cannot be satisfied.H. T. Lin and C. J. Lin [19] also study non-PSD kernel function and its applications to SVM and testify the sigmoid kernel matrix is conditionally positive definite (CPD).When parameters  > 0 and  < 0, the sigmoid kernel is suitable for a valid kernel.The sigmoid kernel has been used in several practical cases, such as support vector machine classification [6,19], decision rules extraction [20], and chaotic time series prediction [21].

The Proposed Support Vector Classifier Based on Vague Similarity Measure
This section presents the proposed method for vague sigmoid kernel-based support vector classifier.It first gives a brief introduction to fuzzy kernel and then focuses on a proposed algorithm.
3.1.Fuzzy Kernel.Several researchers have studied fuzzy kernel.Kwan [22] proposes a simple sigmoid-like nonlinear activation function more suitable for digital hardware implementation as follows: where  is the width of the transition region.
Inspired by the work of Kwan, Soria-Olivas et al. [5] think that the activation function of ( 12) can be drawn in a more natural way by defining the classical activation function by means of the fuzzy logic methodology and propose a fuzzy-based activation function for artificial neural networks, which considers triangular functions due to their simplicity.The fuzzy-based sigmoid function models the hyperbolic tangent function by means of linguistic variables.Camps-Valls et al. [6] extend the fuzzy-based activation function and propose a support vector classifier based on fuzzy sigmoid kernel.The fuzzy sigmoid kernel allows lower computational cost and higher rate of positive eigenvalues of the kernel matrix, which alleviates current limitations of the sigmoid kernel.
Although fuzzy set theory can preferably characterize fuzziness, it has an obvious shortage due to using a singlevalue membership   (  ) ∈ [0, 1] to represent the degree of membership.Fuzzy theory lacks consideration of some nondeterministic factors among samples.In this case, we propose a fuzzy sigmoid kernel support vector classifier based on vague theory.

Vague Value Computation.
The proposed algorithm considers two-class classification for simplicity.We first decide the class center of each class, namely,  *  and  *  in Figures 1-3, and then compute vague values of samples.For the sample  in the training set, if  belongs to class A but does not belong to the intersection area between class  and class B, we define vague value of  as follows: where   is the radius of class .This case is shown in Figure 1.
Similarly, let   be the radius of class B, we get ‖ −  *  ‖/  >1.If  belongs to class B but does not belong to the intersection area between class  and class B, we define vague value of  as follows: where   is the radius of class .This case is shown in Figure 2. We get ‖ −  *  ‖/  >1.If sample  belongs to the intersection area, we label   and   as   = ‖− *  ‖/  and   = ‖− *  ‖/  and define vague value of  as follows: This case is shown in Figure 3. Through a detailed analysis of samples in class  and class B, we find   () = 1 −   () shown in Figures 1 and  2, and vague set reverts back to fuzzy set in two cases.For sample  in Figure 3, we reduce the classification effect of these samples.

Similarity Measure of Vague Value and Its Kernel.
For samples   and   , their inner   ⋅   is obtained typically by computing Euclidean distance between two samples.In this paper, we replace Euclidean distance with similarity measures between vague sets after introducing vague membership.A similarity measure is used for estimating the degree of similarity between two sets.The main idea is described below.We first compute vague values of samples and then represent these vague values with points in the spatial coordinate system.At last, we compute similarity measures between points.
As shown in Definition 2,   in the spatial coordinate system includes three parts, (  ), (  ), and (  ), respectively.The meanings of (  ), (  ), and (  ) are shown in Definitions 1 and 2. Analyzing from the vote model, we consider that some abstentions are likely prone to be in favor, others are likely prone to be against, and others are likely to be abstention.So, we further divide abstention part into three parts: (  )(  ), (  )(  ), and  2 (  ), which represent the cases of being in favor, against, and abstention in all abstentions, respectively.We can use a point in threedimensional space to depict a membership degree of a training sample.
Obviously, (  )+(  )+(  ) = 1.Using Definition 3, we can get a point in three-dimensional space   .For three parts in Definition 3. Let   and   be two points defined as Definition 2; then their similarity measures can be defined as follows: Based on vague value and similarity measure above, we give a computation method of vague sigmoid kernel function.Expression (12) can be readily rewritten as a function of  and  as follows: where  = (  ,   ) + /.

The Proposed Vague Sigmoid-Based Support Vector Classifier.
In order to compute vague values of training samples, we first decide the class center of each class.In this paper, we select fuzzy c-means (FCM) method [24,25] to do it.In fuzzy clustering, FCM method has become one of the most popular techniques.
FCM algorithm starts with an initial guess for the cluster centers, which intends to mark the mean location of each cluster [24,25].The initial guess for these cluster centers is most likely incorrect.Additionally, FCM assigns every data point a membership grade for each cluster.By iteratively updating the cluster centers and the membership grades for each data point, FCM iteratively moves the cluster centers to the "right" location within a data set.This iteration is based on minimizing an objective function that represents the distance from any given data point to a cluster center weighted by that data point's membership grade.Namely, where  is the number of clusters and selected as a specified value in this paper, l is the number of data points,   ∈ [0, 1] denotes the degree to which the sample   belongs to the th cluster, m is the fuzzy parameter controlling the speed and achievement of clustering, (  , V  ) = ‖  − V  ‖ 2 denotes the distance between point   and the cluster center V  , and  is the set of cluster centers or prototypes (V  ∈   ).
When the objects change clusters, the membership values are recalculated according to the following formula: Each cluster center is then calculated by After getting the cluster centers, the algorithm can compute vague values of training sample and measure the similarity of vague values.
According to analysis above, we propose a novel vague sigmoid-based support vector classifier.The algorithm description is as follows.
Step 1. Preprocess data set and classify training data set and testing data set.
Step 2. Use FCM algorithm to compute the membership   of each training sample   to each class k and to obtain the cluster centers.Substep 2.1.Select the number of clusters c, the maximal iterative count  max , fuzziness parameter m (let  = 2) and converge error  > 0. Substep 2.2.Initialize the membership matrix  when | () (, ) −  (−1) (, )| <  or  =  max , stop iteration and return the membership matrix  () and the cluster centers V ()  .
Step 4. Use SVM based on vague sigmoid kernel to train samples with vague values.
The key steps of the proposed algorithm are to compute vague values and to compute vague sigmoid kernel (  ,   ) = tanh(     + ).

Experimental Analysis
Six data sets from the University of California at Irvine (UCI) machine learning repository [26] are used in our experiments.The data sets include Ionosphere, Sonar, Pimadiabetes, Wdbc, Iris and Vehicle.Iris, and Vehicle data are multiclass problems.The other data are binary classification problems.The characteristics of these data sets are shown in Table 1.We tested the proposed vague sigmoid kernel method and compared it to sigmoid-based SVM and fuzzy sigmoidbased SVM.The results of these methods depend on the values of the kernel parameters  and r and penalization parameter C. In order to test the proposed methods, SVM models were trained by using LIBSVM [27].Parameter  was fixed to 1/A, where  is the input dimension of data set, and other parameters of all methods were optimized using gridbased 5-fold cross-validation method.For all the datasets each training set and each testing set were the same for all methods.For all the datasets, we used the 5-fold crossvalidation method to estimate the accuracy of the classifiers.We compare classification accuracy and CPU time (s) for the sigmoid, the fuzzy-based sigmoid, and the vague-based sigmoid kernels.Experimental results are shown in Table 2.
From Table 2, we can find that the sigmoid-based method achieves the better accuracy than the fuzzy sigmoid method and the vague sigmoid method on Ionosphere, Sonar, Iris, and Vehicle data sets.As we can see, the accuracy of the fuzzy sigmoid method and the vague sigmoid method is better than that of the sigmoid-based method on Pima-diabetes and Wdbc data sets.However, the sigmoid-based method has also more CPU times than the other methods.Notice that the vague sigmoid method achieves the same or slightly better accuracy than the fuzzy sigmoid method.
If we consider the cost of accuracy and CPU time, we may prefer the solution found with the vague sigmoid method.We can find that the average CPU time in the all 6 data sets is 22.32 s in the sigmoid method, while there is 19.77 s in the vague sigmoid method; in the meantime, the classification accuracy does not decrease remarkably.

Conclusions
Support vector machine is a novel machine learning method which has been applied to many application fields successfully.The sigmoid kernel was quite popular for support vector machines due to its origin from neural networks.In this paper, we propose a vague sigmoid kernel-based support vector classifier.The proposed method is combined with vague set methodology, which makes the computation of SVM simple.In vague sigmoid kernel, we replace the inner product computation using Euclidean distance between two samples with similarity measures.The experiment is conducted by using 6 data sets from the UCI machine learning repository.The results of classification are evaluated and compared in terms of the performance using accuracy and time.The results obtained from the experiment indicated that the proposed method can reduce the CPU time and maintain the classification accuracy.

Figure 1 :Figure 2 :
Figure 1: Vague value computation in class A.

Figure 3 :
Figure 3: Vague value computation in intersection area between class  and class B.
(  ) is a lower bound on the grade of membership of   derived from the evidence for   ,   (  ) is a lower bound on the negation of   derived from the evidence against   , and   (  ) +   (  ) ≤ 1.It is clear that the grade of membership of   in the vague set  has been restricted in a subinterval[  (  ), 1 −   (  )] of [0, 1].The subinterval [  (  ), 1 −   (  )] is called vague value of   in vague set A.The vague value [  (  ), 1−  (  )] indicates that the exact grade of membership   (  ) of   may be unknown but is bounded by   (  ) ≤   (  ) ≤ 1 −   (  ).

Table 1 :
Characteristics of experimental data sets. )

Table 2 :
Accuracy (%) and time (s) comparison of different methods.