In order to improve utilization rate of high dimensional data features, an ensemble learning method based on feature selection for entity resolution is developed. Entity resolution is regarded as a binary classification problem, an optimization model is designed to maximize each classifier’s classification accuracy and dissimilarity between classifiers and minimize cardinality of features. A modified multiobjective ant colony optimization algorithm is employed to solve the model for each base classifier, two pheromone matrices are set up, weighted product method is applied to aggregate values of two pheromone matrices, and feature’s Fisher discriminant rate of records’ similarity vector is calculated as heuristic information. A solution which is called complementary subset is selected from Pareto archive according to the descending order of three objectives to train the given base classifier. After training all base classifiers, their classification outputs are aggregated by max-wins voting method to obtain the ensemble classifiers’ final result. A simulation experiment is carried out on three classical datasets. The results show the effectiveness of our method, as well as a better performance compared with the other two methods.
Entity resolution (ER) is to find out the ambiguous denotations which refer to the same real world entity. ER has been researched for a long time and it is a crucial stage in data cleaning. It is also called record linkage in statistics, disambiguation in information retrieval [
In the big data era, ER’s researches for big data have become a hot point [
Crowdsourcing is a new ER approach [
Priya et al. [
In order to overcome shortage of human participation for ER in high dimensional data, Cheng et al. [
In order to overcome the difficulties of current ER methods for high dimensional data, we propose an ensemble learning method based on feature selection in this paper. We regard ER process as a binary classification problem, that is, classifying a record pair as matches (similar) or nonmatches (dissimilar). Then we define the measures of classification performance and similarity between binary classifiers which employ SVM as base classifier. Three objectives are applied to optimize each base classifier’s performance, that is, maximizing classifier’s classification accuracy rate, maximizing dissimilarity between classifiers, and minimizing cardinality of features. A modified multiobjective ant colony optimization (MOACO) is designed to solve the optimization model to select complementary feature subset which is adopted to train base classifier. In the end, several binary base classifiers are combined by ensemble learning method to improve performance for ER in high dimensional data.
The paper is organized as follows: Section
According to machine learning technologies, ER’s methods can be divided into four categories: methods based on probability; methods based on rules; methods based on clustering; and methods based on classification [
In this paper, we regard ER as a binary classification problem, and the results contain two classes: matches class and nonmatches class, respectively. Firstly, a candidate record pair is represented by a similarity vector which is obtained by computing their corresponding features’ similarity. Secondly, the similarity vector is adopted as input of a binary classifier to identify whether they are matches or nonmatches. The above process can be described as Figure
Process of entity resolution.
Without loss of generality, we only discuss records in one table. Suppose a record has
As discussed above, we regard ER as a binary classification problem in this paper, so how to measure classifier’s performance is the key to improving its classification effectiveness. In this section, we define some indicators to measure binary classifier’s performance.
Generally speaking, a binary classifier can be seen as a mapping function which is a many-to-one mapping classifier from sample space to class space. That is to say, it maps
Mapping diagram of a binary classifier.
Now we define the measures of binary classifier for high dimensional ER.
Classification accuracy rate
False alarm rate
Fault is not recognized rate
The binary classifier’s output distribution matrix:
Then
Then the classification accuracy rate
So the false alarm rate
The fault is not recognized rate
Then
Based on (
When many classifiers exist, the classifiers which have similar outputs may have similar results under the same ambiguous data, so their combination cannot improve the classification performance. On the other hand, when there are some differences between classifiers’ results, their ensemble will improve the classification performance to a certain extent, which is illustrated in Figure
Examples of classifiers’ ensemble.
Example of two similar classifiers
Example of two dissimilar classifiers
In Figure
Given a set of samples and feature subset
So the classifiers which have complementary results can be obtained by using complementary feature subsets to train them. It is also clear that the similarity of binary classifier
Next we define the measures of similarity between classifiers (similar to diversity) [
Two binary classifiers’ input similarity is defined as the similarity of input feature subsets. For feature subsets
From (
Two binary classifiers’ output similarity is defined as the similarity of classifiers’ output distribution matrix. Given two binary classifiers’ output distribution matrices
If
Suppose
From above analysis, we know that the output similarity of classifiers is stronger than their input similarity, so we adopt output similarity of classifiers to measure the similarity of classifiers. It is also clear that the more dissimilarity between classifiers which are trained by complementary subsets, the more complementarity between them, which may improve utilization rate of high dimensional data features.
There is no single classification algorithm that can solve all kinds of problems according to “no free lunch theorems” [
For high dimensional ER, the proposed model based on feature selection which is used to train binary classifiers can be described as follows: Suppose there are ensemble classifiers which contain
Equation (
The decision function adopted by our ensemble classifiers is “Max-Wins” voting method. Suppose
If the ensemble classifiers contain
A multiobjective optimization problem contains two or more than two objectives which have no order of priority. It can be stated as follows [
The solutions of multiobjective optimization problem are called Pareto optimal solutions which could not be further improved on any objective without harming the rest of objectives. And decision maker should choose one solution based on his or her preference. The goal of multiobjective optimization is to approximate the Pareto front which is composed of Pareto optimal solutions in the objective space [
MOACO is an excellent multiobjective evolutionary algorithm based on the foraging behavior of real ant species. The indirect communication of real ants in the colony uses pheromone trail lying on the ground to guide others to find the shortest path between their food source and the nest, which is called information positive feedback mechanism. As MOACO has that mechanism inherently and adopts reactive search optimization principle, that is, “learning while optimizing” principle [
As our model is a feature selection problem which is also a typical combinatorial optimization problem, we adopt a modified MOACO for solving it.
In order to solve the model by MOACO, we analyze it as follows:
(i) For a given binary classifier, it will have a better computing efficiency and classification accuracy rate if the number of its input features is set between 5 and 10. But we need more features to train the classifier to improve its performance in a high dimensional data. So the number of input features
(ii) Transform (
(iii) When MOACO finishes for a given binary classifier under all
(a) Compare values of all solutions on objective
(b) If there are many solutions with the same max value on objective
(c) If there are many solutions with the same max values on objective
Based on the discussion of Section
Cao et al. [
Lopez-Ibanez and Stutzle [
In MOACO, each ant selects route (feature) according to the transition probabilities. In the case of traveling salesman problem, the probability that ant
Besides, based on our model’s characteristic, the heuristic information
After one cycle, MOACO uses solutions found by all ants to update its Pareto archive based on their Pareto relations. Then we adopt equivalent routes’ policy and solutions in Pareto archive to update the two pheromone matrices.
In summary, the proposed modified MOACO can be described as Algorithm
Initialize parameters, pheromone matrices, and Pareto archive Generate weight parameter Aggregate values from two pheromone matrices by Construct solution by Eq. ( Update Pareto archive Update pheromone matrices by solutions of Pareto archive and Eq. (
We now analyze the time complexity of MOACO in Algorithm
Based on above discussion, the pseudocode of our method is shown in Algorithm
Initialize parameters and each base binary classifier’s Pareto archive Search for the optimal solution (feature subset) by Algorithm Update current base binary classifier’s archive based on analysis (ii) in Section Choose a solution (feature subset) as the current base binary classifier’s input based on analysis (iii) in Section Apply max-wins voting method to aggregate classifiers
Now we analyze the time complexity of Algorithm
We applied three datasets (Colon, GLIOMA, and GLI_85) which come from a well-known website (available from
Characteristics of the three testing datasets.
Dataset | Dimension | Sample | Class |
---|---|---|---|
Colon | 2000 | 62 | 2 |
GLIOMA | 4434 | 50 | 4 |
GLI_85 | 22283 | 85 | 2 |
In order to make the three datasets fit for ER, we must translate them into datasets composed of similarity feature vectors. For each dataset, we first normalized its features’ range between zero and one. Then we chose two samples from the same class as a matching record pair and two samples from different classes as a nonmatching record pair. We used absolute difference value between two records’ corresponding features as their features’ similarities which constitute records’ similarity feature vector. Besides, there are more nonmatching records than matching records in a real world problem, so we adopted uniform resampling method to make the number of matches less than that of nonmatches. We can obtain three new datasets for our experiment after above preprocessing, and they are called Colon_ER, GLIOMA_ER, and GLI_85_ER. The characteristics of them are shown as Table
Characteristics of experimental datasets.
Dataset | Dimension | Matches | Nonmatches |
---|---|---|---|
Colon_ER | 2000 | 176 | 880 |
GLIOMA_ER | 4434 | 200 | 1000 |
GLI_85_ER | 22283 | 200 | 1000 |
We made a comparison between two methods and our proposed method in this section. We adopted a single SVM for ER, it calculated records’ similarity vectors by all features, and this model is called method 1. Our model is called method 2. Cao et al. [
Method 1 applied SVM as a classifier, whose kernel function was “rbf” and
The parameters of method 2 were set as follows: the base binary classifier was SVM whose parameters were set as method1, number of base binary classifiers
The parameters of method 3 were set the same as those in [
The results of three methods on three datasets are shown in Tables
Results of Colon_ER.
Number | Compared methods | | | |
---|---|---|---|---|
1 | Method 1 | 0.4267 | 0.8000 | 0.5565 |
Method 2 | | | | |
Method 3 | 0.6753 | 0.7785 | 0.7233 | |
| ||||
2 | Method 1 | 0.4808 | | 0.6268 |
Method 2 | | 0.8676 | | |
Method 3 | 0.7070 | 0.8053 | 0.7529 | |
| ||||
3 | Method 1 | 0.4800 | 0.9000 | 0.6261 |
Method 2 | | | | |
Method 3 | 0.7021 | 0.8483 | 0.7683 | |
| ||||
4 | Method 1 | 0.4257 | 0.8000 | 0.5557 |
Method 2 | | | | |
Method 3 | 0.7374 | 0.8185 | 0.7758 | |
| ||||
5 | Method 1 | 0.3738 | 0.7000 | 0.4873 |
Method 2 | | | | |
Method 3 | 0.7662 | 0.8209 | 0.7926 |
Results of GLIOMA_ER.
Number | Compared methods | | | |
---|---|---|---|---|
1 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.7778 | 0.4650 | 0.5821 | |
| ||||
2 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.6880 | 0.6828 | 0.6854 | |
| ||||
3 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.6772 | 0.6300 | 0.6528 | |
| ||||
4 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.7149 | 0.6065 | 0.6563 | |
| ||||
5 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.6981 | 0.6877 | 0.6929 |
Results of GLI_85_ER.
Number | Compared methods | | | |
---|---|---|---|---|
1 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.9216 | 0.5000 | 0.6483 | |
| ||||
2 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.8000 | 0.7543 | 0.7765 | |
| ||||
3 | Method 1 | / | / | / |
Method 2 | | | | |
Method 3 | 0.9037 | 0.6667 | 0.7673 | |
| ||||
4 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.7753 | 0.7886 | 0.7819 | |
| ||||
5 | Method 1 | / | 0 | / |
Method 2 | | | | |
Method 3 | 0.9000 | 0.6923 | 0.7826 |
The results on dataset Colon_ER in Table
The results of method 3 on three datasets demonstrate that it has a higher performance than method 1. Feature selection could filter irrelevant features and noise information, which helps to improve the performance of classifier. Method 3 takes classification precision, recall rate, and cardinality of features as optimization objectives for feature selection. So the selected features used to train classifier could make it classify most samples correctly, which improves its classification precision and recall rate. The values of
At last, we can find that method 2 has a higher performance on three datasets than method 1 and method 3 for high dimensional ER. There are three reasons causing this situation. Method 2 takes classification accuracy rate as optimization objective to consider about both false alarm rate and fault is not recognized rate of classifiers to get an improvement for its performance. And Table
Classification accuracy rate of three methods.
Dataset | Method 1 | Method 2 | Method 3 |
---|---|---|---|
Colon_ER | | | |
GLIOMA_ER | | | |
GLI_85_ER | | | |
Table
Objectives’ values of final solutions of method 2 on GLIOMA_ER.
Number | Classifier | | |
---|---|---|---|
1 | 1 | 0.8267 | / |
2 | 0.8385 | 0.5886 | |
3 | 0.8339 | 0.5839 | |
4 | 0.8312 | 0.5819 | |
5 | 0.8394 | 0.5880 | |
| |||
2 | 1 | 0.8475 | / |
2 | 0.8757 | 0.6130 | |
3 | 0.8612 | 0.6052 | |
4 | 0.8448 | 0.5915 | |
5 | 0.8584 | 0.6009 | |
| |||
3 | 1 | 0.8512 | / |
2 | 0.8648 | 0.6067 | |
3 | 0.8675 | 0.6078 | |
4 | 0.8575 | 0.6006 | |
5 | 0.8621 | 0.6034 | |
| |||
4 | 1 | 0.8267 | / |
2 | 0.8385 | 0.5886 | |
3 | 0.8339 | 0.5839 | |
4 | 0.8312 | 0.5819 | |
5 | 0.8394 | 0.5880 | |
| |||
5 | 1 | 0.8240 | / |
2 | 0.8246 | 0.5800 | |
3 | 0.8267 | 0.5787 | |
4 | 0.8249 | 0.5775 | |
5 | 0.8185 | 0.5730 |
In order to improve features’ utilization rate of high dimensional data and reduce the impact brought by irrelevant features, an ensemble learning method based on feature selection is proposed, and some conclusions through experiments can be obtained as follows. It could reduce the impacts brought by irrelevant features and noise information through applying feature selection, which improves classification performance for high dimensional ER. It can get a tradeoff between classification precision and recall rate by taking classification accuracy rate as an optimization objective. The complementary feature subsets obtained by maximizing dissimilarity between classifiers can address local feature relevance problem efficiently. Combining classifiers trained by complementary feature subsets through ensemble learning method can further improve algorithm’s performance for high dimensional ER. Note that the proposed method is very generic. It can be applicable for solving ER from any databases provided there is a way to calculate the distance between two given records without considering the number of dimensions (our method is also suitable for the situation with low dimensions in order to further improve the performance of ER). In order to illustrate the effectiveness of our method, we have used three datasets, but any other datasets could have also been used with suitable distance measure. The proposed method is also very generic with respect to the solution framework. Thus instead of MOACO any other optimization technique could be used. And also the SVM could be replaced by other classifiers.
Though our method has a lot of advantages, there are more problems that need to be solved. We adopt evolutionary algorithm to solve model, but the running time increases with dimensions increasing (e.g., we may need more time to select 20 features from data with 10000 features rather than 5000). Second, there are more nonmatches than matches in ER, which is known as imbalanced data problem, but we do not consider it in our method. Finally, we need more researches to make use of complementary feature subsets to further improve utilization rate of high dimensional data features.
The authors declare that there is no conflict of interests regarding the publication of this manuscript.
This work is supported by the Natural Science Foundation of China under Grant no. 61371196, the China Postdoctoral Science Special Foundation under Grant no. 201003797, the China Postdoctoral Science Foundation under Grant no. 2015M582832 and Grant no. 20090461425, and the Jiangsu Postdoctor Research Fund of China under Grant no. 1402138C.