An Application of Classiﬁer Combination Methods in Hand Gesture Recognition

,


Introduction
Hand gesture language, being one type of sign languages, originates from deaf people communicating with each other.In order to convey a meaning, a person generally needs to simultaneously combine the shape, orientation, and movement of his hands.The complex spatial grammars of hand gesture language are markedly different from the grammars of spoken languages.
Apart from facilitating deaf people to express their thoughts more conveniently, hand gestures are also a very natural part of human communications for common people.In some special situations such as very noisy environments where speech is not possible, they can become the primary communication medium.With the rapid development of hand gesture language, it nowadays has been applied in many fields such as human-computer interaction, visual surveillance, and so on 1-3 .Thus, hand gesture recognition becomes a hot topic in artificial intelligence and computer vision with the goal to automatically interpret human hand gestures via some algorithms.Nevertheless, the great variability in spatial and temporal features of a hand gesture, such as that in time, size, and position, as well as interpersonal differences, makes the recognition problem be very difficult.For instance, different subjects have different hand appearance and may sign gesture in different pace.
Recent works in hand gesture recognition tend to handle the spatial and temporal variations separately and therefore lead to two smaller areas, namely, static posture recognition 4-10 and dynamic action recognition 2, 3, 11, 12 .In static posture recognition, the pose or the configuration of hands should be recognized using texture or some other features.By contrast, hand action recognition tries to interpret the meaning of the movement using dynamic features such as the trajectory of hands and so on.In the current study, we will focus on hand posture classification and recognition.
As for hand posture recognition, Bedregal et al. 4 introduced a fuzzy rule-based method for recognizing the hand gestures of LIBRAS the Brazilian Sign Language .The method utilizes the set of angles of finger joints for classifying hand configurations and classifications of segments of hand gestures for recognizing gestures based on the concept of monotonic gesture segment.Just et al. 5 applied an approach that has been successfully used for face recognition to the hand posture recognition.The used features are based on the modified census transform and are illumination invariant.To achieve the classification and recognition processes, a simple linear classifier is trained using a set of feature look-up tables.Kim and Cipolla 7 attempted to address gesture recognition under small sample size where direct use of traditional classifiers is inappropriate due to high dimensionality of input space.Through combining canonical correlation analysis with the discriminative functions and scale-invariant feature transform SIFT , they developed a pairwise feature extraction method for robust gesture recognition.In the experiments using 900 videos of 9 hand gesture classes, the proposed procedure was seen to notably outperform support vector classifier and relevance vector classifier.Based on a hand gesture fitting procedure via a new self-growing and self-organized neural gas SGONG network, a new method for hand gesture recognition was proposed by Stergiopoulou and Papamarkos 8 .The main idea of this method is as follows.Initially, the region of the hand is detected by applying a color segmentation technique based on a skin color filtering procedure in the Y CbCr color space.Then, the SGONG network is applied on the hand area so as to approach its shape.Based on the output grid of neurons produced by the neural network, palm morphologic characteristics are extracted.These characteristics, in accordance with powerful finger features, allow the identification of the raised fingers.Finally, the hand gesture recognition is accomplished through a likelihood-based classification technique.The proposed system has been extensively tested with success.Furthermore, Flasi ński and Myśli ński 9 presented a novel method for recognizing hand postures of the Polish sign language based on a synthetic pattern recognition paradigm.The main objective is to construct algorithms for generating a structural graph description of hand postures that can be analyzed with the ETPL k graph grammar parsing model.The structural description generated with the designed algorithms is unique and unambiguous, which results in good discriminative properties of the method.
In recent years, classifier combination strategies are rapidly growing and enjoying a lot of attention from pattern recognition as well as many other various domains due to their potential to greatly increase the prediction accuracy of a learning system.So far, these techniques have been proven to be quite versatile in a broad field of real applications such as face recognition, sentiment classification, and so forth 13-15 .Compared with one single classifier, an ensemble classifier has advantages to handle a classification task which is difficult for traditional methods, to achieve much higher prediction accuracy, and so forth.
In the research works of hand gesture recognition, however, there is very little literature about the applications of ensemble classifier methods.Dinh et al. 6 proposed a hand gesture classification system which is able to efficiently recognize 24 basic signs of the American sign language.In the system, computational performance is achieved through the use of a boosted cascade of classifiers that are trained by AdaBoost and informative Harr wavelet features.To adapt to complex representation of hand gestures, a new type of feature was suggested.Some experimental results show that the proposed approach is promising.Burger et al. 10 suggested to apply a belief-based method for SVM support vector machine fusion to recognize hand shapes.Moreover, the method was integrated into a wider classification scheme which allows taking into account other sources of information, by expressing them in the belief theories' formalism.The experimental results have shown that the proposed method was better than the classical methods in avoiding more than 1/5 of the mistakes.In this paper, we will employ several classifier combination techniques to deal with this specific classification problem.On the basis of some hand gesture data at hand, the methods AdaBoost and rotation forest are seen to behave significantly better than all the other considered algorithms, especially a classification tree.The reasons for the success of AdaBoost and rotation forest are then investigated by analyzing the bias-variance decompositions of error for all the compared algorithms.Moreover, the diversity-accuracy patterns of each ensemble classifier are studied via kappa-error diagrams in a visual manner and some promising results are obtained.
The rest of the paper is organized as follows.In Section 2, some commonly used classifier combination methods are reviewed in brief.Section 3 describes our used hand gesture recognition data set.Some experiments are conducted in Section 4 to find out the best method to solve this specific classification task.In the meantime, the reasons for the better performance of some algorithms are also investigated.Finally, the conclusions of the paper are offered in Section 5.

Brief Review of Ensemble Methods
In this section, we will give a brief review of our later used classifier combination techniques.In an ensemble classifier, multiple classifiers which are generally referred to as base classifiers should be first generated by means of applying a base learning algorithm also called base learner to different distributions of the training data, and then the outputs from each ensemble member are combined with a classifier fusion rule to classify a new example.In order to construct an ensemble classifier having better prediction capability, its constituent members should be accurate which at the same time should disagree as much as possible.In other words, the diversity between base classifiers and their individual accuracy are two essential factors for building a good ensemble classifier.However, these two factors are contradictory in practice.Generally speaking, there is a tradeoff between diversity and accuracy: as the classifiers become more diverse, they must become less accurate; conversely, as they become more accurate, the diversity between them must turn to be lower.With respect to different ensemble classifier generation strategies, the difference between them mainly lies in how to achieve a better tradeoff between diversity and accuracy of their base classifiers.
In order to facilitate the following discussions, we first introduce some notations here.Denote by L {x i , y i } N i 1 a training set consisting of N observations, where x i x i,1 , x i,2 , . . ., x i,p T is a p-dimensional feature vector and y i is a class label coming from the set Φ {φ 1 , φ 2 , . . ., φ J }.Meanwhile, let X x 1 , x 2 , . . ., x N T be an N × p matrix containing the training features and Y y 1 , y 2 , . . ., y N T be an N-dimensional vector containing the class labels for the training data.Put in another way, the training set L can be expressed as concatenating X and Y horizontally; that is, L X Y .Furthermore, let T be the number of base classifiers, and let C 1 , C 2 , . . ., C T be the T classifiers to construct an ensemble classifier, say, C * .Denote by W the given base learning algorithm to train each base classifier.
Bagging 16 , random subspace 17 , and random forest 18 may be the three most intuitive and simplest ensemble learning methods to implement.These three methods share the same combination rule, that is, simply majority voting scheme, to combine the decisions of their base classifiers.They only differ in how to use the given training set to generate a diverse set of base classifiers.Breiman's bagging, an acronym of bootstrap aggregating, trains its base classifiers by applying a base learning algorithm to some bootstrap samples 19 of the given training set L. Each bootstrap sample is generated by performing N extractions with replacement from L. As a result, in each of the resulted training sets for constructing base classifiers, many of the original training examples may appear several times whereas others may never occur.The random subspace method 17 obtains different versions of the original training set by performing modifications in the feature space i.e., randomly selecting some features rather than in the example space like bagging.The different training sets are then provided as the input of the given learning algorithm to build its base classifiers.As for random forest 18 , it is an ensemble technique by taking a decision tree algorithm 20 as its base leaner.Beside utilizing bootstrap sampling to obtain different training sets like bagging, random forest tries to produce additional diversity between base classifiers by adding a randomization principle in the tree induction process, which randomly selects a feature subset of size K a hyperparameter of random forest at each nonterminal node and then chooses the best split among it.In Algorithm 1, we present the general algorithmic framework for these three ensemble methods.
Nowadays, boosting can be deemed as the largest algorithmic family in the domain of ensemble learning.Unlike bagging, whose base classifiers can be trained in parallel, boosting is a sequential algorithm in which each new classifier is built by taking into account the performance of the previously generated classifiers.AdaBoost 21, 22 , due to its simplicity where Z t is a normalization factor being chosen so that D t 1 is a probability distribution over L.

Endfor • Output
-The class label for x predicted by the ensemble classifier C * as

Algorithm 2:
The main steps for the AdaBoost algorithm.
and adaptability, has become the most prominent member in boosting family.AdaBoost works by constructing an ensemble of subsidiary classifiers by applying a base learner to successive derived training sets that are formed by either resampling from the original training set 21 or reweighting the original training set 22 according to a set of weights maintained over the training set.Initially, the weights assigned to each training example are set to be equal, and, in subsequent iterations, these weights are adjusted so that the weight of the instances misclassified by the previously trained classifiers is increased whereas that of the correctly ones is decreased.Thus, AdaBoost attempts to produce new classifiers that are able to better predict the "hard" examples for the previous ensemble members.After a sequence of classifiers is trained, which is then combined by weighted majority voting in the final decision.Algorithm 2 lists the main steps for the resampling version of AdaBoost which is utilized in our later experiments.
Based on principal component analysis PCA , Rodríguez et al. 23 developed a novel ensemble classifier creation technique rotation forest and demonstrated that it outperforms several other ensemble methods on some benchmark classification data sets from the UCI repository 24 .With the aim to create training data for a base classifier, the feature set of L is randomly split into many subsets, and PCA is applied to each subset.All principal components are retained in order to preserve the variability information in the data.Thus, some axis rotations take place to form new features for training a base classifier.The main idea of rotation forest is to simultaneously encourage diversity and individual accuracy within an ensemble classifier.Specifically, diversity is promoted by using PCA to do feature axis rotation for each base classifier while accuracy is sought by keeping all principal components and also using the whole data set to train each base classifier.Here, we summarize the detailed steps of rotation forest in Algorithm 3. Note that in this algorithm, there is another parameter, namely, the number of features M contained in each feature subset, which should be specified in advance.For simplicity, suppose that M is a factor of p so that the features are distributed into K subsets with each of them containing M features.Otherwise, the Kth feature subset will have p − K − 1 M features.According to the results reported by Rodríguez et al. 23 , rotation forest with M 3 performs satisfactorily, and it thus provides users a directive advice to choose suitable value for M. Furthermore, Melville and Mooney 25 proposed a new meta-learner DECORATE diverse ensemble creation by oppositional relabeling of artificial training examples that can use any "strong" learner one that provides high accuracy on the training data to build a diverse ensemble.This algorithm is accomplished by adding different randomly constructed instances to the training set when building new ensemble members.The artificial constructed instances are given category labels that disagree with the prediction of the current ensemble, thereby directly increasing diversity when a new learner is trained on the augmented data and added to the ensemble.Based on the experimental results using the J48 an open source Java implementation of the C4.5 algorithm in the Weka data mining tool decisiontree induction as a base learner and the analysis of the cross-validated learning curves for DECORATE as well as some other ensemble methods, Melville and Mooney 25 found that DECORATE produces highly accurate ensembles that outperform bagging, AdaBoost, and random forest low on the learning curve.In order to make this paper self-sufficient, we list the pseudocodes of DECORATE in Algorithm 4 as follows.-Let p i,j x be the probability that x comes from class j supported by the classifier C i .Calculate the confidence for each class by the mean combination rule, that is, where L stands for the real ensemble size.-Assign x to the class with the largest confidence.

Algorithm 4:
The pseudocodes for DECORATE ensemble method.
As can be seen in Algorithm 4, DECORATE builds an ensemble classifier iteratively like all the other ensemble methods.Initially, a classifier is trained on the basis of the given training data L. In each successive iteration, one classifier is created by applying a base learner W to L combined with some artificial data.In each iteration, some artificial training instances are generated according to the data distribution that the given training data only consider the input variables now come from, where the number of instances to be generated is specified as a fraction, R size , of the training set size N.As for the labels for each artificially generated training instance x k , first utilize the current ensemble to predict the class membership probabilities P x k P 1 x k , P 2 x k , . . ., P J x k T that this instance belongs to each class.Then, replace zero probabilities with a small nonzero value and normalize the probabilities to make them form a probability distribution P x k P 1 x k , P 2 x k , . . ., P J x k T .The label y k of the instance x k can then be determined such that the probability for each class i being selected is inversely proportional to the ensemble's prediction; namely, P i x k 1/ P i x k / J j 1 1/ P j x k .The main purpose in doing so is to make the labels for the artificially generated instances differ maximally from the current ensemble's predictions in order to promote the diversity in the constructed ensemble classifier.Thus, the labeled artificially created training set is called diversity data.On the other hand, DECORATE tries to maintain the accuracy of each ensemble member while forcing diversity through rejecting a new classifier if adding it to the existing ensemble decreases its accuracy.The above whole process is repeated until the desired ensemble size is reached or the maximum number of iterations is exceeded.
It is worth to mention that, in DECORATE, the artificially generated training data are randomly picked from an approximation of the training-data distribution.For a numeric feature, the values of it are created from a Gaussian distribution whose mean and standard deviation are computed from the corresponding data in the training set.As for a nominal feature, the probability of occurrence of each distinct value in its domain should be first calculated in which the Laplace smoothing needs to be employed so that nominal feature values not represented in the training set still have a nonzero probability of occurrence.Then, some values can be generated based on this distribution.Another issue that should be pointed out is that we can only specify a desired ensemble size C size when using DECORATE to deal with a classification task.The size L of the finally obtained ensemble may be smaller than C size because the algorithm will terminate if the number of iterations exceeds the maximum limit even if C size is not reached.As for R size , it can take any value in theory.Nevertheless, the experiments done by Melville and Mooney 25 have shown that R size lower than 0.5 adversely affect the performance of DECORATE, and the results with R size chosen in range 0.5∼1 do not vary much.

Data Set
For a hand gesture recognition problem, the task is to design a classifier to recognize different hand gestures where each gesture has a meaning of one simple or compound word.Our used data set contains 120 different signs of the Dutch sign language, each performed by 75 different persons.The images were captured at 640 × 480 pixels and 25 frames per second.Most sign examples include partial occlusions of hands of each other or with the face/neck.For the detailed process to obtain the experimental data, the readers can refer to 3, 26 .The supplementary video can be found on the Computer Society Digital Library at http://www.computer.org/portal/web/csdl/doi/10.1109/TPAMI.2008.123.We briefly introduce the process to collect the experimental data as follows.When a person is making a hand gesture, two cameras are used to independently record the continuous activity of his left and right hands.Because the gesture is made continuously and we would like to obtain some features, three images frames which, respectively, denote the beginning, middle, and ending of the gesture were extracted from the video recorded by one camera, and they were denoted as image 1, 2, and 3 here.Based on each obtained image, some segmentation algorithm was first used to segment two hands from the background, and then 7 invariant moments were computed for each hand.This process was repeated for the video of the other camera.Finally, we obtained 84 features in total through collecting the computed moments corresponding to two hands, three images, and two cameras together.In the original data set, there are totally 120 different gestures among which 29 ones denote compound words and 91 ones indicate simple words.For each gesture, there are 75 objects made by different persons, and every object is described by 84 features which were extracted in the above-mentioned way.
Unfortunately, we encountered a problem during segmenting two hands from one image.Sometimes there may occur an overlap between head, left hand, or right hand, which make it impossible to compute meaningful moments for each hand.Therefore, some corresponding features cannot be obtained in the general manner, and they are indicated as missing.In our experiments, we only considered the objects without missing features.Meanwhile, we tried to select the classes consisting of approximately equal number of objects.Through preprocessing the experimental data in this way, we finally obtained a set having 11 classes with about 70 objects in each class.The data set contains 793 objects in total, and each object is described by 84 features.

Experimental Setting
In this section, we did some experiments by applying six commonly used classifier combination methods to the hand gesture recognition data set that is described previously.The considered ensemble methods include bagging 16 , random forest 18 , random subspace 17 , AdaBoost 22 , rotation forest 23 , and DECORATE 25 .
The experimental settings were as follows.In all the ensemble methods, a decision tree 20 was always adopted as the base learning algorithm because it is sensitive to the changes in its training data and can still be very accurate.The following experiments were all conducted in Matlab software with version 7.7.The decision tree algorithm was realized by the "Treefit" algorithm contained in the "Stats" package of Matlab.The parameters involved in this algorithm, such as the minimum number of training instances that impure nodes to be split should have, were all set to the default values.The implementations of the considered ensemble methods were realized in Matlab by writing programs according to their respective pseudocodes.
The ensemble size was set to 25 since the largest error reduction achieved by ensemble methods generally occurs at the first several iterations.Although larger ensemble size may result in better performance, the improvement achieved at the cost of additional computational complexity is trivial in comparison with that obtained with just a few iterations.As for the hyperparameter K in random forest, which specifies how many features should be firstly selected at each nonterminal node in the process of building a decision tree, the value of it was taken to be log 2 p 1 since some experiments 18 have proven that this choice makes random forest achieve good performance very often.When using random subspace technique to construct an ensemble classifier, one half of features were randomly selected to train its each constituent member.With respect to the parameter M which indicates the number of features contained in each feature subset in rotation forest, we set it to be 3 just like Rodríguez et al. 23 did because they have found that this value was almost always the best choice in their experiments.In DECORATE algorithm, the used parameters except for the ensemble size were all identical to those utilized by Melville and Mooney 25 ; namely, the maximum number of iterations I max to build an ensemble classifier was set to 50, and the factor R size to determine number of artificial examples to generate was chosen to be 1.Here, it should be noted that we can only specify a desired ensemble size for DECORATE algorithm while it may terminate if the number of iterations exceeds the maximum limit even if the desired ensemble size is not reached.

Comparison of Prediction Error
Because there have not been separate training and testing data to use, we employed the 10fold cross-validation method to investigate the performance of the considered classification methods.Specifically, the data was first split into ten subsets with approximately equal sizes, and then nine of them were utilized as a training set to construct a forest while the other one was used to estimate the prediction error of the forest.The experiment was conducted ten times through alternating the role of ten subsets until each of them was used for testing once.We repeated the above process ten times with different random number generating seeds to split the data in order to eliminate the impact of random factor to the performance of each algorithm.
Before utilizing each obtained training set to carry out experiments, we preprocessed the data based on one normalization technique.Given a training set L { x i , y i } N i 1 , the normalization of the values corresponding to each feature X j j 1, 2, . . ., p can be expressed as where x c j and s c j , respectively, denote the mean and variance for class ω c and s is the weighted sum of the variances for each class with weights equal to class prior probabilities.After obtaining x c j and s, the same mapping was applied to the test set.Table 1 reported the mean as well as the standard deviation of the computed test errors for each algorithm.In order to make a complete comparison, the results calculated with a classification tree were also taken into account.In Table 1, the best results were highlighted in bold face to facilitate the comparison.With the aim to make clear whether there is significant difference between the performance of our evaluated ensemble methods on this specific data set, we adopted a one-tailed paired t-test with significance level α 0.01 to carry out some statistical tests between each pair of algorithms.If an algorithm is found to be significantly better than its competitor, we assigned score 1 to the former and −1 to the latter.If there is no significant difference between the two compared methods, they both score 0. Obviously, the higher the score of an approach, the better its performance.In the third row of Table 1, we listed the scores that each classification method gets according to the number of times that it has been significantly better or worse than the other algorithms.
From the obtained mean test errors and scores for each classification method, it can be observed that the prediction error of a single decision tree has been improved greatly by each classifier combination technique, especially by AdaBoost and rotation forest.Among the ensemble algorithms, DECORATE was seen to perform much worse than the other ensemble learning strategies; the reason may be that its main advantage is to deal with classification problems with small training set size while the sample size of the current hand gesture recognition data set is medium.Based on the scores calculated from the statistical tests between each pair of algorithms, rotation forest is seen to be the best method to solve this specific problem, and it performs significantly better than all the other algorithms at significance level α 0.01.Meanwhile, AdaBoost is the second best approach since it was only beaten by rotation forest.However, a single decision tree and DECORATE behave very badly and they should not be selected to deal with this problem.As for the other three ensemble methods, the performance of them is almost equivalent even though their working mechanism is different as described in Section 2.

Bias-Variance Decomposition of Error
In order to investigate the reasons for the better performance of an ensemble classifier than its constituent members, to decompose its error into bias and variance terms is a good choice, and this method has been used by many researchers 27-29 .The decomposition of a learning machine's error into bias and variance terms originates from analyzing learning models with numeric outputs in regression problems.Given a fixed target and training set size, the conventional formulation of the decomposition breaks the expected error into the sum of three nonnegative quantities.
i Intrinsic "target noise" σ 2 .This quantity is a lower bound on the expected error of any learning algorithm.It is the expected error of the Bayes optimal classifier.
ii Squared "bias" bias 2 .This quantity measures how closely the learning algorithm's average guess over all possible training sets of the given size matches the target.
iii "Variance" variance .This quantity measures how much the learning algorithm's guess fluctuates from the target for the different training sets of the given size.
Notice that the above decomposition cannot be directly translated to contexts where the value to be predicted is categorical; a number of ways to decompose error into bias and variance terms in the field of classification prediction tasks have been proposed 30-33 .Each of these definitions is able to provide some valuable insight into different aspects of a learning machine's performance.In order to gain more insight into the performance of the considered ensemble methods on the hand gesture recognition data set, we utilized the bias-variance definition developed by Kohavi and Wolpert 30 in the current research, and they were, respectively, denoted by Bias and Var in the following discussions.
If denote by Y H and Y F the random variables, respectively, representing the evaluated and true labels of an instance, Bias and Var defined for a testing instance x, y can be expressed as Here, the superscript L is used in Y L H to denote that the evaluated class label is predicted by the machine trained on the set L. The term Pr • in the above formulae can be computed as the frequency that the event included in the parentheses occurs in the trials which are conducted with different training sets of the given size.

4.2
To compute the above two statistics, the distribution of where the training data come from should be known in advance.Unfortunately, the knowledge we have in the current hand gesture recognition situation is only a learning sample with medium size.In consequence, the Bias and Var terms should be estimated instead.In our experiments, the method similar to that used by 32 , that is, ten trials of 10-fold cross-validation procedure, was utilized to estimate the bias and variance defined above.Once the cross-validation trials have been completed, the relevant measures can be estimated directly from the observed distribution of results.The use of cross-validation in this way has the advantage that every instance in the available data L is used the same number of times, both for training and for testing.
According to the above approach, the previously defined bias and variance decompositions of the errors for each classification method were estimated for each instance in the given data set, and then their values were averaged over that data set.Detailed decompositions of mean error into Bias and Var for each classification method were provided in the following Table 2.
As can be seen from Table 2, the order of the considered classification methods ranked in terms of Bias value from best to worst is AdaBoost, RotForest, RandForest, Bagging, RandSubspace, DECORATE, and SingleTree.With regard to Var, these algorithms are rated from best to worst as RotForest, RandSubspace, Bagging, AdaBoost, RandForest, DECORATE, and SingleTree.Therefore, the better performance of RotForest and AdaBoost can be attributed to the fact that they reduce both bias and variance of the SingleTree's error to a large degree.RotForest does a better job to reduce variance term while AdaBoost has a small advantage on reducing bias.The working mechanism of RandForest is similar to that of RotForest and AdaBoost, but the reduction achieved by it is not enough.In the meantime, bagging and RandSubspace are observed to mainly reduce the variance term.As for DECORATE, it was seen to only decrease the bias and variance of the SingleTree's error to a small extent.

Kappa-Error Diagrams
On the other hand, many researchers 34-36 have pointed out that the success of an ensemble classifier achieving much lower generalization error than its any constituent member lies in the fact that the ensemble classifier consists of highly accurate classifiers which at the same time disagree as much as possible.Put in another way, with the purpose to construct an ensemble classifier with good performance, we should achieve a good tradeoff between diversity and accuracy.
The kappa-error diagrams developed by Margineantu and Dietterich 37 provide us an effective means to visualize how an ensemble classifier which is constructed by some ensemble learning technique attempts to reach the tradeoff between the diversities and accuracies of its constituent members.For each pair of classifiers, the diversity between them is measured by the statistic kappa κ which evaluates the level of agreement between two classifier outputs while correcting for chance; the accuracy of them is measured by the average of their error rates estimated on the testing data set.A kappa-error diagram is a scatter plot in which each point corresponds to a pair of classifiers C i and C j .On the x-axis of the plot is the diversity value κ and on the y-axis of it is the mean error of C i and C j , say, The statistic κ is defined as follows.Suppose that there are J classes and κ is defined on the J × J coincidence matrix M of two classifiers C i and C j i, j 1, 2, . . ., T where T is the number of classifiers in an ensemble.The entry m k,s of M is the proportion of the testing data set, in which classifier C i labels as ω k and classifier C j labels as ω s .Then the agreement between C i and C j can be measured as where J k 1 m kk is the observed agreement between the two classifiers C i and C j ."ABC," the acronym of "agreement-by-chance," is defined as According to the above definition of κ, low values of it indicate higher diversity.And since small values of E i,j indicate better accuracy, the most desirable pairs of classifiers should lie in the bottom left corner of the scatter plot.Figure 1 illustrates the kappa-error diagrams of the ensemble classifiers constructed by each ensemble algorithm on the hand gesture recognition data.All the constructed forests but the one built by DECORATE consist of 25 trees; therefore there are 300 C 2  25 points in each plot.Because the data set has no separate training and testing parts, we randomly took 90% of observations to build the forests and the remaining to compute the kappa-error diagrams.With regard to each plot, the axes of them were adjusted to be identical so that the comparisons can be easily carried out.Moreover, on the top of each plot in Figure 1, we presented the used ensemble method, its prediction error estimated on the testing set, and the coordinates shown as the red point in each plot for the mean of diversities and that of errors which were averaged over all pairs of base classifiers.
As can be observed in Figure 1, DECORATE gives a very compact cloud of points.Each point has a low error rate and a high value for κ, which indicates that the base classifiers are accurate but not very diverse.The shape of the kappa-error diagrams for bagging and random subspace is similar, but the points for bagging are more diverse while those for random subspace are slightly more accurate.Although the test error of random forest is identical to that of AdaBoost, we can find that AdaBoost is more diverse but is not as accurate as random forest.In the meantime, rotation forest is seen to achieve better diversity than random subspace and DECORATE, but its accuracy is only worse than DECORATE.Thus, the final prediction error of rotation forest is a little higher than that of random forest and AdaBoost in this special situation.
Notice that here we just randomly selected 90% of the data to construct forests and utilized the remaining 10% to estimate the values of κ and mean error for each pair of base classifiers.The experiment was only carried out for one trial.If comparing the results obtained herein with those listed in Table 1, we should be cautious to draw conclusions since the errors for each algorithm as reported in Table 1 were averaged over ten trials of 10fold cross-validation in order to eliminate some random factor that may affect the relative performance of our considered classification methods.However, our aim in this subsection was just to employ the kappa-error diagrams to study the working mechanism of the compared ensemble methods more clearly.

Conclusions
In this paper, we adopted some widely used classifier fusion methods to solve a hand gesture recognition problem.Since the data of this classification task are likely coming from a multi-normal distribution, the ensemble methods are found to be more appropriate to deal with this problem because the performance of them is much better than that of a single classification tree.Among the ensemble techniques, AdaBoost and rotation forest behave significantly better than their rivals and they achieve the lowest generalization error.Through investigating the bias-variance decompositions of error for the considered classification algorithms, the success of AdaBoost and rotation forest can be attributed to the fact that each of them simultaneously reduces the bias and variance terms of the SingleTree's error to a large extent.Rotation forest does a better job to reduce variance whereas AdaBoost has a small advantage on reducing bias.Furthermore, we made use of kappa-error diagrams to visualize how a classifier combination strategy attempts to reach a good tradeoff between diversity and accuracy in the process of constructing an ensemble classifier.The experimental results demonstrate that AdaBoost creates the most diverse base classifiers but with a little higher error.With respect to rotation forest, it is observed to generate very accurate base classifiers while the diversity between them is only medium.

Figure 1 :
Figure 1:The kappa-error diagrams for the compared ensemble methods.
Provide L t as the input of W to train a classifier C t , and then compute the weighted training error of C t as ε t Pr i∼Dt C t x i / y i • takes value 1 or 0 depending on whether the ith training instance is misclassified or by C t or not. 3 If ε t > 0.5 or ε t 0, then set T t − 1 and abort loop.4 Let α t 1/2 ln 1 − ε t /ε t .5 Update the weight distribution D t over L as • Input A training set L { x i , y i } N i 1 ; A base learner W; Number of iterations T ; A new data point x to be classified.• Training Phase Initialization: Set the weight distribution over L as D 1 i 1/N i 1, 2, . . ., N .For t 1, . . ., T 1 According to the distribution D t , draw N training instances at random from L with replacement to compose a new set L t { x Randomly split the feature set F {X 1 , X 2 , . . ., X p } into K subsets F t,k k 1, 2, . . ., K . 2 For k 1, 2, . . ., K a Select the columns of X that correspond to the attributes in F t,k to compose a submatrix X t,k .b Draw a bootstrap sample X t,k with sample size smaller than that of X t,k , generally taken to be 75% from X t,k .c Apply PCA to X t,k to obtain a matrix D t,k whose ith column consists of the coefficients of the ith principal component.3 EndFor 4 Arrange the matrices D t,k k 1, 2, . . ., K into a block diagonal matrix R t .5 Construct the rotation matrix R a t by rearranging the rows of R t so that they correspond to the original features in F.
-Provide XR a t Y as the input of W to build a classifier C t .EndFor • Output -The class label for x predicted by the ensemble classifier C * as C * x arg max y∈Φ T t 1 I C t xR a t y .Algorithm 3: The detailed steps of rotation forest method.
Provide the given training set L as the input of base learner W to get a classifier C i ; Initialize ensemble set C * {C i }; -Compute ensemble error as While i < C size and trials < I max 1 Generate R size × N training instances, R, according to the distribution of training data; 2 Label each instance in R with probabilities that each class label is selected being inversely proportional to those predicted by C * ; 3 Combine L with R to get a new training set L ; 4 Apply base learner W to L to obtain a new classifier C ; 5 Add C to ensemble set C * , namely, let C * C * ∪ {C }; 6 Based on the training set L, compute the ensemble error of C * , say, ε , as that done in equation 3 ; 7 If ε ≤ ε, let i i 1 and update ensemble error as ε ε ; Otherwise, delete C from the ensemble set C * , that is, C * C * − {C }; 8 trials trials 1; -EndWhile • Prediction phase

Table 1 :
The means and standard deviations std. of prediction errors, as well as scores for each classification method on the considered data set.

Table 2 :
The bias-variance decompositions for each classification method on the considered data set.