A novel facial expression recognition algorithm based on discriminant neighborhood preserving nonnegative tensor factorization (DNPNTF) and extreme learning machine (ELM) is proposed. A discriminant constraint is adopted according to the manifold learning and graph embedding theory. The constraint is useful to exploit the spatial neighborhood structure and the prior defined discriminant properties. The obtained parts-based representations by our algorithm vary smoothly along the geodesics of the data manifold and have good discriminant property. To guarantee the convergence, the project gradient method is used for optimization. Then features extracted by DNPNTF are fed into ELM which is a training method for the single hidden layer feed-forward networks (SLFNs). Experimental results on JAFFE database and Cohn-Kanade database demonstrate that our proposed algorithm could extract effective features and have good performance in facial expression recognition.
Facial expression recognition plays an important role in human-computer interaction, and 55% information is transferred by facial expression in face-to-face human communication [
One of the effective methods for facial expression recognition is the subspace-based algorithm [
Recently, the nonnegative matrix factorization (NMF) was introduced into facial expression recognition [
For facial expression recognition, NMF and its variants vectorize the samples before factorization, which may lose the local geometric structures. However, the spatial neighborhood relationships within pixels are critical for image representation, understanding, and recognition [
On the other hand, the choice of classifier plays an important role for recognition. For facial expression recognition, nearest neighbor (NN) and support vector machine (SVM) are the commonly used methods [
In this paper, we propose a novel facial expression recognition algorithm based on discriminant neighborhood preserving nonnegative tensor factorization (DNPNTF) and ELM. It works well in the rank-one tensor space. The simple ELM is adopted to testify its effectiveness for facial expression recognition [
The rest of this paper is organized as follows. The mathematical notations are given in Section
In this paper, a tensor is represented as
The inner product of two tensors
The tensor product of two tensors
A
Here
The mode
In this section, we give a detailed description about the proposed DNPNTF algorithm. Instead of converting into vectors, it processes the samples in rank-one tensor space. The objective function of NTF is adopted, which could learn the parts-based representation and have the sparse property. To discover the spatial local geometric structure and the discriminant class-based information, a constraint is added in the objective function according to the manifold learning and graph embedding analysis. To guarantee the convergence, the project gradient method is used.
Given a
By minimizing (
To incorporate more properties into NTF, different constraints could be added into the objective function. The constraint form of objective function is
Now, we discuss the selection of
Here,
Now, the objective function of constrained NTF becomes
By solving the generalized eigenvalue decomposition problem, the graph embedding criterion in (
The most popular approach to minimize NMF or NTF is the multiplicative update method. However, it cannot ensure the convergence of the constraint forms of NMF or NTF. In this paper, the projected gradient method is used to solve DNPNTF.
The objective function of DNPNTF can be stated as
To find the optimal solution, (
Firstly, we discuss the calculation of
The differential of
And the partial differential for
According to (
To confirm the nonnegative of
Similarly, the update rule of the
Now,
Then we discuss the calculation of
For
According to (
The update step
And the final update rule of
Now
ELM is proposed by Huang et al. [
Given a training set
Equation (
In this section, we apply DNPNTF via ELM to facial expression recognition. We compare DNPNTF with NMF [
Illustration of the preprocess for original images.
Since the results of ELM may vary during each different execution, we repeat the execution for 5 times and take the average value as the final result. It is proved by theory analysis and experiments that the classification performance of ELM is affected by the hidden activation function and the number of hidden nodes [
The JAFFE database [
The average recognition rates of different feature extraction algorithms are shown in Figure
The recognition rate versus dimension curves achieved on the JAFFE database.
The top recognition rates of different algorithms with corresponding dimensions are illustrated in Table
The top recognition rate and the corresponding dimension achieved on the JAFFE database.
DNPNTF | NMF | DNMF | NTF | |
---|---|---|---|---|
Recognition rates (%) | 90.71 | 87.43 | 89.33 | 88.91 |
Dimensions | 98 | 16 | 66 | 101 |
Figure
Basis images obtained on the JAFFE database.
NMF
NTF
DNPNTF
Then we give the experiments to prove the effectiveness of DNPNTF via ELM. The average recognition rates of DNPNTF with ELM, NN, SVM, and SRC are given in Figure
The top recognition rate of different classifiers achieved on the JAFFE database.
ELM | NN | SRC | SVM | |
---|---|---|---|---|
Recognition rates (%) | 94.39 | 90.71 | 93.54 | 91.03 |
Dimensions | 55 | 98 | 63 | 101 |
The recognition rate versus dimension curves of DNPNTF by using different classifiers on the JAFFE database.
The Cohn-Kanade database [
The average recognition rates of different algorithms on the Cohn-Kanade database are shown in Figure
The top recognition rate and the corresponding dimensions achieved on the Cohn-Kanade database.
DNPNTF | NMF | DNMF | NTF | |
---|---|---|---|---|
Recognition rates (%) | 63.61 | 58.17 | 58.83 | 59.72 |
Dimensions | 101 | 76 | 26 | 120 |
The recognition rate versus dimension curves achieved on the Cohn-Kanade database.
Lastly, we give the experiments about different classifiers on the Cohn-Kanade database. The average recognition rates of DNPNTF via ELM, NN, SVM, and SRC are shown in Figure
The top recognition rate and the corresponding dimensions achieved on the Cohn-Kanade database.
ELM | NN | SRC | SVM | |
---|---|---|---|---|
Recognition rates (%) | 86.00 | 63.61 | 73.00 | 72.06 |
Dimensions | 51 | 101 | 116 | 106 |
The recognition rate versus dimension curves of DNPNTF by using different classifiers on the Cohn-Kanade database.
In this paper, a novel DNPNTF algorithm with the application to facial expression recognition was proposed, which adopts ELM as the classifier. To incorporate the spatial information and the discriminant class information, a discriminant constraint is added to the objective function according to the manifold learning and graph embedding theory. To guarantee the convergence, the project gradient method is used for optimization. Theoretical analysis and experimental results demonstrate that DNPNTF could achieve better performance compared with NTF, NMF, and its variant. Then the discriminant features generated by DNPNTF are fed into ELM to learn an optimal model for recognition. In our experiments, DNPNTF via ELM achieves higher recognition rate compared with NN, SVM, and SRC.
The authors declare that there is no conflict of interests regarding the publication of this paper.
This work was supported partly by the National Natural Science Foundation of China (61370127