Conventional incremental PCA methods usually only discuss the situation of adding samples. In this paper, we consider two different cases: deleting samples and simultaneously adding and deleting samples. To avoid the NPhard problem of downdating SVD without right singular vectors and specific position information, we choose to use EVD instead of SVD, which is used by most IPCA methods. First, we propose an EVD updating and downdating algorithm, called EVD dualdating, which permits simultaneous arbitrary adding and deleting operation, via transforming the EVD of the covariance matrix into a SVD updating problem plus an EVD of a small autocorrelation matrix. A comprehensive analysis is delivered to express the essence, expansibility, and computation complexity of EVD dualdating. A mathematical theorem proves that if the whole data matrix satisfies the lowrankplusshift structure, EVD dualdating is an optimal rank
Principal component analysis (PCA) [
In order to obtain the optimal set of normal orthogonal basis, which endues PCA with the minimal reconstruction error, the batchmode PCA can be achieved in two ways: the eigenvalue decomposition (EVD) of the data covariance matrix and the singular value decomposition (SVD) of the data matrix. Both approaches have a high computational cost and a mass demand of storage, in the case of a highdimensional and largescale dataset. In practical applications, not all the observations are available before training. Especially in online usage, samples arise sequentially along with time. In these situations, the batchmode PCA does not satisfy the demand for realtime process due to its requirement to recompute the EVD or SVD of the whole data every time.
To solve this issue, incremental learning has been investigated for more than two decades in both applied mathematics and machine learning community, whose task is to update the learning results without reexecuting the whole process when adding new data points. Various effective incremental PCA (IPCA) methods have been proposed.
In a period of knowledge explosion, the fast growing information is usually adulterated with mock, invalid, or expired data. The presence of a few deviated samples might tremendously contaminate the solved model, such as principal directions in PCA. The overdue instances, which can be regarded as outliers compared to unexpired instances in some degree, could reduce the accuracy of data model. Therefore, for an intelligent learning system, the only function to admit new instances is not enough, but the capability to eliminate aberrant samples is also necessary. This is the aim of decremental learning. Comparing with IPCA, decremental PCA (DPCA) did not receive adequate attention in the literature. Only a few methods have been proposed in the last ten years. Besides, there is no incremental decremental algorithm of subspace learning to the best of our knowledge. Similar works are only about support vector machine (SVM) [
Because the essence of PCA is SVD or EVD in the mathematical form, the task of incremental PCA and decremental PCA is equivalent to updating and downdating SVD or EVD. In existing methods, most IPCA approaches adopt similar strategies via updating SVD. However, these tactics based on SVD may be impossible to be implemented for decremental PCA. Lorenzelli and Yao [
Based on the demand on incremental decremental learning and the difficulty of decremental learning in the analysis above, we introduce a novel online subspace method for simultaneous incremental decremental learning. The contributions in this paper are as follows.
To avoid the problem of lacking right singular vectors in decremental learning, we utilize EVD instead of SVD and propose a dualdating algorithm for eigenspace, that is, EVD dualdating, which can accept and delete samples at the same time. Our algorithm transforms the EVD updating and downdating of the covariance matrix into a SVD updating problem plus an EVD of a small autocorrelation matrix. To the best of our knowledge, it is the first attempt of simultaneous incremental decremental subspace learning and has a simpler and unitized mathematical form, which theoretically guarantees a better performance than the conventional multiplestep implementation.
Several theoretical and computational analyses are presented to further explore the property of EVD dualdating, including the essence and geometric explanation of EVD dualdating, expansive forms of EVD dualdating for data revising and weighted updating, the computation complex of EVD dualdating, a mathematical theorem which demonstrates the optimality of EVD dualdating in the sequential mode if the data matrix satisfies
It is proofed that the change of mean caused by adding or deleting samples in the varyingmean PCA can be transformed into adding and deleting several equivalent vectors in the zeromean PCA. Thus, three online PCA algorithms are derived based on EVD dualdating to cope with changeable mean: incremental PCA (EVDDIPCA), decremental PCA (EVDDDPCA), and incremental decremental PCA (EVDDIDPCA).
The remaining of this paper is organized as follows. Section
Over the past few decades, many efficient incremental PCA methods have been proposed. Generally, existing IPCA algorithms can be divided into three categories. The first category updates eigenvectors without any matrix decomposition. The typical method is the candid covariancefree IPCA (CCIPCA) [
Weng et al. [
Hall et al. [
Except these two approaches above, other incremental PCA methods are based primarily on SVD. Levy and Lindenbaum [
Zha and Simon [
Although a great deal of research has been accomplished about incremental subspace learning, the research on decremental learning is still inadequate in the literature. The merging and splitting eigenspace model developed by Hall et al. [
Beside the accuracy and efficiency, the severest problem faced by SVDbased decremental methods is that it is a NPhard problem without the position information of deleted samples in the data matrix, which might be not obtainable in many practical applications.
In Section
Given a data matrix
When new samples
To solve this problem, Zha and Simon [
(1) Compute QR decomposition,
(2) Compute the rank
(3) The rank
In this section, a thorough discussion of the proposed dualdating algorithm for EVD is presented. Dualdating means updating and downdating together; in other words, we consider the situation of adding and deleting samples simultaneously.
Given a data matrix
Now some old samples
The basic procedure of the proposed EVD dualdating algorithm is as follows. Let
Thus, the covariance matrix of
The basic idea of EVD dualdating is to transform the dualdating problem into a SVD updating problem plus an extra process with a small computation complexity. Firstly, consider the matrix
Take (
Let
Because
Finally, the rank
By (
Although the basic procedure of our EVD dualdating algorithm is given, one problem still remains unsolved: the assumed right singular vectors
Consider the results of the SVD updating algorithm on the rank
Take (
From (
The detailed procedure of EVD dualdating has been presented above. To sum up, the pseudocode of our EVD dualdating algorithm is described in Algorithm
(1) Let equivalent adding data matrix
(2) Compute QR decomposition,
(3)
(4) Let
(5) Compute the EVD of
(6) The rank
In this section, we first analyze the mechanism of EVD dualdating for incremental and decremental learning. Second, some extended forms of EVD dualdating are given for particular uses. Third, the computation complexity on the proposed EVD dualdating algorithm is presented. Fourth, the optimality of EVD dualdating in the sequential usage is demonstrated. Finally, we discuss how to determine the optimal rank
For convenience, when analysing the essence of incremental and decremental learning based on EVD dualdating, we only consider the pure updating or downdating situation and denote the changed matrix as
According to the procedure of EVD dualdating, two key decompositions are the SVD updating of the equivalent adding matrix
Then, divide the columns of
So the matrix
Now, let us observe the situation from the view of geometry shown in Figure
Visualization of EVD dualdating.
To sum up, the aim of EVD dualdating is to obtain the projection matrix caused by the change of sample set, and the essence of EVD dualdating is to transform the EVD of a varying covariance matrix in the data space to the EVD of a varying autocorrelation matrix in a dimensionreduced row space.
From the deduction of EVD dualdating, it can be seen that nearly no restriction is imposed on
The standard dualdating mode for EVD dualdating is adding and deleting samples synchronously. As we mentioned before, when
Expansions of EVD dualdating.
Old data  New data 

Miscellanea  

Update 






Downdate 






Dualdate 






Revise 






Weighted update 




Before analyzing, we define some signs to simplify the representation:
Computation complexity of EVD dualdating and other updating/downdating methods.
Updating  Downdating  

Decomposition  Transformation  Decomposition  Transformation  
MSES  SVD 

SVD 

DCSSVD  QR 


MSVD  QR 

QR 

AIPCA  QR 

QR 



Dualdating  
decomposition  Transformation  


EVDD  QR 

In the pure updating or downdating mode, there are two matrix decompositions in our EVDDualdating algorithm, one more than other pure updating and downdating methods. This may cause EVD dualdating slower than other methods. But taking the dimension and the transformation cost into account, the efficiency of EVD dualdating is close or even better, comparing to other methods. The main advantage of our algorithm can be reflected in the dualdating mode. As the only method achieving simultaneous updating and downdating, EVD dualdating can avoid many repeating processes and decrease the cumulative error. An experimental comparison of efficiency and accuracy on our EVD dualdating and other incremental and decremental methods is presented in Section
In many online applications, it is impossible to store the original data because of the limitation of the physical medium and the consideration about efficiency. Described in a mathematical form, this means that the original data matrix
Zha and Simon [
Let
The lemma above indicates that for the rank
Given a matrix
In the deduction above, the rank of subspace is assumed to be a fixed number
Supposing the truncation operation is not yet executed in steps 4 and 6 of Algorithm
In the deduction of EVD dualdating in Section
Principal component analysis (PCA) is one of the most popular multivariate analysis and dimension reduction methods. The goal of PCA is to find a set of normal orthogonal basis, socalled principal components, which has the best reconstruction performance in the sense of minimum mean squared error (MMSE).
Given a data matrix
When confronting a huge dataset with a high dimension, both batchmode methods, no matter EVD or SVD, cost tremendous time and storage. Besides, for an online learning system, it has to face an awkward circumstance that not all the instances are available before training, or some expired instances need to be deleted after training. Obviously, these problems exceed the ability of the batchmode PCA. The incremental and decremental PCA is a natural solution.
In this section, we consider EVD dualdating with a timevarying mean, and deduce the incremental decremental PCA formula based on EVD dualdating. As mentioned before, EVD dualdating degenerates into SVD updating without right singular vectors in the updating mode, so EVDDIPCA is actually the same as the extended sequential KL algorithm. Nonetheless we still present it in this paper for integrity. The interested reader can find more details in the reference paper [
The key idea of EVDDbased incremental and decremental PCA algorithm is that centralizing the original samples, the added samples, and the deleted samples separately and utilizing some meanrevising vectors to keep the covariance matrix equal to the original one. The methods of determining these meanrevising vectors are introduced in Lemmas
new samples
(1) Update the sample number and mean,
(2) Compute the extra added sample,
(3) Equivalent added data matrix,
(4) Compute
deleted samples
(1) Update the sample number and mean,
(2) Compute the extra deleted sample,
(3) Equivalent deleted data matrix,
(4) Compute
added samples
(1) Update the sample number and mean,
(2) Compute the extra sample,
(3) Equivalent added data matrix,
(4) Equivalent deleted data matrix,
(5) Compute
Let
Let
Let
As an important approach of dimension reduction, PCA is utilized as the preprocessing method for many other machine learning methods, and the feature extraction method in other applications. Because these methods usually work in the subspace of PCA, there is a great demand to achieve simultaneous online incremental decremental subspace learning and data reconstruction. Artac et al. [
In this section, experiments of the proposed algorithms based on EVD dualdating are presented, compared with other classic methods. Because incremental PCA has been discussed a lot in the earlier literature and the proposed EVDDIPCA is actually equivalent to the extended sequential KL algorithm, we do not verify IPCA methods in this paper any more. The interested reader can find the performance analysis and comparison in relative papers [
In order to verify the performance and efficiency of the proposed EVDDDPCA and EVDDIDPCA, four datasets are used, including cases of both high dimension and huge size. The FERET [
Dataset and configuration for DPCA.
Data set  Dimension  Class  Delete  Sample  Training  Testing 

FERET  92 × 112  120  40  6  4  2 
AR  92 × 112  119  39  8  6  2 
Yale B  25 × 30  90  30  45  30  15 
COIL100  25 × 25  100  40  72  42  30 
To compare the performance of decremental learning, we implement the proposed EVDDDPCA algorithm with the batchmode PCA, MSES [
execution time;
weighted angle between PCs of the batchmode PCA and DPCA methods:
recognition rate.
Recalling the analysis of computation complexity in Section
CPU time of DPCA methods on different datasets,
FERET
AR
Yale B
COIL100
In order to evaluate the accuracy, the angles between the resulting PCs of DPCA methods and the batchmode PCA can be adopted. But, we choose the weighted angles by their corresponding eigenvalue, which are more suitable for evaluation because they emphasize the importance of the leading PCs. Figures
The first 50 weighted angles of DPCA methods on different datasets,
FERET
AR
Yale B
COIL100
Error of the first 50 weighted angles of DPCA methods on different datasets,
FERET
AR
Yale B
COIL100
From these figures, our proposed EVDDDPCA algorithm performs the best accuracy of the eigenvector estimation. The accuracy of principal direction depends on the estimation of mean and the cutoff error. The error of mean will cause a bias of the origin for data centralizing, which may cause the direction of the resulting basis totally different in the worst situation. The cutoff error accumulates in the sequential process, so the more times the truncation happens, the lower accuracy the final result remains. The method to update the mean is the same in MSES and EVDDDPCA, whose estimate is equal to the true mean. In DCSSVD, the new mean is updated via the right singular vectors
In the recognition experiment, the resulting PCs are used as the projection matrix to project the testing image to the subspace, then minimum distance classifier (MDC) is utilized for recognition. The advantage of MDC in our online application is that only the mean of each class in the projection subspace needs to be saved. The distance between a sample
Figures
Recognition rate of DPCA methods on different datasets,
FERET
AR
Yale B
COIL100
To compare the performance of incremental decremental subspace learning methods, we implement the proposed EVDDIDPCA algorithm with the batchmode PCA, MSES [
The datasets for IDPCA is the same as in the DPCA experiment and the configuration is shown in Table
Dataset and configuration for IDPCA.
Data set  Dimension  Class  Delete  Add  Sample  Training  Testing 

FERET  92 × 112  120  40  40  6  4  2 
AR  92 × 112  119  39  39  8  6  2 
Yale B  25 × 30  90  20  20  45  30  15 
COIL100  25 × 25  100  30  30  72  42  30 
Figures
CPU time of IDPCA methods on different datasets,
FERET
AR
Yale B
COIL100
Figures
The first 50 weighted angles of IDPCA methods on different datasets,
FERET
AR
Yale B
COIL100
Error of the first 50 weighted angles of IDPCA methods on different datasets,
FERET
AR
Yale B
COIL100
Figures
Recognition rate of IDPCA methods on different datasets,
FERET
AR
Yale B
COIL100
Besides, one important advantage of EVDDDPCA, not reflected by these DPCA and IDPCA experiments, is that the specific position information of deleted and added samples is not needed, which are necessary for DCSSVD, AIPCA, and MSVD.
In this experiment, the selection of the dimension
Results of rank
This paper focuses on the problem of online incremental/decremental subspace learning and reports a novel dualdating algorithm of EVD, namely, EVD dualdating. Different from previous works, the proposed EVD dualdating algorithm can renew the EVD of a data matrix while adding and deleting samples simultaneously. With EVD dualdating, IPCAEVDD, DPCAEVDD, and IDPCAEVDD are presented to handle the changeable mean, where the variation is equivalent to add and delete several additional vectors in the case of zeromean PCA. Plenty of comparative experiments on both realworld and artificial databases demonstrate that our EVD dualdating algorithm has a significant better approximation accuracy and computational efficiency than other stateoftheart incremental and decremental PCA methods.
By definition,
And, the scatter matrix of
By definition,
And, the scatter matrix of
The authors declare that there is no conflict of interests regarding the publication of this paper.
This work is supported by the National Natural Science Foundation of China (Grants nos. 61175028, 61375007), the Ph.D. Programs Foundation of Ministry of Education of China (Grants nos. 20090073110045), and the Shanghai Pujiang Program (Project no. 12PJ1402200).