We address object tracking problem as a multitask feature learning process based on low-rank representation of features with joint sparsity. We first select features with low-rank representation within a number of initial frames to obtain subspace basis. Next, the features represented by the low-rank and sparse property are learned using a modified joint sparsity-based multitask feature learning framework. Both the features and sparse errors are then optimally updated using a novel incremental alternating direction method. The low-rank minimization problem for learning multitask features can be achieved by a few sequences of efficient closed form update process. Since the proposed method attempts to perform the feature learning problem in both multitask and low-rank manner, it can not only reduce the dimension but also improve the tracking performance without drift. Experimental results demonstrate that the proposed method outperforms existing state-of-the-art tracking methods for tracking objects in challenging image sequences.
Object tracking is one of the well-known problems in computer vision with many applications including intelligent surveillance, human-computer interface, and motion analysis. In spite of significant success, designing a robust object tracking algorithm remains still challenging issue due to factors from real-world scenarios such as severe occlusion, scale and illumination variations, background clutter, rotations, and fast motions.
An appearance model-based tracking method, which evaluates the likelihood of an observed image patch belonging to object class, considers some critical factors such as object representation and representation scheme. The object representation can be categorized by adopted features [
Various object tracking methods based on object appearance models can handle only moderate changes and usually fail to track when the object appearance significantly changes. As a result, an appearance model learning process is required for robust object tracking under challenging issues such as object deformation.
Recently, sparse representation-based
Bao et al. applied the accelerated proximal gradient (APG) [
The low-rank sparse tracking (LRST) was proposed by representing all samples using only a few templates [
In spite of these improvements in MTT and LRST in the particle filter framework, the computational cost increases with the number of particles. Furthermore, MTT regards the sparse representations of sampled particles as independent state data without considering relationships between particles. For this reason, MTT may not work if the particles are drawn from specific probability distribution. The LRST-based methods are difficult to apply directly for object tracking in online video processing due to its structural computation complexity such as nuclear norm minimization.
To solve the above mentioned problems, we propose a novel object tracking algorithm based on multitask feature learning using joint sparsity and low-rank representation. We assume that the object representation can be incrementally optimized in the robust principal component analysis (RPCA) framework. The RPCA can be performed by decomposing the observations as the sum of a low-rank matrix and a sparse matrix; thus it can successfully recover the intrinsic subspace structure from corrupted observations. We extract features with low-rank representation within a few frames. After obtaining the subspace basis of object features, the features represented by all possible low-rank and the sparse property are learned using a variant of multitask feature learning framework. Finally, a novel incremental alternating direction method- (ADM-) based low-rank optimization strategy is efficiently applied for update of sparse error and features. The low-rank optimization problem for learning multitask features can be achieved by a few sequences of efficient closed form updating operation for the optimal state variables of object tracking.
RPCA [
The assumption under RPCA framework [
The object tracking problem can be solved under the RPCA framework. Suppose we have
In feature learning literature, the multitask feature learning [
The goal of the proposed object tracking framework is to search particles that have the most similar feature to previous tracking result. Since particles are densely sampled around the current object state, in order to estimate the optimal low-rank features in iterative steps, we focus on all possible nonsmooth sparse errors, joint sparsity of features based on low-rank, and
In the proposed object tracking formulation in (
In order to solve this objective function in (
The above mentioned problem can be minimized using the conventional inexact augmented Lagrange multiplier (IALM) method [
To solve the objective function given in (
The proposed strategy for given variable update is performed by computing sparse coefficients based on the low-rank representation with updated error and jointly sparse features. Thus, we first update the sparse error
For updating low-rank variable
We update the
The ADM can perform minimization with its alternative property; thus we formulate the objective function for updating sparse coefficient
Then, we update Lagrange multipliers as
The convergence of the proposed object tracking algorithm can be guaranteed using the Karush-Kuhn-Tucker (KKT) point approach by referring to linearized ADM with adaptive penalty (LADMAP) approach [
In many problems,
For updating the penalty parameter
The iterations equations (
The above two stopping criteria are based on the KKT conditions of problem (
(1) Input: A data set of (2) Output: Optimal coefficient (3) Initialization: Set
(4) Fix other variables and update Fix other variables and update Fix other variables and update Fix other variables and update Update Update (5)
Given a set of observed images at the
In the motion model, we regard the state variable
The solution of the proposed algorithm in Algorithm
The likelihood function can be formulated by the reconstruction error given in [
This likelihood function, however, cannot deal with severe appearance deformations. For this reason, the likelihood function is modified by an additionally labeled penalty constraint about the similarity in neighboring error matrices for the deformed region of the object such as
The proposed object tracking algorithm is implemented and tested using ten challenging sequences. Major challenging issues in the test sequences include scale variation, shape deformation, fast motion, out-of-plane rotation, background clutter, object rotation, occlusion, illumination variation, out of view, motion blur, and low resolution. The proposed method is compared with a number of state-of-the-art tracking algorithms such as SMTT [
The proposed object tracking algorithm is implemented in MATLAB and processes 1.5 frames per second on a Pentium 2.7 GHz dual core PC without any hardware accelerator such as GPU. For each test sequence, the initial location of the object is manually selected in the first frame. Each image sample from the target and background is normalized to a
Regularization parameters in (
We fix
We test different combinations of
For quantitative performance comparison, center pixel error evaluation method is used and overlap ratio criterion is computed. The center pixel error represents the distance between the predicted and the ground truth center pixels. Figure
Center pixel error. This figure shows center pixel error for ten test sequences. The proposed method is compared with six state-of-the-art methods.
Jogging 1
Jogging 2
Football 1
Freeman 3
Couple
Crossing
Liquor
Car Scale
Subway
Skiing
The overlap ratio criterion represents the ratio between the number of frames for a specific object to be completely tracked and the total number of frames in the image sequence. In order to decide whether the object is successfully tracked, we employ the overlapped score defined in [
Overlap ratio evaluation. This figure shows overlap ratios for ten test sequences. The proposed method is compared with six state-of-the-art methods.
Jogging 1
Jogging 2
Football 1
Freeman 3
Couple
Crossing
Liquor
Car Scale
Subway
Skiing
Jogging 1 and Jogging 2 sequences include a variety of critical conditions, such as disappearing object, out-of-plane rotation, and deformation. Object deformation and out-of-plane rotation make all the existing methods except ASL [
The moving object in the Football 1 sequence undergoes in- and out-of-plane rotation and background clutter. Experimental results demonstrate that the proposed method achieves the best tracking performance in this sequence. Other methods cannot avoid drift at some instances in the neighborhood of the
There is a blurry scale variation in Freeman 3 and Car Scale sequences. For this reason, it is difficult to predict the location of the moving object. Furthermore, it includes drastic appearance change caused by motion blur. The proposed method performs well since it can adapt to the scale and appearance change of the object and overcome the influence of motion blur by using joint sparsity-based object appearance representation.
In addition to scale variation and in- and out-of-plane object rotations, Couple and Crossing sequences include a critical factor such as fast motion. The fast motion can be involved by both camera and object. For tracking an object in Crossing sequence, the proposed method and SPCA [
The Liquor sequence contains severe motion blur, scale and illumination variation, and fast motion. Although full occlusion occurs by a bottle with the similar color, the proposed tracking method successfully tracks the object without drifting.
In the Subway sequence, object's information is lost by discriminant factor such as background clutter and the total occlusion. The proposed method can successfully track the moving object since it preserves the sparsity of object appearance with optimal sparse coding using low-rank feature representation with considering the joint sparsity. Other trackers except ILRFT [
The Skiing sequence contains severely deformed object. The proposed method successfully performs tracking due to its novel object likelihood function defined in (
Object tracking results using seven different algorithms on ten test sequences. They are Jogging 1, Jogging 2, Football 1, Freeman 3, Couple, Crossing, Liquor, Car Scale, Subway, and Skiing, respectively, from the top row to the bottom row.
In this paper, we present an effective, robust object tracking method using a multitask feature learning-based low-rank representation with joint sparsity. In order to overcome the limitation of existing sparse representation-based object tracking methods, we employ the novel optimization process of low-rank representation of objects by using a recently proposed model minimization method. The efficient subspace learning-based sparse coding and simultaneous update method of both optimal sparse codes and error matrix can be appropriately updated in the process of tracking in case of severe appearance variation. Experimental results demonstrate that the proposed method can successfully track objects in various video sequences with critical issues such as occlusion, deformation, plane rotations, background clutter, motion blur, scale and illumination variations, and fast motion.
The authors declare that there is no conflict of interests regarding the publication of this paper.
This work was supported by the MSIP, Korea, under the ITRC Support Program (NIPA-2014-CAU) supervised by NIPA, in part by the Technology Innovation Program (Development of Smart Video/Audio Surveillance SoC and Core Component for Onsite Decision Security System) under Grant 10047788, and by the Ministry of Science, ICT and Future Planning as Software Grand Challenge Project (Grant no. 14-824-09-002).