Research Article An Improved Fractional-Order Variational Optical Flow Model Combining Structure Tensor

Dealing with problems of illumination changes in optical ﬂow estimation, an improved variational optical ﬂow model is proposed in this paper. The local structure constancy constraint (LSCC) is applied in the data term of the traditional HS (Horn & Schunck) optical ﬂow model to substitute the brightness constancy constraint. The fractional-order smoothness constraint (FSC) is applied in the smoothness term of the HS model. Then, the detailed calculation processes from the optical ﬂow model to the optical ﬂow value are explained. The structure tensor in LSCC is an image feature that is constant in the illumination changes scene. The fractional diﬀerential coeﬃcient in FSC can fuse the local neighborhood optical ﬂow vector into the optical ﬂow vector of the target pixel, which can improve the integrity of the motion region with the same motion speed. Combining LSCC with FSC, our improved optical ﬂow model can obtain an accurate optical ﬂow ﬁeld with clear outline in the illumination abnormity scene. The experimental results show that, compared with other optical ﬂow models, our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.


Introduction
Motion detection in the image sequence or video is one of the basic tasks of various image processing projects, which has been widely used in the research and engineering practice of 3D reconstruction, moving object segmentation, moving object tracking, video compression, automatic driving, and so on. e variational optical flow method is one of the most commonly used methods of motion detection. e HS optical flow model [1] is the most classical variational optical flow model. It is composed of a brightness constancy constraint and global smoothness constraint. e brightness constancy constraint requires that the intensity of pixels remains constant in the process of motion, while the smoothness constraint supposes that the motion speed of all pixels in the image changes smoothly. However, both constraints are based on ideal assumptions. In practical applications, these two constraints are often violated. Illumination changes in the scene will break the brightness constant constraint and the global smoothness constraint and lead to the blur of the edge between different motion regions.
Several other constraints on the data term were proposed to improve the performance of the variational optical flow model, such as gradient constancy constraint [2], Laplacian constancy constraint [3], and Hessian constancy constraint [4], but these constraints depend heavily on the intensity difference. Illumination invariant descriptors such as binarybased [5], real value-based [6,7], and neighborhood-based [8,9] were applied to substitute the brightness constancy constraint. But these descriptors fail to identify the edge between different motion regions. e structure tensor [10][11][12], which can embody the shape feature of the motion objective, is a local structure feature that does not change with the environmental factor. e local structure optical flow model (LSOFM) [13][14][15], which is established on the structure tensor feature, can improve the robustness in the illumination changes scene. But when a structure includes different motion parts, LSOFM would fail.
Because the fractional optical flow vector gradient can fuse the characteristics of the local neighborhood optical flow vector into the optical flow vector of the target pixel, the integrity of regions with the same motion speed is enhanced, so it can preserve the discontinuity of the optical flow field between different motion regions. e fractional-order variational optical flow model (FOVOFM) is proposed by Chen et al. [16], which uses the fractional-order smoothness constraint in the smoothness term of the HS model. e dual fractional order variational optical flow model (DFOVOFM) is proposed by Zhu et al. [17], which the fractional-order derivative is applied both in data term and smoothness term of the HS model. e fractional-order TV-L1 optical flow model was proposed in [18]. e fractional-order-based optical flow model neglects the structural information in image, so error would exist on the edge and some tiny motion regions.
An improved variational optical flow model is proposed in this paper. Our model is composed of LSCC and FSC. LSCC cannot only ensure the robustness of the model in scenes of illumination changes but also the contour in the image can be enhanced. FSC cannot only preserve the motion discontinuity between different motion regions but also weaken the influence of illumination changes on optical flow estimation. e model proposed in this paper combines the advantages of the two constraints and solves the illumination change problem and the motion discontinuity problem in optical flow estimation, and different motion parts in a structure can be identified. e contribution of the paper is as follows: (1) A novel optical flow model combining local structure tensor and fractional-order deviation is designed to improve the robustness of illumination changes (2) An applicable model simplification method for our model is presented

Related Work
e research of the optical flow model never stops. An optical flow model using full four-dimensional cost volume was proposed in [19], which applied structural features and stereo matching pipelines to yield high accuracy. To solve the problems caused by untextured areas, motion boundaries, and occlusions in optical flow estimation, post-hoc confidence measures [20] are applied to assess the per-pixel reliability of the flow. Occlusions have remained one of the key challenges. In [21], a symmetric optical flow model was proposed to address occlusion problems of optical flow estimation. In [22], an interpolation correspondence method is proposed to estimate the optical flow, but these methods are usually susceptible to nonrigid motion and large displacements; to solve this problem, robust interpolation of sparse correspondences is proposed in [23].
A coarse-to-fine scheme combined with an efficient random search strategy for efficient dense correspondence is proposed to estimate the optical flow in [24]. A coarse-to-fine PatchMatch with sparse seeds to sparse matches is applied to optical flow estimation in [25]. A novel differentiable neighbor-search up-sampling (NSU) module [26] is proposed to improve the coarse-to-fine technology for avoiding the optical flow error in edges, thin bars, and holes. A novel optical flow method related to the frequency domain that uses TV-wavelet regularization is applied to accurate optical flow estimation with the frequency-domain regularization in [27].
With the rapid development of artificial intelligence, deep learning-based methods have been widely employed in optical flow estimation. e FlowNet [28] demonstrated that CNNs can be used to estimate the optical flow. In [29], three major improvements were designed for the FlowNet: a subnetwork was elaborated on small displacements, a stacked architecture was applied to warp of the second image with intermediate optical flow, and a schedule of the training data. A new deep network architecture called recurrent all-pairs field transforms is designed to estimate the optical flow in [30]. A deformable cost volume neural network which can estimate the multiscale optical flow in a single high resolution was proposed in [31]. PWC-Net [32], which consists of pyramidal processing, warping, and a cost volume, is designed to estimate optical flow. e network uses the current optical flow estimate to warp the CNN features of the second image and then uses the warped features and features of the first image to construct a cost volume. e drawback of deep learning-based methods is large number of ground truth are needed, and retraining the model is needed for different application scenarios.

HS Optical Flow Model
e HS model can be written as follows: where u � (u, v) represents the optical flow vector, u is the optical flow value on x axis, v is the optical flow value on y axis, E data (u) represents the brightness constraint equation, E smooth (u) represents smoothness constraint equation, and λ is a smoothness parameter, and its value is decided by the image quality. In a blurry image with many noises disturbing, the value of λ should be big and vice versa. e brightness constraint equation based on the bright constancy constraint, which supposes that the brightness value of a pixel is constant during the motion, is as follows: (2) e smoothness constraint equation based on the global smoothness constraint, which supposes that the motion speed of all pixels changes smoothly, that is to say, the derivative of the speed is very small, is as follows:

Local Structure Constraint Equation
e structure tensor which is composed of local geometric features of image is an invariable feature to illumination changes, image rotation, image translation, and image scaling. To increase the robustness of the optical flow model, we use the local structure constancy constraint to substitute the brightness constant constraint.

Image Structure Tensor.
e local structure tensor of the two-dimensional image can be expressed as follows: where I represents the brightness function, ∇ � (z x , z y ) is the tangent vector along x and y axes, and T is the structure tensor of a pixel. But T is easily be affected by noise, so it is always enhanced by Gaussian filter G σ :

Local Structure Constancy
Constrain. e local structure constancy constraint (LSCC) is proposed in this paper to enhance the robustness of the optical flow model. LSCC is based on the assumption that the local structure in an image would not change during the motion: where T(x, y, t) represents the structure tensor function from a pixel (x, y) to t frame. en, the pixel will move from (x, y, t) to (x + u, y + v, t + 1), and (u, v) is the optical flow value, and T(x + u, y + v, t + 1) represents the structure tensor of t + 1 frame. Equation (6) means that the structure tensor will not change during the motion. Using the Taylor series expansion and ignoring the minimum term, we can get where T x , T y , and T t are the derivative of the structure tensor function on x, y, and t axes, represents the local neighborhood of the target point, l is the normalization constant, and w represents the weight function.
en, E data (u) can be rewritten as follows:

Fractional Smoothness Constraint Equation
e fractional smoothness constraint (FSC) is applied in our model to further improve the performance of optical flow estimation. We use the fractional-order derivative to substitute the integer-order derivative in the smoothness constraint equation of the HS optical flow model: where D α u represents α-order derivative of the optical flow vector u on x and y axes, D α v represents α-order derivative of the optical flow vector v on x and y axes, α is en, E smooth (u) can be designed as follows:

Improved Fractional Optical Flow Model
Combining LSCC with FSC, we construct our improved optical flow model; using the structure of the HS model, our optical flow model can be described as follows: Using the Euler-Lagrange equation to minimize the energy function (11), a similar procedure can be seen in [14,15]; then, we can get Mathematical Problems in Engineering where T xx (i, j), T xy (i, j), T xt (i, j), T yx (i, j), T yy (i, j), and T yt (i, j) are the second-order derivative of the structure tensor at x, y, and t axes: where D α * x D α x u, D α * y D α y u, D α * x D α x v, and D α * y D α y v denote the dual fractional-order derivative of optical flow value u and v on x and y axes, respectively: where w (α) 0 � 1 and w (α) k � (1 − ((α + 1)/k))w (α) k− 1 , k � 1, 2, . . ., L, which denote the size of the neighborhood. en, we can get where χ(i, j) represents a neighborhood of point (i, j), k ij is equal to max(|i − i|, |j − j|), u(i, j) and v(i, j) are the mean of the optical flow vector in the neighborhood, and u(i, j) and v(i, j) are the coordination of the optical flow vector on x and y axes of each pixel in the neighborhood. en, the optical flow estimation formula is as follows: where λ is the smoothness parameter, according to Huerst and Xie [33]. e value of parameter λ is determined by the image quality; when the image is dark, unidentified, blurry, or noisy, λ should be large. Otherwise, λ should be small. In our algorithm, we choose λ ∈ [5.0, 10.0] in relatively clear images, and λ ∈ [10.0, 15.0] in relatively blurry images. α is the order of fractional derivation in the smoothness term, according to Chen et al. [15], and α � 1.3 or 1.4. e accuracy in the case of α � 1.3 is relatively higher than in the case of α � 1.4, while the latter case needs less iteration times.
We can obtain the accurate optical flow value by repeated iteration utilizing the Gauss-Seidel method. e iteration will stop when the accuracy reaches the requirement (the difference of the optical flow value between two adjacent iterations is less than the threshold).

Experimental Result and Analysis
To testify the superiority and practicability of our optical flow model, Middlebury dataset [34], MPI_Sintel dataset [35], KITTI dataset [36], and an outdoor real scene video are used in experiments. Vertical comparisons are employed to explain the model evolution process; these models include HS model [1], LSCOFM [15], FOVOFM [16], and DFOVOFM [17]. ree widely used high performance models are employed for horizontal comparison: HAST [37], MDP_Flow [38], and PH_Flow [39]. All experiments are run in a PC equipped with Intel Xeon(R) X5600, 3.4 GHZ, and 4 GB memory. e operating system is 64 byte Windows 7, and the software platform is Matlab 10.1. Figures 1-3 show the color code map by optical flow estimation of different algorithms in the Middlebury database. ey are army sequence, grove sequence, and scheflera sequence, respectively. It can be seen in these figures that, from left to right, we can get more and more clear outlines, which means the optical flow results are getting more and more accurate. But, compared to the GT, some highlytextured thin outlines disappear or become adjoined in the results of our model, such as soldier's gun in army, the end of branches in Figure 2(e), and the rhizome parts of leaves in Figure 3(e). at is because using the fractional order derivative, the thin parts would be assimilated by the optical flow field of surrounding pixels. Table 1 shows the AEE/AAE of different algorithms in the Middlebury dataset. We can see that, from top to down, the error rate is becoming more and more small, and the error rate of our model is the lowest. Figures 4 and 5 show the color code map by optical flow estimation of different algorithms in the MPI_Sintel dataset. ey are Cave3 sequence and Shaman1 sequence, respectively.
e results presented show that our method can acquire clear and integral motion outlines under the low contrast scene (Cave3) and illumination insufficient scene (Shaman1). HAST can embody motion details such as the end of the cone in Cave3 and the boy's hair in Shaman1, but fails in large nontextured regions such as the tail in Cave3  Table 2 shows the AEE/AAE of different algorithms in the MPI_Sintel dataset. We can see in Table 2 that, in sequences with existing large displacement and occlusion problems, the accuracy of our method is lower than its components. When comes to sequences with illumination changes and illumination insufficient, the accuracy of our method is higher than other algorithms. e KITTI dataset is a challenging real-world dataset in which most sequences are affected by illumination abnormity, such as illumination changes, illumination uneven, illumination insufficient, and so on. e experimental results are shown in Figures 6-8. ey are sequence 00004, 00009, and 00011, respectively. We can see that our method obtains clear motion outlines in all image sequences. Despite the illumination uneven conditions, the tree outline can be seen clearly in 00004 and the car outline can be seen clearly in 00009. In sequence 00011 with the illumination normal scene, differences between different algorithms are small. Table 3 shows the AEE/AAE of different algorithms in the KITTI dataset. We can see that, in illumination abnormity scenes, the error rate of our model is the lowest, and the average accuracy of our model for the whole sequences in KITTI is the highest.         Mathematical Problems in Engineering Figures 9-12 show the color code map of optical flow estimation by different algorithms in an outdoor video. We can see that compared to LSOFM, DFOVOFM has less noise affected regions, but the outline is not so clear. Our model achieves better results with the most intact object outline and least noise affected regions than other models.
In Figure 10(d), different motion parts in the right sole can be seen, the optical flow field of the first half foot is pink, and the second half is yellow. e motion of one finger in the right hand in Figures 11 and 12 is different from the other part on that hand, and we can see different color regions in Figure 12(d).

Conclusion
An improved variational optical flow model is proposed in this paper to solve problems of illumination changes in optical flow estimation. e model applies LSCC and FSC in the data term and smoothness term of the HS model. en, the detailed calculation processes from the optical flow model to optical flow value are explained. Experimental results in three public datasets and an outdoor video show that our model is superior to other algorithms in illumination abnormity scenes. e disadvantage of our model is that it cannot embody the motion of thin parts. Experimental results testify the performance of our model. Finally, we have conclusions that our model is more suitable for the illumination changes scene and can be employed in outdoor motion detection projects.

Data Availability
Middlebury dataset, MPI_Sintel dataset, and KITTI dataset can be found in references [17][18][19]. Our own outdoor video can be easily obtained by a normal RGB camera.

Additional Points
e model proposed in this paper has some limitations: When the difference between the RGB value of background and foreground all less than "1," the model would fail; because of the large amount of computation, the running speed of the algorithm is 30 seconds per frame, so it cannot be used for real-time application; when the average brightness is less than 5, the optical flow estimation result of the algorithm would not be accurate.

Conflicts of Interest
e authors declare that they have no conflicts of interest. Mathematical Problems in Engineering 9