^{1}

^{1}

^{1}

^{1}

^{1}

^{2}

^{3}

^{1}

^{2}

^{3}

Thanks to the rapid development of hyperspectral sensors, hyperspectral videos (HSV) can now be collected with high temporal and spectral resolutions and utilized to handle invisible dynamic monitoring missions, such as chemical gas plume tracking. However, using such sequential large-scale data effectively is challenged, because the direct process of these data requires huge demands in terms of computational loads and memory. This paper presents a key-frame and target-detecting algorithm based on cumulative tensor CANDECOMP/PARAFAC (CP) factorization (CTCF) to select the frames where the target shows up, and a novel super-resolution (SR) method using sparse-based tensor Tucker factorization (STTF) is used to improve the spatial resolution. In the CTCF method, the HSV sequence is seen as cumulative tensors and the correlation of adjacent frames is exploited by applying CP tensor approximation. In the proposed STTF-based SR method, we consider the HSV frame as a third-order tensor; then, HSV frame super-resolution problem is transformed into estimations of the dictionaries along three dimensions and estimation of the core tensor. In order to promote sparse core tensors, a regularizer is incorporated to model the high spatial-spectral correlations. The estimations of the core tensor and the dictionaries along three dimensions are formulated as sparse-based Tucker factorizations of each HSV frame. Experimental results on real HSV data set demonstrate the superiority of the proposed CTCF and STTF algorithms over the comparative state-of-the-art target detection and SR approaches.

Hyperspectral imaging has been one of the most popular research fields due to its ability of identifying the materials from very high spectral resolution and coverage. In the last decade, researchers focused on the processing and application of hyperspectral image (HSI), such as denoising [

Basically, target detection is a kind of binary classifier with the purpose of labeling every image pixel as a target or background. In HSIs, pixels with a significantly different spectral signature from their neighboring background pixels are defined as spectral anomalies. Anomaly detectors are statistical or pattern recognition methods used to detect distinct pixels that differ from the background. It is worth mentioning that, in spectral anomaly detection approaches [

As mentioned before, the HSIs often suffered from low spatial resolution. To acquire an HSI, the number of sun photons in each spectral band has to be greater than a minimum value, and the number of spectral bands is so huge in an HSI that the spatial resolution has to be sacrificed. Therefore, super-resolution (SR) techniques have aroused great interest in the last decade. Generally, the SR methods of HSI can be classified into four categories: Bayesian [

However, the matrix factorization based schemes cannot fully exploit the spatial-spectral correlations of the HSIs. It is believed that considering HSIs as tensors is better because an HSI can be naturally expressed as a third-order tensor. In this paper, a detection algorithm based on cumulative tensor CP factorization (CTCF) is proposed. The sequential HSV data is expressed as a four-dimensional (4D) cumulative tensor; factor matrices are obtained by decomposing original 4D tensor using CP factorization. When a new frame presents and is added to the time dimension of the original tensor, this 4D cumulative tensor is updated together with the factor matrices. Consequently, a CP tensor approximation of the new frame is computed by updated factor matrices and the fitness between the new frame and the approximation is calculated. After comparing the fitness to a preset threshold, we can make the decision that whether the new frame continues to be used to update the cumulative tensor or the new frame is the key-frame where the target presents. CTCF-based method exploits not only the spatial-spectral correlations of the HSIs by applying tensor model, but also the temporal correlation between adjacent frames of the HSV.

On the other hand, tensor-based analysis has also been widely used in HSI super-resolution [

The remainder of this paper is organized as follows.

In this paper, vectors are denoted by boldface lowercase letters

An important calculation between a tensor and a matrix is the

Given the definition of

For continuous multiplication of a tensor and matrices in distinct modes, the result is not affected by the multiplication order, described by

If the modes are equivalent, equation (

Suppose that

The matricization form of equation (

Moreover, given the tensor

The definition of rank-one tensor is introduced at last. The

CANDECOMP/PARAFAC (CP) factorization decomposes a tensor into a sum of component rank-one tensors [

CP factorization is illustrated in Figure

CP factorization of a third-order tensor.

The factorization result can be expressed by factor matrices of three dimensions. Factor matrices refer to the combination of the vectors from the rank-one components; i.e.,

Following [

On the basis of factor matrices, the mode-

Tucker factorization is another popular tensor decomposing approach [

The Tucker factorization is illustrated in Figure

Tucker factorization of a third-order tensor.

In this subsection, the optimization problem of updating factor matrix is presented, followed with the proposed cumulative tensor CP factorization (CTCF) of third-order tensors. It is then extended to

Flowchart of CTCF-based detection method.

Similar to equation (

The Alternating Least Squares (ALS) algorithm is often applied to obtain factor matrices by solving the following optimization problem:

When the tensor updates, the new tensor can be computed by the updated factor matrices which are given by equation (

Generally, an image is a second-order tensor; then sequential images form a third-order tensor, i.e., a video, adding a temporal dimension on two spatial dimensions. When a new video frame presents and is added to the time dimension of the original tensor, it is defined as a three-dimensional (3D) cumulative tensor. With the number of new frames increasing, the 3D cumulative tensor updates frame by frame.

In conventional CP tensor approximation, whenever a new frame of image is added in the time dimension, ALS algorithm needs to be reused to approximate the new cumulative tensor, which is a time consuming process. In addition, the temporal correlation between neighboring frames is not exploited in the decomposition of the cumulative tensor. This paper proposes CTCF to update the CP factorization of original cumulative tensor, obtain the updated factor matrices, and approximate the new frame.

Given an original 3D cumulative tensor

The updating process is operated in an alternating way. Firstly, temporal dimensional factor matrix

Secondly, factor matrix

Derive

To simplifyequation (

For computing

Since

The update of

Finally, the update of factor matrix

To make the process clearer, the proposed CTCF of third-order tensor is summarized by Algorithm

Step 1: new tensor is added in the time dimension and

Step 2: decompose

Step 3: update

Step 4: update

Step 5: update

Step 6: estimate

On the basis of

Similar to

We also separate original part from new added part; i.e.,

The original part is minimized by fixing the first

The updates of nontemporal dimensional factor matrices

In HSV, the sequential data is expressed as a 4D cumulative tensor; the temporal dimension increases with new frames are added in. Whenever a new frame presents, the results of original cumulative tensor CP factorization are updated to obtain the factor matrices of the new cumulative tensor, and the CP tensor approximation of the newly added frame is obtained at the same time. If the target is absent, the CP tensor approximation will lead to a small error, since the background information is similar between adjacent frames. On the contrary, if the error is large, the target is likely to present. We define the fitness between the new frame and its approximation in

The original 4D cumulative tensor is denoted by

We define the

If the target does not appear, the approximation error is small and the result of fitness is large. Given a preset threshold

If the target appears, the approximation error is large and the fitness is smaller than

The target of each frame will be shown in 2D form by taking the maximum value of every spectrum. In this way, the proposed CTCF-based detection method can extract not only the key-frames where the target presents, but also the approximate region of target in every key-frame. The flowchart of the proposed method is shown in Figure

In

In this subsection, HSIs are represented as 3D tensors with three indexes (

There are two significant characteristics of HR-HSIs [

Sparse-based tensor Tucker factorization of an HSI.

The LR key-frame of HSV

Since

On the basis of equation (

Since the optimization problem in (

The proposed STTF-based SR algorithm is summarized in Algorithm

Initialize

Initialize

Initialize

Step 1: update

Step 2: update

Step 3: update

Step 4: update

Estimate

To highlight the advantages of HSIs, we choose invisible gas plume to be the target. The proposed algorithms can be extended to other types of data reasonably. In this section, the HSV data set is acquired by the infrared imaging spectrometer “HyperCam-LW.” Sulfur hexafluoride (SF_{6}) is chosen to be the target, since it is a kind of odorless and colorless gas plume with a distinct absorption peak in LWIR range. The HSV data set consists of 60 infrared hyperspectral frames with the size of

In SR method, only the middle

For CTCF-based detection method, we compare it with two representative methods: MSD (matched subspace detector) [

For detection methods, receiver operating characteristic (ROC) curves [

For SR algorithms, since we directly process the LR-HSI, there is no original HR-HSI (i.e., the ground truth) for reference. Thus, some popular quantitative metrics are not available, such as RMSE (root-mean-square error) [

Super-resolution aims to introduce more useful information into images, so we may measure the performance of SR methods by calculating the contained information in the experimental results. The entropy is indicated as

The probability of a pixel

Another assessment to measure the performance of super-resolution is the change of the amount of detailed information in the image. We may evaluate the experimental results by average gradient, since it can reflect the ability of expressing the details and measuring the clarity of the image. The gradient increases if the greyscale level rate in one direction of the image varies quickly. The average gradient is formulated as

Besides, the visual quality of output images is an important qualitative metric.

In MSD, we pick 463 spectrums of gas target and 846 spectrums of background from the 12th frame of HSV to build up the training set. The size of the target subspace and background space is ^{−8}; in update stage, the threshold of fitness is 0.9. In the proposed STTF-based SR method, the number of iterations is 5; the parameter

In this subsection, we show the experimental results of the various methods for detection and super-resolution.

After processing the HSV by the proposed CTCF-based method, we compute the values of Frobenius norm of each frame, which are presented in Figure

The value of Frobenius norm of each frame.

The comparison of ROC curves of three detection methods: (a) frame 13; (b) frame 23; (c) frame 34; (d) frame 44.

The ROC curves of 39 key-frames by three detection methods: (a) MSD; (b) CMF; (c) CTCF.

Detection quantitative results (AUC value) of the test methods on key-frames.

Frame | MSD [ | CMF [ | CTCF |
---|---|---|---|

12 | 0.9655 | 0.9980 | |

13 | 0.8462 | 0.9980 | |

14 | 0.8878 | 0.9981 | |

15 | 0.8189 | 0.9965 | |

16 | 0.8734 | 0.9977 | |

17 | 0.9348 | 0.9946 | |

18 | 0.5792 | 0.7477 | |

19 | 0.7894 | 0.9958 | |

20 | 0.9336 | 0.8991 | |

21 | 0.8222 | 0.9966 | |

22 | 0.9001 | 0.9915 | |

23 | 0.8388 | 0.9945 | |

24 | 0.8914 | 0.9983 | |

25 | 0.9169 | 0.9961 | |

26 | 0.9254 | 0.9947 | |

27 | 0.8722 | 0.9951 | |

28 | 0.8503 | 0.9930 | |

29 | 0.9490 | 0.9892 | |

30 | 0.9011 | 0.9341 | |

31 | 0.9157 | 0.9867 | |

32 | 0.8881 | 0.8582 | |

33 | 0.9345 | 0.9771 | |

34 | 0.9007 | 0.9922 | |

35 | 0.9273 | 0.9933 | |

36 | 0.9349 | 0.9950 | |

37 | 0.9528 | 0.9984 | |

38 | 0.9299 | 0.9875 | |

39 | 0.8838 | 0.9962 | |

40 | 0.9295 | 0.9966 | |

41 | 0.9165 | 0.9939 | |

42 | 0.9665 | 0.9986 | |

43 | 0.9660 | 0.9763 | |

44 | 0.9083 | 0.9979 | |

45 | 0.9046 | 0.9935 | |

46 | 0.9156 | 0.9918 | |

47 | 0.9225 | 0.9950 | |

48 | 0.8623 | 0.9967 | |

49 | 0.8894 | 0.9980 | |

50 | 0.8640 | 0.9947 | |

0.8926 | 0.9817 | ||

0.4407 × 10^{−2} | 0.2290 × 10^{−2} | ^{−6} |

The AUC values 39 key-frames by three detection methods.

The target of each key-frame is shown in 2D form (grey image) by taking the maximum value of every spectrum. To save the length of the paper, we choose 8 frames to show the comparison of three detectors, which are shown in Figure

The 2D form of the detection results by three detection methods.

Table

SR quantitative results (entropy and average gradient) of the test methods on key-frames.

Methods | LR frame | Bicubic interpolation | Sparse representation-based SR [ | Sequence information-based SR [ | STTF-based SR | |||||
---|---|---|---|---|---|---|---|---|---|---|

Frame | Entropy | Average gradient | Entropy | Average gradient | Entropy | Average gradient | Entropy | Average gradient | Entropy | Average gradient |

14 | 5.1603 | 0.0076 | 5.3744 | 0.0052 | 5.4996 | 0.0078 | 5.4259 | 0.0090 | ||

15 | 4.7086 | 0.0077 | 5.1407 | 0.0063 | 5.2398 | 0.0089 | 5.2184 | 0.0103 | ||

16 | 5.5521 | 0.0084 | 5.8013 | 0.0060 | 5.8765 | 0.0085 | 5.8203 | 0.0103 | ||

17 | 5.5293 | 0.0086 | 5.7054 | 0.0056 | 5.7918 | 0.0081 | 5.6125 | 0.0094 | ||

18 | 4.2989 | 0.0072 | 4.8339 | 0.0062 | 4.9794 | 0.0088 | 5.0423 | 0.0106 | ||

19 | 4.4843 | 0.0073 | 5.0045 | 0.0063 | 5.1327 | 0.0089 | 5.1831 | 0.0106 | ||

20 | 5.1442 | 0.0075 | 5.4307 | 0.0060 | 5.5039 | 0.0086 | 5.3987 | 0.0099 | ||

21 | 4.8821 | 0.0071 | 5.2234 | 0.0060 | 5.3264 | 0.0086 | 5.2578 | 0.0100 | ||

22 | 4.3472 | 0.0067 | 4.8929 | 0.0065 | 4.9858 | 0.0090 | 4.9409 | 0.0102 | ||

23 | 4.6127 | 0.0067 | 5.0430 | 0.0061 | 5.1534 | 0.0086 | 5.0806 | 0.0098 | ||

24 | 4.4189 | 0.0064 | 4.8688 | 0.0060 | 4.9850 | 0.0085 | 4.8916 | 0.0096 | ||

25 | 4.3273 | 0.0066 | 4.9091 | 0.0066 | 5.0061 | 0.0091 | 4.9607 | 0.0103 | ||

26 | 4.1394 | 0.0064 | 4.7589 | 0.0066 | 4.8438 | 0.0090 | 4.8078 | 0.0103 | ||

27 | 4.1127 | 0.0065 | 4.6995 | 0.0066 | 4.7878 | 0.0091 | 4.7713 | 0.0104 | ||

28 | 3.9657 | 0.0061 | 4.6576 | 0.0066 | 4.7467 | 0.0091 | 4.7507 | 0.0107 | ||

29 | 4.1885 | 0.0063 | 4.7545 | 0.0064 | 4.8611 | 0.0088 | 4.8481 | 0.0101 | ||

30 | 3.9672 | 0.0063 | 4.6345 | 0.0067 | 4.7253 | 0.0092 | 4.7241 | 0.0106 | ||

31 | 3.9440 | 0.0061 | 4.6135 | 0.0065 | 4.7131 | 0.0090 | 4.7349 | 0.0104 | ||

32 | 3.8661 | 0.0060 | 4.5799 | 0.0064 | 4.6914 | 0.0088 | 4.6971 | 0.0101 | ||

33 | 4.0479 | 0.0060 | 4.7100 | 0.0064 | 4.8126 | 0.0088 | 4.8114 | 0.0100 | ||

34 | 4.1691 | 0.0060 | 4.7824 | 0.0066 | 4.8621 | 0.0088 | 4.8462 | 0.0102 | ||

35 | 4.0933 | 0.0062 | 4.7169 | 0.0067 | 4.8010 | 0.0091 | 4.8245 | 0.0108 | ||

36 | 3.9157 | 0.0063 | 4.5995 | 0.0067 | 4.6881 | 0.0092 | 4.6712 | 0.0103 | ||

37 | 3.7810 | 0.0059 | 4.5028 | 0.0064 | 4.6088 | 0.0089 | 4.5984 | 0.0100 | ||

38 | 3.8814 | 0.0061 | 4.5479 | 0.0065 | 4.6483 | 0.0090 | 4.6395 | 0.0101 | ||

39 | 4.3168 | 0.0060 | 4.8397 | 0.0061 | 4.9406 | 0.0084 | 4.9135 | 0.0099 | ||

40 | 3.9333 | 0.0061 | 4.6597 | 0.0067 | 4.7380 | 0.0091 | 4.7209 | 0.0104 | ||

41 | 4.2009 | 0.0063 | 4.7897 | 0.0066 | 4.8711 | 0.0089 | 4.8346 | 0.0102 | ||

42 | 4.1083 | 0.0063 | 4.7514 | 0.0067 | 4.8362 | 0.0091 | 4.8398 | 0.0103 | ||

43 | 4.0485 | 0.0063 | 4.6827 | 0.0067 | 4.7602 | 0.0091 | 4.7117 | 0.0101 | ||

44 | 4.0521 | 0.0062 | 4.6510 | 0.0063 | 4.7587 | 0.0087 | 4.7273 | 0.0097 | ||

45 | 4.3442 | 0.0060 | 4.9011 | 0.0061 | 5.0079 | 0.0085 | 4.9380 | 0.0097 | ||

46 | 4.0006 | 0.0061 | 4.5913 | 0.0062 | 4.7080 | 0.0087 | 4.6467 | 0.0097 | ||

47 | 4.4749 | 0.0062 | 5.0109 | 0.0060 | 5.1015 | 0.0084 | 5.0007 | 0.0095 | ||

48 | 4.0685 | 0.0065 | 4.6843 | 0.0065 | 4.7859 | 0.0090 | 4.7607 | 0.0101 | ||

4.3167 | 0.0066 | 4.8671 | 0.0063 | 4.9651 | 0.0088 | 4.9329 | 0.0101 |

Figure

The SR results of four test methods, from left to right: original LR frame, bicubic interpolation, sparse representation-based SR method [

In this paper, aiming at hyperspectral video, we propose a novel key-frame and target detection method based on cumulative tensor CP factorization, termed as CTCF, and a super-resolution algorithm based on sparse-based tensor Tucker factorization, called STTF. Unlike conventional matrix factorization based methods, CTCF considers hyperspectral video (HSV) as 4D cumulative tensor and approximates new added frames by updating factor matrices. To break the limit of conventional methods and make super-resolution (SR) more practical, STTF exploits the sparsity of HSV frames and factorizes them as a sparse core tensor multiplied by three modes dictionaries. In this way, spatial resolution of LR-HSI is enhanced directly without HR samples. The experimental results systematically prove that the proposed CTCF and STTF methods outperform other state-of-the-art algorithms.

In the future works, we focus on tensor factorization based target tracking methods which are able to extract target region more accurately and clearly. For super-resolution, we aim at exploiting nonlocal similarities in tensor factorization framework, which has been widely used in inverse problems. Besides target tracking and super-resolution, regions of interest (ROI) approaches will be investigated, in order to make HSV target recognition more efficient and full featured. Inspired by [

The optimizations of

Optimization of

The conjugate gradient (CG) method is utilized to solve (

Optimization of

Likewise, CG is used to solve (

Optimization of

We apply CG to solve (

Optimization of

Equation (

Here, the optimizations of

Update

Update

Based on (

However,

Update

The data used to support the findings of this study are available from the corresponding author upon request.

The authors declare no conflicts of interest.

The authors would like to thank Professor Gu from Heilongjiang Province Key Laboratory of Space-Air-Ground Integrated Intelligent Remote Sensing for his selfless help. This work was supported by the National Natural Science Foundation of China (Grant no. 61671184) and the National Natural Science Foundation of Key International Cooperation of China (Grant no. 61720106002).