An Adaptive PHD Filter for Multitarget Tracking with Multispectral Data Fusion

In order to improve the detection and tracking performance ofmultiple targets from IRmultispectral image sequences, the approach based on spectral fusion algorithm and adaptive probability hypothesis density (PHD) filter is proposed. Firstly, the nonstationary adaptive suppression method is proposed to remove the background clutter. Based on the multispectral image sequence, the spectral fusionmethod is used to detect the abnormal targets. Spectral fusion produces the appropriate binary detectionmodel and the computational probability of detection. Secondly, the particle filtering-based adaptive PHD algorithm is developed to detect and track multiple targets. This algorithm can deal with the nonlinear measurement on target state. In addition, the calculated probability of detection substitutes the fixed detection probability in PHD filter. Finally, the synthetic data sets based on various actual background images were utilized to validate the effectiveness of the detection approach. The results demonstrate that the proposed approach outperforms the conventional sequential PHD filtering in terms of detection and tracking performances.


Introduction
The conventional detection scheme of the long range point source target mainly focuses on employing the temporal features of target from broadband (single band) IR sensor [1,2].However, this methodology faces enormous challenges because of the latest decoys in modern electronic countermeasures.It is difficult to detect the decoys and targets by employing the temporal information when they have similar trajectories.It is noted that the spectral information, which is obtained through a large number of narrow contiguous spectral channels, denotes the intrinsic feature of target [3].Therefore multispectral imaging sensors can portray more information than a broadband IR sensor.Multispectral IR image has been widely used to perform target detection by means of fusing the narrowband information [4][5][6].However, the detection mechanism based on multispectral IR image only adopts the spectral data and cannot be used to detect moving targets with low signal-to-noise ratio (SNR).The fusion of the data in time, wavelength, or other observables should be explored to improve the ability to detect the targets with a low SNR.
In addition, multiple target detection and tracking is a challenging problem that arises from heavy clutter, noise, and false alarm.The conventional approaches focused on the data association technique and the appropriate filtering methods (such as Kalman filtering).Typical data association algorithms include multiple hypothesis tracking (MHT), joint probabilistic data association (JPDA), and their own variants [7][8][9].However, it has been demonstrated that data association-based tracking scheme is a NP-hard problem.Thus free data association tracking methods are hot topic in recent years.The Finite Random Set Statistics-(FRSST-) based PHD filter, which incorporates target birth, target spawn, and target death, has been extensively studied in multitarget detection and tracking applications [10][11][12][13][14][15].But PHD filter is essentially approximation of the first moment of multitarget posterior density.This factor makes PHD filter performance degraded in low SNR environment.Moreover, owing to the difficulty of obtaining the suitable multisensor model for PHD filter, the preciously presented PHD filter employed the single sensor model in most cases.To solve this problem, Mahler proposed a sequential PHD filter to fuse measurements from multiple sensors.Some researchers also 2 Journal of Spectroscopy utilized the sequential PHD filter to track radar targets [16].However, the sequential PHD filtering produces different updated PHD depending on the order of the multisensor update [17].In the case that the probability of detection is not high for at least one sensor, the resulting intensity and estimates significantly rely on the order of the multisensor update.Therefore it is hard to determine the correct fusion order of multiple sensors.
Though possessing attractive and promising application, the PHD filtering still has some problems not taken into consideration concretely.Actually, the PHD filter is mainly used in radar target area.The dynamics model and measurement model for radar target are mainly proposed based on active detection scheme.At present the PHD filter is rarely used for infrared targets detection.Two problems arise from IR target detection when using PHD filtering.Firstly the measurements which are produced by IR sensor are nonlinear and it is hard to formulate the measurement model by FRSST framework.Secondly, the low SNR targets immersed in IR images make the PHD filtering hard to implement.
This paper, based on the PHD filtering, is aimed at detecting and tracking multiple targets from IR multispectral image sequences under diverse scenarios.The contribution of the paper can be summarized as follows.
(1) Constructing the general four-dimensional signal model for IR multispectral image sequences: the multitarget detection and tracking problem can be formulated in the adopted model which can be used in the airborne system or in the satellite-borne system.
(2) Proposing a method to suppress the background clutter noise: the proposed method differs from the traditional approaches on the aspect of removing the spatial correlation and temporal correlation in pixels simultaneously.The solution is put forward in accordance with the least square theorem.This method is also considered to be the preprocessing of multispectral image fusion.
(3) Developing the PHD filtering to detect and track multiple targets: an adaptive PHD (APHD) filtering algorithm is derived from using the conventional PHD method and the RX algorithm.The difference between the APHD filtering and the original PHD filtering is that the probability of detection and the probability of false alarm in our study are calculated based on the measured data.Therefore the presented approach may be more appropriate for the real physical model and the final results could have a higher degree of confidence.Additionally, the efficient particle filtering is also developed to implement the APHD filtering.
The experimental results have shown that the proposed algorithms provide good detection and tracking capabilities, even for the targets with low signal-to-noise ratio.The paper is organized as follows.In Section 2, the original PHD filtering is introduced.In Section 3, the general four-dimensional signal model and target imaging model are presented; meanwhile the adaptive suppression of background clutter is proposed.The multispectral data fusion is achieved by means of the RX algorithm.Section 4 derives the adaptive PHD filtering based on efficient particle filtering.The parameters in measurement update are described.In Section 5, the detailed performance results for the proposed detection and tracking algorithm are presented.The discussions and future work are stated in Section 6.The conclusions are given in Section 7.

The PHD Filtering
The PHD filter recursively estimates the first-order statistical moments of multitarget posterior density and is the suboptimal algorithm underlying the optimal Bayesian filtering framework.However, the PHD filter avoids the data association procedure and reduces the computational load.The PHD filter is described as follows by using Bayesian laws.
The posterior intensity of FISST at frame  − 1 is denoted as  −1 (x) and the predicted intensity is given by Bayesian laws.Consider When the new measurement z  is obtained, the intensity  |−1 (x) is updated by where   (x) is the intensity of birth RFS at time ,  In the real applications, all the three sufficient conditions should be taken into account.However, some studies neglected the third sufficient condition.The third sufficient condition is an approximation to update the predicted  |−1 (x) to obtain the posterior intensity   (x).This approximation is justifiable under the scenario that false alarm densities are small [10].In this paper, the third sufficient condition is satisfied via suppression of background clutter and anomaly detection algorithm, which reduce false alarm densities.

Four-Dimensional Signal Model in
Multispectral Image Sequence

Target State and Binary Detection
Model.The multispectral images have  bands.Each band consists of  frames.Each frame consists of  pixels in the -dimension and  pixels in the -dimension.The target state vector is , where (  ,   ) denotes the position and (V  , V  ) refers to the velocity of the target at frame .The time evolution of target state can be modeled as where (⋅) is the nonlinear/linear state function, which depends on the target state x −1 and the process noise n −1 .n −1 is usually subject to Gaussian distribution with zero mean.
It is assumed that each pixel only depends on one target at most.Thus the raw binary measurement signal model can be presented by the four-dimensional signal space.Conisder where   is the received spectral irradiance at the position (  ,   ) of the image.  denotes the spectral irradiance of background and   represents the spectral irradiance of target.

Target Imaging Model.
The target which is distant from the sensor covers one pixel or subpixel in the image.Since the IR imaging system is nonideal, the point source target has several pixels.The target imaging size is decided by the point spread function (PSF).PSF is usually approximated by a truncate 2D Gaussian function.PSF can be represented as where the two constants   and   are the spread parameters.
Both can be obtained via experiments.In this paper we have   =   = 1.

Adaptive Suppression of Background Clutter.
It is necessary to remove clutter noise for the detection of dim point source target embedded in dense background clutter.The conventional methods are based on spatial domain, temporal domain, and frequency domain.However, the spatial correlation exists in the multispectral image sequences besides the temporal correlation.In order to remove background clutter, novel methods for suppressing clutter should be further explored.
The spectral radiant intensity is defined as   at wavelength , and the mean of local spatial pixels is calculated by using the spatial filter.In order to avoid influencing the local spatial mean by the target pixels, it is supposed that the target covers a specific area in the image, which is shown in Figure 1.
In this paper, the spatial correlation and temporal correlation are removed simultaneously.Firstly, let  represent the multispectral image.Let (  ,   ) be a test pixel of interest in .The PSF determines the covering area of target in , which is shown in Figure 1.The corresponding local temporal mean is calculated by four nearest neighbors in temporal domain.The residual error after removing the spatial and temporal mean is presented as where It is denoted as The solution of  can be given by the least square theorem.It is given by

Measurement Model Based on Multispectral Data Fusion.
It is critical to acquire available measurements in the tracking system.The RX algorithm is utilized to produce measurements in this section.RX algorithm, as the benchmark method in multispectral applications, was proposed by Reed and Yu in 1990 [18].It is an abnormal detection algorithm, supposing that the spectral distribution of target and the covariance of background clutter are unknown.The local nonstationary background mean should be removed when using RX to generate detections.This operation has been implemented in Section 3.3.It is supposed that we have  bands images; each residual pixel after background suppression is denoted as x()  = [ 1 ()  ,  2 ()  , . . .,   ()  ]  .Thus sample matrix whose size is  ×  at time  is defined as where  is the number of samples, which come from the outside window of Figure 1.The binary detection model of RX algorithm is given by where the hypothesis H 0 shows that no target exists in the scene.is the signal pattern.W  is subject to approximate Gaussian distribution.The RX detector is derived from the generalized likelihood ratio test (GLRT).Consider where (X  ) denotes the test statistics and  0 is the detection threshold. 0 can be calculated by the probability of detection (  ) and the probability of false alarm ( FA ) according to the Neman-Pearson rules.The   and  FA are given by where ( |  1 ) is a noncentral beta-distribution and ( |  0 ) is a standard beta-distribution.( |  1 ) is represented as where  1 (/2; /2; /2) is the confluent hypergeometric function and  denotes the generalized signal-to-noise ratio.( |  0 ) is presented as The data of spectral domain is fused by RX algorithm and the available measurements are produced.Then the fusion tracking can be implemented by using the test statistics of multiple frames.The measurements in each frame can be formulated by the random finite set.

Adaptive PHD Filtering Based on Particle Filtering
The PHD filter is used to track multiple targets by using the test statistics that were produced by RX.At present two methods are proposed to implement PHD filter: the Gaussian Mixture-(GM-) PHD and the Sequential Monte Carlo-(SMC-) PHD, that is, the particle filtering-based PHD [19,20].Because the target measurement from IR image is nonlinear, the particle filtering-based method is developed to implement PHD filter in this paper.However, the conventional approach based on particle filtering may produce sample degeneration and impoverishment.A new proposal distribution is proposed to deal with sample degeneration and impoverishment.
The SMC approach is utilized to implement the algorithm which provides a mechanism to represent the posterior density by a set of random samples or particles.The posterior 4.1.Time Update.The predicted particles are generated via the following distribution: where ) are a priori distribution.The predicted PHD can be expressed as where  () |−1 is calculated according to (1): 4.2.Measurement Update.When the available measurements are acquired at time , the update PHD can be given by ( 2) and ( 18).Consider where The conventional PHD filtering always sets a fixed   , which rarely exists in the real situation.In this paper, the specified   is substituted by a computed   according to (13).Thus the proposed algorithm is entitled by adaptive PHD (APHD).The likelihood function 21) is defined as (z where   (x |−1 ) and   (x |−1 ) denote the covering area of the target in the image, which has been given in Figure 1.
|−1 ) is the single pixel likelihood, which is expressed via ( 14) and (15).Consider   (z The expected number of targets is calculated by summing up the total weights; that is, Then resampling is performed to avoid the degeneracy.The updated posterior density at time step  is presented as In summary, the processing chain of the proposed approach is presented in Figure 2.

Simulation and Result Analysis
To validate the effectiveness of this algorithm, it is employed to detect and track three typical exoatmospheric targets.Warhead, fragment, and decoy are introduced in the multispectral infrared imagery sequence, which consist of 3 wavebands.The three infrared sensors include the long wave IR, mid-long wave IR, and short wave IR.Each waveband has 30 frames.And each frame size is 180 × 240 pixels.Frame 1 from the image sequence at wavelength 4 m is shown in Figure 3, which is the actual background with clutter noise.
Warhead is introduced at position (25.6, 24.8) in frame 1 and remains within the surveillance region until the last time step.Fragment appears at position (28.3, 47.2) in frame 6 and died in frame 21.Decoy appears at position (56.7,27.4) in frame 11 and disappeared in frame 26.This simulation does not include any spawning of new targets from existing ones.The IR intensity sequences of the three targets are simulated according to the BODE (Bhattacharyya Optical Decoy Evaluation) model [21] and the basic physics parameters for space objects [22].The IR intensity has been normalized, which is shown in Figure 4.The exoatmospheric target state can be approximately modeled as a linear Gaussian process.Therefore (3) becomes where the process noise is n −1 ∼ (0, 0.05 2 ). is the state transition matrix, and  denotes the noise matrix; they are defined as The sampling period  = 1s is adopted.Each target is subject to the same velocity (0.5, 0.5) pixel/frame.The initial prior probability density function (PDF) of velocity is assumed to be uniform within [−1, 1] pixel/frame.It is known according to above parameters that the three exoatmospheric targets appear and move within the image region [1, 100] × [1,100].To observe the particles' distribution conveniently, the image size for displaying is cut to 100 × 100 pixels.Two simulated image frames are shown in Figure 5.Some targets are surrounded by the actual dense background clutter noise.The procedure of implementing the original PHD filter based on particle filtering can refer to [20].Here, we specify the main parameters for implementing the adaptive PHD filtering in Table 1.The number of particles representing one target   = 2000, the number of particles representing new born targets   = 3000, the probability of target survival  , = 0.98, the probability of target spawning   = 0, and the probability of spontaneous target birth   = 0.01.The probability of detection   is calculated according to (13).
The raw infrared image sequence cannot be directly used to detect the targets because of background clutter.The adaptive clutter suppression filter can be used to remove the background clutter.Figure 6(a) shows the target appearance curve which obviously has a nonstationary mean before the suppression of clutter.Figure 6(b) depicts an adaptive suppression curve in which the pixels distribute around zero.Comparing Figure 6(a) with Figure 6(b), the adaptive clutter suppression algorithm removed the background correlation.Thus the residual clutter/noise is approximately subject to the Gaussian distribution.It is noted that since the Gaussian model is the sufficient condition to implement the RX algorithm, the measurements produced by the clutter suppression filter can be utilized in the RX algorithm.The experimental  results demonstrated that the proposed background clutter suppression algorithm performed very well.
The multispectral images are fused by using RX algorithm in each frame.In order to show the potential of multispectral fusion, the single band image and the three band images are processed, respectively.Compared with the single wavelength image, the three band images have good detection output, which was shown in Figure 7(b).In contrast, the target was immersed in the single band image, which was shown in Figure 7(a).The bright spots in Figure 7(b) give a priori positions of the aiming target.The knowledge of data can be used to increase the efficiency of the particle filter.Thus the efficient proposal density function can be designed by the bright pixels.The 50 highest pixels in the image are used to design the proposal density function.The proposal density which includes the latest measurements improves sample degeneration and impoverishment.The experiment shows the effectiveness of the spectral image data fusion, which also reduced the rate of false alarm and clutter density.
The proposed APHD algorithm is performed on the spectral fusion output to detect and track multiple targets.The impact of the number of spectra on the performance of the proposed algorithm is considered to show the potential of multispectral fusion.In addition, the robustness of the proposed algorithm for low SNR targets is investigated.Two scenarios are constructed by selecting different SNR.Actually, low SNR targets are defined as the targets for which the sensor responses have a value of SNR lower than 10 dB in target detection and tracking applications [23].Thus, two typical SNR values of 8.7 dB and 4.8 dB are considered in our work.The definition of SNR refers to [24].The performance of this algorithm is evaluated by using the averaged optimal subpattern assignment (OSPA) [25], which is a good metric for multitarget detection and tracking.The cut-off parameter  and the order parameter  should be chosen in OSPA.To yield smooth distance curves, we have  = 40 and  = 2 in this paper.One hundred Monte Carlo trials are performed for each scenario.Figures 8(a Under the condition that SNR = 8.7 dB, the estimated number of targets is accurate and this algorithm performs very well with no delay in track termination after the targets disappeared when multiple spectrums are used.Although the output of  = 3 is not correct at frames 1, 17, and 28 in Figure 8(a), the continuing estimation of number of targets is still stable.However, the estimated number of targets of  = 1 has a bias, concluding from the curve of OSPA in Figure 8(b).
Under the condition that SNR = 4.8 dB, the estimated number of targets has a little bias and delay appears in track termination after the target disappeared when multiple spectrums are used, which is shown in Figure 9(a).However, the estimated number of targets of  = 1 has a big bias and long delay when the targets disappeared, which can be seen from the curve of OSPA in Figure 9(b).In the circumstances, it is deduced that multiple spectral fusion improved the detection and tracking performance.
Figure 10 shows the estimated error of the number of targets, which is calculated by | − T|, where  is the true number of targets and T denotes the estimated number of targets.From Figures 10(a    In order to show the benefit of the proposed method, we contrast the performance of the proposed approach (APHD) with the SPHD filtering in the same scenario as above (SNR = 8.7 dB,  = 3).The detailed description of SPHD filtering can refer to [17].Since the lack of memory of the SPHD filtering may result in premature death of targets, we used a spectrum with low probability of detection at the start of the iterated update rather than at the end.The probability of detection can be calculated according to the RX algorithm.
We compared the performance of APHD filtering and SPHD filtering in terms of the estimated number of targets and OSPA in Figure 11.It is clear in Figure 11(a) that the proposed algorithm is more stable than the SPHD filtering.The OSPA distance is adopted to compare APHD filtering with SPHD filtering in Figure 11(b), which shows that the SPHD filtering has more fluctuations due to the estimation of target number.As a matter of fact, it is difficult to decide the order of the multisensor update owing to the probability of detection.Even though we obtain the correct order of the multisensor update for the SPHD filtering in advance, the presented APHD filtering has better performances than the SPHD filtering.Thus it is concluded that the proposed APHD filtering outperforms the SPHD filtering.The corresponding results for SNR = 4.8 dB are not shown because of the poor performance of the SPHD filtering in similar cases.

Discussions and Future Work
The fusion of multispectral image sequence improved the detection and tracking performance in multiple targets scenario, especially when the targets had low SNR; that is, SNR < 10 dB (Figures 8 and 9).We found that even though the SNR was 4.8 dB, the detection and tracking for multitarget was still robust (Figures 9(a) and 9(b)).We did observe that the higher the SNR was, the better the detection and tracking performance became (Figures 8, 9, and 10).Compared to  the broadband image sequence, multispectral data fusion produced more information of the detected targets (Figures 7(a For the issues arise from multitarget tracking, previous studies have found that the PHD filtering has superiority.However, the derivation of PHD filtering was based on the single sensor measurement model.We first derived the likelihood function via multispectral data fusion (23) and used this likelihood to update the posterior density.This procedure is centralized data fusion and different from the sequential PHD filtering which is distributed data fusion.The previous studies have shown that centralized data fusion is better than distributed data fusion.Moreover, the probability of detection and the probability of false alarm are always fixed in the conventional PHD filtering.These two parameters usually have variability in the real world.We presented the calculated probability of detection and the calculated probability of false alarm via measured data (( 13), (14), and ( 15)), which replace the two parameters in the conventional PHD filtering.Thus the posterior density in this paper is adaptive to the coming measurements.Additionally, the comparison of the proposed method with the conventional PHD filtering is  Though multispectral data fusion and PHD filtering have been successfully used to study the detection and tracking of multitargets, some limitations and potential work in future should be stated.Firstly, the RX algorithm used whitening data.So the correlation between the background pixels should be removed as much as possible.This paper proposed the linear least square algorithm.The nonlinear method removing background can be studied in the future.Secondly, when the target is far from the sensor, that is, the target has subpixel image, the multispectral fusion method based linear spectral mixing model should be considered.Finally, the particle filtering is a framework to implement the PHD filtering.But the original particle filtering has many issues, such as the degeneracy and diversity of particles.Thus other variants of particle filtering can be proposed to improve the PHD filtering in the future.

Conclusions
In this paper we have considered the multitarget detection and tracking problem arises from IR multispectral image sequences system.We have proposed a fusion tracking approach utilizing the spectral and temporal information of IR multispectral image sequences.The fusion tracking method consists of multispectral data fusion and adaptive probability hypothesis density (APHD) filtering algorithm.The influence of the number of spectrums and the SNR values on the system performance has also been studied.The experimental results have shown that the presented fusion tracking method has better detection and tracking performance than the conventional algorithm, even for targets with low SNR.

Figure 2 :
Figure 2: The processing chain of the proposed approach.

Figure 3 :Figure 4 :
Figure 3: Frame 1 from the image sequence at wavelength 4 m.
) and 8(b) show the estimated number of targets and estimated error, respectively, when SNR = 8.7 dB.Figures 9(a) and 9(b) show the results distinctively when SNR = 4.8 dB.
) and 10(b) one can observe that the multispectral fusion ( = 3) provides significantly higher accuracies than single waveband spectral ( = 1) when estimating the number of targets.

FrameFrameFigure 10 :
Figure 10: The estimated error of number of targets at different SNR.
Targets evolve in time and generate measurements independently of one another.

Table 1 :
The main parameters for implementing the adaptive PHD filtering.