Experimental and Analytical Studies on Improved Feedforward ML Estimation Based on LS-SVR

Maximum likelihood (ML) algorithm is themost common and effective parameter estimationmethod.However, when dealingwith small sample and low signal-to-noise ratio (SNR), threshold effects are resulted and estimation performance degrades greatly. It is proved that support vector machine (SVM) is suitable for small sample. Consequently, we employ the linear relationship between least squares support vector regression (LS-SVR)’s inputs and outputs and regard LS-SVR process as a time-varying linear filter to increase input SNR of received signals and decrease the threshold value of mean square error (MSE) curve. Furthermore, it is verified that by taking single-tone sinusoidal frequency estimation, for example, and integrating data analysis and experimental validation, if LS-SVR’s parameters are set appropriately, not only can the LS-SVR process ensure the single-tone sinusoid and additive white Gaussian noise (AWGN) channel characteristics of original signals well, but it can also improves the frequency estimation performance. During experimental simulations, LS-SVR process is applied to two common and representative singletone sinusoidal ML frequency estimation algorithms, the DFT-based frequency-domain periodogram (FDP) and phase-based Kay ones. And the threshold values of their MSE curves are decreased by 0.3 dB and 1.2 dB, respectively, which obviously exhibit the advantage of the proposed algorithm.


Introduction
Maximum likelihood (ML) estimation depends on the asymptotic theory, which means that the statistical characteristics are shown accurately only when the sample size is infinity.However, burst-mode transmissions always bring problems about short data and severe conditions.Therefore, threshold effect is existing.Namely, the mean square error (MSE) of ML estimation can reach Cramer-Rao lower bound (CRLB) if it is higher than a value, or the performance will be deteriorated rapidly.
Statistical learning theory (SLT) and structure risk minimization (SRM) rule in it are specialized in small-sample learning [1].As their concrete implement, support vector machine (SVM) overcomes the over-fitting and local minimum problems currently existing in artificial neural network (ANN).Least squares support vector regression (LS-SVR) has the following improvements: inequality constraint are substituted by equality one; a squared loss function is taken for the error variable.Hence, we introduce LS-SVR to improve ML estimator and take feedforward single-tone sinusoidal frequency estimation for example, in this study.
Estimating frequency of a single-tone sinusoid has attracted considerable attention for many decades.Rife and Boorstyn exploited the relationship of maximum likelihood estimator (MLE) to discrete Fourier transform (DFT) and proposed a frequency-domain periodogram (FDP) algorithm having two stages: coarse search and fine search [2].In order to reduce the calculation cost, a great deal of improved algorithms have erupted mainly from two sides: interpolationbased and phase-based ones.
During the former, an iterative binary search for the true signal frequency has been presented, which is particularly suited for digital signal processing (DSP) implementation [3].In [4], the same authors have proposed a number of hybrid estimators that combine the dichotomous search with various interpolation techniques in order to reduce the computational complexity, at the expense of acquisition range.And, other modified dichotomous search frequency estimators have been addressed in [5][6][7].Besides, complex Fourier coefficients have been utilized to interpolate the true signal frequency between the maximum and the second highest bin [8].However, it has been shown to have a frequency dependent performance [9].Two improved estimators have been proposed, which were implemented iteratively [10,11].Rational combination of three spectrum lines (RCTSL) has been employed as the fine estimation, because of its constant combinational weights in least square approximation [12].Other methods used for interpolation include Lagrange interpolator [13], L-filter DFT [14], nonlinear filter [15], Kaise window [16], trigonometric polynomial interpolator [17], narrowband approximation interpolator [18], and so on.In the latter, Tretter [19] was the first person to propose a phasebased approach by introducing an approximated and linear model for instantaneous signal phase.Subsequently, a great deal of improvements have erupted mainly in the following three parts: taking differences over one or more delays, which is well-known as Kay and generalized Kay estimators [20][21][22][23][24][25]; introducing autocorrelations and their different functions, such as Fitz, L&R, and M&M estimators [26][27][28][29][30]; and preprocessing by means of lowpass filter, blocking average, and filter banks to increase signal-to-noise ratio (SNR) [31][32][33][34].
In this paper, we present an improved feedforward ML estimation based on LS-SVR, taking single-tone sinusoidal frequency estimation, for example.LS-SVR process is regarded as a time-varying linear filter to increase input SNR of received signals, and accordingly, the threshold value of MSE curve is decreased.Reliability and validation of LS-SVR process are verified by integrating data analysis and experimental simulation.It is verified that not only can the LS-SVR ensure the single-tone sinusoid and AWGN channel characteristics of original signals well, but also increases the input SNR of received signals efficiently and improves frequency estimation performance.During experimental simulations, LS-SVR process is applied to two common and representative single-tone sinusoidal frequency estimation algorithms, the DFT-based FDP and phase-based Kay algorithms.The estimation performance of having the LS-SVR process and not are compared, respectively, to exhibit the advantage of the proposed algorithm, if its parameters are set appropriately.
The remainder of this paper is organized as follows.Section 2 briefly introduces the basic theory of LS-SVR.Section 3 describes the model of single-tone sinusoidal frequency estimation and the classical algorithms including FDP and Kay.In Section 4, the LS-SVR process is concretely explained and analyzed.And Section 5 shows the results of simulations and experiments.The paper is concluded in Section 6 finally.

Theory of LS-SVR
At first, a linear hyperplane (x) = (w ⋅ (x)) +  insofar as for  is assumed to fit all elements of the training set  = {(x 1 ,  1 ), . . ., (x  ,   )} ⊂   × , where w is the high-dimensional coefficient of (x), (⋅) is an inner product operator, and (⋅) is a nonlinear mapping from low to high dimension feature space.Also, -insensitive loss function is defined as denotes the distance from point (x  ,   ) ∈  to (x): According to (2), we optimize (x) through maximizing Then, we proceed to conquer inseparable condition by introducing error variables   and least squares (LS) method, and convert (3) into where penalty factor  is a positive constant to take compromise in LS-SVR's generalization capability and fitting errors, which are denoted by the first and second item of (w, ), respectively.
Next step, we use Lagrange multiplier method and replace where 1 , . . .,   are Lagrange multipliers, Q is a kernel function matrix, and radius basis function (RBF) is adopted in this study, so: where Q  is the (, )th element of Q; the width of RBF ℎ is a positive constant.Ultimately, the discriminant function is described as

Signal Model and Classical Algorithms
3.1.Signal Model.The sinusoid signal polluted by noise is modeled as Here,  > 0,  ∈ [−0.5, 0.5),  ∈ [−, ) are the amplitude, deterministic but unknown frequency, and initial phase, respectively;   is an independent complex additive white Gaussian noise (AWGN) with zero-mean and variance  2 ; and  is the sample size.

LS-SVR Process and Its Analysis
is the real part of  −1 , so it is an independent real AWGN with zero-mean and variance  2 /2, and derive  real (), ( Then, we substitute   into () and get a new series of received signals.
At last, we utilize the classical algorithm to estimate frequency accurately.
Firstly, setting  = 0.15,  = 0,  = 32, SNR = 0 dB, and the parameter of LS-SVR  = 5, the arbitrary amplitude spectrums of Ĝ, while ℎ = 1 and ℎ = 5 are illustrated in Figure 1, respectively.It is shown that when ℎ = 1, the spectrum component of Ĝ in  is much more powerful than other places.It means that now the output of LS-SVR process still keeps the spectrum characteristics of  cos(2+) and can be used to estimate the frequency of  cos(2 + ).However, as ℎ increases, the spectrum component of Ĝ in  inversely decreases and others gradually increase; hence, now the output of LS-SVR process cannot keep the spectrum characteristics of  cos(2 + ).
Furthermore, everything is as in Figure 1, while ℎ = 1, ℎ = 4, and ℎ = 5, the time-domain waveforms of Ĝ are plotted in Figure 2, perspectively.The conclusion of Figure 2 is consistent with Figure 1, just when ℎ = 1, the amplitude of time-domain waveforms of Ĝ is less than  cos(2 + ).
The Euclidean distance between Ĝ and G is defined as follows, where max(⋅) is the operation of taking maximum value: Everything is as in Figure 1, and the number of Monte Carlo experiments is 10000; the values of  with different ℎ are listed in Table 1.Obviously, Ĝ can be very close to G through proper choice of ℎ.Nevertheless, Ĝ will gradually deviate from G as ℎ increases.
Consequently, proper choice of ℎ ensures that LS-SVR process can be used for frequency estimation of single-tone sinusoidal signals.Integrating the above analyses, the value of ℎ must be less than 3.
(2) Through analyzing the covariance function of Ŷ, it is shown that LS-SVR process is feasible and valid with a proper choice of  and ℎ.
From (16), it is obvious that the covariance function is related to  2 and .Taking  = 4, for example, when  = 5, ℎ = 1, (16) is calculated as We can deduce the following by analyzing (18) (A) The elements except main diagonal ones denote the correlations between ŵ in different moments.Everything is as in Figure 1, and the number of Monte Carlo experiments is 10000; Figure 3 illustrates the impact of  on MSE performance, which is consistent with all analyses above, also  = 5 is set in this study.
At the same time, according to Everything is as in Figure 3 other than that  = 5; the impact of ℎ on MSE performance is shown in Figure 4, which is consistent with all analyses above, and ℎ = 1 is set in this study.
(3) Setting  and ℎ appropriately, LS-SVR process can increase SNR of Y and improve the performance of feedforward ML frequency estimations under the condition of small sample and low SNR.

Simulations and Experiments
We apply LS-SVR process to two common and representative single-tone sinusoidal ML frequency estimation algorithms, the DFT-based FDP and phase-based Kay ones, and derive the proposed algorithm called LS-SVR for short, where the number of DFT points of FDP algorithm is  = 32.

Mean Performance.
Everything is as in Figure 3 other than  = 5; Figures 5 and 6 illustrate the mean of these three algorithms with different SNR.As is shown, whether high or low SNR, LS-SVR process can hardly change the unbiased ranges of FDP and Kay algorithms.Also, the unbiased ranges of all three algorithms will degrade with deterioration of SNR.

MSE Performance.
Everything is as in Figure 3 except  = 5; the MSE curves of these three algorithms are both shown in Figures 7 and 8, where CRLB is defined as (1/SNR)(3/(2) 2 ( − 1)(2 − 1)) [2].We can see that LS-SVR process effectively improves the MSE performance of both FDP and Kay algorithm, and their threshold values are decreased by 0.3 dB and 1.2 dB, respectively.

Impact of Sample Size 𝑁.
Everything is still as in Figure 3 except that  = 5, Figure 9 illustrates the impact of  on MSE performance.We can know that the MSE curve of LS-SVR algorithm will decrease as  increases.However, when LS-SVR process is applied to FDP algorithm, its threshold value will increase as  increases; when LS-SVR process is applied to Kay algorithm, its threshold value will keep the same.
The reason is related to the concrete frequency estimation algorithm after LS-SVR process.

Figure 3 :
Figure 3: Impact of  on MSE performance.

Figure 6 :
Figure 6: Mean of Kay and LS-SVR algorithms.

Table 1 :
Values of  with different ℎ.
(B) The main diagonal elements denote the powers of ŵ .ŵ in different moments are independent and identically distributed (i.i.d) by reason of their nearly equal values.And also, it is the premise that the classical algorithms of feedforward ML frequency estimation can be still employed after LS-SVR process.

Table 2 :
Pluses of LS-SVR process with different  and ℎ.