Decision Feedback Blind Equalizer with Tap-Leaky Whitening for Stable Structure-Criterion Switching

The research presented in this paper improves the structure-criterion switching performance of the blind decision feedback equalizer (DFE) which eliminates error propagation effects by optimizing both the structure and the cost criterion. To conquer the complexity of the 64-QAM (quadrature amplitude modulated) signal constellation, the stochastic entropy-gradient algorithm is additionally regularized by the coefficient leaky term to avoid a coefficients norm overgrowth of the received signal whitener. Effectively, the leak of coefficients is employed to ensure a stable structure-criterion switching of DFE between blind and decisiondirected operation modes. The optimization of the resulting whitening algorithm is achieved by means of two free, leaky and entropic, parameters which act in opposition to each other. Both, the influence of the 64-QAM signal on the feedback filter behavior and the parametric optimization of the whitening algorithm are analyzed through simulations.


Introduction
Blind equalization methods are introduced as an alternative approach to the data communication concept employing a specially designed training sequence (pilot) to direct the train of receiver adaptive parameters [1,2].By using blind adaptive equalizers, which work without the assistance of a pilot, it is possible to increase effective system data rates and, also, to realize system applications where the train with a pilot is not possible [3,4].
Unlike a linear equalizer which strives to complete an inverse channel response by a finite impulse response filter, a decision feedback equalizer (DFE) divides equalization task between linear feedforward and nonlinear feedback filters (equalizers).In such a manner, according to the hypothesis of correctly detected symbols, DFE exploits a nonlinear discrete nature of transmitted symbols to eliminate postcursor intersymbol interference (ISI) without a noise enhancement [5] using a relatively small number of coefficients [6].This property of DFE is particularly important in systems characterized by deep spectral nulls channels.On the other hand, the main drawback of a DFE is error propagation phenomena which generally degrades its performance and can lead to an equalization failure depending on the length of error packets.For a blind DFE, the error propagation becomes a particularly critical issue because it appears inherently at the starting phase of equalization.Therefore, blind DFEs appeal for more efficient algorithms and signal processing techniques than their nonblind counterparts [7][8][9][10][11][12][13].
Motivated by the works of Labat [7] and Kim [8] and their coauthors, we have recently developed the blind DFE scheme [10,11], called Soft-DFE, which combines the structurecriterion manipulation method with the blind deconvolution theory by Bell and Sejnowski [14].In contrast to the original "self-optimized" DFE [7] based on the feedback filter (FBF) performing the minimum mean-square error (MMSE) criterion, the specific of Soft-DFE is its soft feedback filter (SFBF) which removes the postcursor ISI by maximizing the joint entropy (JEM) of outputs.The efficiency of the Soft-DFE solution has been verified in the system transmitting 16and 32-QAM (quadrature amplitude modulated) signals over severe ISI time-invariant channels.
This paper considers the extension of the Soft-DFE operation to a 64-QAM signal.This extension is mainly considered 2 International Journal of Digital Multimedia Broadcasting from the perspective of the Soft-DFE's structure-criterion switching robustness to the increased signal complexity from 16-and 32-QAM to 64-QAM.From the statistic point of view, the increase of symbol numbers in QAM signals (through the increase of symbol number level magnitudes) leads to the increase of their second-and fourth-order moments of corresponding statistical constants [15].Thus, the secondorder statistic equalization (whitening) becomes a critical issue in systems typically characterized by deep spectral nulls.In the case of the Soft-DFE scheme, it is shown that JEM algorithms, optimizing SFBF through two operation modes, present the instability of convergence at the time of structurecriterion switching from the blind to the soft transition mode.Typically, by increasing ISI severity, the SFBF adaptation is accompanied by an increasing risk of the catastrophic error propagation effects.To eliminate this weakness, the adaptation of whitener is additionally regularized by introducing the coefficient leaky term into the existing JEM adaptation rule.
The paper is organized as follows.In Section 2 the Soft-DFE scheme is described.Section 3 recalls the theoretical background of the SFBF equalizer model and analyzes its instability in the presence of a 64-QAM signal.The tap-leaky JEM whitener is introduced in Section 4 and the parametric optimization method for the improved SFBF is presented in Section 5.In Section 6 the QAM system simulator is described and the effective equalization characteristics of the Soft-DFE with 64-QAM signal are presented.

Description of Soft-DFE Scheme
The Soft-DFE equalizer, which is presented in Figure 1, has been designed for a system transmitting -QAM,  = {4, 16, 32, 64}, signals through a time-invariant frequency selective channel.Soft-DFE includes four -spaced FIR filters in its recursive and linear parts which are defined with coefficient vectors b  = [ ,1 , . . .,  , ]  and c  = [ ,1 , . . .,  , ]  ,  = 1, 2, respectively.The received signal () is sampled at a rate that is twice bigger than the symbol rate 1/, and then odd and even samples ( 0 +−/2) =  , are alternatively shifted to the delay lines of the corresponding filters.
The Soft-DFE performs equalization through three operation modes named: blind acquisition, soft transition, and tracking.During the blind mode, the Soft-DFE effectively acts as a linear /2 fractionally spaced equalizer (/2-FSE) including four signal transformers ordered in cascade performing tasks with increasing complexity: gain control (GC), whitener (WT), FSE equalizer (TE), and phase rotator (PR), Figure 1(a).Transformers GC and WT are coupled in a pair where GC recovers the transmitted signal energy using single-coefficient equalization rule and whitener WT performs a nonflat channel spectrum equalization based on the JEM cost.At the same time and independently of (GC + WT) the equalizer TE compensates for a phase distortion (introduced by a channel + whitener combination) by using the Constant Modulus Algorithm (CMA-2) [2].In the next stage, named the soft transition mode, one of the two whiteners, selected according to energy criterion, transforms itself back into the SFBF, keeping on JEM adaptation, while the equalizer TE switches adaptation from the CMA to the decision-directed LMS (DD-LMS), Figure 1(b).Effectively, during the soft transition mode, the Soft-DFE is optimized by the combined (MSE + JEM) criterion.Finally, for the signal eye opened enough, the SFBF switches itself into the classical feedback equalizer performing DD-LMS adaptation (tracking mode).
The phase rotator PR is the second-order phase lockedloop modified in a way to evade the increased complexity of the 64-QAM constellation.PR starts the carrier phase acquisition in the blind mode by using the reduced signal constellation including only twelve corner symbols with the largest energy and then continues with the full constellation for enough opened signal eye.
The process of Soft-DFE adaptation is controlled by the MSE monitor that switches both the structure and the criterion for the a priori selected MSE-TL thresholds: for MSE-TL1 from the blind to the soft transition and for MSE-TL2 from the soft transition to the tracking mode.Besides, the threshold MSE-TL3 is introduced to switch PR operation between reduced and full signal operation.Also, it is used as a measure of equalization successfulness.

Soft Feedback Filter: Background and Problem Definition
For the purpose of simplicity, the backgrounds of SFBF operation are considered within a system transmitting a data sequence {  } through a linear noiseless channel where data   represent zero-mean i.i.d.real variables with finite variance and sub-Gaussian distribution, Figure 2. The noisefree data are used to simplify the JEM cost development while the evaluation of the equalizer performance is carried out using additive white noise channels.At the receiver side, the real-valued FBF filter (equalizer) performs data sequence recovery using "soft" neuron unit of the Bell-Sejnowski type [14] instead of a hard detection strategy.The soft FBF equalizer cancels the postcursor ISI iteratively by maximizing the joint Shannon's entropy of outputs   = (  ) where the neuron function (⋅) is a strictly monotone (increasing or decreasing) differentiable nonlinearity and the input   =   + b   r  is a sum of channel outputs   and a convolution sum of neuron outputs   and filter coefficients which are represented by vectors r  = [ ,1 , . . .,  , ]  and b  = [ ,1 , . . .,  , ]  , respectively.Under the hypothesis of correctly detected previous symbols and its maximization is equivalently treated as a minimization of the mutual information where ( , ) is a marginal entropy of the output  , .In other words, by maximizing the entropy   (b  ), the soft FBF removes the statistical dependence between the current output   = (  ) and the previous outputs, which leads to the ISI removal.It is worth to note that soft FBF transforms the sequence   with the arbitrary PDF into the maximum entropy sequence   with PDF approaching the uniformity in the limited range of the given symbol alphabet.Thus, the soft FBF equalizer minimizes the Kullback-Leilbler information divergence ((  ) | (  )) ≥ 0 with equality if and only if   and   have the same distribution [16].
The central point of the soft FBF equalizer design is the selection of its mapping function (⋅).More precisely, the neuron is selected to approximate the expected cumulative distribution of inputs according to the relation () ≈ ∫  −∞ (), () ∈ [0, 1], where () is the probability density function (PDF) of an output .In other words, the slope ()  ≈ () is a PDF matching neuron [17].Since the PDF of ISI, and hence of , is generally unknown and there is also a lack of appropriate nonlinearities, it can be practical to use parametric nonlinearities (, ) where the parameter  varies the "slope" of neuron in a way to be as close as possible to the expected cumulative probability distribution of ISI.
In [10], the basic model of soft FBF is extended into the complex domain (SFBF), and for the complex-valued nonlinearity given by the JEM type stochastic gradient algorithm is derived where  is an adaptation step size,  is a real positive parameter ("slope"), and the operator * denotes complex conjugation.Next, the operation of SFBF is divided into two subtasks performing through the self-optimized Soft-DFE scheme.At the start of equalization, SFBF switches itself into the all-pole filter controlled by the JEM algorithm (JEM-W) (see Figure 1(a)) to perform whitening (decorrelation) of channel outputs  , and then switches itself back to the decision-directed SFBF structure controlled by the JEM-D algorithm (see Figure 1(b)) to continue entropy maximization of   outputs.The optimal {  ,   } parameters for 16-and 32-QAM signals are selected by observing the effective Soft-DFE performing through blind and soft transition operation modes.During the blind signal acquisition, the Soft-DFE acts as a /2-FSE-CMA equalizer maximizing the kurtosis of outputs   given by (  ) = {|  | 4 }/{|  | 2 } 2 [18,19].Supposing a one-by-one correspondence between the stationary points in system (channel-equalizer) and equalizer domains, we have used the absolute normalized kurtosis [7,11] as a measure of kurtosis equalization efficiency.Practically, by gradually increasing   in JEM-W, the efficiency of the received signal whitening is measured through the kurtosis increase at the end of blind mode.Similarly, the performing of JEM-D is varied by   to find a minimal symbol error rate (SER) of Soft-DFE outputs during the soft transition phase.
In this phase of operation, the SER is taken as a measure of error propagation effects suppression.
The SFBF equalization efficiency, with respect to 16-and 32-QAM signals, is verified via intensive simulations, and the corresponding optimal slope parameters {  ,   } are decided in [11] as follows: { ,16 = 1.3,  ,16 = 12} and { ,32 = 1.2,  ,32 = 10}.Also, based on the experimental data, the following conclusion notes have been clarified.(ii) The relations  ,32 <  ,16 and  ,32 <  ,16 indicate that the ISI coming from the 16-QAM signal has a more picked PDF form than the one of 32-QAM signal recall [14] that the optimal slope of neuron maximizing the output entropy is inversely proportional to the variance of its input distribution.
(iii) The estimated position of SER minimum versus   stays practically unchanged by varying   in a relatively wide range about its optimal value while the value of minimum SER is being scaled.This behavior of SFBF indicates a strong influence of   , that is, JEM-W algorithm, on the entire convergence of Soft-DFE despite its suspension at the end of the blind mode.
Having in mind the above results, let us examine Soft-DFE performance in a system transmitting the 64-QAM signal which is characterized by a higher variance of the received signal compared to the one transmitting 16-or 32-QAM signal.Figure 3 presents MSE convergence of Soft-DFE for different slopes   and fixed   = 2 in the case of the worst case channel Mp-E in the given class of multipath channels (see Figure 8).MSE convergence characteristics obtained by one-run test and presented in Figure 3(a) are possibly better (in both convergence speed and residual MSE) for higher slopes   in the range from 0.1 to 0.5.However, the equalization successfulness tests, based on 2000 independent runs, have shown a decreasing trend of the equalization successfulness index (ESI) versus   , Figure 3(c), where ESI [%] represents the ratio between the number of successful equalizations and the total number of Monte Carlo runs.It is verified by additional tests that equalization failures come from SFBF instability at the time of Soft-DFE structurecriterion switching from the blind to the soft transition mode.Figure 3(b) presents the convergence characteristics of SFBF which are given in the term of coefficient vector norm b  and correspond to the MSE characteristics in Figure 3(a).For MSE > MSE-TL1 and MSE ≤ MSE-TL1 the vector (Euclidean) norm is given by ‖b  ‖ = 0.5(‖b ,1 ‖ + ‖b ,2 ‖) and ‖b  ‖ = ‖b , ‖, respectively.As can be seen, for larger slopes   = {0.2,0.3, 0.5}, the norm ‖b  ‖ demonstrates larger drifts (overgrowths) which are accompanied with sharper adaptation instability.
The described behavior of SFBF clearly indicates that the vector setup b , achieved during the blind mode is not the one which is expected by SFBF after equalizer switching and, also, the JEM-D algorithm is not enough robust to conquer such coefficients mismatch.In other words, the given SFBF model is not robust enough to map the input sequence statistic, which is strongly influenced by an increased ISI, into the PDF sequence converging to uniformity.Indeed, a similar behavior of SFBF can also be observed for 32-QAM signal but much less emphasized than with 64-QAM signal.
Based on the above experimental data, we have been motivated to make SFBF more robust to the increased variance of ISI distribution.Practically, it means to extend the operating range of SFBF towards larger values of slope   , which provide a fast initial convergence and, simultaneously, prevent a critical coefficient norm b  overgrowth.Also, it is related to the fact that the received signal whitening with a small slope, for example,   ≤ 0.15, has no sense because the JEM-W efficiency is becoming similar to the extended LMS (ELMS) algorithm [6,7].

JEM Whitener with Leaky Coefficients
The adaptive filtering techniques improved by tap-leaky algorithms ensure less drifting coefficients [20] in a number of telecommunication, automatic control, and signal processing applications [21][22][23][24].For example, the originally developed "tap-leakage" LMS algorithm was introduced in [21] to stabilize a steady-state operation of digitally implemented FSE equalizers.On the contrary, in our case of the Soft-DFE, the coefficient leaky technique is introduced to restrict an overgrowth of whitener coefficient norm and also to regularize the initial phase of equalization process.
The modified JEM cost which penalizes a whitener coefficients overgrowth by means of coefficient leakage is given by where a small positive number  (leakage factor) determines the relationship between entropic and coefficient leakage terms; the index  refers to the blind mode while the index  of the vector b  is dropped for simplicity.Based on the same optimization method applied to the original JEM cost, the corresponding algorithm with coefficient leakage (JEM-L) is given by where the term  , systematically decreases whitener coefficient modules by the rate determined by the leakage factor.In fact, the two terms in (8) controlled by the leaky and slope factors act in opposition to each other.Thus, a suitable balance between   and  has to be achieved for a fast and stable convergence of SFBF.Figures 4 and 5 illustrate the influence of JEM-L on the convergence of the coefficient vector norm ‖b  ‖ and on the kurtosis of symbols   , respectively.It is worth to note that the coefficient leaky term in (8) compensates for the absence of saturation in whitener outputs.In other words, The leak of coefficients can be seen as new information input which protects the SFBF operation from noninformative behavior caused by dropping nonlinearity (⋅) during the blind mode.
From the implementation point of view, a filter coefficient leaky technique is a simple one.The complexity of JEM algorithms is relatively low and it is practically the same as that of the CMA-2 algorithm.Precisely, the complexity of JEM error   (1 − |  | 2 ) in recursion (3) is the same as of the CMA error   (|  | 2 −   );   is statistical constant [2].Also, it should be noted that the difference between the JEM-L (8) and JEM-W (4) algorithms, given only by the leaky term  , , is practically insignificant.

Leaky and Slope Selection for 64-QAM
In this section the optimization of JEM algorithms is carried out for 64-QAM signal using similar methods invented as for 16-and 32-QAM signals.The selection of {,   } parameters is based on the kurtosis statistic given by ( 6), while the slope   is varied to reach a minimal convergence time between thresholds MSE-TL1 and MSE-TL2.The measurement of convergence time between thresholds, in terms of MSE transition time (MSE-TT), is more practical than the measurement of SER (used for 16-and 32-QAM signals) because of high values of SER (typically higher than 0.5) for severe channels.Therefore, the effective MSE convergence time during the soft transition mode is taken as a measure of error propagation recovery time.Figure 5 presents the kurtosis curves versus   obtained for a suitably selected set of leakage factors {0.0, 2 −14 , 2 −13 , 2 −12 } and the channel Mp-E.As can be seen, by increasing leakage from  = 0 (corresponds to the JEM-W in (4)) to  = 2 −12 , the kurtosis statistics are being improved by adjusting the ratio between the secondand fourth-order moments.The improvement is a result of favoring second-order statistic recovery, forced by the leaky whitening, over the fourth-order one forced by the FSE-CMA.This behavior is characterized by the kurtosis curve saturation removal for a relatively wide range of slopes   .Thus, based on the obtained kurtosis curves, the selection of {,   } parameters is decided as follows: firstly, for the roughly selected range of maximal slopes   = (0.85-0.9) corresponding to the curve  = 2 −12 , the kurtosis range Kur = (0.61-0.62) is selected and then, the other {,   } pairs are selected to reach approximately the same kurtosis values as it is presented in Figure 5.More precisely, the next ranges of JEM-L parameters {, (  )} are selected: {0.0, (0.13-0.15)}, {2 −14 , (0.35-0.4)}, {2 −13 , (0.45-0.5)}, and {2 −12 , (0.75-0.85)}.To select the optimal slope   for the JEM-D, the MSE-TT measure is observed for the motivating set of   in the range from 0.5 to 4.0.Figure 6 presents the MSE-TT in symbol intervals versus   for the given class of Mp channels.Independently of channels, the smooth hyperboliclike MSE-TT curves show that their unique minima converge into a relatively wide range of   from 1.75 to 2.25.In addition, to examine the influence of the JEM-L algorithm, that is, its parameters {,   }, on the Soft-DFE convergence after structure-criterion switching, the above experiment is repeated for the previously selected pairs {,   }.As can be seen in Figure 7, the influence of {,   } on the position of MSE-TT minima is negligible which makes the proposed parametric optimization of SFBF easier than it appears at the first glance.And, most importantly, by varying parameters {,   } it is possible to speed up the equalizer convergence.Thus, JEM-L improves the equalizer convergence by the increasing estimation quality of the whitener coefficients.

Soft-DFE Performance Evaluation
In this section the QAM system simulator is described and the final performances of the Soft-DFE with 64-QAM signal are evaluated.The simulator includes the time-invariant frequency selective (three-ray model [19]) channels with signal-to-noise ratio of 30 dB.Channels are involved in the transmitter filter designed with a roll-off factor 0.12. Figure 8 depicts the attenuation response of the Mp channels with attenuation and propagation parameters selected to gradually increase the level of ISI gradually.The length of equalizer is  = 25 and  = 5 in its TE and WT parts, respectively.The blind updating of the TE begins for its initial vectors {c 1 , c 2 } with zero components except for the centered references  1, =  2, = 0.707.The adaptation steps of the stochastic gradient algorithms are taken as a negative power of 2. The  The PR begins a carrier phase estimation using the reduced 64-QAM constellation, including twelve corner symbols with the largest energy, according to the following rule: phase discrimination is active for symbol magnitudes |  | satisfying |  | 2 ≥ 72 while for |  | 2 < 72 a phase error is set to zero.Effectively, during the blind mode PR operates as a carrier phase estimator of the 4-QAM signal.Further, for the constellation opened enough, that is, MSE < MSE-TL3, carrier phase tracking continues with the full 64-QAM constellation.
The selection of switching threshold levels MSE-TL is based on the worst case transmission scenario forcing Mp-(C, E) channels.The thresholds MSE-TL1 = 6.25 (8.0 dB) and MSE-TL2 = 0.610 (−2.1 dB) are chosen to provide the best compromise between the convergence rate and the equalization successfulness ESI during the period of time lasting 50000 symbol intervals.The equalization is successful for MSE < MSE-TL3 = 0.165 (−7.8 dB). Figure 9 presents a sequence of 64-QAM signals sampled at the output of Soft-DFE for the Mp-C channel as follows: at the start of equalization, at the time of passing thresholds MSE-TL1 and MSE-TL2, and at the end of the observed period of time.Note that the data symbols are collected and nearly uniformly distributed over the signal constellation frames at the end of both the blind mode (b) and the soft transition mode (c).
The final results are presented in terms of MSE convergence and equalization successfulness ESI.The simulations are carried out for four different pairs of {  ,   } parameters, {0., 0.14}, {2 −14 , 0.4}, {2 −13 , 0.5}, and {2 −12 , 0.75}, and the fixed   = 2.The selected combinations of parameters {  ,   } provide the maximum values of ESI and in the best way demonstrate the influence of JEM-L on the effective convergence of Soft-DFE.Figure 10  improvements are evident; the MSE convergence for Mp-B and Mp-D channels is practically the same as for Mp-C and Mp-E, respectively.Also, for the purpose of comparison, the convergence characteristics of the self-optimized DFE, named Hard-DFE, are given in Figure 10.The only difference between Soft-DFE and Hard-DFE is the adaptation method applied to their recursive parts.The recursive part of Hard-DFE is adapted by ELMS and DD-LMS algorithms through blind and tracking mode, respectively.
Besides, it is interesting to see how the Soft-DFE optimized by parameters { = 2 −12 ,   = 0.75,   = 2.0} responds to a sudden change of channel conditions.Figure 11 illustrates the Soft-DFE behavior in the situation when Mp channels introduce a strong phase jitter disturbance at the time corresponding to 24500 T intervals after the start of signal transmission.Evidently, after the sudden equalization failure, Soft-DFE demonstrates fast and stable recovery.
The corresponding results of equalization success for the Soft-DFE with JEM-L are presented in Figure 3(c).Based on the previous results, the ESI is particularly evaluated for the fixed leaky  = 2 −12 and the slope   varied in a relatively wide range from 0.5 to 0.9 aiming to verify the equalizer robustness with respect to the JEM-L algorithm.It is proved that Soft-DFE for   in the range from 0.70 to 0.75 reaches the high ESI index of 99.6%, 99.5%, and 98.2% for Mp-A, Mp-C, and Mp-E, respectively; the ESI tests are based on the 2000 independent runs.

Conclusions
In this paper we have shown that Soft-DFE blind equalizer, which has been designed for 4-, 16-, and 32-QAM signals, can be extended to the 64-QAM signal constellation by using the same computationally efficient CMA and JEM

Figure 3 :
Figure 3: (a) One-run MSE convergence of the Soft-DFE with JEM-W.(b) One-run convergence of ‖b  ‖ for JEM-W.(c) Equalization success index versus   : with JEM-W and JEM-L algorithms.

Figure 5 :
Figure 5: Kurtosis of symbols   observed at the end of the blind mode.

Figure 6 :
Figure 6: MSE transition time versus   for Mp channels.

Figure 9 :
Figure 9: Soft-DFE outputs in the window length of 1000 T at the four characteristic phases of equalization: (a) the start of blind mode, (b) MSE-TL1, (c) MSE-TL2, and (d) end of tracking mode.

Figure 11 :
Figure 11: One-run convergence characteristics of Soft-DFE in the 64-QAM system with Mp channels disturbed by a strong phase jitter acting during the 500 symbol intervals.