Eigennoise Speech Recovery in Adverse Environments with Joint Compensation of Additive and Convolutive Noise

The learning-based speech recovery approach using statistical spectral conversion has been used for some kind of distorted speech as alaryngeal speech and body-conducted speech (or bone-conducted speech). This approach attempts to recover clean speech (undistorted speech) from noisy speech (distorted speech) by converting the statistical models of noisy speech into that of clean speech without the prior knowledge on characteristics and distributions of noise source. Presently, this approach has still not attractedmany researchers to apply in general noisy speech enhancement because of somemajor problems: those are the difficulties of noise adaptation and the lack of noise robust synthesizable features in different noisy environments. In this paper, we adopted the methods of state-of-the-art voice conversions and speaker adaptation in speech recognition to the proposed speech recovery approach applied in different kinds of noisy environment, especially in adverse environments with joint compensation of additive and convolutive noises. We proposed to use the decorrelated wavelet packet coefficients as a low-dimensional robust synthesizable feature under noisy environments. We also proposed a noise adaptation for speech recovery with the eigennoise similar to the eigenvoice in voice conversion. The experimental results showed that the proposed approach highly outperformed traditional nonlearning-based approaches.


Introduction
Speech is the most common information in telecommunication systems.Therefore, speech processing has been considered by numeral researchers.Quality and intelligibility of speech are degraded by different distortion sources such as background noises, commonly assumed as additive noises, channel noise, commonly assumed as convolutive noises, and distortion caused by speech disorders.Thus, clean (or undistorted) speech recovery is critical for speech communications.
Present single microphone noisy speech enhancement algorithms have been efficiently used for additive noise but inefficient for convolutive noise because only additive noise can be easily modeled as an independent Gaussian noise [1][2][3][4].Moreover, quality and intelligibility of speech are greatly degraded in adverse environments with joint compensation of additive and convolutive noises, but there is still a lack of efficient methods to solve this problem.
Although multimicrophone models outperform single models [5], the requirement of having more than one microphone is not always practical.Therefore, developing a method for speech recovery in both additive and convolutive noises environments, especially in joint compensation of additive and convolutive noises, when only one microphone source is provided, is a critical and interesting research topic.
In the literature, there are a few researches on learningbased speech enhancement [6][7][8][9][10][11][12][13].Among them, learningbased speech enhancement approach using statistical spectral conversion has been proposed for alaryngeal speech caused by speech disorders [8,9], body-conducted speech [10], NAM-captured speech [11], and bone-conducted speech [12].This approach is adapted from the concept of voice conversion that can be applied for both additive and convolutive noises using only single microphone.This approach can be also applied for other kinds of distortion as in speech disorders.Therefore, it might be a general learning-based approach used for speech enhancement with all kinds of

Noise Modeling in Speech
Noisy environment can be modeled by a background noise () or/and a distortion channel ℎ().In the ideal case, background noises are supposed additive while distortion channels are convolutive [16].
Assume that the clean speech is () and the noisy speech is ().In the ideal case with a convolutive channel noise source ℎ(), the noisy speech is determined as in (1) and Figure 1(a): In the ideal case with  additive background noise source (), the noisy speech is determined as in (2) and Figure 1(b): ( Real noises can be compensated from both background noise () and channel noise ℎ(), and the noisy speech can be modeled as in (3), (4), and Figures 1(c) and 1(d): Noise also is classified into stationary and nonstationary noises.In stationary noises, the noise spectrum levels do not change over time or position.On the contrary, in nonstationary noise, the spectrum levels change in time, and it does not include a trend-like behavior.Most of researches on noise reduction and also this paper are based on the assumption that noise is stationary.

Learning Stationary Information in Speech.
In most of learning-based speech applications, the "stationary" assumption helps us to avoid the learning with big data.An example is given for speaker recognition when speaker identity is characterized by the variation of short-time spectral parameters resulting in that speaker identity can be recognized by an unsupervised short training and short testing utterances [17].This approach is based on the fact that the speaker individual is "stationary" information that is fully represented in short utterances.With an assumption on "stationary" characteristics, any applications can be trained and recognized by a short training and short testing utterances.
In this section, we review some popular learning methods that can be used for learning different kinds of "stationary" information of speech with a short training: those are the neural network, the HMM, and the GMM.
Rosenblatt [18] developed the "perceptron," which was modeled after neurons.It is the starting point for numeral later works on neural networks.The performance of neural networks has been improved far from the starting point.However, neural networks have still high computational cost compared with statistical learning methods.
Statistical machine learning has been proposed with many advantages compared with neural networks [19].There are many statistical machine learning methods and algorithms.The two most popular statistical methods used for speech applications are the Gaussian Mixture Model (GMM) and the Hidden Markov Model (HMM).The probabilistic Advances in Acoustics and Vibration 3 HMM modeling is suitable for text-dependent speech applications such as speech recognition/synthesis [20].However, in text-independent speech applications such as speaker recognition or spectral conversion, the sequencing of sounds found in the training data does not necessarily reflect the sound sequences found in the testing data [21].This is also supported by experimental results in [22] which found text-independent performance was unaffected by discarding transition probabilities in HMM models.
Therefore, GMM might be one of the most suitable learning methods for training with big speech data in textindependent applications such as noisy speech enhancement.In this paper, GMM is used for training, integrated with a sparse low-dimensional speech feature.

GMM-Learning in Spectral Conversion.
As mentioned in previous section, GMM seems one of the most efficient statistical learning methods for training with speech data in text-independent speech applications.GMM is also the most popular training method used in spectral conversions [15,21].In this subsection, we present briefly the training and predicting procedure using GMM-based statistical voice conversion that we used for the proposed noisy speech enhancement method.

Training
where  denotes model parameters.The joint probability density is written as where  is the number of Gaussian mixtures.(;   , Σ  ) denotes the 2D dimensional normal distribution  with the mean   and the covariance matrix Σ  .
The th mixture weight is   which is the prior probability of the joint vector [  ,   ] and it satisfies 0 ≤   ≤ 1, ∑  =1   = 1.The parameters (  ,   , Σ  ) for the joint density (,  | ) can be estimated using the expectation maximization (EM) algorithm.

Conversion
Procedure.The transformation function that converts source feature  to target feature  is based on the maximization of the following likelihood function: where  = { 1 ,  2 , . . .,   } is a mixture sequence.At frame th, (  |   , ) and (  |   ,   , ) are given by where A time sequence of the converted feature ŷ = [ ŷ 1 , ŷ 2 , . . ., ŷ  ]  is computed as follows: The converted features can be estimated by the EM algorithm.

Universal Background Model.
There are one-to-one, many-to-one, and one-to-many VC systems [15].In manyto-one VC, the full training with all sources is expensive and sometime impossible, similar to the full training with all targets in one-to-many VC.Therefore, the source (target) independent model called UBM is introduced in the GMM-UBM speaker verification system [16], while a single, speakerindependent background model is used.The UBM is a large GMM trained to represent the speaker-independent distribution of features.The independent UBM is then used as a representative target with one-to-many VC and a representative source with many-to-one VC.
There are two main approaches that can be used to obtain the UBM model.The first approach is to simply pool all the data to train the UBM via the EM algorithm (Figure 2(a)).The second approach is to train individual UBMs over the subpopulations in the data and then pool the subpopulation models together (Figure 2(b)).In this paper, the first approach is used to train GMM-UBM due to its simplicity.When using this approach, training procedure is the same as presented in Section 3.2.1 but the training data is combined from many noisy speech conditions and environments.

MAP Adaption.
Using UBM-GMM is useful with one-to-many and many-to-one VCs.However, to improve the estimation of the model, the maximum a posteriori (MAP) adaption, also known as Bayesian learning estimation, is used to adapt the UBM into the required source (manyto-one) or target (one-to-many) models [16].In the proposed noisy speech enhancement framework, MAP adaption is used to adapt noise-independent model to the models of specific noisy conditions.Although all weights, means, and variances of GMM can be adapted using MAP, experiments show that only adapting the mean of GMM obtained best performance [16].The MAP adaption for the mean of GMM is represented as below.
Given a UBM and training vectors,  = [ 1 ,  2 , . . .,   ], we first determine the probabilistic alignment of the training vectors into the UBM mixture components.That is, for mixture th in the UBM, we compute We then use Pr( |   ) and   to compute the sufficient statistics for the mean parameter: Finally, these new sufficient statistics from the training data are used to update the old UBM sufficient statistics for mixture  to create the adapted parameters for mixture th with The adaptation coefficients controlling the balance between old and new estimates are    .While LPC has been shown to be sensitive with noises, MFCC is robust with noisy environments and it is the stateof-the-art and standard feature for speech recognitions in both clean and noisy environments [23].The robustness of MFCC is mostly caused by the perceptual Mel-scale, integrated inside MFCC.The nonlinear Mel-scale follows the psychoacoustic model which is natural with human hearing [24].Because humans are capable of detecting the desired speech in a noisy environment without prior knowledge of the noise, modeling speech features closing with human hearing has been improved performance of speech applications in noisy environments.

Noise Robust Synthesizable Wavelet-Based Features
However, MFCC is built based on the concept of Short Time Fourier Transform (STFT), in which fixed length window is used for analysis.The basis vectors of MFCC cover all frequency bands, so corruption of a frequency band of speech by noise affects all MFCC coefficients.Therefore, researchers still attempt to improve the noise robustness of MFCC and to propose other noise robust features for speech applications in noisy environments.

Synthesizability of Traditional Speech Features.
Feature extraction is a critical analysis stage for both recognition and synthesis systems.In recognition tasks, the features can be any parameter that characterizes speech.However, in synthesis tasks, the speech features are usually required invertible or synthesizable.
LPC and MFCC features are two indirectly synthesizable features.To synthesize speech, LPC or MFCC needs to be combined with 0 in a VOCODER, which is one popular source/filter synthesizer widely used in speech coding and synthesis [20].
In VOCODER, 0 and random noise are used for source excitation.In the literature, many researches show that VOCODER produces "buzzy" synthetic speech [25].Therefore, the requirement of combination with 0, which causes the indirect synthesizability of MFCC, limits the efficiency of MFCC in speech synthesis.Currently, MLSA filter, one kind of VOCODER using MFCC and 0, has been still used in state-of-the-art TTS [20].The use of directly synthesizable features without using source/filter model is expected to solve the "buzzy" problem of VOCODER in speech synthesis.

Perceptual Wavelet Packet.
The discrete wavelet transform (DWT) of a general continuous signal () is the family (, ), defined in (15): The indexes of (, ) are called wavelet coefficients.The inverse DWT (IDWT) reconstructs the signal () from wavelet coefficients (, ) as in (16): With the deep mathematical formulation of DWT, IDWT can be found in [26].
In wavelet analysis (wavelet decomposition), using the DWT, a signal is split into an approximation and a detail.The approximation is then itself split into a second-level approximation and detail, and the process is repeated.Wavelet synthesis (wavelet reconstruction) is the inverse process of the wavelet analysis.
In wavelet packet analysis, the details as well as the approximations can be split; therefore, the subband structure can be customized with a user-defined wavelet tree.Recent researches in wavelet show that the integration of the wavelet packet and the psychoacoustic model into the perceptual wavelet packet transform (PWPT) may improve performance of speech applications [27][28][29].
It is shown in the literature that PWPT has significant performances with noisy speech recognition and speech coding in comparison to the conventional wavelet.
In psychoacoustic model, frequency components of sounds can be integrated into critical bands that refer to bandwidths at which subjective response become significantly different.One widely used critical band scale is Mel-scale in MFCC [24], and another popular scale is Bark scale [30].
The nonlinear Mel-scale and Bark scale can be used to design the wavelet tree for the perceptual wavelet packet.PWPT has been used to extract robust and synthesizable features in the literature.

Noise Robustness and Synthesizability of Wavelet-Based
Features.Wavelet has fine time and frequency resolution, and the effects of noise on speech are localized in some specific subbands.Therefore, wavelet is expected to be an efficient tool for noise robust feature extraction.
In the literature, there are many noise robust wavelet speech features that have been proposed.These features can be grouped into two main categories.
The first category computes the sum (or weighted sum) of energies in each subband to form the whole feature [27,28].
The second category simply uses the wavelet coefficients, retaining the time information, to form the feature [29].There are also some mixed categories.
In the first category, the time information in the wavelet subbands is lost into the subband energies.Moreover, this kind of features is noninvertible or nonsynthesizable.
The use of wavelet coefficients in the second category is simple while keeping noise robustness of the wavelet analysis.Moreover, it is known that inverse wavelet transform can perfectly reconstruct signal from wavelet coefficients.Therefore, the feature using wavelet coefficients is robust with noise and synthesizable and this feature is used in the proposed noisy speech enhancement method in this paper.

Features Decorrelation and Compression with DCT.
Although the feature using wavelet coefficients is robust with noise and synthesizable, the simple feature concatenated from all wavelet coefficients in all sub-bands is very highdimensional.Moreover, wavelet coefficients are correlated within and between sub-bands [31].Therefore, if we want to use wavelet coefficients feature in both recognition and synthesis tasks, we need to decorrelate and compress wavelet coefficients feature vector.Although principal component analysis (PCA) [32] is the most popular method to reduce feature dimension, it is difficult to reconstruct original feature for synthesis tasks.The most popular invertible decorrelation transform is DCT [33].
The most common DCT definition of a 1D sequence of length  [33] is  for  = 0, 1, 2, . . ., −1.Similarly, the inverse transformation is defined as for  = 0, 1, 2, . . .,  − 1. () is defined as The DCT is often used in signal processing because it has a strong "energy compaction" property [34] where most of the signal information tends to be concentrated in a few lowfrequency components of the DCT.In this paper, DCT is used to decorrelate wavelet-based feature in this proposed noisy enhancement method.

Eigennoise Speech Recovery Framework
The approach of using eigenfaces for recognition was developed by Sirovich and Kirby [14].A set of eigenfaces can be generated by PCA on a large set of images depicting different human faces.Informally, eigenfaces can be considered a set of "standardized face ingredients," derived from statistical analysis of many pictures of faces.
Developed from the concept of eigenfaces, Ohtani et al. proposed an eigenvoice-GMM voice conversion [15].
In this paper, we call the UBM of a large set of noisy environments "eigennoise," and then we proposed a speech recovery approach using GMM-UBM-MAP based on the joint factor analysis that we called "eigennoise" approach.
The noise robust synthesizable feature analysis and synthesis are described in Figure 3, and we use PWPT to extract wavelet coefficients from input speech.The coefficients in each subband are highly correlated and bear a lot of redundant information, especially that, in high bands, there are a lot of small coefficients or zeros.Therefore, DCT is used in each subband to decorrelate within-band correlations.After concatenating coefficients from all bands to form the whole coefficients, DCT is used again to decorrelate the betweenband correlations.Both PWPT and DCT are completely invertible; thus, the speech is perfectly reconstructed.
The eigennoise training and conversion are presented in Figure 4. Noisy speech with several noisy environments is used for training the noise-independent eigennoise model (GMM-UBM) as shown in Figure 4(a) and presented in Section 3.2.3.The noise-independent model is then adapted to each noise-dependent noise model later as shown in Figure 4(b) and presented in Section 3.2.4.Clean speech is converted from correspondent noisy speech and noisedependent model as shown in Figure 4(c) and presented in Section 3.2.2.

Implementation and Evaluations
6.1.Data Preparation.The clean speech data used in our evaluation was the well-known English MOCHA-TIMIT.The noise database is the NOISEX-92.We created some artificial noisy environments simulating the additive background noise, the convolutive channel noise, and the mixed noise.The noise sources were selected from NOISEX-92.
We simulated the practical open-dataset testing, in which the testing noisy condition does not match the training conditions.The noise inputs of the artificial noisy environment All enhanced noisy speech was evaluated with the objective tests while only mixed noisy speech with SNR of the noisy speech approximated −10 dB, which was the closest with the ultra-high real noises, was evaluated with the subjective test.

Implementation Parameters.
In all experiments, the number of Gaussian components , which should be chosen large enough if we have enough data for training, was chosen as 15.The adaptation coefficient  was initially set to 0.5.The speech data used for all evaluations were resampled at 8 KHz, yielding a bandwidth of 4 KHz, and there are approximately 17 first critical bands among 25 bands in Bark scale.Therefore, the wavelet coefficients feature had 17 coefficients.The order of LP analysis  was chosen as 17 also.The frame size was chosen as 30 ms and the overlapped interval was 15 ms.

Objective Evaluations for
Speech Quality.To evaluate our proposed "eigennoise" approach with two features LP [13] and wavelet, we implemented and compared our methods with the standard nonlearning-based spectral subtraction [4] and Wiener-filter [5] methods.We used Peak-Signal-Noise ratio (PSNR) for objective evaluation.The average PSNR results are shown in Figures 5(a), 5(b), and 5(c).The results reveal that while quality of enhanced speech with nonlearning-based methods depended linearly on the input SNRs, it was acceptable with additive noises but very bad with convolutive and mixed noises.On the contrary, performance of learning-based methods was quite independent with the input SNRs, as well as the kinds of noise.The proposed "eigennoise" speech recovery with wavelet-GMM outperformed the proposed "eigennoise" speech recovery with LP-GMM.In general, learning-based noisy speech enhancements greatly outperformed the nonlearning-based methods.
6.4.Subjective Evaluations for Speech Intelligibility.The speech signals of 100 English words with clean, noisy, and enhanced signals were played in random order in the tests for 5 native English subjects.The subjects were asked to listen to each word only once and write down what they heard.Speech intelligibility could generally be evaluated using the average recognition accuracy scored by all subjects.The results are shown in Figure 6.The subjective evaluation results also support that the nonlearning-based methods reduced the intelligibility of speech while the learning-based methods improved much intelligibility of speech.In addition, the "eigennoise" method with wavelet-GMM outperformed that with LP-GMM method.

Conclusions and Discussions
For adverse environments with joint compensation of additive and convolutive noises, one of the biggest challenges in noisy speech enhancement, the proposed learning-based approach using spectral conversion presented in this paper, is one promising candidate among a few available approaches.
However, the proposed framework and methods have still some remaining issues needed to be studied in the future.The two biggest issues are the requirements of hardware performance for training and the efficient training methods with big data.
There are many kinds of real noise.Thus, to build a practical learning-based noisy speech enhancement usually requires training with several noisy speech conditions, corresponding with the several real noise environments.Therefore, the hardware performance is required to be very high to cope with training of huge corpus.This requirement is not available at the turn of the millennium.There is good news that the hardware performance has been developed rapidly recently.While the first processor Intel 8080 has a clock rate of 2 MHz, speed of modern present processor overcomes 8 GHz [35].In addition, processing performance of computers is increased by using multicore processors, in which processor can handle numerous asynchronous events, interrupts, and so forth.Recent advances in computer hardware researches and applications reduce the difficulty of training with gigantic corpus.Therefore, the first limitation of the learning-based noisy speech enhancement can be overcome with newest hardware technologies.
With the rapid developments of statistical learning methods, learning-based noisy speech enhancement could be much more efficient compared with the beginning results.In this paper, we proposed a wavelet-GMM method that we call eigennoise speech recovery method.The GMMbased training methods have been shown to be efficient with big speech data.However, the computational cost is still necessary to be improved in the future.

Figure 2 :
Figure 2: GMM-UBM training: (a) data from subpopulations are pooled prior to training the UBM via EM algorithm; (b) individual subpopulation models are trained then pooled to final UBM model.
4.1.Noise Robustness of Traditional Speech Features.It is known that noisy speech signal varies largely with different kinds of noise.In learning-based speech applications, it is able to recognize and synthesize perfectly noisy speech as in clean environment if the noisy environment of training data is identical to that of the testing data.Unfortunately, the noisy environment of testing data is seldom known in advance, and it is difficult to train the data in all possible kinds of noisy environment.When the noisy environments of training data are different from the noisy environment of testing data, the recognition and synthesis system typically performs much worse.Therefore, it is necessary to understand and eliminate variance in the speech signal due to the environmental changes and thus ultimately avoid the need for extensive training in different noisy environments.As a consequence, the learning-based speech applications in noisy environments require robust features, which are insensitive with noise environments.State-of-the-art speech recognition is based on source/filter model to extract vocal tract features or spectral envelope features separated with source features.The two most popular spectral envelope features are linear prediction coefficients (LPC), and Mel-Frequency-Cepstral-Coefficients (MFCC).The most popular source feature is fundamental frequency (0).