Speaker Gender Recognition Based on Deep Neural Networks and ResNet50

Department of Computer Science and Engineering, College of Applied Studies and Community Services, King Saud University, P.O. Box 22459, Riyadh 11495, Saudi Arabia College of Computer and Information Sciences, King Saud University, P.O. Box 51178, Riyadh 11543, Saudi Arabia Department of Commerce and Management, Seshadripuram College, Seshadripuram, Bengaluru 20, India Department of Computer Science, College of Computer and Information Sciences, King Saud University, P.O. Box 51178, Riyadh 11543, Saudi Arabia Department of Computer Science and Informatics, School of Engineering and Computer Science, Oakland University, Rochester Hills, MI, 318 Meadow Brook Rd, Rochester, MI 48309, USA -Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Guntur, 522502 Andhra Pradesh, India Gedu College of Business Studies, Royal University of Bhutan, Bhutan


Introduction
Neural networks are state-of-the-art in various classification tasks, including video and audio segmentation. Determining the speaker's identity from an audio clip of their speech is a classification problem of this sort. Nevertheless, the quantity of data necessary to produce acceptable results is one of the most critical trade-offs in neural network training. Gender is a component that results in an average physiological difference. It may increase the identification system's accuracy because males and females have diverse emotional expressions and voice processes. Integrating gender data in the development and testing processes makes the data more trustworthy. The neural network obtains another element for identifying the task-specific voice qualities for the two genders [1]. Gender categorization by the audio signal is crucial for various applications, involving targeted answers and advertising by voice assistants, population statistics via age group analysis, and automated profiling of an individual using speech data to help in a criminal probe. Additionally, for a model with a gender-specific search space, even a small quantity of data will significantly contribute to various audio systems, such as automated voice recognition, speaker identification, and content-based multimedia indexing.
A collection of features is utilized to determine the gender of a sound. The Mel-scaled power spectrogram (Mel), Mel-frequency cepstral coefficients (MFCCs), power spectrogram chroma (Chroma), spectral contrast (Contrast), and tonal centroid are among the most often used features for speech gender detection (Tonnetz). Machine learning (ML) approaches are used to build a high-quality system for distinguishing voice gender using the retrieved attributes. Each classification approach, in specific, generates a collection of hypothesis models and chooses the most optimum one. This model identifies the unknown voice label by acquiring the audio attributes and classifying the voice gender.
As the feature extraction stage has progressed to the point that many academics now see it as feature engineering, intending to develop robust feature vectors that accurately characterize structures in methods relevant to the job at hand. The primary goal of feature engineering is to create features that cluster patterns belonging to the same class together in the feature space while keeping them as far apart as possible from other categories. But autonomous representation learning has attracted increased interest to study deep learning methodologies more readily and extensively. The classification scheme is constructed with deep learning so that the encoder acquires the optimal attributes for describing patterns throughout the training phase. Additionally, because of specialized deep architectures, such as CNN, the input structures are sometimes depicted as a picture. CNN is a specialized architecture for handling the image classification task, among other things. This has prompted academics working with CNNs to create ways for transforming an audio input to a time-frequency image. Hence, another approach tackles the gender categorization issue arose by using the auditory spectrograms as inputs to our system. It sounds spectrograms are visual representations of audio. Spectrograms are very comprehensive and precise representations of speech that have been extensively employed in auditory categorization applications [2][3][4]. Deep neural networks (DNNs) trained on extracted features are very effective in removing data and have been successfully used in applications such as speech recognition [3,5] and picture identification [5][6][7]. CNN's can effectively leverage the invariance inherent in spectrograms for convolutional and pooling operations [8].
The purpose of this article is to analyze deep learning techniques to typical machine learning models trained on handcrafted features to determine if deeply learned attributes are adequate for gender categorization tasks. Deep neural networks (DNNs) and convolutional neural networks are the most accurate classifiers and feature extractors for speech gender detection, according to experimental data (CNN) classification. When used for audio categorization, [9] demonstrated the performance of prominent CNN architectures like AlexNet, VGG, Inception, and ResNet. Their method required decomposing the audio time series using a short-time Fourier transform to generate a spectrogram utilized to enter the CNN. The issue with many of these models is that they are huge and have many learnable parameters. Therefore, we concentrated on a database with just a few thousand samples since it seems improbable that these vast networks could be trained with our sparse data. Transfer learning is one way to circumvent this. This is accomplished by employing a pretrained network, freezing the values of most levels, and retraining just the last few layers using our audio data for training. This is one of the directions we took with this job. The performance achieved by fine-tuning various pretrained CNNs (ResNet 34 and ResNet50 on ImageNet) is optimal for our gender audio categorization issues.
The following are the significant contributions made to the community: (1) Presented the study of impacts of a set of handcrafted voice qualities as possible appropriate features for gender classification algorithms (2) The work contributes to our understanding of the extent to which picking voice signal features aided in the development of machine learning models

Wireless Communications and Mobile Computing
The rest of the paper is organized as follows: Section 2 describes the previous works in the domain; Section 3 details about the methodology, which consists of dataset, preprocessing techniques, feature extraction, model architecture, and evaluation metrics; Section 4 describes the results and Section 5 with discussion; and Section 6 with conclusions followed by a list of relevant references.

Literature Review
Numerous studies have been undertaken to determine the effectiveness of voice classifiers to increase the precision of programs being used. [10] recognized the speaker's gender on the TIDIGITS database with an accuracy rate of 98.65% using two-level classifiers (pitch frequency and GMM classifier). [11] analyzed voices from the Ivie corpus using four classifiers: GMM, multilayer perceptron (MLP), vector quantization (VQ), and learning vector quantization (LVQ). They had a reliability percentage of 96.4%. [12] integrated the acoustic voice levels determined by five distinct approaches into a single score level. The findings were attained using the gender dataset with an 81.7% success rate for the gender category. [13] developed a method to recognize speakers cantered on a fusion score of seven subsystems employing the MFCC, PLP, and prosodic feature vectors on three distinct classifiers: GMM, SVM, and GMM-SV-based SVM. The categorization rate of success for gender identification is 90.4% when utilizing the aGender dataset. [14] used two classifiers to a private dataset to determine gender voice: SVM and decision tree (DT) using the MFCC feature. The total accuracy of gender categorization using MFCC-SVM and MFCC-DT was 93.16% and 91.45%, correspondingly. [15] developed a method to increase the MFCC features and then modify the weighting between the DNN tiers. These enhanced MFCC attributes are assessed using DNN and I-Vector classifiers, which achieve an overall accuracy rate of 58.98% and 56.13%, accordingly. [16] examined two arrangement approaches (DNN and SVM) for robust sound classifications utilizing single and combination feature vectors. The findings indicated that the DNN strategy outperformed the noise approach because of its robustness and poor sensitivity to sound.
[17] predicted age and gender using deep neural networks. Lately, raw waveform processing in speech has become a trend in the speech field, and it has been shown that employing raw audio waveforms improves voice recognition effectiveness [18]. When raw audio-based neural networks are used, voice activity identification [19] and speaker verification [20] have also shown considerable performance gains. [19] attempts to decipher and explain how CNNs 3 Wireless Communications and Mobile Computing categorize audio information. CNN's are learned using both spectrogram and raw audio inputs, and a layer-wise relevance propagation (LRP) technique is employed to examine how the systems choose features and make choices. They demonstrated the distinct patches on the input signal that strongly connect to each output label. The paper's findings show that spectrogram inputs result in greater accuracy than raw auditory data.
In certain works, they employed gender recognition as a factor. [1] used speech recordings to build an emotion recognition model. The suggested approach includes an R-CNN and a gender information block. The suggested approach improves accuracy by 5.6%, 7.3%, and 1.5% in Mandarin, English, and German, respectively, compared to existing highest accuracy algorithms. [21] also illustrates the importance of gender and linguistic variables in vocal expression classification. They found that higher energy emotions like anger, joy, and surprise were easier to discern in male voices speaking a harsh language like German. Disgust and Fear were easier to discern in female voices in each language. They also found that when analyzing emotion across gender and language, signal amplitude, and energy are critical.

Dataset. Familiar
Voice is a corpus of speech data [20] read by users on the Common Voice website (http://voice .mozilla.org/) based on text from various public domain sources, including user-submitted blog posts, old books, movies, and other publicly available speech corpora. Its main goal is to develop and test automated speech recognition (ASR) software. There are 8,64,448 MP3 audio files in the data collection. A .tsv file including the filename, sentence, accent, age, gender, locale, upvotes, and downvotes was also included in the dataset. The audio clips were saved using the same name in the associated.tsv file. The.tsv file was filtered by deleting any missing attributes in gender, age, and accents and choosing the rows with column "downvotes" of 0. Furthermore, the gender options were limited to "male" and "female." As a result, the total dataset is limited to 3,94,818 rows with labels for and age (from diverse geographic areas, each with its own linguistic dialects and accents. We only took a subset of the created dataset for the current work. The number of audio files in the female and male categories was nearly equal to avoid bias ( Figure 1). We separated the .tsv file into two columns for further processing: filename and gender. There were 6995 male audio files and 5662 female audio files in the filtered audio collection. The WAV format converted all audio files for frequency spectrum analysis [22]. Figures 2 and 3 show the time-domain representation of the female and male audio waveforms, respectively. The chromagram of the male and female audio categories is depicted in Figures 4 and 5, which shows how the pitches for twelve different pitch classes change between the gender groups.

Fast Fourier Transform (FFT).
The fast Fourier transform (FFT) is a mathematical procedure frequently used to estimate the discrete Fourier transform of any sequence (DFT). We have a series of amplitudes sampled from a continuous audio stream in this scenario. Using the FFT technique, each frame of those N samples is converted from a time-domain discrete signal to a frequency-domain signal [24]. The FFT is considered an effective computing implementation of the DFT method, which is specified on a set of N samples fx n g as follows:

Mel Spectrogram.
A spectrogram is a visual representation of a signal's frequency spectrum. The Mel scale [25] is a mathematical representation of how the human ear works, demonstrating that people do not perceive frequencies on a linear scale. Humans are more sensitive to differences at lower frequencies than at higher frequencies. In mathematical terms, the Mel scale is the outcome of a nonlinear transformation of the frequency scale [26]. The term "Melfrequency scale" refers to a scale that is defined as where f is the signal values in hertz. mel is the signal values in the Mel scale.
The term "Mel spectrogram" refers to a spectrogram that has been converted to the Mel scale. For example, Mel spectrogram returns a power spectrogram coefficient that has been Mel scaled. Mel spectrogram object is shown in Figure 6(c).  MFCC. MFCC accurately portrays the vocal tract, a filtered shape of a human voice, and a short-time power spectrum envelope. MFCC is nothing but the coefficients that make up the Mel-frequency cepstrum) as shown in Figure 6(d). Generally, the first 13 coefficients of MFCC (the lower dimensions) are considered features because they reflect the spectral envelope. And the higher dimensions that have been deleted express the spectral subtleties. For distinct phonemes, envelopes are sufficient to convey the distinction, allowing us to distinguish phonemes using MFCC. It is a significant feature in various study fields that utilize audio signals [27]. 4.6. Spectral Contrast. The spectral contrast of an audio signal is the energy of frequency at each timestamp. It is not easy to measure energy as most audio files contain a changing frequency. However, spectrum contrast helps to measure energy fluctuation. Spectral contrast is estimated based on [28], where it considers the spectral peak, the spectral valley, and the difference in frequency between each subband. The spectral contrast is depicted in Figure 7(a), which uses the relative spectral distribution rather than the average spectral envelope to represent it. High contrast signals are clear and narrow-band, while low contrast signals are broad-band noise.

Deep Neural Network (DNN).
Five layers form the planned neural network (Figure 8). The retrieved feature vector had a length of 192. Before feeding the whole feature vector into the neural network design, it was normal-ized. The label vector was binary because our objective was to categorize the audio stream into male or female. The first fully connected layer (FCN) is configured to accept a 192dimensional feature vector. The nonlinear ReLU function was used as an activation, followed by a dropout layer. Dropout is a strategy for preventing overfitting in deep neural networks [29], and we provided a dropout rate of 0.5 while designing the network. The design of the second and third hidden levels was identical. The output layer is composed of a single node, and the activation function was Sigmoid due to its superior performance in binary classification. [7] is an exceptional architecture composed of residual layers. Additionally, ResNet is unique in that it uses global average pooling layers at the network's end rather than the more conventional set of fully connected layers. These architectural advancements result in a model that is eight times more detailed than VGGNet while remaining significantly smaller. This study examines ResNet30 (a 34-layer residual network) and ResNet50 (54-layer residual network) ( Table 2). Both CNN's utilized spectrograms created from the audio wave file as input.

4.10.
Training. The test and validation datasets together make up 10% of the total data. The DNN architecture is built using the Keras framework, supported by TensorFlow and written in Python. For the ResNet34 and ResNet 50 network training, we used the FAST AI library built on top of PyTorch. All other processing and analysis were performed on the DNN and ResnNets using NumPy, OpenCV, Scikitlearn, and other open-source tools. The training was conducted on a 32 GB NVIDIA Quadro P1000 GPU. The movement began with a 0.001 learning rate. We used the Adam algorithm as an optimizer [30]. The training could last up to 50 epochs with a batch size of 64. However, if the validation loss does not decrease continuously over a long period, early stopping will occur. It is applied to the test dataset to validate the trained model's performance. These values were evaluated considering the problem as a binary classification problem: 0 (negative) and 1 (positive). Here, we considered the 'male' category as positive and 'female' category as negative.

Results
The research goal was to design a system that could accurately identify a person's gender using their voice. A set of experiments were conducted in order to determine the most appropriate classifier for gender classification. Choosing a classifier for the gender detection problem in multimedia applications is based on identifying genders in data. Other aspects, like the amount of time required for training and classification, can also impact the decision, especially given a large amount of data to be analyzed. We also aimed at identifying whether all the handcrafted features are necessary to develop a highly accurate model. For this reason, we conducted many sets of contrast tests to see whether deep features recovered by CNN or traditional handmade features performed better on the gender dataset. The amounts of speech samples allotted to male and female classes were purposefully kept equal in the dataset.

Classification Based on Manually Extracted Features.
A total of 12 chromatograms, 128 Mel spectrograms, 40 MFCCs, and 12 special contrast and Tonnetz features were retrieved from the Mozilla Voice dataset. Afterward, we used the retrieved features to train machine learning algorithms. Additionally, we designed a DNN suitable for classifying the genders from the extracted features of audio data. The test accuracy is observed and analyzed for these trained models. The experimental results are summarized in Table 3. The DNN shows the best accuracy out of all the models trained. Table 4 provides a detailed performance analysis of the DNN on the test dataset. The accuracy and loss plots are also shown in Figure 9.

Classification Using CNN Models.
We have carried out another experiment to observe the effectiveness of the pretrained CNN models. The models were trained on spectrogram images other than handcrafted features from the audio dataset. ResNet34 and ResNet50 models are initialized with pretrained weights and fine-tuned on features extracted from the spectrograms of each audio file. The ImageNet pretrained models were trained for 50 epochs. The learning rate was 0.001. These two comparatively complex models are of larger size, and the training time for every epoch is much longer. Table 5 displayed the performance of the two models based on the precision, recall, F1 score, and accuracy. The transfer learning performs better than the DNN in terms of all performance metrics. This confirmed our expectation that the lower level features learned by the convolution layers from image data can also be applied to the audio data. Figure 10 shows the train and validation loss graphs of ResNet34 and ResNet 50 models. The more convergence is visible in the ResNet50. Figure 11 also shows the comparison of validation accuracy and error rate for these two models. The results demonstrate that the pretrained ResNet50 outperforms ResNet34. The effectiveness of each gender identification is shown in Figure 12. Male and female prediction scores are almost similar.
The study on the features suggests that the even MEL spectrogram images contain discriminating information for the gender classification. Even though the ResNet50 model is complex and took more training time, it shows impressive increment in the accuracy compared to other models trained on handcrafted features. Overall the architecture with more   [32].
We conducted a performance analysis on these two different datasets in order to better understand the generalization potential of the ResNet50 model that we constructed. The findings of the experiment are depicted in Table 6 and Figure 13.

Discussion
Paralinguistic analysis, which includes tasks such as age and gender detection, is a rapidly increasing research topic. Speech is a time-varying signal generated by the vocal tract, and the form of the vocal tracts varies significantly between males and females. Thus, automatic gender identification using voice has a broad range of applications. The age and gender of the speaker may be used by the interactive voice response system (IVR) to refer the speaker to an appropriate consultant [33] or to play background music suited for the speaker's gender/age group [34]. The variance in pitch across genders is relatively large, making it difficult to categorize the voice into different classes (e.g., emotions and pathologies) [1]. Hence, when used in conjunction with speaker identification, precise gender classification considerably lowers uncertainty in such classification. By narrowing the search space to a class of a particular gender, the classification algorithm will become increasingly accurate and reduce the mutual influence of each other. Compared to gender-neutral systems, automatic speech recognition (ASR) systems using gender-specific models get more excellent recognition rates [35]. We chose the Common Voice corpus for multilingual speech research for a variety of reasons, not the least of which being its immense size. We predicted that the results from this dataset would be more applicable to a wide variety of real-world applications, as it comprises a greater variety of languages and the amount of data per language varies from extremely tiny to relatively large [20].
The first set of classification studies concerns the classification of two gender audio classes using extracted features. As [36] shown, we used five frequency domain features to compare the performance of classical machine learning and the newly designed DNN model. The findings indicated that the proposed DNN achieved the highest accuracy (Table 3) and that the model converges to an optimal weight value for the gender classification problem (Figure 9).
In the second set of experiments, we trained CNNs using voice spectrograms since the human ear similarly perceives sounds in terms of varying frequencies across time [37]. Additionally, a two-dimensional representation of the speech signal serves as an appropriate input for CNN models for speech analysis [38]. This experiment is aimed at determining whether the deep learning method is appropriate for gender classification. If CNN effectively identifies the gender classes, we can bypass the bottleneck associated with manual feature extraction. According to Table 3, the pretrained ResNet50 exhibits the highest accuracy, F1 score, recall, and precision. Additionally, the study implies that pretrained knowledge is crucial in the network's first stages. This is because pretrained models treat spectrograms identically to images. Also, the network's fully connected layers suffer the most change, as they are task-specific [39].
When standard speech features are compared to the deep feature extracted in this study, the latter performs nearly 2% better at gender recognition than the former. As a result, the deep feature extracted in this study is more appropriate for gender representation than the manually extracted features. Additionally, when the number of layers is large, the recognition accuracy is high (Table 3). However, as the number of layers increases, the amount of calculation increases slightly and the duration of classification recognition increases [40]. As a result, we need a trade-off between accuracy and model size. Because the primary objective of this project is accuracy, ResNet50 clearly outperforms other custom and traditional models. However, if we are training models or implementing them on resource-constrained platforms with a tolerance for accuracy loss, our model choices may be different.
6.1. Comparative Analysis. There were numerous attempts on the Mozilla Common Voice dataset to classify gender. The audio samples were transformed into 20 statistical features by [41]. They trained some machine learning models and discovered that CatBoost outperforms all other predictive models with a test accuracy of 96.4%. [42] used a neural network to test the efficiency of the MFCC and Mel spectrograms. They discovered that using a combination of MFCC and Mel spectrograms, the proposed model yielded 94.32%. Following up on their previous work, the same authors [43] propose a 1-D convolutional neural network (CNN) model to recognize gender using several features taken from speech signals such as MFCC, Mel spectrogram, and Chroma. By merging the MFCC, Mel, and Chroma feature sets, the 1-D CNN model achieved a higher accuracy of 97.8%. However, our proposed model surpasses these earlier investigations (Table 7).

Conclusions
Recognizing the gender of the human Voice has been regarded as a difficult task due to its importance in various applications. For this study, we used Mozilla's "Common Voice" database, an open-source, multilanguage collection of voices with information about the speaker's gender. This article examines gender detection from Voice using various 11 Wireless Communications and Mobile Computing machine learning methods. The study discovered that finetuning simple pretrained ImageNet models trained on audio spectrograms result in state-of-the-art performance on the MOZILLA dataset, as well as acceptable performance on the SVD and RAVDESS datasets. We see that pretrained models retain much of their past knowledge, particularly in the early layers during fine-tuning. Only the network's intermediate layers are significantly altered to adapt the model to the audio categorization challenge. Additionally, we discovered that CNN models learnt deep features from energy distributions in spectrograms outperformed handmade feature extraction methods. Gender discrimination results are also rather good, with an accuracy of 98.57%, which is close to the best documented in the literature.

Data Availability
The dataset used in this study is available at https:// commonvoice.mozilla.org/en/datasets

Conflicts of Interest
The author(s) declare(s) that there is no conflict of interest regarding the publication of this paper.