Multithreshold Microbial Image Segmentation Using Improved Deep Reinforcement Learning

Image segmentation technology can eectively extract the foreground target in the image. However, the microbial image is easily disturbed by noise, its greyscale has the characteristics of nonuniform distribution, and several microorganisms with diverse forms exist in the same image, resulting in insucient accuracy of microbial image segmentation. erefore, a multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is proposed.e wavelet transformmethod is used to remove the noise of the microbial image, the threshold number of themicrobial image after denoising is determined by calculating the number of peaks of the grey histogram, and the foreground target of the microbial image is enhanced by the mean iterative threshold segmentation method, the preliminary segmentation of the microbial image is realized, the multithreshold microbial image segmentation model based on ResNet-Unet is constructed, and the cavity convolution and dual Q network mechanism are introduced to improve the segmentation model. e preliminary segmented microbial image is input into the improved segmentation model to realize the segmentation of the multithreshold microbial image.e results show that the proposed algorithm can eectively remove the noise of microbial images.With the increase in the number of thresholds, the peak signal-to-noise ratio, structural similarity, and feature similarity show an upward trend, and the loss rate of the model is less than 0.05%.e minimum running time of the algorithm is 3.804 s. It can eectively and quickly segment multithreshold microbial images and has important application value in the eld of microbial recognition.


Introduction
In people's daily life, microorganisms are everywhere, in the recognition of microorganism images, how to accurately obtain the features of microorganism images is the core problem that needs to be addressed [1]. Extracting image features and obtaining microbial targets by image segmentation is a commonly used e ective method. However, the microbial image segmentation is often susceptible to various factors, resulting in poor microbial image target extraction [2,3]. First, microorganisms are diverse, most of them have nonrigid features, their morphology shows nonregularity, their shapes are very di erent and their sizes vary. Second, in the microbial images, the di erence degree between the image target and background greyscale is small. For multitarget microbial images, the spatial distribution of microorganisms is crossed and overlapping, and conventional segmentation techniques may produce a pseudocontour phenomenon [4]. In addition, noise is another important factor a ecting the imaging e ect of microbial images. Excessive noise will cause the lack of detailed information in microbial images, especially when there is a large number of bright signals, the microbial target will be submerged in the background, resulting in image quality degradation [5].
ese factors can adversely a ect image segmentation and lead to the degradation of image segmentation quality and e ciency. erefore, it is important to study a new multi-threshold microbial image segmentation method.
To address the important research topic of multithreshold microbial image segmentation, Li et al. [6] implemented the segmentation process of greyscale image thresholding by constructing a quantum circuit, which can complete the acquisition of the histogram of pixel distribution and determine the image threshold by obtaining the maximum interclass variance of each greyscale level, to achieve image segmentation. However, the single threshold value determined by this method is not suitable for the segmentation of a wide variety of microbial images. Li et al. [7] proposed to improve the traditional fuzzy C-mean algorithm for poor image segmentation by using a parallel Lévy-grey-wolf optimization algorithm to improve the image segmentation by obtaining the best initial centre. However, this method still results in a serious loss of image detail information. Cai et al. [8] proposed the use of an improved particle swarm optimization algorithm to achieve segmentation processing of thresholded images, and the method obtained a small number of image thresholds, which has limitations in applying it to the segmentation of a variety of microbial images. Jia et al. [9] proposed to improve the multiverse optimization algorithm using the Leyland flight mechanism to improve the accuracy of image segmentation by accurately determining the number of image thresholds. However, this method fails to achieve accurate learning of image information due to premature convergence. Sun et al. [10] proposed an image small target recognition method based on a deep reinforcement learning method using a multilayer convolutional neural network to achieve image target segmentation. However, the convolutional neural network constructed by this method suffers from gradient loss, which reduces the image segmentation performance and thus has a serious impact on the accurate recognition of image targets. To improve the quality and efficiency of multithreshold microbial image segmentation, a new multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is proposed, and the contributions of this paper are as follows: (1) e wavelet transform method is used to remove the microbial image noise, which improves the image quality and reduces the interference of noise to the subsequent image segmentation (2) e foreground target of microbial images is enhanced to achieve the initial segmentation of microbial images and lay the foundation for the subsequent multithreshold microbial image segmentation (3) To improve and introduce a reinforcement learning mechanism to build the relevant segmentation model to achieve the ultimate aim of image segmentation quality and efficiency

Multithreshold Microbial Image
Segmentation. e process of multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is shown in Figure 1.
Analysis of Figure 1 shows that the microbial image is preprocessed to remove the noise contained in the image. e calculation of the number of image thresholds is completed by analyzing the number of wave peaks in the grey histogram to determine the number of thresholds. e foreground target of the microbial image is enhanced to achieve the initial segmentation of the background and foreground. e image segmentation model based on ResNet-Unet is constructed, and the improvement of the segmentation model is completed by introducing the null convolution and double Q network mechanisms, and the initial segmentation results are input to the improved model to obtain the segmentation results of the relevant multithreshold microbial images.

Microbial Image Denoising Based on Wavelet Transform.
In this paper, the wavelet transform method is used to realize the noise reduction of microbial image. Its principle is as follows: set f(i, j), i, j � 1, 2, · · · N as the original microbial image, in which the value of N is among the integer power of 2, (i, j) is the pixel point of the image, and the grey value of the pixel is expressed as f(i, j). When a large amount of noise is covered in the original multithreshold microbial image, the noisy microbial image can be described by the following equation [11]: where b(i, j) is the noise in the microbial image, which meets the Gaussian distribution, the mean value is 0 is, σ 2 is the variance which is independent just as f(i, j). It is processed using the wavelet method, which is described by the following equation: where for noisy microbial images, Z v i,j is the wavelet coefficient. For the original microbial image, Z f i,j is wavelet coefficient. e noise in the microbial image, Z b i,j is wavelet coefficient. e multithreshold microbial image noise reduction process based on wavelet transform is as follows: Step 1: wavelet transform is used to process noisy microbial images v(i, j). After s layers of orthogonal redundancy decomposition, the wavelet coefficient can be determined, expressed as W v i,j (s, j), and j � 1, 2, · · · s.
Step 2: the estimation of noise variance σ 2 is realized from different decomposition levels and directions [12].
e following calculation equation is as follows: Step 3: the parameters of the noise threshold are determined. For the original microbial image, σ f i,j is the variance of Z f i,j . A satisfies a Gaussian distribution whose variance Equation can be described by the following equation. Z v i,j satisfies Gaussian distribution, the calculation equation is as follows: Mathematical Problems in Engineering Step 4: determine the threshold coefficient, expressed as β. rough β, it can update the thresholds of different high-frequency sub-bands in different decomposition layers. e calculation Equation of β is expressed as follows: where for wavelet coefficient after j layers of decomposition, L k is the length of layer k.
Step 5: determine the threshold calculation Equation, described as follows: e wavelet coefficient of the denoised image is determined by the wavelet soft threshold processing method, and its equation is described as follows: where WST(·) is the soft threshold processing function. e method to obtain the original microbial image f(i, j) is to adopt the inverse wavelet transform based on determining Z v i,j (s, j). Based on the above-given process, the noise reduction processing of microbial images can be obtained and realized.

Determination of Multithreshold for Microbial Images.
In the multithreshold segmentation of the microbial image, the selection of threshold number has a direct impact on the segmentation effect of the microbial image [13]. e accurate threshold number can effectively avoid the misclassification of false peaks in the histogram of the microbial image and improve the segmentation effect of the microbial image. For the microbial image, obtain its grey histogram, expressed as h. Suppose U is the number of grey levels contained, i is the grayscale, and i ∈ [0, U − 1]. e mean value of frequency is expressed as h ave whose solution can be obtained by h ave � mean(h). In this paper, the threshold amount of microbial image is determined by the following steps: Step 1: the denoise-colour or microbial image is obtained, perform the grey operation on it, and obtain the grey histogram.
Step 2: according to the obtained grey histogram, the frequency comparison relative to the grey value is completed in corresponding order. For any grey value, when its frequency is greater than that of the adjacent grey value, the determination of the wave crest can be realized, that is, the frequency relative to the grey value.
Step 3: the histogram peak can be obtained through step 2, but there are some pseudopeaks in it. To ensure that the false peak misclassification problem does not occur, set D as the peak spacing, and consider that there is only one peak in D area.
Step 4: most of the pixels in the microbial image are covered by each target. To accurately determine the wave crest [14], it is necessary to further screen the wave crest, and the screening rule is set that its grey value should be higher than h ave .
Step 5: set A i as the wave peak, which can be expressed by the frequency of the grey value of i, and A n refers to the number of wave peaks. e solution Equation is described as follows: k refers to the frequency, and k ∈ [1, 2, . . . , n]. A k i refers to the wave peak with the frequency as k.
Step 6: thus, the threshold number of microbial images can be obtained, expressed as m. e following solution equation is described as follows:

Foreground Target Enhancement of Microbial Image
Using Mean Iterative reshold Segmentation. e mean iterative threshold segmentation method is widely used in multithreshold image segmentation because of its low computational difficulty [15,16]. erefore, this paper uses this algorithm to enhance the foreground target of e principle of this method is to divide the microbial image into two categories, namely, foreground and background, according to the different grey attributes of microbial image foreground and background. e grey levels of each category show different characteristics. e classification of image pixels is realized by threshold T. is method can distinguish the foreground and background of microbial images. e specific steps are as follows: Step 1: set the original threshold value T. e average grey value of the microbial image is taken as its initial value.
Step 2: the foreground and background of the microbial image are distinguished by threshold value T, which are expressed as E1, E2. When the grey value of the pixel is higher than the threshold value T, it is classified into E1. Otherwise, it is classified as E2.
Step 3: get the solution of the mean grey value of all the pixels of E1, E2, which are expressed as μ 1 and μ 2 .
Step 4: according to the threshold number m of microbial image m determined in Section 2.2, the threshold value T is adjusted by the following equation: Step 5: repeat steps 2, 3, and 4, and the algorithm ends when the average change of the last two times is lower than the set value.
Step 6: thus, the new threshold value T ′ can be obtained. rough T ′ , the distinction between the foreground and background of the microbial image is realized.

Multithreshold Microbial Image Segmentation Model
Using ResNet-UNet. Resnet-34 is a residual convolution neural network. Its structure has 34 layers and includes three parts: independent convolution module, residual convolution module, mean pooling, and full connection module [17]. e independent convolution module consists of a size of 7 × 64 channel convolution kernel of 7 and 3 × 3. e former can realize the convolution operation of the input image, and the latter is used to perform the maximum pool processing of the image. e residual convolution part contains four residual convolution blocks with differences. For any residual convolution block, it is realized by twice convolution, batch regularization, and ReLu. Taking the residual convolution block as a processing unit, complete the residual processing of the residual convolution module with 3, 4, 6, and 3 different times, respectively, and then "smooth" the feature vector.
rough mean pooling and full connection, the microbial image is segmented and the segmented image is obtained, to reduce the impact of network gradient disappearance on network performance [18].
e UNet network is composed of the front and rear parts, in which the former is used to obtain microbial image features, and the latter can realize upsampling of image information [19]. According to the dimension difference of the feature map, it can be divided into 5 layers from top to bottom, and 3 layers × 3. e convolution and ReLu adjustable linear unit are used as the structure of each layer. For the adjacent two layers, the feature extraction structure reduces the size of the feature image and improves the feature dimension through feature cutting and downsampling. e upsampling structure enlarges the feature size of the microbial image through feature difference and upsampling and reduces its dimension. For the same layer, the front and back structure of the net network is completed by feature replication and splicing. en, after the last processing of the upper sampling structure, execute 1 × 1. Convolution and ReLu operation can realize the segmentation of microbial image and obtain the target segmentation image.
e ResNet-UNet designed in this paper takes ResNet-34 residual convolution neural network as the feature image extraction network of microbial image. Based on the basic principle of the UNet network, the resolution restoration of microbial images is realized through upsampling, feature replication, and splicing. e specific steps are as follows: take the microbial image processed in Section 2.3 as the input of the ResNet-UNet network, and its size is 512 × 512. e characteristic map of the microbial image is obtained by ResNet -34 network, but its network structure needs to be adjusted, its independent convolution and residual convolution modules are retained, and the mean pooling and full connection modules are deleted. To ensure the tight connection between ResNet -34 network and the U-Net network, the number of convolution channels of the fourth residual convolution module of the ResNet-UNet network needs to be designed to have the same number of convolution channels as the convolution at the end layer of ResNet -34 network. According to the principle of the UNet network, the difference processing of the image is realized through upsampling to achieve the purpose of image resolution restoration. When there are features with the same size in upsampling and extracting the features of the image, they need to be spliced and fused, which can be realized through feature replication and splicing processing. After that, three tests were carried out twice 3 × 3 convolutions, batch regularization, and ReLu processing. e sigmoid function is applied to the last layer convolution of the upsampling structure to ensure that the ResNet-UNet network has a strong learning ability and improves the classification accuracy. en the nonlocal operation is used to realize the integration of microbial image information. rough 1 × 1 convolution and upsampling processing, the initial segmentation of the multithreshold microbial image is realized. To ensure that the ResNet-UNet network has a strong feature extraction ability and make the obtained microbial image feature map contain complete image details, this paper improves the ResNet-UNet network, introduces void convolution, and replaces the original convolution based on the original mean pooling and the deletion of the full connection module. While ensuring that the network parameters do not increase, the receptive field of the image is improved so that the details of the microbial image are not lost. For void convolution, it can be regarded as porous convolution, i.e., inserting pixels with a 0-pixel value between the pixels of the original convolution core, to increase the size of the convolution core and improve the image receptive field [20,21].

Dual Q Network
Mechanism. e improved ResNet-UNet network realizes the panoramic perception of multithreshold microbial image features, obtains the microbial image feature map, takes it as the training data set of reinforcement learning, fully considers the dual Q network mechanism, and introduces the target Net as a network copy of the depth Q value. e target value function is determined through the ResNet-UNet network and provided to the depth Q-value network [22]. While continuously adjusting it, the target Net, give play to the decision-making performance of the deep Q-value network and complete the feature extraction of the microbial map and the collaborative operation of parameters.
rough the positive feedback mechanism, the error of the collaborative operation process is adjusted, to establish the best segmentation mechanism for the multithreshold microbial image. e objective function gradient function equation is as follows: where state information and its collection are represented as s and S, the set of action a is A; the strategy is π, the network parameter is θ. Q π (s, a) is used to describe the expected reward of choosing an action a when the status is s, the probability distribution function is expressed as ρ. According to its value, the determination of the optimal action in the current state can be realized, E is the reward for the best action in the current status.
According to the objective function gradient function of equation (12), the deterministic strategy expression is obtained, use function μ and according to the action information, actions can be obtained. μ is regarded as the optimal action strategy a t � μ(s t |θ μ ).
e quantitative analysis of multi-threshold microbial image segmentation can be realized by the following equation: Under competitive conditions, equation (13) has poor stationarity. erefore, this paper performs the first-order derivation. At this time, the deterministic strategy gradient can be described by the following equation: e compatibility of equation (14) is outstanding. It can complete the segmentation of various multithreshold microbial images in continuous adaptive learning and obtain the best-segmented image. Due to the variety and diversity of microorganisms, there are many characteristics of microorganisms [23]. erefore, in this paper, the strategy network μ is regarded as the actor. e function (s, a) is fitted through the value network and regarded as critical, to complete the recognition and segmentation of different multithreshold microbial images. e expression of the objective function is described by the following equation:

Data Sets and Evaluation Index.
Experimental data sets were used Earth Microbiome Project, Human gut microbiomes, and Human Microbiome Project. Earth Microbiome Project: EMP, founded in 2010, is a large-scale crowdsourcing work aimed at analyzing the global microbial community. e general premise is to study from the perspective of the microbial community itself. e researchers used amplicon sequencing, metagenomics, and metabolomics to analyze 200000 samples from these communities, including 100000 microbial images, which is very suitable for multithreshold microbial image segmentation. Human gut microbiomes: human gut microbiomes refer to all microorganisms inhabiting the human gastrointestinal tract. e different roles of intestinal microbiota in human health and disease have been rerecognized. Metagenomic research has changed our understanding of the taxonomy and functional diversity of human microbiota, such as promoting the maturation of the immune system and providing a direct barrier against pathogen colonization. e data set includes a series of studies on human intestinal microbiota, including images, papers, and patents. In this paper, two thousand microbial images are used for experimental tests to verify the application effect of different algorithms. Human Microbiome Project: the overall mission of the HMP is to generate resources to facilitate the characterization of the human microbiota to further our understanding of how the microbiome impacts human health and disease. e initial phase of the project, HMP1, established in 2008, characterized the microbial communities of 300 healthy individuals, across several different sites on the human body: nasal passages, oral cavity, skin, gastrointestinal tract, and urogenital tract. 16S rRNA sequencing was performed to characterize the complexity of microbial communities at each body site and to begin to ask to investigate whether there is a core healthy microbiome.

Mathematical Problems in Engineering
Metagenomic whole genome shotgun sequencing provided insights into the functions and pathways present in the human microbiome.
A total of five thousand microbial images were selected in the earth microbiome projects data set, human gut microbiomes data set, and Human microbiome project data set, the discount coefficient is 0.95, and the learning rate is 10 − 3 . e value of strategy reward is [− 1, 1], and the reward of action is − 1. e parameters of the multithreshold microbial image segmentation model are adaptively selected to achieve a better effect of model training. Among them, four thousand images are used to train the segmentation model, and the other images are used for image testing. Different algorithms are applied to the segmentation of microbial images in the data set to analyze the image segmentation effect of this algorithm.
(1) Image denoising ability: due to the interference of noise factors when collecting microbial images, the visual effect of the obtained microbial images is poor. erefore, the clearer the image after denoising, the stronger the denoising ability.
(2) Peak signal-to-noise ratio (PSNR): it is an index for evaluating image quality. e calculation Equation of this index is as follows: where MSE refers to the mean square error between an original image and a denoised image, and n refers to the number of bits of the sample value. (3) Structural similarity (SSIM) is an important index to judge the contrast of segmented multithreshold microbial images and whether the structure is suitable and complete.
where x and y refer to the two input microbial images, l(x, y) refers to the brightness comparison function, c(x, y) refers to contrast comparison function, s(x, y) refers to structure comparison function, and α, β, c are different constants. (4) After using the algorithm to segment the microbial image, the similarity of the segmented image features compared with the original microbial image can be reflected by feature similarity (FSIM). e value range of FSIM is [0, 1], and the closer its value is to 1, the closer their characteristics are. e calculation Equation of this index is as follows: where PC m (x) refers to phase consistency judgment function, and S L (x) refers to coupling functions of different terms.
(5) Loss rate: it refers to the rate of loss in the process of segmentation of microbial image by an algorithm. e calculation equation of this index is as follows: where t refers to the planned quantity of microbial image experiment, and i refers to the total amount of microbial images.

Results and Discussion
In this paper, three microbial images randomly selected from the Earth microbiome projects data set, human gut microbiomes data set, and human microbiome Project data set are denoised first. By comparing the microbial images before and after denoising, the image denoising ability of this algorithm is analyzed as shown in Figure 2. According to Figure 2. e three selected microbial images have a lot of noise, which has the problems of too high brightness and too large contrast, resulting in the loss of some details of the microbial image, which seriously affects the visual effect of the microbial image. After denoising with this algorithm, the supersaturation of the microbial image is suppressed, the brightness and contrast of the image are restored, and the microbial image information is complete and rich in detail. erefore, it can be determined that the algorithm in this paper has image noise reduction performance, and the noise reduction effect is outstanding. e threshold number of the microbial image has a direct impact on its image segmentation effect. is algorithm is used to determine the threshold number of Earth Microbiome Project data set, Human gut microbiomes data set, and Human Microbiome Project data set. e Influence of different threshold numbers on the microbial image segmentation effect is analyzed by the PSNR index as shown in Figure 3.
According to the data in Figure 3, PSNR index is used to reflect the visual effect of the segmented microbial image, and the image distortion decreases with the increase of the PSNR index.
is algorithm is used to determine the threshold number of microbial image. With the increasing number of thresholds of the microbial image, the PSNR index of the microbial image after segmentation shows an increasing trend. When the threshold number of the Earth Microbiome Project data set increases to 6, the growth trend of the PSNR index tends to be stable. us, it can be determined that when the number of thresholds is 6, the image can achieve the best segmentation effect; similarly, it can be determined that the threshold numbers of the Human gut microbiomes data set and Human Microbiome Project data set are 8 and 4, respectively.
According to the relationship between the threshold number of different microbial images and SSIM, the image segmentation effect of this algorithm is analyzed. It is shown in Figure 4.
According to Figure 4, with the increasing number of threshold values of microbial images, SSIM indexes show an upward trend. When the number of threshold values is  By analyzing the relationship between FSIM index and threshold number, this paper realizes the analysis of the multithreshold microbial image segmentation effect. It is shown in Figure 5.
e FSIM index showed an upward trend with the increasing number of thresholds. When the number of thresholds increases from 1 to 2, the FSIM increase of the three data sets is small. When the number reaches 4, the three FSIM curves show a rapid growth trend and continue to increase the threshold number. e FSIM index of the Earth Microbiome Project data set changes slightly. After the threshold number of the Human gut microbiomes data set is 8, the FSIM does not change, and after the threshold number of the Human Microbiome Project data set is 6, the FSIM index remains stable. From the above-given analysis, when the threshold number of three data sets is 6, 8, and 4, respectively, the characteristics of the segmented image can be closer to the original microbial image.
Based on the microbial images of the Earth microbiome projects data set, human gut microbiomes dataset, and human microbiome Project data set, the image segmentation accuracy of this algorithm is verified by the loss rate. erefore, this paper verifies the image segmentation accuracy of this algorithm by analyzing the model loss rate index under the conditions of training and testing. It is shown in Figure 6. e training image set is used to complete the training of the multithreshold microbial image segmentation model, and the test image set is input into the trained model. e loss rate indexes of training and test image set gradually decrease with the increasing number of iterations. After 60 iterations, the segmentation model begins to converge, and the loss rate curve under the test condition fluctuates at the beginning of the iteration, which is due to the wide variety of microorganisms and the small variety of microorganisms in the training image set. However, this model can better learn the feature information of the microbial image and reduce the loss rate of the model, which is less than 0.05%. e experimental results show that this algorithm can effectively reduce the loss rate of the model and improve the segmentation accuracy of the multithreshold microbial image. e quantum-based image segmentation algorithm in Li et al. [6], the parallel Lévy grey wolf optimized segmentation algorithm in Li et al. [7], the segmentation algorithm based on the Intelligent Algorithm in Cai et al. [8], the Lévy flight improved multithreshold segmentation algorithm in Jia et al. [9], and the target recognition algorithm of deep reinforcement learning in Sun et al. [10] are used as comparison algorithms.
e above-given algorithm and proposed algorithm were used to segment the three data sets, and the advantages of this proposed algorithm for image segmentation were analyzed by normalized absolute error (NAE), mean square error (MSE), and running time indexes. e results are shown in Table 1.
According to the data in Table 1, different algorithms are used to segment three data sets. e number of thresholds determined by the algorithms in Li et al. [6], Li et al. [7], and Cai et al. [8] is less than other algorithms, which seriously affects the image segmentation performance of the     algorithm. is is the reason why its NAE and MSE indexes are higher than other algorithms, and the operation efficiency of the above-given algorithms is low. e algorithms in Jia et al. [9], Sun et al. [10], and the algorithm in this paper is used to segment three data sets, the same threshold number can be determined. However, the NAE and MSE indexes of this algorithm are the lowest and can realize the efficient operation of image segmentation. e reason is that this algorithm can quickly and accurately learn the characteristic information of microbial image through the improved image segmentation model of deep reinforcement learning, and the loss of microbial image information is low. e experimental results show that the microbial image segmentation ability of this algorithm is outstanding and has significant operational advantages.

Conclusions
To address the problem of poor performance of multithreshold microbial image segmentation existing in traditional methods, this paper proposes a multithreshold microbial image segmentation algorithm using improved deep reinforcement learning.
e results show that the proposed algorithm can improve the accuracy of multithreshold microbial image segmentation and has application performance. It can effectively address the problems of loss of image detail information and poor image visualization caused by noise interference in microbial images. e improved deep reinforcement learning segmentation model can efficiently learn microbial features of different morphologies and shapes and has better generalization performance. However, this paper did not verify the image detail information and image visual effect during the experimental process, which led to a decrease in the persuasive power of the experimental results. erefore, this problem will be investigated in depth in the future as a way to improve the comprehensive performance of the proposed algorithm.

Data Availability
e data used to support the findings of this study are included within the article.

Conflicts of Interest
e author declares that there are no conflicts of interest.