Advanced Image Enhancement Method for Distant Vessels and Structures in Capsule Endoscopy

This paper proposes an advanced method for contrast enhancement of capsule endoscopic images, with the main objective to obtain sufficient information about the vessels and structures in more distant (or darker) parts of capsule endoscopic images. The proposed method (PM) combines two algorithms for the enhancement of darker and brighter areas of capsule endoscopic images, respectively. The half-unit weighted-bilinear algorithm (HWB) proposed in our previous work is used to enhance darker areas according to the darker map content of its HSV's component V. Enhancement of brighter areas is achieved thanks to the novel threshold weighted-bilinear algorithm (TWB) developed to avoid overexposure and enlargement of specular highlight spots while preserving the hue, in such areas. The TWB performs enhancement operations following a gradual increment of the brightness of the brighter map content of its HSV's component V. In other words, the TWB decreases its averaged weights as the intensity content of the component V increases. Extensive experimental demonstrations were conducted, and, based on evaluation of the reference and PM enhanced images, a gastroenterologist (Ø.H.) concluded that the PM enhanced images were the best ones based on the information about the vessels, contrast in the images, and the view or visibility of the structures in more distant parts of the capsule endoscopy images.


Introduction
In the effort to obtain more information about the vessels and structures, particularly, in the darker or distant parts of capsule endoscopic images, the image contrast enhancement is the way to go. There exist several categories and subcategories of contrast enhancement methods in the literature, for example, the Histogram Equalization (HE), Adaptive HE (AHE), and Contrast-Limited AHE (CLAHE) whose details are provided in [1,2] as well as M1 a method proposed in [3], and M2 represents an enhancement method proposed in [4], developed to deal generally with the poor contrast problems in color and grayscale images, which remain nonexhaustive and image dependent in their performance [5][6][7][8][9][10][11][12]. Today, capsule endoscopy is among the newest research and application areas in medicine that caught interest of many researchers because of the advantages that capsule endoscopy provides over the traditional endoscopy in terms of comforting patients while exploring the entire gastrointestinal (GI) tract [3,[13][14][15][16]. To clinically benefit from images obtained thanks to the capsule endoscope (CE), it is important to develop an advanced method that would deal carefully with the poor contrast problem caused generally by poor visibility conditions of the GI tract [17]. In this regard, a novel method using exclusively the bilinear interpolation algorithm has been proposed in [4] to deal with (1) the creation of artefacts leading to unnatural colors of the Histogram Equalization (HE) based methods without the need for converting Red-Blue-Green (RGB) to another color space [9,10] and (2) the disadvantages of the generalized overexposure problem of the method proposed previously in [3]. Although experimental demonstrations showed that the half-unit weighted-bilinear algorithm (HWB), proposed in [4], made considerable improvements over the method proposed in [3], (improvements that can also be seen/noticed in this paper  [21]. This figure has also been used in [4]. could only rate highly 70% of enhanced images presented in experimental demonstrations of [4]. Such a rating was due to overenhancement of the neighborhoods of the brightest areas (i.e., specular areas) by the HWB and bad intensity transitions between darker and brighter areas (as can be seen in this paper's Figures 5(b)-11(b)). Now, since the CE has the main light source, implanted in it, composed of a group of many Light Emitting Diodes, when such a light falls onto a GI surface tissue, some of the beams are reflected back straightaway, specular reflection, while the rest of the beams penetrate it before being reflected (diffuse reflection), thus forming specular highlights on the capsule endoscopic images [18]. Although the sporadic presence of specular areas remains unavoidable, it is not a major one in this direction [19,20], except enlarging them via overenhancement of their neighborhoods. The advanced method, taking into account those possible enlargements and jagged transition of intensities between darker and brighter areas issues, has been proposed in this paper. The disadvantage of the proposed method (PM) is that it does not suppress cognitively (or underenhance appropriately) specular highlight spots. It does not always perform very well or does not achieve the best scores with small sized images. Figure 1 shows the CE device and human GI tract. This paper is organized as follows: the second part gives a brief introduction to the state-of-the-art key algorithm, dealing with darker areas, proposed in [4]. Part three gives the proposed method, as well as its summary. Experimental demonstrations, results, and evaluations by a gastroenterologist are provided in the fourth part. The conclusion is given in the fifth part.

State of the Art
Half-unit weighted-bilinear algorithm (HWB) is a novel RGB image enhancement strategy developed in [4], to significantly remove all Histogram Equalization (HE) based artefacts and disadvantages [9,10], rather than using complex enhancement strategies [23,24]. A key point on which the HWB differs from the conventional bilinear algorithm (CWB) [25][26][27][28] is that it uses a half-unit weighting strategy to calculate new pixel values for each overlapping four-pixel group in the destination matrix or image [4]. The mathematical expression on which the CWB is based is given in (1). ( , ), ( , + 1), ( + 1, ), and ( + 1, + 1) are pixel ( ) locations, on the pixel grid, as shown in Figure 2(b) ( is the number of four nearest neighbors). The mathematical expression for the HWB algorithm is given in (2). Equation (2) is the result of applying a constant half-unit to weights CW in (1): where HW = 2 is the weighting function of the HWB. It is important to note that (2) is the main function used to calculate the pixel values in the preliminary enhancement stage, as explained in [4].

Proposed Method
The threshold weighted-bilinear algorithm (TWB) is a novel algorithm that operates from an empirical threshold value and the Hue-Saturation-Value (HSV) component is developed and proposed, in this paper, to achieve the overall proposed method (PM). By developing TWB, the objective is to back the HWB algorithm and achieve an overall enhancement scheme that leads to a better visibility of distant capsule endoscopic images vessels and structures needed by gastroenterologist in their clinical diagnosis than what was r, c P1 P2 r, c + 1 Δc Δr 1 − Δc previously achieved in [4]. The mathematical expression for the TWB's weighting function (TW) is given as follows: is the denominator of the weighting function TW, whose mathematical expression is given in as follows: where is the TW's denominator initial value whose optimal range, leading to the fact that a better linkage process between the darker and brighter areas (see Figures 3(b)-3(d)) has been experimentally located between 1.45 and 1.50; and is equal to the difference between consecutive denominators. Note that the experimental value for was found to be equal to or less than 0.025 so that the border between darker and brighter areas can become invisible. Here, is the number of weighting steps between the component empirical threshold and maximum values, as shown in (5). This number can be obtained by dividing the component maximum value, , with the component empirical threshold value (a default value for is equal to 0.4).

= .
( Note that is equal to 1 in the component . The mathematical expression for the TWB is given by the following: The PM's mathematical expression is given by (7). The PM's equation is a combination of the (2) and (6). The functioning of this combination is enabled by a set condition where it has to be verified whether the component values are greater or less than the component empirical threshold value . If this condition is true (Yes), the matrix output of (2) is added together with the reference matrix. If the condition is false (No), the matrix output of (6) is added together with the reference matrix. The final matrix mapping, of all output matrices, constitutes the PM enhanced image. The summary of the PM is given in Figure 4. As can be seen/noticed the simplicity in the design of the PM is also another advantage computationally, although the processing time is not the main concern in this work.
where is the reference RGB image (often seen or treated as a poor contrast image) and is any value of the component, ranging between zero and .

Experiments and Results
Experiments using standard image quality metrics, and the evaluation on better visibility of the vessels and structures in the PM enhanced images by a gastroenterologist, are presented here. The PM software has been implemented in MATLAB-R2017a. Image quality metrics used are structural similarity index (SSIM) and feature similarity index (FSIM). The reason is that, for capsule endoscopic applications where the pursuit for diagnostic quality is the main concern, metrics taking into account the image diagnostic structures and features (with reference to the reference image) are more appropriate than those who do not. It is important to note that there exist well-documented and widely available scientific works on such metrics in the literature [29][30][31]. Therefore, explanations, mathematical formulas, demonstrations, and so forth of such metrics are not included in this paper. Also, metrics widely used in statistic of visual representation, such as contrast and intensity enhancement metrics, have been used to measure the contrast and intensity distortion in each RGB channel of the CE images. However, such methods normally apply on grayscale images. Therefore, their results presented were obtained based on processing each RGB channel separately. Blind/Referenceless Image Spatial QUality Evaluator (BRISQUE), which operates in the spatial domain and according to [32] was the best performing metric for capsule images correlating with diagnostic quality, has been used to quantify possible losses of "naturalness" in the enhanced images. Capsule endoscopic images, downloaded from the capsule endoscopy database for medical decision support, have been used as test images [22].
In Tables 1 and 2, M1 represents the method proposed in [3], M2 represents the method proposed in [4], and PM represents the method proposed in this paper.
As can be seen, in images shown in Figures 5-11 11(b), and 11(d). In the ideal world, images obtained from capsule endoscopy would be perfect for a gastroenterologist in the sense of sharpness of image details, brilliant image colors, perfect image contrast, and no artefacts. So far, there exist no such perfect capsule endoscopic images. However, based on evaluation of the reference, proposed, and even upscaled images, an additional experiment whose details are not included in this paper but which was conducted using Lanczos interpolation for three times (3x) upscaling purposes), a gastroenterologist (ØH) concludes that the PM enhanced images were the best ones based on the information about the vessels, contrast in the images, and the view of the structures in the most distant parts of the images. In some of the series the PM enhanced images were brighter, and, hence, it was easier to see the structures also in the distant parts of the images. In some of the series, the upscale images were too blurred to give more information than the PM enhanced images, but most of the upscale images gave more information than the normal sized ones. As mentioned earlier, details about Histogram Equalization (HE), Adaptive HE (AHE), and Contrast-Limited AHE (CLAHE) are provided in [1,2]. As can be seen in Tables 1 and 2, the PM produced the highest SSIM and FSIMc values. The SSIM and FSIMc value closest or equal to one means generally the best quality because, in that case, the similarity (to the reference image diagnostic quality structures and features) is almost maximum or maximum. Unlike in the M1 and M2 cases, the PM produced brighter images, but not too bright, and preserved the hue. It is important to note that Histogram Equalization (HE), Adaptive HE (AHE), and Contrast-Limited AHE (CLAHE) terribly change the hue of reference images, in RGB color space, as demonstrated in [3,9]. Therefore, the capsule endoscopic images enhanced by such methods have not been included in this part. A part from widely known standard image quality metrics, in this paper, metrics widely used in statistic of visual representation have also been used, as stated earlier, to measure the contrast and intensity distortion in each RGB channel of the CE images [16]. It is important to remember that such methods normally are applied on grayscale images. Therefore, results presented in Tables 3 and 4 were obtained based on processing each RGB channel separately. It is also important to note that processing the intensities of each RGB channel is not the same as processing the intensities of a grayscale image. However, in the effort to find out how much enhancement methods affected each channel intensity, each channel was processed separately using those metrics. The results obtained proved that diagnostic quality cannot be correctly assessed based on the highest values of contrast or intensity enhancement in each channel's intensities, between the reference and enhanced images [16]. For example, in Table 3, the method proposed in [3] gave the highest values in terms of contrast enhancement (in all channels, for every image almost) but the corresponding images, shown in Figures 5(a)-11(a), showed that images produced by M1 are too bright and some image details are not visible compared     to M2 and PM. Another example, in Table 4, showed that the HE gave the highest values in terms of intensity enhancement (in G and B channels, for every image) but, as shown in [3], the output of the HE method does not give any usefully diagnostic information because the reference hue is damaged terribly. However, if we analyze the statistics provided by these metrics in another way, for example, assuming that positive statistics means better quality, in this way the enhanced images and statistics by the PM have proven to be the most positively correlating to the reference image without overexposing or overenhancing image details. Figure 12 presents the BRISQUE scores obtained. For image category whose size is equal to 288 × 288 × 3 (i.e., image 5, image 6, and image 7), the PM achieved generally the best scores. For image category whose size is equal to 188 × 188 × 3 (i.e., image 1, image 2, image 3, and image 4), the PM achieved generally the second best scores. This suggests that the PM works better with larger images than with smaller images. However, the PM achieved the best scores compared to all enhancement methods mentioned.

Conclusion
Advanced enhancement method for vessels and structures in capsule endoscopic images has been proposed in this paper. The proposed method used mainly two HWB and TWB algorithms to deal with darker and brighter areas, respectively. It also used additional strategies to create a smooth intensity transition between such areas. The overall enhancement method achieved produced enhanced images with a moderate increase in brightness in darker/distant areas that could preserve the hue of the reference images (without enlarging the specular highlight spots or overenhancing their neighborhoods). Compared to the previous works, more details, especially in brighter areas, could still be seen after the PM enhancement operations because the PM could avoid overenhancing the neighborhood of the brighter areas. In this way, it was easier to see more details about the vessels and structures, for example, in the pursuit of precancerous or polypous tissues or even inflammations, in the PM enhanced images than in the reference images and in both M1 and M2 enhanced images. In the evaluation conducted together with a gastroenterologist (ØH), a gastroenterologist concluded that PM enhanced images were the best ones based on the information about the vessels, the contrast in the images, and the view of the structures in the most distant parts of capsule endoscopic images used. The usefulness of the PM enhanced images was also supported by statistics obtained using the SSIM and FSIMc metrics. Furthermore, in the effort to find out how much the PM affected each channel intensity, each channel was processed separately using contrast and intensity enhancement metrics. The first analysis demonstrated that the pursued diagnostic quality could not be correctly assessed based on the highest values of contrast or intensity enhancement in each channel's intensities, between the reference and enhanced images. The second analysis suggested that the statistics provided by these metrics, in another way, could mean better quality with reference to being closer to zero in a positive direction. And, enhanced images and statistics by the PM proved to be the most positively correlating to the reference image without overexposing or overenhancing brighter areas neighborhoods and their neighborhoods and image textural details compared to M1 and M2 methods' outputs. Future work can be dedicated to the development of an innovative enhancement method enabling the desired gastroenterological sharpness of capsule endoscopic image details, color brilliantness, and artefact-free and that can lead to an underenhancement of specular highlights spots since such spots hide the details in part of the image. On top of that since the PM gave better BRISQUE scores in 2 types of test images of the same size out of 3 and in only 1 type of test images of the same size out of 4 (smaller than the previous category size), future effort will be dedicated to an "intelligent" or "cognitive" method that would lead to the best visibility desired by gastroenterologists and scores in all types of test images sizes (in terms of BRISQUE, FSIM, etc.).

Conflicts of Interest
The authors confirm that the mentioned received funding in  and that there are no any other possible conflicts of interest regarding the publication of this paper.