Image Shadow Detection and Removal Based on Region Matching of Intelligent Computing

Shadow detection and removal play an important role in the field of computer vision and pattern recognition. Shadow will cause some loss and interference to the information of moving objects, resulting in the performance degradation of subsequent computer vision tasks such as moving object detection or image segmentation. In this paper, each image is regarded as a small sample, and then a method based on material matching of intelligent computing between image regions is proposed to detect and remove image shadows. In shadow detection, the proposed method can be directly used for detection without training and ensures the consistency of similar regions to a certain extent. In shadow removal, the proposed method can minimize the influence of shadow removal operation on other features in the shadow region. The experiments on the benchmark dataset demonstrate that the proposed approach achieves a promising performance, and its improvement is more than 6% in comparison with several advanced shadow detection methods.


Introduction
Although the shadow in the image is a kind of image information, it often interferes with object recognition, image segmentation, and other image processing work. erefore, the detection and removal of shadow have been widely concerned. e difficulty of shadow detection lies in the complexity of background and light in the image, as well as the interference of dark objects in the image. e difficulty of shadow removal is to ensure that only the shadow is removed without changing other image features in the shadow region.
Inspired by the relevant articles on shadow detection and removal through region matching and other methods [1,2], this paper proposes a shadow detection and removal method based on interarea material matching without training. is method can accurately detect the shadow region and minimize the impact of shadow removal on other features in the shadow region. e contributions of this paper are as follows. (1) In shadow detection, based on the mutual restriction between image region features and matching region, this method can detect directly without training. (2) In shadow removal, the shadow region is restored according to the region matching. e calculation time is very short. Other features in the shadow region can be kept unchanged to a great extent, and only the shadow is restored. e remainder of this paper is organized as follows. Section 2 introduces the related works of our proposed approach. In Section 3, we present the processes of shadow detection and removal. e experiments and experimental results' analysis are described in Section 4, and finally, this paper is briefly summarized in Section 5.

Shadow Detection
2.1.1. Shadow Feature. At present, shadow detection based on image features is widely studied, mainly to extract some changing and invariant features in shadow. Zhu et al. [3] proposed a variety of features that change and remain unchanged under the shadow in the gray image, including the features of distinguishing dark objects from shadow regions. Lalonde et al. [4] emphasized the local features of the image and determined the shadow boundary quickly by comparing the features of adjacent regions. Chung et al. [5] proposed a single feature threshold method for directly distinguishing shadow regions. Dong et al. [6] proposed the characteristics of brightness and change rate. Some colorrelated features were mentioned in [1,7] to distinguish shadow regions.

Machine Learning Method.
Currently, a lot of works combined the direct distinction of original features with machine learning algorithms to train classifiers to distinguish shadow regions under the premise of data-driven [1,3,4,8]. At the same time, for the models that distinguish each pixel separately, most of the works used CRF, MRF, and other similar models to maintain regional consistency [3,[9][10][11]. With the development of deep learning algorithms in recent years, deep learning models have also been applied to shadow detection [9,10,12]. Deep learning models make the original feature engineering no longer necessary, but these algorithms are often based on a large amount of data and need a long time for model training.

Shadow Removal.
User assistance shadow removal can be completed under two conditions: automatic image shadow detection and input of some image information with user assistance. Different from the earlier user assistance that requires users to clearly input various complex information such as shadow contour [13][14][15], at present, some user assistance methods only need to input a small amount of information to obtain good results [16][17][18].

Shadow Removal Algorithm.
At present, many shadow removal algorithms are based on shadow matting [1,9], that is, to decompose the image into the form of some parameters acting on the original unshaded image, and then remove the shadow by calculating these parameters. On the basis of shadow detection, shadow boundary and image reorganization with shadow can be used to remove shadow [19][20][21]. Vicente and Samaras [2] proposed a method to restore the shadow by using the adjacent region of the shadow region, but this method required the shadow region to have the same material as its adjacent region. At the same time, some recent machine learning algorithms and deep learning methods also have good performance in shadow removal [9,12,22].

The Proposed Method
e proposed method in this paper is based on image region matching and image optical features. Firstly, the Meanshift [23] algorithm is used to segment the image into a total of n regions. Each region is recorded as S i , and the center is C i . en, match the region with the closest material for each region (see Section 3.1). Shadow detection (see Section 3.2) and shadow removal (see Section 3.3) are realized according to region matching. e overall pipeline is shown in Figure 1.

Region Matching Calculation.
Since the color, saturation, and other features will change in the shadow region and the target is looking for the region with the most similar materials, the features used in region matching are the two shadow-invariant features mentioned in reference [3]: gradient feature and texture feature.

Gradient Feature.
e gradient change of the image region is hardly affected by shadow. e gradient similarity between regions with similar materials is stronger. In order to extract this feature, a histogram is calculated using the gradient value of each region in the image, and the similarity between regions is measured through the Euclidean distance of the histograms between two regions.

Texture
Feature. Image surface texture features are almost independent of shadows. Specifically, the method in reference [24] is used to extract texture features. Similarly, the similarity between regions is measured by calculating the Euclidean distance of the histograms of two regions.

Distance between Regions.
In order to ensure the local consistency, the distance between the central points of the two regions is added as the factor to judge the regional similarity. e calculation of similarity between regions is shown in the following equation: where D gradient i,j , D texture i,j , and D distance i,j , respectively, represent gradient feature similarity, texture feature similarity between regions i, j, and distance between center points of regions i, j.

Shadow Detection.
In shadow detection, on the one hand, the shadow is judged by the shadow features; on the other hand, the shadow is judged by the mutual restriction between regions.

Feature Selection.
In color images, color tone is a powerful descriptor that simplifies and dominates feature identification and extraction in visual pattern recognition applications. Shadow detection and removal require separate chrominance and luminance information. Since the image represented by the RGB color space is a mixture of chrominance and luminance information into the three components of R, G, and B, we need some other color models to handle shadow. ere are five invariant color models, HSI model, HSV model, HCV model, YIQ model, and YCbCr model. All of the models consist of independent chromaticity and luminance. We select three common color space models to detect and remove shadows.
(1) Y-channel information in YCbCr color space: convert the original image from RGB color space to YCbCr color space. According to the method mentioned in reference [25], when the value of a pixel on the Y-channel in YCbCr space is less than 60% of the average value of the Y-channel of the whole image, it can be directly considered that the pixel is in shadow. In the algorithm introduced in Section 3.2.2, when the average value of the Y-channel of a region is less than 60% of the average value of the Y-channel of the whole image, this region is considered in the shadow. Take the average value from the region S i and record the feature as Y i . (2) HSI color space information: convert the original image from RGB space to HSI space and then normalize the values on H and I channels to the [0, 1] interval to obtain H e and I e . According to the method in reference [5], the value in equation (2) is extracted for each pixel as another feature of the image. Take the average value from the region S i and record the feature as R i .
R(x, y), H e (x, y), and I e (x, y) represent pixel at position (x, y) in R, image H e , and image I e , respectively. (3) HSV color space information: convert the original image from RGB space to HSV color space. We utilize the chrominance channel H as the other feature. Take the average value from the region S i and record the feature as H i . Figure 2 shows the flowchart of the shadow detection approach. e specific implementation steps of the proposed shadow detection method are as follows.

Variable Preparation.
(1) Segment image with Meanshift algorithm and then obtain n regions; each region is remarked as S i . e center of each region is C i . (2) Compute the disparity between S i and S j and get the corresponding region with the highest similarity for a given region which is denoted as Near i . At the same time, record the information label i of whether the region i is in shadow and initialize label i to 255. e similarity of regions i and j is computed as equation (1). (3) For all R i , 1 < i < n, use K-means algorithm to calculate two centers C shadow and C lit which, respectively, represent whether it is the feature center of the shadow region. Suppose R follows a normal distribution; the standard deviations Std shadow and Std lit are calculated, which correspond to C shadow and C lit , respectively. en, for every R i , F shadow and F lit corresponding to C shadow and C lit are computed. (4) For each region S i , Refuse i represents whether S i is forbidden to be a shadow region because of other regions, and Refuse i is initialized to 0.
Steps. (1) Extract features Y i and R i and prepare relevant variables.

Shadow Removal.
Shadow removal is mainly carried out in HSV color space. e overall idea is to find the corresponding region S j for the shadow region S i to meet label j � 1 and minimize D i,j . erefore, S j is the most similar nonshaded region of S i . Consider using S j to adjust the brightness of S i , remove the shadow on the S i through the histogram matching algorithm, and minimize the impact of the operation on other features. e flowchart of shadow removal is shown in Figure 3.

e Proposed Algorithm.
e specific implementation steps of the proposed shadow removal method are as follows.
(1) Calculate the shadow detection result label and convert the image to HSV space at the same time. (2) Repeat steps (3)-(5) for each shadow region S i .
(3) For region S i , find S j with label j � 1 and minimum D i,j , and use S j to relight S i . (4) In the HSV color space, compute the histograms Hist H,j , Hist S,j and Hist V,j of S j in the three channels H, S, and V, respectively.. (5) Take Hist H,j , Hist S,j , Hist V,j as the template of histogram matching and adjust the three features of S i , so that the feature distribution is close to the template.

Experiments
Experiments are conducted on the shadow dataset SBU [26]. SBU dataset is a large shadow detection dataset, which contains 4727 images, and each image has a ground truth based on pixel markers. In order to quantitatively evaluate the effectiveness of the proposed method and comparative methods, the commonly used indexes such as recall, specificity, and balanced error rate (BER) are used as the evaluation indexes, which are defined as follows: where TP indicates that the real shadow is correctly classified, FN indicates that the real shadow is incorrectly classified as non-shaded, TN indicates that the non-shaded pixels are correctly classified, and FP indicates that the nonshaded pixels are incorrectly classified as shadow pixels. e higher the recall is, the more the shadow pixels are found. e higher the specificity is, the more the non-shaded pixels are recalled. Recall and specificity are considered a pair of contradictions. e comprehensive performance of the algorithm is tested by BER. e smaller the BER, the better the shadow detection performance. Figure 4. e blue line in the figure connects the center of each pair of matching regions. It can be seen that when the image segmentation is more accurate, the nearest region matching is also more accurate.  Computational Intelligence and Neuroscience 5

Shadow Detection.
e parameter setting of shadow detection is mainly in steps (4) and (6) of Section 3.2.2.
In step (4), when equation (4) is satisfied between S i and S j , it is considered that the label properties of R i and R j are opposite, and set Refuse j � 1.
where R i and R j are the features of shadow detection in the region S i and S j . e value of R i ado R j can be computed by equation (2). C shadow is the feature center of the shadow areas, and Std shadow is the standard deviation corresponding to the C shadow .
Step (3) of Section 3.2.2 describes the details. In step (6), when equation (5) is satisfied between S i and S j , it is considered that the label properties of S i and S j are the same, and set label j � 0.
where H i , Y i , R i , H j , Y j , and R j are the shadow detection features of regions S i and S j , respectively.
3 and 2.5 are empirical constants. Tuning of the parameters is outside the scope of the current work. e performance of the proposed shadow detection method is evaluated by comparing it with two commonly used methods: Unary-Pairwise [27] and Stacked-CNN [26].     Computational Intelligence and Neuroscience e Unary-Pairwise method is one of the best statistical methods to detect shadows from a single image. e Stacked-CNN method is a shadow detection method based on the deep learning framework using shadow prior map. Figure 5 shows the shadow detection results of some images in the SBU dataset.

Qualitative Analysis.
It can be seen from the third line of Figure 5 that the Unary-Pairwise method has good shadow detection results in some images, but almost no shadow is detected in columns 3 and 4. e fourth line of Figure 5 shows the shadow detection results of the Stacked-CNN method, which can detect all correct shadows, but the visual effects of some results are blurred.
As can be seen from the last line of Figure 5, the proposed method has relatively small error in image shadow detection, but the detection is basically correct. e consistency of shadow regions is ensured by region matching. It should be noted that in the images in column 1, the black base of the pole is detected as a shadow, which is caused by the similar color features between the black base and the shadow. Objects with color features similar to shadow will reduce the precision of shadow detection. Table 1 shows the results of quantitative analysis on the SBU dataset. e proposed method can improve the detection rate of shadow and nonshaded regions. Obviously, recall and specificity are considered a pair of contradictions. Considering the performance of shadow detection, the BER metric is used, and the proposed method effectively reduces the BER.

Quantitative Analysis.
e Stacked-CNN method can detect most shadow pixels in the dataset, but it also causes pixels in non-shaded areas to be insufficiently recalled. Some non-shaded pixels are wrongly classified as shadow pixels, resulting in blurred shadow detection images, which is consistent with the results of qualitative analysis. e Unary-Pairwise method has the lowest recall of shadow pixels on a large-scale shadow detection dataset. is is because of the false pairing between regions due to the loose constraints. Although the recall of the proposed method is lower than that of the Stacked-CNN method, the precision is higher. On average with BER, the improvement made by the proposed method is more than 6% which is impressive.

Computational Intelligence and Neuroscience
In Table 2, we show the execution time of testing phases of the proposed approach. Our method does not need training, and it works faster than the statistical method and one order of magnitude faster than the deep learning method.

Shadow Removal.
e parameters of shadow removal are set in step (7) of Section 3.3.2, in which Gaussian blur is applied to the points whose distance from the edge pixel is less than or equal to 2, and the parameters of Gaussian blur are set to hsize � 15, σ � 15. e second line of Figure 6 shows the shadow removal process of some images in SBU dataset, in which the red lines connect and match the corresponding shadow region S i and non-shaded region S j . Use S j to relight S i through the algorithm in Section 3.3.2.
e third line of Figure 6 shows the shadow removal results of the images. It can be seen that the regions after removing shadow through region matching maintain the original texture features, but the brightness changes. e image after removing shadow has an obvious boundary around the original shadow region, but the edge effect is significantly weakened after Gaussian smoothing. rough experiments, it is found that not all the shadows of the image can be removed correctly. Two failed shadow removal scenarios are shown in Figure 7. In the first image, the shadow occupies most of the image, and the shadow is evenly distributed in the image, which will lead to the uniform distribution of image feature histogram, and the non-shaded region is not enough to remove the shadow region. In the second image, the shadows of multiple objects are complex, and the fire hydrant is almost completely in shadow, resulting in the wrong shadow removal matching, so the shadows cannot be removed correctly.
In this work, the experiments are run on a PC with a 3.6 GHz CPU and 8G RAM in the Matlab 2018b environment under Windows 10.

Conclusion
Experiments show that the shadow detection and removal algorithm based on region matching is effective. In the aspect of shadow detection, the consistency of images can be guaranteed to a certain extent through the mutual restriction of regions with the same material; in the aspect of shadow removal, the method of brightening the shadow by the region with similar material can minimize the impact of shadow removal on the non-shaded features in the shadow region.
e limitation of the method proposed in this paper is that it is difficult to ensure the accuracy of region matching for complex images. erefore, in shadow detection and shadow removal, it will bring wrong constraints due to wrong matching or restore the region to the wrong material. At the same time, this algorithm is based on Meanshift image segmentation technology, which is difficult to segment accurately in complex images. e further improvement mainly lies in the calculation of the material matching region. First, using more and more datasets, the matching region is given by training classifiers and other methods. Second, adding more shadow-invariant features to improve the accuracy of region matching in complex images will be taken into account.

Data Availability
e data used to support the findings of this study are included within the article.