Shadow Removal Method for Single Image Based on Instant Learning

In this paper, a simple user-aided, robust, and high-quality fast shadow removal method is proposed. Firstly, an instant learning method is used to remove the shadow of an image, guided by input from the user of two rough areas of shadow and lighting pixels from which the initial shadow mask for shadow removal is derived. Secondly, the detected shadow image is offset and corrected to eliminate the influence of ambient light by a simple input of a color sampling line indicating the rate of change of illumination of the shadow region, penumbra region, and luminance region pixels. Clustering and illumination estimation are then performed for the corrected images. Meanwhile, the shadow is removed by calculating the statistical distribution of the shadow-free regions, and the lighting is restored by estimating the illumination scaling factor. Finally, the energy function used for illumination optimization can perform smooth estimation and image enhancement of the area after shadow removal. Experiments show that our proposed algorithm can generate high-quality shadow-free images and deal with difficult scenes with various texture types and uneven shadows.


Introduction
Shadows are a common natural phenomenon and can be used to predict the shape and location of the object that produces the shadow, as well as the source and direction of light. However, shadows can cause many problems in the field of computer vision. For example, in intelligent vehicle systems, they greatly affect image segmentation and the accurate identification of vehicles, pedestrians, and road markings by vision sensors (cameras) mounted on a car. Shadow removal from images is an ill-posed mathematical problem, and it is difficult to remove shadows completely from images. As a hotspot in the field of computer vision, shadow removal is mainly divided into two sub-tasks: shadow detection and shadow removal. One of the difficulties of light restoration in the textured area of a shadow surface is the variability of diverse shadow types. (1) In particular, the following four common attributes of shadows greatly increase the difficulty of shadow removal: rich texture features, high shadow softness, changeable shadow colors, and extreme broken shadows.
In response to the above-mentioned problems and challenges in shadow removal, methods to obtain high-quality shadow-free images have been proposed. However, it is always difficult to remove shadows completely, so most algorithms have been designed for one or more specific problems to improve the accuracy of shadow-free images. Similarly, by observing images with shadows, it has been found that the pixels of objects with the same material will form a color line in the RGB color space with a change of illumination. (2) By using this characteristic of the shadow image, a simple scribble color sampling line is designed to indicate the relationship between the illumination gradient of pixels in the shadow region and those in the lighting region for image offset correction, so as to eliminate possible artifacts in the penumbra region. Then, the shadow is removed and the lighting is restored using the statistics ratio of the shadow area. In addition, we use an energy function for illumination optimization to perform smooth estimation and enhancement of the image region after removing shadows. Through standard evaluation data, it is demonstrated quantitatively and qualitatively that this method is better than some state-of-the-art methods. In this paper, the proposed shadow removal algorithm is mainly applied to the processing of a single image.

Related Work
Shadow removal methods can be divided into automatic (3)(4)(5)(6)(7)(8) and user-aided (1,(9)(10)(11)(12)(13)(14)(15) methods. The difference between them is related to the initial detection of shadows, and the removal of image shadows after detection is an automatic task. In any case, shadow detection and removal are ill-posed problems and are difficult to implement reliably.
Early methods of automatic shadow removal included the use of internal images, a popular solution to this problem. (6) The decomposition of the internal image provides shadow and reflectance information, but such methods are unreliable and may result in excessive processing. Other methods detect shadows by learning the features of shadow regions. (7,8) However, shadow detection is often limited by the scope of training data, the edge detection, and the segmentation quality of image shadows. In addition, in recent years, shadow removal has mainly relied on the deep learning method of a convolutional neural network with end-to-end training. Learning mappings from paired shadows and the ground truth image is a new approach in image shadow removal. (3)(4)(5) Qu et al. developed three sub-networks to extract features from multiple views and embedded these sub-networks in the algorithm framework to remove shadows. (3) In Ref. 4, one conditional generation antagonistic network (CGAN) was used to detect shadows and another CGAN was used to remove shadows. However, such methods require a large number of highquality, paired data sets to be trained. Thus, a shortage of high-quality data sets is a major limitation in the development of in-depth learning methods. Later, Hu et al. examined the use of unpaired shadow image data sets to perform mask-shadow generation antagonistic network (Mask-Shadow GAN) in-depth learning, which can achieve good shadow removal of visual effects. (5) Nevertheless, such antagonistic training still requires a large number of scene data sets for learning and training. Moreover, this algorithm is complex and time-consuming.
User-aided methods usually require different degrees of manual assistance and can obtain more precise shadow removal at the cost of requiring the user to detect some areas of shadow. Wu et al. asked users to confirm shadows and lighting pixels, excluding other uncertain areas. (9) Other methods required users to define the boundary of a shadow. (10,11) Shor and Lischinski only required a shadow pixel as the input, but there were limitations on other areas of the shadow surface, which were not detected in the image. (12) Arbel and Hel-Or required users to define many texture anchors. (13) In addition, the use of natural coatings for penumbra restoration will lead to penumbra artifacts in the final result of treatment. (14) Zhang et al. performed a simple user-aided method to indicate that there is a similar texture feature relationship between lighting pixels and shadow pixels. (15) However, while the user used the illumination factor to perform image extinction, the image processing results are sensitive to the user's initial setting.
Until now, most shadow removal methods (7,11,12) have visually compared some specific shadow images, and only a few studies have performed a quantitative evaluation. These methods of image shadow removal lack high-quality ground truth data for comparison. Quantitative tests were carried out in Ref. 10, but the data used to obtain the test results were not disclosed. Guo et al. first disclosed the real data set of shadowless images they collected and then tested the data. In addition, this type of image data set is very difficult to collect and will be affected by many uncontrollable changes outdoors. (8) Because these changes affect shadow removal results, quantitative testing of such data may be difficult. Therefore, Han and Cosker extended the aforementioned data set according to the diversity of scenes (including four different categories: texture, softness, brokenness, and colorfulness), and presented new division rules. (1) This partition rule will be used to evaluate multiple shadow removal algorithms for selected images in this paper.
Traditional image features and models were used for shadow removal in early studies. In recent years, machine learning methods have been used to improve the accuracy of shadow removal. Our method combines both image features and machine learning methods for fast shadow removal. With the previous literature taken into account, our algorithm has three important contributions: (1) Powerful offset correction. The effect of using tone and contrast to restore brightness may cause inconsistencies in shadow correction areas. A robust and simple user-indicated color sampling line offset correction is used to correct artifacts. (2) Simple and robust image detection based on user inputs. The shadow detection in this paper only requires two simple marks made by the user to provide samples of lighting and shadow pixels. The method in this paper is different from previous methods that require more complex user inputs (9)(10)(11) or simpler inputs, (12) which often compromised the robustness and quality of shadow removal results. (3) Efficient shadow removal. The statistical distribution of the shadow-free area is introduced, and then the shadows are removed and lighting is restored. This method does not require prior training, and can efficiently estimate lighting changes without complicated image training. Compared with some existing methods such as image models and deep learning, our method has certain advantages. These contributions are expected to be very significant for the visual recognition field. Moreover, the validity of the proposed shadow removal algorithm was demonstrated in extensive repeated experiments.

Simple Interactive Shadow Removal Algorithm
In this section, we first briefly introduce the proposed algorithm, and then expand on the technical details of each part in the following sections (see Fig. 1).
(1) Shadow detection (Sect. 3.2). The shadow image is first detected using the K-nearest neighbors (KNN) classifier, which is guided by two simple user scribbles [ Fig. 1(b)]. Next, the detection image is binarized and classified. Then, Gaussian filtering is carried out on the binarized image. In the case without the introduction of artifacts, the shadow ratio is improved by using an energy function for illumination optimization so that most regions in the shadow area correspond to homologous material regions in the luminance area. Finally, the pixel scale factor used for restoring the illumination is used to perform local smoothing on the shadow-free image to achieve a consistent texture and illumination [ Fig. 1(k)].

Shadow image model
According to Ref. 16, an image I c (x) can be expressed as the dot product of the illumination L c (x) and reflectance R c (x): (1) The attenuation factors of direct illumination ( ) S x  in the above formula can be expressed as , .
x shadow pixels S x x other pixels According to Ref. 17, the ambient illumination ( ) a c L x in Eq. (2) can be expressed as an affine function of the offset. Thus, the illumination function L c (x) can be transformed into the following affine function: Therefore, the shadow image can be simplified as follows using Eqs. (1), (2), and (5): Three important results for the shadow image in the RGB color space can be obtained from Eq. (6). Firstly, if the reflectance R c (x) is given, Eq. (6) is an affine function, and the image pixel will form a color line based on the change in direct illumination [see Fig. 2(a)]. Secondly, when the attenuation factor ( ) [ , ] S x ∈ −∞ +∞ is infinitely extended, the color sampling line will not From the shadow image model above, it can be seen that the color sampling lines provide some important clues about the illumination changes of objects of the same material. In addition, when the offset of all different color sampling lines is eliminated, the shadow ratio can be restored by changing the gradient area of the image to remove the shadow.

Shadow detection
To remove the shadows, the shadows in the image should first be detected. Since we focus on the removal of shadows, we adopt the method in Ref. 1 to detect shadows. First, we need two rough user inputs to indicate the light and shadow pixels of the image [ Fig. 1(b)]. The shadow pixels in the RGB color space in the logarithmic domain are used to train features and construct a KNN classifier (K = 3). The distance metric of the classifier is the Euclidean distance, and the classification metric is based on the principle of large numbers for closest-point tie-breaking. We choose the Gaussian kernel for spatial filtering (size 5, standard deviation 3) and apply it to the posterior probability image. Finally, a threshold of 0.5 is used to binarize the filtered image (as shown in Fig. 3) to obtain a rough shadow mask.
After obtaining the rough shadow mask, we use the gradient of the lighting factor S(x) of the sampling line perpendicular to the shadow boundary instead of dealing with time-consuming alignment samples and selected samples. (1) Similarly to in Ref. 19, we can obtain the initial penumbra area by sampling. Finally, as shown in Fig. 3, the scene is divided into the lit area (N), penumbra area (P), and umbra area (U).

Offset correction
In Refs. 20 and 21, fog lines were used for image defogging, and it was found that the pixels in the fog-free image formed color lines in the RGB space. These lines through the origin were derived from the objects within the shadow. When a haze scene appeared in an image, the color sampling line in the haze image no longer passed through the origin, i.e., the offset phenomenon occurred. Inspired by this, in Ref. 21, a line was set for each of these patches for use as a line relative to the origin of the deviation to estimate patches of reflectance. We propose a new algorithm for clearer image processing based on the nonlocal prior. Similarly, we found that the pixels of objects of the same material in the shadow image formed a color sampling line in the RGB spatial distribution with a change in illumination (Fig. 2). Therefore, an instant learning method that uses color sampling lines to indicate changes in shadow pixel illumination is proposed to perform offset correction.
First, the user manually selects an area with similar reflectance u g R and a significant illumination gradient variation g u [e.g., Fig. 4(a)]. Then, principal component analysis is used to obtain the direction of the sampling line v of g u and compute the mean value p c of the area with the illumination gradient variation. The area of the sampling line I c (g u ) can be expressed as where t(g u ) is the independent variable of the sampling line. As described by Eq. (6), different offsets α will cause different offsets. Therefore, when we eliminate the smallest offset, the change in the image pixel values will be minimal or the error is likely to be less. Therefore, the minimum offset ( ) where represents the dot product operation and is the length operation of the vector.
As mentioned above, through this correction of the sampling lines, the direct illumination is fixed globally. Thus, the direction between different pixels is determined by the pixel reflectance. Therefore, the minimum offset ( ) can be estimated as The final offset-corrected image [Figs. 4(c) and 5(d)] can be expressed as

Image clustering and illumination estimation
In this section, all sample lines intersect at the origin after correction. The corrected image ( ) where d(x) is the distance ( )   spherical coordinates. This implies that our algorithm can obtain different sampling lines by clustering similar spherical coordinates. Next, uniform sphere sampling is used to construct a KD-Tree [ Fig. 4(d)] to cluster sampling lines. Different clusters represent different surface materials. For each given sampling line l(g), d(x) can be calculated using the pixel intensity as follows: where ⊗ represents per-pixel multiplication.
In addition, when it is assumed that there are no shadow pixels at the most distant sampling lines, the value of the illuminance attenuation factor is 1. Then, the maximum distance d max (x) of the sampling lines is By combining Eqs. (13) and (14), the illumination attenuation factor S(x) can be obtained as follows: From this, we can estimate the change in illumination of the image after correction (see Fig. 6).

Shadow removal and illumination optimization
According to the offset correction and illumination estimation of the shadow image, our algorithm performs a shadow removal method to ensure global uniform illumination, which has better robustness in complex scenes. The formula used for shadow removal is as follows: We remove the shadows according to the pixel illuminance ratio of each region of the more refined image. First, the umbra is removed by calculating the distribution of the umbra in the bright region: where g is the color sampling line index, E[.] is the average operator, SD [.] is the standard deviation operator, U represents the umbra area, and N represents the lighting area.
Equation (17) only calculates the illumination distribution of the umbra region and lighting region, ignoring the existence of the penumbra. Moreover, with the gradual change of lighting intensity, the penumbra mask cannot be restored in the same way as in the removal of the umbra area. Inspired by Refs. 12 and 22, we calculate the pixel value and illumination attenuation factor S(x) of the shadow-free region ( ) f g c I U , and then estimate the proportion of shadow pixels to restore the illumination of the penumbra. First, we calculate the mean umbra ratio U c k : Then, the penumbra ratio is estimated by amplifying the umbra ratio through the illumination attenuation factor: Finally, using Eq. (19), we use the scale in the penumbra P c k to slowly transform the umbra ratio U c k to 1 when there is no shadow ratio. As a result, we obtain the following image with the penumbra removed: After removing the shadow regions from the original shadow image, we run the above formula in MATLAB [ Fig. 7(b)]. The figure clearly shows that artifacts appear in the shadow of the image, and the illumination of the penumbra area is not well recovered. The obtained shadow-free image is illuminated for optimization. Since most of the pixels in the shadow region of the image correspond to the pixels of similar materials in the bright region, most of the shadow area can be removed using the above algorithm. At the same time, local smoothing is carried out for the area with changed illumination, so as to estimate the illumination in the other areas from the re-illuminated regions. Therefore, we refine the shadow ratio k c (x) by minimizing the following energy function: (23) where σ s (x) is the standard deviation of S(x), which is computed from each sampling line, λ is used to balance the hand smoothing factor [the first term in Eq.
We summarize the main framework of the shadow removal method described in the previous section in Algorithm 1. The user inputs a portion of the region that indicates the lighting characteristics of the scene and a region (sampling line) with significant lighting changes. We use simple user scribbles to obtain high-quality shadowless images.   Table 1 Shadow removal errors for test cases according to four attributes.

Quantitative assessment
In accordance with previous work, (1,12) we calculate the root mean square error (RMSE) per pixel between the image processed by our algorithm and the real shadow-free image. Then the RMSE between the real situation of the unshaded ground truth image and the original shadow image is computed. Finally, the two RMSE values are divided to quantitatively compare the effects of the algorithm proposed in this paper. The normalized calculation can clearly show the robustness of the method in this paper. Table 1 gives quantitative results. We evaluate our method by comparing shadow removal from individual images in the public data sets provided by Refs. 1 and 24. The best score is given in bold. As shown in Table 1 and Fig. 8, our approach is superior to previous methods in most cases, (1,8,15) especially for testing the complex texture and softness of shadows.

Visual comparison
A visual comparison of a typical example of using the proposed shadow removal algorithm is shown in Fig. 8. In general, our approach yields higher quality and more aesthetically pleasing shadow removal results than methods that specifically target shadow boundary restoration. However, small artifacts sometimes occur when the shadow region is highly similar to the background. Figure 9 shows some of the difficult special cases, where there is a broken shadow or shadows similar to the background. Further work is required to solve these special cases.

Original
Guo et al. (8) Zhang et al. (15) Han and Cosker (1) Ours   Fig. 9. (Color online) Special cases where the proposed method fails to remove the shadow.

Conclusion
In this paper, we propose a user-friendly shadow removal method that involves simple and powerful shadow detection based on user inputs. The strong offset correction method is adopted to realize fast shadow removal. After removing the shadow, the image is enhanced and filtered to obtain a high-quality shadowless image.
This algorithm provides some innovations in the field of shadow image processing and recognition as follows. (1) Simple user inputs and intelligent image pixel intensity sampling are used. (2) A novel robust and simple user-indicated color sampling line offset correction is used to correct artifacts. (3) By introducing the statistical distribution of shadow-free areas, changes in illumination can be estimated robustly and effectively for shadow removal and re-illumination without prior training. To evaluate the proposed method, we use the standard data sets in the field. We also visually compare the results of partial shadow removal in the evaluation data set. Through the analysis in this article, some specific situations where it is difficult to remove shadows are identified, for example, soft broken shadows and shadows with strong texture. Future work may be to combine user scribbles and machine learning methods to remove shadows in images.