System for Recommending Facial Skincare Products

1Department of Computer Science and Information Engineering, National Ilan University, Yilan 260, Taiwan 2Department of Engineering Science, National Cheng-Kung University, Tainan 701, Taiwan 3Department of Informatics, Universitas Sebelas Maret (UNS), Jawa Tengah 57126, Indonesia 4Department of Electronic Engineering, Chung Yuan Christian University, Taoyuan 320, Taiwan 5School of Mathematics and Information Engineering, Longyan University, Longyan 364012, China


Introduction
A clean face gives a good first impression on others, so appearance is important. In the past, defects were treated using advice from family members or doctors, but facial care products that are recommended by family members may not be appropriate for specific skin types. Using inappropriate products can exacerbate facial problems. Most of the solutions that are recommended by dermatologists involve taking or applying medicine, but daily maintenance is the most important aspect of any treatment, taking into consideration humidity, acne, and whitening.
Skincare products are mostly chosen using visual or tactile information, and the design of skincare products focuses on these channels. In the age of digital multimedia technology, information is abundant, but its authenticity must be evaluated. Excessive marketing and packaging can make it difficult to distinguish correct and incorrect information, so consumers are unsure about which products are suitable. There has recently been an increase in the number of studies related to this field. (1)(2)(3)(4)(5)(6) Alamdari et al. (1) used a K-means cluster to search for acne. This method achieves a binary threshold using an adaptive method, but it is vulnerable to changes in the brightness of an image, so it is not a reliable basis for a decision. Another study (2) identified the location of acne and provided recommendations to consumers. However, the experimental results in Ref. 3 showed that, although the quality of skin can be identified using a fixed instrument, the brightness cannot be changed, so only specific instruments can be used. In this study, we use the framework of Ref. 3 to identify the skin type. Gupta et al. (4) referred to Ref. 3 to design a new algorithm that relies on the fact that oily skin has a higher light intensity than dry skin. A scale of oiliness was produced by labeling images of the weighted average of the reflected light intensity in the forehead area. As examples of deep learning (DL) methods, Shen et al. (5) designed a binary classifier of skin and nonskin to detect areas of the skin and a seven-point classifier to classify facial acne vulgaris and healthy skin using a convolutional neural network (CNN). Zhao et al. (6) developed a DL model that can assess acne severity as accurately as dermatologists from selfie images. In this study, we employed a supervised method and recruited 11 internal dermatologists to label 4700 selfie images in five categories. Although this method addressed some of the limitations of the instrument in Refs. 3 and 7 and allowed the remote examination of the skin, it did not provide a solution based on the results of the skin examination.
The proposed system analyzes the severity of facial skin and acne to recommend care products for consumers. Computer vision technology allows consumers to instantly obtain an objective and consistent list of recommended facial care products. To allow inexperienced users to find a suitable care product, an image is captured with a camera, which is marked and displayed in a window so that the consumer can verify that the image is of the correct position. The skin type is identified and acne is detected for both cheeks. The system determines whether the skin is oily, normal, or dry, and then detects the number of acne marks. By pushing a bottom on the screen, any misjudgment that is caused by the presence of multiple faces in the picture can be removed to increase the accuracy of the system. The results for the skin condition and the detection of acne are used to recommend skincare products, so that consumers can identify appropriate skincare products without consultation. The remainder of this paper is organized as follows. We describe the system framework in detail in Sect. 2, give experimental results in Sect. 3, and draw conclusions in Sect. 4.

System Framework
The system framework is shown in Fig. 1. The original facial skin image is input, and the region of interest (ROI) is identified as the cheeks. The ROI image is used to identify the skin type and detect acne. The system matches with the dataset for the skin type and detects acne marks in the ROI; then, the database recommends an appropriate skincare product for the consumer. (8)

Image capture, ROI, and skin type identification
The image in the camera is imported into the program, and then Haar-like face detection is used to localize the face in the image. Facemark (9) is then used to identify 68 features of the face, as shown in Fig. 2. The ROI, which is the cheeks, is shown in Fig. 3. The ROI images are imported to the system for detecting the skin type and acne. The input image is in RGB color: red (R), green (G), and blue (B) are added in different proportions to produce a color or RGB image. In this step, the RGB image is initially converted to a gray image and a discrete wavelet transform (DWT) is used to calculate the first feature to determine whether the skin is oily. If the skin is found not to be oily, the blue channel of the input facial image is used to calculate the texture feature as the second feature to determine whether the skin is normal or dry.

DWT
When the ROI image is input, the RGB image is transformed to a gray image, and a twolevel Haar DWT (10) is performed on the gray image, as shown in Fig. 4. A Haar DWT is used because it is orthogonal and double-orthogonal simultaneously. The spatial resolution of the resulting images is low and the quality is high, (11) so it is easily implemented in an embedded system.

Calculation of first feature
The first feature, used to determine whether the skin is oily, was proposed in Ref. 3. The values for the four intervals (3) are combined to give four parameters in the wavelet domain. The coefficients in the two-level decomposition low-low (LL) band (12)  The camera used in this study cannot account for the brightness of the environment and a flash cannot be used, so a button is pushed to capture the facial image such that the face is completely captured before a recommendation is made. It was shown in Ref. 3 that oily and non-oily skin types are different in terms of the first feature accumulation, so here, we classify skin as oily or non-oily using a support vector machine (SVM). This allows classification using known features and the model can be trained using fewer data points. The data are labeled using the opinion of an expert to determine the skin quality in the images taken in different environments. Training occurs after labeling.

Calculation of second feature
The four features of the second feature are entropy, contrast, inverse difference moment (IDM), and deviation. For an N × N pixel image I, Eq. (1) is used to derive the co-occurrence matrix between I(k, l) and I(m, n), where i and j denote gray values, # is the number of elements, and d is distance. If two images have a similar texture, the distribution of the co-occurrence information for each image is similar. The co-occurrence information is normalized using the following probability density function (PDF): where S denotes the total number of elements in the matrix. In contrast, IDM and entropy are respectively defined in Eqs. (3)-(5), and Eg is the maximum gray scale value minus the minimum gray scale value plus one. The same skin sample uses expert labeling for classification as normal or dry.

Acne detection
To account for the effect of light, the RGB image is converted to a grayscale image and then the result is normalized. A V channel image is then extracted from hue, saturation, and value (HSV) images, and the grayscale normalized image is subtracted to obtain the acne distribution image. HSV images intuitively express the shade, hue, and vividness of colors, so colors can be easily compared. When the acne distribution image has been subtracted, the resultant image is binarized, the label is used to remove the noise, and the area of the acne is calculated. This defines the severity of acne.
The converted grayscale images are easier to process than the ROI images. (10) The normalization step scales the pixels of grayscale images to adjust the brightness of the image, as shown in Fig. 5. The light source affects the decision of the image system. (13) In this study, we account for unstable factors. The brightness must be determined to ensure that the image has the correct exposure. The V channel in the HSV image is used to obtain the brightness information. Figure 6 shows images that have been converted to HSV and V images.
The brightness image is subtracted from the normalized grayscale image, and the resulting image is binarized for labeling. In the binary image, the area with acne is white and the area with no acne is black, as shown in Fig. 7. The image is then labeled by tagging the white blocks and calculating their total area. The binary image contains noise that must be denoised. The system defines one-thousandth of the area as noise and removes it. The resulting image is then divided into areas of mild, moderate, and serious acne.
The type of acne is then detected. The system uses a database built with MySQL to store information about skincare products. The attribute ID, product name, price, whether it is used to treat acne, suitable skin type, and detection code are stored in the database. Using the database, the system recommends a skincare product to the consumer that is suitable for the skin type and the severity of acne that are detected.

Experimental Results
We use Microsoft Visual Studio 2015 as the main development tool in a Windows environment and C ++ as the development program with Open Source Computer Vision Library (OpenCV). The user interface (UI) uses C# as the main development program with a related MySQL database management system, as shown in Fig. 8. The UI includes an open camera button, a recommendation button, a clear button, and an exit button. The right side of the window shows the image captured by the camera. The input image uses Facemark (9) to label 68 features on the screen, and the consumer confirms that the system has correctly captured the face.   The acne on the skin and the condition of the skin are factors in the choice of different skincare products. Therefore, we focus on the acne detection and skin type identification. According to the classification results, one of the nine different skin conditions (dry skin with serious acne, dry skin with moderate acne, dry skin with mild acne, normal skin with serious acne, normal skin with moderate acne, normal skin with mild acne, oily skin with serious acne, oily skin with moderate acne, and oily skin with mild acne) is diagnosed. The system was tested on 15 people between 18 and 23 years old. The participants manually evaluated the condition of the face before testing and the test results were shown. The system has an 80% satisfaction rate, as shown in Table 1. The detailed analysis results are given in Table 2.

Conclusions
We developed a system to analyze facial images using computer vision technology to recommend skincare products. In contrast to other proposed systems, our system combines facial detection and skin type analysis. In the future, we plan to incorporate a chat robot into the system and use big data. The chat robot will allow the consumer to communicate easily when using the system and can collect basic information, such as gender and age. This information will produce a more targeted system that is more user-friendly through the use of conversation. The use of big data will involve the collection of comments about skincare products in the network through keywords to increase the comprehensiveness of the database.