Recognition of Vehicle License Plate Based on Hopfield Artificial Neural Network

1College of Mechanical and Control Engineering, Guilin University of Technology, Guangxi 541004, China 2Department of Information Management, Yu Da University of Science and Technology, Miaoli County 36143, Taiwan 3College of Mechanical and Electrical Engineering, Guangdong University of Petrochemical Technology, Maoming Guangdong 525000, China 4Jiangsu Ocean University, Jiangsu Province 222005, China


Introduction
Artificial intelligence (AI) neural network is an emerging technology as it has great potential and adaptability in various fields of industry. (1) One of its applications is image processing that is used in daily lives. The license plate recognition of vehicles adopts image processing technology. (2) For license plate recognition, several steps in image processing, such as grayscaling, binarization, and edge detection, are required.
Grayscaling converts an original image into a gray-scaled image for a more accurate and faster recognition of characters on license plates than the original method. (3) There are several methods of grayscaling: maximum value method, average method, and weighted average method. (4) Among these methods, the weighted average method is regarded as the most appropriate. Image binarization is part of image segmentation. (5) As the segmentation of images by a threshold value has advantages over edge segmentation in terms of calculation and efficiency, threshold segmentation is used in general. There are several methods of threshold segmentation, such as the iterative method, maximum inter-class error method, and local segmentation method. (6) Among them, the maximum inter-class error method is the most practical as it adaptively determines threshold values.
Edge detection reflects the main feature and characteristics of an image. (7) Generally, significant changes in brightness are important in processing an image (8) and thus are used to detect the edge of the image to extract its subsequent features. Edge detection requires appropriate algorithms and operators, such as Sobel, Robert, Prewitt, Log, Laplacian, and Canny operators.
For processing images effectively through the above-mentioned method, an artificial neural network (ANN) is widely used. Among recently developed ANNs, dynamic feedback neural networks are mostly adopted, especially the Hopfield neural network as it is a nonlinear dynamic feedback neural network and is known as the most effective so far in processing images. It has stable performance and reaches a stable state regardless of the initial state by adjusting weights to maintain its stability. The Hopfield neural network enhances the accuracy of recognizing license plates regardless of road conditions and weather. It also processes images taken even by complementary metal-oxide-semiconductor (CMOS) camera modules. A CMOS sensor converts photons to electrons for digital processing and is widely used in digital imaging devices. It is mainly used for low-price devices such as cameras for closed-circuit TVs and digital video cameras. For higher quality images, charge-coupled devices (CCD) are used instead.
In this study, we aim to provide an effective method of recognizing license plates of vehicles even for low-resolution images from CMOS cameras. By developing an appropriate algorithm, we suggest a quick and easy method of recognizing license plates effectively.

Methodology
The main idea of the proposed method is to distinguish target parts from their background in an image according to a threshold value. For example, after a certain value for a gray pixel is assigned as a threshold, the pixel with the largest difference from the gray pixel is defined as the threshold again. Then, in the cluster that contains the pixel, the inter-cluster difference between the target part and its background is calculated. Repeating the process makes differences in values large and errors small. By doing this, the inter-cluster difference in grayscale is calculated to determine the threshold values with the largest difference.

Grayscaling images
We used the weighted average method in grayscaling. The method assigns the values of red (R), green (G), and black (B) colors in a gray image to the R, G, and B colors of an original image before processing. This method is flexible as it adjusts the output of the gray image by using the weights of colors as follows: where φ R , φ G , and φ B are the relative weights of R, G, and B, respectively. Adjusting the weights outputs the transformed gray image. Since the eyes are most sensitive to the green color and least sensitive to blue, the weight of G is the largest, whereas that of B is the smallest. An example of an output image processed by the maximum inter-class error method is shown in Fig. 1.

Image binarization
Image binarization segments an image by a threshold for fast calculation and high efficiency. Among several methods, we used the maximum inter-class error method for image binarization. The maximum between-class error method is practical as it is adaptive to determine the threshold value. The main idea of this method is to distinguish between what is necessary and not based on the characteristics of the threshold value itself. Then, the difference is calculated between the target and the background. A large difference means that the values of the background and target in the image are large with a small error. Thus, how to calculate the difference efficiently is important in image binarization.

Image edge detection
A log operator is required to detect the edge of an image. The log operator is generated by combining Laplacian and Gaussian to suppress noise, which sometimes disturbs several edges instead.

Hopfield neural network
The Hopfield ANN is trained to reduce the noise of information in images by creating a series of digital dot matrices and creating a neural network in MATLAB. MATLAB provides functions for the Hopfield ANN. newhop() function creates a discrete neural network with the call format of net = newhop(T), where T is a matrix of R × Q (Q is a target vector with elements of −1 or 1). As it is not easy to recognize characters on weathered license plates, it is necessary to train the neural network with unchanged points in T. As the Hopfield network has one layer, its activation function needs to be satlins() function that generates the digital dot matrix with noise. Among several methods for simulating the generation of the digital matrix with noise, we use the random noise generation method. When the digital dot matrix with noise is input into the Hopfield neural network, the output of the network becomes the closest target vector to the matrix to realize the associative memory function. Then, the test results are analyzed, compared, and used to verify the Hopfield network, as shown in Fig. 2.
The network state and operating mode of the discrete Hopfield network are defined as follows. Network state where f(•) is the transition function and net j is the net input of the neuron j.
The symbolic linear function is defined as The symmetric saturated linear function is defined as

Operating mode
According to the network state, the first weight adjustment is determined as In the process, the last weight adjustment determines the weight adjustment that determines the next weight adjustment. When the adjustment no longer changes the weight, the system becomes stable. The next weight adjustment is carried out using the following equations:

Vertical projection method
We use the vertical projection method because it is known to be appropriate for the character segmentation of the license plate and for images with a left-right layout. The purpose of vertical projection is to obtain the left and right edge positions of the license plate image. The vertical projection counts the columns in the vertical direction of an image and the number of white pixels in each column. By comparing with a predetermined threshold value, T, a row larger than the threshold value is detected and defined as a valid row.

Results
The first character of the license plate in China is a Chinese character for each province. Thus, the database for network training is different from training alphabets and numbers. The second character in the alphabet of the license plate represents a city in a province. Then, numbers and letters are mixed into five characters.

License plate location
A license plate is located by the following steps: The first step is to grayscale an image, followed by edge detection. Then, the image is divided into several regions by image segmentation. Finally, the license plate is located according to its characteristics. Next, image processing is performed to obtain the edge image using the Hopfield neural network for showing the parts for the license plate in the image. Figure 3 shows the image with the first edge detection.
The figure still needs to fill small blanks and sharpen boundaries in the edges of the parts in the image by open operation, which allows for the extraction of the image of the license plate. The operation for the corrosion and then expansion of an image is called opening operation. It eliminates small objects, separates objects at delicate places, and smoothens the boundaries of larger objects. Closed operation processes an image first with expansion and then with corrosion. It fills small cavities in objects, connects adjacent objects, and smoothens their boundaries. The result is shown in Fig. 3. The edges become clearer by removing the areas that are inconsistent with the license plate. After the opening operation, the image is denoised and expanded. Finally, the license plate is located in the processed image (Fig. 4).

Character segmentation of license plate
For character segmentation, the original image is transformed into a binary image that is projected vertically. After obtaining the histogram of the vertical projection of the license plate, the minimum values of the parts in the image are calculated. The minimum values are combined with the width and distance of the characters on the license plate to segment the characters of the license plate (Fig. 4). On the basis of the vertical projection, the spacing and characters on the license plate are segmented clearly for subsequent character recognition.
Character recognition is carried out by grayscaling an image first. Figure 5 shows the result of grayscaling. By using the Rgb2gray code in MATLAB and the weighted average method, the weight of the red color is adjusted to be greater than those of the green and blue colors. The processed gray image is small in size and easy to store for effective subsequent image recognition.   Then, edge detection is applied to locate the license plate. The morphological processing filters the candidate regions of the license plate on the basis of the features of the license plate. The following results show that the positioning accuracy is still relatively high. First, edge detection is carried out to locate the license plate clearly (Figs. 6 and 7).
After locating the license plate, it is necessary to segment the characters of the license plate. The segmentation method is the vertical projection method mentioned above. This method segments the characters of the license plate quickly and effectively. Figure 8 shows the result of the character segmentation of the license plate.
The result shows that the accuracy of the recognition is acceptable but not complete. In this case, tilting needs to be added as an additional step for license plate recognition. Thus, we adopt a tilt correction method. In recognizing the characters of license plates, tilted images may be taken because of camera shooting angles when vehicles turn and drive on slopes. These situations lead to tilted images of the license plate, which hinders accurate image processing. Therefore, it is necessary to correct the tilted images for license plate recognition. There are several tilt correction methods, such as Hough transform, Radon transform, least square method, and two-point method.
Radon transform is a projection method that projects the image of a tilted target within the range of different tilt angles of 0-179°, and then finds the projection angle for the best result. Then, the projection angle is recorded as the angle of inclination. The image corrected by the method is shown in Fig. 9. Although the correction result does not transform the tilted image into a perfect rectangle, the corrected image has sufficient quality for subsequent image recognition.

Discussion and Conclusions
Low accuracy and efficiency have been problems in license plate recognition. To solve these problems, we adopt the Hopfield neural network. By using the Hopfield neural network, Chinese characters and mixed letters of alphabets and numbers are recognized effectively. Chinese characters are more complicated than the English alphabet and difficult to recognize. The background colors of license plates in China are white, black, green, and yellow, and two colors are mixed in general. Grayscaling in this study is performed by using the weighted average method for weight distribution. The proposed method also recognizes characters in tilted images. The result shows that the proposed method allows accurate license plate recognition. The license plate in China is difficult to recognize as it has Chinese characters, numbers, and alphabets that code for provinces, cities, and registration, respectively. Training an algorithm for recognition requires several databases, including Chinese, English, and digital databases. By using the Hopfield neural network, we tested the proposed method for license plates and obtained a higher accuracy than previous methods. Despite that, the algorithm in this study needs improvement for recognizing characters on old and worn license plates, characters masked by rain or snow, and characters in low-quality images due to the wrong location of cameras. Enhancing the recognition accuracy for various types of license plates should also be studied.