Pattern Feature Matching for Mobile Robot Indoor Service

This paper presents the application of pattern recognition and environmental feature matching to an omnidirectional wheeled mobile robot (WMR). Hue-saturation-value (HSV) color space is used to replace normal RGB color, is similar to human vision, and can intuitively express light, shade, hue, and vividness. For pattern recognition, the back-propagation neural network (BPNN) and adaptive resonance theory (ART) are applied to identify digits. The patterns include elevator buttons, LED numbers, and doorplates. The speeded-up robust features (SURF) matching algorithm is used to guide the robot into a preset working path and to indicate the next direction and action of the robot. If obstacles are encountered along the path, the WMR uses ultrasonic sensors and a laser sensor to avoid them. In addition, fuzzy control is applied to correct the route offset caused by the long-distance movement of the robot. The algorithm used in the proposed system was written in MATLAB 2013. The wide-angle webcam, robotic arms, omnidirectional wheels, ultrasonic sensors, and laser sensor were integrated by LabVIEW 2014. Experimental results show that the proposed control scheme can make the omnidirectional WMR take an elevator to different floors and rooms automatically.


Introduction
The rapid development of science and technology in industry, whether it is for business work or entertainment, makes life more convenient and efficient. In recent years, many prefer the use of multifunction intelligent robots instead of labor for tasks such as home cleaning, hotel room service, and banking service robots. China's first unmanned bank opened in Shanghai, the robotic counter inside can provide more than 90% of the conventional bank counter business services. Robots can also be seen in major science and technology exhibitions and many movies. In the future, robots will be indispensable in human life, so how to make these robots more intelligent and humane will be an important research topic. There are many types of mobile robot, such as wheeled mobile, foot mobile, crawler mobile, and crawling robots. The purpose of this study is to enable a robot to perform office service, so we chose an omnidirectional wheeled mobile robot (WMR). The WMR is more flexible than ordinary mobile robots and can move in complex and narrow environments. (1)(2)(3)(4) When the omnidirectional WMR starts working, it must be given a working path. We use the speeded-up robust features (SURF) matching algorithm as the main direction indicator and fuzzy control to fine-tune the position error, and combine these two methods to achieve indoor navigation. In image processing, hue-saturation-value (HSV) color space is used for color screening, to filter out nontarget backgrounds. We use two algorithms for image recognition, namely, the adaptive resonance theory (ART) for identifying larger and clearer LED numbers, and the back-propagation neural network (BPNN) for identifying smaller and blurred elevator button numbers. During the experiment, the robot will determine whether the door has been opened or not. This depends on the value given by the laser sensor to determine whether the robot should stop or move. Static and dynamic obstacles encountered along the path are detected by ultrasonic sensors. The contribution of this study is the proposed control scheme that can make the omnidirectional WMR take the elevator to different floors and rooms automatically.
Artificial intelligence (AI) is the most hot research topic nowadays, and its most direct application is in robots. Omnidirectional wheel robot technology has been widely used in industrial, commercial, and service industries in recent years. For various needs, related research studies have been intensively carried out and various applications have been extensively presented. Some common robot functions are obstacle avoidance, path planning, visual image processing, object identification, and arm control. Zhang et al. researched on kinematic modeling and path following for a WMR. (5) Muir and Neuman proposed kinematic modeling for the feedback control of an omnidirectional WMR. (6) Chung et al. analyzed a dynamic model of an omnidirectional WMR. (7) Nie used an ant colony algorithm for robot obstacle avoidance and path tracking under a dynamically unknown environment. (8) Feng et al. used a rope method and nonlinear programming to solve the obstacle avoidance problem of a cleaning robot. (9) Ibrahim et al. presented the development of a mobile robot that uses a fuzzy logic approach to control its movement in an area cluttered with static obstacles. (10) Du et al. applied the SURF matching method to robots for multiple object recognition. (11) AI-Jubouri et al. proposed a method different from the general use of SURF matching method in static objects, which is used to identify dynamic zebrafish. A set of local features is extracted from a sequence of image frames collected through a computer vision system. The extracted set of features for each free-swimming fish is then compared with pre-extracted sets of features, which are stored in a database, using the SURF matching method. (12) Park et al. integrated an ART neural model for episodic memory with task memory for the task performance of robots. (13) Despite the advantages of fast and stable learning, ART still relies on an empirically fixed vigilance parameter to determine the vigilance regions of all the clusters in the category field. Meng et al. introduced two methods, the activation maximization rule (AMR) and confliction minimization rule (CMR). They allow different vigilance levels for different clusters, which are incrementally adopted during clustering. Both AM-ART and CM-ART improve the robust performance of fuzzy ART for the vigilance parameter and usually produce higher cluster quality. (14) Fajri et al. applied BPNN to distinguish mango varieties by leaf shape and texture. (15) Liu et al. utilized BPNN for image recognition on a coherent laser radar (LADAR). It is used to recognize vehicles on the ground with different carrier-to-noise ratios (CNRs) of clutters, different target orientations, and different distances between the object and the sensor. (16) Since the patterns to be identified are digit numbers in our study and the computing time should be short, conventional BPNN and ART are thus chosen in this study. Figure 1 shows the omnidirectional WMR that we used in this study. The waistline of the WMR is 150 cm. The omnidirectional WMR's chassis radius is 240 mm. Three omni wheels are configured in a space of 120° and three 12 V DC motors are installed to provide a rated torque of 68 mNm. To measure the distance and avoid obstacles, we set six ultrasonic sensors in front of the carrier of the second layer of the WMR. The second layer includes an electronic module, three motor encoders, and two batteries. The electronic module includes two parts, a voltage step-down circuit board and a control panel. The battery supply voltage is 12 V and the current is 7 Ah. One of the supply voltage services is for motor controllers and the others are for the control panel, additional power supply and voltage step-down circuit board. The voltage step-down circuit board reduces the voltage from 12 to 9 and 6 V, then supplies voltage to robot arms or other devices. The control panel is the kernel of this system, which has two panels. The first panel is Arduino whose main functions are (1) to transmit and receive signals between the note book and the receivers, and (2) transmit control command information to motors. The second panel is an I/O expansion shield for Arduino whose main function is to connect additional receivers. The robot is 600 mm in length, 400 mm in width, and 850 mm in height as shown in Fig. 2. It has two mechanical arms with two fingers each with the ability to grip stuff, and their lengths are 470 and 150 mm, respectively. The robot arms have 6 RX-64 motors, two in the shoulder and one in the elbow on both sides. RX-64 is 40.2 mm in length, 41 mm in width, and 61.1 mm in height. Its stall torque is 5.7 mNm. There are two RX-28 motors in the wrist on both sides. RX-28 is 35.6 mm in length, 35.5 mm in width, and 50.6 mm in height. Its stall torque is 3.77 mNm.

System Description
The kinematic model of the WMR (6) is shown in Fig. 3, where V m is the speed of the target direction m; V 1 , V 2 , and V 3 are the heading speeds of the wheels; m X  and m Y  are the decomposition parts of V m ; and α is the angle between the target and the car. The V m vector consists of m X  and m Y  . We can obtain V 2 for the V m direction as (1)  Then we can also obtain V 1 and V 3 from V m as

Control sequence
The flowchart of the control sequence is shown in Fig. 4. In this study, a system that allows the robot to execute an indoor service automatically is proposed. First, the photos taken by the webcam are constantly used to compare the SURF points so that the robot knows whether it has reached the target position or not. After finding any of the preset target locations, a series of corresponding actions are continued. When the robot arrives at the elevator entrance, it uses digital identification to identify the floor number on the screen and press the up-down button, and takes the elevator to the designated floor. Next, the robot leaves the elevator and moves to the desired door; it uses the HSV image process to check the door number, then knocks on the door and waits for the door to open. The robot leaves the laboratory after recognizing the command pattern. In the corridor, the robot avoids dynamic and static obstacles. Finally, the robot completes the tasks of taking the elevator and executing the service, and stops at the end of the corridor.

Image process
HSV color space (17) is an alternative representation of RGB color space. HSV encapsulates information about color in a way that humans are more familiar with: "What color is this? What is the depth? How is the lightness and darkness?". RGB color space values are transferred to HSV color space values as follows.
The doorplate image taken by a webcam and the processed image in HSV color space are shown in Fig. 5.

Digital identification
BPNN (15) is a supervised learning, which in the process of training requires a target. In the training process, the resulting error is propagated back to the neural network and repeated training makes the error converge to a minimum. We use BPNN to identify smaller elevator buttons. The BPNN structure is shown in Fig. 6. The floor-button panel of the elevator is shown in Fig. 7. The learning patterns and testing process of digit 9 are shown in Fig. 8. The learning patterns and recognition results of elevator button number 9 are shown in Fig. 9. The bar chart shows that the smallest output error of BPNN is the 9th neuron. This means that the unknown pattern is digit 9.  ART (13) is also applied to number recognition. It uses competitive learning rules to identify images. It needs to have a sufficient number of neurons waiting to be used. In this study, standard sampling is first used. Then, the floor numbers indicated on the front panel are recognized by ART. The ART neural network structure is shown in Fig. 10. The corresponding digital patterns of the elevator floor are shown in Fig. 11. Each digit uses one neuron to represent its pattern in ART. Figure 12 shows a sample input pattern of the 7th floor and its identification result. In Fig. 12, the 13th bar is the same as the 7th bar. This means that the unknown pattern is digit 7. ART is also applied to identify the room plate. Room numbers can be identified by the same process. Figure 13 shows the learning patterns of numbers that are captured from the doorplate. Figure 14 shows the input of ART for number 9 and its classification result.

Feature matching
In this study, SURF matching (12) is the main method to get the robot on the right path. The robot keeps taking pictures while moving and uses these pictures to compare SURF points, so as to find the target place and achieve indoor navigation. Before starting the experiment, we measure the maximum number of feature points that each target can be found. Then, we set a threshold, and when the number of matched feature points reaches the threshold, pairing is considered successful. Conversely, when the paired points are insufficient, pairing is considered a failure. SURF matching is shown in Fig. 15.

Fuzzy control
The purpose of adding fuzzy control is to fine-tune the robot's position in order to avoid errors caused by long-distance movements. Our goal is to let the elevator upstairs button appear in the middle of the screen so that the robot arm can accurately press the button. The proposed fuzzy control structure has two inputs and two outputs. Input X and Y are the X and Y coordinate positions of the button, respectively. Fuzzy sets are H (high), M (medium), and L (low). Output XD is the movement in the X direction of the robot, and fuzzy sets are L (left), S (stop), and R (right). Output YD is the movement in the Y direction of the robot, and fuzzy sets are B (back), S (stop), and F (forward). In simple terms, the X coordinate error distance controls the robot to move left and right, and the Y coordinate error distance controls the robot to move back and forward. The fuzzy rules are as follows.    Fig. 16. Figure 17 shows the working environmental setup. The robot takes the elevator from the 8th floor to the 9th floor and performs an office service. The first major task of the robot is to take the elevator. After finding the elevator location [ Fig. 18(a)], the robot observes the elevator floor from B1 to the 11th floor and presses the upstairs button [ Fig. 18(b)], and after moving into the elevator [Figs. 18(c) and 18(d)], the robot presses the 9th floor button to go to the 9th floor [Figs. 18(e) and 18(f)]. The laser sensor provides information for the robot to check the door and move out of the elevator, as shown in Fig. 19. The second major task is to deliver a document

Conclusions
In this study, an automatic control system based on real-time image recognition and environmental feature matching to control an omnidirectional WMR for room service including taking the elevator and delivering documents is proposed. In the control scheme, we used a human-machine interface integrated by LabVIEW 2014, and the algorithm applied in this system was written in MATLAB 2013. In previous studies, only one algorithm was applied to pattern recognition. Here, we used two algorithms to identify multiple targets and compared their differences. Experimental results show that the ART neural network is suitable for identifying large and clear targets. Its advantage is that it does not cost prelearning time. However, it cannot classify a large number of targets at the same time. BPNN is suitable for identifying small and blurred targets. Because indoor GPS navigation is not feasible, a more common method is to track a target point to guide the robot. Here, we used SURF matching as the main direction indicator. In obstacle avoidance, the robot successfully avoids different obstacles by distinguishing dynamic objects from static objects. In addition, the fuzzy control that we used is not very obvious because of the dynamics of the robot wheel, but it is still very helpful for fine-tuning the position. Experimental results show that the proposed control system can successfully lead the omnidirectional WMR to perform indoor service processes including taking the elevator and delivering documents.