This paper presents the application of pattern recognition and environmental feature matching to an omnidirectional wheeled mobile robot (WMR). Hue-saturation-value (HSV) color space is used to replace normal RGB color, is similar to human vision, and can intuitively express light, shade, hue, and vividness. For pattern recognition, the back-propagation neural network (BPNN) and adaptive resonance theory (ART) are applied to identify digits. The patterns include elevator buttons, LED numbers, and doorplates. The speeded-up robust features (SURF) matching algorithm is used to guide the robot into a preset working path and to indicate the next direction and action of the robot. If obstacles are encountered along the path, the WMR uses ultrasonic sensors and a laser sensor to avoid them. In addition, fuzzy control is applied to correct the route offset caused by the long-distance movement of the robot. The algorithm used in the proposed system was written in MATLAB 2013. The wide-angle webcam, robotic arms, omnidirectional wheels, ultrasonic sensors, and laser sensor were integrated by LabVIEW 2014. Experimental results show that the proposed control scheme can make the omnidirectional WMR take an elevator to different floors and rooms automatically.