Sensor Fusion and Gesture Control for Quadcopter Application

In this paper, we present the sensor fusion of a quad rotor aircraft. There are two main applications examined in this study, namely, gesture control and object detection. In gesture control, we can use hand and gesture to control the quadcopter instead of a traditional joystick. In the second application, the quadcopter can be applied to detecting an object surface automatically. The quadcopter can fly along an object to identify its color and circle around the object from a high altitude to a low altitude. If an abnormal area of the object is detected, the quadcopter will hover at that position, and the inspector can determine whether the object needs to be repaired through the images transmitted from the quadcopter. In addition, a voice broadcast system is integrated in the quadcopter, which can report the current status of the quadcopter. The inspector does not need to pay attention to the quadcopter and only needs to listen to the speaker. Ultrasonic sensors with a fuzzy logic system are applied to automatic circling and descending flight control.


Introduction
The development of the quadcopter can be traced back to the early 1920s; Bothezat et al. (1) pioneered the first four-rotor aircraft. It has low stability, but the United States began to vigorously promote unmanned aerial vehicle (UAV) research, and the quadcopter model was gradually taken seriously. In the late 1990s, the United States began to push for more UAV research and development. In recent years, the number of relevant research studies has gradually increased. Most of the early four-rotor unmanned aircraft focused on theoretical research or as a research platform. Altug and coworkers used a camera to obtain feedback images for four-rotor unmanned aircraft attitude research. (2)(3)(4) Bouabdallah and coworkers conducted a series of control studies using the miniature four-rotor UAV research platform that they developed. (5,6) Wendel et al. used four-rotor unmanned aircraft with GPS for automatic navigation missions. (7) In 2007, Hoffmann et al. used experimental methods to verify the thrust and aerodynamic effects of a four-rotor unmanned aircraft. (8) Many researchers used their own or purchased four-rotor unmanned aircraft. (9,10) Only a few are customized for overall design and production. (11,12) For example, in 2002, Pounds et al. (13) probed the design of the structure and production of a quadcopter and used commercially available motors to control the rotation speed. Pounds et al. also published an updated four-rotor unmanned aircraft design in 2006 and 2010. (14,15) There are several control theories such as PID control and fuzzy control that can be applied to a quadcopter. Santos et al. (16) proposed a quadcopter fuzzy controller and verified the results of quadcopter path control through simulations. Dydek et al. (17) proposed a selfadaptive fuzzy controller for quadcopters. The controller is designed for a situation wherein one of the rotors is damaged, but the control system can make the fuzzy system self-adapt and maintain flight balance. Wang (18) combined the PID control and hardware through a fuzzy system to improve the delay of the rotors. In unmanned aerial vehicle research, a traditional helicopter was most commonly used in the laboratory. The open source software ArduPilot and related documentation give basic related knowledge and operating concept at the initial phase. The development environment and assembly operation were from Refs. 19 and 20. Target tracking and the function of obstacle avoidance were tested using the method presented in Refs. 21 and 22. A method of image tracking of massive items, such as cars, ships or airplanes, was proposed in Ref. 23. Gesture control with a quadcopter is presented in Ref. 24. In our previous study, (25) color identification was applied to a quadcopter for object inspection. With the help of the above studies, we focused on a detection quadcopter that can be controlled by hand or automatic mode in this work.
In the past, UAVs were limited to fixed-wing airplanes and traditional autogyros. Flight efficiency with long flying time and high speed are the advantages of the fixed-wing airplanes, compared with the traditional autogyros and multirotors. On the other hand, maintaining a certain flight speed to make the body maintain a sufficient lift and a sufficient runway length with a specific recovery mechanism for taking off and landing are the drawbacks of the fixedwing airplane. Compared with traditional aircraft, a quadcopter can take off and land vertically, hover at the same height in the same place, and has lower cost, smaller size, and higher mobility. A quadcopter has greater capability to move in three-dimensional space. Although the quadcopter is much smaller than an airplane, it can be equipped with a GPS and an inertia navigation system (INS), which are UAV autonomous flight navigation systems. Moreover, wireless communication technology is well developed nowadays, and coupled with flying capability, a quadcopter can overcome terrain factors. Given these advantages and features, it is suitable for a quadcopter to perform aerial investigation and patrol tasks. Using a quadcopter to conduct a reconnaissance mission in war zones or to transport goods in disaster areas is more economical and safer for human life. Besides, a quadcopter with additional sensors can have greater automated and comprehensive capabilities. However, the automated flight technology of quadcopters is not yet mature, and there are still many unexpected situations wherein we have to rely on remote control to stabilize its flight. However, from the operator's viewpoint, the quadcopter remote control operation is not very intuitive, and it is more difficult for a beginner. Therefore, the idea of developing an automatic quadcopter that can be switched to hand control during an urgent situation is proposed in this study. This study has two main parts. In the first part, a body sensor is used to replace the traditional remote control. By using gesture to control the flight attitude of the quadcopter, Leap Motion is applied to transmit data through Xbee between the ground station and the quadcopter. The other part uses a flight control board (Pixhawk) combined with different sensors to perform an inspection task. A Pixy camera is used to track a desired target. An ultrasonic sensor and fuzzy control are applied to determine the flying direction and avoid obstacles during an investigation mission. Figure 1 shows the quadcopter main structure. A PA66+30GF high-strength plastic material is equipped with the flight control board, sensors, motors, batteries, and so forth. In this study, the flight control board used by the quadcopter is Pixhawk, and 3DR radio telemetry is the wireless communication equipment. An ultrasound sensor and a Pixy camera are used for image and distance detection. Figure 2 shows the overall quadcopter hardware block diagram. The image transmission system was originally used as the eyes of the quadcopter to allow the quadcopter to perform the task of detection, but we found that during the entire inspection process, the inspectors must pay attention to the quadcopter in the air, which is very inconvenient. Therefore, we utilized voice broadcast so that the inspectors do not need to pay attention to the quadcopter. They only need to listen to the speaker of the broadcast system to determine the current status of the quadcopter. The voice chip that we used in the voice system is DMA1G08A, as shown in Fig. 3. We first recorded the desired voice files through the voice file translation program. Then we imported the recorded voice files into the voice announcement chip. When the quadcopter was in the autostabilization mode or hovering mode, the voice chip can broadcast the voice file that we imported. The On-Screen Display (OSD) program was modified to control the DMA1G08A voice chip. The OSD program used the Mavlink protocol. The source signal was transmitted to the ground station monitor via audio transmission on the image transmission system, and finally achieved the voice broadcast function as shown in Fig. 4.

Gesture Control
Leap Motion is applied to control the quadcopter by hand and gesture. The Leap Motion sensor can detect hands, fingers, or straight tools, and returns gesture and position information immediately. The Leap Motion sensor assigns an ID to all detected objects, and the ID will not change when the palm, fingers, and hand tools remain within the field of view. As shown in Fig. 5, when the ground station and the quadcopter are ready, we can execute the program and Leap Motion. Hand positions correspond to different English words, and they are transmitted to the quadcopter. The Arduino board that is on the quadcopter will convert the received English words to numbers using the ASCII CODE, and each number corresponds to a different control movement, so that the quadcopter can perform the required flying attitude that we want. The working range of Leap Motion can be established by a 3 × 3 × 3 cube space and split into 27 small cubes, and the commands of each cube are shown in Table 1. The start of the process is waiting for the hand signal. The quadcopter does not have any action and is in the locked state at this time. When the hand in the form of a fist is sensed by Leap Motion, the yaw value is set to the 1900 position ( Fig. 6) so that the quadcopter can be unlocked, then the quadcopter can start flying in response to given different gestures.

Control Scheme
The control sequence of target tracking is shown in Fig. 7. In hover control, there are indoor and outdoor modes. Indoor hover uses a light-sensing element, and an ultrasonic sensor is for altitude holding. In the outdoor mode, the GPS provides coordinates for hover, and barometric pressure is for altitude holding. When the quadcopter is close to the desired object, the position will be locked in the optical flow or GPS mode, and the target will be tracked after switching to the automatic mode. Because the main goal is to check if the object is damaged or rusty, in this study, we use the Pixy camera to track a target and send the captured color images to Arduino Mega 2560. After capturing the target, the quadcopter will hover in front of it. The camera on the quadcopter will transmit images to the ground station through an image transmitter.
In the flowchart, the signal of the ultrasonic sensor HC-SR04 to Arduino Mega 2560 is divided into the direction decision and obstacle avoidance. Before Pixy has detected the target, the ultrasound signal will be used to determine the direction according to the distance between the quadcopter and the object, and send the PWM signal to the Pixhawk flight control board, so that the whole process of investigation is kept under a safe state. For the obstacle avoidance, ultrasonic sensors are installed on the quadcopter and are used to avoid obstacles.

Hover control
Optical flow can help a quadcopter hover at the same position. Otherwise, the quadcopter may drift caused by the inertia in the self-stabilized state. In this study, the indoor experiment uses optical flow to hover at a preset position and the outdoor experiment uses GPS during hovering. Figure 8 shows that the quadcopter uses PX4flow to adjust itself at the same place. Figure 9 shows the quadcopter position and altitude, where the blue, black, and brown lines are the coordinates of X-, Y-, and Z-axes, respectively. The X-and Y-axis coordinates were kept to 0. The accuracy of the altitude of the quadcopter is not good as that of the X-Y plane, but the error was acceptable.

Filtering and obstacle avoidance
The ultrasonic sensors are installed on the quadcopter as shown in Fig. 10. The wiring of ultrasonic sensor is Vcc (positive electrode), trig (control terminal), echo (receiving end), and GND (ground connection). These ultrasonic sensors, trig, and echo are linked to the pin of Arduino Mega. The three faces of the quadcopter can detect if there are any obstacles beside the quadcopter. In the case of automated control, it is important to prevent noise from the sensors, which interferes with the entire control process. Using Arduino to give control commands during the process requires adding a filter to ensure that the AIL, ELE, RUD, and THRO values are stable. Without the filter, the noise pulse would be taken and cause undesired movement. Some of the values of the pulse would be more than 50; thus, every 10 output values by Arduino Mega are averaged. There might be a small pulse, but the value would be about 10. For the quadcopter, this kind of noise pulse will not make the altitude of the quadcopter change considerably. Figure 11 shows the remote controller analyzing data with and without a filter.
It is necessary to determine the direction of the flight through the distance captured by the ultrasonic sensor in front of the quadcopter. The distance, flight direction, and values given by Arduino can be seen on the serial monitor, which confirm that the control signal is entered into the Pixhawk flight control board in the window of the remote control numerical detection of the Mission Planner ground station. When the quadcopter does not detect any object, the ultrasound value will be 4000 in this situation, and the quadcopter will not change its yaw at this time. The safe distance is set to 50 cm; if an object is detected and the distance is less than 50 cm, the quadcopter will then move back. Figure 12 shows the flowchart when the sensor senses an obstacle.

Image identification
Pixy is the image processing sensor that detects and checks the desired target, and sends a hover command to the quadcopter. The Pixy camera port connects to the Arduino Mega board for communication, as shown in Fig. 13. Figure 14 shows the result that the Pixy camera captures the desired red object and the result of two different colors. It can be seen from the figure that even when the objects are overlapped, the result of the processed image can be distinguished between the two colors.
Because the task is to find rust or damage on a hull, it is necessary to test the effect of Pixy on capturing rust in an actual environment. The object of the experiment is the Hikong vessel, which is in Keelung Badozhi Park. There are white, blue, and red colors on this vessel surface, and the rust and hole color is usually black or brown, so it is a suitable experimental object. Figure 15 shows the hull image of the Hikong vessel. Different colors can be predefined at the beginning of each task so that the specific color of the desired object can be detected.

Fuzzy control
The quadcopter needs to face the target surface in order to allow the entire control process to be performed in expected circumstances during the detection process. Because the body of the vessel is rectangle in shape, and the quadcopter is working in a three-dimensional environment, a mechanism is needed to adjust the yaw angle when facing the turning site so that the quadcopter can keep facing the target surface. In this study, we use fuzzy control to adjust the angle of the yaw command. The value of the rotating angle along the z-axis is a problem at the turning site. Under appropriate setup conditions, when the quadcopter sensor does not sense any object signal at its front position, it indicates that the quadcopter can pass over the edge of the object surface. When that occurs, the quadcopter has to make a yaw turn along the z-axis, and then the quadcopter will face the surface of the object again. In the case of using only a single ultrasonic sensor, two situations will happen. The first is that when the quadcopter's yaw angle is not large enough, the next object (the next surface at right angle to the previous surface) will not be in front of the quadcopter. The quadcopter needs to adjust its yaw angle during flight and do it several times; this process results in a zig-zag flight of the quadcopter, as shown in Fig. 16. The second situation is that when the quadcopter's yawing angle is too large (over 90°) and it passes over the object surface, then the sensor cannot detect the surface of the other side of the object and the quadcopter will continue its yaw turn, as shown in Fig. 17. If the quadcopter still could not sense the object surface, it will then maintain its motion and rotate along the z-axis. However, even if the quadcopter detects the object surface, it will only change to situation one and make a zig-zag flight. To solve these problems, two ultrasonic sensors and fuzzy rules (Fig. 18) are used to adjust the yaw angle. In Fig. 18, fuzzy rule-1 shows the case in  which the two sensors at 0 and 45° indicate that the obstacle is close to the quadcopter, then the pitch command is pitch-back strongly and the yaw and roll commands are not changed.

Flight simulation
Because the goal of this study is to use a quadcopter to detect an object surface, we use the MATLAB Simulink tool to simulate the flight path of the quadcopter. The model is from Ref. 25. This model is controlled by the PID controller, and the z-axis is adjusted so the quadcopter can circle the surface of an object. The main quadcopter system is divided into the controller subsystem, attitude subsystem, and displacement subsystem. Figure 19 shows the block diagram of the main quadcopter system.
The controller subsystem is responsible for calculating the commands of angle and altitude needed by the attitude controller. The displacement subsystem provides the output values, which are the real amounts of displacement of the quadcopter calculated using the attitude calculator. Since the self-stabilizing controller is within the quadcopter's flight control board using also the PID control, the quadcopter that simulates the dynamic control system is also using PID control. The PID controller is significantly affected by Kp, Ki, and Kd, which are three control parameters. Figure 20 shows the path whereby the parameters have not been adjusted, and Fig. 21 shows the path whereby the parameters have been adjusted. Comparing Figs. 20 and 21, we can see that the latter has a better flight trajectory than the former.

Experimental Results
The experiments include indoor and outdoor environments. The results for gesture control and auto-fly along an object are presented. Gesture control shows the attitude and response of the quadcopter in indoor and outdoor environments. The autofly experiment shows with and without fuzzy control at the turning site.

Gesture control
The experiment is shown in Fig. 22. The quadcopter uses optical flow to locate its position and uses the QGC ground station to record the attitude of the flight during the experiment. The main screen shows using a hand command to control the quadcopter, and the inserted screen shows that the hand command was transmitted to the quadcopter by the ground station. Since the indoor space is small, to ensure that the control command can be more stably transmitted, we use the position of the middle finger as a command. The flying attitude of yaw and the flying height are shown in Fig. 23.

Autofly experiment
In an indoor environment, we assume the corner as the turning side of an object. The experiment has two parts, one is using a single ultrasonic sensor, and the other is using two ultrasonic sensors (installed on 0 and 45° of the quadcopter) with the concept of fuzzy control to give the control command. An experiment with fuzzy and z-axis controls is shown in Fig. 24. In the outdoor experiment, the quadcopter is set to fly to the right direction along the surface of a container. The tasks of detecting the container and recording the inspection are processed by a camera installed on the quadcopter as shown in Fig. 25.

Hull inspection
The quadcopter can capture the rusty objects on the Hikong vessel using the Pixy camera. Using PixyMon and setting signature 2, we can detect the rusty area (brown color) on the hull. When the range coefficient is 2.640, Pixy can capture most of the rust, and the brown color in this range coefficient is set to be the ultimate goal. The quadcopter will hover in front of the target and send the image back to the ground station, which is double-checked by the ground staff. The rusty areas in the blue and white parts on the hull are shown in Fig. 26. In this experiment, we mainly perform the task of detecting a real ship, but there are too many obstacles around the vessel in the scene, so we only focus on half of the hull as the experimental object (the circling control experiment is shown in Fig. 25). The quadcopter flies along the hull, and when it captures the color that we set, it will hover in front of the target object, as shown in Fig. 27.

Conclusions
To make life more convenient, body sensors have been actively developed nowadays. Leap Motion uses different simple gestures to make it easier to browse a website and play some interactive games. Many studies show that different body sensors have been used in the field of robot research applications, such as using Leap Motion to control a robot arm on an automatic car, but not for quadcopters. In this study, we apply Leap Motion to a quadcopter; the results of the experiment show that there is a slight delay between the quadcopter flying attitude and the action of the hand. However, from the response diagram, we can see that the control commands are very similar to the feedback of the quadcopter, which means that it is stable in the whole process. In this study, we only can control the flight attitude; however, through different gestures, it is feasible to perform more applications. Using the hand to control the quadcopter has a certain entertainment effect, because more young people are familiar with the rotorcraft. Thus, it may be a business opportunity when more gesture control features are developed in the future. In the object detection task of our previous work, (25) owing to the limited control board memory, the quadcopter can only fly and check an object in one circle. In this study, an advanced control board is utilized, and descending flight control while circling an object can be achieved by the software simulation and hardware experiment. The proposed system can perform object inspection automatically. In addition, the quadcopter is equipped with a broadcast system for the safer and easier performance of the proposed task. The operator does not have to pay attention to the quadcopter to perform the task. The status of the quadcopter while in flight can be heard from the speaker of the broadcast system. When a desired target is detected, the information will be sent to the broadcast system; then, the operator can check the monitor in the ground station and make the next step decision. Compared with other systems, (26)(27)(28) most surveillance quadcopters are operated remotely or manually, which require skilled operators during the mission. They also lack gesture control, fuzzy control, image identification, or voice assistance. In this study, automatic object inspection with multiple functions is tested successfully. The proposed quadcopter system has better performance than previous presented systems.