Quadcopter Robot Control Based on Hybrid Brain–Computer Interface System

1Key Laboratory of Complex System Control Theory and Application, Tianjin University of Technology, Tianjin 300384, China 2Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China 3Department of Computer and Network Engineering, College of Information Technology, UAE University, Al Ain, 15551, UAE 4Zhonghuan Information College Tianjin University of Technology, Tianjin, 300380, China 5School of Artificial Intelligence, Xidian University, Xian 710071, China 6School of Computer Science and Engineering, Northeastern University, Shenyang 110189, China 7Department of Electronics and Mechatronics, Tokyo Polytechnic University, Tokyo 243-0297, Japan

A hybrid brain-computer interface (hBCI) has recently been proposed to address the limitations of existing single-modal brain computer interfaces (BCIs) in terms of accuracy and information transfer rate (ITR) by combining more than one modality. The hBCI system also showed promising prospects for patients because the design of a human-centered smart robot control system may allow the performance of multiple tasks with high efficiency. In this paper, we present a hybrid multicontrol system that simultaneously uses electroencephalography (EEG) and electrooculography (EOG) signals. After the preprocessing phase, we used a common spatial pattern (CSP) algorithm to extract EEG and EOG features from motor imagery and eye movements. Moreover, a support vector machine (SVM) was used to solve a multiclass problem and complete flight operations through the asynchronous hBCI control of a four-axis quadcopter (e.g., takeoff, forward, backward, rightward, leftward, and landing). Online decoding of experimental results showed that 97.14, 95.23, 98.09, and 96.66% average accuracies, and 45.80, 43.99, 46.78, and 45.34 bits/min average ITRs were achieved in the control of a quadcopter. These online experimental results showed that the proposed hybrid system might be better in terms of completing multidirection control tasks to increase the multitasking and dimensionality of a BCI.

Introduction
Brain-computer interface (BCI) technology is a direct human-environment interaction system mostly based on translating brain activity into relevant commands. This technology allows direct communication between the brain and any external device that does not depend on the peripheral nervous system and muscles. (1) BCIs can improve the quality of life of patients with cerebral palsy, amyotrophic lateral sclerosis, and stroke. The brain activity of such patients can be measured invasively or noninvasively by either electrocorticography (ECoG) or electroencephalography (EEG), respectively. (2)(3)(4)(5)(6)(7)(8) The basic operation of the BCI system consists of the signal acquisition, preprocessing, feature extraction, classification, control, and feedback phases. At present, the bottleneck problems of existing single-modal BCIs are the number of tasks that cannot meet the requirements of multi-degree of freedom (DOF) control and the low rate of correct recognition task, which limits the practical application of a daily-life BCI system.
To improve the performance of these BCIs, a new BCI, called a hybrid BCI (hBCI), for boosting the performance of existing single-modal BCIs in terms of accuracy and information transfer rate (ITR), has recently been proposed by many researchers (9)(10)(11) who have been trying to combine at least two BCI modalities such as P300-BCI, steady state visual evoked potential (SSVEP)-BCI, and motor-imagery-based BCI. (12)(13)(14)(15)(16) Other researchers tried to combine brain activity with nonbrain activity, such as eye movement activity measured using electrooculography (EOG), (17)(18)(19) muscle activity measured using electromyography (EMG), (20) the electrical activity of the heart measured by electrocardiography (ECG), (21) blood flow changes measured using functional magnetic resonance imaging (fMRI) or functional nearinfrared spectroscopy (fNIRS), (22)(23)(24)(25) and other physiological nonbrain signals to achieve a hybrid multidimensional control. hBCIs have many advantages compared with existing BCIs such as those with multitask, high-accuracy, and high-dimensionality features, plus a flexible hybrid control that reduces the BCI user's fatigue. The hBCI system can also be employed to design a human-centered smart home control system that performs multiple tasks with efficiency. (26) The hBCI system with different characteristics has become a research hotspot in recent years. Pfurtscheller et al. (27) put forward the concept of mixed brain-machine interface, in which the motion imagination event-related desynchronization (ERD) and SSVEP can be recognized as control command. In addition, both the parallel control mode and serial control mode can be used. In parallel control mode, ERD and SSVEP were recognized and used as control command simultaneously. In the serial control mode, ERD is used as the system brain switch and SSVEP is used for the detection of targets as the working and resting states, thereby improving system operability. Li et al. combined the movement of imagination and two P300 control methods, and using the design of a cursor in two-dimensional space to move the BCI system, one can complete web browsing, e-mail tasks and other tasks. (28) Leeb et al. studied the mixed BCI of EEG and EMG fusion and achieved some control tasks. (29) Punsawad et al. designed ERD and EOG models that were mixed and applied to the actual movements for machine control, i.e., the EOG control machine for measurements in the left and right directions, and to the ERD control machine for forward and stop movements with high control accuracy. (30) The mixed BCI system with multiple features has a high target recognition accuracy and more control instructions, which can meet the requirements of controlling complex systems and improving the performance and practicality of the BCI system.
A major challenge for BCIs is how to build a BCI-based control system with a high DOFs and greater asynchronous control, especially by using noninvasive measurements. Compared with a wheelchair, a quadcopter is a very suitable platform for verifying the high-dimensional control performance of a BCI. Researchers have achieved three-dimensional quadcopter robot control in a virtual reality environment (31) or in a real laboratory. (32) Khan et al. proposed a hybrid EEG-NIRS-based BCI for quadcopter control. (33) In this study, a novel hBCI system was designed to control a drone in an asynchronous manner, which combines motor imagery using EEG signals and eye movement using EOG signals. At the same time, the hBCI system processes independent features of EEG and EOG signals, improving its information transmission rate. The microcontrol of miniature four-rotor aircraft requires many DOFs to have multiple control instructions to realize rapid detection and recognition by this aircraft through online decoding.
EEG and EOG signals can be recorded by several devices. Table 1 shows the most commonly used EEG and EOG recording devices. NeuroScan and Eyelink are highly suitable for laboratory use; however, these EOG and EEG devices are not affordable for most handicapped people. Emotiv EPOC can be improved to simultaneously record EEG and EOG signals. Thus, Emotiv EPOC was selected to record signals in this study.
In this study, an hBCI system is designed to realize the microcontrol of the four-rotor aircraft for take-off, forward, leftward, rightward, backward, and landing operations. The experimental results show that the designed hBCI system has good maneuverability and robustness, and that the subjects are able to complete the flight control task of all four rotor operations.

Experimental paradigm and data acquisition
At the time of EOG data acquisition, the subject sits at about 1 m in front of the observation target so that the center position of the subject's eyes is at the same level as the observation target O point, as shown in Fig. 1(a), in the horizontal and vertical directions of the U, D, L, and R points, which are 1.5 m from the O point. The single EOG experiment paradigm for a duration of 8 s is shown in Fig. 1(b). When the "beep" sound is given as a cue, the subject starts scanning. After 2 s, the screen displays an arrow (up, down, left, or right). In accordance with the direction of the arrow corresponding to the direction of scanning for 3 s, the subjects line of sight returns back to the O point 3 s after the arrow disappears, and the subject relaxes for 3 s before the start of the next experiment. In motor imagery, the same test paradigm was used. Two seconds after the experiment starts, the left and right arrows are displayed randomly, and then the corresponding left-and right-hand motor imagery is selected. The imagery lasts 3 s. A total of four experiments were performed, each of which had 10 goals. In this study, an improved Emotiv EPOC EEG acquisition instrument was used to fabricate the EOG signal electrode on the basis of the original Emotiv EPOC. The electrode was connected to AF3, F7, AF4, and F8 channels. This improved Emotiv EPOC can collect not only EEG signals, but also EOG signals, which meets the actual requirements of hybrid system structures. The AF3 and F7 channels were used to collect the left and right eye movement EOG signals, and the F3, FC5, FC6, and F4 channels to collect motor imagery EEG signals. At the Emotiv EPOC sampling frequency of 128 Hz the collected signals were sent to the computer for processing. By observing and analyzing the EOG signals collected by the electrode, we found the electrode to be in good contact and to reflect the direction of eye movement well.
Four healthy subjects took part in the study, and they had not undergone in EEG before our proposed experiment. Signed written informed consent was obtained from all the subjects before the start of experiments, who were informed in detail about the purpose and possible consequences of the experiment. In all the tests, they were required to wear an electrode cap and relax in a comfortable seat with armrests. Then, they were instructed how to control the drone using their brain activity during the experiment.
During the experiment based on the above paradigm, the subjects were asked to follow the instructions (cues) on the screen to record their brain activity (EEG signals) and eye movement (EOG signals). The experiment and model training were performed both offline and online. In the offline experiment and model training phases, the recognized control commands were confirmed by the operator's eye blink. No commands were sent during offline recording or training to avoid the synchronous control of the drone. In the offline training phase, EEG and EOG data were recorded, and classification models were also built. After developing the online decoding algorithm, the commands were sent to achieve the asynchronous control of the drone without requiring the subjects to look at the screen to generate brain activity (e.g., P300 BCI or SSVEP BCI needs synchronous control).

Online decoding hBCI system
In this study, the online control of an hBCI using motor imagery EEG and eye movement systems was investigated. The processes involved included data acquisition, preprocessing, feature extraction, and pattern recognition methods for EEG and EOG signals. Finally, the control flight experiment by the online decoding of the four-axis aircraft was carried out as shown in Figs. 2 and 3.
By extending the EOG signal acquisition electrodes on the Emotiv EPOC EEG acquisition instrument and F3, FC5, FC6, and F4 electrodes for EEG signal acquisition, the collected data was sent back to the computer in real time through Bluetooth communication. For the EEG signals, a 3-24 Hz band pass filter was used to extract the frequency band of rhythm. Then, the features of the EEG signals were extracted on the basis of their common spatial patterns (CSPs), and the feature vectors were classified using a support vector machine (SVM). The details of the method can also be found in our previous work. (34) The classification results of left-and right-hand motor imagery were used to control the takeoff and landing of the four-axis aircraft.
On the other hand, the AF3, F7, AF4, and F8 electrodes were extended for EOG signal acquisition. We first used a hierarchical classification method to identify the vertical and horizontal EOG signals. Then, the vertical EOG signals were classified into the left and right eye movements as the control command, and the horizontal EOG signals were classified into three and four eye blinks as the forward and backward control commands, respectively.  Note that only two eye blinks were used in the offline decoding and model training without commands. The hierarchical classification method will be described in detail in Sect. 2.5. ITR (35) was utilized to evaluate the performance of the hBCI system as follows: where T is the time window length, N represents the number of commands, and P is the correct rate of the CCA algorithm.

CSP algorithm and feature extraction for EEG signal processing
The CSP algorithm (36) has been proved to be an effective method of feature extraction for motor imagery. Assume X A and X B are multichannel EEG signals in two different modes of where X A T is the transpose of X A and trace(X A X A T ) is the sum of the elements on the diagonal matrix X A X A T . The mean covariance matrices A R and B R are obtained by calculating the mean covariance matrices of multiple experimental data. Then, the mixed covariance matrix R is The eigenvalue decomposition of the mixed covariance matrix R is obtained as where U is the eigenvector of the mixed covariance matrix R and Σ is the diagonal matrix of the eigenvalues corresponding to the eigenvectors of R.
The whitening matrix P is constructed as The eigenvalue decomposition of S A and S B is obtained.
It is easy to see that matrices S A and S B have the same eigenvector, and the sum of the eigenvalues of the diagonal groups is the unit matrix.
It can be seen that the maximum eigenvalue of S A corresponds to the smallest eigenvalue of S B . In contrast, the maximum eigenvalue of S B corresponds to the smallest eigenvalue of S A . If the eigenvalues of S A are arranged in descending order, the corresponding eigenvalues of S B are arranged in ascending order. Therefore, two data are classified on the basis of the difference between the first and last m values in the order of Σ A and Σ B . The spatial filter matrix is designed as If we select the first m rows and rear m lines of the eigenvectors U arranged in the order of the eigenvalues to form a new eigenvector U′, we can obtain the spatial filtering matrix as SF′ = U′ T P. (10) For the EEG signal X k , the projected EEG signal Z k is obtained as Z k = SF′X k . According to the m value from the projected EEG signal Z k taking the row Z p as the variance, the following equation is used to obtain the eigenvalues.
where var(•) is the variance, and p = 1, 2, ... , 2m. After processing using the CSP algorithm, we can obtain a 2m dimension eigenvector for classification.

SVM method
SVM (37,38) was used twice in the present work, as shown in Fig. 4. SVM was employed first to classify EEG data into the left and right motor imagery, and then to classify EOG data into vertical EOG (VEOG) and horizontal EOG (HEOG). SVM is based on Vapnik-Chervonenkis theory and a structural risk minimization principle based on a statistical learning theory. It has good generalization ability, which is useful for solving problems with small samples, and nonlinear and high-dimensional pattern recognition ability.
In this study, the classification of motor imagery EEG signals was based on the structure of the SVM classifier. The basic principle of the SVM method is to map the input x in a highdimensional feature space [ ( ) φ = z x ] and look for the optimal decision hyperplane. The decision hyperplane is defined as where w is the normal vector and b is the bias of the separation hyperplane. The decision hyperplane can be found by solving the following optimizing problem: where i ζ is the slack variable that allows an example to be in the margin (0 1 i ζ ≤ ≤ ， also called a margin error) or to be misclassified ( 1 i ζ > ), and C is a penalty factor that can be set by users; a larger C means that the user assigned a higher penalty to errors. In addition, the Gaussian kernel function was used in the training of SVM classifiers 1 and 2.
where K(•) represents the kernel function.

EOG signal processing
The hierarchical classification method was employed to process EOG signals using a 0.5-10 Hz band-pass filter. Then, the filtered EOG signals were decomposed by dual-tree complex wavelet transform (DTCWT). (39) The details of DTCWT can be found in our previous work. (40) where ( ) is an analytic signal that is supported on only half of the frequency axis.
After DTCWT processing, four features, namely, the maximum wavelet coefficient, the area under the curve, the amplitude, and the velocity, were selected to build the feature vector. The details of the feature calculation method can also be found in our previous paper. (17) Then, SVM classifier 2 (Fig. 5) was trained using the data recorded in the offline experiment (the SVM method is described in Sect. 2.4). In the online decoding experiment, an EOG signal was recorded and its features were extracted, and then the feature vector was classified using the SVM classifier well trained in the offline experiment. In this manner, the EOG signal was classified into the VEOG or HEOG signal. In classifying movement patterns, the left-to-right eye scan was considered a pattern of the first category that corresponded to the VEOG signal, and the up and down eyeball movement and eye blinks were considered patterns of the second category that corresponded to the HEOG signal. When classifying different eye movement patterns into the first category, the VEOG data were analyzed to determine the positions of the maximum and minimum amplitudes on the vertical axis. The characteristics of the left and right eye movements are shown in Fig. 3, obtained when HEOG waveform time domain characteristics were used. For the classification rules, we calculated the distance (A, B) and compared it with the threshold T. If the distance (A, B) is greater than T, then the user blinks once only. If the distance (A, B) is greater than T and A is less than B, the eyes of the user move up. If the distance (A, B) is greater than T and A is also greater than B, the eyes move down.
Successive blinks (two to four blinks) were classified into third category using the differential counting algorithm to identify the number of consecutive blinks and convert the recognized number of blinks into the corresponding control command. Then the four-rotor aircraft can be microcontrolled to fly. The differential counting algorithm is the process of identifying continuous blinking signals by the difference method. The subject's EOG data are used to preset the corresponding threshold. The threshold T > 0 cannot exceed the maximum value of the signal, compared with the point amplitude Y and threshold T of each sample. If Y > T, set Y to 1, otherwise set Y to 0. After the above steps, the EOG data will become a number of positive rectangular waves, then each rectangular wave will be a point-by-point difference, each forward rectangular wave will become a number of positive and negative pulses, and finally the negative pulse is removed. The statistics of the number of positive pulses, according to the statistics of the number of positive pulses, can identify different continuous blinks.

Experimental Results
In the experiment, the subjects were trained according to a predesigned experimental task. When the motor imagery or eye movement mode task is started, the acquired signals are processed by the algorithm method in real time. The result of each recognition task is displayed in the task recognition window. If the indicator is lit, then the takeoff indicator light turns on. To confirm to the command controller that the recognition result is correct, the subject blinks twice, and the transmitted recognition result command is displayed in the aircraft control command window, so that the small four-axis aircraft can be controlled. If the recognition result displayed in the task recognition window is incorrect, the subject does not blink and the error command is not sent to the instruction controller, and then the next control task is performed. The GUI control interface designed in this study has good error correction mechanism, which ensures a high correct rate for the control of the aircraft.
Before the experiment, each subject was trained many times to master the use of the GUI. In the experiment, each subject was required to master the control of the four-axis aircraft to takeoff, to move forward, leftward, rightward, and backward, and to land in this sequence. After a number of online experiments, the correct rate of flight control was significantly improved. Each action was completed 30 times in the experiment. The results of each subject's completion of all tasks are shown in Table 2. There are many parameters that can affect EEG signal quality. In addition, the EEG changes over time and from subject to subject. Therefore, each subject performed the tests on 5 different days that were separated by 20 h to ensure the stability of our hBCI performance (See Fig. 6). On each day, the experiments were performed in one session with each action performed 30 times to avoid fatigue. We observed that the hBCI performance improved day by day owing to these practices of the subjects.

Conclusion
An hBCI is composed of two BCI modalities, or at least one BCI and another system based on additional input. The hBCI system can provide classes that are more independent, have more functionalities and higher DOFs. In this paper, we present an asynchronous quadcopter control system using modified portable EEG equipment. To achieve natural and asynchronous control, we combined motor imagery and eye movements to control the drone through online  decoding. The proposed algorithm was designed to classify six classes to control the directions of a quadcopter. Results of the study indicated that successful multiclass control is possible using the hBCI instead of the existing single-modal BCIs. In addition, the performance characteristics, especially the classification accuracies of the proposed hBCI, improved day by day.
We controlled the direction of the drone and realized online decoding by analyzing brain activity and eye movements. When the user imagines or moves his/her eyes, the drone will receive directly the commands with small delay due to the processing time of the hBCI algorithm. There are some limitations of this study. Presently, this system just used forward control. We did not add a feedback correction to our system yet. The users were able to see the feedback but were not able to correct any command. In the future, we are planning to control many parameters of the drone (position tracking or velocity control performance) to achieve an accurate and precise control.
We would also like to develop in the future a new wireless assistive technology based on a portable, noninvasive, and asynchronous hBCI, which allows patients to control a drone using their brain activity and receive a visual feedback. This type of hBCI for building humancentered assistant systems can help patients control smart homes by themselves.