Effect of Electromyography Signals on Single Joint Motion Forecasting

Toward reducing the effect of delay on motion transmission to a remote place, methods of forecasting human motion with subsecond preceding time have been studied. In this paper, we verified whether the prediction of single joint motion could be improved by using surface electromyography (EMG) signals. We used a recurrent neural network to predict the flexion and extension movement of a thigh, and compared the results between the prediction using only the angle and that using both the angle and EMG signals of two muscles. As a result, in the prediction of motion of about 0.5 Hz, the accuracy and delay of the prediction tended to be improved by using the EMG signals (e.g., in 0.3 s ahead prediction, the mean of the root-mean-square error between participants and trials is improved by 0.7°, and that of the prediction delay is reduced by 0.045 s). Such motion forecasting using EMG signals may be useful for improving the operability and stability of medical robots in telerehabilitation and telesurgery.


Introduction
As medical robots become a reality, the real-time transmission of human motion becomes an important issue. Applications of robots with remote communication in medical treatment include telerehabilitation (1) for remotely supporting the movement of a patient by a therapist in rehabilitation training and telesurgery (2,3) for remotely operating a surgical robot. In such applications, since communication delay affects stability and operability, the introduction of a low-latency communication system such as a fifth-generation mobile communication system (5G) is required. (4) However, as the distance increases, the delay becomes inevitable because the speed of light is finite, and it is necessary to deal with the delay using new control methods.
As a method of dealing with such a situation, there are some methods of predicting human motion after a short time from a motion history or a current situation. Horiuchi et al. developed a system that predicted human posture 0.5 s ahead in real time using a five-layer neural network and whole body motion acquired with a motion capture camera (Kinect). (5) Wu and Koike developed a system that made similar predictions using deep learning with RGB camera images. (6) In some studies, motion prediction was attempted by machine learning using a largescale data set of human motion. (7)(8)(9) Although these methods enable the prediction of whole body movements such as jumps using information on many body parts, movements performed in telerehabilitation and telesurgery are often performed only with arms and legs, and optical information alone may not be enough to predict the movement.
Other information that may be useful in predicting motion is surface electromyography (EMG), which is measured on skin and reflects the muscle activity that causes human motion. Several studies have succeeded in estimating joint torque and motion using EMG signals. (10,11) In our study of a real-time motion teaching system using the master-slave assist suit, we have developed a method of predicting the thigh angle of a teacher 0.3 s ahead using EMG signals and postures by a recurrent neural network and communicating it to the learner. (12) As a result, the delay of about 0.4 s in the first version was reduced to about 0.1 s.
In this work, we investigated how the prediction was improved by using EMG signals compared with the case of using only posture information. EMG signals represent commands from the brain to muscles that cause movement, and occurs before the movement of the body part is observed, so it is expected that using EMG signals will improve the prediction accuracy, especially at the start of movement. In this paper, we showed that this conjecture is promising by analyzing the result of the thigh angle prediction by the neural network using the angle or both the angle and the EMG signals.

Data set
In this paper, the posture and EMG signals of four participants (all men; 22-34 years old) who flexed and extended their thighs were used for analysis. Our study was approved by the institutional review board at Tokyo Medical and Dental University. All participants provided written informed consent according to institutional guidelines.
The participants stood upright and flexed and extended their thighs with the lower leg lowered vertically [ Fig. 1(a)]. They were instructed to move their thighs in shorter cycles (approximately 0.5 Hz) for 20 s, longer cycles (approximately 0.2 Hz) for 20 s, and irregular cycles (approximately 0.2 to 0.7 Hz) for 50 s sequentially (Trial 1). This movement was repeated again after a rest (Trial 2). The angle of the thigh during the movement was measured with a motion capture camera (Kinect, Microsoft), and the EMG signals of sartorius and gracilis muscles, which are a flexor and an extensor of the hip joint, respectively, were measured with an EMG measurement system (Bagnoli 16, Delsys) [ Fig. 1(b)]. The angle was sampled at a sampling frequency of about 30 Hz, and EMG signals were sampled at 1000 Hz. The origin of the thigh angle was set to the position when the thigh was lowered vertically, and the flexed posture was assigned to positive values.
Before the training and testing of the neural network, the EMG signals were preprocessed as follows. First, the EMG signals were filtered with a high-pass filter (cutoff frequency: 20 Hz; 2nd-order Butterworth filter) to remove low-frequency artifacts and a band stop filter (stop band: 49-51 Hz; 4th-order Butterworth filter) to cut power supply noise. Then, the EMG signals were rectified and filtered by a low-pass filter (cutoff frequency: 3 Hz; 2nd-order Butterworth filter) like in other studies. (13,14) The processed EMG signals represent the level of muscle activity and correlate with muscle tension. (15,16) The processed EMG signals and the angle were resampled at 50 Hz while interpolating the angle with zero-order hold, and these processed data were used in the subsequent analysis.

Method of motion prediction
In this paper, the prediction method by the recurrent neural network used in the masterslave-type assist suit (12) was used in two patterns, which were the prediction using only the angle and that using both the angle and the EMG signals. Figure 2(a) shows the recurrent neural network with the configuration obtained when only the angle θ was used for prediction, and Fig.  2(b) shows the configuration obtained when the angle θ and the EMG signals u 1 and u 2 were used. This neural network was trained to output a predicted value ˆt θ ∆ of an angle θ Δt , which is the angle θ after time Δt, and received sensor data (the angle or both the angle and EMG signals) in the past 0.2 s (10 samples) and the neural network output in the past 0.12 s (6 samples) as input values. The number of nodes in the hidden layer was set to four as in the previous study. (12) A hyperbolic tangent function was used for the activation function of the hidden layer, and a linear function was used for the activation function of the output layer. The neural network training was performed to minimize the following evaluation function in order to bring the output closer to the angle after Δt: where w is the weight and bias of the neural network, and N train is the length (number of samples) of the training data in discrete time. In the representation of the time series, time is expressed as discrete time. In actual training, validation data were prepared, optimization was completed when the evaluation function in the validation data increased six times in succession, and the weights before the increase were used. Since the weights at the end of optimization differed depending on their initial values, the network was trained 10 times with different initial values and the same training data, and used the result when the evaluation function for the validation data provided the smallest value. Levenberg-Marquardt backpropagation was used as the optimization method. The neural network was implemented using the Neural Network Toolbox of MATLAB R2017b (MathWorks).

Evaluation of prediction performance
To compare the prediction using only the angle and that using the angle and EMG signals, we used two measures: a root-mean-square error (RMS error) and a prediction delay. The RMS error evaluates the difference between the correct value and the predicted value. The prediction delay evaluates the delay that occurs when the neural network cannot predict a change in angle in the designated lead time (Δt) and reacts immediately before or after the change. In either index, the smaller the value, the more accurate the prediction.
The RMS error was defined as where N test is the length of the test data in discrete time.
As the prediction delay, the output delay of the neural network with respect to the ideal prediction obtained by time-shifting the angle was used. The delay between the two signals can be evaluated using the lag time at which the cross-correlation function takes the maximum value. On the basis of the above, the prediction delay was defined as where t s is the sampling period and ˆt is the cross-correlation function between ˆt θ ∆ and θ Δt defined as where ˆt θ ∆ and t θ ∆ are the mean predicted and actual angles in the test data, respectively. The maximum value L max and minimum value L min of the prediction delay were set to 0.5 and −0.5 s, respectively. The first coefficients of Eq. (4) were required for the unbiased estimation of crosscorrelation. (17) In evaluating the predicted performance, Δt of 0.05 to 0.7 s (step 0.05 s), which covers the range that has been mainly used in existing studies, (5,6,8,9) was used. For each trial, the first 80% (40 s) of the movements with irregular cycles were used as the training data, and the next 20% (10 s) were used as the validation data. Then, the evaluation using the above two indices was performed using the last 10 s of the data with about 0.5 Hz motion and the last 10 s of the data with about 0.2 Hz motion, both of which were extracted from the data of the same trial as the training data. The medians of the indices calculated using 5 repetitions of the above evaluation process were used for the analysis because the indices were sometimes very large owing to the instability of the neural network.
For each of the RMS error and prediction delay, we verified the statistical significance of the difference between the prediction using only the angle and that using both the angle and the EMG signals. As we found a linear relationship between the indices and Δt when Δt ≤ 0.4 s, verification was performed by the analysis of covariance (ANCOVA) with Δt as a covariate using the indices with Δt of 0.05 to 0.4. With the linear model represented by Eqs. (5) and (6), statistical tests were performed with the null hypothesis that the variables indicating the effects of the method on the intercept and slope were equal [Eqs. (7) and (8)  : Here, y is the RMS error or prediction delay, α is the common intercept, β is the common slope, α i is the effect of the method on the intercept for each condition [only angle (i = 1), both angle and EMG signals (i = 2)], β i is the effect of the method on the slope for each condition, and ε is an error term. Equation (6) is a constraint for uniquely determining the parameter values. A significance level of 5% was used. Since the analysis assumed that the error term was normally distributed, the indices obtained when the instable behavior of the neural network caused a very large median of the RMS errors were excluded from the analysis. Figure 3 shows two examples of the prediction using the neural network when it was applied to the movement with irregular cycles. In the case when the prediction lead time (Δt) is 0.1 s, the prediction value based only on the angle and that based on both the angle and the EMG signals showed similar results and both results well matched the ideal prediction, which is the time-shifted measured angle [ Fig. 3(a)]. In the case when Δt is 0.3 s, the prediction value based on both the angle and the EMG signals preceded that based only on the angle [ Fig. 3(b)]. The preceding time was larger on the onset of the faster movement (11-15 s) than in other movements. In the following, we evaluated the effects of EMG signals on the predictions using the simple cyclic movements with shorter cycles (about 0.5 Hz) and longer cycles (about 0.2 Hz). Figure 4(a) shows part of the prediction results obtained using the neural network in about 0.5 Hz motion. The top six panels show the measured angle, the ideal predicted angle, the predicted value based only on the angle, and the predicted value based on both the angle and the EMG signals when Δt is 0.1, ..., 0.6 s. The lower two panels show EMG signals at the same time. When Δt = 0.1 s, the predicted value based only on the angle and that based on both the angle and the EMG signals were almost the same, but when Δt ≥ 0.2 s, there were differences between the two cases. In these cases, when looking at the predicted value around the time (0.25, 1.5, 3, and 4 s) when the measured angle started to change, the prediction using only the angle changed rapidly. In contrast, the prediction using both the angle and the EMG signals output values close to the ideal prediction before the angle started to change. EMG signals changed 0.2-0.5 s ahead of the angle.

Results
On the other hand, as shown in Fig. 4(b), with about 0.2 Hz motion, the advantage of the prediction using both the angle and the EMG signals was unclear. In the cases of Δt ≤ 0.4 s, the  onset timings of the two predictions were similar to the ideal prediction. In the cases of Δt ≥ 0.5, the differences of the two predictions could be found, but they could not be distinguished from the general estimation error. Figure 5 shows the prediction performance of the neural network in the motion of about 0.5 Hz. The RMS error [ Fig. 5(a)] and the prediction delay [ Fig. 5(b)], which is the delay from  condition. In the range of Δt ≤ 0.4 s, both indices tended to increase linearly with Δt. In this range, the two indices in the majority of the trials and participants were smaller in the case of the prediction using both the angle and the EMG signals than in the case of the prediction using only the angle, and the difference remained constant or increase with Δt. When Δt = 0.3 s, the RMS errors were 5.3 ± 1.4° (mean ± standard deviation) for the prediction using only the angle and 4.6 ± 1.3° for that using both the angle and the EMG signals, and the prediction delays were 0.125 ± 0.045 s for the prediction using only the angle and 0.080 ± 0.032 s for that using both the angle and the EMG signals.
The results of ANCOVA for all trials of participants are also shown in Fig. 5. In the RMS errors, 7 of the 8 trials showed a significant difference in the intercept or slope of the linear model between the case using only the angle and that using both the angle and the EMG signals [ Fig. 5(a)]. This difference mainly shows smaller RMS errors in the case using both the angle and the EMG signals, except for that in Trial 2 by Participant 4. In the prediction delay, 5 of the 8 trials showed the significant difference in the intercept or slope of the linear model between the case using only the angle and that using both the angle and the EMG signals [ Fig. 5(b)]. This difference shows a shorter prediction delay in the case using both the angle and the EMG signals. In this movement, it was shown that the prediction performance in the case using both the angle and the EMG signals was significantly improved compared with that in the case using only the angle. Figure 6 shows the prediction performance of the neural network in the motion of about 0.2 Hz. In the RMS errors, the linear relationship between Δt and the index found in the motion of about 0.5 Hz was also found in the motion of about 0.2 Hz, but the advantage of the prediction using both the angle and the EMG signals could not be found clearly [ Fig. 6(a)]. In the prediction delay, the majority of the trials and participants showed that the delay was almost 0 s in the range of Δt ≤ 0.4 s [ Fig. 6(b)]. When Δt = 0.3 s, the RMS errors were 3.4 ± 1.2° for the prediction using only the angle and 3.5 ± 1.9° for that using both the angle and the EMG signals, and the prediction delays were 0.045 ± 0.069 s for the prediction using only the angle and 0.028 ± 0.062 s for that using both the angle and the EMG signals. Note that the negative values seen in the prediction delay [ Fig. 6(b)] were accompanied by a large estimation error as seen in Fig.  4(b).
The results of ANCOVA in the case of the motion of about 0.2 Hz are also shown in Fig. 6. In the RMS errors, 6 of the 8 trials showed a significant difference in the intercept or slope of the linear model between the case using only the angle and that using both the angle and the EMG signals. 4 of the 6 trials showed smaller RMS errors in the case using both the angle and the EMG signals, but the other trials showed larger RMS errors in that case [ Fig. 5(a)]. In the prediction delay, only 1 of the 8 trials showed a significant difference in the intercept of the linear model, which showed a shorter prediction delay in the case using both the angle and the EMG signals, but the other trials showed no significant difference between the two predictions [ Fig. 5(b)]. In this movement, the advantage of the prediction using both the angle and the EMG signals was not shown.
When Δt > 0.4 s, the trends found in the range of Δt ≤ 0.4 s changed in different ways for each trial. When Δt = 0.5 s, there were no significant differences in the RMS error and the prediction delay between the case using only the angle and that using both the angle and the EMG signals in both the motion of about 0.5 Hz (paired t-test: RMS error: p = 0.13; prediction delay: p = 0.64) and that of about 0.2 Hz (RMS error: p = 0.42; prediction delay: p = 0.61). From these results, it is suggested that the range of Δt ≤ 0.4 s is the range wherein the EMG signals have the consistent effect on the prediction in the domain covered by this study.
Finally, we investigated the possibility that the EMG signals were dominant compared with the angle in the prediction using both the angle and the EMG signals. We prepared a recurrent neural network using only EMG signals for the prediction of the angles as shown in Fig. 7(a). Using the data used in Fig. 3, we trained the neural network in the same way as the other neural networks used in this study. Then, we compared the prediction based on the EMG signals with that based on both the angle and the EMG signals. Figures 7(b) and 7(c) show the results in which the prediction using both the angle and the EMG signals is the same as that in Fig. 3. In Fig. 7(b), which shows the result when Δt = 0.1 s, the prediction using only the EMG signals has biases at around 11 and 13 s. In Fig. 7(c), which shows the result when Δt = 0.3 s, the estimation error of the prediction using only the EMG signals was larger than that using both the angle and the EMG signals. From these results, it is suggested that the advantage of the prediction using both the angle and the EMG signals was yielded by the combination of the angle and the EMG signals.

Discussion
In the prediction of the angle of the thigh using the recurrent neural network, when the motion was about 0.5 Hz, the prediction using the EMG signals tended to show less error and  prediction delay than that using only the past angle acquired by the motion capture. This would be because in the case of about 0.5 Hz motion, the activity of the EMG signals changed about 0.2 to 0.5 s ahead of the change in joint angle (Fig. 4) and provided information on future angle changes. The phenomenon in which the EMG signals precede the angle would be because the rectified and filtered EMG signals are related to the muscle tension at the same time, (15,16) and the muscle tension causes the movement of the body part through the musculoskeletal system, which is a spring-mass-damper system. The reason why the prediction delay of around 0 s was observed in the motion of about 0.2 Hz is that the phase delay of the musculoskeletal system is small when the movement frequency is low, and the time difference between the changes in EMG signals and joint angle became small as seen in Fig. 4(b).
In this study, we focused only on the case of a single joint movement, and we did not use information on the coordinated movement of many joints, which can be obtained by motion capture. (18,19) Therefore, it is unclear whether there is an effect of using EMG signals in cases with whole body movements, such as in conventional studies. (5,6,8,9) In addition, in this study, only the EMG signals of the flexor and extensor muscles of the hip joint were used, and information on muscle synergy, (20,21) which is a coordination among many muscles including other joints, was not used. The verification of the effect of EMG signals on motion prediction using information on more joints and muscles is a future issue.

Conclusions
In this paper, we showed that in the single joint motion prediction of a thigh, the motion prediction using both the angle and the EMG signals tends to reduce the prediction error and delay compared with that using only the angle depending on the motion frequency. This indicates the possibility that operability and stability can be improved by predicting the movement of the operator by using the EMG signals of the operator when the remote operation of the robot has a delay. Future issues include examining the significance of the use of EMG signals in motion prediction using multiple joints and EMG signals.