With the emergence of the phenomenon of social aging, the elderly have frequent physical movement disorders. In particular, the movement disorder of the ankle joint seriously affects the daily life of the elderly. Rehabilitation robots are of great significance for improving the efficiency of rehabilitation, ensuring the quality of rehabilitation, and reducing the labor intensity of workers. As an auxiliary treatment tool, rehabilitation robots should have rich and effective motion modes. The exercise mode should be adaptable for patients with different conditions and different recovery periods. To improve the accuracy of human-computer interaction of ankle joint rehabilitation robots (AJRR), this study proposes a man-machine collaboration model of an EEG-driven AJRR. The model mainly expands from two levels (1) to establish the connection between EEG and intention so as to identify the intention. In the recognition process, first feature extraction is given on the preprocessed EEG. Convolutional neural network (CNN) is selected to extract the deep features of the EEG signal, and support vector machine (SVM) is used for classifying the deep features, thereby realizing intent recognition. (2) The result of intention recognition is input to the human-computer interaction (HCI) system, which controls the movement of the rehabilitation robot after receiving the instruction. This study truly realizes patient-oriented rehabilitation training. Experiments show that the human-machine collaboration model used can show higher accuracy of intention recognition, thereby increasing the satisfaction of using AJRR.
The rapid rise and popularization of robotics technology have made robots no longer limited to manufacturing, but they are also developing towards the field of medical services [
AJRR is mainly used to assist patients with ankle joint injuries in intelligent rehabilitation training. At the end of the 20th century, rehabilitation robots began to enter the practical stage, and they had already appeared in a few hospitals. The current research on rehabilitation robot technology is mainly on human prosthetics, surgical robots, rehabilitation wheelchairs, rehabilitation training robots, and so forth. Rehabilitation medicine research shows that appropriate and reasonable exercise is an essential rehabilitation process after joint injury [
High-quality rehabilitation training should be actively participated by patients. To improve the quality of rehabilitation training, this study proposes a man-machine collaboration model of an AJRR driven by EEG signals. The specific work is as follows: In order to establish a precise connection between EEG and intent, a deep feature method is used to model and analyze EEG signals. The relationship between the EEG signal and the intention is established to realize the identification of the intention. A human-computer interaction system was developed. The result of intention recognition is input to the interactive system to control the rehabilitation robot to perform rehabilitation training for the patient. Experiments verify the effectiveness of the man-machine collaboration model designed in this paper. Since the intention is accurately recognized, the patient's training instructions can be accurately sent to the rehabilitation robot. Thus, the patient-oriented ankle rehabilitation training is truly realized.
At present, AJRR can be divided into traditional pedestal type [
The development of robot motion control theory can be summarized into three stages: classic control theory, modern control theory, and advanced control theory. In the classical control theory, the Laplace transform is used as the mathematical basis. This theory mainly analyzes the motion characteristics of the system in the time domain and frequency domain. The classical control theory represented by PID is relatively famous. However, it is difficult to obtain a more satisfactory control effect for systems with nonlinear and strong multivariable coupling characteristics. The determination of the controller parameters depends on the debugger. Modern control theory is based on the state space method. For the purpose of optimal control in the time domain, the control system is analyzed and designed by describing the internal state variables of the system. Advanced control theory is used to deal with some tasks, where conventional control strategies cannot achieve satisfactory control results. It has great advantages in improving the stability and robustness of the control system.
In the process of recognizing the patient's intention, deep feature extraction of the collected EEG is required. CNN has stronger advantages than machine learning algorithms [
CNN's multilayer architecture.
CNN’s architecture is relatively fixed and consists of three parts. The first part is the input layer. The combination of multiple convolutional layers and pooling layers forms the second part of the CNN, and the third part consists of a fully connected classifier. With the structure shown in Figure
System frame diagram.
The training process of CNN mainly has two stages: forward propagation of signal and backward propagation of error. The first stage is the forward propagation of the signal. In this stage, the input layer accepts the original information, undergoes step-by-step transformation, and finally transmits it to the output layer. The second stage is backward propagation, which is achieved by adjusting the size of the weights using the error back-propagation algorithm.
Based on the results of intent recognition, this study designed an EEG-driven AJRR man-machine collaboration model. The subject sits on a chair and puts one foot on the AJRR. The intention is to send instructions to the robot, and the robot generates corresponding angle changes according to the received control instructions and finally completes the rehabilitation training. In this article, the brain-controlled AJRR system based on motor imagery is mainly divided into software and hardware. The hardware part includes signal acquisition module and ankle robot control module. The software is divided into signal processing module and rehabilitation training software module. The system structure is shown in Figure
The working principle of the system is as follows: The signal acquisition module is responsible for collecting the EEG electrical signal of the subject, filtering, and amplifying. Send the collected data to the rehabilitation training software in real time. The training data collection function in the rehabilitation training software will save it and use the saved data as training samples. The rehabilitation training software will call the signal processing module to analyze the real-time collected EEG signals and return the processing results. The rehabilitation training software will send corresponding instructions to the robot control module based on the results. The robot first parses the received instructions into angle information for controlling the rotation of the robot. The completion of the subject’s motion imagination task corresponds to the motion of the ankle robot so as to realize the purpose of autonomous rehabilitation of the entire robot according to the patient’s brain.
The movement path of the structural parts of the robot is set according to various specific requirements of the rehabilitation training task. According to the designed motion path corresponding to the control instruction that needs to be sent, the subject performs the corresponding motion imagination task according to the instruction to be sent. In the course of the experiment, after the subject's motor image signal is processed and the result is consistent with the preset instruction, the rehabilitation training software will send the corresponding instruction. After receiving the control instructions sent by the rehabilitation training software, first set the angle that the robot needs to rotate, and then drive the robot to drive the ankle to perform corresponding rehabilitation training. The subjects imagined their corresponding motion control intentions according to the preset instruction sequence in the training task. Then the signal processing classification results are compared with the above-mentioned instruction sequence. If the two are consistent, the instruction is sent to the robot control module. If they are inconsistent, the subjects need to perform the motor imaging task again.
Different human conscious activities will produce corresponding changes in EEG signals. This special effect is often used to recognize human intentions [
Flow chart of EEG-based intention recognition.
In this paper, the EEG generated by the subject’s motion imagination is used as the input of the interactive system. The EEG signals of different modes are generated by performing various motion imaging tasks and then converted into external actions. The final application of the interactive system is to control the external devices, and the core part of the control is the EEG signal processing algorithm. The signal processing process is divided into three parts, namely, signal preprocessing, feature extraction, and classification. The EEG signal extracted from the brain scalp is analyzed by the signal processing module and converted into control commands for external devices.
Use Ag/AgC1 electrode to record the EEG signal on the scalp. The collected data mainly include four kinds of motion imagination data of hands and feet. When collecting, the subject operated according to the prompts. The time to collect data is 11 seconds, a countdown of 3, 2, and 1 appears on the computer screen, and then the data collection officially starts. The subjects were imagined to control their hands and feet according to the four arrows in different directions that appeared on the computer screen. The arrow appears and disappears after 2 seconds. The subjects performed motor imagination on their own. After 3 seconds, the word "End" appears, ending the motor imagination. The subject rested for 3 seconds. According to this collection process, a total of 500 data sets of 10 subjects were collected.
Due to technical limitations of the acquisition equipment, it is inevitable that there will be some noise signals in the process of acquiring EEG signals, such as cerebral cortex electromyography artifacts, eye movement and blinking artifacts, and other types of interference. Among these interference signals, the electromyographic artifacts in the cerebral cortex can be directly filtered out because of the low frequency characteristics. Electrooculogram artifacts and power frequency interference are the main interference factors when collecting EEG signals. In order to facilitate subsequent signal processing, the original EEG signal needs to be preprocessed. For artifact interference, filtering can be used for processing. Specifically, filter out the 10-25 Hz filtering in the EEG signal.
CNN can automatically collect and classify input data, thereby reducing the influence of human factors. When the sample size is larger, the advantages of CNN can be used. However, in the multiclass motor imaging EEG classification task, due to the small training sample size, CNN cannot be fully trained, which will lead to overfitting and affect the final classification effect. To improve the final recognition accuracy, CNN is used for deep feature extraction, principal component analysis (PCA) is used for dimensionality reduction processing on the extracted feature values, and SVM is used for dimensionality reduction feature classification. Figure
In-depth feature extraction and classification process.
The process of deep feature extraction is as follows: In Input layer, the input data is In the convolutional layer, where In the pooling layer, average pooling is used to downsample each map of the convolutional layer. The calculation formula of each map on the pooling layer is as follows: where In the fully connected layer, a mapping matrix where The classification result of the EEG signal is output in the output layer. The error is propagated back by the BP algorithm, thereby updating the parameters of the CNN network. The calculation formula for the value of each neuron where
The classification steps are as follows.
The EEG data set
Select the eigenvectors to form the projection matrix
The classification steps of the sample set Step 1: The feature Step 2: Step 3: According to equation ( Step 4: Input
To verify the classification effect of the method used in this article on EEG, the comparison algorithm has support vector machine (SVM) [
Parameter settings.
Method | Parameter settings |
---|---|
SVM | Penalty coefficient |
Reference [ | Parameter settings are consistent with [ |
TSK | Number of rules |
ELM | Neuron number range |
Our method | Number of convolution kernels |
The evaluation index is
To study the sensitivity of our method to parameters, with the different values of parameter
Comparison of
Number of convolution kernels | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
1 | 0.8536 | 0.8212 | 0.8456 | 0.8004 | 0.8232 | 0.8421 | 0.8263 | 0.8115 | 0.8003 | 0.7967 | 0.8221 |
2 | 0.8489 | 0.8183 | 0.8537 | 0.8064 | 0.8301 | 0.8484 | 0.8318 | 0.8123 | 0.8069 | 0.7993 | 0.8256 |
3 | 0.8339 | 0.8121 | 0.8538 | 0.8012 | 0.8301 | 0.8467 | 0.8305 | 0.8198 | 0.8097 | 0.8002 | 0.8238 |
4 | 0.8562 | 0.8232 | 0.8642 | 0.8117 | 0.8332 | 0.8532 | 0.84220 | 0.8226 | 0.8118 | 0.8041 | 0.8322 |
5 | 0.8502 | 0.8134 | 0.8573 | 0.8101 | 0.8252 | 0.8478 | 0.8327 | 0.8025 | 0.8048 | 0.7910 | 0.8235 |
To compare the superiority of our method, the EEG data of 10 subjects were randomly selected for experiment. The experimental results are shown in Table
Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | Mean |
---|---|---|---|---|---|---|---|---|---|---|---|
SVM | 0.7832 | 0.7899 | 0.7932 | 0.7872 | 0.7742 | 0.7856 | 0.7880 | 0.7923 | 0.7945 | 0.7889 | 0.7877 |
Reference [ | 0.8008 | 0.8032 | 0.7997 | 0.8256 | 0.8102 | 0.8005 | 0.7990 | 0.8025 | 0.8287 | 0.8110 | 0.8060 |
TSK | 0.7789 | 0.8012 | 0.7927 | 0.7897 | 0.8103 | 0.7932 | 0.7790 | 0.8121 | 0.7929 | 0.7978 | 0.7948 |
ELM | 0.7895 | 0.7996 | 0.8011 | 0.7891 | 0.7856 | 0.8002 | 0.7910 | 0.8092 | 0.8124 | 0.7953 | 0.7973 |
Our method | 0.8562 | 0.8232 | 0.8642 | 0.8117 | 0.8332 | 0.8532 | 0.84220 | 0.8226 | 0.8118 | 0.8041 | 0.8322 |
The table shows the classification effect of EEG collected from 10 subjects. On the whole, the classification effects of SVM, TSK, and ELM are very poor, and the accuracy rate does not reach 80%. Reference [
It is easy to mix in noise data during EEG acquisition. Therefore, the method of processing EEG data must have good noise immunity. In order to verify the antinoise performance of our method, here we add 1%, 3%, 5%, 7%, and 9% Gaussian noise to the original data. After adding noise, the experimental results are shown in Table
Performance comparison of each method after adding noise.
Noise\method | SVM | Reference [ | TSK | ELM | Our method |
---|---|---|---|---|---|
1% | 0.7803 | 0.7967 | 0.7902 | 0.7892 | 0.8231 |
3% | 0.7765 | 0.7923 | 0.7863 | 0.7735 | 0.8173 |
5% | 0.7669 | 0.7864 | 0.7754 | 0.7643 | 0.8008 |
7% | 0.7621 | 0.7793 | 0.7671 | 0.7520 | 0.7967 |
9% | 0.7437 | 0.7588 | 0.7583 | 0.7429 | 0.7848 |
It can be analyzed from the data in Table
Ankle dyskinesia restricts people’s movement and brings about troubles to daily life. The methods and techniques of ankle rehabilitation are very important for patients. The AJRR can assist patients in effective ankle joint rehabilitation training. This research designed a more accurate human-computer interaction model of AJRR. This model can effectively control the robot according to the patient's intention. The human-computer interaction of the system is more convenient and efficient. The realization of this study mainly has the following two aspects. One is the effective recognition of exercise intention. For the recognition step, the convolutional neural network (CNN) is used to extract the deep features of the brain wave signal, and the support vector machine (SVM) is used to classify the above features to identify the intention of the machine user. The other is to send instructions to the rehabilitation robot in the form of instructions based on the recognition results. The robot moves in different directions and angles according to the received instructions so as to truly realize patient-oriented rehabilitation training. The human-computer interaction model designed in this research can effectively identify the patient's intention and successfully control the movement of the robot and has a good market application prospect. However, because the extraction of depth features takes a lot of time, the real-time performance of the system needs to be further improved. This is also the direction to be optimized in the next step of this study.
The labeled data used to support the findings of this study are available from the corresponding author upon request.
The authors declare no conflicts of interest.
This work was supported by the Research Team Fund of Fuzhou University of International Studies and Trade in 2018, “Interactive Design” (no. 2018KYTD-08).