A novel hybrid brain-computer interface (BCI) based on the electroencephalogram (EEG) signal which consists of a motor imagery- (MI-) based online interactive brain-controlled switch, “teeth clenching” state detector, and a steady-state visual evoked potential- (SSVEP-) based BCI was proposed to provide multidimensional BCI control. MI-based BCI was used as single-pole double throw brain switch (SPDTBS). By combining the SPDTBS with 4-class SSEVP-based BCI, movement of robotic arm was controlled in three-dimensional (3D) space. In addition, muscle artifact (EMG) of “teeth clenching” condition recorded from EEG signal was detected and employed as interrupter, which can initialize the statement of SPDTBS. Real-time writing task was implemented to verify the reliability of the proposed noninvasive hybrid EEG-EMG-BCI. Eight subjects participated in this study and succeeded to manipulate a robotic arm in 3D space to write some English letters. The mean decoding accuracy of writing task was
Brain-computer interface (BCI) offers a direct communication channel between the brain and external devices without relying on the brain normal output pathways of peripheral nerves and muscles [
Nowadays, there are huge opportunity and necessity for helping handicapped people to enhance or increase their abilities to interact with complex environment, such as rehabilitation training sessions, mind-controlled prosthetic arm applications [
Controlling a robotic arm with noninvasive hybrid BCIs surely provides a desirable alternative, but prior to this study it has not been shown that such hybrid systems could achieve multidimensional control of robotic arm in three-dimensional (3D) space. These systems offer a potentially effective control for complex and naturalistic environment through the combination of brain- and nonbrain-based multifunctional BCI. They can reduce user fatigue by switching from a modality to another and increase the degree of freedom for augmentative and alternative BCI systems. The aim of this study is to improve the performance of BCI system by design a new hybrid EEG-EMG-BCI system (i.e., combination of brain activity (MI and SSVEP) with muscles activity such as teeth clenching). In this paper, a hybrid BCI system was described, including motor imagery-based brain switch, “teeth clenching” state detector, and a steady-state visual evoked potential- (SSVEP-) based BCI. In our proposed hybrid BCI, motor imagery decoding was used as a single-pole double throw brain switch (SPDTBS) which can complete multitasks, combined with 4-class SSVEP-based BCI system. In addition, “stop” command was executed by recognizing facial action by recording EMG artifact from EEG signals. For real-time application, a writing task was implemented to verify the performances of our proposed hybrid system. Healthy subjects succeed to write an English word though our proposed hybrid noninvasive BCI system. In the following sections, we describe our proposed system in detail and its real-time writing application to enhance the user’s abilities to interact with a complex environment.
Our proposed hybrid noninvasive EEG-EMG-BCI system mainly consists of three hardware components which are a portable EEG acquisition device (Emotiv EPOC), a host computer, and a robotic arm (see Figure
Schematic architecture of the experimental setup for the real-time hybrid BCI-controlled robotic arm.
The hybrid BCI consists of MI-, EMG-, and SSVEP-based BCI systems. As shown in Figure
Flowchart of the proposed algorithm for hybrid EEG-EMG-BCI system.
According to each unilateral movement (right or left hand imagination), the SPDTBS was designed. The SPDTBS was combined with four tasks of SSVEP-based BCI modality to provide more commands (i.e., to achieve multidimensional BCI control) for the robotic arm movements such as the forward, backward, left, right, upward, and downward movements (see Figure
The control commands of hybrid BCI.
SPDTBS | SSVEP-based BCI frequency | Control command | |
---|---|---|---|
Brain activity based on imagined unilateral hand movements (motor imagery) and SSVEP | Left hand imagination | 6 Hz | Forward |
7.5 Hz | Backward | ||
8.57 Hz | Left | ||
10 Hz | Right | ||
Idle | No command | ||
Right hand imagination | 6 Hz | No function | |
7.5 Hz | No function | ||
8.57 Hz | Upward | ||
10 Hz | Downward | ||
Idle | No command | ||
|
|||
Muscles activity (EMG artifacts) | “Teeth clenching” state | Stop |
The six possible directions of the robotic arm. (a) Upward and downward movements. (b) Left and right movements. (c) Forward and backward movements.
Eight healthy subjects participated in the experiment (age
Essential steps for the robotic arm to write the word “HI” with the writing result of the robotic arm controlled by our proposed hybrid BCI in the right side.
The following points are used to evaluate the performance of the hybrid BCI system: Time: time required to complete a task. Step count: the number of steps to complete the task in paper. Obvious errors: number of obvious errors. For example, if the writing task is O letter but the result is Q, this result is defined as obvious error. Information transfer rate, which is defined as where
The EEG data were sampled at a frequency of 2048 Hz and then downsampled to 128 Hz for signal processing. The electrodes were placed at 10-20 system locations, AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4, as well as two reference electrodes located above the ears of the subject (i.e., either CMS and DRL or left/right mastoids). Electrodes P7, P8, O1, and O2 were selected to collect SSVEP-based EEG signal. Electrode FC5 and FC6 were used to capture EEG signals in MI. The signals during “teeth clenching” state were collected mainly by electrodes F7 and F8 (see Figure
Position of EEG electrodes used in this study for recording brain and nonbrain signals.
To reduce the effect of signal-to-noise ratio (SNR), discrete wavelet transform (DWT) was employed for preprocessing of EEG signals. Assuming that
EEG signals were decomposed in different layers (5 layers) by using Daubechies wavelet (db4) function and reconstructed by removing frequency components (0–2 Hz).
The Canonical Correlation Analysis (CCA) is a multivariable statistical method, which was used to analyze the potential correlation between two sets of data [
Suppose two multidimensional random variables
The reference signals
The control command
Each subject has different thresholds
The results of canonical correlation analysis coefficients for different SSVEP states.
SSVEP state | Mean ± SD |
---|---|
6 Hz |
|
7.5 Hz |
|
8.57 Hz |
|
10 Hz |
|
Idle |
|
The motor imagery classification based on mu frequency power has been widely used for processing event-related synchronization (ERS) and event-related desynchronization (ERD) [
Assuming a signal of length
In MI-based BCI experiment, while imagining the left hand or right hand movement, the EEG signals are collected with band-pass filtering (0–32 Hz), mu rhythm energy change of FC5 and FC6 was computed, and the energy difference between FC5 and FC6 channels is used to calculate the threshold
When
Accuracy of detecting “teeth clenching” state is higher than other facial states in EEG-based BCI system [
According to the characteristics of different states (“natural” versus “teeth clenching”), the threshold
In the first stage, an offline experiment was held for MI-based BCI and SSVEP-based BCI and “teeth clenching” state detector, respectively. As shown in Figure
Decoding accuracy of the hybrid BCI system.
Performances of writing task.
Eight subjects joined the writing task using a robotic arm. The results were shown in Figures
In this paper, a novel multichannel hybrid BCI system was proposed for multidimensional control purpose, which was composed of a motor-imagery-based brain switch, “teeth clenching” state detector, and SSVEP-based BCI. For achieving a multidimensional control of robotic arm, seven commands which can be up to nine were designed for real-time BCI application. Writing task was held to evaluate the performances of the proposed hybrid system. Eight subjects completed the movement tasks of the robotic arm to write the word “HI.”
Pfurtscheller et al. [
In this study, we found that a group of healthy subjects could willingly use brain and nonbrain activity to control a robotic arm with high accuracy for performing writing tasks requiring human intention, error feedback, and multiple degrees of freedom by combination of MI, SSVEP, and EMG activity (see Supplementary video in Supplementary Material available online at
This paper presented a combination of synchronous and asynchronous control using a novel hybrid EEG-EMG-based BCI which consists of motor imagery, muscle artifacts, and SSVEP to provide a multidimensional control. The synchronous control is based on SSVEP paradigm which requires the user to focus on the screen and the asynchronous control is based on the motor imagery which does not need any synchronization between the user and the screen because it is based on the imagination of the unilateral movements. Users were able to write an English word using our robust real-time control of a robotic arm through the proposed hybrid BCI. This proposed BCI was designed for multiclass control in a complex environment. Results of the study indicated that successful multidimensional control is possible using suitable combination of BCI modalities to detect and classify brain activity in different situations.
In the near future, for rehabilitation, e-learning, and entertainment, we would like to design low cost, portable, noninvasive, and hybrid EEG-EMG-based robotic arm using minimum number of wearable wireless sensors.
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This work was financially supported by Natural Science Foundation of Tianjin City (15JCYBJC51800), Tianjin Key Laboratory Foundation of Complex System Control Theory and Application (TJKL-CTACS-201702), and Young and Middle-Aged Innovation Talents Cultivation Plan of Higher Institutions in Tianjin (Grant no. 20130830).