Electroencephalogram- (EEG-) based brain-computer interface (BCI) systems usually utilize one type of changes in the dynamics of brain oscillations for control, such as event-related desynchronization/synchronization (ERD/ERS), steady state visual evoked potential (SSVEP), and P300 evoked potentials. There is a recent trend to detect more than one of these signals in one system to create a hybrid BCI. However, in this case, EEG data were always divided into groups and analyzed by the separate processing procedures. As a result, the interactive effects were ignored when different types of BCI tasks were executed simultaneously. In this work, we propose an improved tensor based multiclass multimodal scheme especially for hybrid BCI, in which EEG signals are denoted as multiway tensors, a nonredundant rank-one tensor decomposition model is proposed to obtain nonredundant tensor components, a weighted fisher criterion is designed to select multimodal discriminative patterns without ignoring the interactive effects, and support vector machine (SVM) is extended to multiclass classification. Experiment results suggest that the proposed scheme can not only identify the different changes in the dynamics of brain oscillations induced by different types of tasks but also capture the interactive effects of simultaneous tasks properly. Therefore, it has great potential use for hybrid BCI.
Brain-computer interface (BCI) system, also known as a brain-machine interface, is designed to translate human brain signals into commands to control an external device [
EEG-based BCI systems usually utilize changes in the dynamics of brain oscillations for control, such as event-related desynchronization/synchronization (ERD/ERS) with imagined movements, steady state visual evoked potential (SSVEP), P300 evoked potentials, and related components [
In EEG-based BCI, distinctive patterns induced by specific mental task can be identified and utilized for information transmission by the EEG classification algorithm. Effective and accurate feature extraction and classification are of paramount importance for the success of the BCI [
There is an increased interest to represent EEG data as a multiway array named tensor, and tensor decomposition can be applied to exploit the characteristics of data among multiple modes [
Therefore, in this work, we propose an improved tensor based multiclass multimodal analysis scheme especially for hybrid BCI, in which EEG signals of the hybrid tasks are denoted as multiway tensors, a nonredundant rank one tensor decomposition model is proposed to obtain nonredundant tensor components, a weighted Fisher criterion is designed to select multimodal discriminative patterns among hybrid tasks without ignoring the interactive effects of simultaneous tasks, and support vector machine (SVM) is extended to multiclass classification for hybrid tasks. Applications in three datasets suggest that the proposed scheme can not only identify the different changes in the dynamics of brain oscillations induced by different types of tasks, but also capture the interactive effects of simultaneous tasks properly.
The proposed tensor based multiclass multimodal analysis scheme for hybrid BCI is illustrated in Figure
Tensor based multiclass multimodal analysis for hybrid tasks.
Multichannel EEG signals can be added to the spectral modality by wavelet transform and yield a three-way tensor data (this step is conducted in the same way as proposed in our previous work [
Here,
Tensor decomposition can analyze multiway data without losing some potential information among modalities [
For each class of hybrid tasks, the assembled tensor
for set
( ( ( ( ( ( ( ( ( ( ( ( ( (
By the proposed nonredundant rank one tensor decomposition, the assembled tensors
Then those projection coefficients for each hybrid task are concatenated as the feature vectors.
As mentioned before, there is no algorithm focusing on the hybrid task classification. So far, EEG data of hybrid tasks are divided into individual groups and fed into separate processing procedures. In this case, the interactive effects on distinctive patterns are totally ignored when different types of tasks are executed simultaneously. Fisher score has been used to select discriminative features for binary classification in our previously proposed tensor based scheme [
Let
SVM [
In this scheme, a multiclass SVM method is applied for hybrid task classification. For
Three different types of EEG datasets collected in our hybrid BCI study experiment [
In this section, first, the proposed scheme is applied in dataset 1 and dataset 2 to confirm its efficiency in conventional BCI tasks, and then it is applied to dataset 3 to evaluate its performance in hybrid BCI tasks.
For each epoch in dataset 1, as described above, a three-way tensor was generated in multimodes of channel, time, and frequency. The frequency range was set to 5 Hz–30 Hz with 1 Hz spectral resolution and the time range was set to 1–4 s with 0.25 s temporal resolution. The assembled tensor for each MI task was calculated as described in Section
A visualization example of the assembled tensor for the left hand MI task. The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 4 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
A visualization example of the assembled tensor for the right hand MI task. The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 4 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
Figure
The multimodes of channel, time, and frequency of the two selected nonredundant rank one tensor components for MI tasks.
For SSVEP task classification, the continuous data in dataset 2 were segmented into 2 seconds’ epochs. For each epoch, similarly, a three-way tensor was generated in multimodes of channel, time, and frequency. The frequency range was set to 5 Hz–30 Hz with 1 Hz spectral resolution and the time range was set to 1-2 s with 0.25 s temporal resolution. Then the assembled tensor for each SSVEP task was calculated. Figures
A visualization example of the assembled tensor for a SSVEP task (focusing on the 7 Hz stimulus). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 2 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
A visualization example of the assembled tensor for a SSVEP task (focusing on the 8 Hz stimulus). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 2 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
A visualization example of the assembled tensor of a SSVEP task (focusing on the 9 Hz stimulus). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 2 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
A visualization example of the assembled tensor of a SSVEP task (focusing on the 11 Hz stimulus). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 2 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
Figure
The multimodes of channel, time, and frequency of the four selected nonredundant rank one tensor components for SSVEP tasks.
The classification results in those two different types of datasets were compared with two other algorithms in BCIs, that is, CSP and CCA. CSP and CCA are highly successful in classifying MI and SSVEP tasks, respectively, and they are also the most commonly used methods in hybrid MI and SSVEP BCIs [
Table
Classification results (%) for conventional BCI tasks.
Sub. 1 | Sub. 2 | Sub. 3 | Sub. 4 | Sub. 5 | Sub. 6 | Sub. 7 | Sub. 8 | Sub. 9 | Avg. | |
---|---|---|---|---|---|---|---|---|---|---|
CSP for MI |
|
69.2 |
|
|
|
41.7 | 91.7 |
|
|
|
TbMMS for MI | 72.1 |
|
99.1 | 98.7 | 50.0 |
|
|
83.6 | 64.2 | 76.3 |
CCA for SSVEP |
|
57.5 |
|
93.8 | 87.6 |
|
|
|
|
|
TbMMS for SSVEP | 92.1 |
|
97.4 |
|
|
94.0 | 94.3 | 89.8 | 77.3 | 87.9 |
TbMMS denotes the proposed tensor based multiclass multimodal analysis scheme.
Above results confirm that the multiway tensor representation in multidomain of time, frequency, and channel can identify different characteristic changes for different types of tasks, such as MI-based ERD and SSVEP. Moreover, the proposed scheme is efficient in extracting multimodal discriminative patterns for different types of conventional BCI tasks. To investigate its performance in hybrid task classification, we further apply the proposed scheme to dataset 3.
Dataset 3 contains EEG data collected in hybrid MI and SSVEP tasks. For each epoch, a three-way tensor was generated in the previous manner. The frequency range was set to 5 Hz–30 Hz with 1 Hz spectral resolution and the time range was set to 1–4 s with 0.25 s temporal resolution. The assembled tensor for each class of hybrid task was calculated according to the method described in Section
Figure
A visualization example of the assembled tensor for a hybrid of MI and SSVEP task (imagining left hand movements and focusing on the 7 Hz stimulus simultaneously). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 4 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
The multimodes of channel, time, and frequency of the two selected nonredundant rank one tensor components for the hybrid of MI and SSVEP task (imagining left hand and focusing on the 7 Hz stimulus simultaneously).
Figure
A visualization example of the assembled tensor for a hybrid of MI and SSVEP task (imagining left hand movements and focusing on the 8 Hz stimulus simultaneously). The spectrograms are shown at each channel according to channels distribution over scalp, with time ranging from 0 to 4 s and frequency ranging from 5 to 30 Hz. The spectrograms at C3, Oz, and C4 channels are enlarged in the bottom.
The multimodes of channel, time, and frequency of the three selected nonredundant rank one tensor components for the hybrid of MI and SSVEP task (imagining left hand and focusing on the 8 Hz stimulus simultaneously).
Those results show that there are some interactive effects when the different types of tasks are executed simultaneously and the proposed scheme could extract those patterns properly.
Table
Classification results (%) for the hybrid BCI tasks.
Sub. 1 | Sub. 2 | Sub. 3 | Sub. 4 | Sub. 5 | Sub. 6 | Sub. 7 | Sub. 8 | Sub. 9 | Avg. | |
---|---|---|---|---|---|---|---|---|---|---|
CSP for MI |
|
75.0 |
|
98.3 | 58.3 |
|
96.7 | 69.1 | 53.2 | 75.7 |
TbMMS for MI | 73.1 |
|
|
|
|
51.0 |
|
|
|
|
CCA for SSVEP | 82.0 |
|
97.5 | 90.3 | 90.0 |
|
98.0 | 81.3 | 63.9 | 84.1 |
TbMMS for SSVEP |
|
58.0 |
|
|
|
94.0 |
|
|
|
|
TbMMS denotes the proposed tensor based multiclass multimodal analysis scheme.
In this paper, we propose an improved tensor based multiclass multimodal scheme especially for EEG analysis in hybrid BCI. Compared to current signal analysis methods for hybrid BCI, EEG data need not to be divided into individual groups and fed into the separate processing procedures. In this scheme, owing to tensor representation on multimodes of channel, time, and frequency, different characteristics of EEG signals can be presented simultaneously. Moreover, a nonredundant rank one tensor decomposition algorithm is proposed to obtain nonredundant rank one tensor components, and a weighted Fisher criterion is designed to select multimodal discriminative patterns among hybrid tasks without ignoring the interactive effects of the simultaneous tasks. Finally, SVM is extended to multiclass classification for hybrid tasks. Applications in three datasets suggest that the proposed scheme can not only identify the different changes in the dynamics of brain oscillations induced by different types of tasks, but also capture the interactive effects of simultaneous tasks.
This work presents a novel method of EEG analysis for hybrid BCI by considering the interactive effect of the simultaneous tasks and demonstrates that it is helpful to improve the classification results for hybrid tasks. Although the proposed scheme is not suitable for online BCI because tensor generation and decomposition are very time-consuming, it is still very useful for developing hybrid BCI. It could help to learn the difference when two or more tasks are executed simultaneously rather than individually and obtain the multimodal information of the difference, including channel, time, and frequency. Taking advantage of the revealed multimodal information, some simple methods for online BCI could be improved and acquire better results. Therefore, the proposed scheme is a potential efficient tool in EEG analysis for hybrid BCI.
The authors declare that there is no conflict of interests regarding the publication of this paper.
The work was supported by the National Natural Science Foundation of China (Grant no. 61105122) and the Fundamental Research Funds for the Central Universities.