We present a methodology for a hybrid brain-computer interface (BCI) system, with the recognition of motor imagery (MI) based on EEG and blink EOG signals. We tested the BCI system in a 3D Tetris and an analogous 2D game playing environment. To enhance player’s BCI control ability, the study focused on feature extraction from EEG and control strategy supporting Game-BCI system operation. We compared the numerical differences between spatial features extracted with common spatial pattern (CSP) and the proposed multifeature extraction. To demonstrate the effectiveness of 3D game environment at enhancing player’s event-related desynchronization (ERD) and event-related synchronization (ERS) production ability, we set the 2D Screen Game as the comparison experiment. According to a series of statistical results, the group performing MI in the 3D Tetris environment showed more significant improvements in generating MI-associated ERD/ERS. Analysis results of game-score indicated that the players’ scores presented an obvious uptrend in 3D Tetris environment but did not show an obvious downward trend in 2D Screen Game. It suggested that the immersive and rich-control environment for MI would improve the associated mental imagery and enhance MI-based BCI skills.
Gamification is the application of game-design elements and game principles in nongame contexts [
Brain-computer interface (BCI) is a direct communication pathway between an enhanced or wired brain and an external device [
One of important factors improving the efficiency of MI-based BCI is the experiment paradigm, because the motivational experiment paradigms for MI provide more enlightenment and guidance for users to study neural control of movement. Allison et al. [
How can a BCI experimental paradigm be more attractive? Though games can provide strong motivation for practicing and achieving better control for users within a rehabilitation system, the amount of information interaction during gaming should be adjusted to a proper range. The idealized experimental environments would not only be attractive to players (to reduce distraction) but also enhance the performing efficiency of motor imagery and help inexperienced users. So experimental objectives should be the core design principles of experimental design; meanwhile, content and forms should be vivid and rich. Marshall et al. designed a system to encourage rapid generation of mental commands and enhance the user’s experience in motor imagery-based BCI [
Based on the reasons mentioned above, we conjectured that an immersive 3D game environment could promote characteristic brain state generation in the context of motor imagery. We implemented in a Game-BCI system for 3D Tetris game playing, which was a hybrid brain-computer interface (BCI) system, with the recognition of motor imagery based on EEG and blink EOG signals. A hybrid BCI system usually contained two or more types of BCI systems. And BCI system also could be combined with another system which is not BCI-based, for example, combining a BCI system with an electromyogram- (EMG-) based system. The research on hybrid BCI has been a mainstream research direction in BCI field. Many works [
The main content of paper can be divided into five parts. In Sections
Ten players (3 females and 7 males) without previous BCI experience participated in the experiment voluntarily. All these players were right-handed, and their mean age was 24.6 years with a standard deviation of 3.3 years. All these players were conducted in accordance with the highest ethical standards of Xi’an Jiaotong University and signed the declaration file to declare they volunteered for the research experiment.
We used the 40-channel NuAmps system (America, Neuroscan Co.) to acquire EEG and EOG data. The system collected and transformed data using the TCP/IP (Transmission Control Protocol/Internet Protocol) protocol. The sampling rate was 1000 Hz. EEG data was recorded from 25 scalp electrodes, placed as shown in Figure
Positions of 25-channel EEG electrodes on players’ scalps.
Before 3D Tetris game playing, all players went through a process of MI training. We familiarized them with the feeling of performing of four kinds of motor imagery. In the MI training phase, the participant sat in a comfortable armchair in front of a computer screen (Dell S2316 M LED monitor, maximum resolution: 1920 × 1080) for sixty centimeters. We instructed participants to imagine right hand, left hand, foot, and tongue movements corresponding to visual cues showed on the computer screen. Each trial began with a 2 sec interval in which the screen was blank. Then players took 4 secs to do motor imagery. The screen then was again blanked to begin the next trial. The flow of one single trial for MI training was showed in Figure
The flow of one single trial for MI training.
In the 3D Tetris experiment, we divided the 10 players into two equal groups: One group experienced the traditional asynchronous BCI paradigm and the other group experienced the 3D Tetris paradigm. The 3D Tetris procedure was a puzzle game that used a three-dimensional playing field, as opposed to the traditional two dimensional pattern mentioned in literature [
3D Tetris Scene.
During game playing, we used the names of standing planes to label the direction of motion of the block groups. In coordinates of block group, Foot Plane represents
In this research, the data processing showed in Figure
The illustration of data handling procedures.
In both offline calculation and online control, preprocessing steps included power frequency filtering, EOG extraction, and baseline correction of EEG. We used all EEG data collected in the MI training phrase in feature component extraction and algorithm training (classification and feature extraction). Trial data striping and feature component extraction only occurred in offline calculation.
Ten players participated in the MI training phrase. For each player, we collected 240 trials of EEG data, giving 60 trials for each kind of motor imagery. For each kind of motor imagery, we averagely separated the data of each player into 6 parts. Each part contained 10 trials EEG data related to given kind of motor imagery. For each trial of EEG data, we applied CAR spatial filtering to each of the 25 data channels firstly and then selected the data recorded after 4 seconds of the MI cue presentation. Chebyshev I Bandpass filters of order 10 were used for extracting multiband data, with the range from 0 Hz to 60 Hz and frequency band 2 Hz wide. Subsequently, the filtered data was separated into components labeled by frequency band and electrode.
We calculated the spectral power for each selected component and the average
In this investigation, we proposed a method of multifeature extraction. That procedure combined independent component analysis and common spatial patterns in a renovated mode.
In this method, we described the time pattern of the sources by a stationary autoregression model
The assumption which was important to the least squares estimation method used in linear regression analysis required residuals to have the statistic characteristics
Based on this equivalence relationship, the correlation among all independent components in the temporal model was measured with minimization of mutual information.
In order to compare the multifeature extraction to traditional CSP, we define two computation processes. First, we let the feature components be the processing objects of the CSP spatial filter directly. The spatial features obtained in this way are called cspW_Data. Second, we let the feature components go through the independent component analysis and then used CSP spatial filtering to process those independent components. The spatial features obtained with the method of multifeature extraction were called cspW_IC. By comparing the quantitative differences between spatial feature cspW_Data and cspW_IC, we tried to demonstrate the effectiveness of the method of multifeature extraction.
In this work, we used the small world neural network (SWNN), discussed in previous research [
During classifier training, we defined four 4-bit gray codes to stand for the four kinds of motor imagery. If the SWNN produced a 4-bit gray code different from the four desired ones, we defined this brain state as idle. There was no “idle” data collected in the MI training phase, but players would exhibit idle states during game playing. The features extracted from idle state data would not produce a 4-bit gray code to be one of the four predefined ones.
In the original 3D Tetris game, the coordinate system of the 3D space and the local coordinate system of the block group were predefined. So the BCI system just took advantage of the original definition of the coordinate systems to adjust the movement and rotation of the block groups. In the proposed control strategy, the BCI system recognized the player’s mental states (four kinds of motor imagery) and translated them into control commands. The correspondence between MI and control command was determined in the procedure of secondary development of 3D Tetris (Table
The correspondence between motor imagery, object control command, and game effect.
Motor imagery | Control command | 3D Tetris coordinate |
---|---|---|
Foot motion | Moving to Foot Plane | Positive |
Tongue motion | Moving to Tongue Plane | Negative |
Left hand motion | Moving to Left Plane | Positive |
Right hand motion | Moving to Right Plane | Negative |
In addition, two kinds of blink detected from EOG recordings yielded rotation commands for block group control. The block group could be rotated about the
State transition for movement and speed control.
Input | Current state | |||||
---|---|---|---|---|---|---|
Start/ |
|
|
|
|
N_B | |
Left |
|
|
|
|
||
Right |
|
|
|
|
||
Tongue |
|
|
|
|
||
Foot |
|
|
|
|
||
|
|
|
|
|
||
|
|
|
|
|
||
|
|
|
|
|
||
Touch |
|
|
|
|
|
|
Fallen | N_B | N_B | N_B | N_B | ||
Cross | Reset | Reset | Reset | Reset | Reset | Reset |
|
|
The block group descended at a constant speed in the 3D game space. Players used mentally generated control to move and rotate the block groups in two dimensions. During the BCI game,
We defined the alphabet
Through the preprocessing of motor imagery training data, we picked up the most suitable characteristic components for the classification of motor imagery described in Table
The frequencies and electrodes of all feature components.
Player | Electrode | Frequency | [ |
---|---|---|---|
Player 1 | Cz | 8–12 Hz | [0.49 ± 0.024] |
C3 | 12–16 Hz | [0.48 ± 0.032] | |
Fz | 14–16 Hz | [0.35 ± 0.03] | |
F4 | 20–22 Hz | [0.26 ± 0.022] | |
T7 | 24–26 Hz | [0.23 ± 0.032] | |
|
|||
Player 2 | C4 | 16–20 Hz | [0.49 ± 0.03] |
Cz | 20–24 Hz | [0.38 ± 0.025] | |
C3 | 24–26 Hz | [0.32 ± 0.031] | |
F4 | 10–12 Hz | [0.30 ± 0.042] | |
T3 | 24–28 Hz | [0.22 ± 0.02] | |
|
|||
Player 3 | C4 | 16–18 Hz | [0.43 ± 0.024] |
Cz | 20–24 Hz | [0.42 ± 0.04] | |
C3 | 26–28 Hz | [0.40 ± 0.048] | |
P3 | 18–22 Hz | [0.37 ± 0.01] | |
Pz | 10–18 Hz | [0.32 ± 0.024] | |
|
|||
Player 4 | C4 | 20–16 Hz | [0.49 ± 0.024] |
F3 | 12–10 Hz | [0.37 ± 0.01] | |
C3 | 20–22 Hz | [0.32 ± 0.01] | |
T3 | 22–26 Hz | [0.32 ± 0.022] | |
Cz | 14–16 Hz | [0.26 ± 0.024] | |
|
|||
Player 5 | Cz | 10–14 Hz | [0.58 ± 0.062] |
F3 | 18–22 Hz | [0.37 ± 0.050] | |
C4 | 20–24 Hz | [0.37 ± 0.075] | |
T7 | 8–14 Hz | [0.34 ± 0.700] | |
C3 | 10–14 Hz | [0.21 ± 0.062] | |
|
|||
Player 6 |
|
12–16 Hz | [0.47 ± 0.022] |
|
20–24 Hz | [0.36 ± 0.032] | |
|
24–26 Hz | [0.36 ± 0.059] | |
|
8–16 Hz | [0.35 ± 0.03] | |
|
22–24 Hz | [0.3 ± 0.042] | |
|
|||
Player 7 |
|
10–12 Hz | [0.52 ± 0.062] |
|
20–26 Hz | [0.44 ± 0.070] | |
|
22–24 Hz | [0.33 ± 0.055] | |
|
10–14 Hz | [0.31 ± 0.700] | |
|
10–12 Hz | [0.28 ± 0.062] | |
|
|||
Player 8 |
|
16–22 Hz | [0.49 ± 0.03] |
|
20–24 Hz | [0.48 ± 0.042] | |
|
20–24 Hz | [0.44 ± 0.032] | |
|
16–22 Hz | [0.44 ± 0.031] | |
|
10–18 Hz | [0.37 ± 0.05] | |
|
|||
Player 9 |
|
18–24 Hz | [0.55 ± 0.03] |
|
22–28 Hz | [0.52 ± 0.01] | |
|
24–28 Hz | [0.38 ± 0.032] | |
|
18–22 Hz | [0.42 ± 0.03] | |
|
22–26 Hz | [0.33 ± 0.01] | |
|
|||
Player 10 |
|
10–18 Hz | [0.43 ± 0.024] |
|
18–22 Hz | [0.42 ± 0.04] | |
|
24–28 Hz | [0.41 ± 0.048] | |
|
26–28 Hz | [0.32 ± 0.01] | |
|
10–14 Hz | [0.32 ± 0.024] |
We took Player 1 as example to interpret the output of the verification program (Figure
Comparisons of results from cspW_Data and cspW_IC. The upper left part is the frequency domain relief topographic map (FDRM) of feature components relevant to the motor imagery of foot (MI_F) and left hand (MI_L). The upper right part is the frequency domain relief map of independent components relevant to the motor imagery of foot and left hand.
The CSP spatial filters trained from two kinds of components were called cspW_Data and cspW_IC, respectively. The lower left part of Figure
To verify the effectiveness of EEG features extracted by multifeature extraction, we compared the performances on EEG data for each player among SWNN, RBF neural network, BP neural network, and least squares support vector machines (LS-SVM) techniques. The average accuracy or error rate was over 10 runs of the 10 × 10-fold cross-validation procedure. We implemented the LS-SVM multiclass with one versus one decomposition strategy, using MATLAB (ver. 7.7, R2009b) using the LS-SVMlab toolbox (Version 1.8). The details about parameter setting for these three algorithms and algorithm toolboxes using are in the literature (Table
The mean accuracy of classification from four classifiers based
SWNN (mean) | RBF (mean) | BP (mean) | LS-SVM (mean) | |||||
---|---|---|---|---|---|---|---|---|
cspW_Data | cspW_IC | cspW_Data | cspW_IC | cspW_Data | cspW_IC | cspW_Data | cspW_IC | |
Player 1 | 87.10 | 86.6 | 78.61 | 85.2 | 82.74 | 80.6 | 68.37 | 72.0 |
Player 2 | 79.66 | 82.9 | 72.11 | 74.72 | 75.90 | 77.5 | 71.64 | 68.0 |
Player 3 | 65.29 | 74.0 | 83.67 | 76.1 | 62.37 | 72.8 | 67.20 | 72.2 |
Player 4 | 76.40 | 76.4 | 66.81 | 67.51 | 59.31 | 71.2 | 71.59 | 70.4 |
Player 5 | 60.80 | 63.6 | 59.72 | 53.92 | 61.54 | 63.3 | 58.20 | 59.4 |
Player 6 | 74.60 | 78.5 | 66.27 | 77.2 | 54.87 | 74.6 | 62.81 | 67.5 |
Player 7 | 56.30 | 76.3 | 49.52 | 74.97 | 72.10 | 69.6 | 52.61 | 60.1 |
Player 8 | 66.94 | 81.3 | 49.83 | 79.30 | 53.30 | 72.8 | 57.22 | 62.0 |
Player 9 | 72.13 | 77.45 | 65.81 | 73.62 | 65.26 | 77.3 | 63.70 | 68.95 |
Player 10 | 71.16 | 83.6 | 50.6 | 82.0 | 57.0 | 75.1 | 59.77 | 74.7 |
Mean | 71.03 |
|
64.3 |
|
64.4 |
|
63.3 |
|
|
|
|
|
|
The classification results from four classifiers indicated that cspW_IC produced more quality features than cspW_Data. To investigate the statistical significance of the accuracies, we performed an analysis of variance (ANOVA) on each player’s result based on all classification accuracies (10 runs of the 10 × 10-fold cross-validation procedure). The
In the control task, ten players were divided into two equal sized groups. One group (Group S) experienced the traditional asynchronous BCI paradigm. The other group (Group 3D) experienced the 3D Tetris paradigm. Group S contained Player 1 (S1), Player 2 (S2), Player 3 (S3), Player 4 (S4), and Player 5 (S5). Group 3D contained Player 6 (3D_1), Player 7 (3D_2), Player 8 (3D_3), Player 9 (3D_4), and Player 10 (3D_5). All players went through the given paradigm for 10 runs in one day. The control task lasted ten days.
For Game-BCI 3D Tetris, the rules and mechanisms were described in Sections
The traditional asynchronous BCI paradigm used as contrast experiment in this paper was called the Screen Game; it ran in a 2D environment (Figure
Screen Game Scene.
Just as prior knowledge of the physiological processes underlying motor imagery does, hand motor imagery will stimulate the electroactivities focusing on contralateral regions over the motor cortex area containing Mu or Beta event-related desynchronization (ERD) and ipsilateral event-related synchronization (ERS) activity. Both ERD and ERS patterns localizing in the midcentral or parietal area are significant for the foot motor imagery. Otherwise, only ERS activity in this area is sufficiently dominant for tongue motor imagery [
ERD/ERS produced by players in the two games used in the experiment across 10 test days.
In Figure
We performed a 2 (groups: Group S, Group 3D) × 10 (test days) two-way ANOVA, with repeated measures over day, on these quantitative differences. The main effect for days was significant,
In order to investigate the impact of individual variability on the effect of ERD/ERS, we applied Welch’s
In this research, though 3D Tetris brought the entirely different operating experiences to players compared to 2D Screen Game and a lot of incomparable elements existed between these two BCI paradigms, they all were the method to test the player’s spontaneous ERD/ERS production ability.
In the 3D Tetris Game-BCI, the score represented the number of layers of disappearing Block-heaps. So a higher score represented a better ability to control the block objects using mind control. From training day 1 to day 4, players’ scores did not show an upward trend,
Distribution of players’ scores from training day 1 to day 10 in 3D Tetris Game-BCI.
So we separated the 10 training days into two stages: Stage I (S_I) covered from day 1 to day 4 and Stage II (S_II) covered from day 5 to day 10. The details of the 3D Tetris Game-BCI experiment were described in Table
The details of the 3D Tetris Game-BCI experiment.
3D_1 | 3D_2 | 3D_3 | 3D_4 | 3D_5 | ||||||
---|---|---|---|---|---|---|---|---|---|---|
S_I | S_ II | S_I | S_II | S_I | S_II | S_I | S_II | S_I | S_II | |
Number of right hand MI | 52 | 76 | 32 | 83 | 89 | 173 | 87 | 183 | 72 | 176 |
Number of left hand MI | 41 | 33 | 25 | 95 | 82 | 116 | 95 | 106 | 68 | 188 |
Number of Foots MI | 38 | 44 | 47 | 66 | 71 | 105 | 114 | 127 | 92 | 109 |
Number of Tongue MI | 21 | 35 | 22 | 56 | 79 | 119 | 73 | 98 | 64 | 124 |
Single blink EOG | 33 | 40 | 30 | 46 | 36 | 70 | 52 | 62 | 42 | 77 |
Double blink EOG | 47 | 49 | 26 | 34 | 18 | 26 | 18 | 19 | 12 | 21 |
Number of Block | 31 | 96 | 48 | 102 | 51 | 132 | 74 | 101 | 42 | 94 |
Mean Duration of a run | 477 s | 1440 s | 720 s | 1530 s | 754 s | 1980 s | 1260 s | 1710 s | 630 s | 1880 s |
For the 2D Screen Game, the player’s mission was to balance numbers relevant to different motor imagery categories. The score was the standard deviation of these four percentage numbers, which meant that a lower score represented better ability to generate motor imagery. However, from training day 1 to day 10, players’ scores did not show an obvious downward trend,
Distribution of players’ scores from training day 1 to day 10 in Screen Game.
In this study, we have shown that the combination of video game and BCI is a new design approach to enhance the stimulation and feedback ability of BCI systems. We implemented a Game-BCI system for 3D Tetris game playing with motor imagery indicated by EEG and blink EOG elements. We proposed and tested two key techniques, multifeature extraction and shared control, for enhancing player’s BCI control ability, to demonstrate the feasibility that 3D game environment could enhance player’s spontaneous ERD/ERS production ability. Taking the 2D Screen Game as a contrast, we compared the quantitative differences between spatial features extracted from motor imagery EEG collected in two experiments separately. The results of the analysis of ERD/ERS and game scores both suggested that an immersive and rich-control environment would improve user’s MI ability and BCI control skills.
The method of multifeature extraction, combining independent component analysis and common spatial patterns, is a renovated mode for EEG feature extraction. Independent component analysis (ICA) is a standard tool for data analysis in the area of neural networks and signal processing. The typical application is blind source separation of EEG signals. In raw EEG signals, there are electrooculograms, electromyography, and other artifacts, as well as mutual interferences between signals. The most direct phenomenon is the submergence of small power components exported from other leads, when there is a large power component from a given lead. Extraction via decorrelation of independent components in a multilead time domain with mixed signals could help indicate the energy distribution of each independent component during a certain period or a special cerebral state. The identification of temporal independence is one part of EEG signal processing. Spatial features illustrate EEG expressions of various mental tasks from the perspective of time-varying features of signal energy in the whole brain. In this way, unlike the extraction of time domain features, the spatial domain emphasizes spatial correlations among original signals or among certain types of components. Instead of merely analyzing energy features of a single channel EEG signal, the algorithm considering frequency spectrum variation correlations between different channels facilitates the creation of connections between EEG feature distribution and complex mental tasks. The common spatial pattern method (CSP), based on the theory of matrix simultaneous diagonalization, involves searching for a set of spatial filters, under the effects of which the variance of one type of signal reaches a maximum and the other type of signal reaches a minimum, thereby achieving classification. Because the EEG variance within a specific frequency band is related to its signal energy, the common spatial pattern method was able to achieve optimal classification of EEG signals based on waveband energy features.
In this study, we applied a time model-based residual mutual information minimization independent source signal extraction method based on artifact elimination and characteristic component extraction of EEG signal of limb motor imagery. This method reduces the correlations components under conditions of preserving temporal structures of EEG signals and so provides clear observation of signal characteristics of each component.
To validate the efficiency of multifeature extraction, two computation processes were derived. The spatial filter cspW_Data was trained with feature components. After multifeature extraction, the spatial filter trained with independent components was called cspW_IC. The results of spatial filtering demonstrated that, compared to cspW_Data, cspW_IC could produce more prominent quantitative differences between spatial features extracted from different motor imagery signals.
In this research, as a means to assess the utility of the MI control methodology we developed, we integrated BCI design into a 3D Tetris game. The goal was to improve the function in motor imagery training of the BCI system. This attempt follows the idea of gamification for rehabilitation highly respected frontiers. Studies under this new concept, which wants to gamify the process of rehabilitation, have gained wider attention in the rehabilitation field. For example, the Wellapets video game helps teach children how to manage asthma [
Rehabilitation is complex. It involves an ever-changing interaction of the rehabilitation patient with different clinical environments and healthcare providers. It has gone beyond simply creating a fun and exciting application in which to complete rehabilitation exercises and interventions. A delicate balance of the task and the patient’s abilities must be achieved. For BCI systems, the created system should be usable across experimental paradigms and at different phases in the rehabilitation training process. Sollfrank et al. [
In our research, the game part contained more of a gambling element compared to the Game-BCI system above. The 3D visual environment did not completely immerse players but felt more like an operating space. Players paid most attention in the ERD/ERS pattern generation. In order to make players feel that they were completing a complicated control mission with four motor imagery and two EOG commands, an interpretation method of physiological signal was formed based on the concept of shared control. Through evaluating the significance of ERD/ERS generation, we found that 3D Tetris Game-BCI provided an effective approach for players to enhance MI-based BCI skills. During 10 training days, the rapid growth of scoring rate appeared in the last five days. We interpret that outcome to mean that players were willing to use the 3D Tetris Game-BCI system after they mastered the needed skills. So we claim that the pattern of Game-BCI will be a tremendous advance in BCI research field.
The algorithm called one-versus-rest (OVR) CSP is an extension of a well-known method called common spatial patterns to multiclass case, to extract signal components specific to one condition from electroencephalography (EEG) data sets of multiple conditions.
In this research, the details of the one-versus-rest CSP algorithm are as follows.
Here,
Here,
However, the variances of only a small number
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This work was supported by the State Key Laboratory for Manufacturing Systems Engineering in Xi’an Jiaotong University and the School of Computer Science in Xi’an Polytechnic University.