Evaluation of a Compact Hybrid Brain-Computer Interface System

We realized a compact hybrid brain-computer interface (BCI) system by integrating a portable near-infrared spectroscopy (NIRS) device with an economical electroencephalography (EEG) system. The NIRS array was located on the subjects' forehead, covering the prefrontal area. The EEG electrodes were distributed over the frontal, motor/temporal, and parietal areas. The experimental paradigm involved a Stroop word-picture matching test in combination with mental arithmetic (MA) and baseline (BL) tasks, in which the subjects were asked to perform either MA or BL in response to congruent or incongruent conditions, respectively. We compared the classification accuracies of each of the modalities (NIRS or EEG) with that of the hybrid system. We showed that the hybrid system outperforms the unimodal EEG and NIRS systems by 6.2% and 2.5%, respectively. Since the proposed hybrid system is based on portable platforms, it is not confined to a laboratory environment and has the potential to be used in real-life situations, such as in neurorehabilitation.

NIRS is an emerging neuroimaging modality that records the cortical hemodynamic response based on changes in local optical transmission as measured by pairs of nearinfrared light sources and detectors placed on the scalp surface [20]. This method is less sensitive to motion artifacts compared to EEG [21]. NIRS has by now been recognized as a promising neuroimaging modality that has overcome some of the drawbacks of EEG [22]. Recently, portable and costeffective NIRS systems have become available [23] and have been actively used in the field of rehabilitation [24][25][26]. A defining characteristic of NIRS is the inherent delay of the measured hemodynamic response on the order of several seconds (typically > 5 s) [27], which limits its use in timecritical BCI applications and which requires a relatively long interstimulus interval (ISI) to gain task-relevant responses of reasonable quality. The resulting increase of the experimental time not only drops the overall information transfer rate usually quantified by bit rate per minute but can also exhaust the NIRS-based BCI users more easily [28].
To overcome the disadvantages of these individual methods, NIRS-EEG hybrid (HYB) BCI systems have been suggested to take advantage of superior performance provided by combining both modalities [29][30][31][32][33][34][35][36]. However, despite the comparatively low cost and compactness of both EEG and NIRS systems, the experimental setup of a hybrid system still poses practical challenges, even in a laboratory environment. Until now, each system required an individual amplifier, recording platform, and its own leads, which need to be affixed to the scalp with reliable optical and electrical contact. This poses added challenges and leads to generally increased setup times for HYB systems [32,37,38].
To date, to the best of our knowledge, no study that aims to reduce the complexity of hybrid NIRS-EEG BCI systems and validate their performance has been reported. In this study, we implement a lightweight and portable NIRS-EEG hybrid instrument and demonstrate its use for a hybrid BCI that has the potential for mobile and continuous use. We recorded NIRS and EEG signals simultaneously while the subject performed a word-picture matching test using simple mental arithmetic (MA), which is similar to the task in Power et al. [39]. The proposed hybrid system was validated by comparing its classification accuracies to those of the unimodal systems (EEG and NIRS).

Subjects.
Eleven right-handed healthy subjects participated voluntarily in the experiment (1 male and 10 females, average age: 25.7 ± 3.2 years [mean ± standard deviation]). None of them had a history of neurological, psychiatric, or other disorders that might affect the experimental results. A written experiment summary was given to the participants, and each participant signed a written consent form prior to the experiment and obtained a financial reimbursement after the experiment. This study was approved by the Ethics Committee of the Institute of Psychology and Ergonomics, Berlin Institute of Technology (approval number: SH 01 20150330).

Apparatus.
In the experiment, 14 EEG electrodes and eight NIRS probes (5 sources and 3 detectors) were placed on the scalp by means of a stretchy fabric cap (EASYCAP GmbH, Herrsching am Ammersee, Germany). The EEG system used was an EPOC device (Emotiv Inc., San Francisco, USA) and was selected for easy setup, wireless form factor, and, in particular, its economical price. The system had been verified in previous studies to show comparable performance to other commercial EEG devices with much higher prices [40][41][42][43][44][45][46][47][48][49].
In its original state, the EPOC uses a rigid headpiece of headphone-like appearance, which would not have allowed easy integration with NIRS. In a recent study, Debener et al. [50] demonstrated performance enhancement of the EPOC system by replacing the original head gear with a traditional fabric cap and ring electrodes. Following Debener et al. 's instructions, we dismantled the original hardware and moved the amplifier electronics into a small custom plastic case attached to the back of the cap. To provide good skin contact, we used passive Ag-AgCl ring electrodes (EASYCAP GmbH) with conductive gel. To measure taskrelated brain activation, a custom channel layout was chosen according to the international 10-10 system [51]. Fourteen electrodes were placed on frontal (F7, F3, Fz, F4, and F8), motor/temporal (C3 and C4/T7 and T8), and parietal (P7, P3, Pz, P4, and P8) areas. Reference and ground electrodes were attached on the left (TP9) and right (TP10) mastoids, respectively. The EEG signals were sampled at a 128 Hz sampling rate with provided software named "test bench" from the manufacturer. A portable NIRS system (NIRSport, NIRx Medical Technologies, NY, USA) was used to map hemodynamic responses. Five sources and three detectors were located over the prefrontal area around Fpz, Fp1, and Fp2 with an interoptode distance of 30 mm. Adjacent pairs of source and detector optodes comprised nine physical channels. NIRS signals were recorded at a 12.5 Hz sampling rate with NIRStar software, provided by the manufacturer. Figure 1 shows the channel layout of NIRS optodes and EEG electrodes and the headgear setup on a phantom head.

Experimental Protocol.
Subjects sat still in a comfortable armchair in front of a 24-inch LCD monitor. NIRS and EEG signals were acquired simultaneously from each subject while performing MA as a cognitive task and rest condition as a baseline task (BL). During MA, the subjects were instructed to subtract a single-digit number (between 6 and 9) from a random three-digit number and subtract it again from the result over and over as fast as possible until the trial ended (e.g., 544−7 = 537, 537−7 = 530, and 530−7 = 523). During the BL task, they were instructed not to think anything to maintain a low cognitive load state, while moving the body as little as possible. Even though the subject was instructed not to move the head and body, unintended subtle movement and unavoidable ocular movement might occur during the experiment. The quality of the EEG signal is easily affected by such artifacts, while the quality of the NIRS signal is less vulnerable to them. Figure 2 presents a timing sequence of a single trial. The experiment was designed as a Stroop word-picture test. A similar task was used in Power et al. [39]. A single trial was composed of task presentation (congruent or incongruent task, 2 s), followed by initial MA problem presentation (2 s), task period (10 s), and rest period (15-17 s). At the task presentation stage, two pictures (e.g., animals, fruits, and sport activities) were displayed on the screen side by side, and the name of either of the two objects was shown on the top of the screen. First, the left picture was highlighted using a red box. After 2 s, a random MA problem replaced the word. After 2 s, the problem was replaced by a black fixation cross with a short beep (250 ms) and the task period started. After the task period, a rest period with a random length of 15-17 s started with a short beep (250 ms), in which a large black fixation cross was displayed in the middle of screen. After the trial was finished, the same procedure was iterated with the right picture highlighted instead of the left one. If the displayed name matched the picture (congruent), subjects were asked to perform the MA task. On the other hand, if they were not matched (incongruent), subjects were asked to try not to think anything as a baseline task during the task period. During the rest period, subjects were instructed to relax and think nothing (BL). Therefore, congruent and incongruent trials were presented in a row as a pair, either "congruent firstincongruent later" or "incongruent first-congruent later," for the same picture set. They were presented in a random order. A single trial consisted of both MA and BL trials, and a session consisted of 10 trials (i.e., 10 MA + 10 BL). After finishing a single session, a short break was given in which subjects were allowed to move their bodies but not to leave the seat. The session was repeated three times constituting three sessions. Overall, although the number of "congruent firstincongruent later" and "incongruent first-congruent later" trials might not be equal within each session, a total of 30 MA and 30 BL were acquired across the three sessions.

Data Analysis
is a measure of correlation between a dichotomous variable and a continuous variable. The r-value was estimated to determine the spectral and spatial distribution of separability. The r-value at the time of interest is defined as [52] where ∈ [1, 2, . . . , ] and t is the length of the time of interest and 1 and 2 denote the total number of trials of class 1 and class 2 (MA and BL in this study), respectively. denotes the data points that belong to class label = 1 or 2.
[⋅] and [⋅] are mean and standard deviation operators, respectively. The -value was also utilized to calculate the most discriminative frequency band for the EEG feature extraction and a spatial distribution of separability for the NIRS temporal response.

Preprocessing.
Offline EEG and NIRS data analyses were performed using MATLAB 2013b (The MathWorks, Natick, USA), in particular with the EEGLAB toolbox and BBCI toolbox [2,53]. For NIRS data, raw light intensity signals were band-pass filtered (3rd-order Butterworth zerophase filter with a passband of 0.01-0.2 Hz). Concentration changes of oxyhemoglobin (Δ[HbO]) and deoxyhemoglobin (Δ[HbR]) were then calculated according to the modified Lambert-Beer law [54,55]. Baseline correction was performed using 5 s of prestimulus period. For EEG, the data The whole process was done twice consecutively for congruent and incongruent tasks, which comprised a single trial. "Congruent first-incongruent later" and "incongruent first-congruent later" tasks were randomly presented. At the task presentation, the left-or right-side picture was sequentially selected. The name of either picture was displayed for 2 s. At initial mental arithmetic (MA) problem presentation, an example of a three-digit number minus a one-digit number (6 to 9) was shown instead of the name for 2 s. In a task period starting with a short beep (250 ms) and black fixation cross, subjects performed MA or baseline (BL) task if the word and picture were matched (congruent) or mismatched (incongruent), respectively. After 10 s, a rest period started with a short beep (250 ms), and a large black fixation cross was displayed at the center of the screen.

Classification.
We performed a single trial classification of NIRS and EEG data to discriminate MA-and BL-related responses [29,57]. To examine classification accuracy change with respect to different time windows, a sliding time window was used to extract the features of both modalities (window size: 5 s, step size: 1 s) between −5 and 25 s from stimulus onset to account for the hemodynamic delay with respect to brain activation [58]. The relatively long window size was chosen to consider the relatively slow hemodynamic responses compared to EEG, thereby increasing the performance of each modality as well as the HYB system. Both NIRS and EEG features were calculated for each sliding time window. All NIRS and EEG channels were used for feature extraction and classification (9 and 14 channels). For NIRS, the mean values and average slopes of Δ[HbO] and Δ[HbR] of each channel were calculated as NIRS features, which are widely used for NIRS data classification [7]. For EEG, the common spatial pattern (CSP) algorithm was applied to the preprocessed EEG data. EEG features were calculated as the log-scaled variance of CSP-filtered data (first and last 2 components containing the most discriminative information). The feature vectors of each sliding time window were independently used for the classification. Tenfold cross-validation was performed 10 times for each sliding window. For classification, shrinkage linear discriminant analysis (sLDA) was used [52]. The shrinkage parameter was estimated as described previously [59,60]. In order to confirm the advantage of adding EEG data to NIRS data, the correct answer ratio was estimated not only for the EEG or NIRS data individually but also for a combination of both modalities. For the latter case, a metaclassification approach based on sLDA was used. Normalization is not necessary when concatenating EEG and NIRS features for metaclassification. This is because, for metaclassification, both EEG and NIRS individual classifiers yield LDA-projected EEG and NIRS features with the same scale, which are combined for the input of the metaclassifier. The detailed information regarding the metaclassifier is provided in Fazli et al. [32].  hand, fewer ERD/ERS patterns were observed during BL task. Thus, the distinct difference of ERD/ERS between MA and BL was widely observed in the corresponding frequency band. In Figure 3, -band or -band (8-30 Hz) is included in the passband of 12 (85.7%) or 9 (64.3%) of 14 subjects, respectively. Figures 4(a)-4(d) show the grand average of CSP patterns that correspond to the eigenvectors for the highest and lowest two eigenvalues ( ) for CSP [56]. Note that frontal and parietal areas are mainly associated with task-relevant activation. Figures 5(a) Table 1 denotes the maximum accuracies of each subject among the tested time windows. Eight of eleven subjects showed the EEG accuracy exceeding the BCI performance threshold (>70% for binary communication [61]) and scored 82.0 ± 11.2% on average. All subjects exceeded the threshold accuracy when NIRS data was used (HbR + HbO) and scored 85.7±4.9% on average. For all three cases combining EEG data with NIRS data, classification performance was significantly improved (e.g., HbR: 81.4 ± 7.2% versus HbR + EEG: 86.3 ± 8.4%; Wilcoxon rank-sum test, < 0.05). Since HbR + HbO scored the highest mean accuracy among the tested NIRS chromophores (HbR, HbO,   and HbR + HbO), HbR + HbO represents the NIRS result hereafter. The classification of HbR + HbO + EEG could enhance the accuracy by 2.5% and 6.2% compared to NIRS and EEG alone, respectively.

Classification.
The grand average classification accuracies with error bars indicating the standard errors are presented in Figure 6.
During the task period (gray shaded period), EEG accuracy reached the highest value at t = 6 s. Due to the hemodynamic delay, NIRS showed the highest value 4 s after the end of the task (t = 14 s). The classification performance of the hybrid modality was significantly higher than that of EEG or NIRS for most time periods or was at least comparable. In Figure 6, red and blue asterisks represent time windows in which the classification accuracies of the HYB were significantly higher than those of EEG or NIRS alone, respectively. Figure 7 shows the performance comparison between the NIRS and HYB. The comparisons were made where the EEG, HYB, and NIRS scored the maximum accuracy according to the results shown in Figure 6 (t = 6, 11, and 14 s, resp.). At t = 6, the performance comparison between the EEG and HYB was also provided (see red circles). The number on the upper left side denotes the percentage of the improved results by HYB. All subjects' performances were improved by HYB at t = 6 and 11 s ( < 0.01). At t = 14 s, the HYB was not capable of showing significantly better performance than NIRS ( = 0.482). This might be caused by less contribution of EEG features to the performance after t = 10 s, when less task-relevant activation was produced after the task period.

Discussion
We aimed to establish a lightweight hybrid BCI system by combining a portable NIRS with an economical EEG system. The classification results verified that the simultaneous use of EEG and NIRS data was beneficial to improve classification performance. In particular, all subjects (except one: VP001) showed increased performance when the hybrid modality was used (see Table 1). Some previous studies have already confirmed that a hybrid BCI system combining NIRS with EEG can improve system performance, but they used stationary and bulky devices, thereby limiting application outside the laboratory setting [32,37]. Since our hybrid system was implemented by combining a portable NIRS with an economical EEG system, it can be widely used and easy to handle not only in laboratory settings but also in out-of-lab scenarios. Even though we verified the feasibility of the hybrid neuroimaging system in a typical BCI scenario, it may also be used for neurorehabilitation purposes, such as restoring motor functions lost in neurological disorders. In this study, MA was selected as a cognitive task to demonstrate the usability of our system because it is one of the stable and consistent cognitive tasks that can produce distinct task-relevant brain activation. As the light and convenient NIRS optodes can easily be reorganized to configure the channels, they are able to measure signals from different brain areas such as motor or occipital areas. However, we must note that careful hair preparation is necessary to avoid interference with the signal acquisition in this case.
In this study, we implemented a MA-based BCI system to demonstrate the feasibility of our hybrid EEG-NIRS neuroimaging device. This device generally showed low operation speed (10 s is theoretically required for producing one command) compared to other paradigms, such as P300 and steady-state visual evoked potential (SSVEP) [62]. However, as the EEG electrodes and NIRS optodes of our hybrid device can be easily reorganized, BCI systems employing other brain areas or paradigms could also be implemented using our hybrid neuroimaging device. For example, it is possible to develop an SSVEP-based BCI system by moving the recording sensors of our hybrid system to occipital areas. It has been well documented that an SSVEP-based BCI system shows high operation speed, and, in particular, a recent study demonstrated that the simultaneous use of EEG and NIRS can further increase the operation speed of an SSVEP-based BCI system [63]. Thus, our hybrid EEG-NIRS recording device may also be used to develop a highspeed BCI system for other BCI paradigms by appropriately modifying the configuration of recording sensors.
It is generally acknowledged that increased Δ[HbO] and decreased Δ[HbR] are induced in task-relevant brain areas during performance of the corresponding task. As seen in Figure 5(a), the opposite pattern to the typical NIRS signal pattern was observed, in that the increasing trend of Δ[HbR] was synchronized with the decreasing trend of Δ[HbO] from the task onset. Note that these opposite patterns are also frequently shown in the literature [64,65]. Particularly, significant Δ[HbO] decrease and Δ[HbR] increase were observed during MA tasks in frontal areas [64][65][66].
As mentioned previously, system performance improvement with respect to the classification accuracy was not accomplished by HYB after t = 14 s. This likely results from the lack of task-relevant EEG signals after task termination, and, therefore, EEG does not contribute to the enhancement of system performance at this time. Moreover, before t = 6 s, because of inferior temporal responsiveness of NIRS due to inherent hemodynamic delay, the system performance improvement is also not observed. It is worth mentioning that, because of the delayed responses of NIRS, it would be hard to implement a high-speed BCI system using solely NIRS; nevertheless, NIRS is helpful as a second modality when incorporated in a hybrid BCI with EEG. Based on changing performance over time, an optimal task time length can be determined between 6 and 12 s for HYB. However, for t = 6 s, the degree of performance improvement is less than that for t = 12 s, while 12 s may degrade the usability of the NIRS system owing to the relatively long task time. The tradeoff between the time period to make a decision and performance should be considered based on whether system speed or performance is preferred.

Conclusion
Recently, various easily wearable commercial EEG devices have been released [67][68][69][70]. These devices have lightweight and easy-to-use configurations. They are used in the field of rehabilitation as well as in entertainment. In this study, we verified the usefulness of a lightweight hybrid BCI system by combining a portable NIRS and a cost-effective EEG system. Our hybrid EEG-NIRS system allowed for improved classification performance. Despite probable doubt with respect to system stability and reliability of the economical EEG system, we verified that the proposed system is capable of stably enhancing the system performance. The concurrent use of the portable NIRS and EEG systems can help us to use the combined system more practically in a cost-effective way. Therefore the proposed system has a high potential for future BCI research in out-of-lab scenarios at low cost.