This study proposes a home care system (HCS) based on a brain-computer interface (BCI) with a smartphone. The HCS provides daily help to motor-disabled people when a caregiver is not present. The aim of the study is two-fold: (1) to develop a BCI-based home care system to help end-users control their household appliances, and (2) to assess whether the architecture of the HCS is easy for motor-disabled people to use. A motion-strip is used to evoke event-related potentials (ERPs) in the brain of the user, and the system immediately processes these potentials to decode the user’s intentions. The system, then, translates these intentions into application commands and sends them via Bluetooth to the user’s smartphone to make an emergency call or to execute the corresponding app to emit an infrared (IR) signal to control a household appliance. Fifteen healthy and seven motor-disabled subjects (including the one with ALS) participated in the experiment. The average online accuracy was 81.8% and 78.1%, respectively. Using component N2P3 to discriminate targets from nontargets can increase the efficiency of the system. Results showed that the system allows end-users to use smartphone apps as long as they are using their brain waves. More important, only one electrode O1 is required to measure EEG signals, giving the system good practical usability. The HCS can, thus, improve the autonomy and self-reliance of its end-users.
Individuals with locked-in syndrome (LIS), amyotrophic lateral sclerosis (ALS), spinal cord injury, and congenital or accidental nerve injury may experience serious obstacles in developing motor skills in their limbs, yet most of them have normal brain function [
Locked-in syndrome (LIS) is a condition in which a patient is aware but cannot communicate verbally or move because of complete paralysis of nearly all voluntary muscles in the body except for vertical eye movements and blinking [
ALS is a relatively rare neurodegenerative disorder characterized by gradual loss of both upper and lower motor neurons in the brain, brainstem, and spinal cord [
A BCI system is a system that connects the human brain and its surroundings. It enables people to communicate with others using their brain waves without muscle movement [
There are four different types of EEG-based BCI modalities: event-related desynchronization/synchronization (ERD/ERS), steady-state visual evoked potentials (SSVEP), event-related potentials (ERP), and slow cortical potentials (SCP). Among these, ERP and SSVEP-based BCIs are more practical than others because they support large numbers of output commands and need little training time [
Table
Recent studies of BCI-based systems implemented in real-world scenarios.
Study | Main function | Stimulation modality | Electrodes | Subjects | Accuracy (%) | Bit rate |
---|---|---|---|---|---|---|
[ | Speller | ERP: motion-onset-P300 | Fz, Cz, Pz, Oz, P7, and P8 | 10 CS | N2-91.5, P3-72.4 | N2-15.91, P3-12.84 |
[ | Chinese speller | ERP: motion-onset-N2P3 | F3, F4, C3, C4, P3, P4, O1, O2, Fz, Cz, and Pz | 7 CS | 80% using O1 only | 27.8 |
[ | Speller | ERP + SSVER RC paradigm | Cz, Pz, P3, P4, O1, O2, POz, PO7, and PO8 | 14 CS | After 8 trials: >95 | 53.6 |
[ | Robot control | ERP: motion-onset-N2P3 | O1 | 12 CS | 80% using O1 only | 353.33 s for 26.33 comm. |
[ | Robot control | EOG + EEG: flash on eight direct | Fz, Cz, Pz, Oz, P7, P3, P4, and P8 | 13 CS | After 5 trials:>99.04 | — |
[ | Speller | ERP + SSVER RC paradigm | Fz, Cz, Pz, P3, P4, PO7, PO8, POz, Oz, O1, and O2 | 13 CS | 95.18 for hybrid | 50.41 for hybrid |
[ | Healthcare BCI syst. | ERP + SSVER RC paradigm | Cz, Pz, O1, O2, and Oz | 5 CS | ERP: 95.5SSVER:93 | — |
[ | Environmental control | ERP-P300RC paradigm | Fz, FCz, Cz, CPz, P7, P3, Pz, P4, P8, O1, Oz, and O2 | 6 MDS, 2 CS | 89.6 | 734.3 s for 30 comm. |
[ | Use of social networks | ERP-P300 RC paradigm | Fz, Cz, Pz, P3, P4, PO7, PO8, and Oz | 18 MDS, 10 CS | 80.6 for MDS, 92.3 for CS | 1.47 OCM for MDS, 2.06 for CS |
RC paradigm: the row-col paradigm; “N” indicates the number of subjects; “CS” stands for control abled subjects; “N2” stands for N200 evoked potential; “P3” stands for P300 evoked potential.
Table
Event-related potentials (ERP), proposed by Sutton in 1965, are a series of potentials of a user’s brain waves elicited by external stimuli. These potentials are time-dependent voltage fluctuations triggered by specific physical or psychological events [
ERP research provides an impersonal and workable discrimination method for a BCI system [
A waveform showing several ERP components, including the N200 (labelled N2) and P300 (labelled P3). Note that the ERP is plotted with negative voltages at the top, a common, but not universal, practice in ERP research [
However, conventional BCIs have not become practical because they lack high accuracy and reliability and have low information transfer rate and user acceptability [
A flashing stimulation paradigm such as that the row-col paradigm [
Second, the fewer the electrodes are, the more comfortable the user is. Based on the previous research results of our laboratory, when the user stares at the GUI of a vBCI system, the ERPs acquired from electrode O1 or O2 in the occipital area of the skull (the visual region of the human brain) can achieve a statistically significant difference between the target and nontarget stimuli [
Third, the waveform and the amplitude of ERP (N200 and P300) of the target stimulus vary from person to person. Thus, the BCI system needs to use a more stable ERP component to increase its accuracy. Based on the results of our previous experiments, the accuracy from using component N2P3 is significantly higher and steadier than the accuracy obtained using any other ERP component. Herein, the N2P3 value of one option is the potential value (
According to the work of Huggins et al., if caregivers are absent, BCI users may want to perform tasks such as controlling the room temperature and lights or make emergency calls by themselves and feel more comfortable than they would using text communication [
Because of the popularity of smartphones, several studies have applied BCI systems to control smartphones. Most of these studies explore merely dialing numbers [
This study develops a BCI-based home care system (HCS). The HCS allows the end-users to control their household appliances by themselves. Thus, end-users can reduce their dependence on the caregivers. In this study, there are two functions of a smartphone: to make an emergency call and to act as an adjustable infrared- (IR-) band remote controller. Thus, the user’s smartphone must have or install an IR transmitter first. The corresponding app in the user’s smartphone can, then, emit an IR signal to control the required household appliance.
Today, short-distance remote controls for devices in daily life make wide use of IR [
Improving the personal autonomy and the self-reliance of end-users and giving them the ability to communicate with others are two of the primary missions of the HCS. Since the assessment of BCI systems with end-users is essential for ensuring a fair evaluation [
The subjects used in this study were 15 healthy people (six females, aged 19–55), six motor-disabled people, and one man with ALS (SE7). Table
Clinical data of the motor-disabled participants.
Subject | Age | Gender | DD | Disease |
---|---|---|---|---|
SE1 | 35 | M | Moderate | Spinal cord injury |
SE2 | 37 | M | Moderate | Tetraplegia |
SE3 | 46 | M | Moderate | Spinal cord injury |
SE4 | 42 | M | Mild | Spinal cord injury |
SE5 | 39 | M | Moderate | Spinal cord injury |
SE6 | 43 | M | Mild | Spinal cord injury |
SE7 | 50 | M | Marked | ALS |
In this study, the BCI module in the home care system is derived from the BCI module of our Chinese spelling system [
System architecture of the proposed HCS, including the ERP-based vBCI system and its applications.
The proposed vBCI module of the prototype, such as other human-machine interface systems for communication or control, comprises input/output processes. The BCI module requires the input of signals gained from the user’s brain waves through an EEG. The EEG device in this system, which contains 32 channels, uses a typical noninvasive method [
In all GUIs of the BCI module, there are several graphic options arranged in sequence on each GUI, as shown in Figure
Four GUIs in the BCI module. (a) Main screen with four options; (b) TV control screen with six options; (c) air conditioner control screen with six options; and (d) TV channel shift screen with 12 options.
The vBCI module outputs communication signals to the user’s smartphone via Arduino and the HC-5 Bluetooth module. The communication signals first execute an app on the user’s smartphone designated ICAI1101, an application developed by the author. ICAI1101, then, triggers the corresponding app to make an emergency call or to send an IR signal to control a household appliance.
In this study, there are four GUIs in the BCI module, including the main screen, TV control screen, air conditioner control screen, and TV channel shift screen, as shown in Figure
When the BCI module identifies the option the user wants, it will send a command signal to the user’s smartphone to control ICAI1101. Then, the GUI of the user’s smartphone changes based on the user’s selection. If there is a caregiver around the user, they can follow-up on the user’s demands and help the user by using the GUI of the user’s smartphone directly. Figure
The caregiver GUIs of the ICAI1101 smartphone app: (a) main app screen on the smartphone; (b) TV remote controller; (c) air conditioner remote controller; and (d) TV channel shift screen.
In this study, a stimulus is defined as the motion-strip of an option moving from right to left once, about 200 ms, as shown in Figure
A stimulation schematic of one trial for the four options on the main screen. There are six instances of stimulation for every option in a single trial.
In Figure
The ERPs of four options on the main screen from the output of the first trial of SE3. The red circle indicates the potential of N200, while the green circle represents the potential of P300.
In Figure
Each subject took about 0.5 to 1 hour to complete the experiment, depending on the accuracy of the trials. Figure
The first step of the experiment was to attach electrodes to the subject’s scalp and check the signals. The BCI module, then, connects with the user’s smartphone via its Bluetooth module. Next, each subject performs 15 trials during the experimental procedure. In each trial, the user must choose an option from the GUI designed for the BCI module on the computer screen.
In each test, the subject had to gaze at the blue motion-strip of the option they wanted to choose. The system, then, collected the ERPs of all options available on the GUI from the EEG. After that, the BCI module identified the highest potential from the ERP components N200, P300, or N2P3. The option with the highest N2P3 potential should be the one the user was gazing at during the trial. The BCI module then sent a command signal to the user’s smartphone via Bluetooth to make an emergency call or to control a household appliance via IR.
Figure
Flowchart of the vBCI operating procedures. The solid arrow line shows that the system sends an instruction to switch the screen of the system to the target GUI. The dotted arrow line shows that the system is only sending a command to do something, and the screen remains on the same GUI.
Each subject performed their experimental procedure using the four GUIs. The TV and the air conditioner were under control during the process. They also made an emergency call before the end of the procedure. Table
Details of the 15 trial sequences.
Step | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
GUI | Main screen | TV screen | TV screen | TV screen | TV channel shift | TV channel shift | TV channel shift | TV screen | Main screen | Air condi. | Air condi. | Air condi. | Air condi. | Main screen | Main screen |
Option to choose | Enter TV | Volume up | Next channel | Enter channel shift | Choose channel 1 | Choose OK | Back TV | Back to main | To air condi. | Turn on AC | Temp down | Air Flow | Back to main | Emergency call | System off |
The equipment used to acquire the EEG data included a 32-channel EEG amplifier, an ISO-1032CE, and the control unit, CONTROL-1132, produced by Braintronics B.V. Company. The system uses PCI-1713 to convert analog data to digital data. The authors wrote the vBCI module using Borland C++ Builder and wrote the ICAI1101 smartphone app in Java. The BCI module used an Arduino Uno and HC-05 Bluetooth module to communicate with the user’s mobile phone.
In the EEG acquisition settings, the sampling rate is 500 Hz, and the impedance remains below 10 kΩ. The EEG acquires the subject’s brain waves from electrode O1 on the user’s scalp. The electrodes for eye movement detection are FP1 and FP2. The reference electrodes are A1 and A2. The ground electrode is FPz [
(1). ERPs acquisition: in each trial, each motion-strip moves from right to left six times, as shown in Figure
(2). ERPs analysis: the system saves and analyses the final ERPs for each option in each trial. The system, thus, finds the N200 value and the P300 value of each option and, then, determines the N2P3 value. Next, the system compares the N2P3 value of all options to each other to identify which option the user selected.
(3). Instruction output: the system translates the ERP analysis results into a BCI instruction and sends it to the user’s smartphone via Bluetooth. When the smartphone receives a BCI instruction, it runs the application to make an emergency call or to control an appliance via IR.
In addition to the accuracy rate, the rate at which information per unit of time is obtained is particularly important for evaluating a BCI system. To calculate the number of bits available per minute, the bit-rate calculation in this study uses the definition of Wolpaw [
Table
Figure
Two ERP samples from one motor-disabled subject. (a) The ERPs from the main screen of the BCI system (4 options); (b) the ERPs from TV control screen of the BCI system (6 options).
In Figure
In Figure
Figure
The ERP values (
Options | N200 | P300 | N2P3 | Result | |
---|---|---|---|---|---|
Online | Offline | ||||
TV | −1.7969 | 2.2845 | 4.0814 | ✓ | N200, N2P3 |
AC | 0.6518 | 3.5418 | 2.8900 | P300 | |
EC | 0.0030 | 1.0302 | 1.0272 | ||
Off | −1.7847 | −0.9263 | 0.8584 |
Figure
The ERP values (
Options | N200 | P300 | N2P3 | Result | |
---|---|---|---|---|---|
Online | Offline | ||||
Next channel | −1.6458 | 2.7013 | 4.3471 | ✓ | N200, N2P3 |
Channel shift | −0.3196 | 2.5717 | 2.8913 | ||
Volume up | 0.9145 | 3.0962 | 2.1817 | P300 | |
Prev. channel | −0.5131 | 1.4089 | 1.9220 | ||
Main screen | 0.8788 | 1.0479 | 0.1691 | ||
Volume down | 1.3293 | 1.1093 | −0.2200 |
Fifteen healthy subjects participated in the experiment. The experimental results showed that the feature of N2P3 enabled the best discrimination. The average accuracy across all 15 users was 81.78%, meaning that of all 15 commands, 12 were performed right. The precision attained by 10 of the 15 subjects was greater than 80%. The accuracy of E10 was even 100%. However, the accuracy of E1 and E15 was unacceptable. These two subjects may not be able to adapt to the BCI system or were disturbed by other factors, resulting in reduced efficiency. Figure
The accuracy levels and bit rate attained by all 15 healthy subjects.
The average bit rate attained by all 15 healthy subjects is 27.11. It is better than that of other studies [
Figure
This bar chart summarizes the number of correct selections for each healthy subject for each trial. For example, if the system uses N2P3 to interpret the EEG, the choices from 14 of all 15 users are correct in the first trial, while N200 13 are correct, and only 10 are correct for P300.
Table
Paired-sample t-test results of all trials for the 15 healthy subjects.
Case | |||
---|---|---|---|
N200 | Targeted vs. nontargeted | 0.747 | 0.467 |
P300 | Targeted vs. nontargeted | 6.225 | 0.000 |
N2P3 | Targeted vs. nontargeted | 8.998 | 0.000 |
N2P3 vs. P300 | N2P3-targeted vs. P300-targeted | 2.276 | 0.039 |
Six motor-disabled people and one man with ALS participated in the experiment. The experimental results showed that the feature of N2P3 offered the best discrimination. The average accuracy across all seven users was 78.10%, meaning that of all 15 commands, 11 were performed right. However, the accuracy of SE2 is not acceptable. This subject may not be able to adapt to the BCI system or was disturbed by other factors, resulting in reduced efficiency. Figure
Accuracy levels and bit rate attained by all motor-disabled subjects (including one ALS, SE7).
The average bit rate attained by all seven disabled subjects is 22.37. Although the average bit-rate attained by the disabled group is lower than that of the healthy group, it is also better than that of other studies [
Figure
This chart summarizes the correct selections of all motor-disabled subjects for each trial. For example, if the system uses N2P3 to interpret the EEG, the choices from 7 of 7 users are correct in the first trial, while 4 are correct with N200 and 5 with P300.
Table
Paired-sample
Case | |||
---|---|---|---|
N200 | Targeted vs. nontargeted | −4.509 | 0.004 |
P300 | Targeted vs. nontargeted | 0.346 | 0.741 |
N2P3 | Targeted vs. nontargeted | 6.953 | 0.000 |
In this study, we conducted experiments with 15 healthy subjects, six motor-disabled subjects, and one ALS. The average accuracy attained by the 15 healthy subjects was 81.78% if using N2P3 (online) for interpretation, while the average accuracy attained by the seven motor-disabled subjects was 78.10%. The disabled group has a lower accuracy level than the healthy group. However, both groups had an average accuracy of more than 75%.
We compared the results of these two independent samples, as shown in Table
Independent-sample
Case | ||||
---|---|---|---|---|
N200 targeted | Healthy vs. disabled | 0.1596 | 3.1693 | 0.0048 |
P300 targeted | Healthy vs. disabled | 0.2428 | 2.9396 | 0.0081 |
N2P3 targeted | Healthy vs. disabled | 0.2817 | 0.6258 | 0.5385 |
Bit rate | Healthy vs. disabled | 0.2576 | 0.8793 | 0.3897 |
The average bit rate attained by all 15 healthy subjects is 27.11 (Figure
The aim of the present study is two-fold: (1) to develop a BCI-based home care system to help end-users control their household appliances and (2) to assess whether the architecture of the HCS is easy for motor-disabled people to use.
First, we designed and developed a BCI-based home care system (HCS). We designed the HCS to make an emergency call or control the household appliances, such as TV and air conditioner, via a smartphone. Thus, end-users can improve personal autonomy and reduce their dependence on caregivers.
Second, most previous research has not experimented with end-users. Thus, the second purpose of this study was to assess the usefulness of the system with motor-disabled subjects. We conducted experiments with both healthy and motor-disabled subjects. One subject had ALS.
Previous researchers attempted to improve the performance of their BCI systems [
Most BCI studies use Fz, Cz, Pz, Oz, and other electrodes to collect data [
To use an electrode, Q1, to gain the data, we ask the user to stare at the GUI when using the system. The system asynchronously shows the stimuli to shorten stimulation times. In the HCS there are four options on the main screen, six options on the TV and AC control screen, and 12 options on the TV channel shift screen. Figures
Previous studies have stated that component P300 provided an excellent level of discrimination [
When using component P300, the healthy and the motor-disabled subjects had an average efficiency of 73.78% (SD = 14.79) and 52.38% (SD = 18.23), respectively. Although the precision attained by 8 of the 15 healthy subjects was greater than 80%, only SE7 (ALS) obtained an accuracy of 80% when using P300 for interpretation. However, for the online experimental results (using component N2P3), the healthy and the motor-disabled subjects exhibited an average efficiency of 81.78% (SD = 13.69) and 78.10% (SD = 10.69), respectively. The precision attained by 10 of the 15 healthy subjects was greater than 80%, and the precision obtained by three of the seven motor-disabled subjects was greater than 80%. Furthermore, Figure
Although the disabled group has a lower accuracy level than the healthy group, the difference is not significant (
The second issue in system construction is making the system easy for end-users. This question included two key points: whether the GUI of the HCS is friendly and whether the remote controls for all appliances can be integrated into one remote control.
First, Figure
Second, there are often two common household appliances, TVs and air conditioners, in the same room. Every home appliance has a dedicated remote control. If all home appliances can share the same remote control, the system can be easy for end-users to use. In this study, we added a smartphone to the HCS. A smartphone with IR communication technology may act as a remote control. It can emit a distinct IR wavelength to control any household appliance. Thus, we developed an app called ICAI1101 to control the TV and air conditioner using a smartphone via IR. Such a BCI system would make it easier for the end-user to control their home appliances.
ICAI1101 is first installed on a smartphone. When the user makes a choice during use, ICAI1101 can act following the user’s choice. Furthermore, ICAI1101 not only makes emergency calls and controls the TV or air conditioner but can also integrate other apps. The rapid growth of the Internet and the popularity of smartphones have had an immense impact on human life in the last two decades [
Tables
In this study, we have proposed a home care system that combines BCI with a smartphone. The HCS helps end-users, motor-disabled people, make an emergency call or control their household appliances. Thus, end-users can take care of themselves with only eye muscle movement. Fifteen healthy subjects and seven motor-disabled subjects (including one with ALS) participated in clinical trials. Because of the high accuracy-rate and rapid response of the system during the online experimentation, most subjects of both groups can rapidly complete the experimental procedure in less than the preset time, 35 minutes. In the offline analytics, the data collected enabled us to evaluate and improve the performance of the system. The results showed that the disabled group has a lower accuracy level than that of the healthy group, but the difference is not statistically significant. The average accuracy of the disabled group (78.10%) not only exceeded the chance level but was also higher than 75%. The bit-rate analysis yielded conclusions similar to those of the accuracy analysis. Thus, when a user chooses an option, the accuracy of the target option in a short period exceeds three-fourths. We, therefore, reason that the HCS is a viable system for motor-disabled people.
The HCS is a system that can be used without prior training. The bit rate of the HCS is close to that of a previous study, the Chinese Spelling System, performed in our Lab [
Data are available on request. E-mail:
The authors declare that there are no conflicts of interest regarding the publication of this paper.