Advances in neural interfaces have demonstrated remarkable results in the direction of replacing and restoring lost sensorimotor function in human patients. Noninvasive brain-computer interfaces (BCIs) are popular due to considerable advantages including simplicity, safety, and low cost, while recent advances aim at improving past technological and neurophysiological limitations. Taking into account the neurophysiological alterations of disabled individuals, investigating brain connectivity features for implementation of BCI control holds special importance. Off-the-shelf BCI systems are based on fast, reproducible detection of mental activity and can be implemented in neurorobotic applications. Moreover, social Human-Robot Interaction (HRI) is increasingly important in rehabilitation robotics development. In this paper, we present our progress and goals towards developing off-the-shelf BCI-controlled anthropomorphic robotic arms for assistive technologies and rehabilitation applications. We account for robotics development, BCI implementation, and qualitative assessment of HRI characteristics of the system. Furthermore, we present two illustrative experimental applications of the BCI-controlled arms, a study of motor imagery modalities on healthy individuals’ BCI performance, and a pilot investigation on spinal cord injured patients’ BCI control and brain connectivity. We discuss strengths and limitations of our design and propose further steps on development and neurophysiological study, including implementation of connectivity features as BCI modality.
Advances in neural interfaces including implantable neural prosthetics and brain-computer interfaces (BCIs) have recently demonstrated remarkable results in the direction of replacing [
Despite recent technological breakthroughs in BCI research, in terms of reliability, accuracy, and speed, the best results in robotics and neural prosthesis control have been demonstrated by invasive technology (neural implants) [
Even past the challenges and limitations of BCI systems, the design of a robotic arm for medical engineering applications, such as rehabilitation and assistive technologies for disabled individuals, constitutes a challenge on multiple fronts, including engineering problems, design requirements, and budget cost issues [
While programmable automation design can be traced back to Ancient Greece [
To that end, social robotics and Human-Robot Interaction (HRI) are considered important—yet sometimes overlooked—aspects of robotics development [
In our previous work we have already presented the conceptual design and development of the Mercury robotic arm for biomedical applications and dealt with construction standards and validation tests [
In the remainder of this paper we present our progress and goals towards developing off-the-shelf BCI-controlled robotic arms for assistive technologies and rehabilitation applications. In Materials and Methods, we first account for further development of the robotic arms and electronics, including a qualitative assessment study of the BMI module. We subsequently report on the implementation of the BCI control module using an off-the-shelf EEG-BCI system and the development of BCI-robotics communication; then we present two illustrative experimental applications of the BCI-controlled robotic arms. The first experiment is a study on healthy individuals to compare MI modalities for optimal BCI performance. The second experiment regards a comparative pilot investigation on SCI patients and healthy individuals for noninvasive control of multiple robotic arm motions and functional connectivity [
The Mercury robotic arm system has been developed as a customized design by our team for two technological generations so far [
The Mercury robotic platform comprises a robotic arm currently capable of movement along 8 DoFs (at shoulder, elbow, wrist, hand gripper, and thumb joints), as well as a choice between two control modules [
Current generation of Mercury robotic arm: (a) the robotic arm in position during an illustrative experiment, (b) the 3D-printed gripper (in focus circle), and (c) schematic of the 8 DoFs of the robotic arm. Mercury arms are house-built, of low cost, and anthropomorphic.
The BMI control module for the Mercury robotic arm has been described in previous work in terms of design, construction, cost, and features [
During the design process of the EPSN, emphasis was placed on rapid capture and transfer of control signals to the Mercury robotic arm, allowing it to replicate the movement of the human operator’s arm in a fluid, anthropomorphic fashion. For this purpose analogue classical automation control circuits were used to calculate analogue control signals subsequently fed to an Atmel ATmega2560 microprocessor. The microprocessor handled digitization, interface to a PC, and generation of the control signals for the Mercury robotic arm. Initial experiments using the EPSN focused on HRI, specifically the time required for first-time human operators to develop the skills to control the Mercury robotic arm and perform basic tasks such as knocking, gripping, lifting, and placing small objects [
Advances in both hardware and software technologies rendered real-time EEG processing a possibility, including detection and identification of brain activity features for use in BCIs. Currently there are several BCI systems available commercially, one of which is the Emotiv EPOC (USA), sold around $300, which is significantly lower than most medical EEG devices. It is a portable, wireless EEG recording device that has 14 dry electrodes arranged according to the international 10–20 System and can be easily mounted to the user’s head. The device operates at an internal sampling rate of 2048 Hz and the data are transmitted wirelessly at 2.4 GHz to a USB dongle with a sampling rate of 128 Hz. The BCI capabilities of the device are accessed by the Cognitiv suite and rely on Event Related Desynchronization (ERD). The user initially needs to record a resting state EEG after which he is able to train up to four mental commands, using a machine-learning pipeline to teach the BCI how he visualizes. The pipeline operates along the stages of preprocessing, feature extraction, reduction of dimensionality, and classifier training. Following the training, the suite will continually attempt to identify the trained commands by analyzing the user’s EEG. During this process, the suite presents a floating box that will execute any mental command that it identifies, and the action power, corresponding to the level of confidence of each classification.
In order to achieve online communication between the commercial BCI application and the robotic arms, the trained BCI classes are mapped in real-time to computer controls, using a combination of the BCI’s native Emokey application (Emotiv, USA) and an in-house script, developed in Matlab environment (Mathworks, USA). In our implementation, the BCI is trained in only three classes: one for resting state and two for general “left” or “right” directions, using either visual or kinesthetic motor imagery. Each BCI class is linked to a specific key button, which is enabled when the detected mental state corresponds to that class. Then the script accepts the corresponding command as input and transmits it through a serial port, with Baud Rate 9600, to the on-board microcontroller unit for each Mercury robotic arm (Figure
Schematic of Brain-Computer Interface loop: using off-the-shelf EEG-BCI for control of house-built robotic arms.
The Bioethics & Ethics Committee of Faculty of Medicine, Aristotle University of Thessaloniki, approved the experimental protocol. All experiments were conducted after the participants providing informed consent and no remuneration was given. To facilitate the integration of the robotic arms (or the limb presentation during EEG recording) into the participants’ own body schema, their arms and body were covered with a black curtain [
All experimental parts that involved the use of the robotic arms were conducted in the Thessaloniki Active and Healthy Ageing Living Lab technology showcase room (Thess-AHALL, member of ENoLL,
Overview of the experimental setup in the Thess-AHALL Living Lab. The figure is modified with authors’ permission [
The experimental parts that involved the use of high-resolution EEG recording were conducted in a specially designed magnetic shielded room for recordings with presentation capabilities and audiovisual monitoring. The participants sat on an inclined armchair inside the room, while facing a 21′′ computer monitor located a meter away. Recordings were obtained using the 10-5 international electrode system for high-resolution EEG [
The first of the two illustrative experimental applications was a qualitative assessment study, comparing MI modalities for control of the BCI-controlled robotic arms by healthy individuals with regard to BCI training and optimal performance [
In total thirty healthy participants were included in the study, 18 male (60%) and 12 female (40%), ranging from 19 to 46 years (median age 24 years). All 12 female and 14 of the male participants declared that they were right-handed. From the rest of the male participants, 2 declared being left-handed and 2 being ambidextrous.
Kinesthetic motor imagery (KMI) modality was trained first. The participants were asked to relax and resting state EEG with eyes-open was first trained as the neutral BCI class. All participants were then asked to strongly imagine a commonly performed (daily routine) movement for each hand (left and right) without actually moving their limbs. A “left” and “right” BCI class was trained accordingly, 20 times each. Always the “left” class was trained first and training was conducted in blocks of five training cycles. Each cycle consisted of 8 seconds of continuous recording of the mental state (“training”) and 2 seconds of rest (Figure
The training procedure of the qualitative assessment experiment.
The participants rested for 10 minutes after the KMI control trials and then the VMI modality was trained. The training procedure was the same but instead of imagining a movement, during the “training” cycle, a video played on the TV monitor (left or right forearm pronation). Again when the VMI training was concluded, 2-minute rest intervened before the participants attempted to control the robotic arms.
The participants attempted to control the “elbow/wrist” rotational DoF of each robotic arm. First they attempted to move the right robotic arm 10 times with the “right” BCI class and then the left robotic arm 10 times with the “left” BCI class. Each trial cycle lasted 10 seconds with a 2-second rest between them and a successful trial was marked by any detection of the correct BCI class during the 10-second period. When the correct class was detected the relevant DoF moved (corresponding to pronation), while it remained idle otherwise.
For the control trials using KMI, only a command was given to the participants to attempt to control the robotic during the trial cycle. For the control trials using VMI, during the trial cycle, on the TV monitor the same video that the participants were trained to played and no command was given (Figure
Overview of robotic arm control trials during the qualitative assessment experiment.
The second of the two illustrative experimental applications is an ongoing pilot study that involves SCI patients and healthy individuals controlling multiple DoFs of the robotic arms, as well as an investigation of their brain connectivity [
Three SCI patients were already recruited for participation in the pilot study, one female (28 years old) and two male (52 and 47 years old), as well as three age and sex matched healthy individuals as control group. The patients’ neurological level of injury was C4, C4, and T7, respectively and their Asia Impairment Scale classification was D, C (incomplete injuries), and A (complete injury), respectively. The protocol involves a full neurological examination using the International Standards for Classification of Spinal Cord Injury [
While under high-resolution EEG recording (as described in Section
28-year-old female SCI patient participating in the pilot investigation: (a) 1st part of the experiment, 128-channel EEG recording during oddball presentation of multiple limb movements (visual imagery); (b) 2nd part of the experiment, control of robotic arms using a commercial EEG-BCI headset, employing mental rehearsal of movements (kinesthetic imagery).
In the second part of the experiment, the participants used the commercial EEG-BCI to control the robotic arms. Three BCI classes were trained: resting state, left, and right. The participants were asked to visualize the videos they were presented during the previous part during the training of left and right. Each direction was trained 20 times, each cycle lasting 8 seconds, followed by 2 seconds of rest.
After the system’s training to each participant’s brainwaves, they were asked to follow presented instructions, corresponding to specific DoFs of the robotic arms and to specific direction of movement. The participants attempted to visualize the same movements to achieve control (KMI), without moving their limbs, while the BCI detected one of the three aforementioned classes. The presentation followed a pseudorandom routine that included an instruction to perform each of 32 possible arm movements once. Each instruction lasted 30 seconds, followed by 5 seconds of rest period. The participants’ performance in each movement was rated on a 0–5 scale and an overall percentage score was calculated to denote overall BCI performance.
Regions of interest (ROIs) for connectivity analysis at the cortical level: (a) midline surface, left hemisphere, (b) top view, both hemispheres, and (c) lateral view, right hemisphere. (1): SAC, (2): S1F, (3): S1H, (4): S2, (5): CMA, (6): M1F, (7): M1H, (8): M1L, (9): SMA, (10): pSMA, (11): PMd, and (12): PMv.
The Godspeed questionnaire consists of five semantic differential scales, equipped with Likert type scaling evaluating the attitude towards robots in the subcategories of Anthropomorphism, Animacy, Likeability, Perceived Intelligence, and Perceived Safety [
As with the original one, in the translated version, each semantic differential scale represents a key concept enclosing a short questionnaire. Each short questionnaire results in a score adding the ratings of the respondent. However, in the last two questions of Perceived Safety subcategory reversed rating was used, to associate the lower scores to the negative assessment, as is the case with the other items of the questionnaire [
Participants achieved higher median skill training percentage using KMI. That for the left arm was 26.5% (1st training block), 56.5% (2nd), 75.5% (3rd), and 72.5% (4th) for KMI and 20,5%, 54.5%, 71.5%, and 73%, respectively, for VMI. For the right arm it was 14.5%, 26%, 36%, and 24.5% for KMI and 8.5%, 17%, 17,5%, and 14.5% for VMI. There was a fatigue effect, median skill training percentage dropping from 3rd training block to 4th in all settings but left arm VMI.
Statistical testing resulted in significant difference between KMI and VMI skill training score only for the right hand extracted by training block 1 (
Median success score was 7 for both left and right arm VMI, 5.5 for left arm KMI, and 5 for right arm KMI (Figure
KMI against VMI success scores for 24 participants above action power threshold. Most participants performed better with VMI but the difference was not statistically significant.
While participants appeared to perform better using VMI rather than KMI as an imagery modality for BCI control, our analysis did not prove a statistically significant correlation [
Our experimental paradigm allows control of multiple DoFs of two robotic arms using a 3-class BCI implementation along with VMI training and the use of AI algorithms. As we have also shown in the proof-of-concept [
Performance in BCI control of patient and healthy participants in comparison to (a) their g-SCIM-III total score, (b) VVIQ score, and (c) Godspeed total score. Also (d) evaluation of the robotic arms in each separate Godspeed subcategory by participant.
As this is an ongoing investigation and subject recruitment continues, we hereby only provisionally present results from connectivity analysis, while a comprehensive assessment of performance, psychometric evaluation, and functional connectivity will be performed with the conclusion of the study. Healthy participants scored 77, 75, and 56 (out of max 80) in the VVIQ questionnaire, while SCI patients scored 54, 69, and 72 (Figure
In Figure
Functional connectivity networks formed in alpha brainwave band during different visual motor imagery tasks performed by an SCI patient and a sex and age matched healthy control participant (connections > 60% max power displayed).
Functional connectivity holds promise in classifying imagery of multiple classes (multiple different movements) or complex motions, based on imagery modalities. A possible automated approach would be to identify significant connections for each task using Network-Based Statistics (NBS). In our opinion, semiautonomous algorithms and AI should be part of a strategy to control multiple DoFs of robotic arms. Our BCI approach uses a 3-class implementation to achieve control of many (32 possible) DoFs but currently relies on research intervention. The low-class approach employed could be feasible both for BCI training and neurophysiological investigation. While training and functional connectivity study is performed using high-resolution EEG, it is highly impractical to use such systems for everyday BCI applications. Therefore, we aim to downscale the findings from high-resolution EEG regarding functional connectivity to control features for commercial low-resolution EEG-BCI headset. Moreover, other investigations could include trauma-induced brain reorganization with a focus on possible rehabilitation opportunities.
A natural milestone for future development is the integration of the BCI and robotic arms system into the operator’s perceived body mental image [
Another important aspect of perceived body mental image that needs to be taken into account in further development is anthropomorphism. In our current technological generation, user perception of the robotic arm, as measured by Godspeed, did not correlate with performance [
There are several paradigms for sensorimotor BCI implementation that vary from machine learning to signal processing perspective. Current BCIs are capable of easily recognizing two classes, which translates to control of 1 DoF but usually fails to work with more classes. One of the biggest challenges of noninvasive motor imagery BCIs is the low spatial resolution of EEG, due to volume conduction effect [
Cortical current density estimation methods can be deployed to compensate for the low spatial resolution of EEG, by reconstructing activation of cortical sources using EEG data and realistic head model, so essentially transforming sensor data to a higher dimension space, where spatial resolution is higher. Several studies concluded that features extracted from source space are superior over sensor based [
One of the strongest requirements of Mercury BCI algorithm is natural control of a multi-DoF robotic arm, corresponding to multiclass in terms of decoding. Decoding brain activity is still an open challenge especially when it turns to multiple classes [
Such a processing pipeline is highly computational demanding, and at this stage of its development we work with offline analysis until the results are encouraging to proceed to real-time implementation. Recent advancements in Graphic Processing Units (GPUs) and Field-Programmable Gate Arrays (FPGAs), which have proven to be effective in rendering computationally demanding applications in real-time, could be employed for an online implementation of our paradigm.
High-resolution EEG data and its analysis for functional connectivity from multiple motor imageries are expected to provide insight in brain network adaptive and maladaptive reorganization that occurs after SCI [
In the context of our design and experiments, we encountered several limitations. First of all, although demands for portability, ease of use, low cost, and availability made the selection of a commercial dry electrode EEG headset necessary, the accompanying commercial (and undisclosed) BCI algorithm did not meet the needs of our neurophysiological experimentation [
Advances in BCIs have demonstrated remarkable results in the direction of replacing and restoring lost sensorimotor function in human patients. Novel paradigms and recent advances in noninvasive BCI protocols aim at progressively improving past technological and neurophysiological limitations. Neurophysiological changes in the brain network level, induced by SCI, could prove critical in designing and developing robust and durable noninvasive BCIs for motor restoration and rehabilitation. Moreover, successful rehabilitation strategies should take into account user perception, satisfaction, and overall experience, alongside performance. We presented our implementation of BCI-controlled 8-DoF anthropomorphic robotic arms, using noninvasive off-the-shelf BCI technology. Moreover we presented two illustrative experimental applications on healthy individuals and SCI patients. Current, state-of-the-art, BCI technology is unable to control multiple DoFs but semiautonomous AI algorithms and connectivity-based BCIs could provide solutions towards that direction. Individual differences appear to play a role in motor imagery based BCIs and multiple training sessions are always encouraged in order to improve performance in robotic arm control. Functional connectivity holds promise in classifying imagery of multiple classes (multiple different movements) or complex motions, based on imagery modalities. Future development aims at facilitating the integration of BCI and robotic arm system into the operator’s perceived body mental image, thus requiring rapid, fluid, accurate, predictable system performance and improved anthropomorphism. Online implementation of connectivity-based classifiers, although currently too computationally demanding, is expected to be soon feasible. High-resolution EEG data and its analysis for functional connectivity from multiple motor imageries are expected to provide insight in brain network adaptive and maladaptive reorganization that occurs after SCI and, subsequently, into promoting or preventing it accordingly [
This study was conducted in accordance with the Declaration of Helsinki (1964) and its following amendments. The institutional Ethical Committee approved the study.
All experiments were conducted with the subjects’ understanding and written informed consent.
The Cervical Spine Research Society-European Section (CSRS-ES) had no involvement in the study design, writing, or decision for the submission of this paper.
The authors declare that there are no conflicts of interest regarding the publication of this article.
This study was conducted as part of the development of the project CSI:Brainwave, which was partially supported by a “Mario Boni” Research Grant, awarded by Cervical Spine Research Society-European Section (CSRS-ES). The project continues by funding of the Lab of Medical Physics through the LLM care self-funding initiative (