Brain-computer interfaces (BCIs) are interactive systems that aim at providing users with an alternative way of translating their volition into control of external devices. Their most popular applications lie within the scope of rehabilitation and motor restoration for patients with severe neurological impairment [
The use of brainwaves to control robotic devices has produced promising clinical results in terms of feasibility [
Given these recent technological advances, we have focused our research efforts in noninvasive, minimally intrusive, and low-cost BCI. We have designed, partly implemented, and tested an electromechanical robotic system to investigate the capabilities and limitations in combining these technologies for biomedical applications [
Design requirements included fast robotic movement that approximates the natural movement of a human operator’s arm, an intuitive and immersive interface, portability, potential for further development, scalability, and relatively low cost (i.e., less than $3000). We targeted investigative comparative neurophysiological scenarios in which an operator remotely controls a 6-degree-of-freedom (DOF) robotic arm using their arm movement, their brainwaves, or both.
The hybrid system under development comprises two input devices and a robotic arm. The input devices are an exoskeletal position sensing harness (EPSH) and a commercially available dry electrode BCI headset. The robotic arm is the actuator device in our system and was also developed entirely by the authors. There is no feedback to the human operator in the proof-of-concept prototype described in this system.
The human arm, excluding the hand, possesses the ability to move along 7 DOF: pitch, yaw, and roll at the shoulder joint, pitch and roll at the elbow, pitch and yaw at the wrist. Only three DOF are needed to move the hand at a particular point in space, while the remaining 4 DOF permit humans to approach and grasp objects from different angles.
In order to simplify the design of both the EPSH and the robotic arm, we made the choice to omit wrist yaw, since it does not significantly affect the ability of the robot to manipulate objects. Still, a 7th DOF was added in order to operate the gripper.
All parts were designed using computer-aided design (CAD) software, manufactured using computer numerical control (CNC) and laser cutting machines, and assembled and tested by our team.
An EPSH was developed as a form of minimally intrusive, intuitive, and immersive interface for the robotic arm. It copies the operator’s arm movement with measurable and repeatable accuracy (Figure
CAD diagram of the 6-DOF exoskeletal position sensing harness. The curved arrows show the axis of 7 axes of rotation (the 7th DOF is used to control the robotic gripper). The harness is worn around the human operator’s arm.
The harness sensors and accompanying electronics provide real-time data on the position of the human operator’s joints from wrist to shoulder. Hand and finger position is not sensed; however, a finger-operated switch allows the user to operate the robotic arm’s gripper in order to pick up and release items.
Copying of the human operator’s movements is achieved through the EPSH which is worn around their arm. The harness measures the angles between the different parts of the arm and hand. Electronic output originating from the sensing harness is passed on to the robotic arm’s control circuit, which also takes into account feedback output from the robotic arm itself. During the design phase, we, therefore, faced a classic automation closed-loop control challenge: the input is provided by the EPSH worn by the operator, the robotic arm is the recipient of the control output, and a feedback loop takes into account the actual position of the arm.
The robotic arm is capable of the 6 following types of movement. 2 DOF for the “shoulder” joint (“right–left” and “up–down”). 1 DOF for the “elbow” joint. 1 DOF for the “wrist” joint. 2 DOF for rotation between the “shoulder–elbow” and “elbow–wrist” parts.
The robotic arm (Figure
CAD diagram of the 6-DOF robotic arm and gripper. The 6 dark cylinders are DC motors, each accommodating a degree of freedom.
The selection of a commercially available BCI headset depended on the number of sensing channels, signal quality, price, and ease of use [
Furthermore, the ideal BCI headset would need to meet the requirements of multiple data acquisition channels, low weight, and low cost. Frequency-based automatic classification of mental states by the hardware device and the ability to export the raw EEG signal were considered, the former being a strong factor for preference, the latter a decisive requirement. The maximum number of automatically detected mental states was not considered a priority, since this feature can be provided by the signal processing capabilities of our experimental setup. Still, the capacity of a commercial system to automatically detect multiple mental states was considered an indirect indication of the quality and breadth of its sensing capabilities. Thus, it was deemed a desirable—albeit less significant—advantage.
For this reason we examined two low-cost, commercially available headsets (Figure
The two commercially available dry electrode BCI headsets considered. The NeuroSky MindWave (a) and Emotiv EPOC (b).
Both devices export raw EEG as well as processed, automatically classified mental state data. Our choice between them depended on their sensing capabilities where the NeuroSky MindWave uses one sensor that can provide only three values: attention, meditation, and eye blinking. The Emotiv EPOC uses a series of 16 sensors, which are capable of detecting specific conscious thoughts, levels of attention, facial expressions, and head movements (the latter using the embedded gyroscope). The sampling frequency of the Emotiv EPOC is 4 times greater than the NeuroSky MindWave making it comparable to more complex and expensive virtual rehabilitation EEG devices [
The drawback of both devices is the occasional unreliability of signal quality, primarily associated with the use of dry electrodes. For this reason, the designers of the Emotiv headset suggest that users further improve skin conductance by the moistening of the sensors using a saline solution. While this procedure is not ideal for our purposes, it was considered the least inconvenient among commercially available low-cost solutions.
Despite this drawback, we selected the Emotiv EPOC for use in our hybrid system design, since it integrates the largest number of sensors at the highest sampling rate among all portable low-cost BCI headsets available in the market at the time this paper was submitted.
In the MERCURY hybrid setup, frequency-based EEG data classification is performed both inside the Emotiv EPOC device [
The first prototype exoskeletal position sensing harness, which is worn around the extended arm of a human operator. All parts used for the assembly were manufactured by the developers to design requirements.
Development of the first two components of the hybrid system, the robotic arm and EPSH, has recently been completed. Validation and characterization tests have been performed in order to measure response times, angular velocity and acceleration, maximum payload, and power consumption (Figure
The first prototype robotic arm developed by our team. It is capable of moving at angular speed comparable to the natural motion of a human operator’s arm. It comprises 6 DC motors (one for each DOF) and 2 servomotors for the gripper.
The average response time of M2, the motor operating within the shoulder joint of the robotic arm, was measured to be 120 ms ± 10. This was measured using an oscilloscope, measuring (10 repetitions average) the drop in current consumption by the motor as soon as the rotor started rolling. This motor is the slowest in the robot, so this measurement is used to formulate worst-case scenario comparisons. The aforementioned value was obtained using an oscilloscope to measure the initial setup time before the motor starts accelerating continuously. Anecdotal evidence from initial tests indicates that this delay is hardly noticeable by human operators.
A typical range of values for the average angular velocity of a human arm has been reported in the literature [
We experimentally measured the average angular velocity of motors M2 (shoulder) and M4 (elbow) of our robotic arm throughout their full range of motion, 180° and 150°, respectively [
The experiments were set up so that the robotic arm be in an upwards movement (impeded by gravity) and were performed twice, with and without an additional 50 gr load. The robot input was to move as fast as possible, and multiple measurements were made in order to obtain average values. Results are summarised in Table
Experimental results of average angular velocity and acceleration measurements from the robotic arm. Motor M2 moves the shoulder joint, M4 moves the elbow. The load used was 50 gr.
Avg angular velocity (°/sec) | Avg angular acceleration (°/sec2) | |
---|---|---|
M2 (free) | 108 ± 1 | — |
M2 (load) | 102 ± 1 | 1535 ± 10 |
M4 (free) | 134 ± 1 | — |
M4 (load) | 128 ± 1 | — |
Even though the experimental results are not directly comparable, focusing on the slowest response times of the robot—a deliberately pessimistic scenario—leads to some useful conclusions. The robotic arm is capable of combining speed and acceleration that exceed the typical range of regular human arm movements [ is capable of accelerating faster than the human arm, has an average speed reduction of less than 6% when carrying a small 50 gr load (averaged across the full range for motion for any joint), has a qualitative attribute, not quantified yet, informally indicated by testing engineers: motion is smooth and the control is immersive and “feels natural”; combined with the EPSH and the BCI headset, the entire integrated device is portable and suitable for carriage by a single person.
Complete qualitative assessment studies are planned in the immediate future, prior to the comparative experiments mentioned in following sections.
The EPSH measures 21 × 21.4 × 69 cm and weighs 3 kg. Including its wooden base, the robotic arm weighs 5.9 kg. When extended vertically, the robotic arm measures 25 × 73.9 × 30.2 cm. The moving part of the arm measures 25 × 12.5 × 46 cm. Its working space is approximately a hemisphere with a radius equal to its reach (46 cm), approximately 60% the reach of an adult human arm [
The robotic arm prototype has a rated maximum payload of 300 gr. The maximum payload is 750 gr. These figures are a direct consequence of striving to maintain low development cost and can be dramatically improved in future prototypes. No tests were performed with loads greater than 750 gr in order to prevent damage to the prototype.
The robotic arm is powered by a 24 V DC power supply. The peak current is 3.5 A ± 0.1, and the average power consumption is 25.3 W ± 0.1. All measurements were made with gravity impeding movement without additional load.
With respect to electromechanics and electronics, the next development steps involve integrating the selected dry electrode headset with the EPSH and robotic arm into a hybrid device. A PC will be capable of recording experimental sessions in which the operator uses the headset, the harness, or both to control the robotic arm. Once experiments are concluded with the hybrid setup, there are further plans to introduce a feedback loop so that the operator gains a tactile feeling of resistance when the robotic arm touches an object or obstacle.
With respect to software development, MATLAB-based (the MathWorks Inc., Natick, Massachusetts, USA) software code will process the output of the BCI headset, evaluate it against precise motion data captured by the harness and arm, and draw comparative conclusions. Our team is interested in comparing the BCI output to that of a conventional 10/20 EEG data acquisition system as well as to the output of the EPSH.
The novelty of our experimental setup is the ability to capture all 6 degrees of freedom of the human arm’s physical movement electromechanically, at low cost, with relative accuracy, and in real time. Furthermore, we plan to proceed with comparing our results to those acquired through the use of both a commercial BCI headset and a more conventional multichannel EEG-BCI paradigm, while the subject actually performs motor execution tasks. It is also among our research plans to comparatively investigate the role of virtual reality (VR) in controlling a virtual prosthetic arm versus controlling an actual robotic arm.
A series of neurophysiological experiments involve comparison of brain activation results between scenarios of motor execution (ME) versus motor imagery (MI) [
The role of functional connectivity (FCN) of the brain in the fluid motions of the arm is also within our scope of experimentation. We wish to explore whether graph analysis of FCN during ME and MI of the arm can contribute towards the goal of making BCI systems more intuitive, easy to learn and easy to use.
A hybrid man-machine interface (MMI) and brain-computer interface (BCI) systems offer numerous investigative research advantages. Fluid and intuitive control of a prosthetic robotic arm using BCI is yet to be demonstrated, due to limitations of current BCI data acquisition and classification technologies. Those limitations could be addressed using the added benefit the MMI-control parameter for the robotic arm and artificial intelligence classifiers. The main question to be answered, as far as this hybrid system is concerned, can be identified in whether the unintuitive, hard-to-adjust-to and limited in functionality BCI systems which are currently commercially available can benefit from this approach. Are BCIs after all destined to be exclusively used for research and clinical purposes or could they evolve into a mature mainstream technology?
We have developed an intuitively controlled 6 DOF robotic arm and accompanying operator’s sensing harness, satisfying the design requirements for an immersive, hybrid robotic control system. The first proof-of-concept prototype has been developed, evaluated, and deemed adequate for the next development step: integration with a commercial dry-electrode BCI headset. The intended research objectives for this system include BCI optimization through comparative experiments, using the motion and position sensing harness, the dry electrode BCI headset, and a combination of both. The ultimate research goals are to better understand the function of the motor cortex, improve neurofeedback training for people suffering from neurological disorders, and optimize robotic prosthetics.
The research leading to these results has received funding from the European Union’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no 288532. For more details, please see