Control of a Robot Arm Using Decoded Joint Angles from Electrocorticograms in Primate

Electrocorticogram (ECoG) is a well-known recording method for the less invasive brain machine interface (BMI). Our previous studies have succeeded in predicting muscle activities and arm trajectories from ECoG signals. Despite such successful studies, there still remain solving works for the purpose of realizing an ECoG-based prosthesis. We suggest a neuromuscular interface to control robot using decoded muscle activities and joint angles. We used sparse linear regression to find the best fit between band-passed ECoGs and electromyograms (EMG) or joint angles. The best coefficient of determination for 100 s continuous prediction was 0.6333 ± 0.0033 (muscle activations) and 0.6359 ± 0.0929 (joint angles), respectively. We also controlled a 4 degree of freedom (DOF) robot arm using only decoded 4 DOF angles from the ECoGs in this study. Consequently, this study shows the possibility of contributing to future advancements in neuroprosthesis and neurorehabilitation technology.


Introduction
Brain machine interface (BMI) is a communication tool for quadriplegia between the brain and external devices such as a robot [1]. Since electroencephalography (EEG) has high temporal resolution, prominent EEG-based studies have been used in various paradigms, such as a computer cursor control [2], direction intention of hand movements [3,4], a P 300 speller device [5], and neurofeedback systems for rehabilitation [6][7][8][9]. ese noninvasive methods are useful for classification of movement direction or intention. e prediction of time-varying trajectories, however, is very difficult because of insufficient spatial resolution and low signal-to-noise ratio.
After the first electrocorticogram-(ECoG-) based BMI succeeded in one-dimensional cursor control in humans [10], it has come into the spotlight as an alternative recording approach for less invasive BMIs because ECoG signals could offer higher spatial resolutions than the classical scalp EEG signals. ECoG signals have also shown direct potential as a stable in long-term recording method [1,[11][12][13][14]. Many studies using ECoG have already succeeded in the classification of movement direction [15,16], grasp type [17], and the prediction of hand trajectory [11][12][13]18]. Our previous studies based on the brain rhythmic bands with SLiR algorithm also have been reported to predict muscle activities and arm trajectories from ECoG signals [13,14,[19][20][21][22].
Despite these successes, however, there still remain works for the realization of ECoG-based BMIs. e brain and CNS system allows skillful manipulation of the body to interact with the external environment. is sophisticated and flexible operation involves intrinsic (kinetic) information such as force and stiffness, and as well as extrinsic (kinematic) information such as desired angle or velocity.
e neuromuscular system naturally modulates mechanical stiffness and viscosity in order to obtain proper interaction force to the environments. Since stiffness, viscosity, and force change with our muscle activation in order to interact with environments, decoding muscle activities, as well as joint angles, are important components for realizing neuroprosthesis capable of controlling interaction force or stiffness. Although many studies have achieved to decode arm trajectory, a few studies have demonstrated that muscle activity could be decoded from spike signals [23][24][25], EEG [26,27], and ECoG [19,22]. e purpose of this study is to propose a basis concept for a neuromuscular BMI system. e schematic outline of this concept is shown in Figure 1. A well-trained Japanese monkey performed a series of reaching, grasping, pulling, and releasing movements. We simultaneously recorded 16 ECoG signals of the primary motor cortex (Ml) and 12 electromyography (EMG) signals in the right arm. We also measured and calculated joints angle using a 3D marker tracking system. We decoded three joint angles for the shoulder, elbow joint angle, and muscle activities from ECoG signals using sparse linear regression (SPR). Our results indicate that ECoG signals from primary motor cortex involves both intrinsic (kinetic) and extrinsic (kinematic) information. We could also predict multiple muscle activities (intrinsic) and joint angles (extrinsic) from ECoG signals simultaneously. In addition, we controlled a 4 degree of freedom (DOF) robot arm using decoded joint angles and muscle activation in offline simulation.

Behavioral Task.
All experimental procedures were performed in accordance with the Guidelines for Proper Conduct of Animal Experiments of the Science Council of Japan and approved by the Committee for Animal Experiment at the National Institutes of Natural Sciences (Approval No. 11A157). e animal welfare and steps taken to ameliorate suffering were in accordance with the recommendations of the Weatherall report, " e use of nonhuman primates in research. " We note that we reused our database reported in our previous work [19] for verifying the possibility of realizing neuromuscular BMI device. We explain briefly experimental setup in this paper as follows. A Japanese macaque (female, at 4.7 kg) was trained to perform reaching and grasping tasks with the right hand as shown in Figure 2. First, the monkey placed her hand on a home button located in front of the chair. Second, the monkey tried to reach for the knob after a "go" cue was given in a beep sound. ird, the monkey then had to pull the knob and release. When the monkey successfully pushed the home button and pulled the knob to the required displacement (6 cm), it received a juice reward. We extracted continuous data (total length; 500 s) from our database to control the robot arm. ese data involved a total number of 248 trials. Each trial duration averages and standard deviations (STD) were 1.16 ± 0.29s.

Data Recording.
A platinum ECoG array (Unique Medical Corporation, Tokyo, Japan), which had 16 (4 × 4 grid) channel electrodes was implanted over the left primary motor cortex (M1) as shown in Figures 2 and 3.
EMG signals of the right forelimb muscles were recorded from chronically implanted pairs of multistranded stainless steel wires (Cooner Wire, Chatsworth, CA, USA). ey were subcutaneously tunneled to the following target muscles: adductor pollicis (AP), abductor pollicis longus (APL), flexor digitorum profundus (FDP), and extensor digitorum communis (EDC) for hand muscles; flexor carpi ulnaris (FCU), and extensor carpi radialis (ECR) for wrist muscles; brachioradialis (BRA) and triceps lateral head (TRA) for elbow muscles; biceps long head (BIL) and triceps long head (TRO) for shoulder-elbow double joint muscles; and pectoralis major clavicular head (PECM) and deltoid clavicular part (DELP) for shoulder muscles. ECoG and EMG signals were sampled at 4 kHz. e 3-D positions were recorded using the optical motion capture system (Eagle digital system; Motion Analysis Corporation, Santa Rosa, CA). e system used twelve infrared cameras operating at 200 frames/s to track the positions of multiple reflective markers (4 mm-diameter spheroids). A total of fourteen markers were attached to the right arm of the monkey from the shoulder to the fingers. e motion data were down-sampled to 100 samples per second.

Preprocessing of ECoG and EMG Data.
ECoG signals were preprocessed with our previously proposed method [19]. First, the signal data were re-referenced with a common average reference. Second, each ECoG signal was divided into nine frequency bands (δ: 1.5-4 Hz; θ: 4-8 Hz; α: 8-14 Hz; β1: 14-20 Hz; β2: 20-30 Hz; c1: 30-50 Hz; c2: 50-90 Hz; c3: 90-120 Hz; and c4: 120-150 Hz) using fourthorder bandpass Butterworth filters. Second, the bandpass filters split 16-channel ECoG signals into nine band-passed signals to produce M channels of bandpass filtered signals x i (t). ird, these band-passed signals were digitally rectified and smoothed with a Gaussian filter (width: 0.1 s, σ: 0.04 s).   Computational Intelligence and Neuroscience Fourth, the obtained signals at time t were normalized to the standard z-score as follows: where μ i and σ i are the mean and the standard deviation of x i , over a 2 s interval before the time t. Finally, the signals were down-sampled to 100 Hz to match the motion data. EMG signals were rectified and passed through a fourthorder lowpass filter with a cutoff frequency of 4 Hz and further down-sampled to 100 Hz, resulting in muscle activation [28,29].

Prediction of Muscle Activation and Joint Angles from
ECoG Signals. We used the Variational Bayesian Sparse Regression toolbox [30] to decode muscle activation and joint angles. e decoded muscle activation pEMG k (t) and joint angles pY h (t) at time t, are described as where ω k ij and ω h ij are the weight coefficient of the k-th muscle and h-th joint angle for the i-th signal source at a delay time jΔt, ω k 0 and ω h 0 are the bias terms, x i (t) is the i-th ECoG source at time t, and Δt is a discrete-time step-size of 20 ms. e muscle activity at time t was predicted using 10 time points (N � 10) starting 200 ms before the target time t.
e joint angles at time t were predicted using 25 time points (L � 25) starting 500 ms before the target time t.
2.5. Analysis. Accuracy of the predictions was evaluated using a 5-fold cross validation. e entire 500 s of experiment data were divided into two parts, 400 s of training data and 100 s of test data. We calculated the coefficients of determination (R 2 ) to evaluate the similarity between actual and predicted muscle activities. Accuracy was also evaluated using normalized rootmean-square error (nRMSE) between actual and predicted.
2.6. Robot Arm. We specially designed a life-sized robot arm for monkey to reproduce the movements of arm as shown in Figure 3. For the purposes of this study, the robot arm is modeled as a four DOF kinematic linkage, consisting of two links i.e., upper arm and forearm along with the hand and two joints i.e., shoulder joint and elbow joint, with a fixed wrist joint. e shoulder joint has 3 DOFs (S1: abduction/adduction; S2: flexion/extension; S3: rotation), and the elbow is simplified as a revolute joint with 1 DOF (E1: flexion/extension). e upper arm and forearm are driven by the brushless DC servo motors with 512 ppr encoder (Faulhaber Motors: 2342S024CR). e PID controller is implemented on the base, and basic trajectory tracking tasks were executed in joint space.
We also made an offline simulator. We made ECoG database at a 100 Hz sampling time for the simulator. e simulator reads 10 time points of ECoG data per cycle. We used first-in-first-out (FIFO) method with 2 s data length. e simulator calculates the desired joint angles and sends them to the control computer using TCP/IP protocol. en, the control computer can calculate and update θ ref in the PID controller on the robot arm with at a 200 Hz sampling rate ( Figure 3): where θ des and θ pre are the desired angle and the present angle and θ ref is input value for robot arm. K gain should be controlled by the intrinsic information i.e., stiffness. We used an index of muscle cocontraction around the joint (IMCJ) instead of stiffness. Osu et al. reported the linear relationship between IMCJ and joint stiffness measured by the perturbation [29,31]. We defined shoulder gain K s (t) and elbow gain K e (t) as follows: where β s and β e are constant gain and adjusted manually for stable movements.

Prediction of Muscle Activation.
We applied the decoding model to continuous test data. One typical example of continuous prediction is shown in Figure 4., where the prediction was stable even for repetitive trials over 100 s.  Figure 3: Algorithm flowchart of control of a robot arm using decoded joint angles and muscle activation from ECoG signals. We note that the results showed the similar pattern in the agonistic muscles and also even the antagonistic muscles such as TRO, ECR, and EDC. e arrows shown in Figure 4 indicated local difference between measured and estimated EMGs. ese differences have appeared occasionally in this work. is measuring noise might be a result from the crosstalk among the EMG electrodes or the wobbling electrode during the monkey's motion. Although the results involved some failures, these results clearly show that the proposed method could realize neuromuscular BMI system in an online fashion.

Computational Intelligence and Neuroscience
In the results of the 5-cross validation, means and standard deviation of R 2 and nRMSE for each muscle ranged from 0.0199 ± 0.0062 (ECR) to 0.6333 ± 0.0033 (FDP) and 0.1303 ± 0.0053 (APL) to 0.1825 ± 0.0098 (FCU) as shown in Table 1. e median and standard error of R 2 and nRMSE were 0.3765 ± 0.0164 and 0.1527 ± 0.0028, respectively. In the case of TRO, R 2 value was worse but nRMSE was good among muscle prediction. It might be the reason that the hardly used muscle such as TRO had small range of activation. e bold numbers in Table 1 show the best value among the test data.

Prediction of Joint Angle and Robot Control.
We also predicted joint angles from the ECoG signals with the SPR model. e desired angles calculated from offline simulation showed good correlation with the actual angles of movements as shown in Figure 5. In the results of the 5-cross validation, means and standard deviations of R 2 and nRMSE of angle prediction between the measured angles and the predicted ones were from −0.0333 ± 0.3547 (S3) to 0.6359 ± 0.0929 (S2) and from 0.1261 ± 0.0111 to 0.1596 ± 0.0286, respectively (Table 2). e median and standard error of R 2 and nRMSE were 0.6171 ± 0.1415 and 0.1373 ± 0.0083, respectively. e bold numbers in Table 2 also show the best value among the test data. ese results suggest that the arm movement can be estimated from ECoG signals.
e accuracy values of S1 (shoulder abduction/adduction), S2 (shoulder flexion/extension), and E1 (elbow flexion/extension) were higher than those of S3 (rotation). It could not affect the robot control because the range of shoulder rotation had small. (see also video S1 about the robot arm).

Discussion
Since ECoG signal is the origin of EEG signals, we tried to divide ECoG signals into specific brain rhythmic bands based on the traditional EEG studies. In addition, Chen et al. [13] also reported that the brain rhythmic based method produced the same or better results than the nonphysiological fractionized frequency method. e delta and gamma bands were superior to the other bands in this study. Our previous studies [15-17, 19, 20, 27] also reported that delta and gamma bands have plentiful information to estimate the trajectories and force. However, the weights of all frequency bands disappeared after applying the SPR algorithm. is phenomenon stood out in intrinsic (kinetic) prediction. is might indicate that all sensorimotor rhythms of ECoG are needed to predict EMG signals and angles.
Current rehabilitation robots can perform sophisticated operations including stiffness control [32,33]. e human musculoskeletal system has stiffness and viscosity properties essential to interaction with our surroundings. e perturbation method has been used to measure stiffness of the human arm with a manipulator. Since the results are the averages of many experimental trials, stiffness could not be measured in time series. ere are some trials that estimate stiffness using EMG signals because stiffness changes with muscle activation. Osu and Gomi tried to rebuild joint stiffness from EMG signals using conversion factors to match the EMG to the measured stiffness using PFM [34]. Osu et al. [31] consequently proposed IMCJ that was defined as the summation of absolute values of muscle quasitension. ese earlier models used parameters or "gains" with no physiological basis, thus compromising constructional validity. Our previous studies suggested the myokinetic (Mykin) model which can estimate the angle, torque, and stiffness of joints from muscle activities [25,29,33]. In this study, we used IMCJ instead of Mykin model, because Mykin model should need the parameter calibration from the relationship between joint torque and muscle activation. Nonetheless, we could control the robot arm using both angles (extrinsic) and IMCJ (intrinsic). erefore, decoding muscle activity and joint angles is an important component for realizing BMI systems capable of controlling interaction.

Conclusion
is study describes the prediction muscle activities and joint angles from ECoG signals. We displayed a novel attempt to control the 4 DOF robot arm using the decoded

Data Availability
e data used to support the findings of this study were supplied by prof. Nishimura and prof. Isa under license and so cannot be made freely available. Requests for access to these data should be made to prof. Nishimura (yukio@nips. ac.jp).