Exploration of Muscle Fatigue Effects in Bioinspired Robot Learning from sEMG Signals

1School of Computing, Electronics and Mathematics, Plymouth University, Drake Circus, Plymouth PL4 8AA, UK 2Shenzhen Research Institute, The Chinese University of Hong Kong, Shatin, Hong Kong 3Key Laboratory of Autonomous Systems and Networked Control, College of Automation Science and Engineering, South China University of Technology, Guangzhou 510640, China 4School of Automation, Beijing Institute of Technology, Zhong Guan Cun South Street, Haidian District, Beijing 100081, China 5Changzhou Key Laboratory of Robotics and Intelligent Technology, Changzhou 213022, China 6Jiangsu Key Laboratory of Special Robots, Hohai University, Changzhou 213022, China 7College of IoT Engineering, Hohai University, Changzhou 213022, China


Introduction
Muscle fatigue is a complicated phenomenon which is relevant to the functionality of muscles.Generally speaking, it is the decline in ability of a muscle to generate force, which could be a result of excessive exercise.According to the physiological mechanisms causing fatigue, there are two classes of muscle fatigue; among them, neural fatigue is due to the limitations in generating sustained signal by the nerve, while the metabolic fatigue is caused by the falling contraction capacity of muscle fibre.Fatigue limits the sport performance of people.Research found that fatigued muscles produce reduced voluntary force, throwing velocities, and kicking power.Meanwhile, less accuracy and endurance capacity in performing these activities has been reported by Montgomery et al. [1] and Knicker et al. [2].Pathologists take fatigue assessment as crucial information source of disease progression for diagnosis and treatment.In clinical practice, nowadays qualitative measurements like subjective questionnaires and clinical rating scales are taken as the main protocols of assessing fatigue (Féasson et al. [3] and Mcdonald et al. [4]).

Complexity
The EMG is a bioelectricity generated by muscle cells.It is known to reflect the activation of neurons controlling muscle contractions.sEMG signals can be collected by attaching surface electrodes on the skin where the target muscle is just beneath.Raw sEMG signals are usually first band filtered to remove noises and retaining the useful information.sEMG related features are characteristics describing the property of the signal in various forms, for example, the amplitude and frequency of the signal, which are computed within a finite time segment of the signal (Wang et al. [5]), the power spectrum (Phinyomark et al. [6]), and so on.Since its noninvasive nature, sEMG has been applied as a research technique to assess muscle recruitment in a variety of scenarios for about three decades.Fatigue generally causes increases in sEMG data, which relates to exercise/sport performance degradation (Carneiro et al. [7] and Beneka et al. [8]).The usefulness of mean frequency (MNF) and median frequency (MDF) of the sEMG signal has been widely investigated in clinical studies and engineering applications (Phinyomark et al. [6] and Mugnosso et al. [9]).To estimate the MNF and MDF of the sEMG signal, a Fourier transform of the autocorrelation of the sEMG signal, that is, the power spectrum or the power spectral density of the sEMG, is first obtained, usually through Periodogram method.The average frequency estimated from the power spectrum is defined as the MNF of the signal.MDF is the frequency that divides the total sum of the sEMG power spectrum into two segments that are equal in amplitude (Oskoei et al. [10]).It has been found that a decreasing MDF over time indicates fatigue (Hollman et al. [11]).Besides these frequency-based features, the other spectral variables which have been employed in sEMG signal analysis include the power spectrum, root mean square values, peak frequency, and the spectral indexes (Farina and Merletti [12] and Phinyomark et al. [6]).
With the advance of robot control strategies under various dynamic scenarios or with even uncertain manipulators, robots nowadays play more proactive roles in coordinating and collaborating with human beings in accomplishing complicated tasks, with skills such as haptic identification (Yang et al. [13]) and human-robot interaction (HRI) (Wolf et al. [14]).Neural interfaces have been introduced for user to control a robot or other intelligent systems through neural commands, such as brain-computer interfaces (BCIs) (Muelling et al. [15] and Yang et al. [16]) or via haptic interaction like teleoperation (Yin et al. [17]).These technologies have been widely employed in complex dynamic scenarios such as robot-assisted surgical operation (Taylor et al. [18]) and disability assistance and rehabilitation (Kaufmann et al. [19]).For tasks requiring physical movements, for example, in manufacturing, sEMG-based neural interface is widely adopted in HRI scenarios (Zucker et al. [20] and Yang et al. [21]).Knowledge and implementation skills transfer by sEMG to telerobot makes it controllable in a remote place and manageable when accomplishing hazardous jobs (Artemiadis and Kyriakopoulos [22], Fukuda et al. [23], and Yang et al. [24]).In our previous studies, sEMG signal has been employed to estimate robot joint stiffness and related muscle tensions (Wang et al. [5]).The sEMG amplitudes from only the very high frequency band have been engaged in impedance and force control, where desirable robot control performance has been achieved.Nowadays, the development of modern controllers has been employed in complex robotic systems, including telerobots.Controller parameters have been estimated to minimize the tracking errors in robot manipulation (Na et al. [25] and Wang et al. [26]).Adaptive methods like unknown input observer and time-varying parameter estimation have also been proposed and applied in complex systems such as vehicle engine system (Na et al. [27]).
TbD is another important HRI research subject.A TbD method is to teach a robot to perform specific tasks through showing the operation by an experienced demonstrator, at once or at multiple times.This flexible way of HRI enables the robot to acquire human-like manipulation skills without complicated steps and at the same time ensures the robot's adaptation to new working conditions (Calinon et al. [28] and Peternel et al. [29]).Nowadays, the taught tasks become more and more complex, for example, carrying on heavy load and obstacles avoidance (Zhang et al. [30]).In these dynamic scenarios, the demonstrator has to conduct multiple demonstrations, where muscle fatigue might occur.Yang et al. have made efforts to remove the tremor's influences on skills transfer when teleoperating a robot (Yang et al. [31]).However, in current TbD research, there has little fatigue-related information been considered.In the field of HRI, fatigue phenomena widely exist, and the challenge of investigating these effects lies in how to inspect and estimate them in dynamic environments.Unlike in clinical studies where fatigue could be measured by subjective ratings, for an engineering application, a systematic method to quantify one's developing fatigue condition over time is essential.On the other hand, the underlying fatigue mechanism is still under research, and their effects in many scenarios are quite subtle to detect, which increases the difficulty of examining them precisely.
In this paper, we present a fatigue-aware robot learning method where the human muscle fatigue issues have been taken into account in the human-robot interaction process.The sEMG data from the human demonstrator have been acquired through a MYO armband worn on the forearm.Fatigue-related parameters have been generated from the collected sEMG recordings thereby.These parameters are derived from the low frequency bands of the sEMG signal where muscle fatigue is generally examined.The instantaneous amplitude and frequency variables which are time-varying sequences of the subband sEMG signal have been estimated.These variables are viewed indicators of the fatiguing status of the demonstrator and are taken into consideration in the robot trajectory learning process.In a complex robot learning application requiring multiple human demonstrations, the detected muscle fatigue status forms the basis of weighing among the different trials.The final learning outcomes will then be generated from the weighted training samples through model adaptation techniques.This work delivers the early efforts of exploring the muscle fatigue effects resulting from real-world HRI tasks, which has brought new perspectives into this research area.With hand-free sEMG measurements, the proposed working approach will not cause any disturbance to the human operator.Meanwhile, the computational costs required for fatigue scrutiny are manageable in real time as well.With the growing employment of HRI related techniques in a wider social and industrial community nowadays, the efforts made here are expected to inspire more future contributions.

Muscle Fatigue Assessment from sEMG
During multiple demonstrations, the human demonstrator will feel tired.It is mainly reflected by the emerging muscle fatigue, which will affect the quality of robot learning.The changes in muscle electricity can describe the underlying muscular activity.As a noninvasive method, sEMG has been taken to measure the muscle fatigue.According to Petrofsky et al. [32], muscle length change causes the amplitude and frequency components of sEMG to vary throughout the duration of fatiguing exercise.This section investigates the intrinsic amplitude and frequency variables in sEMG and how they connect with human muscle fatigue.
For a band-limited signal of a sufficiently small bandwidth, for example, the subband sEMG signal, there is a dominant frequency component assuming the majority of signal power.The amplitude and frequency of this component are viewed as the primary features of the signal.This amplitudefrequency representation is defined as follows (Maragos et al. [33]): where () and Θ() stand for signal ()'s amplitude and phase sequence, respectively.The instantaneous frequency is defined as Ω() = Θ() − Θ( − 1) = Ω  + () = (2/  )  + (), where   and () denote the sampling frequency (200 Hz) and frequency modulation variable, respectively.() delivers any noise and possible errors.Figure 1 illustrates a segment of sEMG signal and its subbands obtained from band-pass filtering.In each of these subbands, the envelope sequence is estimated and presented by magenta dashed line.Over a short time interval, the mean value of the envelope sequence has been calculated to indicate the dominant amplitude of the subband in green color.Due to the sampling frequency limitation, we choose threeband signal decomposition (<10 Hz, 10-50 Hz, and >50 Hz).We name the average instantaneous amplitude of subband sEMG signal from the three bands as AIE [0,10]   b , AIE [10,50]   sb , and AIE [50,100]   sb , respectively.The identified parameters in Figure 1 are AIE [0,10]   sb = 39.55,AIE [10,50]   sb = 22.29, and AIE [50,100]   sb = 16.81,respectively.In Figure 2, the instantaneous frequencies of the subband sEMG signals have been tracked, and the principal frequency components are identified as the mean values of the instantaneous frequency sequences over time.Likewise, they have been named as AIF [0,10]   sb , AIF [10,50]   sb , and AIF [50,100]   sb , respectively.The detected dominant frequencies Lower frequency sEMG signals have been found more prominent in identifying muscle fatigue than the higher frequency bands (Petrofsky et al. [32] and Venugopal et al. [34]).The following exploration of fatiguing effects hereby only involves the lower band of sEMG to avoid other irrelevant factors.

Robot Movement Representation
Dynamic movement primitive (DMP) is to model the movement streams of a system in discrete time, for example, the trajectory of robot motions (Schaal [35]).DMPs are motion elements or short segments, formed as nonlinear dynamic systems.The robotic joint space can be formulated by multiple DMPs, one per joint state.A DMP system is   :  →  is assumed as a nonlinear continuous bounded function and  ∈  is the state of a first-order dynamic system formulated by where   > 0 is a time constant and the state  is regarded as a phase variable.This system is referred to as a canonical system.Since it is not dependent on time, the described DMP system in (2) is autonomous, which will finally converge to zero.Empirically, the initial value of  0 is set to be 1.DMP is found stable and robust in the presence of external disturbances.It can also adjust the speed of the discrete movement it represents by spatial and temporal scaling.
The original DMP predefines the nonlinear function (), which can be expressed as where   ∈  are the weights of   () and   () are normalized radial basis functions defined by where   > 0 are the centres and ℎ  > 0 are the widths of the  Gaussian basis functions, with  = 1, . . ., .

Robot Learning from Multiple Human Demonstrations
In order to model data from multiple demonstrations, Gaussian mixture models (GMMs) have been employed.
GMMs are a probabilistic model with unknown parameters, consisting of a group of Gaussian distributions.Each individual demonstration trial produces a set of joint angles of the robot.Therefore, there are  groups of joint angles { , , θ , , θ , }   , =0,=1 in our data set, where  is the number of demonstration trials.For the th trial,  , ∈  is the joint angle and   is the duration of demonstration.Additionally, the  , ∈  is the state of system (3) at time step , and a muscle fatigue factor from the corresponding sEMG data has been calculated as MF  , where  = 1, . . ., .
To learn from the data vector set {, }, where  are values of function () in system (2), GMMs have been applied to formulate the joint distribution (, ) as described by Given the dimension  of the input data vector equal to 2, the probability distribution of a Gaussian component N(, ;   , Σ  ) is elaborated as follows: The GMMs are composed of  Gaussian components here, where the unknown parameters are   ,   , and Σ  , with  = 1, 2, . . ., .   ≥ 0 is the prior probability satisfying ∑  =1   = 1, while   ∈  2×1 and Σ  ∈  2×2 are the mean and covariance matrix of the th Gaussian component referred as follows, respectively: These unknown parameters can be retrieved through the Expectation-Maximization (EM) algorithm.We propose to involve weighted Gaussian components rather than the conventional Gaussians in this work in order to take the muscle fatigue effects into account.Considering the fatiguing status during the demonstration phase is varying, the weights across all demonstration data have to vary accordingly.Therefore, compared with the conventional GMMs algorithm, the training data of weighted GMMs (W-GMMs) is scaled by the fatigue factor first.The modified EM algorithm for W-GMMs is elaborated as follows: (1) Initialization: set initial values of the parameters.
(3) -step: update model parameters by maximum likelihood estimation at new round of iteration to be (4) Repeat the -step and -step until convergence.
The f function can be estimated through Gaussian mixture regression (GMR) algorithm.In GMR, the conditional probability ( | ) is elaborated by where According to the linear transformation property of Gaussian distributions, ( | ) can be approximated as , where η is the expectation.

Experiments
In this section, we setup our experimental platform and verify the proposed method by a robot teaching by demonstration experiment which consists of a demonstration phase, a learning phase, and a reproduction phase.

Platform Setup. The Baxter robot has been employed
to verify the proposed method.Baxter is a semihumanoid industrial robot developed by Rethink Robotics, whose head is of 2-DOF and the two arms are of 7-DOF (shoulder joint: 0 and 1; elbow joint: 0 and 1; wrist joint: 1, 2, and 3).Figure 3 gives us a picture of the Baxter robot.Besides industrial usage, it has been employed in HRI and humanrobot collaboration (HRC) research widely (Reddivari et al. [36]).In the experiment, the human demonstrator will drag the end effector of the robot to complete a demonstration task, during which the robot joint angles will be recorded.Figure 4 shows the way of sEMG data collection in the experiments.During the manipulation, MYO armband is worn on the arm of the demonstrator to record the sEMG signals generated, where the movement of human arm is mainly driven by biceps brachii.The MYO armband is composed of 8 sEMG electrodes and a 9-axis inertial

Demonstration Phase.
As shown in Figure 5, there is a pen fixed at the end effector of the Baxter robot.The rest place of the pen is on the top of the obstacle (the stacked cylinder).Dragged by the human demonstrator, the robot will complete the following action: at first, the pen at the end effector pokes the first circle (A); it then bypasses the obstacle and pokes the second circle (B); and then it bypasses the obstacle again and pokes the third circle (C), and finally it returns to its rest place.In this way, the passing path of the pen is A This demonstration is repeated 30 times.In the course of each demonstration, the joint angles of the robot and the sEMG signals of the human demonstrator are recorded.Although the passing path is fixed, the trajectory of robot motion and the duration of teaching in each trial vary.However, all trajectories share the exact same starting and end places.attached to the biceps brachii, since the movement of human arm is mainly driven by it.As described in Section 2, we extract the instantaneous amplitude sequence of sEMG subband [0, 10 Hz] captured by the second channel of MYO armband and compute the mean values of that timevarying sequence at every 2 s time interval to acquire the fatigue parameters across all 30 demonstrations.To reveal the most significant trend and remove fluctuations, a 3rd-order polynomial function has been used to fit the data samples AIE [0,10]   sb .Figure 6 shows the smoothed data samples over all 30 trials after curve fitting.We define an Onset of Fatigue (OF) index as the flex point of the cubic function in the figure .When training W-GMMs, six groups of data have been chosen to learn the new robot trajectory; three of them are from demonstrations before the OF, while the others are from trails after the OF.Their muscle fatigue factors are taken as weights in the W-GMMs training process, which are calculated and normalized to be 0.9, 0.8, 0.7, 0.4, 0.3, and 0.2, respectively.

Learning Phase and Reproduction
Both the duration and the data amount recorded in each demonstration vary from one to another.In Figure 7(a), we randomly select three samples of data to observe, which are the angles of shoulder joint 0.It is found that they are different in duration and misaligned in time.Dynamic Time Warping (DTW) algorithm is therefore applied to align the data along the timeline.Figure 7(b) shows the aligned data after the DTW processing.
As mentioned, six groups of data, including three before the OF, and three after that, as well as their corresponding weights are taken as training data to build the W-GMMs.Figure 8 illustrates the learning process of the W-GMMs.Angle data of all seven joints, including two shoulder joints 0 and 1, two elbow joints 0 and 1, and three wrist joints 1, 2, and 3, have been taken into account in the robot learning experiment.Figure 9 illustrates the learning results for each joint from the W-GMMs.The Baxter robot then successfully accomplishes the taught task by the human demonstrator after receiving these learned joint angle data.Figure 10 demonstrates the trajectory of the Baxter end effector in Cartesian space when accomplishing the task.

Conclusion
Robot learning manipulation skills from human demonstration is a straightforward yet effective way of knowledge transfer.However, the manipulation, especially for complex tasks with heavy loads, will make the human demonstrator feel tired.The purpose of this study is to inspect the influences of muscle fatigue on robot learning from human demonstration.We propose new fatigue-indicating sEMG features, which have been explored from analyzing the varying amplitude and frequency components within the sEMG during biceps brachii muscle contractions.These newly presented features have been employed to identify the fatigue onset in the process of a fatiguing TbD task, which has been further taken to guide the learning curve of the robot.A weighted GMM has been employed to learn the robot joints angle from data of multiple demonstration trials, where the muscle fatigue factors identified in each of these trials have been considered.In this way, the regenerated movement trajectory shows its closeness with the ideal condition when less fatigue detected.In the TbD experiments carried out, the Baxter robot successfully completes the task taught by the human demonstrator with desirable performance.This work describes the efforts on investigating the muscle fatigue effects under the environments of HRI manipulation.With growing interests and needs of applying HRI techniques in various working conditions, it is expected that the efforts made here would bring new perspectives into relevant research areas.

Figure 1 :
Figure 1: The subband sEMG signals and their envelopes (in magenta dash line).The green dot-dash line in each subband indicates its dominant amplitude.

Figure 2 :
Figure 2: The instantaneous and dominant frequencies of subband sEMG signals.

Figure 6 :
Figure 6: The smoothed curve of the proposed fatigue-related features (average instantaneous amplitude of sEMG signal) across all trials during the human demonstration to robot.

Figure 8 (
a) shows the training data samples in the joint space.

Figure 8 (
b) reveals the GMMs after training, which encodes the joint distribution with sixteen Gaussian components.

Figure 8 (Figure 7 :
Figure 7: (a) Three original samples of the joint angle 0.(b) The processed samples after DTW time alignment.

Figure 8 :
Figure 8: The learning process of the W-GMMs.(a) Training data samples in the joint space 0.(b) Gaussian mixture components formed by the training data.(c) Regenerated (learned) trajectory from the W-GMMs.

Figure 9 :Figure 10 :
Figure 9: The seven joint angles of the Baxter robot right arm.