Effects of Error Modulation-Based Visual and Haptic Feedback Fusion Strategies on Motor Learning and Motivation

Visual or haptic feedback based on error modulation has been used to improve the effect of robot-assisted rehabilitation training. However


Introduction
It is reported that about 75% of stroke patients have upper limb motor impairment, which is caused by brain damage and severely affects their daily life [1]. Improving upper limb motor function has been clinically recognized as a research priority [2]. Conventional rehabilitation is commonly treated with medicines and simple manual or instrumental therapy.
However, the tedious nature of manual or instrumental rehabilitation leads to low patient compliance and initiative, which harms the effectiveness of rehabilitation training. In recent years, end-effector upper limb rehabilitation robots (EULRR) have been widely applied in the rehabilitation process of stroke patients [2]. Virtual reality technology is primarily applied to provide patients with context-rich interactive training tasks and audio-visual feedback. Additionally, human-robot interaction technology is made to provide patients with the appropriate haptic feedback. Through repetitive, goal-oriented practice and multisensory feedback, the rehabilitation robotic system improves patients' motor learning abilities [3]. A major technical challenge that needs to be solved is how to enhance the training effects of rehabilitation robots through the optimal integration of multisensory feedback [4].
Visual or haptic feedback with error amplification or error reduction has been extensively explored and considered a promising human-robot interaction-based rehabilitation technology. A systematic review found that haptic error amplification was overall more effective than conventional repetitive practice and haptic error reduction on motor performance [5]. Shum et al. [6] found that visual error amplification improved bilateral upper limb recovery in hemiplegic cerebral palsy patients. However, most of the existing studies focus only on error-based single-sensory feedback. They discovered that visual error amplification had the potential to trigger motor recovery by awakening the damaged nervous system [7], visual error reduction was more beneficial for older with limited cognitive resources to enhance motor control [8], haptic error amplification had more potential to facilitate motor learning in continuous and discrete tasks [9], and more severe patients were suitable for haptic error reduction with higher stiffness gains [10]. Substantial evidence suggested that multisensory feedback fusion might further improve motor learning in stroke patients. The visual and haptic error amplification feedback fusion group showed faster motor adaptation than the visual error amplification group, higher attention levels, and satisfaction [11]. Visual and haptic error amplification feedback fusion promoted upper limb motor recovery in stroke patients [12,13]. The benefits of visual error amplification combined with haptic error amplification have been studied, but there is a lack of study on the effects of visual error reduction combined with haptic error amplification on motor learning.
Previous studies have also shown that visual or haptic feedback with error amplification or error reduction has different effects on motivation. Visual error reduction increased patients' self-confidence since they were highly sensitive to failure [14], while visual error amplification was specially challenging [15]. Haptic error reduction improved perceived competence and satisfaction [16], while haptic error amplification decreased perceived competence and enjoyment [9]. Overall, visual or haptic error reduction feedback has positive effects on motivation, while visual or haptic error amplification feedback may have negative effects on motivation. We want to know if the combination of visual error amplification and haptic error reduction or visual error reduction and haptic error amplification has positive effects on motivation. However, there is a lack of studies on the impact of error amplification and error reduction feedback fusion with different sensory channels on training motivation.
In this study, we designed a trajectory-tracking task to compare the effects of three feedback fusion strategies on motor learning and training motivation. The no error modulation (NEM) group received visual and haptic feedback without error modulation, the visual amplification haptic reduction (VAHR) group received visual error amplification combined with haptic error reduction, the visual reduction haptic amplification (VRHA) group received visual error reduction combined with haptic error amplification. Based on previous studies, we hypothesized that subjects in VAHR and VRHA groups would have better performance on motor learning and motivation than the NEM group. This study aims to provide some new insights for multisensory feedback fusion technology in the application of rehabilitation robots.

EULRR and Surface Electromyography (sEMG)
Acquisition System. EULRR is a five degree-of-freedom rehabilitation robot device for upper limb rehabilitation designed by the Motor Rehabilitation and Medical Robotics Team at the Ningbo Institute of Materials Technology & Engineering, Chinese Academy of Sciences [17]. Figure 1 shows the components of the system; the rehabilitation robot body consists of three servo motors. According to the characteristics of the trajectory-tracking task, the Z-axis of EULRR was not utilized in this experiment; subjects only manipulated the rocker in the X and Y planes. TRIGNO wireless sEMG system (Delsys Inc., Massachusetts, USA) was used to record the sEMG signals [18], and the sampling frequency was set at 2,000 Hz. The four upper limb muscles involved in the trajectory-tracking task were selected: anterior deltoid (AD), biceps brachii (BB), flexor carpi radial  Applied Bionics and Biomechanics (FCR), and extensor carpi radial (ECR). Before the start of the experiment, the four muscles of the subject's right arm were cleaned with scrub cream and alcohol to reduce the resistance between the electrodes and the skin, and the sEMG sensors were attached to the four muscles parallel to the muscle fiber direction, as detailed in Figure 1. After the end of the experiment, we preprocessed the collected sEMG signals offline. We used a Butterworth bandpass filter at 50-400 Hz to obtain the useful frequency band of the sEMG signals. To eliminate direct current offsets, we subtracted the average value of the sEMG signals. Finally, we applied a full-wave rectification and a low-pass filter at 5 Hz to obtain the linear envelope of the sEMG signals.

A Trajectory-Tracking
Task and the Generalization Experimental Protocols. The virtual environment was designed using Unity 2019.1.0a8 (Unity Technologies, California, USA): a trajectory-tracking task. The task was displayed in two dimensions on the computer screen, and the position of the rocker was mapped to an agent point in the virtual environment. After the task started, the target point moved on the trajectory at a special rate. The subject's goal is to move the rocker to catch up with the target point to get a high score as possible, and performance-based concurrent visual feedback was provided during the execution of the task. When the target point arrived at the starting point, it was regarded as the end of a consecutive tracking movement. At this moment, the audio would emit auditory feedback of "ding," and the score as results-based visual feedback would be displayed, see Figure 1 for details. Equation (1) describes the tracking error D i between the actual point (x i ; y i ) and the target point (x 0 ; y 0 ). Equation (2) describes the method for calculating the average tracking error, and N is the collection sample size in a trajectory-tracking movement. The score is calculated based on the average tracking error D average and the maximum error D max [19], see Equation (3) for details. D max was set to 0.04 m in this experiment, which was obtained by repeated measurements. When D average is greater than D max , the subject is considered to have a low ability to perform the task, at which point the score is zero. The score was provided only as results-based visual feedback at the end of each consecutive movement and was not used for assessment.
The generalization experiment was designed to evaluate motor learning transfer from one task to another new, similar task. In the generalization experiment, the trajectory was rotated by 90°, and the working range changed from rightto-left to top-to-bottom. All other parameters remained the same as the trajectory-tracking task, such as the speed of the target point and guidelines for calculating scores.  [8]. Visually reduced the error between the target point and the actual point. Equation (4) describes the position of agent point under visual error reduction modulation (x ER ; y ER ). The position of the agent point was the position where the actual tracking error was reduced with a proportional gain k 1 (Figure 2). Based on a previous study [8], the visual error reduction parameter was 0.5 in this experiment.

Visual Error Reduction
2.3.2. Visual Error Amplification [7]. Visually magnified the error between the target point and the actual point. Equation (5) describes the position of the agent point under visual error amplification modulation (x EA ; y EA ). The position of the agent point was the position where the actual tracking error was increased with a proportional gain k 2 ( Figure 2). According to a previous study [13], the visual error amplification parameter was 1.5 in this experiment. Applied Bionics and Biomechanics Haptic feedback without error modulation: EULRR only provided friction and gravity compensation; it did not provide assistance or resistance force. Subjects can move freely in the X and Y planes. [9]. In addition to providing friction and gravity compensation, an assistant force F ER was provided to help the subject reach the target point. Equation (6) describes the tracking error of the actual point relative to the target point. Equation (7) describes the change of F ER with e under the modulation of haptic error reduction gain k 3 . F ER proportionally decreased with e decreased (Figure 3). Through repeated tests before the start of the experiment, k 3 was −300 N ⋅ m in this experiment.

Haptic Error Reduction
2.3.4. Haptic Error Amplification [9]. In addition to providing friction and gravity compensation, a resistance force F EA was provided to prevent the subject from reaching the target point. To prevent too large a force to become dangerous [20], haptic error amplification is limited to a certain area. When the tracking error D i is greater than the maximum error D max , haptic feedback without error modulation is provided. Equation (8) describes the change of F EA with e under the modulation of actual haptic error amplification gain k max . F EA proportionally increased with e decreased. Equation (9) describes the relationship between k max and D i . k max gets maximum when D i is equal to zero, and k max gets minimum when D i larger or equal to the maximum error D max . Through repeated tests before the start of the experiment, haptic error amplification gain k 4 was 300 N · m in this experiment.
2.4. Subjects. Thirty healthy, right-handed adults (16 males and 14 females, with average ages of 24.9 AE 2.5 years) without any history of neurological disorders were recruited to participate in this experiment. Subjects were randomly divided into three groups: NEM, VAHR, and VRHA. There were no significant differences in gender, age, and average tracking error among the three groups of subjects after statistical analysis (P >0:05), detailed information is shown in Table 1. The study protocol was approved by the Ningbo Institute of Materials Technology & Engineering, Chinese Academy of Sciences, and all subjects gave written informed consent.

Experimental Protocol.
A parallel controlled experiment was conducted on separate 3 days, and each subject went through the same test phases, as detailed in Figure 4. On day 1, subjects started with a Familiarization phase to adapt the EULRR and the trajectory-tracking task. They performed 20 consecutive tracking movements under visual and haptic feedback without error modulation. After a 5-min break, subjects performed the Baseline1, where they played again 20 consecutive tracking movements under the visual and haptic feedback without error modulation. The purpose of the phase was to test the initial skill level of subjects, who would be divided into three groups with nonsignificant differences in average skill level.
On day 2, the subjects' sEMG signals were recorded once every 20 consecutive tracking movements to understand muscle activation of subjects' upper limbs during the trajectory-tracking task and observe the differences in physical effort under different feedback strategies. Subjects performed 20 consecutive tracking movements under the visual and haptic feedback without error modulation during Base-line2, and the intrinsic motivation inventory (IMI) scale was completed after Baseline2. After a 5-min break, subjects performed the Baseline generalization, where the trajectory was rotated by 90°. They repeated 20 consecutive tracking tasks under visual and haptic feedback without error modulation. After a 5-min break, subjects completed 180 consecutive tracking movements in NEM, VAHR, or VHAR groups with different feedback strategies during Training. Training1  refers to the initial 60 movements, Training2 to the middle 60 movements, and Training3 to the last 60 movements.
There was a 5-min break at the end of every 20 consecutive movements, and IMI questions needed to be completed at the end of every 60 movements. During Short-term assessment and Short-term generalization assessment, subjects underwent the same experimental procedures as the Base-line2 and Baseline generalization. On day 3, occurred within 1-3 days after day 2, subjects were invited to perform Long-term assessment and Longterm generalization assessment. To evaluate the influence of long-term retention on motor learning, these phases utilized the same experimental procedures as the Baseline2 and Baseline generalization.
After Baseline2, Training1, Training2, Training3, Shortterm assessment, and Long-term assessment, subjects were asked to complete the IMI scale. IMI questionnaire has been widely used to assess motor learning motivation [9,15,16,21]. The full questionnaire consists of seven motivational subscales [22], three of which were selected in this experiment: interest/ enjoyment, perceived competence, and effort/importance, as detailed in Table 2. Subjects chose a Likert scale between 1 and 7 points according to their subjective feelings, with 1 indicating "I disagree completely" and 7 indicating "I agree completely." The subjects' previously filled-out contents can be seen.

Assessments of Motor Learning and Training Motivation.
All subjects' kinematic information, the sEMG signals, and IMI questionnaire results were collected.
Average tracking error was used to assess the accuracy of the movement. The change in the average tracking error from Baseline1 to Short-term assessment or Long-term assessment for evaluating the short-term or long-term effects of motor learning. The change in the average tracking error from Baseline generalization to Short-term generalization assessment or Long-term generalization assessment was used to evaluate the short-term or long-term effects of motor learning transfer. In addition, exponential curves were fitted to evaluate the motor learning effects of different feedback strategies during the Training period [7]. As in Equation (10), y is the average tracking error, and x is the number of groups during Training (every 20 consecutive tracking movements as a group). Based on this, a is the amount of learning (the differences of the average tracking error between the beginning and the end of training), b is the rate of learning (the number of groups corresponding to when 63% amount of learning is reached), c is the value of the average tracking error convergence at the end of Training.
The root mean square (RMS) of the sEMG signals was used to evaluate the effects of different feedback strategies on muscle activation. After obtaining the envelope of the collected sEMG signals, we calculated the RMS value of the sEMG signals containing 20 consecutive tracking movements offline, as shown in Equation (11), E i is the ith sEMG signals, and N is the collection sample size.  Applied Bionics and Biomechanics The IMI scale was used to evaluate motor learning motivation. Subjects' responses to interest/interest, perceived competence, and effort/importance were collected, with Baseline2 used as a reference to compare the amount of change in responses in other periods.

Statistical Analysis.
To evaluate motor learning effects, we analyzed the changes in average tracking error between Baseline1 and Training1, Short-term assessment (last 10 consecutive tracking movements), and Long-term assessment (first 10 consecutive tracking movements) using independent t-tests within a group. To evaluate motor learning transfer, we analyzed the changes in average tracking error between Baseline generalization and Short-term generalization assessment and Long-term generalization assessment using independent t-tests within a group. A one-way analysis of variance (ANOVA) was used to analyze the average tracking error between groups during Baseline1, Training, and assessment, the reduction in average tracking error between groups during generalization, and the RMS of sEMG signals between groups during all periods. If the ANOVA was significant, multiple comparisons were carried out using the least significant differences method. We compared the changes in responses at each phase using the nonparametric Kruskal-Wallis one-way ANOVA with training strategies as the main factor. The differences were considered statistically significant at P <0:05. Data processing and statistical analysis were performed in Microsoft Office Excel 2013 (Microsoft, Washington, USA), and measurement information was presented as x AE s.

Effect of Training Strategies on Motor
Learning. The exponential curve fits for the different feedback strategies during Training can be seen in Figure 5, which shows that subjects in the VRHA group had the biggest a value and smallest b value, and subjects in the VAHR group had the smallest c value. Table 3 shows the average tracking error for the three groups of subjects at Baseline1, Training1, and assessment. During the Training1 period, the average tracking error significantly decreased in NEM and VAHR groups (P <0:05) while significantly increased in VRHA group compared to Baseline1 (P <0:05). In the Short-term and Long-term assessments, the average tracking error was significantly decreased in all three groups (P <0:05). Figure 6 compares the differences of average tracking error between groups. There were no significant differences between groups at Baseline1 (P >0:05). During Training, there were significant differences between groups (P <0:05); subjects in VRHA group had the biggest average tracking error, while subjects in VAHR group had the smallest. During both Short-term and Long-term assessments, there was a smaller average tracking error in VRHA group, but the error reduction was nonsignificant between groups (P >0:05). Table 4 shows the average tracking error for the three groups of subjects during the generalization phases. In the Shortterm generalization assessment, only the VRHA group significantly decreased the average tracking error compared to the Baseline generalization (P <0:05). In the Long-term generalization assessment, all three groups of subjects significantly decreased the average tracking error compared to the Baseline generalization (P <0:05). Figure 7 shows the reduction in the average tracking error in Short-term and Long-term generalization assessment. The VRHA group significantly reduced the average tracking error than the NEM group (P <0:05), and the VAHR group also reduced the average tracking error more than the NEM group, although the reduction did not reach significance (P >0:05).

Effect of Training Strategies on Muscle Activation.
The RMS values of four muscles' sEMG for a typical subject (subject no. 5) from the NEM group are shown in Figure 8(a). Figure 8(b) shows the RMS values of one muscle's sEMG for three typical subjects (subjects nos. 5, 14, 22) from NEM, VAHR, and VRHA groups. There were no significant differences between groups in the RMS of sEMG signals at Base-line2 and Baseline generalization (P >0:05). Figure 9 shows the RMS of sEMG signals in the Training, Short-term assessment, and Short-term generalization assessment. In general, subjects in the VRHA group had a lower RMS than those in the NEM group. During Training, the AD of subjects in the VRHA group had significantly lower RMS than the NEM group (P <0:05). In the Short-term assessment, the AD of subjects in the VRHA group still had significantly lower RMS than the NEM group (P <0:05). In the Short-term generalization assessment, subjects in the VRHA group showed significantly lower RMS of BB and AD than the NEM group (P <0:05), and significantly lower RMS of AD than the VAHR group (P <0:05). Figure 10 shows the changes in responses to the IMI questionnaire after Baseline2, Training, and assessment. In the Short-term and Long-term assessments, subjects in the VRHA group significantly increased perceived competence (P <0:05). There were no significant differences in interest/enjoyment and effort/ importance between groups in Baseline2, Training, and assessment (P >0:05). In particular, subjects in the NEM group significantly improved perceived competence only during Training1 (P <0:05), subjects in the VAHR group significantly improved perceived competence during Training (P <0:05), and subjects in the VRHA group significantly improved perceived competence in all periods (P <0:05). Only subjects in the VAHR group significantly improved effort/importance during Training1 (P <0:05).

Discussion
Visual or haptic feedback based on error modulation has been extensively studied to improve the effectiveness of robot-assisted rehabilitation training. However, there are several investigations on the effects of error amplification and error reduction feedback fusion with different sensory channels on motor learning and motivation. In this study, we designed a trajectory-tracking task to evaluate the effects of three error modulation feedback fusion strategies (i.e., NEM, VAHR, VRHA) on motor learning and motivation.   During the Training period, subjects in the VAHR group had the best motor performance, and the VRHA group had the worst motor performance. This is in line with previous studies on haptic error modulation [9,23], which discovered that haptic error reduction enhanced subjects' performance while haptic error amplification degraded it. Based on the trajectory-tracking task, visual error amplification enhanced performance by repeatedly correcting for errors [7], while visual error reduction degraded performance may be due to incorrect delivery of task-relevant information [24]. Therefore, visual error amplification combined with haptic error reduction feedback fusion further enhanced performance, and visual error reduction combined with haptic error amplification feedback fusion further degraded performance. However, the relatively high motor performance of subjects in the VAHR group may limit their learning potential at the beginning of Training. In contrast, the relatively poor motor performance of subjects in the VRHA group may   NEM, visual and haptic feedback without error modulation; VAHR, visual error amplification combined with haptic error reduction; VRHA, visual error reduction combined with haptic error amplification. * 0:01<P ≤ 0:05; * * 0:001<P ≤ 0:05; * * * P ≤ 0:001. 8 Applied Bionics and Biomechanics lead to more improvement at the beginning of Training. Therefore, the VRHA group had the largest amount of motor learning and the fastest learning rate during Training, which was indicated by the results of the exponential fit curves in Figure 5.
During the Short-term and Long-term assessment periods, the subjects in all three groups had significant improvement comparing with Baseline1, indicating that the motor learning did happen in our study. The subjects in the VRHA group had a relatively better motor performance with the lowest average tracking error, while there were no statistical differences between groups. This may be explained by the specificity-of-learning hypothesis [25], which states that motor learning is most effective when subjects receive the most important sensory information needed to complete the task during the Training period. In our experiments, subjects may mainly rely on performance-based concurrent visual feedback to catch up with target points during the Applied Bionics and Biomechanics trajectory-tracking task. Therefore, all three groups significantly improved performance in the assessment period may be due to the real-time visual feedback play a major role in the trajectory-tracking task.
Generalization is an important aspect of motor learning to evaluate transfer. The subjects in all three groups had significant improvement comparing with Baseline generalization in the Long-term generalization assessment. Only subjects in the VRHA group improved transfer ability in the Short-term generalization assessment. Both in the Short-term and Long-term generalization assessments, the VRHA group had significantly higher transfer ability than the NEM group. The trajectorytracking task requires both temporal and spatial alignment with the target point. It has been demonstrated that haptic feedback is as effective as visual feedback when learning spatiotemporal aspects of a motor task, and haptic feedback does not further facilitate motor learning when visual feedback provides sufficient information [24]. Furthermore, a related study [15] also found that haptic error amplification enhanced the transfer of the practiced asymmetric gait pattern to free walking. It is possible that subjects in the VAHR and NEM groups received sufficient information provided by visual feedback, causing haptic feedback was redundant and helpless for task learning. However, when subjects in the VRHA group found that performance-based feedback was inconsistent with results-based feedback, they may rely on haptic error amplification to learn the trajectory-tracking task. This task-related information transmitted by the haptic error amplification may cause subjects in the VRHA group significantly improve transfer ability.
The results of sEMG showed that visual feedback may have a significant effect on motor control. A previous study [26] has shown that haptic error amplification leads to higher muscle activation, and haptic error reduction leads to lower muscle activation. However, this study showed that subjects trained under visual error reduction combined with haptic error amplification had lower muscle activation. The subjects may subjectively decrease physical effort due to good performance-based visual feedback provided by visual error reduction [27]. On the other hand, they may reduce unnecessary muscle use due to relying more on haptic error amplification to learn to perform the task. A study [24] has shown that experts generally achieve maximum accuracy with minimum effort, so visual error reduction combined with haptic error amplification may stimulate subjects' potential and help them quickly reach the expert level.
Motivation is a key factor for engaging trainees in extensive, repetitive training tasks. Previous studies [9,16] have reported haptic error amplification has negative effects on motivation. However, the results of the present study showed that visual error reduction combined with haptic error amplification feedback fusion improved perceived competence. A large score increment in the VRHA group may enhance subjects' satisfaction and self-efficacy. On the other hand, it is also possible that the perceived competence brought by visual error reduction overshadows the frustration brought by haptic error amplification. Therefore, maintaining subjects under the condition with increasing scores can lead to higher perceived competence, which may indicate new ideas for designing motivation-enhancing rehabilitation training strategies.
Modern rehabilitation theory views neurorehabilitation as a motor relearning process, and motor learning promotes the reorganization of brain neural circuits to encode new learning experiences and enable behavioral change [28]. Therefore, motor learning is related to stroke recovery and neurorehabilitation. Providing multisensory feedback during motor learning may help correct movement, strengthen the relationship between external stimulus and internal responses, and increase self-efficacy and motivation. Our findings may have important implications for optimizing multisensory feedback and improving the effectiveness of robot-assisted rehabilitation. Visual error reduction combined with haptic error amplification could accelerate the motor learning process, enhance motor skill transfer, decrease muscle activation, and increase motor learning motivation. The results of motor learning and motivation found in this study may be applied in the robot-assisted rehabilitation training of stroke patients; the task difficulty can be reduced by modifying trajectory or decreasing the haptic error amplification feedback parameter to match the motor ability of stroke patients.
There are still several limitations in this study. First, a previous study [9] showed that the effectiveness of the training strategy depended on motor task characteristics. This study was conducted only for a trajectory-tracking task; a variety of task types should be considered in the future. Second, according to challenge point theory [29], optimal learning is formed when the difficulties of the task match the individual's skill level, and it has also been shown that there are optimal error modulation parameters to maximize the effectiveness of the training strategy [30]. The present study used fixed error modulation parameters, and the difficulty of the task remained constant as the skill level increased, which was detrimental to subjects' motor learning in the later phase. Finally, the sample size was small, and the subjects were all healthy. Although motor learning may share the same brain mechanisms as neurorehabilitation [24], stroke patients have a different learning status and motivation during the rehabilitation. The investigations on error modulationbased visual and haptic feedback fusion for rehabilitation patients are needed in the next study.

Conclusion
In summary, we designed a trajectory-tracking task to evaluate the effects of three error modulation feedback fusion strategies (i.e., NEM, VAHR, VRHA) on motor learning and motivation. The results showed that all three groups significantly improved motor learning ability, particularly subjects in the VRHA group had higher transfer ability and perceived competence and reduced unnecessary muscle activation. Our study provided a new research idea to optimize the multisensory feedback technology for rehabilitation robots. Providing visual error reduction and haptic error amplification feedback strategies during rehabilitation training may not only improve motor learning efficiency and active participation motivation but also have a potential for facilitate a higher dose of rehabilitation training. Adaptive optimization of visual and haptic feedback fusion strategies based on personalized rehabilitation requirements of patients may accelerate the recovery of upper limb motor function; this will be studied in the future.

Data Availability
Data supporting this research article are available from the corresponding author or first author upon reasonable request.

Ethical Approval
This research was conducted by the Declaration of Helsinki (1964) and approved by the Ningbo Institute of Materials Technology & Engineering, Chinese Academy of Sciences.