An Approach to Applying Feedback Error Learning for Functional Electrical Stimulation Controller : Computer Simulation Tests of Wrist Joint Control

Feedback error-learning (FEL) controller that consists of a proportional-integral-derivative (PID) controller and an artificial neural network (ANN) had applicability to functional electrical stimulation (FES). Because of the integral (reset) windup, however, delay or overshoot sometimes occurred in feedback FES control, which was considered to cause inappropriate ANN learning and to limit the feasibility of the FEL controller for FES to controlling 1-DOF movements stimulating 2 muscles. In this paper, an FELFES controller was developed applying antireset windup (ARW) scheme that worked based on total controller output. The FELFES controller with the ARW was examined in controlling 2-DOF movements of the wrist joint stimulating 4 muscles through computer simulation. The developed FEL-FES controller was found to realize appropriately inverse dynamics model and to have a possibility of being used as an open-loop controller. The developed controller would be effective in multiple DOF movement control stimulating several muscles.


Introduction
A spinal cord injury or a stroke causes paralysis of human motor function.Even in those cases, if peripheral muscles and nerves maintain electrical excitability, electrical stimulation to them can activate the muscles, and therefore the motor function can be restored.This is the principle of "Functional Electrical Stimulation (FES)".FES has been found to be effective clinically, especially in controlling paralyzed upper limbs [1][2][3].
FES controller is required to determine appropriate stimulation data for restoring desired functional movements.However, FES control of limbs is very difficult subject because responses of electrically stimulated musculoskeletal system have strong nonlinearity, large latency, time variability, fatigue and so on.Redundancy in stimulation intensity determination also makes control more difficult because the number of stimulated muscles is generally larger than the number of controlled joint angles of limb movements.
An effective way of FES movement control of highly nonlinear musculoskeletal system with large latency is to use a hybrid regulator with a feedforward and a feedback controller [4][5][6][7].The hybrid controller has advantages of the both controllers: improvement of tracking control of fast movements by the feedforward controller and compensation for disturbance by the feedback controller.However, a lot of controllers were evaluated on simple movements such as flexion and extension stimulating one or two muscles.In addition, most hybrid controllers tested in the previous studies required separate control sessions for initial adjustment or learning of the feedforward controller before using it in FES control.
The controller using feedback error-learning (FEL) scheme [8] is also a hybrid controller.The feedforward controller is realized by an artificial neural network (ANN) that learns the inverse dynamics of controlled object using outputs of the feedback controller as teacher signals during controlling online.The FEL controller can also be applied to control the redundant musculoskeletal systems if the feedback controller can solve an ill-posed problem in stimulation intensity determination.Therefore, an application of the FEL scheme to FES controller is expected to simplify and shorten clinical tasks for parameter adjustment of a feedforward controller, which reduces the burden of adjustments of feedforward controllers remained in the previous hybrid controllers.
In our previous study, the FEL controller for FES (FEL-FES controller) was found to be feasible in controlling 1-DOF movements of the wrist joint stimulating 2 muscles through control tests with normal subjects and computer simulation tests [9].In the previous FEL-FES controller, a proportional-integral-derivative (PID) controller was used as a feedback controller, which was designed to solve the illposed problem in stimulation intensity determination [10].However, ANN in the previous FEL-FES controller was not trained appropriately in some cases, especially in controlling 2-DOF movements of the wrist joint stimulating 4 muscles.A possible reason of that was in the PID controller, which was the windup of the integral action of the PID controller since the musculoskeletal system has saturation characteristics.The integral (reset) windup causes large delay or overshoot in responses of the musculoskeletal system in feedback FES control.Inappropriate stimulation causing these undesirable responses in feedback control are considered to cause inappropriate learning of the ANN and to limit the feasibility of the FEL-FES controller to controlling 1-DOF movements stimulating 2 muscles.
The purpose of this paper was to realize an FEL-FES controller that could solve the reset windup problem.An antireset windup (ARW) algorithm was applied to the FEL-FES controller, in which the ARW worked based on the total output of the FEL-FES controller.The developed FEL-FES controller with the ARW algorithm was examined through computer simulation using musculoskeletal model with delay in responses to electrical stimulation.First, the controller was compared in controlling 2-DOF movements of the wrist stimulating 4 muscles before ANN learning with the previous controller and previous experimental tests with normal subjects.Then, the FEL-FES controller was examined in ANN learning of the inverse dynamics of the complicated 2-DOF movement control evaluating on open-loop control.

Feedback Error-Learning Controller for FES
Figure 1(a) shows the block diagram of the FEL-FES controller used in this study.The controller consists of a PID controller and an ANN that realizes an inverse dynamics model (IDM) of the controlled limb [9].The sum of stimulation outputs from the PID controller (I PID ) and the IDM (I ANN ) is applied to each muscle after adding offset (threshold value of electrical stimulation intensity) and clipping out with the limiter to prevent excessive stimulation.

Artificial Neural Network.
A three-layered ANN shown in Figure 1(b) was used as a feedforward controller.The inputs of the ANN were time series of angles, angular velocities and angular accelerations of target movements at continuous 6 times, from n to n + 5 (50 ms interval).In this paper, tracking control was performed on 2-DOF movements of the wrist joint (dorsiflexion/palmar flexion and radial flexion/ulnar flexion) as shown in Figure 2, in which muscles to be stimulated were extensor carpi radialis longus/bravis (ECRL/B, which are expressed by "ECR" in this paper since the same stimulation was applied to them), extensor carpi ulnaris (ECU), flexor carpi radialis (FCR) and flexor carpi ulnaris (FCU).Therefore, the total number of neurons of the input and output layers were 36 and 4, respectively.The number of neuron of the hidden layer was 36.In Figure 1(b), input and output of the ANN are shown by, θ d1 and θ d2 meaning dorsiflexion/palmar flexion and radial flexion/ulnar flexion, and I ANN1 ∼ I ANN4 for 4 stimulated muscles, respectively.

Feedback Controller.
Outputs of the PID controller were described by Here, e(n) is an error vector showing the difference between target and measured joint angles at time n.K P , K I and K D are PID parameter matrices.To solve the ill-posed problem in closed-loop control, the generalized inverse matrix of the matrix which transforms stimulus intensities into joint angles was calculated and applied to determine parameter values of the PID controller expanding the Chien-Hrones-Reswick (CHR) method [10].
In the previous studies, a common PID controller shown by (1) was used in FEL-FES controller, although negative values of the controller output are ineffective for FES control.The negative stimulation intensities of the PID controller are necessary for error signals in learning of the IDM, while positive stimulation intensities can activate a muscle.The PID controller is required to output both positive and negative values of stimulation intensities for the FEL.

Antireset Windup Algorithm for FEL-FES Controller.
The PID controller shown in the previous section, however, has a problem in controlling the system that has the saturation property.For example, in the FES control, negative outputs of the PID controller are ineffective and therefore cause integration wind up of the PID controller (reset windup).In order to avoid the integration windup, antireset windup (ARW) scheme can be applied directly to the PID controller.However, such method of applying the ARW limits the output of the PID controller to positive values less than the maximum intensity, which eliminates negative outputs of the PID controller in ANN learning process.Negative outputs of the PID controller are necessary in the ANN training when ANN output should be decreased.Therefore, in this study, the function of the integral term of the PID controller was suspended when the total output of the FEL-FES controller (sum of the outputs of the ANN and the PID controller) saturated.The ARW used in this paper is considered to be able to suppress ineffective large decreasing or increasing of PID controller outputs when a controlled object has saturation property, which is expected to improve delay and/or overshoot in responses of feedback control.prepared, in which those values were determined randomly: −0.25∼0.25 for the weights between input and hidden layers, and −0.5∼0.0 between hidden and output layers.Values of learning coefficient were 10.0 for the weights between input and hidden layers, and for those between hidden and output layers, 0.1, 0.05 and 0.03 were used for cycle period of 3 s, 6 s and 10 s, respectively.Iteration number of training was fixed at 50.Six different subject models were prepared, whose parameters were adjusted to approximate the muscle properties of 6 normal subjects.The total number of simulation conditions was 45 for each subject model.ANN output power increases as the learning progresses and then the ANN realizes the IDM of the controlled limb after the learning.To evaluate performance of the controller, Advances in Artificial Neural Systems mean error (ME) and power ratio (PR) were calculated:

Computer Simulation Tests
Here, e(n) represents the error between target joint angle and desired one.N represents the number of sampled data.P ANN (n) and P PID (n) represent the square value of stimulation intensity (output power) of the ANN and the PID controller, respectively.PR is the proportion of ANN output to the total controller output, which shows large value if the ANN learns the inverse dynamics appropriately.ME was calculated for each movement direction and PR was done for each muscle.

Musculoskeletal
where l and v were muscle length and contraction velocity, respectively.F A shows the force produced by contracted muscle fibers during electrical stimulation, which actively contributes to develop movements.Active torque τ A produced by electrical stimulation was calculated by Moment arm r f (θ) was modeled by an polynomial equation as a function of joint angle θ for each movement developed by each muscle [11].
The recruitment property of electrically stimulated muscle u(I) was modeled by the following [12]: where s c , s h , x c , and y c were constants.Electrical stimulation was expressed by stimulation intensity I(t), which was the sum of I ANN , I PID and offset in this paper.I ANN and I PID were normalized stimulation intensity determined by the ANN and the PID controller shown in Figure 1(a), respectively, which had values between 0 and 1.The muscle activation a m was described by the following dynamics using the recruitment property with different two time constants, t r and t f [13]: Here, L shows delay in response to electrical stimulation.The value of L was set to 50 ms in order to make approximately latency of 100 ms and time constant of 300 ms when the step response was represented by the first-order delay with latency.These values relating to delay were determined by step responses measured with healthy subjects.The length-force relationship k(l) was described by (7) [14].l o means optimum muscle length whose value was determined from [11] The velocity-force relationship h(v) was modeled with maximum contraction velocity v max [14,15].Parameter value of v max was determined as 10 × l o from [16] The maximum muscle force produced by electrical stimulation F max was determined by PCSA (physiological cross sectional area) as follows [11]: Value of PCSA was determined from [11].
The passive viscoelastic element developed passive torque τ P calculated by (10) for each joint movement [17].The range of motion was also represented by where θ and θ were joint angle and angular velocity, respectively.Constants k 0 , b 0 , k 1 , and k 2 were determined for each joint movement.

3.3.
Results.An example of control results by the PID controller is shown in Figure 5.By using the ARW scheme, feedback FES control was improved reducing delay and inappropriate responses.In the case of previous controller without the ARW, the PID controller output large negative values, while those of controller with the ARW were almost positive as seen in Figure 5(b).
Figure 6 shows ME in the control by the feedback controller before ANN learning.The ARW applied to the FEL-FES controller worked appropriately decreasing ME with all subject models for different movement velocities.For reference, control results from 3 normal subjects [10] were also shown in Figure 6.In experiments with the subjects, 5 s of cycle period was used.In the computer simulation tests, however, 6 s of cycle period was used instead of 5 s, because of making clear difference between cycle periods.In the control tests with the subjects, the PID controller was modified to prevent the reset windup problem.That is, the output of the PID controller was simply limited to the values between the stimulation thresholds and the maximum intensities, which means only positive values were used in the control.Although the ARW algorithm used in the FEL-FES controller was not the same as the controller used with subjects, values of ME were almost same between them.
Values of PR after ANN learning is shown in Figure 7.Both average and minimum values of PR increased clearly by applying ARW scheme, which shows improvement of the ANN learning.Especially, the minimum values of the PR increased from 0.0% to 68.6% or more.Even in the cases of 0% of PR with the previous controller, the values of PR were not 0% in the process of the ANN learning.That is, after the 50th learning trial, the ANN output converged 0 because of large negative outputs of the previous PID controller.Thus, the cases that the ANN could hardly learn at all in the previous FEL-FES controller were improved by applying the ARW.
For comparing values of ME, open-loop control test with the ANN after learning was carried out, removing compensating effects of the PID controller.The results of ME in the open-loop control shown in Figure 8 were smaller than or almost same as values of ME by feedback controller shown in Figure 6.Therefore, it is suggested from this result that the trained ANN can be used as an open-loop controller.In controlling faster movements (cycle period of 3 s), values of ME by the controller with the ARW were smaller than those without the ARW.However, in controlling slow movements (cycle periods of 6 s and 10 s), the ME values were almost same between with and without the ARW.
An example of open-loop control results after the ANN learning is shown in Figure 9.The results were obtained under the same simulation condition as that of Figure 5. Delay in the response seen in Figure 5(b) was decreased by the ARW scheme as seen in Figure 9(b).The open-loop control was performed appropriately, since the ANN learning was improved by the ARW scheme.

Discussion
The ARW was applied to the FEL-FES controller considering saturation of its total output.The ARW worked effectively in the feedback FES control and also improved the ANN learning by the FEL.In the feedback control, values of ME were almost same as the results of control with normal subjects.The method of preventing the reset windup used in the controller tested with normal subjects [10] cannot be applied in the FEL-FES controller because outputs of the previous PID controller were limited to positive values.Therefore, the different ARW was applied to the FEL-FES controller in this paper.
Values of ME in open-loop control for movements with the cycle period of 3 s decreased significantly by applying the ARW.On the other hand, in controlling slow movements, the values of ME were almost same between with and without the ARW.Since those values of ME were similar to those by the feedback controller, the previous type controller is suggested to have sufficient ability to control slow movements.However, as seen in results of PR shown in Figure 7, there were some cases of highly-inappropriate ANN learning.Because the control was also inappropriate in such cases, the FEL-FES controller using the ARW scheme would be effective to provide stable control performance.From the results of Figures 5 and 9, mean error less than 2 or 3 deg is seemed to be desirable for a control performance, although it may be changed between control movements or by results from control tests with subjects.
Values of PR increased significantly after ANN learning by applying the ARW scheme.However, ME was not always smaller than the previous controller for slow movements.As seen in Figure 9, stimulation intensities of all muscles increased by using the ARW, which were generally larger than those without the ARW.In such cases, movements were considered to be controlled under the condition of high stiffness.The stiffness control has not been realized in the current FEL-FES controller, although it will be possible by modifying the PID controller.Stiffness control is desired for more appropriate control.
In this paper, the FEL-FES controller using the ARW scheme was found to realize appropriately inverse dynamics model of the electrically stimulated musculoskeletal system by ANN learning in controlling 2-DOF movements stimulating 4 muscles.Target angle trajectories, however, were similar patterns for all movements although both of the center of the trajectory and movement speed were changed.Various target trajectories have to be tested considering practical application.
The improvement of ANN learning has crucially important meaning in FES control.It is difficult to realize appropriate control by only feedback controller, because characteristics of electrically stimulated muscles of paralyzed patients usually show strong nonlinearity and large difference between patients.Therefore, a learning controller such as the FEL-FES controller is expected to be an effective controller for the musculoskeletal system that has various nonlinear characteristics with large delay.Since the IDM Advances in Artificial Neural Systems can be realized appropriately by the developed FEL-FES controller, it is possible to use the IDM as a feedforward controller removing sensors for feedback control in addition to using it in a hybrid controller.

Conclusions
In this paper, the feedback error-learning (FEL) controller for FES with antireset windup (ARW) scheme was developed and examined.The ARW worked based on the total output of the FEL-FES controller.The computer simulation tests of controlling 2-DOF movements of the wrist joint showed that the FEL-FES controller with the ARW decreased error in tracking control before ANN learning and realized appropriately inverse dynamics model after ANN learning.The realized IDM was also found to have a possibility to be used as an open-loop controller.The EFL-FES controller with the ARW developed in this study was found to be effective in multiple DOF movement control stimulating several muscles.It is expected to test the FEL-FES controller with more practical target trajectories and to study stiffness control for getting practical to restore motor function of paralyzed patients.

Figure 1 :Figure 2 :
Figure 1: Block diagram of the FEL controller for FES (a) and structure of ANN used in the FEL-FES controller (b).θ d and θ represent the desired and the measured joint angles.I PID and I ANN are outputs of the PID controller and the ANN.The ANN realizes the IDM of the controlled limb after learning.

3. 1 . 5 5Figure 3 :
Figure 3: Target joint angle trajectories on the joint angle plane for ANN learning.Small closed circle shows start point of tracking control on the target trajectory.

Figure 4 :Figure 5 :
Figure 4: Outline of the musculoskeletal model for FES.

6 AdvancesFigure 6 :Figure 7 :
Figure 6: Mean error (ME) in feedback control before ANN learning.Average and SD are shown.For normal subjects, 9 control results (3 trials with 3 subjects) were averaged for each cycle period, and result with 5 s of cycle period was shown for comparison in 6 s of cycle period.

Figure 8 :
Figure 8: Mean error (ME) in open-loop control with the trained ANN.Average and SD are shown.

Figure 9 :
Figure 9: An example of open-loop control with the trained ANN.Joint angles and ANN outputs are shown.Three of six cycles in one control trial were shown.(model A, Palmar, cycle period of 3 s).Values of ME were 7.1 deg for palmar flexion/dorsiflexion and 4.8 deg for radial flexion/ulnar flexion in (a), and 2.1 deg and 2.0 deg in (b), respectively.

Table 1 :
Movements and muscles included in computer simulation.