Task-Oriented Parameter Tuning Based on Priority Condition for Biologically Inspired Robot Application

This work gives a biologically inspired control scheme for controlling a robotic system. Novel adaptive behaviors are observed from humans or animals even in unexpected disturbances or environment changes. This is why they have neural oscillator networks in the spinal cord to yield rhythmic-motor primitives robustly under a changing task. Hence, this work focuses on rhythmic arm movements that can be accomplished in terms of employing a control approach based on an artificial neural oscillator model. Themain challenge is to determine various parameters for applying a neural feedback to robotic systems with performing a desired behavior and self-maintaining the entrainment effect. Hence, this work proposes a task-oriented parameter tuning algorithm based on the simulated annealing (SA). This work also illustrates how to technically implement the proposed control scheme exploiting a virtual force and neural feedback. With parameters tuned, it is verified in simulations that a 3-DOF planar robotic arm traces a given trajectory precisely, adapting to uneven external disturbances.


Introduction
Researches on a robotic control of allowing natural behaviors have been given increasing attention due to the fact that animals are capable of naturally behaving under uneven environmental changes.For instance, animal natural movements, such as bipedal or quadrupedal walking and fluttering wings, are realized by means of interaction between the musculoskeletal system and neural oscillator networks, termed central pattern generators (CPGs), on spinal level in the central nervous system [1,2].In humans, specifically, each limb segment system joined with muscles and tendons shows a mechanical spring-like motion connected to a mechanical link by neural signals.This is called the neuromusculoskeletal system of humans.Hence, if a robot is built similarly as the human-like system with CPGs then it may be possible to show natural behavior.Here, the function, known as the entrainment property, of CPGs may be replaced with an artificial neural oscillator network [3].Thus our attention focuses on the robotic application of the entrainment function to adapt the nervous system to the natural frequency corresponding to a sensory feedback.
The application with an artificial neural oscillator to a robot control was started from Matsuoka's work [4,5].He showed mathematical proof on the steady state rhythmic patterned output including the output property according to various networks.Inspired by his works, his neural model has expanded for the various applications to the fields of robotics such as various walking robots for adaptive locomotion [6][7][8][9], crawling, swimming [10,11], and flying [12].In addition, employing other models, biologically inspired motions also have been proposed and verified [13].
Although the CPG-inspired control approaches contribute to enhancing adaptive mobility for a locomotive robot irrespective of uneven environments, novel potential from biological inspiration has been limited till now owing to the difficulty of producing a desired motion with many parameters and nonlinearity.In another application, a rhythmic arm motion of neuromechanical system coupled with the network of neural oscillators was performed under unknown collisions successfully [14].However even in the application to the robot arm control [15], precision control [16,17] according to a given motion was not exposed clearly.Recently, although parameter optimization methods [18,19] for satisfying performance requirements were proposed successfully, the dynamic performance of the coupled system interacting with various environments was not considered.Here, many parameters and the nonlinear property of the neural oscillator with the sensory feedbacks from unknown environments make parameter tuning difficult.
From early works, in order to follow a given trajectory including complicated behaviors in dexterous manipulation, meaningful works [20,21] have been performed with substantial difficulty of selecting the parameters.But such approaches are not clear and much time and efforts are needed in viewpoint of precisely performing an energyefficient motion.Thus this paper presents a parameter tuning framework based on simulated annealing (SA) according to a priority condition.In the proposed approach, the control gains and parameters of the neural oscillator networks are selected in terms of cost energy function sets prioritized as closely as possible without interfering with higher-priority constraints [22].This method is incorporated to a three-link robotic arm coupled with the neural oscillators for performing a given task considering task failures.In addition, extensive simulations for the verification explore dynamic performances with respect to the robustness of the tuned parameters and adaptive behaviors against unknown disturbances.
In this paper, the artificial neural oscillator as CPGs employed, at first, is briefly illustrated with the entrainment property.And the control method of coupling a robotic system with the neural oscillator is described.Section 3 gives how to tune the control gains with the oscillator's parameters based on the proposed optimization process in order for an imposed task.The verifications of the proposed tuning methodology are dealt with in Section 4 and Section 5 gives the properties of novel self-adaptive behavior in the proposed control approach.Finally, Section 6 draws conclusions.Matsuoka.It has been known that the CPG in vertebrates generates rhythmic-motor patterns and allows them to adapt to uneven external disturbances using sensory feedbacks.In order for technically applying such novel characteristics to a robotic system, the neural oscillator model suggested by Matsuoka is employed in this work.The model is composed of two neurons arranged for mutual inhibition as seen in Figure 1.If the neural model is coupled to a robotic system, the entrainment property of the neural oscillator enables the coupled system to show biologically inspired behavior that easily interacts with various environments:

Neural Model Proposed by
where   and   are the inner state of the th neurons for  = 1 ∼ .Here, the subscripts "" and "" denote the extensor and flexor neurons, respectively.V () indicates the degree of adaptation.The adaptation constant of the th neuron is represented by .The output of each neuron  () is taken as the positive part of   and the final output   of the oscillator is the difference in the outputs between the extensor and flexor neurons.  is a connecting weight from the th neuron to the th neuron:   are 0 for  ̸ =  and 1 for  = .    indicates the total input from the neurons arranged to excite one neuron and to inhibit the other, respectively.  and   mean the time constants of the inner state and the adaptation effect, respectively, and   is an external input with a constant rate. () is a weight of the extensor neuron or the flexor neuron and   indicates a sensory input from the coupled system which is scaled by the gain   .

Entrainment Property of Neural
Oscillator.In Figure 2, the entrainment procedure caused by the sensory feedback of the neural oscillator is well illustrated.There are many unknown parameters related with the entrainment property.From our previous works [16,17,20], insight and technique for parameter tuning have been given with various applications.Here, the neural dynamics should satisfy the condition [21] for the stable limit cycle behavior, and then the sensory feedback gain  which is most sensitive to entrainment should  be properly tuned.In simulation, the gain  was set with 0.02, 0.2, and 0.576 as seen in Figures 2(a) to 2(c), respectively.When  is set to 0.2 and 0.24, the output of the neural oscillator does not entrain the sensory feedback signal (see Figures 2(a) and 2(b)).In Figure 2(c), when  is tuned manually as 0.576, the closely entrained output is generated from the neural oscillator.It can be verified from the result that the sensory feedback to be tuned enables the neural oscillator to follow an unknown input signal, although a nonperiodic input is fed into the neural oscillator [16].
In general, in order to apply the entrainment property to the robotic systems control, a desired torque input to the joint can be expressed as where   indicates the proportional gain of the joint,  is the damping coefficient,  is the joint angle, and   is the output of the neural oscillator that produces the neural command corresponding to the joint.The neural oscillator senses the sensory signal  fed again from the joint and its output may be changed within a range corresponding to the sensory input.This characteristic is called "entrainment" that can be regarded as the tracking of sensory feedback signals.This causes that the coupled system shows the performance on adaptive behaviors interacting with various environments.In Figure 3, the simulation results indicate the dynamic responses including three characteristics based on the entrainment property with manually tuned parameters of the neural oscillator: (1) motion limitation employing the saturation effect of the output of the neural oscillator caused from a sensory feedback, (2) fully entrained output for interacting with various unknown environments, and (3) motion synchronization with a neural network for a multi-DOF robotic systems control.The first characteristic on the saturation effect is well illustrated between 5 s and 20 s.This property can be applied to the motion limitation using high gains.And the entrained motion for interacting with unknown environments through the sensory feedback is illustrated from 25 s to 30 s.In addition, in the next step, while entraining, we examined whether the synchronized motion by means of the network is possible or not.In A, (a) and (b) of Figure 3 have the phase difference  when the network of the neural oscillators is not connected.If the network between the neural oscillators is connected with a network gain, the phase becomes synchronized as shown in B of Figure 3.This is the third property (3) of the neural oscillator mentioned above.

Control of the Coupled Mechanical System with Virtual
Force and Neural Oscillator.If dynamics with th DOFs is considered for controlling a robot as seen in Figure 4, the governing equation could be expressed as where q = [ 1 ,  2 , . . .,   ] T ∈ R  denotes a vector of joint positions and H(q) ∈ R × and (1/2) Η() + S(q, q ) ∈ R × denote an inertia matrix and a centrifugal and Coriolis matrix, respectively.g(q) ∈ R  is a vector of gravitational forces or torques and  ∈ R  is a control input vector at the level of generalized forces described in (3): where Examine solution of the lowest cost function level in stores (database) Cooling schedule (reduce temperature) where   (> 0) and (> 0) are the spring stiffness and damping coefficient, respectively, for the virtual components.C 0 is the joint damping in which  is chosen within 0 <  < 1. k  and q  are the proportional gain and output of the neural oscillator that produces neural commands for entrainment, respectively.x  is the specified target position vector and x is the vector of the current position in Cartesian coordinates.In (4), the neural oscillator based control is composed of a virtual force and neural sensory feedback.The first term indicates the task space control of producing a desired force in terms of a spring and a damper.In the second and third terms, the desired input q  is yielded by an entrainment rule of the neural oscillator as described in Section 2.2.This enables a robot coupled with the neural oscillator to show self-reflexive motions against uneven environmental perturbations.And a joint damping, C 0 , plays an important role in restraining a certain self-motion under a redundant system.

Parameter Tuning Based on Simulated Annealing
Because rhythmic motions of the dynamic system are generated by the neural oscillator, it is usually difficult to obtain the desired motion required in the target task.This is the reason that there are many parameters in the nonlinear neural oscillator required to be tuned, and various responses are produced according to the interoscillator dynamics and the effect of the entrainment property in terms of sensory feedbacks (see Figure 3).Thus, for strengthening the advantageous of the proposed control approach employing the neural oscillator, this section informs a parameter tuning method considering the priority of the cost functions which is based on simulated annealing (SA) [23,24].
From the optimization problem to minimize cost functions   , the parameters () can be adopted by means of the proposed tuning method as seen in Figure 5.The parameters related with the virtual force and the neural oscillators are Note that if Δ  < 0, the state   corresponding to the th iteration is accepted and saved to database.If not, the transition probability, Prob  (  ), gives the new state by where a random value  is set from 0 to 1 uniformly.The temperature cooling schedule is   =  ⋅  −1 , where  is the effective annealing gain and temperature-dependant normalization factor notes ().In case that Δ  > 0 and Prob  (  ) <  or Prob  (  ) = 0, the state   is denied.
The parameter tuning procedure is terminated when the temperature equals zero.
In addition, the SA algorithm cannot be applied to parameter tuning for the robot task control directly because it is difficult to distinct whether the objective task is attained properly with the selected parameters or not.Hence, the optimization process is modified and upgraded in this work.First, each energy level calculated by the cost function sets is weighed and prioritized with a given objective and total energy is evaluated.Then, the task completion judgment step is newly considered as shown in Figure 5.If the desired task is not satisfied, the algorithm searches for and reselects the parameter set which gives the lowest energy level.

Application and Verification
In this section, whether a desired motion for multi-DOFs robotic systems coupled with the neural oscillators can be attained or not is verified.Also it is confirmed that the main characteristics of the neural oscillators illustrated in Figure 3 are helpful to keep the joint repeatability in a redundant system.For this, each joint of a three-link robotic system is coupled with the three neural oscillators, respectively, as shown in Figure 6.And the specilized SA algorithm for the parameter tuning is disclosed.The dynamics parameters and the initial control gains for the dynamic computer simulation are described in Table 1.

Implementation to a
Real Three-Link Robotic Arm.The desired trajectory is set to be Δx of (4).In simulation, the circular trajectory is given to the end-effector of a 3-DOF planner robotic arm (see Figure 6).The SA-based parameter tuning algorithm proposed in Figure 4 is additionally upgraded, since many parameters related with the virtual forces and the neural oscillators effect on attaining an objective task and embodying the entrainment property of the proposed control approach.Hence energetic cost functions for dynamic performance [25] of the robot arm and a tuning process are newly considered as seen in Figure 7.The proposed parameter tuning rule basically follows a hierarchical priority condition illustrated in Figure 7 for the reason that the solutions to consider the entrainment effect in the cost functions of the second step are subjected to the solution sets of satisfying the constraint cost function for a desired motion of the first step.In addition, in each process, the energetic cost functions (  ) are minimized for amplifying dynamic performance such as energy consumption and energy-efficient motion.Here, we define the parameter tuning rule based on the SA incorporating the three steps of Figure 7.
(1) Step 1. Set the initial parameters of the neural oscillator exploiting the following cost functions: subject to where  and   is the natural frequency and the amplitude of the neural oscillator for generating a desired motion.And   indicates the natural frequency of the output produced in the neural oscillator. max and  min mean the maximum and minimum amplitude constraints, respectively.V is the gain for the performance.
(2) Step 2. After the initial procedure of Step 1, the parameter tuning algorithm considering the cost functions corresponding to the first priority is operated as illustrated in Figure 7.
Here the priority rule is decided and divided due to the multiple objectives.In this step, the parameters are tuned for attaining the desired motion, considering the control input of the neural oscillator for entrainment.The cost functions are expressed as a weighted linear combination: where where   is the constraint cost function for tracing a given trajectory,  1 indicates the energy applied to the 3-DOF robotic arm,  2 represents the dissipative energy from the viscous friction of each joint, and the energy additionally loaded from the neural oscillators can be described as  3 .  and   denote stiffness and damping values as a weight, respectively.
(3) Step 3. The parameters selected in the procedure of Step 2 are fixed.Then, the third step focuses on enhancing the entrainment effect that contributes to repeatability and adaptability of the joints of a multi-DOF robotic arm.Thus the parameters related with the internal neural dynamics are tuned by means of the following cost functions: where   = (1/) ∫  1/2 ⋅ k  ⋅ (q  − q)  (q  − q), where   indicates a spring constant.

Verification through Simulation.
Incorporating Steps 1 to 3 in sequence of the proposed tuning method, we are able to acquire the appropriate initial and tuned parameters as seen in Tables 1 and 2. Figures 8 and 9 show the tuning procedure of the parameters described in Table 2 corresponding to Steps 2 and 3, respectively.Figures 8(a  Even though the optimized parameters correspond to the lowest cost energy level needed to execute the required task, completion of the task may fail.Hence, the proposed SAbased approach confirms whether the constraint condition on the position error is satisfied or not through dynamics simulation.From Figures 10 and 11, if the parameters of the proposed controller are properly selected for a desired task such as the circular motion, the given motion could be achieved clearly.In addition, the motion repeatability for a redundant robotic system is compared in Figures 10(b This is why each joint is tightly coupled with the neural oscillators.It is verified that each joint motion in Figure 11(b) is fully entrained with the outputs of the neural oscillators.This property enables the arm to sustain the given task against changes in parameters of arm kinematics and dynamics as well as disturbances.

Discussion on Entrainment Property
We, in this section, investigate the dynamic performances of the three-link robotic arm coupled with the neural oscillators.The parameters of the neural oscillators are set for enhancing the entrainment property by the proposed tuning algorithm.Entrainment implicates that their outputs lock onto sensory feedbacks to interact with unknown environments in phase as mentioned in Section 2.2.In simulation, during the circular motion of the end-effector shown in Figures 10 and 11, the robot arm collides with the unknown virtual walls: a vertical wall (Case I) and sphere wall (Case II).Here, it is confirmed that the end-effector of the robotic arm coupled with the neural oscillators traces the changed paths compliantly according to the surface of an unknown wall.The parameters of the neural dynamics were set in Table 2 and remained unchanged for all cases.In particular, Case III finally examines whether the control approach tuned precisely can be applied even in the change of the desired trajectory.
(2) Case II.A virtually built-up sphere wall.
(3) Case III.A variability of the desired motion.
In Figures 12 and 13, unknown environmental contact models such as a vertical and sphere wall for various collision effects make use of the unilateral spring-damper elements in simulation.The end-effector of the robotic arm is collided against the virtually modelled walls from 4.6 s to 7.5 s, respectively, as seen in Figures 12(a), 12(d), 13(a), and 13(d).Note that the neural model-coupled robot arm shows good adaptive motion as illustrated in Figures 12(d  comparing with Figures 12(a)-12(c) and 13(a)-13(c), although there happens to be collision with unknown walls irrespective of the shape.The outputs generated from the neural dynamics are helpful in keeping each joint motion.This is confirmed from the dash dotted lines of Figures 12(f) and 13(f).The blue line and red line of Figures 12(b), 12(e), 13(b), and 13(e) indicate the real motion drawn by the end-effector of the robotic arm and the desired motion, respectively.In addition, without coupling with the neural model, the repeatability of each joint motion is not guaranteed even in a rhythmic motion (see Figures 12(c) and 13(c)).Differently from this, comparing (d) of Figures 12 and 13 with Figures 12(a) and 13(a), respectively, the repeatability of the joint position in the 2nd joint is verified from A and B in Figures 12(d

) and 13(d).
A and B are the joint motions before and after the collision.In both of them, it is observed that the 2nd joint draws a specific motion repeatedly.
The additional novel feature of the tuned coupled system described in Case III is explored in Figure 14.Experimental results of Figure 14 indicate that there are significant improvement and possibility on the variability of the motion in the coupled robotic arm with the neural oscillator.Although the desired motions of the end-effector are given as the vertical and horizontal motions as seen in Figures 14(b) and 14(e), respectively, the coupled robotic arm is able to follow the new desired motions regardless of the parameter tuning as shown in Figures 14(a) to 14(b) and 14(d) to 14(e).It also can be verified from Figures 14(c) and 14(f) that each joint of the robotic arm keeps the motion repeatability.
As concluding remark of this section, reflex function by means of reciprocal action between the flexor and extensor neurons contributes to enhancing the natural behavior of a robot arm.In particular, irrespective of the change of a desired motion, the proposed control approach properly tuned for an imposed motion is possible to exhibit the entrained motion corresponding to a variety of environments adjusting the control outputs in an adaptive way.

Conclusion
This work introduces the parameter tuning algorithm for the application of the biologically inspired control approach to the skillful manipulation of a robotic arm.This can be regarded as an example for the embodiment of a human-like behavior property using a robotic arm whose joints were coupled to neural dynamics composed of a network.Differently from previous works that were only capable of rhythmicmotor patterned outputs, the main contribution of this work is to allow the end-effector of a robotic arm to precisely follow a predefined given trajectory with entrainment.The entrainment properties are helpful in keeping the motion repeatability and enhancing the compliance of the robotic arm.For technically accomplishing this, we propose a taskoriented parameter tuning method for properly determining the parameter values of the neural dynamics considering the priority condition of the cost functions based on the simulated annealing method.The novel performances of the proposed control and parameter tuning method were verified with various simulations.In addition, the detailed demonstration that the 3-DOF robotic arm compliantly responds to unknown environmental changes was performed with the parameters tuned.Finally, it could then be confirmed that the neural oscillator based control exhibits the variability of the robotic arm motion with the parameters set for only a certain motion.

Figure 2 :
Figure 2: The responses of the neural oscillator with respect to an arbitrary sensory input with the initial gain  (a) and roughly tuned gain  (b).(c) The response of the neural oscillator with respect to an arbitrary sensory input with the tuned gain .

Figure 3 :
Figure 3: Simulation results on main characteristics of the output of the neural oscillator with the entrainment property and the network.Case (a) is simulated at equivalent position 0 ∘ and the starting position of (b) is set with about 8.6 ∘ .The blue thin line is the output of neural oscillator and the red dotted line is the output of the coupled pendulum.

Figure 4 :
Figure 4: Schematic robot arm control model coupled with neural oscillators.

Figure 5 :
Figure 5: Flowchart of task-oriented parameter tuning based on the modified SA.

Figure 6 :
Figure 6: Schematic model for verification of a given task of threelink planar arm whose joints are coupled to the neural oscillator.

Figure 7 :
Figure 7: The modified SA-based parameter tuning rule according to priority level.
) and 9(a) indicate the cooling state in terms of the cooling schedule.The cooling or annealing gain  was set to 0.98.

Figure 9 :
Figure 9: (a) Transition of total cost function level; (b) transition of inhibitory connection weight; (c) transition of adaptation constant; (d) transition of rising time constant; (e) transition of sensory gain; (f) transition of tonic input.
) and 11(b).Differently from Figure 10(b), each joint motion shown in Figure 11(b) sustains the repeatability of the joint motion.

Figure 10 :Figure 11 :
Figure 10: (a) The circular motion rotated by the end-effector of the three-link arm without coupling to the neural oscillator.(b) The joint trajectories actuated by the three-link arm.

Figure 12 :
Figure 12: A comparison of the adaptive behavior in collision with a vertically modelled-vertical wall: (a), (d) snapshots of the robotic arm, (b) the trajectory, and (c) the angles drawn by the end-effector and each joint of the robot arm without coupling to the neural oscillator and with coupling to the neural oscillator, respectively; (e) the trajectory drawn by the end-effector; (f) the angles measured at each joint of the robot arm and the output of the neural oscillators.

Figure 13 :
Figure 13: A comparison of the adaptive behavior in collision with a vertically modelled-sphere wall: (a), (d) snapshots of the robotic arm, (b) the trajectory, and (c) the angles drawn by the end-effector and each joint of the robot arm without coupling to the neural oscillator and with coupling to the neural oscillator, respectively; (e) the trajectory drawn by the end-effector; (f) the angles measured at each joint of the robot arm and the output of the neural oscillators.

Table 1 :
Initial parameters (for neural oscillator and control method) and mechanical model.

Table 2 :
Each parameter selected by the proposed tuning algorithm.