A Review of Gait Optimization Based on Evolutionary Computation

Gait generation is very important as it directly affects the quality of locomotion of legged robots. As this is an optimization problem with constraints, it readily lends itself to Evolutionary Computation methods and solutions. This paper reviews the techniques used in evolution-based gait optimization, including why Evolutionary Computation techniques should be used, how fitness functions should be composed, and the selection of genetic operators and control parameters. This paper also addresses further possible improvements in the efficiency and quality of evolutionary gait optimization, some problems that have not yet been resolved and the perspectives for related future research.


Introduction
Compared to wheeled robots, legged robots usually possess superior mobility in uneven and unstructured environments.This is because they can use discrete footholds to overcome obstacles, climb stairs, and so forth, instead of relying on a continuous support surface.
A gait is a cyclic, periodic motion of the joints of a legged robot, requiring the sequencing or coordination of the legs to obtain reliable locomotion.In other words, gait is the temporal and spatial relationship between all the moving parts of a legged robot [1].Gait optimization is very important for legged robots, because it determines the optimal position, velocity and acceleration for each Degree of Freedom (DOF) at any moment in time, and the gait pattern will directly affect the robot's dynamic stabilization, harmony, energy dissipation and so on.Gait optimization determines a legged robot's quality of movement.

Gait Generation Is a Multiconstrained, Multiobjective
Optimization Problem.Gait generation, which incorporates mobility and stability, is a very challenging task for legged robots, because their system of locomotion has multiple DOFs and a variable mechanical structure during locomotion.As a result a large number of parameters have to be established.For example, 54 motion parameters have to be considered for the walk gait of the Sony AIBO robot [2].
To obtain a natural and efficient gait for a legged robot, two kinds of strategies for sequencing or coordination of the leg movements can be followed.
The first strategy assumes that the gaits of humans or animals are optimal, as otherwise they would not have been able to survive the competition and natural selection proposed by Darwin's Theory of Evolution.This assumption has been proved accurate [3].The constrained optimization hypothesis suggests that gait parameters are selected to optimize (minimize) the objective function of the cost of transport (metabolic cost/distance) within the limitations of imposed constraints [4].A lot of research has shown that humans and animals move in a way that minimizes the metabolic cost of locomotion and validates the idea that the gait synthesis of legged robot is a constrained optimization problem [5][6][7][8][9][10][11][12].
Robots simulate human or animal behavior [13].Therefore, it is quite natural to use biological locomotion data to control the gait of robots.For example, Human Motion Captured Data has been adopted to drive a humanoid robot [14].However, some research indicates that biological locomotion data cannot be used directly for a legged robot due to kinematic and dynamic inconsistencies between humans/animals and the legged robot.This implies the need for kinematic corrections when calculating joint angle trajectories [14].
The second strategy formulates the gait generation problem of the legged robot as an optimization problem with constraints.It generates the optimal gait cycle by minimizing some performance indexes, for example, velocity of motion, stability criteria, actuating forces, energy consumption, and so forth.The gait generation problem of legged robots often has several objectives, and some of these objectives may be contradictory to each other (for example, speed and stability).Thus the gait generation problem can be stated as a multi-constrained and multi-objective optimization problem [15].
These two gait generation strategies may reach the same goal by different routes because both of them actually solve the gait synthesis problem as a multi-constrained multi-objective optimization problem.Once a database of precomputed optimal gaits has been created, the robot can cover the entire interval of precomputed optimal gaits by interpolation and thus realize smooth real time locomotion.

Evolutionary Computation Is Suitable for the Gait Optimization Problem.
The dynamic equations of legged robot locomotion are high order highly coupled and nonlinear, and gait optimization for legged robots requires searching a set of parameters in a highly irregular, multidimensional space.As a result, the standard gradient search-based optimization methods are not useful for legged systems with high DOF [2,16,17].
Evolutionary Computation (EC), including the Genetic Algorithm (GA), Genetic Programming (GP), Evolutionary Programming (EP), and Evolutionary Strategy (ES), is a natural choice for the gait optimization of legged robots.
First, EC uses optimization methods based on Darwin's Natural Evolution Theory.According to this theory, the locomotion mechanisms of life forms resulted from natural selection and the interaction between individuals and the natural environment.This makes the use of EC a natural choice, as it is biologically inspired and can generate biologically plausible solutions [18].
Second, from the computational point of view, EC also fits well with the gait optimization of legged robots [2,18,19], because of the following: (a) Gait optimization problems can have multiple criteria, multiple constraints, as well as multiple design variables, and EC has been shown effective for these kinds of large-dimension, multi-objective, multiconstraint optimization problems.(b) EC has been seen to be robust for search and optimization problems and has been used to solve difficult problems with objective functions where local information such as continuity, differentiability, and so on is not available, even though it is very important for gait optimization, as the objective functions of gait optimization may be very complex and it is very difficult to obtain this local information.(c) Because of the complexity and high DOF of the mechanical structure, it is difficult to obtain a precise dynamic model of a legged robot [20].EC will be efficient as this method is resistant to noise in the evaluation function and offers a model-free approach to optimization, only requiring feedback from the environment to improve performance when online evolution is deployed with a real robot.(d) EC has strong global search capability and is also insensitive to the initial population.Therefore EC decreases the risk of being trapped in a local minimum for finding a true optimum solution.(e) EC can easily be parallelized.Since gait optimization of legged robots is often a large-scale problem and the objective function and constraints are often complex, the process of evolutionary optimization may be very time-consuming because of the high computational cost of EC due to iterative evaluations of candidate solutions.Therefore it is advantageous to use parallel implementations of EC to gain efficiency and improve the solution quality of EC-based gait optimization.1.This offers a first glance at the application of EC technique for gait optimization of legged robots.A lot of EC models have been adopted to solve gait optimization problems.The gaits most often studied include the gaits of biped, quadruped, and hexapod robots engaged in walking, running, negotiating sloping surfaces, and going up and down stairs [21][22][23][24][25][26].

How to Evolve the Optimal Gait
The Genetic Algorithm (GA) is the gait optimization tool which is most often used, and some modifications can be introduced to fit the specific problems of gait optimization [25,27,28].For example, interpolating and extrapolating operators [29], two-point crossover, Gaussian mutation, overlapping populations [2], and Elitism strategy have been adopted.The explicit fitness sharing mechanism [30] has also been adopted to prevent premature convergence to suboptimal extremes.This speciation technique divides the population into a fixed number of species, where each species contains individuals that are similar to each other, and can force similar members of the same species to "share" one representative score, thereby penalizing species with a large number of individuals and allowing new species to form even if they do not perform as well as other, larger, species [30].
Adaptability that can adaptively change the probabilities of crossover and mutation is introduced in GA to balance global and local exploitation and exploration towards the progress of evolutionary optimization.For instance, Adaptive GA is used to optimize the gait of a humanoid robot ascending and descending a staircase by searching optimal trajectory parameters in blending polynomials [31].
Adaptive mechanisms may also be applied to control mutation rate [2].This method places radiation (the level of radiation decreases over time) into the middle of a region where a large group of individuals is clustered within the same locality to dramatically increase the mutation rate in this area, causing all the individuals to mutate in the next generation and to disperse to other areas of the space.It is reported that this mechanism can be useful in controlling the learning behavior of GA and makes GA more robust with respect to noise in parameter evaluations preventing premature convergence to suboptimal extremes.
Genetic Programming (GP) [32] and Grammatical Evolution (GE) are often used to evolve the gait of robots.
Simulation results obtained using GP on an AIBO quadruped in the Webots environment are reported much better than those obtained using simple GA-based approaches [17].In this approach, the gait is defined using joint angle trajectories instead of locus of paw to reduce the search space of optimization.An elite archive mechanism (EAM) is used to prevent premature convergence and improve the search capability of GP.EAM can preserve elite individuals at an early stage and flow them into in later stage.In this way, genetic material from elite individuals at an early stage is used to refresh an evolutionary convergent state and to create a role for preserving diversity as long as possible.
GE is one of the most popular forms of grammarbased GP.The advantage of GE lies in that it allows the user to conveniently specify and modify the grammar, whilst ignoring the task of designing specific genetic search operators.Thus GE can be used to optimize pre-existing motion data or generate novel motions.Using a Fourier gait representation to encode the chromosome and the dynamic similarity principles as a constraint, GE is employed to optimize the gait retargeting problem in animation.It successfully modified one animal's gait cycle data into a different animal's gait cycle data in computer simulations of animal locomotion [33].The same method can also be used to optimize the gait of a walking horse from a veterinary publication into a physics-based horse model [34].
ES is also employed to solve the gait optimization problem [35,36], and some encouraging results have been obtained.
Using a hand-tuned gait as a seed, the bipedal gait is directly evolved on a physical robot by an ES approach with parametric mutation and structural mutation.After hundreds of evaluations significant improvements were obtained for a functioning but nonoptimized bipedal gait that improved the walking speed by around 65% compared to the hand-tuned gait [37].
A hybrid approach of space-time optimization and covariance matrix adaptation evolution strategy (CMA-ES) has been proposed to generate gaits and morphologies for legged animal locomotion [38].It effectively generated dynamic locomotion gait of bipeds, a quadruped, as well as an imaginary five-legged creature by simulation.The gaits and morphologies produced are reported lifelike and exhibit many qualitative traits seen in real animals.This hybrid approach may combine the efficiency in high-dimensional spaces and the ability to handle general constraints of spacetime optimization with the ability to handle nondifferentiable variables and to avoid many local minima from CMA.
Apart from traditional EC and its variations, some relatively new types of EC have also been applied to gait optimization research.
Estimation of Distribution Algorithms (EDAs) are evolutionary algorithms based on probabilistic models that replace the operators of mutation and crossover used in GAs.The main advantage of EDA lies in the fact that the knowledge about the problem acquired previously can be used to set the initial probability model, and the global statistical information about the search space can be extracted directly by EDA to modify the probability model with promising solutions.This can reduce the search space and obtain good solutions in a shorter time interval.For this reason EDA has been used to study the gait optimization problem [15,39,40].For example, EDA has been applied to optimize the gait of the AIBO robot.A fitness function based on direct evaluation of the robots was adopted, and significant improvement of the previous gait was achieved over a short training period [41].
In some cases of gait optimization, the performance of a gait cannot be directly measured or calculated based on certain functions.In this case human preferences, intuition, emotions, and other psychological aspects can be introduced into the target system.Interactive evolutionary computation (IEC) is a form of evolutionary computation where the fitness function can be replaced by the user.A prominent advantage of IEC is that it can reflect user preference and allow optimization of the solution with a minimum of required knowledge in the problem domain [42].
Staged Evolution, which evolves the result in a number of stages, has also been proposed for gait optimization [20,42,43].This approach employs a strategy of divide and conquer.By introducing a staged set of manageable challenges, it decreases the search space and thus improves the convergence rate of EC and obtains rapid evolution of behavior towards a given goal.
The multiobjective multiconstraint problem is often solved by combining the multiple objectives and constraints into a single scalar objective problem using weighting coefficients.To do this, some problem-specific information is needed, and the relative importance of the objectives and constraints should be decided.In the complex problem of gait optimization, it is difficult to know this information in advance.In addition, there is no rational basis for determining adequate weights for these competitive or conflicting criteria, and the objective function that will be formed may lose significance due to the combination of noncommensurable objectives [44].Therefore, more and more gait optimization problems are parameterized and optimized using tailored Multiple Objective EC procedures [45], for example, the Strength Pareto Evolutionary Algorithm [46] and Nondominated Sorting Genetic Algorithm with Fitness Sharing method [44], and the obtained Pareto-optimal gaits, which is a set of nondominated or noninferior gaits that satisfies different objective functions.These methods have shown good performance [44,[46][47][48].

Gait Representation and Chromosome
Encoding.The gait of a legged robot may be represented in three-dimensional space [30,49] or in joint space [17,20,50].
In order to control a legged robot's movement, it is necessary to generate the trajectories of all the joints.Therefore, gait is usually represented by a sequence of key poses (states) extracted from one complete gait cycle [51], and phases between these key poses are approximated by a polynomial function, for example, 3rd, 4th, or 5th order polynomials.These polynomial functions are adopted because they can insure that the joint trajectories are smoothly connected with first-order and second-order derivative continuity.First order derivative continuity guarantees the smoothness of velocity, while the second order guarantees the smoothness requirements of acceleration or the torque in the joints [20].As a result, the gait of robot will look natural.
If only the foot placement point of these key poses is specified, once the foot trajectories are generated, inverse kinematics should then be used to convert the locus of foot into the joint angles required to generate the foot placement curves for a particular gait [17,20,30,47,[52][53][54].
To make the robot optimally move from its current position/stance to a goal position/stance, other parameters apart from those of the leg joint trajectories should also be considered [47,55], for example, parameters describing the position and orientation of the body, how the robot's weight shifts during walking, whether or how much the arms swing, and so forth.
The joint angles in these states, the coefficients of the polynomials, and some of the other parameters mentioned above are the design variables to be optimized by EC [18,39,56,57].These design variables, when treated as genes and arranged in an array, make up a chromosome of EC [16].
A variety of chromosome encoding methods, including the gray code representation [43], real number coding [57], mixed encoding of floating point number, and binary number [1], have been adopted, but the most often used encoding is the real coded method.This is due to difficulties associated with binary representation when dealing with a continuous search space with large dimension [44].

3.3.
Composing the Fitness Functions.EC are a family of objective function driven optimization algorithms.The objective/fitness functions represent the problem environment and decide how well the individual solves the problem.Therefore, the construction of a fitness function is very important for the correct functioning of EC, and researchers should define these objective functions appropriately according to the task to be accomplished so that each individual's actual behavior can be evaluated correctly and efficiently.
A lot of criteria can be used to construct the fitness functions of EC for gait optimization, and at present most studies mainly emphasize only one or a part of the following aspects [1].
(a) Maximum Velocity.The gait should help the robot attain maximum velocity, so the robot's speed of locomotion is a basic performance index [47,51,55,58].
(b) Minimum Consumed Energy.The criterion most often used for gait optimization is the minimum consumed energy (MCE) [1,20,39,40,44,46,56,57,[59][60][61][62][63][64][65][66] as an energy-efficient locomotion pattern results in a more natural walking motion.In fact, the MCE gait of a biped robot is similar to that of human.Another advantage of MCE criteria is that the consumed energy should be reduced so as to maximize battery operation time.
(c) Minimum Torque Change.The criterion of minimum torque change (MTC) is based on smoothness at the torque level [44,46,57,59,60].This may result in a more stable motion due to a smoother change in link acceleration.
(d) Stability.In order to make the robot move in an environment and avoid falling down, it has to have a stable gait.Stability is the most important constraint and is most often used in gait optimization.Stability can be static or dynamic.Static stability can be verified via the center-of-gravity index [18,59], while the dynamic stability is often verified via the zeromoment-point (ZMP) [1,14,16,20,39,40,46,47,57,60,67], which has an important role in gait optimization, especially for the biped gait.If the robot has to focus on how to restore balance rather than constantly trying to maintain dynamic balance, the foot placement estimator (FPE) can be adopted [68].
(e) Geometric Constraint.This ensures the feasibility of robot gaits from the point of physical structure [1,14,40,58,67,69].When the robot is passing through obstacles or climbing stairs, the gait should not lead to a collision between the robot and its environment.
When the robot walks, the swing limb has to be lifted off the ground at the beginning of the step cycle and has to be landed back at the end of it.
(f) Smooth Transition Constraint.To have a continuous periodic motion, the initial posture and velocities should be identical to those at the end of the step [44,56,62,69].For a humanoid robot, the horizontal displacements of the hip during the single and double support phases must also be continuous.When a walking robot's swing limb makes contact with the ground (heel strike), the effect of impact should be minimized so that it does not influence the motion stability of the robot [1,14,40,57,58].
More criteria can be added to achieve other practical requirements in gait generation and optimization, and the constraints can be formulated as equalities and inequalities.These criteria will serve as objective functions for evolutionary-based gait optimization.
There are two ways to evolve gait for a robot, namely on-line evolution and off-line evolution.On-line evolution evolves gait directly on a real robot [32,43,47,49,51,70,71], while the off-line method evolves gait on a simulator [18,59,72].In the case of off-line evolution, solutions are evaluated using the objective functions mentioned above.In the case of on-line evolution, the fitness may not be directly calculated, instead it will be determined based on measurements, that is, the solutions have to be tested by letting the robot actually walk with the parameters encoded by the chromosome, and the fitness for each individual is evaluated using the robot's sensors (digital camera, infrared sensor, and gyro-sensor) or directly evaluated by the user [30,42].

The Genetic Operators and Control Parameters of EC.
In order to make EC work properly, a set of control parameters and some genetic operators, for example, selection, crossover, and mutation should be predefined.
(a) Selection.Selection is performed so that better individuals are chosen for breeding and surviving.There are quite a lot of methods for selecting individuals for genetic operations.The mostly commonly used method is the roulette wheel procedure [16,47], which assigns a higher probability of selection to an individual if its fitness is determined to be better.Tournament selection is another commonly used selection procedure.Binary tournament randomly selects two individuals from the population at each time and chooses the fitter one [42].Other tournament algorithms simultaneously select individuals as parents and other specific individuals to be replaced [32,49].The parent(s) is the individual(s) with higher fitness, and the individual with the lowest fitness is replaced by the offspring of the parent(s).Elitism strategy is commonly adopted.This guarantees that the fittest individuals will always be retained into the next generation.
( Therefore, the population size should be carefully decided according to the size or difficulty of the problem, and a compromise between efficiency of computation and diversity of solution should be made.This is why Chernova suggested that a population size of 30 is a good choice [2]; however Eperješi reported that a population of the same size lost most of the genetic material quite quickly and converged to local minima [42]. (e) Initial Population.The initial population can be randomly created in two ways.One generates the initial population by mutation using a hand-tuned gait as a seed [29,51], and the other generates the initial population with a uniform distribution over the given search range [32,49,50,71].
(f) Maximum Generation Criterion.EC will iteratively apply the genetic operations until a certain termination criterion is met.Typical termination criteria employed in EC include: the realization of a predefined total number of iterations, having reached a maximum number of iterations without improvement, and even more complex mechanisms based on estimating the probability of being at a local optimum.Maximum generation is the most often adopted termination criterion in gait optimization.
The value of the maximum generation may vary between 30 to 5000 iterations [16,18,20,44,47,51,56,58,67,69,73].This should also be carefully determined according to the complexity or difficulty of the problem.Due to the stochastic nature of EC, each evolutionary optimization experiment may have to be repeated a number of times to ascertain the global optima [18].

The Effect and Efficiency of Evolutionary Gait Optimization.
Almost all research, whether it is simulation or physical experiment, has shown that EC achieves good results.The advantages of evolutionary gait optimization include the following.
(i) As the joint torques and link accelerations change smoothly, the final motion of the robot is very smooth, and the impact of the foot with ground is minimal.Therefore, the best-evolved gaits determined by this method outperform the best manually developed gaits in their ability to move straighter and faster, while at the same time being more flexible and reliable [2, 18, 32, 44, 49-51, 57, 71, 74].
(ii) The energy consumption of the optimal motion was dramatically reduced [44,56,57].The optimal gait pattern was natural and very similar to that of humans [44,59,73], and battery actuated robots can thus prolong their operation time.
(iii) EC can autonomously and more efficiently search for high performance gait parameters for various surface conditions and different robot platforms compared to using a manual approach [2,50].Furthermore, the EC-based approach was able to match the best previously known AIBO gait within a matter of hours even starting from a random population [2].Multi-objective evolution was able to find optimal humanoid robot gaits with completely different characteristics efficiently in one simulation run [44].

Comparing EC with Other Global Optimization Approaches
Besides evolution-based optimization techniques, other global optimization approaches that adopt a nonevolutionary metaphor have also been employed in gait optimization.These also search for the global optimum of the cost function without using the differential information of a given cost function.
Particle Swarm Optimization (PSO) can be used to optimize the stable and straight movement patterns (gaits) of a humanoid robot with the control signals of the joint angles produced by a Truncated Fourier Series (TFS) [75,76].It is reported that PSO optimized TFS significantly faster and better than GA to generate straighter and faster humanoid locomotion because PSO bypassed a local minimum that GA was caught in [76].The authors therefore concluded that PSO is better than GA as a learning method for the gait optimization problem in a non-deterministic environment.
We argue that GA may not necessarily be inferior to PSO in gait optimization, even in a non-deterministic environment such as the one in this experiment.This is because the PSO employed in this experiment was Adaptive PSO, which has a dynamically adjustable nonlinear parameter of inertia weight w to control the balance between global and local exploration.A larger inertia weight facilitates a global search, while a smaller inertia weight facilitates a local search [77].The GA employed in this experiment is just a canonical paradigm with roulette wheel selection and a fixed rate of crossover and mutation.This may be the reason why PSO can speed up the search and perform better than GA in this experiment.
EC of course can employ the same mechanism to improve its efficiency.For example, Adaptive GA can adaptively change the probabilities of crossover and mutation during the process of evolution.In ES, the step size or mutation strength is often governed by self-adaptation (evolution window), and the individual step sizes for each coordinate or correlations between coordinates are either governed by self-adaptation or also by covariance matrix adaptation (CMA-ES) [38].
Adaptive PSO is used to optimize the fastest forward gaits of the quadruped robot AIBO with the whole learning process running automatically on the physical robot [78].Starting with randomly generated parameters instead of hand-tuned parameters, several high-performance sets of gait parameters are obtained, and these gaits were reported as being among the fastest forward gaits ever developed for the same robot platform.
Parallel PSO was applied to large-scale human movement problems, and experimental results show that PSO was outperformed by the gradient-based algorithm [19].It is reported that a single run with a gradient-based nonlinear least squares algorithm produced a significantly better solution than did 10 runs using global PSO.Thus the authors do not recommend using the PSO algorithm for solving largescale human movement optimization problems possessing constraints or competing terms in the cost function.
The results of this experiment may be a fortunate exception.The objective functions of large-scale gait optimization problems with hundreds of design variables will no doubt be massively multimodal and the landscape must be very rugged.Therefore a gradient-based algorithm will certainly be trapped in a local minimum, and the global search ability of EC is absolutely necessary for decreasing this risk.We agree with the suggestion of the authors that a global local hybrid algorithm may be necessary for PSO and other global optimizers to solve large-scale human movement problems efficiently.
As far as we have seen from the literature, Ant Colony Optimization (ACO) has not yet been used in the field of gait optimization though this too is a famous metaheuristic of Swarm Intelligence (SI) similar to PSO and has been widely used to solve a lot of kinds of optimization problems.
The univariate dynamic encoding algorithm for searches (uDEAS) has also been applied to the gait optimization problem of a biped model walking up and down a staircase.The simulation results show that uDEAS outperforms adaptive GA with a 17 s versus 126 s run time on average and a slightly smaller minimum for best cost values [79].The authors attribute this result to the effectiveness of describing trajectories with the blending polynomial of uDEAS.
The problem representation method and the genotype encoding method directly determine the size and the characteristic of the search space and as a result directly affect the efficiency of EC optimization.Therefore, we suggest that researchers should pay attention to both the problem representation method and the genotype encoding method when studying the EC-based gait optimization problem.For example, TFS is reported to be a good gait representation approach that can generate suitable angular trajectories for controlling bipedal locomotion.This is because it does not require inverse kinematics, and stable gaits with different step lengths and stride frequencies can be readily generated by changing the value of only one parameter in the TFS [76].
Though some comparison of performance between EC and other non-evolutionary global optimization approaches has been reported, no systematic comparative study has been carried out.Such a systematic comparative study may be not necessary or not feasible because we often search for a set of satisfactory solutions instead of an absolutely global optimal solution.Both the robot platform and the objective functions of gait optimization will be different in each case, and thus it is difficult to find a benchmark robot and a set of benchmark objective functions to optimize.
Both EC and SI approaches are population-based iterative algorithms, even though they adopt different metaphors.Thus they share the same advantages and disadvantages in gait optimization, for example, a similar global and highdimensional search capability, multi-objective optimization capability, as well as a lot of control parameters which require tuning.One thing that can be said for sure is that EC, and SI approaches are proven good tools for gait optimization for legged robots, and further research should be done to improve their performance in the field of gait optimization.

Conclusions and Perspectives for Future Research
The most important conclusion that can be drawn from the literature mentioned above is that EC is indeed capable of achieving good performance in gait optimization and that this direction of research is very encouraging.However, it is obvious that we have not yet taken full advantage of EC, and some questions still need to be resolved for EC-based gait optimization.
To solve these problems, we suggest that future research should focus on the following issues.

Studying EC Approaches That Are Gait Optimization
Niches.One of the most obvious problems in EC-based gait optimization is that of computational efficiency.Evolutionary methods generally require the maintenance of a population of candidate solutions and an iteration of a large number of generations.Thus it can be very timeconsuming to evaluate every candidate gait especially when the experiment is carried out on a physical robot over several days while requiring constant manual supervision [32].
Though many researchers have optimized the gait of legged robots using EC, they mainly just transplanted the EC from other optimization applications with a few modifications.In order to solve gait optimization problems properly, it is necessary to study EC-based algorithms that are especially suitable for solving gait optimization problems, that is, study EC that can evolve the gait of robots with a high level of efficiency and quality.Some researchers have noticed the necessity of studying the gait optimization niching EC paradigm, operators, and parameters and have done some work in this area.For example, using interpolation and extrapolating operators instead of a crossover has been found to reduce the size of the population [29], which is an essential element for robotic applications because the time required for optimization may be considerably reduced.The ideas of parameter tuning and the use of a global-local hybrid algorithm for the global optimizers to solve large-scale gait optimization problems have also been suggested in [19,38].Some fragmentary information has also been established on the characteristics of the design space of the gait optimization problem, for example, the penalty terms of the constraints could result in the minimum of the objective function being located within a narrow "channel", and the shape of the design space could made it difficult to locate the global minimum without the use of gradient information, and so forth [19].This may be helpful for designing EC approaches.
The problem representation method and the genotype encoding method should also be studied as they can lead to different efficiency of EC.For instance, two different gait definition methods, a finite state machine based on the joint angles of the robot legs, and an Elman's recurrent neural network were studied [80], and the performance of the neural controller was reported superior (more stable, better displacement) for a simulated legged robot navigating on an irregular surface.TFS has also been reported to be a good gait representation approach and has been mentioned above [76].
Yet another method that could dramatically improve the gait optimization efficiency of EC has been proposed [81].This method harnesses the general purpose computing on graphics processing units (GPGPU) to produce hardwareaccelerated simulations without significant loss in fidelity, and it has achieved results that are orders of magnitude faster than software-only simulation (a 10-50-fold increase in speed has been reported).
We believe that EC-based gait optimization research should continue in this direction.The characteristics of the search space of gait optimization should be studied.This is determined by objective functions and constraints.Different applications of legged robot can have different objective functions and constraints, and different combinations of objective functions and constraints can have different search space characteristics.Based on the characteristic information of a search space, some benchmark test functions should be designed and classified to reflect to a certain extent the characteristics of the gait optimization objective functions.These benchmark test functions will facilitate the study of the corresponding EC paradigms, the genetic operators and control parameter sets of EC, providing researchers of legged robot gait optimization with a suite of high-performance optimization tools that can work well "out of the box" [19].In particular, we suggest that both the EC hybrid with local search and the GAs that can learn linkage may play an important role in gait optimization.The former can integrate the global search ability of EC with the local optimization ability of local search operators, and the latter can evolve optimal code sequencing by learning the linkage among the genes and therefore enhancing the search ability of EC in tackling complicated, large scale problems of gait optimization.

Investigating the Objective Functions of EC.
The objective function plays a critical role in EC.It serves as the environment for judging whether a solution represented by a chromosome in the simulated evolution is good or bad.It thus directs the search direction.The fitness functions in the research mentioned above were all different from one another.These were dependent on each researcher's insight into the nature of the problem, the different nature of each robot model or platform, and the performance requirements of each robot.
When constructing the objective functions for gait optimization, one has to compromise between the quality of solution and the velocity of evolution.A complex objective function including more performance indexes will, of course, lead to good solutions, but it can also greatly increase the size of the search space, and hence the computation cost of evaluation as well as the duration of the evolution.A lot of experience and techniques are needed to compose the objective functions (and the penalty weight set) properly.Therefore, it would be beneficial to construct a set of general guidelines for selecting the evaluation functions, especially for a typical application scenario or for a special robot platform [43].

Bridging the Reality Gap.
Another important problem in EC-based gait optimization is that the gait evolved by simulation usually does not yield the same behavior once it is transferred to a real robot, since simulation always includes some simplifications when modeling the real world.This is called the reality gap problem in the field of evolutionary robotics.It is a branch of artificial intelligence concerned with the automatic generation of autonomous robots [74] and of course includes the gait generation of legged robots.Though evolving gait directly in real robots is an attractive goal with certain advantages, it is also very time-consuming requiring heavy user-intervention.
A possible scheme for tackling the problem of the reality gap in gait optimization nowadays is to integrate off-line and on-line evolution with respect to the reality gap.
Staged evolution may be a good method for this purpose [32,42,43,82,83].This method evolves the gait by simulation first so that the gait may get a preliminary evaluation faster, and then the simulation results are transferred to the physical robot where the process of evolution is continued on the real robot.In this way, both the efficiency of EC and the quality of the solution may be significantly improved.A general multistage process, which minimizes the reality gap between real and simulated robots with respect to the behavior of actuators and their interaction with the environment, has been reported to be transferable to different kinds of legged robots [83].
Another method worth considering is the back to reality algorithm [74].This method employs a coevolutionary construction that allows continuous robot behavior adaptation interleaved with simulator adaptation.It was able to solve the gait optimization problem in real robots using fewer evaluations than most of the currently existing approaches.It was used for the gait optimization of AIBO and is claimed to be the first work in which the simulated gait been successful and constantly transferred to reality [74].
In conclusion, a lot of work still remains to be done.Evolutionary gait optimization for legged robots is a promising field of research, and future encouraging and interesting results can be expected.

Figure 1 :
Figure 1: A general block diagram for EC-based gait optimization.