Optimizing High-Dimensional Functions with an Efficient Particle Swarm Optimization Algorithm

,e optimization of high-dimensional functions is an important problem in both science and engineering. Particle swarm optimization is a technique often used for computing the global optimum of a multivariable function. In this paper, we develop a new particle swarm optimization algorithm that can accurately compute the optimal value of a high-dimensional function. ,e iteration process of the algorithm is comprised of a number of large iteration steps, where a large iteration step consists of two stages. In the first stage, an expansion procedure is utilized to effectively explore the high-dimensional variable space. In the second stage, the traditional particle swarm optimization algorithm is employed to compute the global optimal value of the function. A translation step is applied to each particle in the swarm after a large iteration step is completed to start a new large iteration step. Based on this technique, the variable space of a function can be extensively explored. Our analysis and testing results on high-dimensional benchmark functions show that this algorithm can achieve optimization results with significantly improved accuracy, compared with traditional particle swarm optimization algorithms and a few other state-of-the-art optimization algorithms based on particle swarm optimization.


Introduction
In both science and engineering, the particle swarm optimization (PSO) algorithm [1] is an important optimization technique that has been extensively used to find the global optima of multivariable functions. e PSO algorithm searches the variable space of a multivariable function by simulating the social behaviors of a group of animals. e animals in the group act cooperatively to find the potential location of the global optimum point, and the search pattern of each particle in the group is adjusted based on the exploring experiences of its own and other particles in the group.
Although the PSO algorithm can effectively search the variable space of a large number of multivariable functions and accurately determine their global optima, it has two disadvantages that may adversely affect its performance on some multivariable functions. Specifically, the speed of convergence is slow, especially in cases where the function that needs to be optimized is defined on a high-dimensional space. In addition, the algorithm is not guaranteed to find a global optimum and may converge to a local minimum point when the dimensionality of the function is large. A large number of techniques thus have been developed to further improve the performance of the PSO algorithm in the past two decades.
One strategy to improve the performance of the traditional PSO algorithm is to divide the individuals in the swarm into subgroups. In [2], the concept of subpopulation and a reproduction operator is introduced into the PSO algorithm. In [3], a dynamic multiswarm PSO algorithm is developed. Recently, a symbiotic particle swarm optimization (SPSO) algorithm is developed in [4] to optimize the neural fuzzy networks. In [5], the traditional PSO algorithm is modified to solve multimodal function optimization problems. In [6], the particles in the swarm are divided into mentors, mentees, and independent learner groups based on differences in fitness values and the Euclidian distances from the particle with the best fitness value. As a result, a new dynamic mentoring and self-regulation-based particle swarm optimization (DMeSR-PSO) algorithm is developed. Testing results show that DMeSR-PSO algorithm can achieve excellent and reliable performance on a number of benchmark datasets.
Another technique often used to improve the performance of PSO algorithms is to develop new rules for updating the positions of the particles. In [7], the cognitive and social behaviors of the swarm are randomized based on chaotic sequences and Gaussian distribution. Particles can thus search in a much broader area than the traditional PSO algorithm. In [8], a chaotic PSO algorithm is developed based on a virtual quadratic objective function constructed from the optimum point of an individual particle and the global optimum point. Recently, in [9], an improved PSO algorithm is developed to improve the performance of the traditional PSO by using a new strategy to move each particle. Specifically, particles in the PSO fly to their own predefined target instead of the location of the global optimum that has been found so far.
A third class of methods improves the performance of optimization by designing new strategies to update the velocities of particles. In [10], a self-adaptive PSO algorithm (SAPSO-MVS) that uses a novel velocity updating strategy to balance the exploring and exploiting capabilities of the PSO algorithm is developed to improve the performance. In [11], a new PSO-based optimization algorithm that can adaptively adjust the particle velocities in each step of iteration is developed. e adjustment is based on the current distance of each particle from the location with the best fitness found so far by all particles. Recently, in [12], a modedependent velocity updating approach is developed to balance the capabilities of the PSO algorithm for local search and those for the global search, which can effectively reduce the chance to be trapped within local minima.
Recently, in addition to the methods discussed above, a large number of other techniques [2,3,6,8,[13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31] have been developed to further enhance the performance of PSO-based optimization algorithms, and a comprehensive survey of these methods is available in [31]. In general, most of the existing techniques require sophisticated processing of the locations and fitness values of particles to determine the location and velocity of each particle for the next step of iteration [32,33]. ese approaches thus may become computationally inefficient when the function to be optimized is of a large dimensionality [34]. In practice, the optimization of these functions is often important [35]. For example, a protein sequence generally contains a few hundred amino acids. An amino acid usually consists of around 10 atoms. A protein sequence of moderate length is thus comprised of a few thousand atoms. e native folding of a protein sequence is generally predicted by minimizing the free energy of the system formed by the atoms in the sequence. e free energy of a protein sequence is thus often a function of a few thousand variables, and the accuracy of its minimization is crucial for determining the native folding of a protein sequence.
In this paper, we develop a new efficient PSO algorithm for the optimization of high-dimensionality functions. e algorithm performs optimization by a number of large iteration steps. Each large iteration step consists of two stages. In the first stage, an expelling force field is applied to each particle in the swarm to significantly enlarge the area the particles can explore. In the second stage, the standard PSO algorithm is applied to the swarm to converge to a global optimum. After a large iteration step is completed, a translation procedure is applied to each particle in the swarm to further expand the region explored by the swarm. An iteration step in this new algorithm can be efficiently performed, and the algorithm thus can be applied to the optimization of high-dimensional functions. Our analysis and testing results on high-dimensional benchmark functions show that this new algorithm can achieve significantly improved performance on optimization, compared with the traditional PSO algorithm and its a few existing variants.

The Proposed Algorithm
e standard PSO algorithm performs the optimization of a function by simulating the social behavior of a swarm of animals. Specifically, a swarm of particles with randomly generated initial locations is created in the variable space of the function. e fitness of a location in the variable space of the function is defined to be the function value of the location. Each particle is also assigned a randomly generated velocity. In each iteration step, the location with the best fitness value that has been found by all individuals in the swarm is determined. In addition, the location with the best fitness in the trajectory of each individual is also determined. e velocity of each particle is adjusted based on the location with the globally best fitness value and the one with the best fitness value in its own trajectory. e location of each particle is then updated based on the current velocity of the particle.
Given a swarm of mparticles, let x i,1 , x i,2 , . . . , x i,m be the locations of the particles in the ith step of iteration and v i,1 , v i,2 , . . . , v i,n be their velocities, respectively. g b,i denotes the location with the globally best fitness value that has been found by the swarm after i steps of iteration and b k is the location with the best fitness value in the trajectory of particle k(1 ≤ k ≤ m). e velocity of particle k in the i + 1th step of iteration can be computed as follows: where c 1 and c 2 are two positive constants and r 1 and r 2 are two randomly generated numbers between 0 and 1; ω is a positive constant between 0 and 1. e location of particle k in the i + 1th step of iteration can be computed from v i+1,k as follows: To effectively enhance the ability of a PSO algorithm to explore within the variable space of a high-dimensional function, we propose to perform the procedure of optimization by a number of large iteration steps. Each large iteration step contains two stages. In the first stage, the particles in the swarm are expelled from the center of mass for the particles in the swarm to explore a wider area in the variable space. is stage is the exploring stage of the optimization procedure. After the exploring stage is completed, the standard PSO algorithm is then applied to compute the global optimum of the function. e second stage is thus the converging stage of the optimization procedure.
In the exploring stage, the velocity of particle k in the i + 1th step of iteration can be computed from its location x i,k and velocity v i,k in the ith step of iteration as follows: where c 1 , c 2 , r 1 , r 2 , and ω are the same as those shown in equation (1); c 3 and β are positive numbers and r 3 is a random number between 0 and 1. y i is the center of mass for the particles in the swarm for the ith iteration step and can be conveniently computed as follows: e exploring stage executes for a certain number of iteration steps to perform an extensive exploration of the variable space. e converging stage starts execution after the exploring stage is completed. In the converging stage, equations (1) and (2) are used to update the velocity and location of each particle in the swarm until the specified number of iterations has been executed.
Before a new large iteration step starts, a translation procedure is applied to the particles in the swarm such that each particle is relocated to a different position. e velocity of each particle remains unchanged after the translation occurs. e translation of particle i is performed as follows: where p i is the location of particle i before the translation is applied and z i is its location after the translation; c 4 is a positive constant and r 4 is a random vector of the same dimensionality as p i , and each component of r 4 is a random number between− 1.0 and 1.0.

Analysis of the Algorithm
In this section, we show that the algorithm can effectively explore the variable space of a high-dimensional function in the exploring stage when appropriate values are selected for parameters c 1 , c 2 , and c 3 and the population size. From equations (3)- (5), no coupling exists between any two different dimensions in the coordinate of a particle in the swarm. Each individual dimension can thus be analyzed independent of others. e analysis of the algorithm can be simplified by choosing one particular dimension in the space and analyzing the behavior of the swarm in this dimension.
In the rest of this paper, we assume that r 1 , r 2 , and r 3 all follow a uniform distribution in interval [0, 1].

Definition 1.
Given positive constants c 1 , c 2 , c 3 , ω, and β and a positive integerm, let I m be the m × m identity matrix and A m be an m × m matrix where every element in A m is 1. e partial transition matrix P(c 1 , c 2 , c 3 , ω, β, m) is defined as follows: Definition 2. Given positive constants c 1 , c 2 , c 3 , ω, and βand a positive integerm, let I m be the m × m identity matrix. e complete transition matrix C(c 1 , c 2 , c 3 , ω, β, m) is a 2m × 2m square matrix and it can be determined from the partial transition matrix P � P(c 1 , c 2 , c 3 , ω, β, m) as follows: We assume that the variable space of a high-dimensional function is a "cubic" region in a multidimensional space, where the value of the lth dimension is within interval [L l , U l ] and both L l and U l are positive numbers. e following theorem shows that, for certain values of c 1 , c 2 , c 3 , ω, β, and m, the expected value of the lth dimension of each particle explores the corresponding dimension of the cubic region with a function of exponential order. Theorem 1. Given a "cubic" region in an m-dimensional space, where the value of the lth dimension is within interval [L l , U l ] and both L l and U l are positive numbers, x l (i, k) is the lth dimension of particle k in iteration step i, and e l (i, k) is the expected value of x l (i, k). Let λ 1 , λ 2 , . . . , λ 2m be the eigenvalues of the complete transition matrix constructed from c 1 , c 2 , c 3 , ω, and β and the population sizem, λ 1 , λ 2 , . . . , λ 2m are mutually different, and both Proof. Let v l (i, k) be the lth dimension of the velocity of particle k in iteration step i, g l (i) be the lth dimension of the best location found by the swarm in iteration step i, and b l (i, k) be the lth dimension of the best location in the trajectory of particle k in iteration step i. From equations (4) and (5), when i > 1, x l (i, k) can be written as follows: where r l (i, k) is defined to be Taking the expected value for equation (8), we have Mathematical Problems in Engineering where and H l (i, k) as follows: Equation (10) can then be written as follows: where C 1 and F l (i − 1, k) are defined as follows: From the fact that Equation (14) can be further written as (18) can be written as Define 2m-dimensional vectors X l (i) and Q l (i) as follows: Equation (19) can be written as follows: Since C(c 1 , c 2 , c 3 , ω, m) has 2m different eigenvalues, it can be decomposed into a form as follows: where Λ is a 2m × 2m diagonal matrix with λ 1 , λ 2 , . . . , λ 2m on its diagonal positions and the columns of Ψ are the corresponding eigenvectors of C(c 1 , c 2 , c 3 , ω, m). Based on equation (23), equation (22) can be further written as follows: where From equation (25), we are able to obtain the following equation: We assume that all particles in the swarm are within the "cubic" region in the iteration steps from 1 to i. From equations (12) and (13), we have From equation (16), the following equation holds for F l (i − 1, k) when i is sufficiently large: where c 1 and c 2 are selected to guarantee that . . ≤ |λ 2m |, both Ψ and Λ are constant matrices and X l (i) � ΨY l (i) and Q l (i) � ΨV l (i). We immediately obtain from equations (17), (26), and (29) that when i is large enough, there exist positive constants M 1 and N 1 such that e theorem thus follows.
Next, we show that c 1 , c 2 , c 3 , ω, β, and m can be selected to guarantee that |λ 2m | > 1 holds. e proof is based on the following well-known Gershgorin circle theorem. □ Theorem 2. Let A be an n × n matrix and R i (1 ≤ i ≤ n) denotes the circle in the complex plane with center a ii and radius j≠i,1≤j≤n |a ij |, that is, where C denotes the set of complex numbers. e eigenvalues of A are contained within R � ∪ 1≤i≤n R i . Moreover, the union of any k of these circles that do not intersect the remaining contains precisely k (counting multiplicities) of the eigenvalues.
It is clear that whenc 3 (β − 1) > c 1 + c 2 , r 0 > r 1 + 1 holds and R 1 and R 2 are thus disjoint and there is thus at least one eigenvalue λ r contained in R 1 . We have e lemma thus follows. From eorems 1 and 3, it is clear that when appropriate values are selected for c 1 , c 2 , c 3 , ω, and β, each particle in the swarm is able to enlarge its searched area with the order of an exponential function and the entire "cubic" region can thus be extensively explored by the swarm in a logarithmic number of steps. In practice, the value of |λ 2m |should be appropriate to guarantee a both efficient and extensive search in a desired region. eorem 3 only provides a sufficient condition for |λ 2m | > 1 to hold, and experiments are often needed in practice to determine the appropriate values for c 1 , c 2 , c 3 , ω, β, and m.

Testing Results
We have implemented this new PSO-based optimization algorithm into a computer program PSOTS in MATLAB and evaluated its performance on the minimization of a few high-dimensional functions in the Benchmark Dataset for PSO functions, which can be downloaded for free from the website at https://www.cil.pku.edu.cn/resources/somso/ 271214.htm. Its performance is also compared with that of a few other PSO-based algorithms, including the traditional PSO algorithm, the AsynLnPSO algorithm [14], the LinWPSO algorithm [13], the GPSO algorithm [19], the CCAS algorithm [21], and the VPSO algorithm [30]. GPSO, CCAS, and VPSO are state-of-the-art PSO-based algorithms particularly designed for the optimization of high-dimensional functions. A large amount of testing has been performed to determine the parameters that can optimize the performance of each of the four algorithms. For PSOTS, the population size is determined to be 200, the values of c 1 ,c 2 , and c 3 are set to be 0.7, 0.5, and 0.9, respectively. β is 1.0 and ω is 0.8. e performance of all four algorithms is then evaluated and compared. To evaluate the performance of a given algorithm on a function, the algorithm is executed 10 times on a randomly rotated version of the function. It is also interesting to observe that the best minimized function value obtained with PSOTS on Ellipse is less than that obtained with the traditional PSO algorithm when the dimensionality is 100. However, the mean of the minimized function values obtained with PSOTS is larger than that obtained with the traditional PSO algorithm in the same dimensionality. When the dimensionality increases from 100 to 150, PSOTS outperforms the traditional PSO algorithm in both the best and mean of minimized function values. is particular example suggests that the exploration ability of PSOTS is stronger than that of the traditional PSO, and the advantage of its exploring ability over the traditional PSO algorithm is further enhanced when the dimensionality of the function that needs to be optimized increases.
We then increase the dimensionality of the functions to higher values and tested the performance of all seven algorithms on the same set of functions when the dimensionality is 200, 300, and 500, respectively. Tables 3-6 show the performance of the seven algorithms on 10 functions from the benchmark dataset when the dimensionality is selected to be 200, 300, 500, and 1000, respectively. It is clear from the tables that PSOTS significantly outperforms the other six algorithms in 9 tested functions in both the best and mean of minimized function values. Although the other six algorithms still outperform PSOTS on function Schwefel, the relative performance of PSOTS is improved significantly when the dimensionality increases. Table 5 shows that PSOTS outperforms the traditional PSO and AsynLnPSO in the best of the minimized function values of Schwefel when the dimensionality of the function reaches a value of 500.
Compared with other PSO-based optimization algorithms, a certain portion of the iteration steps of PSOTS is dedicated to the exploration of the variable space of a function.
is exploring process may not always lead to Mathematical Problems in Engineering 5 improved performance of optimization. As a direct result of the exploration, fewer iteration steps are available for the convergence process. e convergence rates of PSOTS thus may be lower than those of other PSO-based algorithms on functions that do not have a large number of local optima. Function Schwefel in the benchmark set is an example of such functions. In addition, the global minimum of Schwefel is located in a straight line with small gradient values in the variable space. e existence of this straight line may further slow down the convergence process of PSOTS. Specifically, in the converging process of any PSO-based algorithm, particles tend to move toward the straight line rapidly while converging slowly along the direction of the straight line to the global minimum. In PSOTS, since particles are expelled apart from one another during the exploring process, the converging process first moves them to points that have  Mathematical Problems in Engineering longer distances from the global minimum along the direction of the straight line and then let them converge to the global minimum. e longer distances to the global minimum clearly lead to optimization performance inferior to that of other PSO-based algorithms. e performance of PSOTS on functions with a dimensionality of a few thousand is also tested and evaluated. e dimensionality of two functions, including Ellipse and Sphere, is selected to be 5000, and all seven algorithms are used to minimize them. Each algorithm is executed 10 times. Table 7 shows the result of minimization obtained with the seven algorithms. It is clear from the table that PSOTS significantly outperforms the other six algorithms on both functions when the dimensionality is 5000.

Conclusions
In this paper, a new efficient PSO algorithm is developed for the optimization of high-dimensional functions. Based on a simple expelling force field and a translational procedure, the algorithm can explore in a significantly enlarged area in the variable space of a high-dimensional function and the performance of optimization can thus be significantly improved for functions that have a large number of local minima. Our analysis and testing results on high-dimensional benchmark functions also show that this new algorithm can achieve significantly improved performance on the optimization of high-dimensional functions, compared with the traditional PSO algorithm, a few of its variants, and a number of state-of-art algorithms for the optimization of high-dimensional functions.  Since a large number of PSO-based algorithms have been developed for the optimization of multivariable functions, it remains unknown whether this approach can outperform the PSO-based algorithms that have not been tested in our experiments on high-dimensional functions or not. A much more comprehensive testing and comparison is thus needed to evaluate the performance of this algorithm.
Recently, due to the rapid growth in the amounts of datasets with dimensionalities larger than ten thousand, the extraction of crucial features from such datasets has become an important problem in the area of data mining. Such datasets are often called ultrahigh-dimensional datasets. e exploring ability of the proposed approach may deteriorate significantly in such ultrahigh-dimensional spaces, which may adversely affect the application of this approach to the accurate processing of ultrahigh-dimensional datasets. Improving the optimization performance of this approach on ultrahigh-dimensional functions thus constitutes an important part of our future work.

Data Availability
e source code and testing data of this work are freely available upon request.

Conflicts of Interest
e authors declare that they have no conflicts of interest. Mathematical Problems in Engineering 9