This paper presents an experimental study that aims to compare the practical performance of wellknown metaheuristics for solving the parameter estimation problem in a dynamic systems context. The metaheuristics produce good quality approximations to the global solution of a finite smalldimensional nonlinear programming problem that emerges from the application of the sequential numerical direct method to the parameter estimation problem. Using statistical hypotheses testing, significant differences in the performance of the metaheuristics, in terms of the average objective function values and average CPU time, are determined. Furthermore, the best obtained solutions are graphically compared in relative terms by means of the performance profiles. The numerical comparisons with other results in the literature show that the tested metaheuristics are effective in achieving good quality solutions with a reduced computational effort.
The present paper addresses the problem of finding a set of parameter values in a dynamic system model that calibrates the model so that it can reproduce the existing experimental data in the best possible way [
Sörensen and Glover [
Most metaheuristics use random procedures that invoke artificial intelligence tools and simulate nature behaviors and their performance does not depend on the properties of the problem to be solved. They are alternative methods to find good approximations to optimal solutions of GO problems. In addition, they are derivativefree and easy to implement [
Previous use of metaheuristics, mainly those that use metaphors based on natural evolution and on behavior of animal swarms, to solve some DMbPE problems showed that they are able to provide good quality solutions when real experimental data and noisy artificial data are considered. In [
Since the problem of estimating the parameters of a dynamic model is important, the contribution of this study is concerned with the implementation and practical comparison of five very simple and easy to implement metaheuristics, hybridized with a local intensification phase, when solving the finite nonlinear programming (NLP) problem that arises when a numerical direct method of a sequential type is used to locate a global optimal solution to the DMbPE problem. The unknown parameters are the decision variables of the NLP problem.
The selected metaheuristics are very popular and have been used to solve a variety of realworld applications. The selection includes the FA [
The remainder of the paper proceeds as follows. Section
To solve the DMbPE problem, the sequential numerical direct method is used. For completeness, the DMbPE problem reads as follows. Find
To solve problem (
In direct methods, the optimization present in (
We use the sequential direct method and the system of ODE is numerically integrated by the Matlab function
To compute global optimal solutions to NLP problems, stochastic or deterministic methods are available. Stochastic GO methods are able to provide a nearoptimal solution in a short CPU time, although it may not be globally optimal. In contrast, deterministic GO methods provide an interval within which the global optimal solution falls, although they require very large computational efforts [
Most stochastic metaheuristics are classified in terms of source of inspiration as natureinspired algorithms. The wellknown swarmintelligencebased algorithms belong to a wider class called the bioinspired algorithms, and these are a subclass of natureinspired algorithms [
We use the notation
The FA is a bioinspired metaheuristic algorithm that is capable of converging to a global solution of an optimization problem. It is inspired by the flashing behavior of fireflies at night [
Table
Nomenclature for the FA.

Number of fireflies in the population 

Attractiveness of a firefly 

Absorption coefficient/variation of attractiveness 

Randomization parameter 

The brightest firefly 

The less brighter firefly 
The HS algorithm was developed to solve GO problems in an analogy with the music improvisation process where music players improvise the pitches of their instruments to obtain better harmony [
Nomenclature for the HS algorithm.
HM  Harmony memory 

Size of the HM 

The best solution in HM 

The worst solution in HM 

Harmony memory considering rate 

The pitch adjusting rate 

Distance bandwidth 
The HM contains
The DE is a bioinspired populationbased algorithm that relies on three strategies—
Nomenclature for the DE algorithm.

Number of points in the population 

Amplification parameter 

Mutant point 

Trial point 

Crossover parameter 
The initial population of points,
The components of the mutant vector are then mixed with components of the
The ABC algorithm is an optimization algorithm based on the intelligent behavior of honeybee swarms [
Nomenclature for the ABC algorithm.

Number of bees in the colony 

Number of food sources 

Food source/solution 

Mutant solution 
“limit”  Limit for abandonment 
At the initial stage, a set of food source positions are randomly selected by the bees; that is, the positions are randomly generated in the search space
On the other hand, during the onlooker bee phase, the food sources are randomly chosen according to probability values,
The tabu search (TS) algorithm, introduced to continuous optimization in the paper [
The DTS method developed in [
Nomenclature for the DTS algorithm.
TL  Tabu list 
VRL  Visited region list 

Ratio of accepting diversification point 

Edge length (with 

Region radius in VRL ( 
During the exploration procedure, DTS uses an adaptive pattern search strategy to generate an approximate descent direction (ADD) for the objective function
A pattern search method directs the search towards a minimizer using a pattern of specific number points. At least
For a fair comparison, and to improve the quality of the produced solutions, we propose the implementation of the HJ intensification phase with all the abovementioned metaheuristics. Based on the final solution provided by the metaheuristic, the HJ local search algorithm is invoked and allowed to run for
This section aims to present and analyze the statistical significance of the numerical results that were obtained when the metaheuristics are used to solve the DMbPE problems in the sequential direct method context. The dynamic system models were coded in the Matlab programming language and the computational application of the sequential direct method with the FA, HS, DE, ABC, and DTS codes was developed in Matlab programming environment. The computational tests were performed on a PC with a 2.2 GHz Core i72670QM and 8 GB of RAM. The parameter values for the tested algorithms are set as shown in Table
Parameter values.
FA  





0.25 

1 

1 

0.2 


HS  





0.95 


DE  





0.9 
Mutation  DE/randtobest/1 


ABC  




“limit”  100 


DTS  



1 

0.1 

2 
Nine case study problems have been selected to analyze the performance of the described stochastic algorithms. A comparison is made considering the quality of the solutions and the time spent to reach the solution after a threshold number of function evaluations. The problems and the experimental data can be found in the Appendix.
Table
Comparison of
Inst. 

MH 






St.D. 


St.D. 


St.D. 
 

5  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













3  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













4 
FA 









HS 










DE 










ABC 










DTS 













4  FA 









HS 










DE 










ABC 










DTS 













3  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 













2  FA 









HS 










DE 










ABC 










DTS 









To analyze the statistical significance of the results, we use the Matlab function
When applied to
Estimates of the 95% confidence intervals for
When the test is applied to
A similar analysis is made for
Estimates of the 95% confidence intervals for
When the statistical analysis is extended to the distributions of
Estimates of the 95% confidence intervals for
We now show in Table
Best results (
Inst.  MH 










 

FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 










FA 






HS 







DE 







ABC 







DTS 






Performance profiles for the five metaheuristics, using
The higher the percentage, the better. A higher value for
Therefore, the metaheuristic DE is the most successful since it has the highest probability of achieving the lowest
Similarly, Figures
Performance profiles for the five metaheuristics, using
Finally, Figures
Performance profiles for the five metaheuristics, using
Tables
Comparative results for
Source 








753  
[ 

– 

(9518 iter.) 
[ 



1163 
[ 


–  9148 
[ 

n.a. 

(2 iter.) 
Comparative results for
Source 








434  
[ 

n.a. 

(10000 iter.) 
Comparative results for
Source 








245  
[ 



(9028 iter.) 
Comparative results for
Source 








256  
[ 


–  – 
[ 



(26 iter.) 
[ 



– 
[ 

– 

– 
[ 



(67 iter.) 
[ 



(182 iter.) 
Comparative results for
Source 








449  
[ 

n.a.  –  – 
[ 

n.a. 

(38 iter.) 
[ 



– 
[ 

– 

– 
[ 

n.a. 

(37 iter.) 
[ 

n.a. 

(4 iter.) 
Comparative results for
Source 








447  
[ 


–  – 
Comparative results for
Source 








953  
[ 

n.a. 

(56 iter.) 
Comparative results for
Source 








374  
[ 

n.a. 

(349 iter.) 
[ 

n.a. 

(40552 iter.) 
Comparative results for
Source 








255  
[ 


–  – 
Comparative results for
Source 








448  
[ 


–  – 
Comparative results for
Source 








440  
[ 

n.a. 

(1000 iter.) 
[ 

n.a. 

(536 iter.) 
Overall, after all the numerical comparisons, it is possible to conclude that the selected metaheuristics when enhanced with the local intensification phase are effective in achieving good quality solutions with a reduced computational effort. Furthermore, we have also shown that the sequential direct method combined with the selected metaheuristics competes very favorably with exact methods and other metaheuristics available in the literature.
In this paper, we have analyzed the performance of five wellknown metaheuristics when solving parameter estimation problems in dynamic system models. The sequential numerical direct method is applied to the DMbPE problem and the resulting optimization is performed directly making use of a numerical integration formula for solving the system of ODE. Using nine DMbPE problems and different experimental data, errorfree and random erroradded data, a total of 12 instances have been used in the comparative experiments. The solutions produced by the metaheuristics have been analyzed in terms of quality, by stopping the algorithms after a specified number of function evaluations, and compared using statistical hypotheses testing. The statistical tests show that the average obtained solutions and the average CPU time are considered to be mostly significantly different, at a significance level of 5%. The best solutions produced by the metaheuristics are analyzed by means of the performance profiles. After the graphical comparisons for different numbers of allowed function evaluations, we are able to conclude that the DE is the metaheuristic that has the highest probability of giving the lowest value of the objective function
Experimental values for
1230.0  3060.0  4920.0  7800.0  10680.0  15030.0  22620.0  36420.0  


88.35  76.4  65.1  50.4  37.5  25.9  14.0  4.5 

7.3  15.6  23.1  32.9  42.7  49.1  57.4  63.1 

2.3  4.5  5.3  6.0  6.0  5.9  5.1  3.8 

0.4  0.7  1.1  1.5  1.9  2.2  2.6  2.9 

1.75  2.8  5.8  9.3  12.0  17.0  21.0  25.7 
Erroradded generated experimental values for 14 time instants, available in [
1  2  3  4  5  6  7  



1.4  6.3  10.4  14.2  17.6  21.4  23.0 


9  11  14  19  24  29  39  



27.0  30.5  34.4  38.8  41.6  43.5  45.3 
Erroradded experimental values for 20 time instants, available in [
0.025  0.050  0.075  0.100  0.125  0.150  0.175  



0.7307  0.5982  0.4678  0.4267  0.3436  0.3126  0.2808 

0.1954  0.2808  0.3175  0.3047  0.2991  0.2619  0.2391 


0.200  0.225  0.250  0.300  0.350  0.400  0.450  



0.2692  0.2210  0.2122  0.1903  0.1735  0.1615  0.1240 

0.2210  0.1898  0.1801  0.1503  0.1030  0.0964  0.0581 


0.500  0.550  0.650  0.750  0.850  0.950  



0.1190  0.1109  0.0890  0.0820  0.0745  0.0639  

0.0471  0.0413  0.0367  0.0219  0.0124  0.0089 
Generated experimental values for 10 time instants, available in [
0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1.0  


0.606  0.368  0.223  0.135  0.082  0.050  0.030  0.018  0.011  0.007 

0.373  0.564  0.647  0.669  0.656  0.624  0.583  0.539  0.494  0.451 
Generated experimental values for 20 time instants, available in [
0.5  1  1.5  2  2.5  3  3.5  4  4.5  5  



1.341  0.899  0.602  0.404  0.271  0.181  0.122  0.082  0.055  0.037 

0.609  0.933  1.077  1.110  1.079  1.011  0.925  0.833  0.742  0.655 


5.5  6  6.5  7  7.5  8  8.5  9  9.5  10  



0.025  0.016  0.011  0.007  0.005  0.003  0.002  0.001  0.001  0.001 

0.575  0.503  0.438  0.380  0.329  0.285  0.246  0.213  0.184  0.158 
Errorfree generated experimental values for 20 time instants, available in [
0.05  0.1  0.15  0.2  0.25  0.3  0.35  



0.8241  0.6852  0.5747  0.4867  0.4166  0.3608  0.3164 

0.0937  0.1345  0.1654  0.1899  0.2094  0.2249  0.2373 

0.0821  0.1802  0.2598  0.3233  0.3738  0.4141  0.4461 


0.4  0.45  0.5  0.55  0.6  0.65  0.7  



0.2810  0.2529  0.2304  0.2126  0.1984  0.1870  0.1780 

0.2472  0.2550  0.2613  0.2662  0.2702  0.2733  0.2759 

0.4717  0.4920  0.5082  0.5210  0.5313  0.5395  0.5460 


0.75  0.8  0.85  0.9  0.95  1.0  



0.1709  0.1651  0.1606  0.1570  0.1541  0.1518  

0.2779  0.2794  0.2807  0.2817  0.2825  0.2832  

0.5511  0.5553  0.5585  0.5612  0.5632  0.5649 
Erroradded experimental values for 20 time instants, available in [
0.05  0.1  0.15  0.2  0.25  0.3  0.35  



0.8261  0.6782  0.5721  0.4817  0.4226  0.3698  0.3114 

0.0917  0.1335  0.1644  0.1939  0.2111  0.2229  0.2313 

0.0826  0.1772  0.2628  0.3213  0.3598  0.4201  0.4511 


0.4  0.45  0.5  0.55  0.6  0.65  0.7  



0.2710  0.2499  0.2354  0.2216  0.1974  0.1890  0.1780 

0.2398  0.2510  0.2703  0.2602  0.2732  0.2733  0.2769 

0.4797  0.4990  0.5122  0.5200  0.5281  0.5305  0.5500 


0.75  0.8  0.85  0.9  0.95  1.0  



0.1729  0.1701  0.1606  0.1490  0.1531  0.1568  

0.2709  0.2754  0.2797  0.2817  0.2825  0.2792  

0.5601  0.5533  0.5485  0.5612  0.5632  0.5599 
Experimental data for 22 time instants and for
4.5  8.67  12.67  17.75  22.67  27.08  



0.0514  0.01422  0.01335  0.01232  0.01181  0.01139 


32.00  36.00  46.33  57.00  69.00  76.75  



0.01092  0.01054  0.00978  0.009157  0.008594  0.008395 


90.00  102.00  108.00  147.92  198.00  241.75  



0.007891  0.00751  0.00737  0.006646  0.005883  0.005322 


270.25  326.25  418.00  501.00  



0.00496  0.004518  0.004075  0.003372 
Experimental data available in [
1  2  3  4  5  6  7  



1.1529  0.9333  0.7806  0.6675  0.5801  0.5104  0.4535 

0.6747  0.4944  0.3828  0.3083  0.2558  0.2173  0.1881 


8  9  10  



0.4060  0.3658  0.3314  

0.1653  0.1473  0.1327 
Erroradded generated experimental values for 10 time instants.
1  2  3  4  5  6  7  



0.8170  0.8525  1.2451  1.0423  0.7868  0.9834  1.2771 

1.1142  0.8876  0.9619  1.1419  1.0144  0.8694  1.0247 


8  9  10  



0.8776  0.7760  1.1412  

1.1231  0.9538  0.8797 
Erroradded experimental values for 10 time instants, available in [
1  2  3  4  5  6  7  



0.7990  0.8731  1.2487  1.0362  0.7483  1.0024  1.2816 

1.0758  0.8711  0.9393  1.1468  1.0027  0.8577  1.0274 


8  9  10  



0.8944  0.7852  1.1527  

1.1369  0.9325  0.9074 
The authors declare that there are no conflicts of interest regarding the publication of this paper.
The authors would like to acknowledge the financial support of CIDEM, R&D Unit, funded by the Portuguese Foundation for the Development of Science and Technology (FCT), Ministry of Science, Technology and Higher Education, under the Project UID/EMS/0615/2016, and of COMPETE: POCI010145FEDER007043 and FCT within the Projects UID/CEC/00319/2013 and UID/MAT/00013/2013.