^{1}

^{1}

Multilayer feed-forward artificial neural networks are one of the most frequently used data mining methods for classification, recognition, and prediction problems. The classification accuracy of a multilayer feed-forward artificial neural networks is proportional to training. A well-trained multilayer feed-forward artificial neural networks can predict the class value of an unseen sample correctly if provided with the optimum weights. Determining the optimum weights is a nonlinear continuous optimization problem that can be solved with metaheuristic algorithms. In this paper, we propose a novel multimean particle swarm optimization algorithm for multilayer feed-forward artificial neural networks training. The proposed multimean particle swarm optimization algorithm searches the solution space more efficiently with multiple swarms and finds better solutions than particle swarm optimization. To evaluate the performance of the proposed multimean particle swarm optimization algorithm, experiments are conducted on ten benchmark datasets from the UCI repository and the obtained results are compared to the results of particle swarm optimization and other previous research in the literature. The analysis of the results demonstrated that the proposed multimean particle swarm optimization algorithm performed well and it can be adopted as a novel algorithm for multilayer feed-forward artificial neural networks training.

Artificial neural networks (ANNs) are a vital component of artificial intelligence. Machine learning and cognitive sciences depend on ANNs to solve various complex nonlinear mapping relationships [

In this study, we propose a multimean particle swarm optimization (MMPSO) algorithm that makes novel use of PSO for solving continuous optimization problems. To assess the performance of the proposed MMPSO algorithm, it is applied on multilayer feed-forward artificial neural networks (MLFNNs) training and compared with PSO and other algorithms used in previous research. Analysis of the experimental results shows that the proposed MMPSO algorithm improved the classification accuracy of MLFNNs and showed a better performance than other algorithms.

The paper is organized as follows: in Section

MLFNNs can be defined as a system by modeling the human brain functions. MLFNNs consist of artificial neural cells linked to each other in various forms and are usually organized in layers. They can be implemented as hardware in electronic circuits or as software in computers. In accordance with the brain information processing method, MLFNN has the ability to store and generalize information after a learning process [

Structure of MLFNNs.

When we look at Figure

Metaheuristic algorithms are generally used in many areas for solving different problems such as optimization, scheduling, training of ANNs, fuzzy logic systems, and image processing [

PSO is a population-based metaheuristic optimization technique developed by Dr. Eberhart and Dr. Kennedy in 1995 [

Multiswarm optimization (MSO) is a technique that is used to predict the optimal solution to nonlinear continuous optimization problems. The effectiveness and the productivity of many metaheuristic algorithms worsen as the dimensionality of the problem increases [_{1 }and_{2} are two positive constants representing acceleration factor, and_{1} and_{2} are two random numbers in the range

Determining the optimal weights of MLFNNs is a nonlinear optimization problem so metaheuristic algorithms can be used for MLFNNs training. An application of metaheuristic algorithms to MLFNNs training is explained step by step in the following text.

The normalization process, which is a data preprocessing technique, is applied to the dataset to be classified. Thus, the dataset becomes more regular and suitable for MLFNNs. This normalization process is done using a min–max normalization function, which is shown in [

In this study, the datasets are organized in two different ways for two different experiments. In the first experiment, 5-fold cross validation is used for comparing the proposed MMPSO algorithm to the PSO algorithm. In the second experiment, 80% training and 20% testing are used for comparing the proposed MMPSO algorithm to previous research in the literature.

The numbers of inputs and the number of outputs are determined according to the characteristics of the dataset. The number of inputs is equal to the number of attributes of the dataset. Similarly, the number of outputs is equal to the number of classes of the dataset. The number of hidden layers is set to one for all problems and the number of nodes in the hidden layer is determined with GA, which is reported in a previous study by the authors [

A well-trained MLFNNs should have optimum weights and determining the optimum weights is a nonlinear optimization problem. Metaheuristic algorithms can be used to solve this problem owing to the structure of the metaheuristic algorithm. Generally, metaheuristic algorithms initialize with a random population. The fitness of each individual in the population is calculated according to the SSE of MLFNNs. The goal of the metaheuristic algorithms is to minimize the SSE. Therefore, metaheuristic algorithms search the problem space locally and globally and update the global best solution. The metaheuristic algorithms run until the stopping criteria, such as the number of iterations or the error rate, are met.

In order to determine the performance of the MLFNNs training with metaheuristic algorithms, the classification accuracy is calculated according to (

The application of the proposed MMPSO algorithm and the PSO algorithm to the MLFNNs is implemented using C# Microsoft Visual Studio Ultimate 2013. All experiments are carried out using a computer with an Intel Core i7 3840QM@2.00 GHz processor with 8 GB of memory with Microsoft Windows 8 operating system. Ten different benchmark datasets from the UCI repository [

The characteristics of the ten datasets used.

| | |||
---|---|---|---|---|

| | | | |

Lymphography | 18 | 0 | 4 | 148 |

Iris | 0 | 4 | 3 | 150 |

Wine | 0 | 13 | 3 | 178 |

Glass | 0 | 9 | 6 | 214 |

Shuttle-landing | 6 | 0 | 2 | 253 |

Ionosphere | 0 | 33 | 2 | 351 |

Balance-scale | 4 | 0 | 3 | 625 |

Breast cancer | 0 | 9 | 2 | 699 |

Diabetes | 0 | 8 | 2 | 768 |

Thyroid | 0 | 21 | 3 | 7200 |

In general, the structure of MLFNNs is represented by

Furthermore, to determine the optimum structure of the MLFNNs is an optimization problem and the classification accuracy of the MLFNNs is directly affected by it (Ibrahim, Jihad, and Kamal, 2017). The determined structures of the MLFNNs according to the ten benchmark datasets which are shown in Table

The structure of the MLFNNs.

| | | | |
---|---|---|---|---|

Lymphography | 18 | 15 | 4 | 349 |

Iris | 4 | 5 | 3 | 43 |

Wine | 13 | 10 | 3 | 173 |

Glass | 9 | 12 | 6 | 198 |

Shuttle-landing | 6 | 8 | 2 | 74 |

Ionosphere | 33 | 4 | 2 | 146 |

Balance-scale | 4 | 5 | 3 | 43 |

Breast cancer | 9 | 8 | 2 | 98 |

Diabetes | 8 | 6 | 2 | 68 |

Thyroid | 21 | 12 | 3 | 303 |

For the proposed MMPSO algorithm and the PSO algorithm, the acceleration constants

The results of 5-fold cross validation experiment of the proposed MMPSO and PSO.

| | | | |
---|---|---|---|---|

Lymphography | PSO | 23.46 | 89.32 | 78.67 |

MMPSO | | | | |

| ||||

Iris | PSO | 6.72 | 91.00 | 90.67 |

MMPSO | | | | |

| ||||

Wine | PSO | 3.86 | 92.25 | 87.22 |

MMPSO | | | | |

| ||||

Glass | PSO | 83.60 | 66.35 | 55.84 |

MMPSO | | | | |

| ||||

Shuttle-landing | PSO | 2.07 | | 94.12 |

MMPSO | | | | |

| ||||

Ionosphere | PSO | 24.77 | 93.07 | 85.35 |

MMPSO | | | | |

| ||||

Balance-scale | PSO | 58.21 | 90.20 | 86.56 |

MMPSO | | | | |

| ||||

Breast cancer | PSO | 73.80 | 98.38 | 95.14 |

MMPSO | | | | |

| ||||

Diabetes | PSO | 202.39 | 76.78 | 72.08 |

MMPSO | | | | |

| ||||

Thyroid | PSO | 528.55 | 92.95 | 92.83 |

MMPSO | | | |

When the results of the 5-fold cross validation experiment in Table

Additional advantages of the proposed MMPSO algorithm are that it searches the global space more efficiently and convergences the optimum results more rapidly. To provide these advantages, the proposed MMPSO algorithm must minimize the fitness function SSE more rapidly than the PSO algorithm in the training process. The minimization of SSEs according to the iteration number in the training process is given in Figure

The minimization of the SSE according to the iteration in the training.

Furthermore, for analyzing the computational complexities of the proposed MMPSO and the PSO algorithms, the CPU running times of each algorithm were measured by Microsoft Process Explorer utility in seconds and are given in Table

CPU running times of the PSO and MMPSO algorithms in seconds.

| | |
---|---|---|

Lymphography | 6,48 | |

Iris | 2,16 | |

Wine | 4,37 | |

Glass | 7,59 | |

Shuttle-landing | 4,34 | |

Ionosphere | 6,52 | |

Balance-scale | 8,03 | |

Breast cancer | 14,44 | |

Diabetes | 11,29 | |

Thyroid | 479,08 | |

As shown in Table

Finally, the performance of the proposed MMPSO algorithm is compared with the performance reported in the literature for the HSA [

The results of the 80% training and 20% testing experiment for six datasets.

| | | | |
---|---|---|---|---|

Iris | MMPSO | | | |

FWA | 0.52 | | | |

KHA | 21.28 | 99.59 | | |

HS | 18.00 | 98.33 | 96.67 | |

GA | 96.00 | 90.00 | 90.00 | |

| ||||

Glass | MMPSO | 47.56 | | 62.79 |

FWA | 94.33 | 61.99 | 60.47 | |

KHA | | 58.79 | 58.14 | |

HS | 355.85 | 70.12 | | |

GA | 544.00 | 57.89 | 67.44 | |

| ||||

Ionosphere | MMPSO | | | 91.54 |

FWA | 25.28 | 95.71 | 90.14 | |

KHA | 31.0 | 89.00 | 91.43 | |

HS | 106.4 | 95.00 | | |

GA | 152 | 93.21 | | |

| ||||

Breast cancer | MMPSO | | | 97.85 |

FWA | 66.11 | 93.92 | 96.43 | |

KHA | - | - | - | |

HS | 126.37 | - | | |

GA | 172 | - | 98.57 | |

| ||||

Diabetes | MMPSO | | | |

FWA | 267.20 | 65.96 | 66.88 | |

KHA | - | - | - | |

HS | 856 | - | 77.27 | |

GA | 1108 | - | | |

| ||||

Thyroid | MMPSO | | | |

FWA | 749.11 | 93.21 | 93.82 | |

KHA | 320.3 | 94.81 | 92.90 | |

HS | 3146.4 | 93.06 | 92.78 | |

GA | 3416.0 | 92.58 | 92.57 |

When the results of the 80% training and 20% testing experiment in Table

In this paper, a novel MMPSO algorithm is proposed for MLFNNs training. The proposed MMPSO algorithm based on MSO technique has two advantages according to the PSO algorithm. Firstly, the proposed MMPSO algorithm strengthens the particles to carry out a local search in the search space range. Secondly, the proposed MMPSO algorithm has multiple swarms and takes into account both the best solution of each swarm and the best solution of all swarms and thus it gets closer to the optimum solution. To evaluate the performance of the proposed MMPSO algorithm experiments were conducted on ten benchmark datasets from the UCI repository. According to the experimental results, the proposed MMPSO algorithm yielded better performance than PSO for all datasets. Furthermore, the obtained experimental results were compared with the previous researches in the literature for six datasets. According to this comparison, the proposed MMPSO algorithm showed a competitive advantage over the reported algorithms. In conclusion, the proposed MMPSO algorithm showed good performance and can be adopted as a novel algorithm for MLFNNs training.

For future work, the proposed MMPSO algorithm will be used by intelligent systems to solve complex real-life optimization problems in various fields such as: design, identification, operational development, planning, and scheduling.

The data used to support the findings of this study are available from the corresponding author upon request.

The authors declare that there are no conflicts of interest regarding the publication of this paper.

This paper is supported by BAP Coordination Office of Necmettin Erbakan University.