Multistrategy Learning of Self-Organizing Map (SOM) and Particle Swarm Optimization (PSO) is commonly implemented in clustering domain due to its capabilities in handling complex data characteristics. However, some of these multistrategy learning architectures have weaknesses such as slow convergence time always being trapped in the local minima. This paper proposes multistrategy learning of SOM lattice structure with Particle Swarm Optimisation which is called ESOMPSO for solving various classification problems. The enhancement of SOM lattice structure is implemented by introducing a new hexagon formulation for better mapping quality in data classification and labeling. The weights of the enhanced SOM are optimised using PSO to obtain better output quality. The proposed method has been tested on various standard datasets with substantial comparisons with existing SOM network and various distance measurement. The results show that our proposed method yields a promising result with better average accuracy and quantisation errors compared to the other methods as well as convincing significant test.

In classification process; normally, large classes of objects are separated into smaller classes. This approach can be very complicated due to the challenge in identifying the criteria especially for procedures involving complex data structures. In this scenario; practically, the Machine Learning (ML) techniques will be used and introduced by many researchers as alternative solutions to solve the above problems. Among the ML methods and tools, Artificial Neural Network (ANN), Fuzzy Set, Genetic Algorithm (GA), Swarm Intelligence (SI), and rough set are commonly used by researchers.

However, the most popular ML method widely used by the practitioners is ANN [

ANN and evolutionary computation methodologies have each been proven effective in solving certain classes of problems. For example, neural networks are very efficient at mapping input to output vectors and evolutionary algorithms are very useful at optimization. ANN weaknesses could be solved either by enhancing the structures of ANN itself or by hybridizing it with evolutionary optimisation [

The searching implementation with evolutionary method such as ANN learning may overcome the gradient-based handicaps. However, the convergence is in general much slower, since these are general purpose methods. Kennedy and Eberhart [

Early studies have shown that the multistrategy learning of PSO-SOM approach was first introduced by Shi and Eberhart [

Moreover, Chandramouli [

Sharma and Omlin [

Since 1993, Extension of SOM network topologies such as self-organization network has been implemented in many applications. Fritzke [

Hybridization of SOM and evolutionary method was proposed by Créput et al. [

The quality of the Kohonen map is determined by its lattice structure. This is because the weights for each neuron in the neighborhood will be updated by these lattice structures. There are many types of SOM lattice structures: circle lattice structure, rectangular, hexagonal, spherical (Figure

Spherical SOM [

Torus SOM.

Spherical and Torus SOM representing the plane lattice give a better view of the input data as well as provide closer links to edge nodes. They make the 2D visualisation of multivariate data possible using SOM’s code vectors as data source [

According to Middleton et al. [

Astel et al. [

Wu and Takatsuka [

Due to limitations of the previous studies focusing on the improvement of SOM lattice structure, this study enhanced SOM lattice structure with improved hexagonal lattice area. In SOM competitive learning process, wider lattice are needed for searching the winning nodes as well as for weights adjustment. This allows SOM to get a good set of weights for improving the quality of data classification and labeling. Particle Swarm Optimisation (PSO) is developed to optimize SOMs’ training weights accordingly. The hybridisation of SOM-PSO architecture, so-called Enhanced SOM with Particle Swarm Optimisation (ESOMPSO) is proposed with improvement on the lattice structure for better classification. The performance of the proposed ESOMPSO is validated based on the classification accuracy and quantization errors (QE). The error deviations between the proposed methods are computed to further illustrate the efficiency of these approaches accordingly.

In this study, we proposed multistrategy learning with the Enhancement of SOM with PSO (ESOMPSO) and improved formulation of hexagonal lattice structure. Unlike conventional hexagonal lattice (as given in (

The weights of all neurons within this hexagon are updated with

For each input vector

Initialisation—set initial synaptic weights to small random values, say in a interval

Competition—for each output node

Cooperation—identify all output nodes

Adaptation—adjust the weights:

Iteration—adjust the learning rate and neighborhood size, as needed until no changes occur in the feature map. Repeat step (ii) and stop when the termination criteria are met. The improved hexagonal lattice area consists of six important points: right_border

The proposed lattice structure for enhanced SOM (ESOM).

Subsequently, the weights of ESOMPSO learning are optimised by PSO. Particle Swarm Optimisation (PSO) is one of the Swarm Intelligence (SI) techniques that are inspired by social behavior of bird flocking and fish schooling. The pioneers of the PSO algorithm are Kennedy, Eberhart, and Shi in 1995 [

The rectangular topology and hexagonal lattice structure of the SOM is initialized with feature vectors

Input feature vector

Initialise the population array of particle representing random solutions for

For each particle, the distance function is evaluated,

The personal best

The global best

Update the velocity

Update the position

Repeat steps 2 to 9 until all input patterns are exhausted in the training.

To investigate the effectiveness of PSO in evolving the weights from SOM, the proposed method has been performed in the testing and validation process. In the testing phase, data is presented to the network with target nodes for each input sets. The reference attributes or classifier computed during training process is used to classify input data set. The algorithm identifies the winning node that will be used for determining the output of the network. Then, the output of the network is compared to the expected result to decide the ability of the network for classification phase. This classification stage will classify test data into correct predefined classes obtained during training process. A number of data is presented to the network, and the percentage of correct classified data is calculated. The percentage of the correctness is measured to obtain the accuracy and the learning ability of the network. The result is validated and compared using several performance measurements: quantisation error (QE) and classification accuracy. Later, the error differences between the proposed methods are computed for further validations.

The performance measurement of the proposed methods is based on quantisation error (QE) and classification accuracy (CA). QE is measured after SOM’s training, and CA is the analysis for testing. The efficiency of the proposed methods is validated accordingly; if QE values are smaller and the classification accuracy is higher, then the results are promising. QE is used for measuring the quality of SOM map. QE of an input vector is defined by the difference between the input vector and the closest codebook vectors. QE describes how accurately the neurons respond to the given dataset. For example, if the reference vector of the BMU calculated for a given testing vector

Quantization Error:

While the classification accuracy indicates how well the classes are separated on the map, the classification accuracy of new samples measures the networks generalisation for better quality of SOM’s mapping.

Classification accuracy,

The goal of the conducted experiments is to investigate the performance of the proposed methods. The comparisons are done on ESOMPSO, SOM with PSO (SOMPSO), and enhanced SOM (ESOM). The results are validated in terms of classification accuracy and quantisation error (QE) on standard universal machine learning datasets: Iris, XOR, Cancer, Glass, and Pendigits. From the conducted experiments, it shows that the proposed methods, ESOMPSO and SOMPSO, give better accuracy despites higher convergence time. As PSO and improved lattice structure are being implemented, the convergence time is increasing. This scenario is due to the PSO process in searching for the

Self-Organizing Maps (SOM) has two layers: input and output layers. The basic SOM architecture consists of a lattice that acts as an output layer with its input nodes fully connected. In this study, the network architecture is designed based on the selected real world classification problems. Table

Data information.

Data type | Iris | XOR | Cancer | Glass | Pendigits |
---|---|---|---|---|---|

Input node | 4 | 4 | 30 | 10 | 16 |

Output ode | 1 | 1 | 1 | 1 | 1 |

Data size | 150 | 8 | 569 | 214 | 10992 |

Training size | 120 | 6 | 379 | 149 | 494 |

Testing size | 30 | 2 | 190 | 65 | 498 |

The input layer is comprised of input pattern with different nodes that is randomly chosen from training data set. Input patterns are presented to all output nodes (neurons) in the network simultaneously. The number of input node determines the number of data required to be fed into the network, while the numbers of nodes in the Kohonen layer represent the maximum number of possible classes. Table

Class information.

Dataset | Number of class | Classes |
---|---|---|

Iris | 3 | Class 1: Iris Virginia |

Class 2: Iris Sentosa | ||

Class 3: Iris Versicolor | ||

XOR | 4 | Class 1: 0 |

Class 2: 1 | ||

Cancer | 2 | Class 1: Benign |

Class 2: Malignant | ||

Glass | 6 | Class 1: Building windows float |

Class 2: Processed building | ||

Class 3: Windows float | ||

Class 4: Processed building | ||

Class 5: Windows nonfloat | ||

Class 6: Processed containers | ||

Class 7: Tableware Headlamps | ||

Pendigits | 10 | Class 0: Digit 1 |

Class 1: Digit 2 | ||

Class 2: Digit 3 | ||

Class 3: Digit 4 | ||

Class 4: Digit 5 | ||

Class 5: Digit 6 | ||

Class 6: Digit 7 | ||

Class 7: Digit 8 | ||

Class 8: Digit 9 |

The training starts once the dataset has been initialised and input patterns have been selected. The learning phase of the SOM algorithm repeatedly presents numerous patterns to the network. The learning rule of the classifier allows these training cases to organize in a two-dimensional feature map. Patterns which resemble each other are mapped onto a specific cluster. During the training phase, the class for randomly selected input node is determined. This is done by labeling the output node that is more similar (best-matching unit) to the input node compared to other nodes in the Kohonen mapping structure. The outputs from the training are the resulting map that contains the winning neurons and its associated weight vectors. Subsequently, these weight vectors are optimised by PSO. The quality of the classification accuracy is calculated to investigate the behavior of the network in the training data.

In the testing phase, for any input patterns, if the

It is often reported in the literature that the success of the Self-Organizing Maps (SOM) formation is critically dependent on the initial weights and the selection of main parameters of the algorithm, namely, the learning rate parameter and the neighborhood set [

There is no guideline in suggesting good learning rates to any given learning problem. In standard SOM, too large and too small learning rates can lead to poor network performance [

The accuracy of the map also depends on the number of iterations of the SOM algorithm. A rule of thumb states, for good statistical accuracy, number of iterations should be at least 500 times the number of neurons. According to [

Parameter settings for ESOMPSO.

Parameter | Dataset | ||||

Iris | XOR | Cancer | Glass | Pendigits | |

Input vector (Training) | 120 | 6 | 379 | 149 | 7494 |

Input vector (Testing) | 30 | 2 | 190 | 65 | 3498 |

Input dimension | 4 | 4 | 30 | 9 | 16 |

SOM's Mapping Dimension | 10 × 10 | 10 × 10 | 10 × 10 | 10 × 10 | 10 × 10 |

SOM lattice structure | Standard | Standard | Standard | Standard | Standard |

ESOM lattice structure | Improved | Improved | Improved | Improved | Improved |

Learning rate | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |

Number of runs | 10 times | 10 times | 10 times | 10 times | 10 times |

Epoch | 1000 | 1000 | 1000 | 1000 | 1000 |

2.0 | 2.0 | 2.0 | 2.0 | 2.0 | |

2.0 | 2.0 | 2.0 | 2.0 | 2.0 | |

0.1 | 0.1 | 0.1 | 0.1 | 0.1 | |

Number of particles | 100 | 100 | 100 | 100 | 100 |

PSO problem dimension | 10 × 10 | 10 × 10 | 10 × 10 | 10 × 10 | 10 × 10 |

Stop condition (minimum error) | 0.0000193 | 0.0000193 | 0.0000193 | 0.0000193 | 0.0000193 |

The experiments were conducted with various datasets and distance measurements: Euclidean, Manhattan, and Chebyshev distance. The comparisons were conducted between standard SOM and standard SOM with improved hexagonal structure, so-called ESOM. Standard SOM was trained using standard hexagonal lattice, while ESOM with improved hexagonal lattice. The choice of distance measure influences the accuracy, efficiency, and generalisation ability of the results. From Table

Summarization of SOM and ESOM results.

SOM | ESOM | ||||||

EUC | MAN | CHEBY | EUC | MAN | CHEBY | ||

IRIS | Quantization error | 0.0358 | 0.0419 | 0.0244 | 0.0275 | ||

60.0000 | 70.0000 | 73.3333 | 74.333 | ||||

XOR | Quantization error | 0.2060 | 0.2159 | 0.2458 | 0.2077 | ||

68.4525 | 72.5632 | 80.4462 | 84.2561 | ||||

CANCER | Quantization error | 0.4913 | 0.5037 | 0.4771 | 0.4781 | ||

37.8947 | 43.1579 | 34.7368 | 71.5789 | ||||

GLASS | Quantization error | 0.0307 | 0.0350 | 0.0122 | 0.0117 | ||

13.8462 | 36.9231 | 50.7692 | 44.6154 | ||||

PENDIGITS | Quantization error | 0.2006 | 0.2103 | 0.1957 | 1.2008 | ||

44.5969 | 72.6415 | 52.9445 | 69.1252 |

EUC: Euclidean distance, MAN: Manhattan distance, CHEBY: Chebyshev distance.

Similar experiments were conducted for standard SOM with PSO, so-called SOMPSO and ESOMPSO with Euclidean, Manhattan's, and Chebyshev’s distance measurements. SOMPSO was trained using standard hexagonal lattice, while ESOMPSO was trained with improved hexagonal lattice. The results were compared in terms of classification accuracy, quantisation error, and convergence error. As illustrated in Table

Summarisation of SOMPSO and ESOMPSO results.

SOMPSO | ESOMPSO | ||||||

EUC | MAN | CHEBY | EUC | MAN | CHEBY | ||

Iris | Epoch | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |

Quantisation error | 4.0979 | 4.0875 | 2.0125 | 2.0565 | |||

Convergence error | 0.0318 | 0.0358 | 0.0322 | 0.0243 | 0.0587 | 0.0347 | |

Convergence time | 22 sec | 22 sec | 22 sec | 240 sec | 240 sec | 240 sec | |

89.24 | 90.45 | 90.11 | 90.75 | ||||

XOR | Epoch | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |

Quantisation error | 0.6455 | 0.5866 | 0.0250 | 0.0145 | |||

Convergence error | 0.2500 | 0.3204 | 0.3050 | 0.1916 | 0.2591 | 0.2641 | |

Convergence time | 10 sec | 10 sec | 10 sec | 17 sec | 17 sec | 17 sec | |

85.25 | 88.47 | 86.14 | 90.24 | ||||

Cancer | Epoch | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |

Quantisation error | 0.0145 | 0.0102 | 0.0125 | 0.0078 | |||

Convergence error | 0.5951 | 0.6523 | 0.6424 | 0.4422 | 0.5371 | 0.4823 | |

Convergence time | 80 sec | 80 sec | 80 sec | 110 sec | 110 sec | 110 sec | |

75.23 | 78.89 | 77.35 | 82.05 | ||||

Glass | Epoch | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |

Quantisation error | 0.0087 | 0.0052 | 0.0060 | 0.0048 | |||

Convergence error | 0.0435 | 0.0541 | 0.1242 | 0.0157 | 0.0324 | 0.0224 | |

Convergence time | 40 sec | 40 sec | 40 sec | 60 sec | 60 sec | 60 sec | |

80.98 | 84.66 | 82.45 | 84.87 | ||||

Pendigits | Epoch | 1000 | 1000 | 1000 | 1000 | 1000 | 1000 |

Quantization error | 0.4752 | 0.4777 | 0.5143 | 0.4221 | |||

Convergence error | 0.2060 | 0.2365 | 0.2241 | 0.1405 | 0.1569 | 0.1478 | |

Convergence time | 110 sec | 110 sec | 110 sec | 205 sec | 205 sec | 205 sec | |

70.25 | 72.48 | 70.85 | 72.89 |

EUC: Euclidean distance, MAN: Manhattan distance, CHEBY: Chebyshev distance.

Figures

Accuracy SOM, ESOM, SOMPSO, and ESOMPSO.

Quantization errors of SOM, ESOM, SOMPSO, and ESOMPSO.

However, this tradeoff, that is, higher accuracy with more convergence time and vice versa, does not give big impact on the success of the proposed methods due to the concept of

From the findings, it seems that the selection of SOM’s lattice structure for better learning is crucial in updating the neighbourhood structures for network learning. The standard formulation for basic and improved hexagonal lattice structure is illustrated in Figure

Improved hexagonal lattice structure.

By using improved hexagonal lattice area, the nodes will be updated to 33. The coverage area is better compared to the basic hexagonal lattice, and the potential nodes differences are 22.61. This formulation improves the neighborhood updating process; hence, better results of ESOMPSO are quite promising. The proposed methods are validated using the Kruskal-Wallis [

Kruskal-Willis ranks for the proposed methods.

Methods | Number of datasets | Mean rank based on accuracy (Euclidean distance) | Mean rank based on accuracy (the Manhattan distance) | Mean rank based on accuracy (the Chebyshev distance) |
---|---|---|---|---|

SOM | 5 | 3.60 | 4.60 | 6.00 |

ESOM | 5 | 8.40 | 7.60 | 7.00 |

SOMPSO | 5 | 14.20 | 14.40 | 13.80 |

ESOMPSO | 5 | 15.80 | 15.40 | 15.20 |

Total | 20 | |||

0.004 | 0.008 | 0.025 |

This paper presents multistrategy learning by proposing Enhanced Self-Organizing Map with Particle Swarm Optimization (ESOMPSO) for classification problems. The proposed method was successfully implemented on machine learning datasets: XOR, Cancer, Glass, Pendigits, and Iris. The analysis was done by comparing the results for each dataset produced by Self-Organising Map (SOM), Enhanced Self-Organising Map (ESOM), Self-Organizing Map with Particle Swarm Optimization (SOMPSO) and ESOMPSO with different distance measurements. The analysis reveals that ESOMPSO with Euclidean distance generate promising results based on the highest accuracy and the least quantization errors (referring to Figures

Authors would like to thank the Research Management Centre (RMC), Universiti Teknologi Malaysia, and the