A hybrid learning scheme (ePSO-BP) to train Chebyshev Functional Link Neural Network (CFLNN) for classification is presented. The proposed method is referred as hybrid CFLNN (HCFLNN). The HCFLNN is a type of feed-forward neural networks which have the ability to transform the nonlinear input space into higher dimensional-space where linear separability is possible. Moreover, the proposed HCFLNN combines the best attribute of particle swarm optimization (PSO), back propagation learning (BP learning), and functional link neural networks (FLNNs). The proposed method eliminates the need of hidden layer by expanding the input patterns using Chebyshev orthogonal polynomials. We have shown its effectiveness of classifying the unknown pattern using the publicly available datasets obtained from UCI repository. The computational results are then compared with functional link neural network (FLNN) with a generic basis functions, PSO-based FLNN, and EFLN. From the comparative study, we observed that the performance of the HCFLNN outperforms FLNN, PSO-based FLNN, and EFLN in terms of classification accuracy.

In recent years, higher-order neural networks [

An easy way to avoid these problems consists in removing the hidden layers. This may
sound a little inconsiderate at first, since it is due to them that nonlinear
input-output relationships can be captured. Encouragingly enough, the removing
procedure can be executed without giving up nonlinearity, provided that the input
layer is endowed with additional higher-order units [

Genetic algorithms and particle swarm optimization can be used for training the FLNN
to reduce the local optimality and speed up the convergence. But training using
genetic algorithm is discouraging because of the following limitations: in the
training process, it requires encoding and decoding operator which is commonly
treated as a long-standing barrier of neural networks researchers. The important
problem of applying genetic algorithms to train neural networks may be
unsatisfactory because recombination operators incur several problems, such as
competing conventions [

Unlike the GA, the PSO algorithm has no complicated operators such as cross-over and
mutation. In the PSO algorithm, the potential solutions, called as particles, are
obtained by flowing through the problem space by following the current optimum
particles. Generally speaking, the PSO algorithm has a strong ability to find the
most optimistic result, but it has a disadvantage of easily getting into a local
optimum. After suitably modulating the parameters for the PSO algorithm, the rate of
convergence can be speeded up, and the ability to find the global optimistic result
can be enhanced. The PSO algorithm search is based on the orientation by tracing

The remainder of this paper is organized as follows. Some the recently proposed
functional link neural networks (FLNNs) are reviewed in Section

FLNNs are higher order neural networks without hidden units introduced by
Klassen et al. [

In contrast to the linear weights of the input patterns produced by the linear links
of artificial neural network, the functional link acts on an element of a pattern or
on the entire pattern itself by generating a set of linearly independent functions,
then evaluating these functions with the pattern as an argument. Thus, class
separability is possible in the enhanced feature space. For a

However, the dimensionality of many problems is itself very high and further
increasing the dimensionality to a very large extent that may not be an appropriate
choice. So, it is advisable and also a new research direction to choose a small set
of alternative functions, which can map the function to the desired extent with an
output of significant improvement. FLNN with a trigonometric basis functions for
classification, as proposed in [

Haring and Kok [

Dash et al. [

With the encouraging performance of FLNN [

In [

A Chebyshev functional link artificial neural networks have been proposed by
Patra and Kot [

In [

Majhi and Shalabi [

Misra and Dehuri [

Two simple modified FLANNs are proposed by Krishnaiah et al. [

With this discussion, we can conclude that a very few applications of HONNs have so far been made in classification task. Although theoretically this area is rich, but application specifically in classification is poor. Therefore, the proposed contribution can be another improvement in this direction.

It is well known that the nonlinearly approximation of the Chebyshev orthogonal polynomial is very powerful by the approximation theory. Combining the characteristics of the FLNN and Chebyshev orthogonal polynomial the Chebyshev functional link neural network what we named as CFLNN is resulted. The proposed method utilizes the FLNN input-output pattern, the nonlinearly approximation capabilities of Chebyshev orthogonal polynomial, and the evolvable particle swarm optimization(ePSO)-BP learning scheme for classification.

The Chebyshev FLNN used in this paper is a single-layer neural network. The architecture consists of two parts, namely transformation part (i.e., from a low-dimensional feature space to high-dimensional feature space) and learning part. The transformation deals with the input feature vector to the hidden layer by approximate transformable method. The transformation is the functional expansion (FE) of the input pattern comprising of a finite set of Chebyshev polynomial. As a result, the Chebyshev polynomial basis can be viewed as a new input vector. The learning part uses the newly proposed ePSO-BP learning.

Alternatively, we can approximate a function by a polynomial of truncated power
series. The power series expansion represents the function with a very small
error near the point of expansion, but the error increases rapidly as we employ
it at points farther away. The computational economy to be gained by Chebyshev
series increases when the power series is slowly convergent. Therefore,
Chebyshev series are frequently used for approximations to functions and are
much more efficient than other power series of the same degree. Among orthogonal
polynomials, the Chebyshev polynomials converge rapidly than expansion in other
set of polynomials [

Evolvable particle swarm optimization (ePSO) is an improvement over the PSO
[

Initially, the iteration counter

We can say a particle in

At each iteration

Specifically, the set

The parameters

This process of updating the velocities

We now briefly present a number of improved versions of PSO and then show where our modified PSO can stand.

Shi and Eberhart [

Again Shi and Eberhart [

Fourie and Groenwold [

Clerc and Kennedy [

Da and Ge [

In this work, the inertial weight is evolved as a part of searching the optimal
sets of weights. However, the evolution of inertial weight is restricted between
an upper limit

In addition, the proposed method also uses the adaptive cognitive acceleration
coefficient

The ePSO-BP is an learning algorithm which combines the ePSO global searching
capability with the BP algorithm local searching capability. Similar to the GA
[

Initialize the positions and velocities of a group of particles randomly
in the range of

Evaluate each initialized particle’s fitness value, and

If the maximal iterative generations are arrived, go to Step 10, else, go to Step 4.

The best particle of the current particles is stored. The positions and
velocities of all the particles are updated according to (

Adjust the value of

Adjust the inertia weights

Evaluate each new particle’s fitness value, and the worst
particle is replaced with the stored best particle. If the

If the current

Use the BP algorithm to search around

Output the global optimum

The parameter

Learning of a CFLNN may be considered as approximating or interpolating a
continuous multivariate function

Let

As the dimension of the input pattern is increased from

Input the set of given

Choose the set of orthonormal basis functions.

For

Expand the feature values using the chosen basis functions.

Calculated the weighted sum and then fed to the output node.

End for

If the error is tolerable then stop otherwise go to (9).

Update the weights using ePSO BP learning rules and go to step (3).

This section is divided into five subsections. Section

The availability of results, with previous evolutionary and constructive
algorithms (e.g., Sierra et al. [

Summary of the datasets.

Dataset | Patterns | Attrib. | Clas. | Patterns in class 1 | Patterns in class 2 | Patterns in class 3 |
---|---|---|---|---|---|---|

IRIS | 150 | 4 | 3 | 50 | 50 | 50 |

WINE | 178 | 13 | 3 | 71 | 59 | 48 |

PIMA | 768 | 8 | 2 | 500 | 268 | — |

BUPA | 345 | 6 | 2 | 145 | 200 | — |

HEART | 270 | 13 | 2 | 150 | 120 | — |

CANCER | 699 | 9 | 2 | 458 | 241 | — |

All the algorithms have some parameters that have to be provided by the user. The
parameters for the proposed hybrid CFLNN are listed in Table

Description of the parameters.

Symbol | Purpose of the symbol |
---|---|

Size of the swarm | |

Inertia weight | |

Upper limit of the inertia | |

Lower limit of the inertia | |

Cognitive parameter | |

Left boundary value of cognitive parameter | |

Right boundary value of cognitive parameter | |

Social parameter | |

Left boundary value of social parameter | |

Right boundary value of social parameter | |

Maximum iterations for stopping an algorithm |

The values of the parameters used in this paper are as follows. We set

In the case of BP learning, the learning parameter

In this subsection, we will compare the results of hybrid CFLNN with the results
of EFLN with polynomial basis functions of degree 1, 2, and 3. The choice of the
polynomial degree is obviously a key question in FLNN with polynomial basis
functions. However, Sierra et al. [

Possible number of expanded inputs of degrees ONE, TWO, and THREE.

Dataset | Attributes | Degree 1 | Degree 2 | Degree 3 |
---|---|---|---|---|

IRIS | 4 | 5 | 15 | 35 |

WINE | 13 | 14 | 105 | 560 |

PIMA | 8 | 9 | 45 | 165 |

BUPA | 6 | 7 | 28 | 84 |

HEART | 13 | 14 | 105 | 560 |

CANCER | 9 | 10 | 55 | 220 |

For the sake of convenience, we report the results of the experiments conducted
on CANCER and BUPA and then compared with the methods EFLN [

Comparative results of HCFLNN with EFLN for the cancer and PIMA dataset by considering the average training error (MTre), average validation error (MVe), and average test error (MTe).

Dataset | HCFLNN | EFLN | ||||

MTre | MVe | MTe | MTre | MVe | MTe | |

Cancer 1 | 4.01 | 2.76 | 2.57 | 4.27 | 1.89 | 2.09 |

Cancer 2 | 3.95 | 3.97 | 4.66 | 4.37 | 2.96 | 3.96 |

cancer 3 | 4.13 | 3.51 | 4.43 | 3.29 | 3.01 | 4.65 |

BUPA 1 | 16.26 | 21.98 | 22.62 | 19.07 | 22.44 | 23.29 |

BUPA 2 | 17.90 | 24.12 | 22.35 | 19.84 | 18.63 | 20.37 |

BUPA 3 | 15.34 | 19.92 | 21.96 | 16.68 | 17.81 | 24.44 |

Here, we will discuss the comparative performance of hybrid CFLNN with FLNN using three datasets IRIS, WINE, and PIMA. In this case, the total set of samples are randomly divided into two equal folds. Each of these two folds are alternatively used either as a training set or as a test set. As the proposed learning method ePSO BP learning is a stochastic algorithm, so 10 independent runs were performed for every single fold. The training results obtained in the case of HCFLNN, averaged over 10 runs, are compared with the single run of FLNN. Similarly, the performance of both classifiers in test set is illustrated herein.

The plotted results clearly indicate that the performance of HCFLNN is competitive with FLNN, whereas in other classification problems like WINE and PIMA, the HCFLNN is showing a clear boundary.

The comparative performance of HCFLNN with FLNN [

Comparative average performance of HCFLNN and FLNN [

Dataset | HCFLNN | FLNN |
---|---|---|

IRIS train | ||

Test set | ||

WINE train | ||

Test set | ||

PIMA train | ||

Test set |

Comparative Average Performance of HCFLNN and FLNN [

Dataset | HCFLNN | FLNN |
---|---|---|

IRIS Train | ||

Test set | ||

WINE Train | ||

Test set | ||

PIMA Train | ||

Test set |

In this subsection, we will explicitly examine the performance of the HCFLNN
model by considering the heart dataset with the use of the 9-fold cross
validation methodology. The reason for using 9-fold cross validation is that to
compare the performance with the performance of few of the representative
algorithms considered in StatLog Project [

The procedure makes use of a weight matrix, which is described in Table

Weight Matrix of classes to Penalize.

Real Classification | Model Classification | |

Class 1 | Class 2 | |

Class 1 | 0 | |

Class 2 | 0 |

The purpose of such a matrix is to penalize wrongly classified samples based on
the weight of the penalty of the class. In general, the weight of the penalty
for class 2 samples that are classified as class 1 samples is

Table

Heart disease classification performance of FLANN models.

Data subset | Error in training set | Error in test set | ||||

Class 1 | Class 2 | Class 1 | Class 2 | |||

Heart1 | 13/133 | 14/107 | 1/17 | 1/13 | 0.35 | 0.2 |

Heart2 | 14/133 | 12/107 | 2/17 | 1/13 | 0.31 | 0.23 |

Heart3 | 13/134 | 15/106 | 4/16 | 2/14 | 0.37 | 0.47 |

Heart4 | 13/133 | 10/107 | 1/17 | 4/13 | 0.26 | 0.7 |

Heart5 | 13/133 | 16/107 | 3/17 | 2/13 | 0.39 | 0.43 |

Heart6 | 13/134 | 14/106 | 6/16 | 0/14 | 0.35 | 0.2 |

Heart7 | 15/133 | 13/107 | 0/17 | 3/13 | 0.33 | 0.5 |

Heart8 | 18/133 | 17/107 | 1/17 | 0/13 | 0.43 | 0.03 |

Heart9 | 20/134 | 9/106 | 2/16 | 1/14 | 0.27 | 0.23 |

Mean | 0.34 | 0.33 |

Table

Heart disease classification performance of HCFLANN models.

Data subset | Error in training set | Error in test set | ||||

Class 1 | Class 2 | Class 1 | Class 2 | |||

Heart1 | 13/133 | 14/107 | 1/17 | 1/13 | 0.35 | 0.2 |

Heart2 | 13/133 | 12/107 | 1/17 | 2/13 | 0.30 | 0.36 |

Heart3 | 12/134 | 13/106 | 5/16 | 1/14 | 0.32 | 0.33 |

Heart4 | 13/133 | 10/107 | 4/17 | 1/13 | 0.26 | 0.30 |

Heart5 | 13/133 | 15/107 | 3/17 | 2/13 | 0.37 | 0.43 |

Heart6 | 13/134 | 12/106 | 5/16 | 1/14 | 0.30 | 0.30 |

Heart7 | 14/133 | 13/107 | 1/17 | 2/13 | 0.33 | 0.37 |

Heart8 | 16/133 | 16/107 | 0/17 | 2/13 | 0.40 | 0.33 |

Heart9 | 18/134 | 10/106 | 2/16 | 1/14 | 0.28 | 0.23 |

Mean | 0.32 | 0.31 |

The classification results found by the HCFLNN for the heart disease dataset were
compared with the results found in the StatLog project [

Comparative classification performance of HCFLNN, FLNN with the
algorithms considered in [

Methods | ||
---|---|---|

HCFLNN | 0.31 | 0.32 |

FLNN | 0.33 | 0.34 |

0.37 | 0.59 | |

Bayes | 0.37 | 0.35 |

In this paper, we developed a new hybrid Chebyshev functional link neural network (HCFLNN). The hybrid model is constructed using the newly proposed ePSO- back propagation learning algorithm and functional link artificial neural network with the orthogonal Chebyshev polynomials. The model was designed for the task of classification in data mining. The method was experimentally tested on various benchmark datasets obtained from publicly available UCI repository. The performance of the proposed method demonstrated that the classification task is quite well in WINE and PIMA whereas showing a competitive performance with FLNN in IRIS. Further, we compared this model with EFLN and FLNN, respectively. The comparative results of the developed model is showing a clear edge over FLNN. Compared with EFLN, the proposed method has been shown to yield state-of-the-art recognition error rate for the classification problems such as CANCER and BUPA.

With this encouraging results of HCFLNN, our future research includes: (i) testing the proposed method on a more number of real life bench mark classification problems with highly nonlinearly boundaries, (ii) mapping the input features with other polynomials such as Legendre, Gaussian, Sigmoid, power series, and so forth, for better approximation of the decision boundaries, (iii) the stability and convergence analysis of the proposed method, and (iv) the evolution of optimal FLNN using particle swarm optimization.

The HCFLNN architecture, because of its simple architecture and computational
efficiency, may be conveniently employed in other tasks of data mining and knowledge
discovery in databases [