A Hybrid Approach for Modular Neural Network Design Using Intercriteria Analysis and Intuitionistic Fuzzy Logic

,


Introduction
One of the open and important questions in biology is the ability of biological systems to adapt to new environments, a concept termed evolvability [1].A typical feature of evolvability is the fact that many biological systems have modularity; especially many biological processes and structures can be modeled as networks, such as metabolic pathways, gene regulation, protein interactions, and brains [1][2][3][4][5].This feature has motivated important concepts in intelligent systems, such as modular neural network and evolutionary computation.
Neural networks are considered modular if they are comprised of highly connected clusters of nodes that are connected to nodes in other clusters [4,6,7].Despite importance and continuous research in this area, there is no agreement on why modular biological systems can evolve [4,8,9].There is evidence that modular systems look more adaptable in nature [10] than the monolithic networks [11,12].Consequently, there are many papers dedicated to this problem, for example, the work in [12].
In this paper, we introduce a hybrid combination between the intercriteria analysis (ICA, see [13][14][15][16][17][18]) method and modular neural network models.The ICA employs the apparatus of the intuitionistic fuzzy sets (IFS) for detecting possible correlations between pairs of criteria.Introduced in [19], IFSs are one of the extensions of Zadeh's fuzzy sets [20].In contrast to fuzzy sets, IFS [21][22][23][24] have two degrees: of membership (validity, etc.,   ) and of nonmembership (nonvalidity, etc., ]  ), so that for each element  of the universe, over which an IFS  is defined, the following inequality is valid: 0 ≤   () + ]  () ≤ 1.In this case, a pair ⟨, ⟩, where , ,  +  ∈ [0, 1], is called an intuitionistic fuzzy pair (IF pair).The ICA method produces the so-called positive and negative consonance coefficients between the different criteria used for evaluation of different objects.
The main contribution of the paper is the proposed hybrid approach combining intuitionistic fuzzy logic (through the ICA method) with modular neural networks for designing a powerful neural model for classification.The neural network model is tested with benchmark problems and a real world case to show the advantages of the proposed approach.As regards existing works that could be considered similar to this one, we can mention that intuitionistic fuzzy logic has not been considered in conjunction with modular neural networks previously, so it can be considered an original contribution to the area of computational intelligence that combines the advantages of the two methods.For the purpose of testing the proposed method for preprocessing the information going into MNNs, we use data from the LUKOIL Neftochim Burgas AD from the measurements of a set of crude oil probes (objects, in terms of ICA) against a set of technological properties (criteria, in terms of ICA), which precedes and conditions the process of production of petrochemical products from the crude oil [25], dataset for iris plant [26], and glass types [27].
The remainder of the paper is organized as follows.In Section 2 some short remarks about the intercriteria analysis method are given, which is based on intuitionistic fuzzy logic.Section 3 describes basic concepts about modular neural networks.Section 4 describes the simulations and a discussion of the results.Finally, Section 5 offers the conclusions and outlines future work in this area.

Short Remarks on the Index Matrices and Intercriteria Analysis Method
As we mentioned above, the ICA method [13,14] is based on two main concepts: intuitionistic fuzzy sets and index matrices.A brief description is offered below for completeness.Index matrices allow summarizing the criteria relevant to a particular decision making problem.
For any two IMs, a series of relations, operations, and operators have been defined.The theory behind the IMs is described in a more detailed fashion in [13].
Here, following the description of the ICA approach, given by [14], we will start with the IM called  with index sets with  rows { 1 , . . .,   } and  columns { 1 , . . .,   }, where for every ,  (1 ≤  ≤ , 1 ≤  ≤ ),   is an evaluated object,   is an evaluation criterion, and    ,  is the evaluation of the th object against the th criterion, defined as a real number that is comparable according to relation  with all the remaining elements of the IM .
From the requirement for comparability above, it follows that for each , ,  the relation (   ,  ,    ,  ) holds.The relation  has a dual relation , which is true in the cases when the relation  is false, and vice versa.For instance, if  is "greater," the dual relation  is "less." For the requirements of the proposed method, pairwise comparisons between every two different criteria are made along all evaluated objects.During the comparison, a counter is maintained for the number of times when the relation  holds, as well as another counter for the dual relation.
Let   , be the number of cases in which the relations (   ,  ,    ,  ) and (   ,  ,    ,  ) are simultaneously satisfied.Let also  ] , be the number of cases in which the relations (   ,  ,    ,  ) and the dual  = (   ,  ,    ,  ) are simultaneously satisfied.As the total number of pairwise comparisons between the objects is given by (-1)/2, it can be verified that the following inequalities hold: For every , , such that 1 ≤  ≤  ≤  and for  ≥ 2 two numbers are defined: The pair constructed from these two numbers plays the role of the intuitionistic fuzzy evaluation of the relations that can be established between any two criteria   and   .In this way, the IM  that relates evaluated objects with evaluating criteria can be transformed to another IM  * that gives the relations detected among the criteria, where stronger correlation exists where the first component    ,  is higher while the second component ]   ,  is lower.
From practical considerations, it has been more flexible to work with two IMs   and  ] , rather than with the IM  * of IF pairs.IM   contains as elements the first components of the IFPs of  * , while  ] contains the second components of the IFPs of  * .Once the intercriteria pairs have been calculated, for example, using the software described in [28], the question arises about defining the thresholds against which the membership and the nonmembership parts are evaluated [29,30].
As has been discussed in some publications on ICA, for example, in [31,32], the ICA results are very close to those obtained with the correlation analyses of Spearman, Pearson, and Kendall.It is worth noting the so far empirically observed fact that when in the data there are mistakes (e.g., shift of the decimal separator) these three correlation analyses give a larger deviation of the value than ICA; that is, ICA is less sensitive, so the use of them together can be used as a way of detecting errors in the input data.

Modular Neural Networks
Modular neural networks [33,34] are one of the models that can be used for object recognition, classification, and identification (see Figure 1).A modular neural network can be viewed as a set of monolithic neural networks [35][36][37] that deal with a part of a problem, and then their individual outputs are combined by an integration unit to form a global solution to the complete problem.The main idea is that a complex problem can be divided into simpler subproblems that can be solved by simpler neural networks and then the total solution will be a combination of the outputs of the simple monolithic neural networks.
In the proposed hybrid approach each of the MNN Modules takes as its input the result of the application of ICA method over three datasets (dataset for production of petrochemical products from the crude oil [25], iris plants [26], and glass types [27]).Every module is a two-layer Multilayer Perceptron and the output of the second layer of the ANN is  2  (for  ∈ {1, . . ., }), where  is the maximal number of modules.
The output  is calculated according to the following equation, which basically performs a weighted integration of the module outputs: where  2  is output of the module  ∈ [1, 2, . . ., ];   is average deviation of the output values of the module ;   is coefficient of existence of module .The coefficient of existence of every module shows the presence/absence of the respective module, with respect to the need of that module in the particular case.
For illustration purposes and simplifying calculations, we can choose to reduce the structure of the MNN and use 2 modules from the structure from Figure 2 ( 1 = 1,  2 = 1,  3 = 0, . . .,   = 0).The first module considers all independent inputs (weak dissonance, dissonance, and strong dissonance).The second module takes the inputs that have strong negative consonance, negative consonance, and weak negative consonance, and weak positive consonance, positive consonance, and strong positive consonance.
In the second module, we can reduce some of the inputs if they have very strong positive consonance.In this case, we can remove one of the inputs.In the case that they have strong negative consonance, this means that we can also remove some of the inputs.Other configurations for the modular neural network are possible, depending on how the inputs are selected.

Discussion of Results
For verifying the accuracy of the proposed method the data of the real dataset with measurements of crude oil probes, glass, and iris plants were used.

4.1.
Testing with Data for Crude Oil Probes.For the learning process of MNN, we set the following parameters: performance (MSE) = 0.00001; validation check = 15.The dataset was divided into three different parts: training (from 1 to 100); validation (from 101 to 120), and testing (from 121 to 140).For the purpose of learning algorithms, one of the variants of the backpropagation algorithm is used, namely, Levenberg-Marquardt.As a target for the model, the values from the databases were used.
In the MNN, the data for 140 crude oil probes, measured against 8 physical properties ("criteria" in the ICA terminology) were used (for the input data, see [34]).The eight criteria are as follows: (1) density at 15 ∘ C g/cm 3 ; (2) 10% (v/v) ASTM D86 distillation, ∘ C; (3) 50% (v/v) ASTM D86 distillation, ∘ C; (4) 90% (v/v) ASTM D86 distillation, ∘ C; (5) refractive index at 20 ∘ C; (6) H 2 content, % (m/m); (7) aniline point, ∘ C; (8) molecular weight g/mol.Using the ICA approach, we are seeking for correlations between these eight criteria on the basis of the 140 crude oil probes ("objects" in ICA).Using the ICA software [28] applied to the whole 140 × 8 matrix with measurements, we obtain 28 IF pairs, giving the pairwise correlations between the eight criteria.For easier processing of the result of ICA application, the output is given in the form of two IMs, containing the membership and the nonmembership parts of the IF correlations discovered between each pair of criteria (Tables 1 and 2), which in Table 3 are sorted in descending order according to the membership part of the intercriteria pairs, from strong positive consonance (i.e., pair 1-5 with degree ⟨0.956; 0.042⟩), through dissonance, to negative consonance (i.e., pair 5-6 with degree ⟨0.134; 0.866⟩).
On this basis, we separate these 28 intercriteria pairs in two groups, where Module 1 of the MNN takes as input the dissonant pairs, and Module 2 takes as input the consonant pairs, either positive or negative.In this way, we aim to reduce the number of input parameters of the MNN, yet keeping high enough level of precision.
In Table 4 we are presenting the values from 22 neural network simulations and the identifiers of criteria that we provide at the input on each module.These 22 simulations

Testing with Data for Iris Plants.
As another test case for the proposed method, we also consider the dataset [26] that contains four parameters (criteria): (1) sepal length in cm, (2) sepal width in cm, (3) petal length in cm, and (4) petal width in cm.There are 3 classes of 50 instances each [26], where each class refers to a type of iris plant.The targets are Iris setosa, 1, Iris versicolor, 2, and Iris virginica, 3.
The ICA method was applied to the 150 × 4 matrix in the same fashion as previously (see Tables 5, 6, and 7), and the results are given in Table 5, sorted in descending order according to the membership part of the intercriteria pairs.In the neural network inputs were assumed with the following parameters: parameter 2 (in the first neural network input) and parameters 1, 3, and 4 in consonance (in the second neural network input).In Table 8 the values from 7 neural network simulations are presented and a number of parameters that we defined for each module are described.

Testing with Data for Glass.
Finally, as a third case for testing the proposed approach we consider the benchmark glass data.In this case, in the input of this modular neural network, we use the experimental data from [27] for obtaining the type of glass.We work with data for 214 building and vehicle window glasses, measured against 9 criteria: (1) RI: refractive index; (2) Na: sodium (unit measurement: weight percent in corresponding oxide); (3) Mg: magnesium; (4) Al: aluminium; (5) Si: silicon; (6) K: potassium; (7) Ca: calcium; (8) Ba: barium; (9) Fe: iron.In the output (as a target) we give information about the glass type.
As was done in the previous sections, the input parameters (criteria) are divided in two groups: parameters in dissonance (parameters 2, 3, 4, 5, 6, and 9) and parameters in consonance (see Table 9), with the results sorted in descending order according to the membership part of the intercriteria pairs.In Table 10, the values from 7 neural network simulations are presented with the respective identifiers of the criteria that are feeding the inputs of each module.
The information is divided into two groups: the first part is the group of independent evaluation criteria (see Tables 11  and 12): criteria 2, 4, and 7 (see Table 4 for the crude oil probes data), criterion 2 (see Table 5 for the iris plant data), and criteria 2, 3, 4, 5, 6, and 9 (see Table 8 for the glass types data).These data were used on the inputs of the first module of the neural networks.The other criteria (dependent parameters) were considered on the inputs of the second module of the neural networks.
The inputs on the first module are not removable, because the ICA approach tests the independence of the criteria.The results from applying the ICA method over tree types of data show that most of the parameters are selected in a proper way (criteria in dissonance).For some of the parameters that were provided on the inputs of the second module, an approach for reasonable elimination of some of these criteria can be adopted.

Discussion.
In the first step we use neural networks without any removal of information and without removing of the inputs.The corresponding results can be found on the first rows in Tables 4, 8, and 10.
On the second step the data that have the highest and lowest consonance coefficients (strong positive consonance, positive consonance, strong negative consonance, and negative consonance) were removed.For example, in Table 4 we use parameters 2, 4, and 7 in Module 1 and parameters 3, 5, 6, and 8 (without 1) in Module 2. In the study, parameter 1 is removed due to the high value of membership coefficient  = 0.956 (between parameters 1 and 5).After simulating the neural network an average deviation of the MNN  3 = 2.1562 ( 1 = 2.2139,  2 = 2.1650) is obtained.
The removed parameters do not have substantial influence on the result, because of decreasing the number of the weight coefficients, and along with this the error of the output values also decreases.
In the same way the second group of parameters (from the pairs) were successively removed.This process continues until finishing the pairs in the highest and lowest consonance.
The next step is to remove two parameters, for example, in row 5 (Table 8) the situation with all parameters on Module 1, and parameter 4 (without 1 and 3) in Module 2 was presented.In the study, parameters 1 and 3 are removed due to the high value of membership coefficient  = 0.81852 (between parameters 1 and 3).After simulation of the neural network, an average deviation of the MNN  3 = 0.5846 ( 1 = 0.5725,  2 = 0.0631) is obtained.The removed parameters 1 and 3 do not have substantial influence on the result, because of decreasing the number of the weight coefficients, and along with this the error of the output values also decreases.
The process can continue iteratively in the same way.In summary, a new approach for building modular neural networks based on ICA, which uses intuitionistic fuzzy logic, has been proposed and tested with three benchmark databases.The simulation results are good, in this way verifying the advantages of the hybrid method.
In Table 13 comparative analysis of the results of the three different datasets is made.The table includes a short description of each dataset, the initial number of inputs (number of the inputs before reduction), the number of inputs after reduction, and average deviation of the output values of the neural networks ( 3 ), as well as minimal value of average deviation of the output values of the neural networks ( 3 , min), the number of weight coefficients without reduction and with reduction, the percentage ratio between the average deviation of the output values of the neural networks and the minimal value of average deviation of the output values of the neural networks, respectively, and the number of weight coefficients without and with reduction.
From the data in the table, it is seen that reducing the number of input parameters and, respectively, the number of inputs of the MNN, the value of average deviation of the output values of the MNN decreases (93,65%-98,2%).On this basis, we further observe decrease of the number of weight coefficients (60%-80%).

Conclusions
The number of the neurons is one of the major parameters that need to be defined during the realization of the MNN.Of course, the higher number of neurons in a neural network has the effect of slowing down the learning process.Here, we use the intuitionistic fuzzy logic based method of intercriteria analysis to reduce the number of input parameters of the modular neural network.This leads to reduction of the weight matrices and thus allows implementation of the neural network in limited hardware, saving time and resources in training.The method can also be used to assess the independence of the criteria against which data are measured.Three different real datasets with measurements of glass types, crude oil probes, and iris plant specimens were used to verify the accuracy and efficiency of the proposed approach.As future work, we envision using the proposed approach with other types of complex problems or more real life situations, like in the design of intelligent controllers for robotic systems  Complexity or in the design of pattern recognition systems for human identification based on biometric measures.We believe that for real world situations some pertinent modifications or improvements to our model could be needed, but the essence of the solution to the problems is already provided in the proposed approach.As another leg of future investigation in the proposed direction, we consider a three-module MNN, which will be fed with the following inputs: one with the intercriteria pairs exhibiting positive consonance, another with the intercriteria pairs exhibiting negative consonance, and the third one with the pairs exhibiting dissonance.

Figure 1 :Figure 2 :
Figure 1: The structure of the MNN.

Table 1 :
Membership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations between measurements of crude oil probes.

Table 2 :
Nonmembership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations between measurements of crude oil probes.

Table 3 :
Relations among the criteria with specific values.Based on the values from Table3, we offer the use of the relations of positive consonance and negative consonance, since both affect in the same way the inputs of the neural network.The parameters  1 ,  2 , and  3 are the average deviations of the output values of Module 1, Module 2, and whole neural network, respectively.A detailed discussion of the results follows in Section 4.4.

Table 4 :
Results from the simulations.

Table 5 :
Relations among the criteria.

Table 6 :
Membership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations between measurements of iris plant.

Table 7 :
Nonmembership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations between measurements of iris plant.

Table 10 :
Results from the simulations.

Table 11 :
Membership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations of glass.

Table 12 :
Nonmembership parts of the intuitionistic fuzzy pairs, giving the intercriteria correlations.

Table 13 :
Results from the simulations.