Desirability Improvement of Committee Machine to Solve Multiple Response Optimization Problems

Multiple response optimization (MRO) problems are usually solved in three phases that include experiment design, modeling, and optimization. Committee machine (CM) as a set of some experts such as some artificial neural networks (ANNs) is used for modeling phase. Also, the optimization phase is done with different optimization techniques such as genetic algorithm (GA). The current paper is a development of recent authors’ work on application of CM in MRO problem solving. In the modeling phase, the CM weights are determined with GA in which its fitness function is minimizing the RMSE. Then, in the optimization phase, the GA specifies the final response with the object to maximize the global desirability. Due to the fact that GA has a stochastic nature, it usually finds the response points near to optimum. Therefore, the performance the algorithm for several times will yield different responses with different GD values. This study includes a committee machine with four different ANNs. The algorithm was implemented on five case studies and the results represent for selected cases, when number of performances is equal to five, increasing in maximum GD with respect to average value of GD will be eleven percent. Increasing repeat number from five to forty-five will raise the maximumGD by only about three percent more. Consequently, the economic run number of the algorithm is five.


Introduction
Multiple response optimization (MRO) problems need to find a set of input variable values (x's) which get a desired set of outputs (y's).The current study develops a proposed algorithm in recent authors' work to solve MRO problems [1].MRO solution methodologies usually include three phases: experiments design, modeling, and optimization.
There are some techniques for experiments design.Some methodologies in this phase are as follows: design of experiments (DOEs) knowledge such as factorial design and fraction factorial design, response surface methodology (RSM) such as central composite design (CCD), and Box Behnken [2,3].Furthermore, Taguchi orthogonal arrays [4][5][6][7] are derived from the Taguchi method.
Modeling as the second phase is done using different mathematical or statistical models such as multiple linear and nonlinear regressions in the form of polynomials [2,8,9] and artificial neural networks (ANNs).Due to the existence of complicated relationship between inputs and outputs, usually ANNs are mostly used for modeling rather than for polynomials.One famous artificial neural network (ANN) is back propagation neural network (BPNN) that is used in many engineering problems [10,11].Cheng et al. [12] utilized MANFIS (multiadaptive neuro fuzzy inference system) for modeling and showed that the results are superior to RSM polynomial models.
The last phase is optimization, which is usually done on a performance metric such as global desirability function.In this process, each predicted response is converted to a value between 0 and 1.Finally, a composite function is defined which converts all desirability functions to a unique number by global desirability function (GDF).Also, Chatsirirungruang and Miyakawa [13] proposed a combination of Taguchi   and GA to get more accurate responses by using the benefits of both techniques together.

Neural Networks and Committee Machine
Different kinds of neural networks are used to model in complicated prediction problems.Four neural networks are used in this study that include feed forward neural networks (FF) [14], radial basis function networks (RBFNs) [15], generalized regression neural network (GRNN) [16], and adaptive neural fuzzy inference system (ANFIS) [17,18].
A committee machine (CM) is a collection of some intelligent systems named experts and a combiner which combines the outputs of each expert (Figure 1).The advantage of CM is that it reaps the benefits of all work with only little additional computation.Independent variables are entered for experts, and all experts' outputs are transferred to a combiner to get the final response.
One of the most popular methods to combine the experts' outputs is the simple ensemble averaging method according to (1) [19].Furthermore, a combiner could be an intelligent system such as a neural network.Consider where   is the weight coefficient of th expert,   is the estimated response from th expert, and  is the total number of the experts [20].
Genetic algorithm could be used to yield the experts' contribution (weights) in a committee machine.Equation (2) represents that the committee machine gives smaller errors than the average of all the experts [20,21]: where   =   ANN−   real is the error of predicted and real response of each expert and  2  is the squared error for the th expert.Error ave is the average error for all experts and Error CM is the error of CM.

Global Desirability and Genetic Algorithm
Overall, desirability or global desirability function is used to transmit multiple responses to a single response case.Desirability function converts each estimated response into a dimensionless desirability value   .It gets   values according to the kind of objects in the problem.These conditions are shown in (3), (4), and (5) [31,32].

Desirability Functions Formula with Different Objects.
The desirability for goal of "Target:" The desirability for goal of "Maximum:" The desirability for goal of "Minimum:"    [22] 2008 RSM RSM Graphical Chang [5] 2008 Taguchi ANN SA Chatsirirungruang and Miyakawa [13] 2009 Taguchi Taguchi GA Cheng et al. [12] 2 0 0 2 R S M A N N G A Cojocaru et al. [23] 2009 Full factorial MLR Graphically Martinez Delfa et al. [24] 2009 RSM RSM, ANN Mathematically Mukherjee and Ray [10] 2008 N/A ANN Modified TS Nagesh and Datta [25] 2010 Fractional factorial design MLR, ANN GA Noorossana et al. [11] 2008 RSM ANN, FS GA Pasandideh and Niaki [9] 2 0 0 6 R S M R S M G A Patnaik and Biswas [26] 2007 Taguchi Taguchi (S/N) Weighting Pizarro et al. [27] 2006 Taguchi RSM Graphically 4 Advances in Artificial Neural Systems  where the parameters  and  in the formulas are convexity coefficients and specify how strictly the target value will be desired.In the current study,  and  are equal to one.Global desirability (GD) function is according to (6): Equations from (3) to ( 5), yield the single desirabilities for different objects and ( 6) calculates the global desirability (GD).Both   's and GD values range vary from zero to one.In the MRO problems, it is important that all responses optimize simultaneously, and GD is a suitable performance metric to achieve this target.Genetic algorithm (GA) is a population-based search technique, which can quickly and reliably solve problems that are difficult to tackle by traditional methods.One advantage of GA is that it is extensible and can interface with existing models and hybridize with them and optimizes the fitness function [33,34].
Also Brie and Morignot [35] state that genetic algorithm has stochastic nature, and consequently, the results may highly vary from test to test, even for the same problem and parameter set.
Different methods have been proposed in the literature for the optimization of multiple response problems.Table 1 shows corresponding techniques.In this table, some include only investigation for analysis and comparison not optimization.
As a consequence, by reviewing the above works and other works in the literature, since the genetic algorithm has been widely implemented by the researchers for optimization phase of MRO problems with respect to other techniques, this metaheuristic algorithm was selected as the optimization technique.

Methodology
First of all, an important matter is the selection of data for training and testing of model.Dixit and Chandra [36] have suggested a selection method for ANNs.According to their suggestions, for n inputs, the minimum number of training set should be such that it includes the corners of n-dimensional space with respect to more contribution to input variables with more influence on output.In the current investigation, this suggestion was applied for corners of lower and upper limits for all independent variables.Also, training and testing dataset numbers were 80 and 20 percent, respectively.
Different criteria are used to assess forecasting models performance.Two criteria were selected in the current work, which compare models' results with the observed or real data.They are root mean square error (RMSE) [37] and correlation coefficient () [38].
where ŷ is th predicted value or model output,   is the th actual value, and  is the number of data used for prediction.Furthermore,  and ŷ are the means of actual and predicted values [39].The current work includes two conditions to build ANNs model: first is that RMSE for all data is the minimum and the second condition is that the correlation coefficient of testing data is positive.Usually, MRO solution includes three phases.Phase one is experiments design, in which in the current work, all data are selected from the literatures.The second phase is modeling which is done by building four different neural networks and a committee machine.ANNs include feed forward, RBF, GRNN, and ANFIS models.All neural networks have the same inputs and one output, and so the number of ANNs in each model is equal to the number of responses (Figure 2) [1].
A committee machine (CM) was made by a combination of all four ANN models (Figure 3). inputs are entered for each expert of CM simultaneously, and  responses are multiplied to their weights and then are added together to get the final response.Committee machine combiner is an ensemble averaging.Genetic algorithm (GA) computes CM weights with the object to minimize RMSE of CM response.So the weight matrix is an  *  matrix.
The object of the current study is to find the economic performance number of the committee machine and genetic algorithm to get the best responses in MRO problems solving.Therefore, firstly, four ANNs and one committee machine were created separately.Committee machine weights were calculated by means of GA with the object of minimizing overall RMSE.Then in the optimization phase, GA yields the best responses with the object of maximizing global desirability.The result is  * and  * with the highest possible GD.These calculations of finding CM weights and  * were repeated 45 times.The schematic of the methodology is shown in Figure 4 and corresponding algorithm (Algorithm 1).

Results and Discussion
Genetic algorithm is applied in two steps.The first step is to find CM weights with the object of minimizing the overall RMSE of CM, and the second step is to find the 's by GA and ANNs with the object of maximizing global desirability.In both steps, GA specifications are listed in Table 2.
The current algorithm is implemented on five MRO problems.These problems include different numbers of inputs and outputs and different numbers of experiments.Table 3 represents their properties.
Case 1.The first problem is based on the wire-bonding process in the semiconductor industry.Table 4 represents the process inputs and outputs.Different neural networks Case 2. The problem is to optimize the yield of recombinant Oryza sativa nonsymbiotic hemoglobin 1 in a medium containing byproduct glycerol.Table 7 represents the input and output variables of this case.
Case 3. The problem is multiple response optimization of styrene-butadiene rubber (SBR) emulsion batch polymerization.Table 8 lists the input and output variables.
Case 4. The object of this case is to optimize process variables, electrolysis voltage, and treatment time for the electrocoagulation removal of hexavalent chromium (Cr(VI)).Table 9 represents the input and output variables.
Case 5.The problem is to optimize multiple characteristics in CNC turning of AISI P-20 tool steel using liquid nitrogen as a coolant.Table 10 lists the input and output variables.
In all five cases, the CM responses that include GD and RMSE were calculated 45 times.The results of GD are listed in Tables 11 and 12, representing the statistical results.Case 1 was eliminated in the calculations and the reason is due to the existence of zero values in GD; the increasing of maximum GD to average GD is very high and this can mislead us to unmoral results.So only cases from two to five are considered and this will yield smaller increase, but more reliable.
The GD ratio is defined in formula (8) and represents ratio of increasing maximum GD to average GD: GD ratio (for  runs) = Max GD − Ave.GD Ave.GD × 100.(8) Advances in Artificial Neural Systems Also, to investigate for ANNs behavior, the results of five runs are listed in Table 13.For abstract only Case 2 is listed.Table 14 represents statistical results of this case.It is obvious that in both CM and ANNs, RMSE is constant for all run numbers.Table 13 shows this reality for Case 2 with ANNs models.Table 13 shows there is no significant difference between GD values with respect to run numbers for different ANNs runs.Table 13 represents, for all case, that there is an increase in the mean of GD ratio (or mean of increasing the maximum GD to average GD) with respect to increasing the run number.
Figure 5 shows the corresponding results graphically and it illustrates that for committee machine, if the program performs, for example, 3 times, the maximum to average will increase to 7.8%.In addition, it shows that if the program runs 5 times, the maximum value of GD can increase to 11.7% with respect to average.From run numbers 5 to 8, there is a slight rise about 1.6%.From run numbers 8 to 10, there is a relatively fall in GD ratio.Then from run numbers ten to forty-five, there is no significant rise in GD ration and it is only 2.3 percent (from 12.8% to 15,1%).Consequently, the economical run number for the algorithm is five times.Because by consuming time from 5 to 45 times will increase GD ratio about 3.2% (15.1%-11.7%)whereas run number equal five times has 11.7% and more than 3 times.
Table 13 shows for different ANNs run numbers, there is only about 1% increasing in GD ratio for run numbers more than one and this rise is not noticeable, because increasing 1% is due to nature of GA.So to run more than 3 times for neural networks models has no noticeable effect to increase GD ratio.

Conclusion
Multiple response optimization (MRO) problem solving is usually done in three phases that include experiments design, modeling, and optimization.Committee machine (CM) as a collection of some experts such as some artificial neural networks (ANNs) can be used in the modeling phase of MRO.Genetic algorithm is used to find CM weights in the modeling phase and also as main optimization techniques in the optimization phase.
The current study modifies a proposed algorithm from recent works of authors that had used CM and GA to solve MRO problems.Due to stochastic nature of GA, the final solutions vary together and different performances will yield different responses with related global desirability (GD).So since object of MRO is to find responses with highest GD, to know economic run number will be useful to obtain best responses in minimum possible time.According to this investigation and for the selected MRO problems, the results represent that the economic run number of the algorithm is five.With five run numbers, maximum global desirability of final solution can increase about 11 percent in concern with average of GD.Whereas, to run the algorithm from five to forty-five numbers, the maximum of global desirability can increase only about 3 percent more.

Figure 1 :
Figure 1: A typical architecture of a committee machine based on static structure.

Figure 2 :
Figure 2: Inputs and outputs of every model.

Figure 5 :
Figure 5: GD ratio increasing with respect to number of runs.

Table 1 :
Classification of some works in MRO subject in the literature.

Table 4 :
,  * and GD( * ) using GA for goal of maximizing in Global desirability Input and response variables and optimization criteria for every response (output) in Case 1. Target  4 : maximum temperature at position B ( ∘ C) Target  5 : beginning bond temperature at position B ( ∘ C) Target  6 : finish bond temperature at position B ( ∘ C) Target * 1 : flow rate (SCFM)  1 : maximum temperature at position A ( ∘ C) Target  2 : flow temp ( ∘ C)  2 : beginning bond temperature at position A ( ∘ C) Target  3 : block temp ( ∘ C)  3 : finish bond temperature at position A ( ∘ C)

Table 7 :
Input and response variables and optimization criteria for every response (output) (Case 2).

Table 8 :
Input and response variables and optimization criteria for every response (output) (Case 3).

Table 10 :
Input and response variables and optimization criteria for every response (output) (Case 5).

Table 11 :
GD values according to run number of CM.

Table 12 :
Statistical results of GD values according to run number of CM (Cases 2-5).

Table 13 :
Results of five runs for Case 2.

Table 6 .
Case 3 has deferent specifications to get acceptable results.Four neural networks that include feedforward (FF), radial base function (RBF), GRNN, and ANFIS were consisted in each response for each problem data.So every problem finds (4 * no. of responses) models.A committee machine was set with the object to minimize the overall RMSE.