An Ensemble Learning for Predicting Breakdown Field Strength of Polyimide Nanocomposite Films

1School of Applied Science, Harbin University of Science and Technology, Harbin 150080, China 2College of Computer Science and Engineering, Dalian Nationalities University, 18 Liaohe West Road, Dalian Development Zone, Dalian 116600, China 3Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian 116023, China 4Faculty of Engineering, Mudanjiang Normal College, Mudanjiang 157012, China


Introduction
As one of material products that have been developed for a long time, polyimide film (PI film) has been mainly applied in high and new technology industries such as aerospace, machinery, electrical and electronics engineering, optical communication, LCD, automobile, precision instrument, gas separation, and microelectronics [1].With the development of nanotechnology, nanoparticles of different sizes, percentage compositions, and components have been mixed with PI by more and more researchers to produce high-quality polyimide nanocomposite films [2,3].
Breakdown field strength is an important characteristic parameter to characterize polyimide nanocomposite films.It can be calculated by (breakdown voltage)/(film thickness).Many researchers have already made studies and analysis on the breakdown field strength of nanocomposite films [4][5][6].
There are many factors that can impact the breakdown field strength, including the type of nanoparticle, dielectric constant, electric conductivity, coefficient of thermal conductivity, composition, nanoparticles' size and specific area, and composite film thickness.In order to establish the knowledge base for the material property regarding breakdown field strength of polyimide nanocomposite films, large quantities of experiments have to be prepared and measure related characteristics.Nevertheless, it is also well known that getting properties data is very costly in terms of time and materials.It is for this reason that developing a fast and efficiency method to predict the breakdown field strength of polyimide nanocomposite films is very much in demand.
Intelligent computing and neuronal network have been widely applied in performance prediction, identification, and optimization of nanocomposite films.Yang et al. used a generalized regression neural network (GRNN) to predict the friction coefficient of Cr 1− Al  C film [7].Cho et al. optimized the characteristics of ITO/Al/ITO multilayer films by advantages of neural network and genetic algorithm [8].Bahramian made use of an artificial neural network to predict the growth rate of TiO 2 nanostructured film [9].Ensemble learning has now become a new hotspot of intelligent computing [10][11][12][13].By using ensemble, several weak classifiers can be constructed into a strong classifier.Some common methods of ensemble learning include boosting, bagging, and stacking.Boosting [14] keeps upgrading weights during data extraction and revises the weights of data set that have been classified wrong.In the end, several weak classifiers are obtained and can be constructed into a strong classifier.Bagging [15] is used the reiterative training to get several classifiers based on a training set.Stacking [16] contains two layers.In the first layer, different algorithms are employed to generate several weak classifiers.At the same time, a new data set with the same size of the original data set is also generated.Then, the new data set together with a new algorithm can be used to construct the classifier of the second layer.
The purpose of this paper is to develop a Stochastic Gradient Boosting + SMO-SVR model (SGBS model) to predict the breakdown field strength of polyimide nanocomposite films.In what follows, film preparation and prediction model are introduced first.The experimental details for sputtering systems and materials are described next.Then, the experimental results are described and the establishment of SGBS model with 10-fold cross validation results is carried out.Comparison experiments between linear regression, BP neural network (BP), general regression neural network (GRNN), SVR (support vector regression), and SMO-SVR models are conducted.To verify our SGBS model, we have designed and prepared in this paper nanoparticle samples with different types, sizes, ratios, and thicknesses.The SEM image of surface appearance of pure PI is given in Figure 1(a).The surface of film is smooth and its tightness is well.Figure 1(b) shows the SEM image of surface appearance of PI/BaTiO 3 composite film doped with content of 60 wt%.There are large amounts of BaTiO 3 nanoparticles exposing on the surface of film with the size from 100 nm to 300 nm.Comparing with the other films, the surface appearance of film has been changed to be

Nanocomposite Film Preparation and Prediction Model
where  means input and  means output.The training sample needs to be turned into a linearly separable problem when it is a linear inseparability.Generally, the sample space can be mapped from a low-dimensional space into a high dimensional space by using a kernel function.The nonlinear regression function (estimation) is where  is a threshold,  is a weight, and () is the nonlinear mapping function.The loss function is shown as follows: By importing a Lagrange multiplier, the minimization of the objective function can be expressed as Transform ( 4) into a dual problem, and we get max where  is a penalty factor and  *  and   are Lagrangians; then the regression function is where  is the number of support vectors in (6) [18].In this paper, a normalized polynomial kernel has been taken as the kernel function.John Platt, from Microsoft Research, proposed the SMO (sequential minimal optimization) algorithm in order to shorten the training time of support vector regression in 1998 [19].SMO can optimize the  value of two samples at one time.Through the loop iteration with a given times , the  value of all samples can be optimized, as shown in Figure 2.For the data in Table 2, the method of 10-fold cross validation is used to make the model training on the standard SVR and SMO-SVR.The time of modeling SVR and SMO-SVR is 0.26 s and 0.03 s, respectively.SMO can promote the training efficiency of support vector regression while reducing the training time of the model.

Promoting the Model by Stochastic Gradient Boosting.
Boosting [20], as one of the most important ensemble learning methods, is to obtain a predictive function by construction and ensemble of a series of predictive functions.The core idea of gradient boosting, proposed by Friedman [21], is to construct an ensemble learning machine by calculating a loss function and letting the function descend along its gradient.In other words, it is to calculate the loss function of previous model, so as to build a new model along the descending gradient direction of the loss function.Eventually, a regression  (5) Fit the SMO-SVR by using least squares to obtain the fitting model ℎ(  ,   ).
(7) Generate a new model (8) Stochastically extract % of the training sample to fit the SMO-SVR during every time of iteration; then we can get the SGBS model after  times of iterations:

Experiment and Result Analysis
where   represents predicted values and   represents real values.RMSE (root mean squared error) is the square root of the ratio of the quadratic sum of deviations between predicted values and real values to the times () of predictions.It is sensitive to maximum or minimal errors of a group of predicted values and therefore can well reflect prediction accuracy.RMSE is inversely proportional to the prediction accuracy.The smaller the RMSE is, the more accurate the predictor can be.It can be expressed as follows:     prediction accuracy.The smaller the RSE is, the higher the prediction accuracy can be: RRSE (root relative squared error) can be calculated as follows: RRSE is also inversely proportional to prediction accuracy.The smaller the RRSE is, the higher the prediction accuracy can be [22].

Experimental Results and Analysis.
To verify the SGBS prediction model, the type, dielectric constant, electrical resistivity, thermal conductivity, size and specific area of nanoparticles, and the thickness of films in Table 2 are taken as the input , and the breakdown field strength of hybrid PI is taken as the output .In this paper, we use Macbook Pro (CPU: Intel I7-2640 M; memory: 16G) as the hardware for experiments and use Matlab 2012a to program prediction model.
Experiment 1. Use the method of 10-fold cross validation to fit the data in Table 2. Main parameters of the SGBS model include kernel function,  and , of which  is the penalty factor of SMO-SVR and  is the extracting ratio when training the sample.First of all, normalize the sample data.Three kernel functions-normalized polynomial kernel, polynomial kernel, and RBF kernel-are employed to test the sample.The results are listed in Table 3. From the table we know that the normalized polynomial kernel has the highest correlation coefficient but lowest mean absolute error, root mean squared error, relative absolute error, and root relative squared error.Therefore, we choose it as the kernel function for the SGBS model.
For optimization function, the penalty factor of outliers is selected by experience as well as by experiments.Figure 3 shows the comparison on prediction performances of SGBS model when  varies from 1 to 1.8. Figure 3(a) demonstrates that the correlation coefficient reaches its peak values when  = 1.6, 1.7, and 1.8, which are chosen as the value of , for correlation coefficient is proportional to prediction accuracy.By analyzing Figures 3(b) and 3(d) we know that the mean absolute error and relative absolute error reach their minimal values when  = 1.7,only larger than the values when  = 1.6 and 1.8.Therefore, they reach optimal performance when  = 1.7, for mean absolute error and relative absolute error are inversely proportional to prediction accuracy.
By analyzing Figures 3(c) and 3(d) we know that the smallest root mean squared error and root relative squared error appear at  = 1.6, followed by  = 1.7 and  = 1.8.Based on the above analysis, this paper takes  = 1.7 as the optimum value.
is the specify shrinkage rate in Stochastic Gradient Boosting, namely, the proportion of the stochastically extracted training sample.Figure 4 shows the impact of variation of  on prediction performance when  = 1.7.When  = 1, correlation coefficient reaches its peak value, and the prediction attains the best performance as shown in Figure 4(a).By analyzing Figures 4(b)-4(e) we know that when  = 1, mean absolute error, root mean squared error, relative absolute error, and root relative squared error reach their minimal values, signifying the best prediction performance.
According to the above analysis we know that when  = 1.7 and  = 1 are taken as the optimum values of the SGBS model, the prediction performance is the best.Figure 5 shows the degree of fitting between predicted values and real ones.Figure 6 shows the absolute error ratio of prediction.The tables and figures demonstrate that the error ratios of sample 8 (70 wt%, 100 nm, BaTiO 3 ), sample 17 (15 wt%, 40 nm, SiO 2 ), and sample 30 (15 wt%, 7 nm, SiO 2 ) are all larger than 15%, indicating an ordinary fitting, while the error ratios of other samples are less than 15%, indicating a better fitting.For all the six multicomponent TiO 2 film samples with the thickness of 35 m, the prediction errors are not larger than 15%.For the eight BaTiO 3 film samples, there is one sample of which the prediction errors are larger than 15%.For all the nine multicomponent Al 2 O 3 film samples, the prediction errors are not larger than 15%, while for the nine multicomponent SiO 2 film samples, there are also two samples of which the prediction errors are larger than 15 wt%.In this model, the prediction performance of the multicomponent Al 2 O 3 and rutile TiO 2 with different thicknesses is better than that of the BaTiO 3 and SiO 2 film samples.For the thirty-two nanocomposite films with different components, mixtures, and thicknesses, there are twenty-nine film samples of which the prediction errors are lower than 15%, proving that the model is of practical value in actual engineering works.
Experiment 2. In order to further verify the SGBS model, it needs to be compared to other models, namely, the linear regression, BP neural network, GRNN neural network, SVR (support vector regression), and SMO-SVR, under the same conditions.Comparison results of the prediction performance of these models are shown in Table 4.The correlation coefficient of SGBS model is 0.962, larger than that of the models of linear regression [23], BP neural network [24], GRNN neural network [7], SMO-SVR [25], and SVR [26], proving that the linear regression relationship of the SGBS model is better than that of the other five models.In the SGBS  model, the root mean squared error and root mean squared error are 20.8668 and 26.520%, respectively, lower than those of the other models, certifying its better prediction performance.
Experiment 3. In order to validate the generosity of the model, the actual measurement results conducted by Shi et al. in [27] are chosen to do the prediction.In this preference, the particle size is 30 m, and the doping ratio is 2 wt% with PI/nano-Al 2 O 3 composite films.The thickness of film is 30 m.Its breakdown field strength is 233 kV/mm.Adopting the samples in Table 2 as the training set of SGBS model.For the parameters of the model,  = 1.7,  = 1.Using the 2 wt%-PI/nano-Al 2 O 3 composite films in [27] as testing sample, the results could be obtained as shown in Table 5.For the test result of breakdown field strength that is 233 kV/mm, its prediction value is 225.9667kV/mm by SGBS model.The error ratio is 3.0185%.The MAE, RMSE, RAE, and RRSE are 7.0334, 7.0334, 24.5976, and 24.5976.The prediction and actual data are in good agreement.This method could predict the PI/nano-Al 2 O 3 composite films effectively.

Conclusions
This paper presents an ensemble learning method for predicting breakdown field strength of polyimide nanocomposite films.By using the method of Stochastic Gradient Boosting, ten SMO-SVR prediction models are constructed into a strong prediction model (SGBS model) that is efficient in predicting the breakdown field strength.Through analyzing the experiment data we obtain following conclusions: (1) In prediction of thirty-two nanocomposite films of different components, particles, and thicknesses (25-30 m) by using the method of 10-fold cross validation, there are twenty-nine samples of which the prediction errors are lower than 15%, proving that the SGBS model is efficient in predicting the breakdown field strength of polyimide nanocomposite films.
(2) Comparisons show that the SGBS model has a larger correlation coefficient than that of linear regression, BP, GRNN, SVR, and SMO-SVR models but smaller root mean squared error and root relative squared error.Hence, prediction performance of the SGBS model is better than that of the other five models.
(3) The SGBS model shows a better prediction on Al 2 O 3 and rutile TiO 2 films than on BaTiO 3 and SiO 2 films.
Next, some other ensemble learning methods will be employed to predict the corona resistance, dielectric constant, dielectric constant, and thermal properties of polyimide nanocomposite films.

Figure 1 :
Figure 1: SEM image of the nanocomposite film.(a) The SEM image of pure PI film; (b) the SEM image of surface appearance of PI/BaTiO 3 composite film doped with content of 60 wt%; (c) the SEM image of surface appearance of PI/TiO 2 composite film doped with content of 5 wt%; (d) the SEM image of surface appearance of PI/Al 2 O 3 composite film doped with content of 20 wt%.

Figure 3 :
Figure 3: The impacts of variation of penalty factor () on prediction performance.(a) The correlation coefficient with different penalty factors (); (b) the mean absolute error with different penalty factors (); (c) the mean absolute error with different penalty factors (); (d) the root mean squared error with different penalty factors (); (e) the root relative squared error with different penalty factor (). 0

Figure 4 :
Figure 4: Impact of  on prediction performance.(a) Correlation coefficient relative to shrinkage rate () of training samples; (b) mean absolute error relative to shrinkage rate () of training samples; (c) root mean squared error relative to shrinkage rate () of training samples; (d) relative absolute error relative to shrinkage rate () of training samples; (e) root relative squared error relative to shrinkage rate () of training samples.

Figure 5 :
Figure 5: Comparison between real values and predicted values of the SGBS mode.
Al 2 O 3 , rutile TiO 2 , BaTiO 3 , SiO 2 , and ethanol.The detailed purchase information of the above experimental materials is shown in Table1.Firstly, put PMDA into the solution of ODA in DMAc to produce an amount of polyamide acid of certain viscosity.Secondly, add in different nanoparticles.Finally, let the mixture go through paving membrane heat treatment and imidization transform.

Table 2 :
Data of breakdown field strengths of nanocomposite films.

Table 2
Take polymethylphenylsiloxane fluid with high insulation strength (which can be up to 16 kV/mm) as the medium, with the rate of voltage rise of 500 V/s.Test the breakdown field strength of the nanocomposite films prepared in Section 2.1.Select 20 samples as a group for testing each hybrid PI, and take the mean value of the 10 middle breakdown field strengths as the breakdown field strength of the group.Then, calculate the standard deviation.After the experiment we get the data of breakdown field strengths as shown in   (independent variable) and   (response variable) of the model.It is a coefficient of a simple linear correlation between   and its estimated values.CC is larger than zero and ranges from 0 to 1.A larger CC means a more relevant linear regression relation- 3.1.Construction of the Experimental Sample.Test the breakdown field strength by using a withstanding voltage tester (type: CS2674C).Voltage range is 0-50 kV; test error is ±5%; leakage current measurement range is 0.5-20 mA; measurement error is ±5%. ∑ =1           ,

Table 3 :
Comparison on prediction performances of three kernel functions.

Table 4 :
Comparisons of models' prediction performance.

Table 5 :
The comparison of the predicted and tested values of breakdown field strength.