This paper reviews methods to fix a number of hidden neurons in neural networks for the past 20 years. And it also proposes a new method to fix the hidden neurons in Elman networks for wind speed prediction in renewable energy systems. The random selection of a number of hidden neurons might cause either overfitting or underfitting problems. This paper proposes the solution of these problems. To fix hidden neurons, 101 various criteria are tested based on the statistical errors. The results show that proposed model improves the accuracy and minimal error. The perfect design of the neural network based on the selection criteria is substantiated using convergence theorem. To verify the effectiveness of the model, simulations were conducted on realtime wind data. The experimental results show that with minimum errors the proposed approach can be used for wind speed prediction. The survey has been made for the fixation of hidden neurons in neural networks. The proposed model is simple, with minimal error, and efficient for fixation of hidden neurons in Elman networks.
One of the major problems facing researchers is the selection of hidden neurons using neural networks (NN). This is very important while the neural network is trained to get very small errors which may not respond properly in wind speed prediction. There exists an overtraining issue in the design of NN training process. Over training is akin to the issue of overfitting data. The issue arises because the network matches the data so closely as to lose its generalization ability over the test data.
Artificial neural networks (ANN) is an information processing system which is inspired by the models of biological neural networks [
From the development of NN model, the researcher can face the following problems for a particular application [
How many hidden neurons can be used?
How many training pairs should be used?
Which training algorithm can be used?
What neural network architecture should be used?
The hidden neuron can influence the error on the nodes to which their output is connected. The stability of neural network is estimated by error. The minimal error reflects better stability, and higher error reflects worst stability. The excessive hidden neurons will cause over fitting; that is, the neural networks have overestimate the complexity of the target problem [
Prediction plays a major role in planning today’s competitive environment, especially in the areas characterized by high concentration of wind generation. Due to the fluctuation and intermittent nature of wind, prediction result varies rapidly. Thus this increases the importance of accurate wind speed prediction. The proposed model is to be implemented in Elman network for the accurate wind speed prediction model. The need for wind speed prediction is to assist with operational control of wind farm and planning development of power station. The quality of prediction made by the network is measured in terms of error. Generalization performance varies over time as the network adapts during training.
Thus various criteria were proposed for fixing hidden neuron by researchers during the last couple of decades. Most of researchers have fixed number of hidden neurons based on trial rule. In this paper, new method is proposed and is applied for Elman network for wind speed prediction. And the survey has been made for the fixation of hidden neuron in neural networks for the past 20 years. All proposed criteria are tested using convergence theorem which converges infinite sequences into finite sequences. The main objective is to minimize error, improve accuracy and stability of network. This review is to be useful for researchers working in this field and selects proper number of hidden neurons in neural networks.
Several researchers tried and proposed many methodologies to fix the number of hidden neurons. The survey has been made to find the number of hidden neurons in neural network is and described in a chronological manner. In 1991, Sartori and Antsaklis [
In 1995, Li et al. [
In 1996, Hagiwara [
In 1997, Tamura and Tateishi [
In 1999, Keeni et al. [
In 2001, Onoda [
In 2003, Zhang et al. [
It has high first hidden layer and small second hidden layer.
Weights connecting the input to first hidden layer can be prefixed with most of the weights connecting the first hidden layer and second hidden layer that can be determined analytically.
It may be trained only by adjusting weights and quantization factors to optimize the generalization performance.
It may be able to overfit the sample with any arbitrary small error.
In 2006, Choi et al. [
The quality of prediction made by the network is measured in terms of the generalization error. Generalization performance varies over time as the network adapts during training. The necessary numbers of hidden neurons approximated in hidden layer using multilayer perceptron (MLP) were found by Trenn [
In 2009, Shibata and Ikeda [
In 2010, Doukim et al. [
In 2012, Hunter et al. [
The other algorithm used to fix the hidden neuron is the data structure preserving (DSP) algorithm [
Another approach to fix hidden neuron is the sequential orthogonal approach (SOA). This approach [
The researchers have been implemented various methods for selecting the hidden neuron. The researchers are aimed at improving factors like faster computing process, more efficiency and accuracy and less errors. The proper selection of hidden neuron is important for the design of neural network.
The proper selection of number of hidden neurons has been analyzed for Elman neural network. To select hidden neurons in order to to solve a specific task has been an important problem. With few hidden neurons, the network may not be powerful enough to meet the desired requirements including capacity and error precision. In the design of neural network, an issue called overtraining has occurred. Over training is akin to the problem of overfitting data. So fixing the number of a hidden neuron is important for a given problem. An important but difficult task is to determine the optimal number of parameters. In other words, it needs to measure the discrepancy between neural network and an actual system. In order to tackle this, most researches have mainly focused on improving the performance. There is no way to find hidden neuron in neural network without trying and testing during the training and computing the generalization error. The hidden output connection weights becomes small as number of hidden neurons become large, and also the tradeoff in stability between input and hidden output connection exists. A tradeoff is formed that if the
The input and output neuron is to be modeled, while
There exists various heuristics in the literature; amalgamating the knowledge gained from previous experiments where a near optimal topology might exist [
Elman network has been successfully applied in many fields, regarding prediction, modeling, and control. The Elman network is a recurrent neural network (RNN) adding recurrent links into hidden layer as a feedback connection [
For the considered wind speed prediction model, the inputs are temperature
Let
Let
Let
Architecture of the proposed model for fixing the number of hidden neurons in Elman Network.
From Figure
Weight vector of input to hidden vector,
Weight vector of hidden to recurrent link vector,
Weight vector of recurrent link layer to input vector,
Generally, neural network involves the process of training, testing, and developing a model at end stage in wind farms. The perfect design of NN model is important for challenging other not so accurate models. The data required for inputs are wind speed, wind direction, and temperature. The higher valued collected data tend to suppress the influence of smaller variable during training. To overcome this problem, the minmax normalization technique which enhances the accuracy of ANN model is used. Therefore, data are scaled within the range
The realtime data was collected from Suzlon Energy Ltd., India Wind Farm for a period from April 2011 to December 2012. The inputs are temperature, wind vane direction from true north, and wind speed in anemometer. The height of wind farm tower is 65 m. The predicted wind speed is considered as an output of the model. The number of samples taken to develop a proposed model is 10000.
The parameters considered as input to the NN model are shown in Table
Input parameters that applied to the proposed model.
S. no.  Input parameters  Units  Range of the parameters 

1  Temperature  Degree. Celsius  24–36 
2  Wind direction  Degree  1–350 
3  Wind speed  m/s  1–16 
Collected sample inputs from wind farm.
Temp 
Wind vane direction from true north (degree)  Wind speed (m/sec) 

26.4  285.5  8.9 
26.4  286.9  7.6 
25.9  285.5  8.6 
25.9  284.1  8.9 
31.9  302.7  3 
25.9  285.5  8.1 
25.8  282.7  7.9 
33.8  307.4  6.7 
25.8  281.2  7.9 
25.9  282.7  7.9 
25.9  282.7  8.4 
25.8  282.7  7.9 
The normalization of data is essential as the variables of different units. The data are scaled within the range of 0 to 1. The scaling is carried out to improve accuracy of subsequent numeric computation and obtain better output. The minmax technique is used. The advantage is preserving exactly all relationships in the data, and it does not introduce bias. The normalization of data is obtained by the following transformation (
Setup parameter includes epochs and dimensions. The training can be learned from the past data after normalization. The dimensions like number of input, hidden, and output neuron are to be designed. The three input parameters are temperature, wind direction and wind speed. The number of hidden layer is one. The number of hidden neurons is to be fixed based on proposed criteria. The input is transmitted through the hidden layer that multiplies weight by hyperbolic sigmoid function. The network learns function based on current input plus record of previous state. Further, this output is transmitted through second connection multiplied with weight by purelin function. As a result of training the network, past information is reflected to Elman network. The stopping criteria are reached until the minimum error. The parameters used for design of Elman network are shown in Table
Designed parameters of Elman network.
Elman network 

Output neuron = 1 ( 
No. of hidden layers = 1 
Input neurons = 3 ( 
No. of epochs = 2000 
Threshold = 1 
For the proposed model, 101 various criteria were examined to estimate training process and errors in Elman network. The input neuron is taken into account for all criteria. It is tested on convergence theorem. Convergence is changing infinite into finite sequence. All chosen criteria are satisfied with the convergence theorem. Initially, apply the chosen criteria to the Elman network for the development of proposed model. Then, train the neural network and compute statistical errors. The result with the minimum estimated error is determined as the fixation of hidden neuron. The statistical errors are formulated in (
The collected data is divided into training and testing of network. The training data was used to develop models of wind speed prediction, while testing data was used to validate performance of models from training data. 7000 data was used for training, and 3000 data was used for testing data. The training can be learned from past data after normalization. The testing data was used to evaluate the performance of network. MSE, MRE, and MAE are used as the criteria for measuring performance. Based on these criteria, the results show that proposed model can give better performance. The selection of proposed criteria is based on the lowest errors. These proposed criteria are applied to Elman network for wind speed prediction. The network checks whether performance is accepted, otherwise goes to next criteria, then train and test performance of network. The errors, value is calculated for each criterion. To perform analysis of NN model, 101 cases with various hidden neuron are examined to estimate learning and generalization errors. The result with minimal error is determined as the best for selection of neurons in hidden layer.
Based on the discussion on convergence theorem in the Appendix, the proof for the selection criteria is established henceforth.
Lemma
Suppose a sequence
It has limit
The proof based on Lemma
According to convergence theorem, parameter converges to finite value:
If sequence has limit, then it is a convergent sequence.
The considered 101 various criteria for fixing the number of hidden neuron with statistical errors are established in Table
Statistical analysis of various criteria for fixing number of hidden neurons in Elman network.
Considered criteria for fixing number of hidden neurons  no. of hidden neurons  MSE  MRE  MAE 


6  0.1329  0.0158  0.1279 

46  0.0473  0.0106  0.0858 

12  0.0783  0.0122  0.099 

31  0.0399  0.0117  0.0944 

52  0.0661  0.0106  0.0862 

43  0.0526  0.0113  0.0917 

76  0.0093  0.0084  0.0684 

26  0.083  0.0135  0.1097 

77  0.0351  0.008  0.065 

14  0.0361  0.0116  0.094 

27  0.0388  0.086  0.0701 

62  0.0282  0.0055  0.0444 

33  0.0446  0.0096  0.0778 

53  0.0447  0.0097  0.0782 

1  0.1812  0.0239  0.1937 

75  0.017  0.0057  0.0462 

25  0.0299  0.0105  0.0853 

10  0.1549  0.0153  0.1238 

34  0.0414  0.0105  0.0854 

81  0.0285  0.0065  0.0526 

101  0.0697  0.0069  0.0561 

45  0.0541  0.01  0.0813 

28  0.038  0.0112  0.0904 

32  0.028  0.0105  0.0848 

24  0.0268  0.009  0.0727 

54  0.0155  0.0086  0.0695 

44  0.0599  0.0112  0.0907 

63  0.0324  0.0094  0.0758 

74  0.0208  0.0109  0.0881 

13  0.1688  0.0167  0.1349 

64  0.0359  0.0133  0.1079 

100  0.0194  0.0058  0.0472 

55  0.0618  0.0166  0.1342 

2  0.217  0.028  0.2272 

29  0.0457  0.008  0.0651 

11  0.0547  0.0118  0.0959 

65  0.0232  0.0098  0.0798 

66  0.0155  0.006  0.0487 

78  0.0171  0.0086  0.0697 

35  0.1005  0.0138  0.116 

15  0.0838  0.0109  0.088 

82  0.0293  0.0061  0.0497 

96  0.0295  0.0106  0.0859 

79  0.072  0.009  0.0731 

16  0.0588  0.0115  0.0934 

30  0.0831  0.0115  0.0935 

67  0.037  0.0079  0.0636 

47  0.0527  0.0097  0.0784 

56  0.0157  0.0093  0.0755 

36  0.0188  0.0117  0.0948 

86  0.0236  0.0132  0.1066 

3  0.1426  0.0218  0.1763 

87  0.0118  0.0064  0.0519 

97  0.0319  0.01  0.0813 

5  0.1339  0.0208  0.1685 

57  0.021  0.0078  0.0629 

68  0.0249  0.0081  0.0654 

17  0.039  0.0114  0.0925 

80  0.0558  0.0098  0.0797 

58  0.0283  0.0079  0.064 

69  0.1152  0.0127  0.1028 

18  0.0437  0.0094  0.0759 

98  0.0283  0.0079  0.064 

42  0.0136  0.0099  0.0806 

61  0.0249  0.0101  0.082 

71  0.0162  0.0077  0.0623 

23  0.1142  0.0156  0.1264 

8  0.0894  0.0142  0.1149 

89  0.0202  0.01  0.0808 

40  0.0163  0.0102  0.083 

50  0.0184  0.0092  0.0744 

90  0.0188  0.0049  0.0395 

41  0.0329  0.0118  0.0957 

91  0.0405  0.0071  0.0573 

94  0.0258  0.0127  0.1028 

22  0.049  0.0088  0.0713 

4  0.0723  0.0118  0.0954 

51  0.0278  0.009  0.0728 

60  0.035  0.0106  0.0861 

37  0.0198  0.0078  0.0633 

88  0.0336  0.008  0.0648 

21  0.1008  0.0134  0.1089 






49  0.0636  0.0095  0.077 

92  0.0332  0.011  0.089 

9  0.1091  0.0123  0.1 

84  0.0169  0.009  0.0727 

95  0.0103  0.0064  0.0517 

85  0.0293  0.0104  0.0842 

72  0.0209  0.0091  0.0739 

38  0.0735  0.0101  0.0815 

19  0.0771  0.0125  0.101 

59  0.0422  0.0101  0.0815 

70  0.0366  0.0083  0.067 

7  0.0727  0.0161  0.1301 

48  0.0183  0.0118  0.0958 

83  0.0407  0.0092  0.0747 

73  0.0493  0.0089  0.0721 

20  0.142  0.0157  0.1271 

93  0.0133  0.0093  0.0753 

99  0.0377  0.012  0.0971 
The actual and predicted wind speeds observed based on proposed model is shown in Figure
Actual/Predicted output waveform obtained from proposed model.
Several researchers proposed many approaches to fix the number of hidden neurons in neural network. The approaches can be classified into constructive and pruning approaches. The constructive approach starts with undersized network and then adds additional hidden neuron [
The salient points of the proposed approach are discussed here. The result with minimum error is determined as best solution for fixing hidden neurons. Simulation results are showing that predicted wind speed is in good agreement with the experimental measured values. Initially realtime data are divided into training and testing set. The training set performs in neural network learning, and testing set performs to estimate the error. The testing performance stops improving as the
Performance analysis of various approaches in existing and proposed models.
S. no.  Various methods  Year  Number of hidden neurons  MSE 

1 
Li et al. method [ 
1995 

0.0399 
2 
Tamura and Tateishi method [ 
1997 

0.217 
3 
Fujita method [ 
1998 

0.0723 
4 
Zhang et al. method [ 
2003 

0.217 
5 
Jinchuan and Xinzhe method [ 
2008 

0.0299 
6 
Xu and Chen method [ 
2008 

0.0727 
7 
Shibata and Ikeda method [ 
2009 

0.1812 
8 
Hunter et al. method [ 
2012 

0.0727 
9  Proposed approach 


In this paper, a survey has been made on the design of neural networks for fixing the number of hidden neurons. The proposed model was introduced and tested with realtime wind data. The results are compared with various statistical errors. The proposed approach aimed at implementing the selection of proper number of hidden neurons in Elman network for wind speed prediction in renewable energy systems. The better performance is also analyzed using statistical errors. The following conclusions were obtained.
Reviewing the methods to fix hidden neurons in neural networks for the past 20 years.
Selecting number of hidden neurons thus providing better framework for designing proposed Elman network.
Reduction of errors made by Elman network.
Predicting accurate wind speed in renewable energy systems.
Improving stability and accuracy of network.
Consider various criteria
A convergent sequence has a limit.
Every convergent sequence is bounded.
Every bounded point has a limit point.
A necessary condition for the convergence sequence is that it is bounded and has limit.
An oscillatory sequence is not convergent, that is, divergent sequence.
A network is stable meaning no there is change occurring in the state of the network regardless of the operation. An important property of the NN model is that it always converges to a stable state. The convergence is important in optimization problem in real time since it prevents a network from the risk of getting stuck at some local minima. Due to the presence of discontinuities in model, the convergence of sequence infinite has been established in convergence theorem. The properties of convergence are used in the design of realtime neural optimization solvers.
Discuss convergence of the following sequence.