^{1}

^{2}

^{2}

^{1}

^{2}

This paper presents an improved model of echo state networks (ESNs) and gives the definitions of energy consumption, energy efficiency, etc. We verify the existence of redundant output synaptic connections by numerical simulations. We investigate the relationships among energy consumption, prediction step, and the sparsity of ESN. At the same time, the energy efficiency and the prediction steps are found to present the same variation trend when silencing different synapses. Thus, we propose a computationally efficient method to locate redundant output synapses based on energy efficiency of ESN. We find that the neuron states of redundant synapses can be linearly represented by the states of other neurons. We investigate the contributions of redundant and core output synapses to the performance of network prediction. For the prediction task of chaotic time series, the predictive performance of ESN is improved about hundreds of steps by silencing redundant synapses.

Artificial Intelligence is a branch of science and it studies and develops theories, methods, techniques, and applications for simulating and extending human intelligence [

As a kind of artificial recurrent neural network, the echo state network was proposed by Jaeger et al. [

In the classical chaotic time series prediction, the precision of ESN improved several thousand times compared with traditional methods which solved the bottleneck problem in the previous research of neural network. ESN has rapidly become the research focus due to its excellent predictive performance in various areas [

Compared with the computer, human brain has higher energy efficiency with powerful capacity [

To find out and remove redundant synapses in the output connections, further to improve the predictive performance of ESN, this paper proposes an improved ESN model. The definitions of energy consumption, energy efficiency, etc. were given. We analyze the relationship between energy consumption and sparsity, the relationship between predicted steps and sparsity, and the relationship between energy efficiency and sparsity in ESN and then verify the existence of redundant synapses. We find that silent redundant synapses have little influence on the weights of other synapses. We also study the contributions of redundant and core output synapses to prediction performance in the improved ESN and discover that the contribution of redundant output synapses to predictive performance is close to zero. At the same time, the energy efficiency and the predictive steps are found to have the same variation trend when silencing different synapses. Thus, we propose a method to locate redundant output synapses based on energy efficiency of ESN system, which is computationally efficient. Numerical simulations of different chaotic systems are presented to demonstrate the feasibility and the effectiveness of the proposed approach in large-scale ESN. Compared to a fully connected network, the predictive performance is improved about hundreds of steps by silencing redundant synapses for the task of chaotic time series prediction.

An echo state network is an artificial recurrent neural network. It can maintain an ongoing activation even in the absence of input and thus it can exhibit dynamic memory. As shown in Figure

Schemas of the traditional ESN approach and our improved ESN approach. (a) The traditional ESN approach. Input signals

At the sampling time

Synapses are the sites by which neurons can contact with each other and transmit information. The long-term transmission function of synapses can be depressed and potentiated. It has a particularly important influence on the advanced function of the brain while maintaining the computation, memory, and learning powers of the brain. It is a difficult problem to accurately find the redundant output synaptic connections and remove them in order to improve the network function in artificial neural networks.

To solve the above problem, the traditional model of ESN is improved. As shown in Figure

We explain the improved ESN approach based on a prediction task of chaotic time series. The Mackey-Glass system (MGS) is a standard benchmark system for the study of time series prediction. It generates an irregular time series. The prediction task has two steps: (

At the sampling time

Since the reservoir can maintain an ongoing activation in the absence of input and it can exhibit dynamic memory. It is not necessary for the state update of the reservoir to have the input signal

In the prediction of the classical chaotic sequence, the precision of ESN is several thousand times than those of traditional methods. So we select four typical chaotic systems determined by differential equations to generate the teacher sequence.

The Mackey-Glass system (MGS) is a standard benchmark system in the research on time series prediction. It generates a subtly irregular time sequence. Almost every available technique for nonlinear system modeling and prediction has been tested on the MGS system.

A teacher sequence

The initialization process of Echo State Network by MGS is given as follows. We create a random partially connected ESN with 1000 neurons, and the sparsity of the reservoir is 1%. We generate a

The Lorenz system is governed by the following three-dimensional differential equation

The initialization process of Echo State Network by Lorenz system is given as follows. We create a random network with 1000 neurons. Its spectral radius is 1.6, and its sparsity is 0.02. Output feedback connection weights are sampled from an uniform distribution over

The three-dimensional differential equation of Rössler System is given as follows:

The initialization process of Echo State Network by Rössler system is given as follows. This process is the same as the initialization setting of Lorenz system. We only should adjust the sparsity of ESN to be 0.01 to adapt to the Rössler chaotic system.

The equation of Chen chaotic system is presented as

The initialization process of Echo State Network by Chen system is given as follows. Similar with the network initialization process of Mackey-Glass system, we create a random network with 1000 neurons. The spectral radius of the network is 1.7 and its sparsity is 0.02. Output feedback connection weights are sampled from the uniform distribution over

In this paper, we present an improved model of ESNs and use many physical quantities. Here we give the definitions of these physical quantities.

Predicted step

Sparsity is the small connection probability between the neurons in the reservoir or between the neurons in the reservoir and the output neurons, and the sparsity is given by

Energy consumption

Energy efficiency

Contribution

Now we consider the experimental results. In Figure

The experimental results verify that there are redundant synapses in the output connections of ESN. All the experimental results are averaged over 100 independent trials per sparsity. (a) Curves about the relationship between the predicted steps and the sparsity of ESN. The corresponding teacher sequences are generated by three different chaotic systems (Mackey-Glass system, Lorenz system, and Rössler system), respectively. (b) Curves about the effects of the sparsity on the energy consumption

In order to further verify the existence of redundant output synapses in ESN, we analyze the influences of the sparsity of output synapses on energy consumption

Figure

Based on all the experimental results in Figures

In the above subsection, we analyzed the existence of redundant output synapses in ESN. Now we verify the existence of redundant output synapses in small-scale ESN. To facilitate the observation about the variations of energy efficiency and the predicted steps, we randomly create an ESN with 10 neurons. Since the predictive performance of small-scale ESN is limited, we create the periodical sequence to adapt to the performance of ESN. A 500-step teacher sequence is generated from the equation ^{th}, and 7th synapses are silenced, respectively, the energy efficiency and the predicted steps of ESN are close to 0; i.e., these synapses have great influence on the predictive performance of ESN. It means 1st, 5^{th}, and 7th synapses are core synapses. At the same time, we discover from Figure

Experimental results to verify the existence of redundant output synapses in the small-scale ESN. (a) Curves of the energy efficiency (blue solid line) and the predicted steps (red dotted line) when we silence a synapse each time. The abscissa representing the ^{th}, and 7th synapses to predictive performance of ESN as time increases. The abscissa is the time, the ordinate represents the energy contribution of synapses to the predictive performance of ESN, and the curves with different colors and different marks represent energy contributions of 1st, 2nd, 4th, 5^{th}, and 7th synapses to predictive performances of ESN, respectively. (c) Curves of all synaptic weights when the 2nd and 4th synapses are silenced, respectively, and all the synapses are activated. The abscissa represents the ^{th}, and 7th synapses are silenced, respectively, and all the synapses are activated.

Figure

Figure

Figure ^{th}, and 7th synapses are not redundant synapses and they cannot be removed. From the above experimental results, we can see the existence of redundant output synapses in small-scale ESN.

From the perspective of mathematics, now we begin to analyze the reason for the existence of redundant output synaptic connections. Without loss of generality, we take the 4th synapse for example to illustrate this.

We can consider

The form of (

The states of the 2nd or 4th neuron at all moments can be linearly expressed by the states of other neurons in the state matrix. Other vector elements are independent and they cannot be linearly expressed by other appropriate elements. Similarly for the states of other redundant synapses, there exists the characteristic of linear representation in the long-term prediction.

Here we analyze the reason for the existence of redundant output synaptic connections by table data. We consider the circumstance that the 2nd or 4th synapse is silenced, respectively, and there are 10 neurons in reservoir. In Table

Weights of output synapses when all output synapses are activated and the 2nd or 4th synapse is silenced, respectively. Here are the ratio between the weights of synapses when the 2nd or 4th synapse is silenced and the weights when all synapses are activated.

neurons | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
---|---|---|---|---|---|---|---|---|---|---|

Weight when all are activated | 1130.428 | 134.006 | −308.004 | −0.545 | −2375.857 | 346.081 | −3668.496 | 24.762 | −135.418 | −301.811 |

| ||||||||||

Weight when 2th synapse is silenced | 1119.316 | 0 | −313.068 | 6.100 | −2349.637 | 343.812 | −3680.391 | 20.728 | −82.074 | −231.207 |

| ||||||||||

| 0.990 | 0 | 1.016 | −11.183 | 0.988 | 0.993 | 1.003 | 0.837 | 0.606 | 0.766 |

| ||||||||||

Weight when 4th synapse is silenced | 1130.079 | 131.702 | −305.079 | 0 | −2377.723 | 348.913 | −3664.580 | 25.212 | −135.811 | −300.373 |

| ||||||||||

| 0.999 | 0.982 | 0.990 | 0 | 1 | 1.008 | 0.998 | 1.018 | 1.002 | 0.995 |

From Table

Based on the analytical results, we find that there are redundant output connections in ESN. Locating redundant synapses in ESN and then removing them are helpful to increase the predictive steps of ESN. Therefore, it is necessary to propose a method to find or locate redundant output synapses. According to the analytical results, the same variation trend is found between the energy efficiency and the predicted steps with the changes of sparsity.

Inspired by this rule, we propose a method to find redundant output synapses. The main mechanism of this method is to use the repeated iterations to search multiple redundant output synapses. If we silence these redundant output synapses, then ESN will have the highest energy efficiency. By iterations, potential redundant output synaptic connections are gradually reduced in ESN, thus the sparse output synaptic weight matrix

The detailed steps to find out redundant output synapses are given as follows.

Ergodic search: silence some output synapse, and record the energy efficiency of ESN after this synapse is silenced.

Repeat Step

Locate redundant output synaptic connections. In the output result of Step

Cycle repeatedly in order to find the optimum. Repeat Steps

In the above section, we analyzed the reason for the existence of redundant output synapses in small-scale ESN. Now we verify the existence of redundant output synapses in large-scale ESN and further verify the effectiveness of the proposed method by analyzing redundant output synapses. We create an ESN with 1000 neurons with reservoir sparsity 0.9% to predict Mackey-Glass system, Lorenz system, Rössler system, and Chen system, respectively. The parameter values of these four chaotic systems are given.

Figures

The variation of predicted steps for ESN with the increment of silent synapse. In each subgraph, there are five curves which represent different connection states of output synapses, where

From Figures

We emphasize the general character of the proposed method because this method can be used by other types of artificial neural networks to find out redundant synapses. Our approach is readily applicable to these situations, such as perception and BP network. It is worth considering whether other neural networks with synaptic connections also have such characteristic. Similarly, we can propose the corresponding method to improve the optimization of network structures and the predictive performance of neural networks based on the study of the energy efficiency of human brain. Furthermore, we can extend our discovery and the proposed method of ESN to the research on the networks of human brain. However, this important aspect has received little attention in the existing studies. The ideas of the proposed method inspired by the research on the energy of networks can be extended to the research processes of many human behaviors such as government service and cooperation layoff. For example, the government achieves interdepartmental data reduction and shares to improve the efficiency of government service by removing redundant data.

In summary, we have analyzed the reason for the existence of redundant output synaptic connections in ESN from the perspectives of mathematical analysis and numerical simulations. This paper presents an improved model of echo state network (ESN) and gives the definitions of energy consumption, energy efficiency, etc. In this paper, the data generated by four different chaotic systems (i.e., Mackey-Glass system, Lorenz system, Rössler system, and Chen system) are selected as teacher sequences to train ESN. We investigate the relationship between the sparsity of output connections and the predicted steps of ESN. We take the prediction of M-G system as example to investigate the relationship among energy consumption, energy efficiency, and the sparsity. And we discover that it is difficult to obtain high energy efficiency and more predicted steps simultaneously. For a small-scale ESN, we investigate the relationship among the predicted steps, the energy efficiency, and silent output synapses by numerical simulations, and we find that the energy efficiency of output synapse and the predicted steps of ESN have the same variation trend when the corresponding output synapses are silenced. We investigate the contributions of redundant and core output synapses to the predictive performance of ESN and find that the energy contribution of redundant synapses to predictive performance is close to 0 in ESN. We investigate the variations of all synaptic weights when redundant and core synapses are silenced and all synapses are activated, and we find that silencing of redundant synapses does not have much influence on the synaptic weights, while silencing of core synapses has a significant influence on other synaptic weights which can lead to lower predictive performance of ESN. Based on the relationship between the energy efficiency and silent synapses, we have presented a general approach to find out the redundant synapses. The advantage of our approach is that redundant synapses can be accurately located in ESN. We have presented numerical results of the predictions for different chaotic systems to demonstrate the feasibility and the effectiveness of the proposed approach in large-scale ESN. With the increment of silent synapse, we give the variation curves of the predicted steps of ESN in five different connected states of output synapses. Compared to a fully connected network, the predicted steps are improved for the prediction of nonlinear time series when we apply the proposed method to remove the redundant synapses in ESN. Finding the redundant synapses in ESN to improve the predictive performance is one of the uttermost important problems in science and engineering and it is of extremely broad interest as well. We hope our work will stimulate further efforts in this challenging area.

The data used to support the findings of this study are included within the article.

The authors declare no conflicts of interest.

This work is supported by the National Key R&D Program of China (Grant Nos. 2016YFB0800604, 2016YFB0800602), the National Natural Science Foundation of China (Grant No. 61573067), and the “13th Five-Year” National Crypto Development Fund of China (Grant No. MMJJ20170122).