Horseshoe Chaos in a 3D Neural Network with Different Activation Functions

This paper studies a small neural network with three neurons. First, the activation function takes the sign function. Although the network is a simple hybrid system with all subsystems being exponentially stable, we find that it can exhibit very complex dynamics such as limit cycles and chaos. Since the sign function is a limit case of sigmoidal functions, we find that chaos robustly exists with some different activation functions, which implies that such chaos in this network is more related to its weight matrix than the type of activation functions. For chaos, we present a rigorous computer-assisted study by virtue of topological horseshoe theory.


Introduction
Since substantial evidence of chaos is found in biological studies of natural neuronal systems, researchers have realized that chaos is much helpful for neural networks escaping the local minima and may play an essential role in the storage and retrieval of information [1][2][3].Thus, a thorough investigation on chaotic dynamics of neural networks is significant for neural networks studies, which has become a popular research field in recent decades.A lot of artificial neural networks have been proposed in order to realize chaotic and hyperchaotic attractors [4][5][6][7][8][9][10][11][12].
Generally, neural networks in real world have very high dimension, which is too hard to study.Fortunately, research in anatomy and physiology shows that neurons in biological brains are grouped together into functional circuits [13,14].This implies that a first step before studying chaos in highdimensional neural networks should be to have detailed investigations on chaos in low-dimensional networks with only a few nerons [15][16][17][18][19].
The nonlinearity of neural systems usually comes from the activation functions, which is the reason causing chaos.There are many types of sigmoid functions used in literature, such as the hyperbolic tangent function, piecewise linear functions, the Logistic (sigmoid) function, and the sign functions.So, we are interested in whether chaos in a neural network can take place with all these functions or whether chaos can take place for any type sigmoidal activation function.
In order to answer the two questions, this paper will take a limit of the sigmoidal functions by zooming out the input scale and study a small Hopfield neural network (HNN) with hard switches.Such sign function is not only extremely easy to implement, but also of dynamical and biological significance in gene regulatory networks [4,5].An interesting phenomenon we find in this paper is that the HNN can demonstrate chaos, although it is a switching system that only consisted of stable subsystems; such chaos still exists even when we replace the sign function with many other activation functions.
The following paper is organized as follows.Section 2 presents the HNN and demonstrates its chaotic behavior with different activation functions; Section 3 first recalls some theoretical results of topological horseshoe and then presents computer-assisted proof of the existence of chaos; Section 4 draws conclusions.

Chaos in the Small Network with Different Activation Functions
The Hopfield neural network is described by [20] where   is a sigmoidal activation function and  = (  ) is an  ×  matrix, called weight matrix or connection matrix describing the strength of connections between neurons.
In this paper, we only consider three neurons, that is,  = 3 and take   = 0, then the small network can be written with the following equation in vector form: where  = [ 1 , where  is a parameter.Then we can easily solve the isolated equilibrium points of (2) while the parameter  from −∞ to +∞, as illustrated in Table 1.Since ( 2) is symmetric with respect to the origin, its equilibria appears in pairs, and the origin is always an equilibrium.Numerical computation suggests that the origin is always unstable.Since  only takes value of −1, 0, and 1, it is easy to see from the equation that the other equilibria are always exponentially stable.
To explore complex dynamics in (2), we simulate the system with  < 0.61 then numerically find that there are limit circles and strange attractors, as illustrated with the bifurcation plot in Figure 1.The dynamics relies greatly on its initial condition; that is, different initial value may result in different kind of dynamics.During our simulation, we chose (0) = [0.2,0.3, −0.4]  .The bifurcation diagram indicates that the system (2) is very likely to be chaotic when  is near to zero.
After careful numerical computation, we obtain an invariant set called attractor in the sense that almost every trajectory with initial point near this set tends to this set, while this set contains a trajectory, that is, dense in it, as illustrated in Figure 2 for  = 0.It seems a chaotic attractor.In the next section, we will prove that this attractor is indeed chaotic with the topological horseshoe theory.
Since the sign function is a limit of most sigmoidal functions with large-scale input, we in this section will study existence of chaos for the HNN (1) with other sigmoidal activation function.For this purpose, we considerate the following equations: where  is a positive scale factor.Replacing  with  −1 , we have the equivalent system  So for any type sigmoidal activation function () with its output range from −1 to 1, if  is great enough, () will be sufficiently close to the discontinuous sign function.The robustness of (2) suggests that (4) could exhibit chaos with the same weight matrix (3).Taking the consideration of the bias, (1) may exhibit chaos for any type sigmoidal activation function when  = 3.
To illustrate this fact, we will give examples with a hyperbolic tangent function, a Logistic function, a piecewise linear function, and a very complicated function.
Case 1.  takes the hyperbolic tangent function.Let () = tanh().In order to make the neural system (5) chaos, we take  = 60.Then we get an attractor shown in Figure 3.The Lyapunov exponents are 0.156, 0.000, and −1.4273.The first one is positive showing that the attractor is most likely chaotic.
The three Lyapunov exponents are 0.169, 0.000, and −1.450.The first one is positive suggesting the attractor is chaotic.
Case 4.  is a complicated piecewise linear function.For the system (4) and the weight matrix (3) at  = 0, we take a more complicated (  ) randomly as shown in Figure 5.Let  = 100, a chaotic attractor appears as shown in Figure 6.Since the activation function is too complicate, it is not easy to compute the Lyapunov exponent of the attractor in Figure 6 with enough accuracy; so, in next section we will prove that this attractor is indeed chaotic with the topological horseshoe theory.
It is clear to see from Figures 3 to 6 that although their activation functions are much different, the chaotic attractor exists in all four cases and they all look similar.All the above four cases suggest that the existence of chaos in this neural network is robust on the type of activation functions.In another word, the complex dynamics of a small neural network should be more related to its weight matrix.

Computer-Assisted Proof of Chaos
The existence of a topological horseshoe is recognized as one of the most important signatures of chaos.With the horseshoe theory, one can not only prove chaos rigorously but also reveal the mechanism of chaos with its invariant set.So we will present computer-assisted proof of chaos in the HNN in this section.First, let us recall a theorem on topological horseshoe and the m-shift and then present our main results.
Let  be a metric space,  is a compact subset of , and  :  →  is map satisfying the assumption that there exist  mutually disjoint compact subsets  1 ,  2 , . . .,   of , the restriction of  to each   , that is,  |   is continuous.Definition 1.Let  be a compact subset of , such that for each 1 ≤  ≤ ,   =  ∩   is nonempty and compact; then  is called a connection with respect to  1 ,  2 , . . .,   .Let  be a family of connections s with respect to  1 ,  2 , . . .,   satisfying property:  ∈  ⇒ (  ) ∈ .Then  is said to be -connected family with respect to  1 ,  2 , . . .,   .Theorem 2. Suppose that there exists a -connected family with respect to  1 ,  2 , . . .,   .Then there exists a compact invariant set  ⊂ , such that  is semiconjugate to -shift.
In this theorem, the -shift is also called Bernoulli shift sometimes, denoted by  : Σ  → Σ  , where Σ  is the collection of all bi-infinite sequences  = {. . .,  − , . . .,  −1 ;  0 ,  1 , . . .,   , . ..} , and the shift map  is defined as It is well known that Σ  is a Cantor set, which is compact, totally disconnected, and perfect.As a dynamical system defined on Σ  ,  has a countable infinity of periodic orbits consisting of orbits of all periods, an uncountable infinity of aperiodic orbits, and a dense orbit.A direct consequence of these three properties is that the dynamics generated by the shift map is sensitive to initial conditions.Since  is topologically semiconjugate to , which means that there exists a continuous surjection ℎ : Σ  →  such that  ∘ ℎ = ℎ ∘ ,  must be also sensitive to initial conditions.Mathematically, the complexity of the system  can be measured by its topological entropy, which roughly means the exponential growth rate of the number of distinguishable orbits as time advances.As another result of the semiconjugate, the topological entropy of , denoted by ent(), is not less then .When  > 1, ent() > 0, therefore the system is chaotic.For more details of the above symbolic dynamics and horseshoe theory, we refer the reader to [21][22][23][24].
In what follows we will study existence of horseshoes embedded in the attractor in Figure 2.For this purpose, we will utilize the technique of cross section and the corresponding Poincaré map.Consider the section plane  :  3 = 0.2, as shown in Figure 2. The Poincaré map  :  →  is chosen as follows.For each  ∈ , () is taken to be the second return point in  under the flow with the initial condition .
To find the horseshoe, we use the efficient method proposed in [21,25] which has been implemented with a MATLAB toolbox called "a toolbox for finding horseshoes in 2D map" (download: http://www.mathworks.com/matlabcentral/fileexchange/14075).The method is so powerful that it has been successfully applied in a number of chaotic systems [26][27][28][29], a fractional-order system [30],even a hyperchaotic system [31].
After many attempts, we carefully pick two quadrangles By means of interval analysis, we compute the Poincaré map on | 1  1  1  1 | and | 2  2  2  2 | and then have the following statement.

Theorem 3. For the Poincaré map 𝜋 corresponding to the cross sections |𝐴
Proof.To prove this statement, we will find two disjoint compact subsets  1 and  2 of , such that the existence of a -connected family can be easily derived.
The first subset  1 takes | 1  1  1  1 | as shown in Figure 7, and the Poincaré map sends this subset to its image showing that  1  1 is mapped left of the side  1  1 and  Generally, (2) can be regarded as a switching system consisting of eight very simple continuous subsystems and 12 quarter planes called switching planes; for detail, see [5].In this system, every trajectory in the neighborhood of the attractor transversely intersects with the switching plane, and the Poincaré map  can be regarded as a composition of a series of continuous submaps by the subsystems.Since each subsystem is linear, it is easy to prove that for the quadrangle region It is easy to see from Theorem 3 that the entropy of  is not less than log 2, so the attractor in Figure 2 is a chaotic attractor indeed.
So we can prove the existence of chaos in the attractor shown in Figure 6 The images of | From the topological horseshoe theory we have ent() ⩾ log 2 after similar arguments.The positive entropy suggests that the attractor in Figure 6 is chaotic indeed.

Conclusions
In this paper, we have studied a 3D Hopfield neural network with the sign activation function.Computer simulation shows that this HNN can exhibit chaotic attractors and limit cycles with respect to .In order to verify the chaotic behavior, we present a computer-assisted verification for the existence of horseshoes imbedded in this system.We also show evidence that chaos could possibly be exhibited by the HNN with any type of sigmoidal activation function.In another word, such chaos should be more related to its weight matrix than the type of activation functions.In addition, since the HNN is a switching system only consisting of stable subsystems, this fact suggests that the dynamics of a hybrid system could be much more complex than we used to think.This may be of interest to researchers of neural networks, nonlinear dynamics, and so on.