Hopﬁeld Neural Networks with Unbounded Monotone Activation Functions

For the Hopﬁeld Neural Network problem we consider unbounded monotone nondecreasing activation functions. We prove convergence to zero in an exponential manner provided that we start with su ﬃ ciently small initial data.


Introduction
Of concern is the following system: where a i (t) ≥ 0, b i j (t),c i (t), i, j = 1, . . . , m are continuous functions, and f j are the activation functions which will be assumed continuous and bounded by some nondecreasing (and possibly unbounded functions). This system appears in Neural Network theory [1,2]. As is well-known, Neural Networks are an important tool in business intelligence. Their architecture differs from the one of standard computers in that it consists of a large number of processors (neurons) with high connections between them. In contrast to computers with a single processor, (Artificial) Neural Networks perform their computations in parallel.
Just as the human brain, the neurons receive weighted signals from the neurons in the input layer, sum up these inputs and test against a threshold value. Then they decide to fire or not.
The applications are numerous, we may cite few: modelling soil behavior, design of tunnels, image processing, graph flow, data deconvolution, energy demand forecasting, ecosystem evaluation, scheduling optimization, targeted marketing, medical diagnosis, time series analysis, and stock market.
Neural Networks are able to analyze and evaluate many phenomena in real world business as well as in industry. Some of their advantages over the conventional computers are forecasting, strategy planning, and predicting many phenomena.
Different methods have been used by many authors to study the well-posedness and the asymptotic behavior of solutions [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20]. In particular, a lot of efforts are devoted in improving the set of conditions on the different coefficients involved in the system as well as the class of activation functions. Regarding the latter issue, the early assumptions of boundedness, monotonicity, and differentiability have been all relaxed to merely a global Lipschitz condition. Since then, it seems that, this assumption has not been weakened further considerably although there is a great need for that [21]. A slightly weaker condition: x * i is the equilibrium, has been used in [22][23][24] (see also [25][26][27]).
Here we assume that the activation functions f j are bounded by continuous monotone nondecreasing functions g j , that is, 2 Advances in Artificial Neural Systems The functions g j are not necessarily Lipschitz continuous and they may be unbounded (like power type functions with powers bigger than one). We can also consider activation functions with discrete delays as is explained below. We prove that, for sufficiently small initial data, solutions decay to zero exponentially.
The local existence and existence of equilibria is standard (see, the Gronwall-type Lemma 1 below) and the global existence follows from the estimation in our theorem below. However, the uniqueness of the equilibrium is not trivial.
Here, as we are concerned with the convergence to zero rather than stability of equilibrium, the uniqueness of equilibrium is put aside.
The next section contains the statement and proof of our result as well as a crucial lemma we will be using.

Exponential Convergence
In this section it is proved that solutions converge to zero in an exponential manner when the activation functions are (or bounded by) continuous nondecreasing and unbounded functions. To this end we need a lemma due to Bainov and Simeonov [3].
Let I ⊂ R and let g 1 , g 2 :

Lemma 1. Let a(t) be a positive continuous function in
. . , n are nonnegative continuous functions for α ≤ s ≤ t < β which are nondecreasing in t for any fixed s, g j (u), j = 1, . . . , n are nondecreasing continuous functions in R + , with g j (u) > 0 for u > 0 and u(t) is a nonnegative continuous functions in J. If g 1 ∝ g 2 ∝ · · · ∝ g n in (0, ∞), then the inequality implies that where ω 0 (t) := sup 0≤s≤t a(s), and β 0 is chosen so that the functions ω j (t), For the statement of our theorem we will need the following notation: where Proof. From (1) and our assumption on f j we see that or where D + denotes the right Dini derivative. Hence Advances in Artificial Neural Systems 3 In virtue of (13) we derive and thereafter (see [28]) where Applying Lemma 1 we obtain the existence of β 0 such that where ω 0 (t), ω j (t), and G j (u), j = 1, . . . , m are as defined in (7)-(9).

Remark 3.
To have global existence we need β 0 = ∞ and this is possible when Remark 4. Assuming that ω n (t) grows up at most polynomially, we see that the rate is exponential.
Remark 5. Note here that our assumptions in the previous remarks involve a smallness condition on the initial data.

Applications
Using Kirchhoff 's law, Hopfield demonstrated that electrical circuits could behave as a small Neural Network. His original system has the form: where C i > 0: Capacity, R i > 0: Resistance, I i : Bias (external action on the ith neuron), u i : Input (voltage) of the ith neuron, v i : Output of the ith neuron, T i j : The coupling constants of the jth neuron with the ith neuron, and g j (u j ): Activation functions. T i j are called elements of the weight matrix or connection matrix. This matrix describes the strength of connection between neurons. The expression 1/R i is sometimes called the feedback factor.
The functions g j (u j ) are nonlinear functions characterizing the response of the ith neuron to changes in its state. Typical activation functions are the "Step function", the "Sign function", the "Gaussian" function, the "Hyperbolic function", and the "Exponential type function". However, it has been established that many other activation functions arise in practice which are not of these forms. Therefore there is a need to enlarge these classes of functions to more general ones.
In Neural Network Theory researchers are rather interested in designing models which are globally asymptotically stable. That is, the models must have a unique equilibrium which attracts all the solutions. Of course the rate of convergence is extremely important and it is preferable to have an exponential convergence rate. In the present work (for the case of variable coefficients) we prove that if solutions start close enough to zero then they will be attracted by zero. Our theorem shows that solutions remain bounded by as long as t < β 0 defined as a bound for the interval of existence of the ω j 's (see (8)). In Remark 3 we gave a sufficient condition ensuring the existence of the ω j 's for all time. That is conditions for which β 0 = +∞. It follows then that, under these conditions, the states actually converge to zero as t goes to infinity with an exponential rate in case ω m (t) does not grow too fast and t 0 a(s)ds → ∞ as t → ∞. The example below represents a possible practical situation for which our argument applies. Again we establish a sufficient explicit condition leading to exponential convergence to zero provided that the initial data are small enough. Example 6. Consider the special (but common) functions g j (x) = x ni , n i > 1, i = 1, . . . , m. The order g 1 ∝ g 2 ∝ · · · ∝ g m means n 1 ≤ n 2 ≤ · · · ≤ n m . Clearly, in this , and for t > 0 4 Advances in Artificial Neural Systems The value β 0 will be the largest value of t for which for all j = 1, . . . , m. As we are interested in the long time behavior of solutions it is necessary that these conditions hold for all t. Our theorem then implies that solutions are bounded by the expression which provides us with an exponential decay under some fairly reasonable assumptions.

Discrete Delays.
The case where we have discrete delays in the activation functions, that is, where τ i j are different finite delays, can be treated similarly. We use the following functional to get rid of the delayed terms and replace them by terms without delays.