Discrete Pseudo-SINR-Balancing Nonlinear Recurrent System

Being inspired by the Hopfield neural networks (Hopfield (1982) and Hopfield and Tank (1985)) and the nonlinear sigmoid power control algorithm for cellular radio systems inUykan andKoivo (2004), in this paper, we present a novel discrete recurrent nonlinear systemand extend the results inUykan (2009), which are for autonomous linear systems, to nonlinear case.Theproposed systemcan be viewed as a discrete-time realization of a recently proposed continuous-time network in Uykan (2013). In this paper, we focus on discrete-time analysis and provide various novel key results concerning the discrete-time dynamics of the proposed system, some of which are as follows: (i) the proposed system is shown to be stable in synchronous and asynchronous work mode in discrete time; (ii) a novel concept called Pseudo-SINR (pseudo-signal-to-interference-noise ratio) is introduced for discrete-time nonlinear systems; (iii) it is shown that when the system states approach an equilibrium point, the instantaneous Pseudo-SINRs are balanced; that is, they are equal to a target value. The simulation results confirm the novel results presented and show the effectiveness of the proposed discrete-time network as applied to various associative memory systems and clustering problems.


Introduction
Artificial neural networks have been an important research area since 1970s.Since then, various biologically inspired neural network models have been developed.Hopfield Neural Networks [1,2] have been one of the most widely used neural networks since the early 1980s whose applications vary from combinatorial optimization (e.g., [3,4]) to image restoration (e.g., [5]) and from various control engineering optimization problems in robotics (e.g., [6]) to associative memory systems (e.g., [7,8]).For a tutorial and further references about Hopfield neural networks, see, for example, [9,10].
In [11], we introduce a novel pseudo-signal-to-interference-noise ratio concept for discrete-time autonomous linear systems.Our main motivation in this paper is to investigate a nonlinear extension of [11].Furthermore, the proposed system can be viewed as a discrete-time realization of a very recently proposed continuous-time network called doublesigmoid continuous-time Hopfield neural network in a brief letter [12].And our investigations in this paper yield various interesting key novel results in discrete time, some of which are as follows: (i) a novel concept called Pseudo-SINR (pseudo-signal-to-interference-noise ratio) is introduced for discrete-time nonlinear systems; (ii) it is shown that when the network approaches to one of its equilibrium points, the instantaneous Pseudo-SINRs are balanced; that is, they are equal to a target value; (iii) the proposed network outperforms its Hopfield neural network counterpart as applied to various associative memory systems and clustering applications.The disadvantage of the proposed network is that it increases the computational burden.
The paper is organized as follows.The proposed recurrent network and its stability features are analyzed in Section 2. Simulation results are presented in Section 3, followed by conclusions in Section 4.

Discrete Pseudo-SINR-Balancing Recurrent Neural Networks
Being inspired by the nonlinear sigmoid power control algorithm for cellular radio systems in [13] and the Hopfield neural networks [2], we propose the following discrete nonlinear recurrent network: x ( + 1) = x () +  () f 1 (−Ax () + W 2 (x ()) + b) , where  represents the iteration step, A, W, and b are defined as in (2), and f  (⋅),  = 1, 2, represents a vectoral mapping from R  to R  .For an -dimensional vector e = [ 1  2 ⋅ ⋅ ⋅   ]  , f  (e) = [  ( 1 )  ( 2 ) ⋅ ⋅ ⋅   (  )]  where   (⋅) is chosen as the sigmoid function; that is, for a real number   , the output is   (  ) =   (1−(2/(1+exp(−    )))), where   > 0,   > 0. We will call the network in (1) as discrete sigmoid-pseudo-SINR-balancing recurrent neural network (D-SP-SNN).The name comes from the fact that the proposed network balances an entity called Pseudo-SINR, as will be seen in the following.In this paper, we choose sigmoid function because it's used both in Hopfield neural network and the power control algorithm in [13].Furthermore, the proposed D-SP-SNN can be viewed as a discrete-time implementation of a very recently proposed continuous-time network called double-sigmoid continuoustime Hopfield neural network in the brief [12].In this paper, we focus on discrete-time analysis and provide various novel key results concerning the discrete-time dynamics of the proposed system: In (2), A shows the self-state-feedback matrix, W with zero diagonal shows the interneurons connection weight matrix, and b is a threshold vector.
The proposed network includes both the sigmoid power control in [13] and the traditional Hopfield neural network (HNN) as its special cases by choosing the  1 (⋅) and  2 (⋅) appropriately.The Euler approximation of the continuoustime HNN is given as Let us call the network in (3) HNN-Euler, which is a special case of the proposed D-SP-SNN in (1).From (1), where () is the step size at time .Let's define the error signal   () as where Then, the performance index is defined as  1 -norm of the error vector in (5) as follows: where In what follows, we examine the evolution of the energy function in (6) in synchronous and asynchronous work modes.Asynchronous mode means that at every iteration step, at most only one state is updated, whereas synchronous mode refers to the case that all the states are updated at every iteration step according to (4).

Proposition 1.
In asynchronous mode of the proposed network D-SP-SNN in (4) with a symmetric matrix W, for a nonzero error vector, the  1 -norm of the error vector in (6) where  2 = 0.5  (5).Writing (5) in vector form for steps  and  + 1 results in Using the error signal definition of ( 5) in (4) gives So, the error signal for state  is obtained using ( 10) and ( 11) as follows: = −   ()  1 (  ()) .
From ( 12) and ( 13 where  1 (⋅) is a sigmoid function, which is lower and upper bounded.Since the sigmoid function  1 (⋅) has the same sign as its argument and  1 (  ) = 0 if and only if   = 0, then it is seen that () can easily be chosen small enough to satisfy |  ()| > ()  | 1 (  ())| according to the parameter   and the slope of sigmoid function  1 (⋅).Above, we examined only the state  and its error signal   ().In what follows, we examine the evolution of the norm of the complete error vector e( + 1) in (10).From the point of view of the  1 norm of the e( + 1), the worst case is that when |  ()| decreases, all other elements |  ()|,  ̸ = , increase.So, using (10), (12), and ( 14 then The  (15), which implies (16).This completes the proof.
Proposition 4. In asynchronous mode, choosing the slope of  2 (⋅) relatively small as compared to  1 (⋅) and choosing   > 0 and () satisfying (8), the D-SP-SNN in (4) with a symmetric matrix W is stable and there exists a finite time constant such that the  1 -norm of the error vector in (6) approaches to an vicinity of the zero as its steady state, where  is a relatively small positive number.If   =  tgt  at the converged point, then it corresponds to a prototype vector as defined above.
(2) x( + 1) = x() if and only if e() = 0; that is, (3) Examining the (11), (12), and ( 13) taking the observations (1) and (2) into account, we conclude that any of the   (),  = 1, . . ., , does not go to infinity and is finite for any .So, the D-SP-SNN in (4) with a symmetric matrix W is stable for the assumptions in Proposition 4. Because there is a finite number of insaturation states (i.e., the number of all possible insaturation state combinations is finite), which is equal to 2  , there exists a finite time constant such that the  1norm of the error vector in (6) approaches to an -vicinity of the zero as its steady state, where  is a relatively small positive number.
From (18), if   =  tgt  at the converged point, then it corresponds to a prototype vector as defined in the previous section, which completes the proof.
Proposition 6.The results in Propositions 1 and 4 for asynchronous mode hold also for synchronous mode.
In synchronous mode, all the states are updated at every step  according to (5).So, from ( 5) Using ( 5) in (34) and writing it elementwise give From ( 34) and ( 35), we obtain which is equal to (15) in Proposition 1 and ( 20) in Proposition 4.
It is well known that the performance of Hopfield network may highly depend on the parameter setting of the weight matrix (e.g., [8]).There are various ways for determining the weight matrix of the Hopfield networks: gradient-descent supervised learning (e.g., [16]), solving linear inequalities (e.g., [17,18] among others), Hebb learning rule [19,20], and so forth.How to design D-SP-SNN is out of the scope of this paper.The methods used for traditional Hopfield NN can also be used for the proposed networks D-SP-SNN.As far as the simulation results in Section 3 are concerned, we determine the matrices A, W, and b by using a Hebb learning-based algorithm [19] presented in Appendix B.

Simulation Results
In the simulation part, we examine the performance of the proposed D-SP-SNN in the area of associative memory systems and clustering problem.In Examples 7 and 8, we present some toy examples one with 8 neurons and one with 16 neurons, respectively, where the desired vectors are orthogonal.Lyapunov function of the HNN at time  is given as In Examples 7 and 8, we use discrete-time HNN just for comparison reasons, which is given by where W is the weight matrix and x() is the state at time , and at most one state is updated at a time.
Example 7. In this example of discrete-time networks, there are 8 neurons.The desired prototype vectors are as follows: The weight matrices A and W and the threshold vector b are obtained as follows by using the outer-product-based design (Hebb-learning [19]) presented in Appendix B and the slopes of sigmoid functions  1 (⋅) and  2 (⋅) are set to  1 = 10,  1 = 10, and  2 = 2,  2 = 1, respectively, and  = 0,  = 0.1: Figure 1 shows the percentages of correctly recovered desired patterns for all possible initial conditions x() ∈ (−1, +1) 8 , for the proposed networks D-SP-SNN as compared to traditional discrete Hopfield network.In the proposed network D-SP-SNN,  1 (⋅) is a sigmoid function.Establishing an analogy to the traditional fixed step 1-bit increase/decrease power control algorithm (e.g.[21,22]), we replace the sigmoid function by the sign function and call corresponding network as fixed-step pseudo-SINR neural network (FSPSNN).For comparison reason its performance is also shown in Figure 1.
As seen from Figure 1 the performance of the proposed network D-SP-SNN is remarkably better than that of the traditional discrete Hopfield network for all Hamming distance cases.The FSP-SNN also considerably outperforms the Hopfield network for 1 and 2 Hamming distance cases while the all the networks perform poorly (less than 20%) at 3-Hamming distance case.
Example 8.The desired prototype vectors are The weight matrices A and W and threshold vector b are obtained as follows by using the outer-product-based design (Hebb-learning [19]) in Appendix B: Figure 2 shows the percentage of correctly recovered desired patterns for all possible initial conditions x() ∈ (−1, +1) 16 , in the proposed D-SP-SNN and FSP-SNN as compared to discrete Hopfield network.
As seen from Figure 2 the performance of the proposed networks D-SP-SNN and FSP-SNN is the same as that of discrete Hopfield Network for 1-Hamming and 2-Hamming   distance cases (%100 for all networks).However, the D-SP-SNN and FSP-SNN give better performance than the discrete Hopfield network does for 3-and 4-Hamming distance cases.
Typical plots for evolution of states in Example 8 by the D-SP-SNN are shown in Figure 3.The evolution of corresponding pseudo-SINRs is given by Figure 4.The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
Evolutions of the Lyapunov function in (37) for the states of Figure 3  Figure 6 shows desired pattern 1, a distorted pattern 1 where the Hamming Distance (HD) is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition.As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while the HNN-Euler fails for the same parameters and weight matrix.
The evolutions of the Lyapunov function in (37) and the norm of the difference between the state vector and equilibrium point for pattern 1 in Figure 6         Figure 11 shows the evolutions of pseudo-SINRs of states in Example 9 for pattern 2 by D-SP-SNN.The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
The evolutions of the norm of the difference between the state vector and equilibrium point for pattern 2 in Figure 8 are shown in Figure 12.As seen from the figure, the proposed D-SP-SNN converges much faster than its HNN-Euler counterpart.
Figure 13 shows desired pattern 3, a distorted pattern 3 where the HD is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition.As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while its HNN-Euler counterpart fails for the same parameters and weight matrix.
The evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point for pattern 3 in Figure 13 are shown in Figure 14.As seen from the figure, (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network, and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix for this example.
Figure 15 shows desired pattern 4, a distorted pattern 4 where the HD is 5, the result of HNN-Euler, and the result of the D-SP-SNN using the distorted pattern as initial condition.As seen from the figure, the proposed D-SP-SNN succeeds to recover the number while its HNN-Euler counterpart fails for the same parameters settings.
The evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point for pattern 4 in Figure 15   problem.Clustering is used in a wide range of applications, such as engineering, biology, marketing, information retrieval, social network analysis, image processing, text mining, finding communities, influencers, and leaders in online or offline social networks.Data clustering is a technique that enables dividing large amounts of data into groups/clusters in an unsupervised manner such that the data points in the same group/cluster are similar and those in different clusters are dissimilar according to some defined similarity criteria.The clustering problem is an NP-complete, and its general solution even for 2-clustering case is not known.It is well known that the clustering problem can be formulated in the form of the Lyapunov function of the HNN.The weight matrix is chosen as the distance matrix of the dataset and is the same for both HNN-Euler and D-SP-SNN.
In what follows, we compare the performance of the proposed D-SP-SNN as compared to its HNN-Euler counterpart as applied to clustering problems for the very same parameter settings.Two-dimensional 16 data points to be bisected are shown in Figure 17.The clustering results are also shown in Figure 17.As seen from the figure, the D-SP-SNN finds the optimum solution for this toy example.HNN-Euler also gives the same solution.
The evolutions of states in the clustering by HNN-Euler and by D-SP-SNN are shown in Figures 18 and 19, respectively.As seen from the figures, the states of the proposed D-SP-SNN converge faster that those of its HNN-Euler counterpart.
The evolutions of psuedo-SINRs of states in the clustering by D-SP-SNN in Example 10 ( = 16) are given by Figure 20.The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point.
The evolutions of Lyapunov function and the norm of the difference between the state vector and equilibrium point in        States for HNN-Euler approx.
States for HNN-Euler approx.
(a)       As seen from the figure, while -means fail to find the optimum clustering solution for this example (for a randomly given initial values), the proposed D-SP-SNN succeeds in finding the optimum solution (for the same initial values).
Figure 23 shows the evolution of pseudo-SINRs of states by the D-SP-SNN.The figure shows that the pseudo-SINRs approach to constant value 1 as states converge to the equilibrium point, as before.
Evolutions of the Lyapunov function and the norm of the difference between the state vector and equilibrium point are shown in Figure 24.The figure confirms the superior convergence speed of the D-SP-SNN as compared to its HNN-Euler counterpart.

Conclusions
In this paper, we present and analyze a discrete recurrent nonlinear system which includes the Hopfield neural networks [1,2] and the nonlinear sigmoid power control algorithm for cellular radio systems in [13], as special cases by properly choosing the functions.This paper extends the results in [11], which are for autonomous linear systems, to nonlinear case.The proposed system can be viewed as a discrete-time realization of a recently proposed continuous-time network in [12].In this paper, we focus on discrete-time analysis and present various novel key results concerning the discretetime dynamics of the proposed system, some of which are as follows: (i) the proposed network is shown to be stable in synchronous and asynchronous work mode in discrete time; (ii) a novel concept called Pseudo-SINR (pseudo-signalto-interference-noise ratio) is introduced for discrete-time nonlinear systems; (iii) it is shown that when the network approaches one of its equilibrium points, the instantaneous Pseudo-SINRs become equal to a constant target value.
The simulation results confirm the novel results (e.g., Pseudo-SINR convergence, etc.) presented and show a superior performance of the proposed network as compared to its Hopfield network counterpart in various associative memory systems and clustering examples.Moreover, the results show that the proposed network minimizes the Lyapunov function of the Hopfield neural networks.The disadvantage of the D-SP-SNN is that it increases the computational burden.where  = 0.5 is the global Lipschitz constant of the sigmoid function.

B. Outer Product-Based Network Design
Let us assume that  desired prototype vectors are orthogonal and each element of a prototype vector is either −1 or +1.
Step 1. Calculate the sum of outer products of the prototype vectors (Hebb Rule, [19]) Step 2. Determine the diagonal matrix A and W as follows: So, the prototype vectors {d  }  =1 correspond to equilibrium points.

) Proposition 5 .
In asynchronous mode, in the D-SP-SNN in (4) with a sufficiently small () and with a symmetric matrix W, the   () is getting closer to  tgt  at those iteration steps  where   () ̸ = 0; that is, |  ( + 1)| < |  ()|, where index  shows the state being updated at iteration .

) 8 Figure 1 :
Figure 1: The figure shows the percentage of correctly recovered desired patterns for all possible initial conditions in Example 7 for the proposed D-SP-SNN and FSP-SNN as compared to traditional Hopfield network (8-neuron case).

16 Figure 2 :
Figure 2: The figure shows the percentage of correctly recovered desired patterns for all possible initial conditions in Example 8 for the proposed D-SP-SNN and its 1-bit version FSP-SNN as compared to traditional Hopfield network (16-neuron case).

Figure 3 :Figure 4 :
Figure 3: Typical plot for evolutions of states (a) 1 to 8 and (b) 9 to 16 in Example 8 by the D-SP-SNN.
in Example 8 are given in Figure 5.The figure shows that the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network with the same weight matrix.Example 9.In Examples 7 and 8, the desired vectors are orthogonal.In this example, the desired vectors represent numbers 1, 2, 3, and 4, which are not orthogonal to each other.The numbers are represented by 25 neurons.The weight matrix is determined by the Hebb learning as in the previous examples.In the rest of the examples in this paper, we set  1 = 1,  1 = 10,  2 = 10,  2 = 1, and () = 0.01, for all .

Figure 8
Figure8shows desired pattern 2, a distorted pattern 2 where the HD is 5, the result of HNN-Euler, and the result of D-SP-SNN using the distorted pattern as initial condition.As seen from the figure, the proposed D-SP-SNN succeeds to
are shown in Figure 16.As seen from the figure, (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield Neural Network, and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix for this example.Example 10.In this and in the following example, we examine the performance of the proposed D-SP-SNN in clustering
Example 10 are given in Figure 21.The figure confirms that (i) the proposed D-SP-SNN minimizes the Lyapunov function of Hopfield neural network and (ii) the proposed D-SP-SNN converges faster than its HNN-Euler counterpart with the same weight matrix.Example 11.In this example, there are 40 data points as shown in Figure 22.The figure also shows the bisecting clustering results by -means algorithm and the proposed D-SP-SNN.

Figure 21 :Figure 22 :
Figure 21: Evolution of (a) Lyapunov function and (b) norm of the difference between the state vector and equilibrium point in Example 10 ( = 16).

Figure 24 :
Figure 24: Evolution of (a) Lyapunov function and (b) norm of the difference between the state vector and equilibrium point in Example 11 ( = 40).