Dynamics of Moment Neuronal Networks with Intra-and Inter-Interactions

A framework of moment neuronal networks with intraand inter-interactions is presented. It is to show how the spontaneous activity is propagated across the homogeneous and heterogeneous network. The input-output firing relationship and the stability are first explored for a homogeneous network. For heterogeneous network without the constraint of the correlation coefficients between neurons, amore sophisticated dynamics is then explored.With random interactions, the network gets easily synchronized. However, desynchronization is produced by a lateral interaction such as Mexico hat function. It is the external intralayer input unit that offers a more sophisticated and unexpected dynamics over the predecessors. Hence, the work further opens up the possibility of carrying out a stochastic computation in neuronal networks.


Introduction
The theory of moment neuronal networks (MNN) was developed recently [1], which takes into account both the first and the second order statistics of spike trains, generalizing the case of Poisson synaptic inputs to more biologically plausible renewal processes.Therefore, the MNN framework can be considered as an attempt toward a general framework of computation with stochastic systems.Further a more biologically reasonable MNN with intra-and interlayer interactions was developed by introducing intralayer inputs [2], which is analogous to the "networks with context units" [3] in ANNs.It shows that even a single unit with such system, as analogous to [4][5][6], is able to perform various complex nonlinear tasks like the XOR problem and that it can reach the trade-off between the output bias and variance by determining the optimal penalty factor due to a specific learning task.
In this letter we will explore how the spontaneous activity is propagated across the feedforward network with both homogeneous and heterogeneous connections, derived from the intralayers inputs.The input-output firing relationship and the stability are first explored for a homogeneous network.Synchronization or desynchronization is then presented for heterogeneous network by random or lateral interactions.It is the external intralayer input unit that offers a more sophisticated and unexpectedly dynamics over MNN.This paper is organized as follows.The framework of MNN with intra-and interlayer interactions is presented in Section 2. The dynamics of homogeneous and heterogeneous network are explored in Section 3.

Moment Neuronal Networks with Intra-and Inter-Interactions
For the decay rate , when the membrane potential  +1  () of the th neuron in the (+1)th layer is between its resting state   and its threshold   , it satisfies the following dynamics: ) and the coefficient of variation ( ()  1 ) are reported on the abscissa and the ordinate, respectively.Points labeled 1 correspond to the first layer; points corresponding to successive layers are connected by lines.Results were obtained for  = 100,   =    = 0,  (1)  1 = 50 Hz (solid lines), 100 Hz (dashed lines), and  = 0.1 (top),  = 0.5 (middle), and  = 1.5 (bottom).In each panel, the ratios from intralayer (  ) are evenly spaced.where the synaptic input  (+1) ,syn () is given by (

Mathematical Problems in Engineering
Here  ,(),  and  ,(),  are the magnitudes of excitatory postsynaptic potentials (EPSPs) and inhibitory postsynaptic potentials (IPSPs), { ,(),  ()} and { ,(),  ()} are renewal processes arriving from the th and th synapse, and  () and  () are the total number of active excitatory and inhibitory synapses in the th layer.All notations with the superscript ia imply that they receive the external stimulus from intralayer interactions rather than from the network itself, similar to the "networks with context unit" [3].It would present a more biologically reasonable network architecture than the moment neuronal networks.When  (+1)  () crosses the membrane threshold   from below, a spike is generated and the membrane resets to its resting potential   .
In terms of Siegert's expression [7], we have the expression of the mean and variance of the output ISIs: where As discussed in [2], for the correlation between the inputs and outputs, we can assume that the following heuristic relationship holds: Note that the right-hand side of ( 10) is the correlation of the inputs to the th and the th neurons in the ( + 1)th layer, which includes the intra-and interinput correlations.The above relationships between inputs and outputs lay the foundation of the moment neural networks with intraand interinteractions (MNNIII).

Dynamics of Networks
The question we intend to address here is how a spontaneous activity can be maintained in a feedforward network.For all simulations, the values of the decay rate, threshold, and resting potential were set equal to  = 1/20 ms −1 ,   = 20 mV, and   = 0 mV, respectively.The same parameters have been employed elsewhere [8] and are thought to be in the physiological range for visual cortex cells and in agreement with most published results [9,10].The coefficient of variation, CV = √var()/⟨⟩, is used to quantify the irregularity of a spike train.If CV = 0, the spike train is regular; otherwise the spike train is random.As analogous to [1], we stop our simulation when the firing rate is slower than 0.001 Hz.All simulations were carried out with MATLAB.

Dynamics of Homogeneous Network.
In this section, we focus on a homogeneous network, where all weights, afferent means, and variances were set to be identical.Now the quantities in (6)

𝑗
, and  () 12 =  ()  =  () = ,  (), 12 =  (),  =  (), =   ,  ̸ = .As we discussed before, the propagation of correlation becomes trivial in such case since all cells become fully correlated after the first layer.To avoid this, we clamped the correlation coefficient; that is, we set  ()  12 =  ()  =  () = ,  (), 12 =  (),  =  (), =   ,  ̸ = .In simulations we set   =  = 0 or 0.1, in agreement with experimental data reported in the literature [1,11,12], and we assume that each intralayer input, including numbers of neurons, afferent means, and variances, is identical; that is,  (1), =  (), ≡   ,  (1), 1 =  (), 1 , and First we choose  = 0.1, 0.5, 1.5 to illustrate how the activity is propagated across the networks, varied from   and   , on assumption that  () 1 =  (), 1 = 1.Unless otherwise specified, the initial CV in all simulations is always 1, and the weight is 0.5.In Figure 1, we show the results obtained for various values of  (1) and  (1) (we reported the coefficient of variation  () =  () / () ).As stated in [1], each data point ( () ,  () ) is connected with ( (+1) ,  (+1) ) to illustrate how the activity is propagated across the networks.After the first few layers, for  = 0.1 (top), neurons are found to be either silent or firing at relatively high frequency for all   and   .For  = 0.5 (middle), however, the situation has changed.On the one hand, the left panel (middle) shows that even if there is no inhibitory input (  = 0) for a small number of intraneurons (  = 10), to be silent is certainty on fewer intrainput neurons.On the other hand, the right panel (middle) shows that not all networks are certainty to be silent.As stated earlier, for  = 0.5 with fewer intrainput neurons (  = 10), the network is certainty to be silent (left).However, with increasing size of intrainput units, the network might be fixing or firing at relatively high frequency (right).Instead, they fire at low or relatively high frequency; for example,   ≥ 30 for   1 = 0.1.It actually shows that the networks can be stable at low frequency; see discussions later.In addition, for  = 1.5 (bottom), there is another kind of properties that neurons fire at relatively low frequency after a few times oscillations.Unlike a homogeneous MNN [1], neurons are certainty to be silent with a larger ratio (e.g.,  ≥ 0.4) after the first few layers.It is obvious that the intrainputs cause a more sophisticated dynamics of network than MNN.
Figure 1 implies that the stable solution existed in some networks.Thus we explore the stable solution of such network.Figure 2 shows how the input-output firing rate relationship is varied from   and   , for  = 0.5 and  = 1.5.As analogous to [1], roughly speaking, the stable solution appears at low firing rate for networks of increasing size (either  or   size) with strong inhibition (either  or   ).However, the preferred appearance of stable solution at low frequency is not the increasing size of both inter-and intralayer; for  = 1.5 (top right, dashed lines), for example, any ratio   from intralayer (from 0 to 1.5) appears as a fixed point at low firing rate when  = 300,   = 100 (top right), instead of  = 300,   = 300 (bottom right).We further explore the stability of such network by showing that the stable outputs varied from (  ,   ). Figure 3 shows the stable outputs versus   and   , with a fixed ratio from interlayer inputs  = 0.8 (top) and  = 1.5 (bottom) and with the number of neurons from interlayer inputs  = 100 (left) and  = 300 (right).The initial inputs fix 5 Hz in the current and latter simulations.It is found that the intralayer input unit appears to have a sophisticated role in the stability of such network, depending on the ratio   and the size   of context unit, together with the ratio  and the size  of network itself, that is, interlayer.For example, for  = 0.8 (top), the stable output appears to increase with the increasing size   and the decreasing ratio   of context unit, which plays a general role; for  = 1.5 and  = 300 (bottom right panel), however, the situation has changed to play a negative role; see   ⩾ 300.
We also explore the stability of such network by showing that the stable outputs varied from numbers of intralayer neurons   and of interlayer . Figure 4 shows that the network with a small size of intra-and interlayer will be either silent or firing at low frequency.Top panel shows that the stable output of such network seems to be symmetry between numbers of intralayers and interlayers when  = 1.However, there are obvious asymmetry for  < 1 or  > 1; see middle and bottom panels in Figure 4. Middle panel shows that the network with a relatively large size of interlayer, for  < 1, will be firing at relatively high rate and that the size of intralayer has negligible effect on the stable outputs such as  > 250.Bottom panel presents unexpected results of  > 1 that relatively high frequency is firing at the large size of intralayer and small size interlayer, rather than the large size of intralayer and interlayer.In particular, for  > 300 and   > 300, the network is unexpectedly firing at low frequency when  = 1.5 and   = 0.5; see the left of bottom panel.All those imply that the current network can lead to more sophisticated dynamics than MNN and the intrainput unit plays a very sophisticated role in dynamics of such network.
Finally, we explore the stability of such network by showing that the stable outputs varied from the ratio of both inter-and intralayers, that is,  and   .Figure 5(a) shows the stable outputs versus  and   with  =   = 300.It can be seen that the network will be either silent or firing at low frequency when  > 1 and will be firing at relatively high rate when  < 1, which implies that the ratio (between inhibitory and excitatory input) from the network itself plays a more important role than that from intrainput unit with such parameters.However, the network will converge to silent or to low firing frequency with strong inhibition  > 1 or to high frequency with weak inhibition  < 1.Unlike the stable output of MNN in [1] (a high level of inhibition is necessary to maintain a spontaneous activity, that is, a low output firing rate), a weaker inhibition ( < 1 approaching 1), even with weaker intrainhibition (  < 1 approaching 1), can converge to low firing frequency; see, for example,  ∈ (0.8, 1) and   ∈ (0.5, 1).It is the intrainput unit to maintain the low frequency activity.Thus there is an important gain in computational power over MNN.Note the appearance of oscillation for a homogeneous MNNIII as singularity, for example, with  = 2,   = 0.1,  =   = 300, and   =    = 0.1 (Figure 5(b)).

Dynamics of Heterogeneous Network.
We now turn to explore a more sophisticated dynamics of a heterogeneous network.We shed light on, for instance, how synchronization or desynchronization is produced by random or lateral connections when we remove the constraint on the correlations between neurons.
First we generated random connections  ()   ,  (1),  ∈ [0, 0.5] and inputs  (1)   ,  (1),  ∈ [0, 100] Hz.We also assumed that  (1)   1 =  (1), 1 = 1 and that  (1)   ,  (1),  ∈ [0, 0.1] for ,  = 1, . . ., 324. Figure 6 shows the output CV versus the mean firing rate in the first 10 layers of the network itself (left) and the correlation coefficients for one cell in the first, second, and fourth (or fifth) layer (right) for  = 1.2,   = 1.5;  = 0.5,   = 0.5; and  = 1.5,   = 0.9.Simulations show that all neurons synchronize after the 5th layer, that is,  ()  = 1 for  ≥ 5, and that the network is stable.Top panels show that the network will be stable at relatively low frequency (<100 Hz) after the 4th layer; middle panels show that the network will be stable at relatively high rate (>100 Hz) after the 5th layer; however, the last panels show that the network can be stable at a very low frequency (<1 Hz).For example, Figure 7 shows the mean firing rate for different layers, corresponding to Figure 6 (top).Note the different scale for each layer.It can be seen from Figures 6 and 10 (top) that the ratios  and   change the stable outputs but do not change the synchronization.This result, in general, is in agreement with numerical experiments of feedforward spiking neuronal networks, showing that neurons get synchronized quite easily [13].In fact, desynchronization rather than synchronization seems to be the major problem for a spiking neuronal network.
Figure 8 shows the propagation of activity in a heterogeneous MNNIII with Mexico hat type connections for  = 1.2,   = 1.5.The results indicate that the activity in the network becomes stable after the 6th layer (top left).Also, as indicated by the values of the correlation coefficients in the top right panel, the neurons do not synchronize.Middle and bottom panels show the mean firing rate for different layers.Figure 9 presents the correlation coefficients between the central cell and the other cells in layer 3 and in layer  (10)   153,i (b) Figure 9: The correlation coefficients between the central cell and the other cells in layer 3 (left) and in layer 10 (right), corresponding to Figure 8.
10, corresponding to Figure 8; on average the correlation coefficient between neurons was around zero.
It is interesting to note that, from the first to the second layer, there is a general reduction in firing rate.Then, after a few transition layers, the neuronal activity becomes stable.In summary, the activity in a heterogeneous MNNIII becomes stationary after a few layers.With random interactions, the network gets easily synchronized.However, with lateral interactions by Mexico hat function, desynchronization is presented; that is, firing rates of individual neurons tend to spread out.
To compare the results with others, Figure 10 shows the propagation of activity in a heterogeneous MNNIII with random connections (top) and with Mexico hat type connections (bottom) for  = 1.5,   = 1.5.It only shows the output CV versus the mean firing rate in the first 10 layers (left) and the stable mean firing rate in layer 10 (right), respectively.Figure 10 implies that the MNNIII with random connections gets synchronized after the 6th layer, and with Mexico hat type connections it becomes stationary and fires with the bell-like firing rates after the 7th layer.
First, the weight connections are key to propagate the activity of such network.A homogeneous network with clamped weight connections, as stated in Section 3.1, converges to silent or to low/high firing frequency.However, a heterogeneous network gets synchronized or desynchronized due to random or lateral weight connections.It is the lateral weight connections introduced by the Mexican hat that push cells to fire with more widely spread firing rates than in a network with random interactions; see the former discussions in this subsection.In other words, weight connections determine the type of dynamics of network (say, convergence, synchronization, and desynchronization) and subsequently change the size of stable outputs.For  = 1.5,   = 1.5, for example, a homogeneous network with initial input 50 Hz converges to 16.09 Hz after the 5th layer (approximate to  = 1.5,   = 1.5 in the left panel of Figure 5); Figure 10 shows that a network with random connections gets synchronized at 9.36 Hz after the 6th layer (top) and with Mexico hat type connections it gets desynchronized at the stable like outputs with a mean value of 19.56 Hz after the 7th layer (bottom).It is the weight connections that change the stable outputs of network.Additionally, the network simulations reported in this subsection show that a highly irregular output firing is also the result of correlations between neuronal activities.
Second, all simulations show that the ratio of inhibition from inter-or intralayer ( or   ) does not determine the type of dynamics but influences the stable outputs of network.For homogeneous network, it was verified in Section 3.1.For a network with random connections, Figures 6 and 10 (top) show that the ratio  or   changes the stable outputs but does not change the synchronization.Figures 8 and 10 (bottom) show that the lateral weight connections introduced by the Mexican hat push cells to fire with more widely spread firing rates than in a network with random interactions and that the ratio  only changes the stable outputs; for example, the mean of stable bell-like outputs in Figure 8 (18.05Hz) is bigger than that in bottom panel of Figure 10 (11.45 Hz).It is the inhibition from network itself or from external intrainput that can enhance or weaken the (stable) firing outputs of network.
Finally, all parameters influence the speed of the stability of network.For homogeneous network, it was included in discussions in Section 3.1.For heterogeneous network, the weight connections can influence it, when other parameters are identical; see, for example, Figure 8, which shows that the speed of synchronization by random connections is slightly faster than that of desynchronization by lateral connections.Further the ratio of inhibition influences it; see, for example, Figures 6 and 10  Mexican hat type connections.In addition, the heterogeneous MNNIII simulations reported in this subsection show that synchronization/desynchronization seems to be slightly slower or faster than MNN [1] with other identical parameters, since the external intralayer input unit can enhance or weaken the firing outputs.Thus, the network reported here offers a more sophisticated and unexpected dynamics over the MNN.

Discussion
In the current paper, we have focused on the dynamics of feedforward MNNIII, akin to the dynamics of MNN.Synchronization or desynchronization is presented by random or lateral interactions.Due to more biologically reasonable intralayer inputs, such network offers a more sophisticated and unexpected dynamics over the MNN.As stated earlier, the intrainput unit plays a very sophisticated role in dynamics of such network.It can enhance or weaken the firing outputs by varying on the ratio   and/or the size   of intralayer, incorporating with those of interlayer, that is,  and/or .
In the future we might shed light on the application in engineering, the dynamics, and/or learning rule of MNNIII with other network architectures such as recurrent and RBF network [14].

Figure 2 :
Figure 2: The input-output firing rate relationship for a homogeneous MNNIII varied from numbers of intralayer neurons   and the ratio of intrainputs   .Results were obtained for   =    = 0.1 and  = 0.5 (solid lines),  = 1.5 (dashed lines).

Figure 5 :
Figure 5: (a) Stable outputs for a homogeneous MNNIII varied from the ratio of both inter-and intralayer ( and   ).Results were obtained for  =   = 300, with   =    = 0.1.(b) The appearance of oscillation for a homogeneous MNNIII.

5 Figure 7 :
Figure 7: The mean firing rate for different layers, corresponding to top panel in Figure 6.

10 Figure 8 :
Figure 8: Propagation of activity in a heterogeneous MNNIII with Mexico hat type connections.The output CV versus the mean firing rate in the first 10 layers (top left).The correlation coefficients for three cells in layer 10 (top right).The mean firing rate for different layers (middle and bottom).Results were obtained for  = 1.2,   = 1.5.
(top), which show its validity for random connections, and Figures8 and 10

Figure 10 :
Figure 10: Propagation of activity in a heterogeneous MNNIII with random connections (top) and Mexico hat type connections (bottom).The output CV versus the mean firing rate in the first 10 layers (left).The stable mean firing rate in layer 10 (right).Results were obtained for  = 1.5,   = 1.5.