Single Gaussian Chaotic Neuron: Numerical Study and Implementation in an Embedded System

Artificial Gaussian neurons are very common structures of artificial neural networks like radial basis function. These artificial neurons use a Gaussian activation function that includes two parameters called the center of mass (cm) and sensibility factor (λ). Changes on these parameters determine the behavior of the neuron. When the neuron has a feedback output, complex chaotic behavior is displayed.This paper presents a study and implementation of this particular neuron. Stability of fixed points, bifurcation diagrams, and Lyapunov exponents help to determine the dynamical nature of the neuron, and its implementation on embedded system illustrates preliminary results toward embedded chaos computation.


Introduction
The study of dynamical systems that involves artificial neurons is very complex; usually, numerical solutions are required when analytical solutions are difficult to be made.These simulations require models from single neurons to interconnected neurons and help to explain the behavior of complex neural structures.A similar work about analysis of complex dynamic systems with groups of neurons is [1], where results show that the periodically forced Hodgkin-Huxley oscillator exhibits not only periodic motions but also non-periodic motions (quasi-periodic or chaotic oscillation) depending on the amplitude and the frequency of the stimulating current, the exhibited complex motions were similar to motions found experimentally in squid giant axons.Inspiration on neural structures from the nature is given by the work of Wang [2] where a complex dynamic behavior is generated from the Bonhoeffer-van der Pol model, and it is due to the interaction between the periodic stimulating current and Bonhoeffer-van der Pol oscillations.A complete work about the analysis of nature-inspired models was made by Korn and Faure [3].Authors use different models of excitable cells with different modes of firing of busting neurons.These models that simulate different structures of the brain are complex nonlinear systems that generate oscillations inclusive chaotic ones.With this in mind, the signals of the brain generated by neural structures could be chaotic.Soriano et al. presents a similar work where the Fitzhugh-Nagumo neural model is analysed using a new method called cloned dynamics approach used for calculating the Lyapunov spectrum of dynamical systems [4].
Single neurons have been studied recently, in part because simple configuration could exhibit high complexity in its dynamics that includes chaos.Li and Chen [5] demonstrate that a single neuron can show a chaotic behavior due to the nonlinear dynamics where dependency of parameters is strong, generating switching between coexisting chaotic attractors and connected attractors [5].
A comparison between single neurons models is made in the work of Sungar et al. [6] to establish a model for cognitive purpose, where synchronization conditions are explored.A neural model with a high propensity to synchronization could be more plausible for modeling neural activity.El Boustani and Destexhe [7] use microelectrodes inserted in the cat cerebral cortex that show neuronal discharges very similar to stochastic variables or single neurons; in this case, single neurons show stochastic dynamics, but on large scales the behavior is more coherent with high dimensional chaos [7].Other example is the study of structures from thalamocortical neurons that can be explained using a model of the generated current.An analysis made on this model to enervate one-dimensional maps captures the essentials of the chaotic behavior [8].Cellular neural networks (CNN) are another related work.Single cells make interactions between neighbors only and generate complex behavior.Roska and Chua work with discrete time CNN using the Chua's local activity principle and nonhomogeneous spatiotemporal patterns are induced by initial setting in these systems [9].A complete review of these systems can be found in the work of Roska and Chua [10].
The study of single neurons as an isolated system is important because very complex dynamics can be generated in a simple structure.Usually, single neurons are modelled with differential equations with a time delay [11].The study of these systems is diverse and extended; however, it is possible to take advantage of the richness in this behavior in a variety of applications like chaos computing [12,13].In all cases, phase plots, waveform plots, and power spectra (in others cases, Lyapunov exponents) are used to confirm chaoticity.
Implementation of single neurons has been made usually with electronic circuits, where the nonlinear behavior is analysed by means of electronic simulation software [14], or by designing analog electronics using the same way to solve differential equations by analog computers [15,16].Due to the social demands for sophisticated products and the requirements on the daily industrial processes to cover the human needs, it is important the implementation of new scientific tools on embedded systems.The embedded systems are often defined as a collection of programmable parts surrounded by ASICs (application-specific integrated circuits) and other standard components that interact continuously with an environment through sensors and actuators.The programmable parts include microcontrollers and DSPs [17,18].The use of embedded systems simplifies the design procedure involved and reduces the time process implementation, so a prototype has a rapid launch.
In this paper, a single neuron is represented as a nonlinear discrete dynamic system.Bifurcation diagrams and Lyapunov exponents are used to explain the behavior of the system and determinate effects of the parameters to generate periodicity, stability, and chaos.Finally, we presented an implementation of a single neuron with a Gaussian activation function in an embedded system based on the open-source platform Arduino, which is a micro-controller-based electronic board of 8-bit microcontroller ATmega168 [19].Experiments are made using two potentiometers to establish the parameter values and as results, some bifurcation diagrams comparable with numerical results are made.

A Single Gaussian Neuron
A neuron is an information-processing unit that is fundamental to the operation of a neuronal network [20]  called simple-processing unit.The block diagram of Figure 1 shows the classical model of a neuron, identifying three basic elements: (i) A set of synapses or connecting links, each of which is characterized by a weight of its own.Specifically, an input signal   , connected to the neuron, is multiplied by the synaptic weight   .
(ii) An adder to add the input signals, pondered by the respective weight.
(iii) An activation function for limiting the amplitude of the output of a neuron.Normally this function is nonlinear such as Gaussian, sigmoidal or hyperbolic tangential forms.
In this paper we deal with a single recurrent artificial Gaussian neuron (SRAGN) in which, as can be seen in Figure 2, the output  is feedback to the main input , while the other two inputs, cm and , are parameters of the activation function.
The Gaussian activation function is given by and has bell form as can be seen in Figure 3.
A single neuron can be considered as a difference equation that can generate complex behavior when the input is a feedback of its output and the activation function is a Gaussian map, a commonly studied nonlinear discrete system [21].
Thus, the discrete response of this SRAGN is governed by where (i)  = {1, 2, 3, . . ., } denotes the number of the execution of (2), with   and   as the neuron response and input for execution , respectively.
(ii) Consider that input  +1 is the feedback response   , with  1 =  0 = 0 as initial condition.
(iii) It is considered that each execution  takes  seconds to complete the computation of   (⋅).
Then, discrete response of the SRAGN (2) can be rewriten as a one-dimensional nonlinear discrete dynamical system For certain parameter values  and cm, the response   of system (3) can display highly complex behavior and even chaotic phenomena.For instance, with values  = 0.7 and cm = 0.25, response   tends to a fixed point, see Figure 4(a).For values  = 0.4 and cm = 0.25, response   is a periodical oscillation shown in Figure 4(b).For values  = 0.15 and cm = 0.50, response   seems to be an irregular or chaotic response shown in Figure 4(c).In the next section, the dynamical behavior for SRAGN (3) is analyzed.

SRAGN Dynamical Behavior Analysis
Firstly, consider the case in that response   of system (3) tends to be a fixed point and remains in it.Formally, a fixed point is defined as follows.
Using Definition 1, it is possible to determine the fixed points of period one for the SRAGN (3); however, fixed points  * are difficult to calculate as an analytical solution because the function ( 3) is exponential and has two parameters As with other dynamical systems, the fixed points of period one can be attracting, repelling, or indifferent.The type of fixed point is determined from the gradient of the tangent to the function (  ) at the fixed point  * .A test for stability of fixed points for nonlinear discrete systems is established with the following result.
Theorem 2 (see [22]).Suppose that map () has a fixed point at  * ; then the fixed point is stable if and it is unstable if When |( * )/| = 1, the fixed point is neither repelling nor attracting.Using Theorem 2, stability test of some fixed points  * computed for (3) is shown in Table 1.
Then, there is one unstable period-one fixed point at  * = 0.657708295 when  = 0.5 and cm = 0.2.When  = 0.9 and cm = 0.5, (3) has one stable fixed point of period one at  * = 0.754050244206.Exploring values of  and cm, there are different stable or unstable period-one points of (3).Thus, a question arises: where do the orbits go if not to these unstable points of period one?
One answer is that response   tends to infinity because system (3) is unstable, but there are cases in that response   remains in a bounded domain as in Figure 4(b).The value in which a fixed point changes from stable to unstable is called bifurcation point.When the system reaches a bifurcation point a doubling period occurs.
Definition 3 (see [22]).For system  +1 = (  ), a fixed point of period  is a point at which  + =   (  ) =   , for all .Fixed points of period two are numerically computed for (3), and with values of  = 0.5 and cm = 0.2 there are two stable period-two fixed points at  * = 0.293828883932 and  * = 0.982546391135.
In summary, if the period-one fixed point  * is stable, iterative responses will be attracted to that point, which means that response   stabilizes to a constant value after a sufficiently long time.If  = 0.5 and cm = 0.2, fixed point of period one becomes unstable and stable fixed points of period two are created; the response   will alternate between two values on each iterative step after a sufficient amount of time.As  and cm pass through 0.15 and 0.25, respectively, the fixed points of period two lose stability and fixed points of period four are created.As with other dynamical systems, all of this information can be summarized on bifurcation diagrams.Setting cm = 0.25, a bifurcation diagram is illustrated in Figure 5, where a value close to  = 1 establishes a stable fixed point.For 0.2 <  < 0.65, two stable fixed points of period two are created.For 0.1 <  < 0.2, the fixed points of period two will be unstable and fixed points of period four will appear.If parameters are changed even further, stability is lost again, which marks the rise of a period eight cycle for (3) and so on.The periods of the attractive cycles are 1, 2, 4, 8, 16, and 32, and the bifurcations are called period-doubling bifurcations.Thus, SRAGN exhibits a variety of behaviors as  and cm take different values; inclusive there are responses of complex behavior for certain parameter values, which seem chaotic, as in Figure 4.At this point, chaos is understood in the sense of deterministic dynamical systems.
No definition of the term chaos is universally accepted yet, but almost everyone would agree on the three ingredients used in the following working definition [21].Chaos is an aperiodic long-term behavior in a deterministic system that exhibits sensitive dependence on initial conditions.
Aperiodic long-term behavior means that there are trajectories which do not settle down to fixed points, periodic orbits, or quasiperiodic orbits as  → ∞.Deterministic means that the system has no random or noisy inputs or parameters.Sensitive dependence on initial conditions means that nearby trajectories separate exponentially fast.
An attractor is a closed set  with the following properties.
(1)  is an invariant set.Any trajectory () that starts in  stays in  for all time.
(  then the distance from () to  tends to zero as  → ∞.This means that  attracts all trajectories that start sufficiently close to it.The largest such  is called the basin of attraction of .
(3)  is minimal.There is no proper subset of  that satisfies conditions (1) and (2).
A strange attractor is an attractor that exhibits sensitive dependence on initial conditions [21].
The Lyapunov exponents are a criterion used for diagnosis of whether or not a system is chaotic.One of the properties of chaos is the sensitivity to initial conditions.This means that if two trajectories start close to one another in phase space they will move exponentially away from each other for small times on the average.However, it is known that an orbit on a chaotic attractor for a bounded system also returns to all accessible states with equal probability.This property is known as ergodicity.Thus iterates return infinitely closely, infinitely often to any previous point on the chaotic attractor.The Lyapunov exponents give an indication as to whether two orbits starting close together diverge or converge.In order to determine if responses of SRAGN are chaotic for certain parameters, Lyapunov exponents will be computed by means of the next results.Definition 4 (see [22]).The Lyapunov exponent  computed using the derivative method is defined by where   = ()/ represents differentiation with respect to  and  1 ,  2 , . . .,   are  successive samples.The Lyapunov exponent may be for a sample of points near the attractor to obtain an average Lyapunov exponent.
The divergence of chaotic orbits can only be locally exponential, because if the system is bounded, as most of the SRAGN responses are,   cannot go to infinity.Thus, the Lyapunov exponent ( 6) is a measure of this divergence of orbits.Then the criterion for chaos becomes [23]  > 0 (chaotic) ,  ≤ 0 (regular motion) .
Then, the signs of the Lyapunov exponents provide a qualitative picture of a system's dynamics.One-dimensional maps are characterized by a single Lyapunov exponent which is positive for chaos, zero for a marginally stable orbit, and negative for a periodic orbit [24].
The above criterion is useful to determine the chaotic behavior of trajectories   of (3) that start and stay in a bounded domain for all time.Three cases are considered for analysis of the behavior of the SRAGN (3) by means of bifurcation diagrams and Lyapunov exponents setting cm in 0.25, 0.50, 0.75 and varying the  parameter from 0 to 1.
In Figure 6, parameter cm = 0.25, the chaotic behavior is presented in the narrow interval 0.05 <  < 0.1, the period- region is presented in the interval 0.1 <  < 0.65, and the region of stable fixed point of period one corresponds to 0.65 <  < 1.
In Figure 7, parameter cm = 0.50, the chaotic behavior is presented in the interval 0.06 <  < 0.2, the period- region is presented in the interval 0.2 <  < 0.5, and the region of stable fixed point of period one corresponds to 0.5 <  < 1.
In Figure 8, parameter cm = 0.75, the region of chaotic behavior is vague near to  = 0.1, the period- region is presented in the interval 0.1 <  < 0.3, and the fixed point of period one is presented in a wide region given by 0.3 <  < 1.
From illustrated cases, if any parameter  or cm takes values close to one, the behavior of responses will tend to a stable fixed point independent of the other parameter.From bifurcation diagrams and Lyapunov exponents computed, there are indicia that chaotic behavior is present in SRAGN for certain parameter intervals.

Implementation of Single Neurons in an Embedded System
Arduino is an open-source platform to work with electronic, control, robotic, and informatics prototypes.The platform consists of a microcontroller-based board and an integrated development environment (IDE) software to write, debug, and download programs to the board.Both the board and the IDE are opened to the users, either hardware or software modifications can be made.There are some versions of this board according to the desired features; the implementation presented in this paper uses the Arduino Diecimila one, which is based on the 8-bit microcontroller ATmega168 whose main features are as follows: operation at 5 Vdc, 16 MHz clock speed, 16 KB FLASH memory program, and 1 KB SRAM data memory.This board is the main component of the prototype and has in its memory the complete neuron working.Also two potentiometers were connected to modify the value of the parameters, each one in the range of [0, 5] Vdc that are converted to a [0, 1] floating point normalized range by doing the following.
So, the calculation of the neuron response is in the [0, 1] normalized range, but this value is converted to a [0, 5] Vdc representation that can be measured by an instrument such as an oscilloscope by doing the following.
(1) Variable in a [0, 1] floating point normalized range.Figure 9 shows a schematic diagram of the system implementation.ADC and PWM modules are implemented internally in the microcontroller.The pins used for these modules are mapped on board terminals.
The program was written using the programming language provided by the Arduino platform, and the compiler was responsible for the code optimization and translation to machine code.The ATmega168 microcontroller includes native instructions to work with signed/unsigned/fraction/ nonfraction multiplications and 8-bit number comparisons which make this microcontroller a good option to work with.Assignation of the initial condition  1 = 0 Do an infinite loop with containing these 5 steps: (1) Read data: read ,  and cm from the analog input pins of the board.
(2) Compute neuron response using (3).Notice that if  = 0 then the division in (3) renders undefined, so assignment must be   = 1.(3) Write the value of the response in an analog output pin of the board.Next, we present the pseudocode of the program which can be easily extended to more than one SRAGN (Pseudocode 1).
It is important to notice that the feedback of the output is made by software instead of connecting a wire from output pin to an input pin.This action avoids noise and improves the calculation.Finally, note that around 300 or even more neurons can be processed in this embedded system.

Experimental Results
In this section, some bifurcation diagrams from embedded single neurons are presented.A personal computer was used to acquire the data from the embedded system and make the plots.All bifurcation diagrams are made using the output versus the change of the parameter  while cm remains fixed in each diagram.The value of cm starts at 0.05 until 1.0 with increments of 0.05 because the 8-bit precision defined by the PWM and ADC modules.The period of execution  has been fixed to 10 milliseconds.The experimental results are illustrated in Figures 10,11,12,13, and 14.

Conclusions
Single neurons can show a variety of behaviors; this particular neuron that involves the use of a Gaussian activation function depends on only two parameters.Chaoticity is shown using numerical simulation by bifurcation diagrams and Lyapunov exponents.The implementation of this simple neuron on an embedded system generates the same results as numerical ones.An interpretation for these bifurcation diagrams could be that, for each value of cm,  chooses the type of response: fixed points, oscillating, or chaotic.For small values in cm, the predominant response (in all  range) tends to be oscillating, for medium values in cm, the predominant response is chaotic, and finally for bigger values in cm (close to 1), the predominant response is stable.By "predominant response" we mean the graphic form in the greater part of the range of ; for example, in the last plot (cm = 1) we can see that the predominant response is a fixed point; however the general form is something as a unitary pulse, that is, a bistable nonlinear system.A multiple interaction of these neurons could produce interesting structures that have complex behaviors, so the future work is to study more this neuron, what the cause of these behaviors is, the structures that can be formed, and also the control or adjustment of the parameters to get desirable behaviors, for example, the representation of oscillations observed in heart pulses, the activation of muscles for walking or breathing, learning and recognition.

Figure 3 :
Figure 3: Gaussian bell.Its two parameters are displayed in the plot.

Table 1 :
Fixed points of period one with  = 0.5 and stability determination for different values of cm.
(cm and ).Approximated solutions  * can be computed using numerical methods.As an example, setting  to 0.5, fixed points  * are computed and shown in Table1for different cm values.