^{1}

^{2, 3}

^{1}

^{1}

^{1}

^{2}

^{3}

For decades, Mackey-Glass chaotic time series prediction has attracted more and more attention. When the multilayer perceptron is used to predict the Mackey-Glass chaotic time series, what we should do is to minimize the loss function. As is well known, the convergence speed of the loss function is rapid in the beginning of the learning process, while the convergence speed is very slow when the parameter is near to the minimum point. In order to overcome these problems, we introduce the Levenberg-Marquardt algorithm (LMA). Firstly, a rough introduction is given to the multilayer perceptron, including the structure and the model approximation method. Secondly, we introduce the LMA and discuss how to implement the LMA. Lastly, an illustrative example is carried out to show the prediction efficiency of the LMA. Simulations show that the LMA can give more accurate prediction than the gradient descent method.

The Mackey-Glass chaotic time series is generated by the following nonlinear time delay differential equation:

Originally, Mackey and Glass proposed the following equation to illustrate the appearance of complex dynamics in physiological control systems by way of bifurcations in the dynamics:

They suggested that many physiological disorders, called dynamical diseases, were characterized by changes in qualitative features of dynamics. The qualitative changes of physiological dynamics corresponded mathematically to bifurcations in the dynamics of the system. The bifurcations in the equation dynamics could be induced by changes in the parameters of the system, as might arise from disease or environmental factors, such as drugs or changes in the structure of the system [

The Mackey-Glass equation has also had an impact on more rigorous mathematical studies of delay-differential equations. Methods for analysis of some of the properties of delay differential equations, such as the existence of solutions and stability of equilibria and periodic solutions, had already been developed [

The Mackey-Glass chaotic time series prediction is a very difficult task. The aim is to predict the future state

In this paper, we will predict the Mackey-Glass chaotic time series by the MLP. While minimizing the loss function, we introduce the LMA, which can adjust the convergence speed and obtain good convergence efficiency.

The rest of the paper is organized as follows. In Section

A multilayer perceptron (MLP) is a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs. A MLP consists of multiple layers of nodes in a directed graph, with each layer fully connected to the next one. Except for the input nodes, each node is a neuron (or processing element) with a nonlinear activation function. The multilayer perceptron with only one hidden layer is depicted as in Figure

Multilayer Perceptrons.

Mackey-Glass chaotic time series.

In Figure

The output of the multilayer perceptron described in Figure

MLP provides a universal method for function approximation and classification [

Given a set of observed data, sometimes called training examples, we search for the parameters

One of the serious problems in minimizing the mean square error is that the convergence speed of the loss function is rapid in the beginning of the learning process, while the convergence speed is very slow in the region of the minimum [

In mathematics and computing, the Levenberg-Marquardt algorithm (LMA) [

The LMA is interpolates between the Gauss-Newton algorithm (GNA) and the gradient descent algorithm (GDA). As far as the robustness is concerned, the LMA performs better than the GNA, which means that in many cases it finds a solution even if it starts very far away from the minimum. However, for well-behaved functions and reasonable starting parameters, the LMA tends to be a bit slower than the GNA.

In many real applications for solving model fitting problems, we often adopt the LMA. However, like many other fitting algorithms, the LMA finds only a local minimum, which is always not the global minimum.

The least squares curve fitting problem is described as follows. Instead of the unknown true model, a set of

The LMA is an iterative algorithm and the parameter

The Taylor expansion of the function

As we know, at the minimum

Levenberg’s contribution is to replace this equation by a “damped version,”

The damping factor

In the process of iteration, if either the length of the calculated step

Levenberg’s algorithm has the disadvantage that if the value of damping factor

In this section, we will derive the LMA when the MLP is used for the Mackey-Glass chaotic time series prediction. Suppose that we use the

To implement the LMA, what we should do is calculate the Jacobian matrix

The derivatives of

As we know,

We will conduct an experiment to show the efficiency of the Levenberg-Marquardt algorithm. We choose a chaotic time series created by the Mackey-Glass delay-difference equation:

Such a series has some short-range time coherence, but long-term prediction is very difficult. The need to predict such a time series arises in detecting arrhythmias in heartbeats.

The network is given no information about the generator of the time series and is asked to predict the future of the time series from a few samples of the history of the time series. In our example, we trained the network to predict the value at time

In the simulation, 3000 training examples and 500 test examples are generated by (

Let

The initial values of the parameters are selected randomly:

The learning curves of the error function and the fitting result of LMA and GDA are shown in Figures

The learning curves of LMA and GNA are shown in Figures

As far as the fitting effect is concerned, the performance of LMA is much better than that of the GDA. This is very obvious from Figures

All of these suggest that when we predict the Mackey-Glass chaotic time series, the performance of LMA is very good. It can effectively overcome the difficulties which may arise in the GDA.

The GDA learning curve of the error function.

The LMA learning curve of the error function.

Fitting of Mackey-Glass chaotic time series, GDA.

Fitting of Mackey-Glass chaotic time series, LMA.

In this paper, we discussed the application of the Levenberg-Marquardt algorithm for the Mackey-Glass chaotic time series prediction. We used the multilayer perceptron with

The authors declare that there is no conflict of interests regarding the publication of this paper.

This project is supported by the National Natural Science Foundation of China under Grants 61174076, 11471152, and 61403178.