^{1}

^{2}

^{1}

^{2}

This paper reviews the major developments of modeling techniques applied to nonlinear dynamics and chaos. Model representations, parameter estimation techniques, data requirements, and model validation are some of the key topics that are covered in this paper, which surveys slightly over two decades since the pioneering papers on the subject appeared in the literature.

The field of nonlinear dynamics experienced a very quick and intense development in the last thirty years or so. To determine the starting point of any subject is very difficult simply because we all stand on somebody elses showlders in any “new” attempt we make. For the sake of presentation, and because we feel that this understanding is not absurd anyway, the origins of what we nowadays call the field of nonlinear dynamics can be traced back to the work of Henri Poincaré.

In his studies on nonlinear dynamical systems Poincaré figured out that, since no analytical solution to most of nonlinear systems can be obtained, the whole set of solutions can be investigated in the so-called phase space spanned by the set of variables required for a complete description of states of the system [

From the publication of Lorenz's paper up to the mid of the 70 s not many papers were published. Among those who contributed to the emergence of “chaos theory,” we can quote Ruelle and Takens paper on turbulence [

An important turning point happened around 1980 (Figure

Number of papers published per year up to 1994. The search machine used was ISI Web of Science using Topic=(“nonlinear dynamics" OR chaos). Up to 1994 the total number of entries found was 8 400 (shown in this figure). Up to 2008 the total number of entries is 39 180 and only in 2008 over 2 700 papers were published within the topics nonlinear dynamics or chaos.

When, in the early 1980, it became accepted that a scalar-reconstructed attractor could be equivalent to the original one, a great deal of work was devoted to developing tools to quantitatively characterize chaotic attractors. In the following decade, geometric measures such as dimensions [

Another turning point which can be detected in the number of papers published in the field of nonlinear dynamics and chaos happened in the end of the decade 1980 (Figure

Before actually addressing how the modeling of nonlinear dynamics and chaos developed within the field, it is important to say that a significant amount of work was developed independently, and sometimes previously, in the field of nonlinear system identification. The basic goals were very similar to those that later were pursued by the nonlinear dynamics community but there were some important differences. Nonlinear system identification, which has its origins in the field of engineering, is usually concerned with nonautonomous systems, discrete-time models, disturbance modeling (this is vital to correctly deal with noisy data), and hardly ever was concerned with chaos. On the other hand, modeling nonlinear dynamics and chaos, with its origins in physics and applied mathematics, usually concerned with autonomous systems, very often considers continuous-time models, does not typically model disturbances, and is strongly focused on chaotic systems.

Having established such main distinctions between the two fields, let us point out that this paper is a survey of modeling techniques applied to nonlinear dynamics and chaos. Many such techniques have indeed been developed in the field of nonlinear system identification and will be mentioned in this survey only to the extent in which they were applied to modeling nonlinear dynamics and chaos. In the reminder of this section, for the sake of completion, we present some issues in nonlinear system identification which will be relevant in the discussion of modeling techniques applied to nonlinear dynamic and chaos. In [

By the end of the 1970 linear system identfication was well established. The typical problem was to build a model, usually a transfer function of the form

The approach to nonlinear system identification happened along different lines of action. One such line of action was to build so-called block-oriented models which consisted of a linear transfer function model (see (

A third, and last one to be mentioned here, line of action was to rewrite (

After presenting a brief introduction (Section

In what follows, we begin by mentioning explicitly five pioneering works, the first four of which were of great influence; [

The first papers on modeling nonlinear dynamics and chaos seem to have appeared in 1987. One of them is related to local linear modeling, the rest is generally concerned with some aspect of global modeling. Because of its historical importance, in this subsection, the main points concerning the local linear predictor [

Suppose that a measured time series

Several choices for

A common difficulty of such approaches is that the data have to be separated into neighbourhoods. Thus given a point in the embedded space the closest neighbours to such a point must be found. It is well-known that for many methods most of the CPU time is spent in searching for close neighbours in the embedding space within the data [

Another drawback has to do with model representation. The local linear predictor does not have a closed form, in other words, it is not a global model. This prevents using the obtained model in any kind of analytical investigation.

The local linear predictor has been applied to several real-time series including Wolf's sunspot numbers [

In 1987 James Crutchfield and Bruce McNamara published a paper that became a reference in the field [

In [

In closing, it should be said that more than a method, Crutchfield and McNamara put forward a general philosophy for modeling chaotic data. With the exception of some issues that are specific to such data, their procedure does not differ substantially from works in the field of nonlinear system identification using linear-in-the-parameter models widely available at that time (see [

The paper by David Broomhead and David Lowe is among the four pioneering works that we have selected to start surveying the main developments in modeling of nonlinear dynamics and chaos [

A typical model, as proposed by Broomhead and Lowe, can be written as

This paper is curious in some aspects. First of all, it is clearly devoted to machine learning for purposes of classification. Several pages are devoted to the problem of approximating the exclusive-OR function. The last examples in the paper seem to be an afterthought and consist of the modeling of the doubling map

From the abstract of the paper we clearly read the objective of the paper: “numerical techniques are presented for constructing nonlinear predictive models directly from time series data” [

With respect to the model class, this paper is remarkable because it compared polynomial, rational, and RBF models, plus local linear predictions. Very few papers present results for more than one model class, a more recent exception to this rule is [

As for the use of “strict interpolation” (Broomhead and Lowe) or simply “interpolants” (Casdagli) or “approximants” (Casdagli), the situation is not completely clear. The paper mentions both. When describing general techniques (polynomial and rational models), Casdagli mentions the least squares solution. However, when presenting the RBF model the only case considered is that of choosing the model structure so as to have an inverse problem with unique solution. At the end of that section, the least squares solution is mentioned again, but as “ongoing research.” One of the critical problems of the “strict interpolation” approach is the total lack of robustness to noise. In fact, Casdagli does not consider any noise in the “global modeling” examples, but only in the “distinguishing noise from chaos” examples. In two occasions interesting remarks are made. In discussing modeling difficulties Casdagli says: “this does not appear to be due to numerical errors in the least-squares fitting and matrix inversion algorithms” [

The RBF models considered by Casdagli were of the form (compare with (

Casdagli investigated the accuracy of short-term predictions based on model iteration and the

By model class we mean the

The early paper by Cremers and Hübler had the clear objective to “obtain a concise description of an observed chaotic time sequence” [

The authors discuss a few aspects of embedding and the effect of dynamical noise. In their numerical examples no noise is considered. They mention sucessful estimation of the parameters for two systems: Lorenz and an autonomous version of the van der Pol oscillator, for which the following 4th-order approximation was used:

Breeden and Hübler later remarked that the 1987 paper also considered the discrete-time case, which is definitely not obvious. Then, so far the field of nonlinear dynamics is concerned, this paper seems to mark the beginning of building differential polynomial equations from data. Although no mention on important practical issues is made in this paper, such as structure selection, derivative and parameter estimation, it has the merit of having opened the way for a prosperous modeling class of techniques, to be surveyed in Section

The citation relationships among these five pioneering papers are illustrated in Figure

Quotation diagram.

In the description to follow, the order of the model classes mentioned is somewhat arbitrary. We follow a rough chronological order of the first papers in each class.

A number of papers appeared in the early nineties which had as a common goal fitting ordinary differential equations (ODEs) to observed data. As pointed out in Section

A similar representation was used by Gouesbet who started investigating under which conditions it was possible to reconstruct the vector field

Other polynomial expansions are to be found in the literature. Giona and colleagues investigated the use of a basis of polynomials to approximate both continuous-time and discrete-time dynamical systems [

Practical implementation of global modelling using continuous-time model requires an integration scheme. For instance, the explicit Euler integration scheme

Structure selection issues for continuous-time polynomials have been discussed in [

Other approximations for functions

A neural network is a model class that resembles some aspects of a brain. Conventional simplifications made for perceptron models are: (i) to take only one hidden layer of nodes, (ii) to consider the output node linear, and (iii) to consider all the activation functions

Schematic representation of some general model classes. (a) Typical perceptron model, (b) single-type basis function models, (c) multitype basis function models. The solid lines correspond to parameters that must be estimated. Dashed lines indicate absence of parameters. Therefore (a) is nonlinear in the parameters whilst (b) and (c) are linear in the parameters. Usually

The first papers to build this kind of models from chaotic data seems to have been the references [

A rare study of bifurcation diagrams achieved by neural models has been presented in [

The RBF model class is shown in Figure

A number of papers have described the application of RBF networks in the modeling of nonlinear dynamical systems and chaos [

The model class illustrated in Figure

The choice of monomials as basis functions constrains the resulting models to those cases in which the dynamics underlying the data can be approximated by a linear combination of nonlinear monomials. For systems that are more strongly nonlinear, other basis functions should be preferred. On the other hand, the choice of monomial basis functions enables building models which are more information dependent than models for which all the basis functions are of the same type.

Discrete-time polynomial models have been used in a number of papers within the field of nonlinear dynamics and chaos. Bagarinao and colleagues used this model class to reconstruct bifurcation diagrams from data [

Rational models are composed by the division of two polynomials such as (

Rational models for continuous-time systems were considered in [

Wavelet models, wavelet networks, or just

One of the first papers to use wavenets in order to model a chaotic system was [

Fuzzy models use internal variables which are linguistic. At a certain point of modeling the numerical variables must become linguistic by taking labels such as small negative large positive. The core of a fuzzy model then consists of a set of rules of the type: if

From the list of model classes surveyed in this paper, it seems to us that fuzzy logic models have been the least used to model nonlinear dynamics and chaos. One of the first papers on the subject seems to be [

Input-output models are not a class of models but simply mean that the model class caters for the use of input signals. By input we mean external, time-dependent signal(s). Therefore an input-output model will be nonautonomous.

For the modeling of input-output models it is necessary to record not only the output, but also the input. As a general rule, global differential equations built from data are autonomous (for an exception, see [

There have been attempts to develop an embedding theory for input-output models [

Nonautonomous chaotic models have been obtained in [

As surveyed above, there are various methods for building continuous-time models, typically in the form of ODEs. However, the data available is always discrete in time, and the simulation of ODEs is also carried out on digital computers. Therefore, at some stage some type of discretization of the ODEs must occur. In some methods such discretization is intentional [

A more fundamental question concerns the choice of the integration step of numerical integration schemes. Given an ODE, say that produces a chaotic attractor, what happens to the attractor if the integration is varied? If it becomes too great, certainly the attractor will change, but will the “new” attractor still be some attractor of the original system? These questions have been addressed in [

In few words, the problem of structure selection is that of deciding

In the field of system identification, the issue of structure selection for nonlinear models gained much attention by the late 80 s. However, in the field of nonlinear dynamics and chaos this trend was delayed a few years. In the early (and sometimes late!) 90 s several papers simply assumed the structure known [

Soon the practical importance of model structure selection started to be recognized and dealt with in various ways and in varying degrees of success [

The key point in structure selection is to choose a model structure that is as simple as possible, but also sufficiently complex to capture the dynamics underlying the data. One of the easiests (and less efficient) methods for model selection is called zeroing-and-refitting [

An alternative and simple way of tackling this problem for nonlinear systems has been proposed in [

One way of addressing the structure selection problem are to define some measure of complexity for a given model. In their paper Crutchfield and McNamara were concerned with quantifying and limiting the complexity of their models. They chose models that minimized the model entropy

Unfortunately, there is no definite solution to the model structure selection problem so far. Situations in which the current methods fail abound. Brown and colleagues report failure of the MDL criterion [

A key issue in modeling nonlinear dynamics is that of selecting an appropriate embedding space. In principle, this would include two stages: the choice of observables [

It has been duly pointed out that the problem of choosing an embedding in the context of model building is a bona fide stage of the modeling procedure [

To enable irregular embeddings (see Figure

Before, it is noted that synchronization has been used in parameter estimation problems [

One of the most commonly used algorithms for estimating unknowns in linear-in-the-parameters models is the least-squares algorithm or some generalization of it [

All such algorithms are based in some norm of one-step-ahead prediction error. An alternative would be to minimize some norm of a

In what concerns parameter estimation, an important difference between [

For

Breeden and Packard have discussed the use of genetic algorithm and evolutionary programming for solving a number of optimization problems which occur in the modeling of nonlinear dynamics and chaos [

The issue of model validation is vast. In order to cover such a wide subject in limited space, we will base this section on the paper [

Before actually starting to describe some results in the literature, a few remarks are in order. First and foremost, the challenge of model validation or of choosing among candidate models should take into account the intended use of the model. Hence, a model could be good for one type of applications and, nonetheless, perform poorly in another. In the context of this paper, the main concern is to assess the model dynamics. A different concern, though equally valid, that would probably require a different approach would be to assess the forecasting capabilities of a model. Second, it should be realized that two similar though different problems are: (i) model validation, which usually aims at an

Although very popular in other fields, the computation of various

Subjective though it is, the

Still in relation to the visual inspection of attractors, it should be noticed that in many practical instances there is not much more that can be done consistently. For instance, in the case of slightly nonstationary data, to compare short-term predictions with the original data is basically the best that can be done. Building a model for which the free-run simulation approximates the original data in some sense is usually a nontrivial achievement. (In this respect we rather disagree with [

Other

Meaningful validation can only be accomplished by taking into account the intended use of the model. A model that provides predictions consistent with the observed data will probably not be a good model to study, say, the sequence of bifurcations of the original system.

The use of model free-run simulations in the context of surrogate data analysis for model validation was suggested in [

A related method has been described in [

Graphical interpretation of (a) consistent prediction and (b) inconsistent prediction [

In the following two subsections we briefly discuss two other procedures which the authors have developed for model validation. The first one, following an early paper by Brown and colleagues [

As done in Figure

The rationale behind this procedure is as follows. Assume that the data

The difference

As it often happens in the realm of model validation, this procedure also is highly subjective, since it requires an ad hoc threshold, mentioned in the previous paragraph. In what concerns dissipative synchronization, it is well-known that in many cases by increasing the strength of the coupling (matrix

Therefore although the concept of synchronization could be useful in the context of model validation, it becomes apparent that some adjustments are required to render the procedure more practical. In [

A topological analysis usually starts by computing a first-return map to a Poincaré section of the phase portrait. This is indeed useful for extracting periodic orbits using a close return method although that other techniques can be used. In the best cases, the first-return map is made of

Chaotic trajectories and the periodic orbits constituting their skeleton are thus encoded over the symbol set

All topological properties are encoded in the template. Thus it is possible to extract topological invariants like linking numbers from a template construction. Linking number between two periodic orbits is the most often used topological invariant. It can be counted on a regular plane projection of the two periodic orbits. Each crossing (in the plane projection) between the two orbits is associated to

Topological analysis is surely the strongest validation method. Unfortunately, it is up-to-now restricted to 3D dyanmics. Topological validation of global models was performed for the copper electrodissolution experiments [

We start this section with a quotation from one of the pioneering papers of the field: “If extra information is available about a system in addition to a time series, such as explicit underlying partial differential equations or symmetries, then the inverse approach as presented here does not exploit such information. Consequently, for such systems it may be possible to improve significantly upon the inverse approach using other techniques” [

Despite opinions as the one just quoted, the problem of building models from data

The general setting is to have the dynamical data

In the realm of nonlinear dynamics, procedures have been put forward for building models using auxiliary information. A number of fixed-points were used in [

As early as 1987, the issue of modeling spatiotemporal nonlinear dynamical systems can be found in the literature [

Over fifteen years ago Robert Gilmore, referring to model building, mentioned: “There are at present two distinct methods to model data. One is analytic, and has not been extensively used. The other is topological, and has been used even less” [

The view presented in this work, as any review paper, is strongly influenced by the authors' experience. A varied list of introductory and review articles was included. A rather short list of benchmark models and widely available data with indication of works that have used such models and data were provided. Although every effort has been made to survey such a wide subject in a comprehensive way, it goes without saying that a complete list of papers is outside the authors' reach. Nevertheless we are sure that the present survey will serve as a good starting point for future works in modeling nonlinear dynamics and chaos.

As a helpful aid to those involved in developing new tools for modeling nonlinear dynamics and chaos, in this appendix we list some benchmark toy models and benchmark data sets (widely available) and point out papers that have discussed their modeling and analysis.

The logistic map was originally investigated by Robert May [

The logistic map was originally proposed in [

The Ikeda map was introduced in [

The autonomous van der Pol oscillator settles to a limit cycle and therefore can be modeled. This oscillator has been considered in [

The Lorenz system was introduced in [

Fuzzy models for this system were built in [

The well-known Rössler system was proposed in [

The Makey-Glass delay-differential equation was introduced in [

It must be pointed out that a nonautonomous oscillator or order

This oscillator was considered

This oscillator appeared in [

Several papers in the literature illustrate the proposed techniques using measured time series. It would be impossible to list all of them. In this subsection, however, we list those time series that are most commonly used because they are widely available and therefore constitute typical benchmarks.

In 1994 the Santa Fé Institute promoted a time series prediction competition, which is described in [

The Laser data was considered in [

These data can be found at

These data can be found at

These data results from the recrods of the Hudson Bay Company regarding the populations of lynx and hares. They can be found at

The electronic oscillator described in [

The so-called Chua's circuit [

This work has been partially supported by CNPq and CAPES (Brazil) and CNRS (France). The authors are grateful to the editors for their encouragement and continued assistance in the preparation of this survey.