This study deals with the external validation of simulation models using methods from differential algebra. Without any system identification or iterative numerical methods, this approach provides evidence that the equations of a model can represent measured and simulated sets of data. This is very useful to check if a model is, in general, suitable. In addition, the application of this approach to verification of the similarity between the identifiable parameters of two models with different sets of input and output measurements is demonstrated. We present a discussion on how the method can be used to find parameter deviations between any two models. The advantage of this method is its applicability to nonlinear systems as well as its algorithmic nature, which makes it easy to automate.
The external validation of simulation models is an important topic in many engineering problems. Some methods are known to support this process, but there is no general method which can deal with a broad class of systems. In this study, a method based on differential algebra is shown to be an efficient solution to the external validation problem. Differential algebra can be used to prove that a set of nonlinear polynomial differential equations are able to represent the input and output behaviour of a set of measurement or simulation data. Estimating any parameters of the system for this proof is not necessary. The advantage of this method is the decoupling of the validation of the mathematical model structure from the validation of the numerical, maybe physical, parameters of the system. This decoupling makes the external validation process much simpler. In addition, differential algebra can be used to show that different sets of measurement data were generated by the same system with the same identifiable set of parameters. In other words, the proposed solution can prove that the system structure and its numerical parameters did not change during several measurements.
The external validation of mathematical models is one of the major challenges in simulation technology. Model validation is substantiating that the model, within its domain of applicability, behaves with satisfactory accuracy consistent with the study objectives. Model validation deals with building the right model [
A large number of model validation methods are known, as well addressed by Balci [
Forrester and Senge proposed to conduct a model on the basis of the model-builder’s personal knowledge and then extend the structural verification test by including criticisms from others with direct experience of the real system [
Currently, it can be concluded from a thorough review of the literature that no general automatic method for the model validation problem is available. The reasons for this may be attributed to the fact that several promising methods available for use are only applicable to systems with special properties such as linear systems or to academic examples. Other broad ranges of methods have only limited benefits compared with the well-known and simple methods, including the direct comparison of simulation results with measurement data.
Given the importance of validation methods, it is surprising how only few model validation methods are implemented in standard software packages. One example of a successful implementation is the “Reality Check” in the simulation software “Vensim.” The idea is based on the definition of cases the model has to fulfil and the automatic check, if the model behaves as suggested. Each “Reality Check” test consists of a test input coupled to an expected behaviour. It is important to notice that these tests refer only to the behaviour and not to the structure [
In this study, an interesting method from the field of differential algebra is introduced, which has the potential to overcome some of the current model validation problems.
Section
A promising approach to model validation was shown by Ljung and Glad [
Differential algebra originated in 1950 based on the work of mathematician Ritt [
Starting from a set of differential polynomials,
A key concept of differential algebra is the algorithmic reduction of the differential equations [
Two sets of differential polynomials which are reduced with respect to each other are called autoreduced sets, where the sets with the lowest possible rank are called the characteristic sets [
On the basis of these concepts, Ljung and Glad [
In particular, in large and complex models with a high number of degrees of freedom, probing the reason for deviations between the simulations and measurements is very difficult. Moreover, proving that two or more sets of measurement data were generated by the same system with the same parameterization is advantageous. In general, the deviations may be due to structural or parametric problems of the simulation model. Structural problems are problems with the mathematical structure of the equations. Parametric problems are problems with the numerical values of the (possibly physical) parameters of the model. In other words, structural problems are those where the equations of the simulation model are not able to represent the dynamics of the system under measurement. This is the case if the model assumptions are not correct, for example, due to unmodelled dynamics. In contrast to structural problems, parametric problems are caused by insufficient numerical values of the model parameters. In the case of mechanical systems, these values might be the mass or spring constants.
To improve the simulation model, it is advantageous to distinguish between structural and parametric errors, which is difficult in practice. One widely accepted method for the improvement of models is the identification of the parameters of the system. The identified parameters may differ from the real system parameters, for example, due to unmodelled dynamic effects or measurement noise. Unfortunately, the parameter identification algorithms may not determine perfect parameter values. Furthermore, the reason can be a structural problem of the model, which implies that no optimal set of parameters exists for this model. Moreover, other reasons for failure in parameter identification could be, for example, numerical problems or inappropriate start values for the iterative parameter identification. If the parameter identification succeeds and the numerical parameters are close to the expected and physically meaningful parameters, it is possible to conclude that there are no structural problems inside the simulation model. However, unsuccessful parameter identification is not a helpful indicator to find the reason for model deviations, because no conclusion as to the source of the deviation can be derived. If the model has a unique set of parameters and the parameter identification is initiated with meaningful values close to the real values, then this could serve as a useful method for the improvement of the model.
One general requirement for the estimation of a unique set of parameters is the global identifiability of the parameters. Following the work of Ljung and Glad [
Since structural identifiability is only a property of the system structure and independent of the measurement data, it is a very important property for a guaranteed validation of the model. It is known that the parameters of structural unidentifiable models can only be estimated in combination with other parameters. For example, if the product of two parameters is the only identifiable one, a perfect accordance of the input and output behaviour with the estimated parameters can mask significant errors in the individual parameters. A high value of one parameter can be compensated by a low value of the second parameter which leads to an inconclusive result of the validation exercise [
One elegant algorithmic method for identifiability analysis based on differential algebra was presented by Ljung and Glad [
Starting from a very general description, the system
To compute the characteristic set, it is necessary to define a ranking for the variables and the inputs and outputs of the system. A ranking is a total ordering of variables, which determines the sequence of variables to be eliminated. To obtain a parameter-free characteristic set
It was shown by different authors in [
A characteristic set generates the same differential ideal as generated by the differential polynomials of the model to be validated but in a special form, which is easier to validate. With the knowledge of a characteristic set in the base ring of the model, finding an input and output relation without any states and without any parameters of the model is possible [
It is to be mentioned that the differential algebraic elimination algorithm requires equations which are polynomial in
A general algorithm for transforming nonlinear systems with linear control inputs (
The characteristic set
Quarter car model.
The first example shows the application of the characteristic set for the structural validation of a model. All parameters of the model are shown in Table
Parameters of the quarter car model.
Parameter | Value |
---|---|
|
362.5 kg |
|
35000 N/m |
|
4000 Ns/m |
|
48 kg |
|
239333 N/m |
|
0.1 Ns/m |
Road roughness | “Very poor” according to ISO 8606 |
The displacement of the wheel relative to the chassis can be measured and denoted as
After the discretization of (
Quarter car model with exact structural accordance.
To show that the proposed approach is able to detect deviations in the structure of the model, the linear system equation, (
Quarter car model with nonlinearity and unaccomplishable structural accordance.
Quarter car models with different input and output signals.
Due to the high potential of the structural validation of models, this approach is extended and further developed in the current section. We will show how differential algebra can be used to prove that two sets of measurement data can only be generated by a model with the same structure and the same parameters. For this proof, the knowledge of these numerical parameters is again not necessary.
The question on whether two measures are related to only one system becomes interesting when we want to know whether the parameters change during one or several tests. This is very useful to prove, for example, if a system is time-invariant or if the different sets of data belong truly to one and the same system. This becomes all the more important if the simulation model answers perfectly to one excitation signal and poorly to another excitation signal.
By assuming for simplicity a single-input single-output system (SISO) and two sets of measurement data, the following ranking can be obtained:
The main idea behind this new ranking is to assume that two systems with the same structure of the underlying differential equations have the same physical parameters but different input and output signals. After the elimination of these equal and constant parameters, by using differential algebra, the resulting characteristic set would vanish, if the correct input and output signals are inserted into the characteristic set. The phrase “correct signals” here implies that the assumption of equal equations and equal parameters of both systems is correct.
The third example below shows how the characteristic set can be used to prove the equality of the parameters of one model and the different sets of measurement data. Again, the quarter car model with the same parameters as in Example 1 is used for the generation of two different input and output signals. The only difference is another street profile for the second set of data, all parameters being equal. The characteristic set is generated by modelling the same model of (
The equations of the system are therefore equal, except for the variables of the inputs and outputs:
Structural accordance of the quarter car models corresponding to the system in Figure
As a counterexample, the chassis mass
Quarter car models with different chassis mass.
This simulation shows that the approach is very useful to detect changes in parameters between different sets of measurement data, without any knowledge of their numerical values.
If the structural validation shows a perfect accordance of the structure with all signals and additionally a change of the parameters has been detected with the proposed method, inferring which parameters have changed is possible. In order to accomplish this, the elimination ranking from (
Suppose that the first parameter of the system is the only one that has changed; the following ranking can be used to generate such a characteristic set:
If a test of the equality of parameters shows changes between different sets of measurement data, as in Example 4, it is necessary to isolate the changed parameters. The isolation can be performed by introducing different variables for the parameters of the two models.
In this example, the chassis mass
Quarter car model with different input and output signals and different chassis mass.
Equations (
Quarter car models with different input and output signals and different chassis mass.
As expected, the residual is very small compared with the input and output signals of the model. This shows that the residual of the validation equation, that is, (
As a counterexample, the damping
Quarter car models with different input and output signals as well as different chassis mass and different damping.
It was already mentioned earlier that unambiguous conclusions about changed parameters are only possible for identifiable parameters or identifiable parameter combinations. The reason is the possible equality of the input and output behaviour of the systems with different unidentifiable parameters.
By deriving the transfer function from (
Quarter car models with different input and output signals as well as different chassis mass, damping, and spring constant.
The previous example showed that for a clear assessment about changed parameters it is necessary only to consider the identifiable parameters of the system or to substitute the nonidentifiable parameters by parameter combinations which are identifiable. This result is only restrictive if it is assumed that a few parameters may change because different combinations of parameters can be nonidentifiable. In the last example, in the event of an error in the displacement sensor, with which the relative velocity is also obtained by numerical differentiation, a multiplicative fault of this sensor cannot be distinguished from a changed chassis mass. Both changes can lead to the same input and output behaviour. If it is only necessary to check whether any of the parameters are equal, like in Example 3, this problem is not as restrictive as it seems to be. The reason is that a change of one parameter has to be compensated exactly by another parameter which is improbable. For example, a multiplicative fault of the sensor, like in Example 7, can be detected with our proposed approach in this study.
We have demonstrated that methods from differential algebra can be very useful for external validation of simulation models. Because the examples presented were linear in their parameters, the validation problem with a common parameterization could have been solved by simple linear algebra as well. However, the advantage of the proposed differential algebraic approach is that it can also handle nonlinear systems. In addition, these methods have an algorithmic nature, which allows for ease of automation. With algorithms from differential algebra, the complex external validation problem can be divided into much simpler problems and we can test whether the equations of a model are generally able to fulfil a set of measurement data. Our study has shown examples to prove that two or more sets of data are generated by the same model parameterized by the same set of identifiable combinations of parameters. It was also shown that if two or more sets of data were generated by the same structure of a model, differential algebra can answer which parameters changed between different sets of measurement data. However, for usage with industrial relevant systems, a residual which is less sensitive against measurement noise and ideally linear in terms of the structural accordance has to be defined. Moreover, there is a need to enhance the efficiency of the algorithms for the computation of the characteristic sets. Nevertheless, to conclude, differential algebra shows great potential for the external validation of simulation models.
It has to be mentioned here that there are several other applications for this method, which have to be considered further in future. This method may be used to check the exact approximability of systems over time and to prove that requirements of systems can be fulfilled by at least one parameterization of the considered system, during a technical development process. As an example, this might be used to prove that there is at least one set of parameters for a selected controller with which the system has exactly the desired input and output behaviour.
The authors declare that there is no conflict of interests regarding the publication of this paper.