Clustering Input Signals Based Identification Algorithms for Two-Input Single-Output Models with Autoregressive Moving Average Noises

This study focused on the identiﬁcation problems of two-input single-output system with moving average noises based on unsupervised learning methods applied to the input signals. The input signal to the autoregressive moving average model is proposed to be arriving from a source with continuous technical and environmental changes as two separate featured input signals. These two input signals were grouped in a number of clusters using the K -means clustering algorithm. The clustered input signals were supplied to the model in an orderly fashion from cluster-1 up to cluster-K . To ensure that the output signal can be best predicted from the input signal which in turn leads to selecting good enough model for its intended use, the magnitude-squared coherence (MSC) measure is applied to the input/output signals in the cases of clustered and nonclustered inputs, which indicates best correlation coeﬃcient when measured with clustered inputs. From collected input-output signals, we deduce a K -means clustering based recursive least squares method for estimating the parameter of autoregressive moving average system. The simulation results indicate that the suggested method is eﬀective.


Introduction
Most systems that have been inferred to prove many of the assumptions proposed in the area of system identification focused on single-input single-output systems [1][2][3][4]. Recently, the focus has been on studying system identification of multivariable systems in order to deal with an appropriate modelling and estimation of dynamic systems operating in industrial applications and process control [5][6][7].
Several methods and techniques addressed system identification problems in multivariable models as in [8][9][10][11][12]; the researchers in these studies depend on hierarchical identification principle that decomposes a multi-input system into two subsystems, one containing a parameter vector and the other containing a parameter matrix [10]. e proposed method in [13] was dependent on the concept of coupling identification to avoid matrix inversion in multivariable least squares in order to reduce computational time in the algorithms. Many algorithms are used to implement these methods such as least squares recursive algorithms or iterative least squares algorithms without mentioning the impacts of the excitation signals on the proposed models.
ere are identification methods focusing on the modified effects of the excitation signals on the design of system identification experiments and they yielded acceptable results in modelling and estimation process of multivariable system. ese proposed methods are based mainly on applying filtering techniques to the collected inputoutput data in order to generate more accurate results as in [14][15][16][17]. We aim at the same objectives of these researches by considering effective preprocessing on the excitation signals of multivariable systems.
In this paper, we tried to focus on the collected input data to identify multiple-input single-output systems by introducing unsupervised learning approach such as the clustering techniques applied only to input signals by means of K-means clustering algorithm. To ensure that the system output is best predicted from the proposed excitation signals, the similarities in the frequency contents between modified input signals and generated output signal are investigated using the magnitude-squared coherence measure [18].
Most of the works that relied on the use of clustering in the system identification process were based on collected and recorded data from input and output without even assuming any slightly modified effect on the input signals. In [19], the submodels of the PieceWise AutoRegressive systems with eXogenous input (PWARX) are obtained through an algorithm inspired by competitive learning. In particular, the proposed method exploits a process of fuzzy clustering to obtain a subset of representatives from the original data set. Reference [20] assumed the number of modes of the PWARX system to be unknown and proposed a split-and-merge clustering algorithm to estimate the correct number of modes. Reference [21] discusses the use of correlation clustering algorithms for robust identification of PWARX models with reduced complexity. A PWARX model of the electronic throttle regulating the air inflow of a car engine has been identified based on clustering in [22]. e refinement algorithm proposed in [23] is repeatedly applied to the estimated clusters to improve both the data classification and the parameter estimation. Since, in complex engineering system, reliability is considered as the main issue, K-means clustering is used in [24] to estimate the reliability of the complex engineering system to obtain three or more subsystems called cluster systems and analyze the key performance parameters of constructs, which are known as family system that contains only subsystems with similar performance; then the contribution of these subsystems in the cluster system leads to estimating reliability of the whole system. Many of the above-mentioned research works were based on clustering the regressor vector to divide the system model into multiple candidate models for investigation and analysis. We pointed out here that the regressor vector contains both input and output signals. e fundamental idea of the proposed method in this paper is to cluster only the input signals using K-means clustering algorithm without clustering the whole regressor vector. Based on this idea, we present a clustered input signals based recursive least squared algorithm. Given this recursive algorithm, we can produce more accurate parameter estimation compared to existing multivariable recursive estimation methods, for example, [14,15]. Compared with the clustering techniques used in a PieceWise affine ARX model [19][20][21], the modified recursive algorithm in this paper can produce highly accurate estimation.
Furthermore, since the clustered input signals are assumed to be similar due to the similarity measure of the Kmeans algorithm, the same system properties will be excited and hence better identification performance will be obtained [25]. e rest of this paper is organized as follows. Section 2 gives the general model description and the formulation of the autoregressive moving average (ARMA) identification model with some important definitions. Section 3 derives a clustering input signals based algorithm for two-input single-output model. Section 4 gives the recursive extended least squares algorithm used for comparison. Section 5 gives the magnitude-squared coherence measure used to obtain the degree of similarities between input signals and the output signal of the proposed model. Section 6 provides the steps used to perform the K-means clustering algorithm. An illustrative example to demonstrate the effectiveness of the proposed algorithms is provided in Section 7. Finally, concluding remarks are given in Section 8.

The Model Description
Firstly, let us introduce some important notations. e superscript T denotes the matrix transpose, the symbol I n stands for an identity matrix of order n, 1 n represents an n-dimensional column vector whose elements are 1, ‖X 2 ‖ stands for the norm matrix X, X :� A denotes that X is defined by A, the estimate of parameter η at time t is denoted by η(t) , z − 1 represents a unit backward shift operator defined as z − 1 x(t) :� x(t − 1), and p 0 is a large positive number [26].
Consider the two-input single-output system, described by the autoregressive moving average model, as shown in Figure 1, with a mathematical model as where u 1 (t), u 2 (t) are the two features of the input data points of the system; y(t) represents the output data points of the system; v(t) { } is a white noise sequence with zero mean and variance σ 2 ; and P(z), Q 1 (z), Q 2 (z), and R(z) are the polynomials with known degrees in the unit backward shift operator z − 1 as in the following equation: Assume that y(t), u 1 (t), u 2 (t), and v(t) are strictly proper; that is, their values are 0 for t ≤ 0, and the orders n p , n 1 , n 2 , and n r are known.
From two inputs u 1 (t), u 2 (t) and the output y(t) , the model parameters are to be identified.
Define the parameter vector η and the information vector ϕ(t) as 2 Complexity η :� η s η n ∈ R n , n :� n p + n 1 + n 2 + n r , Based on recursive least squares in [27], the identification model of ARMA model can be expressed as e parameter vector η contains all parameters to be estimated.

The K-Means Clustering Algorithm
K-means clustering is an unsupervised learning approach. e unsupervised learning approach is given by grouping data with similar characteristics; this grouping process is known as clustering, which consists in separating each group with different characteristics in sets called clusters [28]. As in [29,30], K-means clustering algorithm works with three main steps: initialization step, assignment step, and update step. Firstly, in the initialization step, the process starts with selecting data U ij (i � 1, . . . , n; j � 1, . . . , m) where n is the amount of data to be carried out in clustering and m is the number of variables (data dimension), and, initially, the centre of each cluster c kj (k � 1, . . . , k; j � 1, . . . , m) is determined arbitrarily [30]. Secondly, in the assignment step, the distance such as Euclidian distance (other types of distance measures can be applied depending on the application) of data-i to centroid k is calculated according to the following equation [30]: where d ik is distance of object i and centroid k, m represents data dimension, u ij denotes coordinates of object i in dimension j, and c kj denotes coordinates of object k in dimension j.
If the distance of data to specific cluster centroid k has the smallest value when compared to another cluster centroid, then the data will be a member of the cluster k. is can be examined according to the following criterion index [31]: Finally, in the update step, after classification of data for each cluster, centroid value can be calculated by finding the average value of the data which are members of the cluster using the following equation [31]: where u ij ∈ cluster k and p denotes number of cluster k members. Figure 1: Two-input single-output autoregressive moving average model.

Complexity 3
As in [30], the steps of the K-means clustering algorithm to cluster input signals used in the proposed model are listed as follows: (1) Suppose given input data matrix U � u ij with i � 1, 2, . . . , n ; j � 1, 2, . . . , m.
(2) Determine the number of clusters (k), with centroid chosen randomly. (3) Calculate the distance of each input data point to the centroid using equation (5). (4) Classify input data into clusters with the shortest distance using equation (6). (5) Calculate the new centroid using equation (7). (6) Repeat steps 3 through 5 until there is no more input data movement to another cluster.
In our proposed method, we apply the above algorithm to the model input signals which are assumed to be represented as two input features collected from different sources. e optimal number of clusters was chosen according to the elbow method in [32]. In elbow method, a decreasing cost function is analyzed for different values of k and it can be defined as the sum or average distance from all points in the data set to its centroid. Figure 2 shows the candidate elbow point as an optimal point to select the best number of clusters according to the best total sum of distances. As seen from Figure 2, the optimal value chosen for the illustrative example used to verify the proposed method is k � 2.

The Clustered Input Signals Based Recursive Least Squares (CIB-RLS) Algorithm
Our goal in this paper is the identification of the ARMA model based on clustered input signals. We start with the assumption that the two-input signals represent two different attributes supplied from a source with continuous technical and environmental changes. e first step is to apply these signals to the K-means algorithm to group them in several k-clusters to share a common statistical property.
Let the two-input data set signals u 1 (t) and u 2 (t) be applied to the K-means algorithm; the algorithm grouped these data sets into k-clusters, known as cluster-1 to clusterk; see Figure 3 below. Every cluster contains part of data set belonging to u 1 (t) and u 2 (t). Assume this data set to be known as u 1,k (t), which represents a part of u 1 (t) in the cluster k, and u 2,k (t), which is a part of u 2 (t) in the same cluster k; Figure 3 illustrates the assumptions. At this point, we select the parts of input signals u 1k (t) and u 2k (t) from each generated cluster to form new modified inputs as u 2modified (t) :� u 2,1 (t), u 2,2 (t), . . . , u 2,k (t) T .
e modified input signals, represented by equation (8) and equation (9), are then applied to the general system under investigation as shown in Figure 4. e proposed CIB-RLS algorithm will be as follows: η n (t) � r 1 (t), r 2 (t), . . . , r n r (t) T .
(2) Let t � 1; set the initial values of the parameter estimate vectors and the covariance matrix according to (20), and give the parameter estimation precision ε � 0.01. (3) Collect input data u 1 (t) and u 2 (t).
(4) Select the optimal number of clusters K according to u 1 (t)and u 2 (t) collected by using elbow method.

The Recursive Extended Least Squares (RELS) Algorithm
e main algorithm used to estimate the model parameters in our proposed method is the recursive extended least squares (RELS) algorithm. With the aid of the algorithm in [14], the recursive least squares algorithm of the ARMA model described by equation (1) with its identification model expressed in equation (4) can be applied. Before we proceed, based on this identification model, the unknown variables v(t − i) in the information vector ϕ(t) in (3) are replaced with their estimates v(t − i) [14]. So, the recursive extended least squares (RELS) algorithm [33] is as follows: , p 2 (t), . . . , p n p (t), q 11 (t), q 12 (t), . . . , q 1n 1 (t), q 21 (t), q 22 (t), . . . , q 2n 2 (t) T , η n (t) � r 1 (t), r 2 (t), . . . , r n r (t) T . (21)

The Magnitude-Squared Coherence (MSC)
e magnitude-squared coherence (MSC) is a measure that estimates the extent to which one real-or complexvalued signal can be predicted from another real-or complex-valued signal [18]. In our work, we used these measures to investigate the similarities in the frequency content between input signals and the output signal used to identify the proposed ARMA model. Comparisons were done between the cases where the two inputs are clustered before being applied to the model and the two inputs are applied directly without passing to the clustering algorithm. If we define the power spectrum at frequency ω of the input signal u(t) and the output y(t) as S u (ω) and S u (ω), respectively, and the cross power spectrum as S uy (ω), then the magnitude-squared coherence can be given as where S u (ω) , S y (ω) ≠ 0. e value of MSC lies in the range 0 ≤ M uy (ω) ≤ 1 which is analogue of the correlation coefficient in statistics.
In our proposed model, the MSC gives the better correlation between the clustered applied input signals and the 6 Complexity measured output signal compared to unclustered input signals with measured output. Figures 5 and 6 represent the MSC for the illustrative example in Section 7 with noise variances of σ 2 � 0.10 2 and σ 2 � 0.40 2 , respectively. e dotted line represents the case in which the input signals are unclustered, whereas the solid line represents the case of clustered input signals.
As seen from these figures, it is clear that best measure is obtained in the case of clustered input signals. According to the type of signals, we can choose the best values of sampling rate, number of overlaps, and number of fast Fourier transform points (FFT) to achieve suitable results of MSC calculation.

Illustrative Example
In this illustrative example, we use MATLAB version R2020a and Wolfram SystemModeler software for simulation [34] to run recursive lest squares with K-means algorithms and to collect input and output data, respectively.
Consider the electric circuit model with two-input voltage and a single-output voltage measured from a capacitor C as shown in Figure 7. From the circuit of the Figure 7 and the concepts in [35][36], we can obtain the state-space model as e circuit parameter is assumed to be according to Table 1, where x 1 represents the inductor current i L as the first state and x 2 represents the capacitor voltage v c as the second state. From the above state-space model, we can find the continuous and discrete transfer functions [35], respectively, as y(t) � 0.2273s + 0.2503 s 2 + 1.7238s + 0.8552 u 1 (t) + 0.1697 s 2 + 1.7238s + 0.8552 Magnitude-squared coherence, the output with unclustered inputs Magnitude-squared coherence, the output with clustered inputs Magnitude-square coherence, the output with unclustered inputs Magnitude-square coherence, the output with clustered inputs  Figure 7: System schematic diagram with two modified inputs signals u 1modified and u 2modified . e output y is measured from the voltage across the capacitor 2.2 F, and the circuit is simulated with Wolfram SystemModeler software for simulation [34]. 8 Complexity From the circuit of Figure 7 and the concepts in [35,36], we can obtain the state-space model as Table 2: e parameter estimates and their errors with recursive extended least squares algorithm (σ 2 � 0.10 2 ).  Table 3: e parameter estimates and their errors with clustering inputs based recursive least squares algorithm (σ 2 � 0.10 2 ).     with sampling time T s � 0.05 sec. Consider the discrete polynomials model in (25) above, the proposed ARMA model to be identified can be written in the form e parameter to be estimated is e inputs u 1 (t) , u 2 (t) are taken as a chirp signal with different ranges of frequency and different amplitudes and carry the characteristics of persistent excitement using simulation program in [34], and v(t) is taken as a white noise sequence with zero mean and variances of σ 2 � 0.10 2 and σ 2 � 0.40 2 , respectively.
Applying the recursive extended least squares algorithm for a comparison and clustering inputs based recursive least squares algorithm to estimate the parameter of the electric system, the parameter estimates and their errors are shown in Tables 2-5, and the estimation errors δ � ‖η(t)t − nη‖/‖η‖ versus t for σ 2 � 0.10 2 and σ 2 � 0.40 2 are shown in Figures 8  and 9, respectively.
rough Tables 2-5 and Figures 8 and 9, we can infer the following observations: (i) With a suitable selection of the input signals and choosing the optimal value of k in K-means clustering algorithm, the parameter estimation error decreases gradually with the data length t increasing. is, in turn, shows that the proposed algorithm gives effective results. (ii) Since, at a certain time, the model is excited with clustered input signals that share the same characteristics with a high degree of similarity due to the K-means clustering algorithm, the same system properties could be excited and the accuracy of the model identification is improved. (iii) Compared with the recursive extended least squares algorithm, the proposed clustering inputs based recursive least squares algorithm has better identification performance. (iv) Using clustering inputs based recursive least squares algorithm, the parameter estimates converge to their true values faster than using recursive extended least squares algorithm.

Conclusions
is paper investigates one aspect of unsupervised learning effects on the system identification field. e clustering inputs based recursive least squares algorithm for the twoinput single-output system with moving average noise is proposed by introducing clustering techniques to the excitation input signals. e simulation results indicate that the proposed algorithm can yield more accurate parameter estimates under certain conditions (i.e., selection of the optimal value of K-means) compared with the recursive extended least squares algorithm. e proposed method can be extended to multiple-input multiple-output systems and stretched forth to investigate the identification of nonlinear systems and time-delay systems.

Data Availability
e data used to support the findings of this study are included within the article.

Conflicts of Interest
e author declares that there are no conflicts of interest regarding the publication of this paper.