Optimal Control for a Linear System Subject to a General ARIMA Disturbance

A novel run-to-run (R2R) control algorithm based on Kalman filter approach is proposed to deal with a linear system with a general ARIMA(p, d, q) process, in the presence of measurement error and adjustment error together with a random initial bias. We mathematically prove its optimality. The performance of the newly proposed controller and the proportional-integral (PI) controller is evaluated and compared under multiple scenarios through Monte Carlo simulations. Almost all the results reflect the new controller’s superiority.


Introduction and Motivation
Optimizing a sequence of actions to obtain some future goal is the general topic of control theory (see [1]).Many optimal control methods have been widely used in actuarial science (e.g., [2,3]), production management (e.g., [4][5][6]), and quality management (e.g., [7][8][9]).Specifically, disturbance rejection is one of the major concerns in quality control problems, such as machine setup adjustment problems and R2R process control problems in semiconductor manufacturing (see [10,11]).Grubbs [12] originally studies the machine setup adjustment problem and develops an optimal adjustment rule for a linear system with normal disturbances, which is referred to by Trietsch [13] as the "harmonic rule." Vander Wiel et al. [14] give an optimal control algorithm for a linear system with ARMA(1, 1) disturbances.For a linear system with IMA(1, 1) disturbances, Ingolfsson and Sachs [15] and Box et al. [16] introduce the exponential weighted moving average (EWMA) control algorithm and prove its optimality; He et al. [17] recently develop an optimal control algorithm, the ARMA controller, for a linear system with a general ARMA(, ) disturbance.
In this paper, we will extend the results in [17] and develop an optimal control algorithm for a linear system with a general ARIMA(, , ) disturbance, where  = 1, 2, . ... The ARIMA disturbance has been widely used to describe process dynamics (see [18][19][20][21]).Since ARIMA(, , ) processes can be used to model a large class of nonstationary disturbances, many machine setup problems and R2R control problems can be solved under this new framework.Similar to [17], the newly proposed controller will consider a more realistic case such that both measurement error and adjustment error exist, and the initial bias of the process is a random variable.We present the problem as follows.

Mathematical Problems in Engineering
Neither   nor   can be measured or observed directly.At time  ≥ 1, suppose we need to make an adjustment of magnitude Δ  to   to bring the process output to target  in the next run.That is, where   is the adjustment error and   ∼ WN(0,  2  ).In practice, the process adjustment Δ  is assumed to be done by adjusting one controllable factor,   , via the following model: where  is called the process gain and Δ  =  +1 −   .Assume (    ) = (    ) = (    ) = 0,  = 0, ±1, ±2, . ... The initial value  0 is assumed to be a random variable with ( 0   ) = ( 0   ) = ( 0   ) = 0,  = 0, ±1, ±2, . .., and ( 0 ) = ĝ0 , Var( 0 ) =  2  .Without loss of generality, the target  is assumed to be 0 in the rest of this paper.For a positive time index , we hope to determine the optimal Δ  ,  = 0, 1, . . .,  − 1 that satisfy min where   (⋅) is the expectation operator conditional on all the information until time .Problem ( 6) is a finite-horizon problem.When  → ∞, we get the infinite-horizon problem that searches the optimal Δ  ,  = 0, 1, . . .satisfying min The purpose of this paper is to find the solutions to both problems ( 6) and (7).The rest of this paper is organized as follows.In Section 2 we derive the state-space representation of the system and get the recursive estimation formulae of the system states using Kalman filter.In Section 3, we develop a control algorithm for the system and prove its optimality without normal distribution assumptions.We further give the implementation steps of the controller in practice.Simulation studies are done in Section 4 under multiple scenarios.Section 5 gives an illustrative example for the application of the control algorithm.Concluding remarks are included in Section 6.
Theorem 1.The linear system (1)-( 5) has a state-space representation as follows: where  and  are the  × 1 and  ×  matrices defined by if  > 1;  and  are the  × 1 and  ×  matrices defined by A proof of Theorem 1 is presented in Appendix A. For simplicity, in the rest of this paper we denote where 1  is defined to be a  × 1 vector of ones.That the problems of optimal control and state estimation can be decoupled in certain cases is one of the most fundamental principles in feedback control theory.This is known as the separation principle (see [23]).The Kalman filter produces the statistically optimal estimate of the state of system (8)- (9).Note the fact that the disturbances of ( 8) and ( 9) are correlated to each other, so we should use the Kalman filter formulae for correlated measurement and process noise.At any time , let us define R− +1 =   (R +1 ) and  − +1 = Var  (R +1 ) and assume R− 1 = [ ĝ0 0 0]  and  − 1 = diag( 2  , 0, 0).Using the results on page 123 in Lewis' book [24], we can directly get the following Lemma 2. Lemma 2. For the systems ( 8)-( 9), we have the following recursive formulae: where   is called the modified Kalman Gain that

The Optimal Control Algorithm
In this section, we will derive an optimal control algorithm without the normal distribution assumptions on R 1 ,   ,   , and   ,  = 0, ±1, . ... This optimal control algorithm can be applied to both the finite-horizon and infinite-horizon problems.
Proof.For a given positive time index , let ) and   ≡ 0. We can get the Bellman equation as From ( 8), it is evident that .Using the fact that and the property of trace operator, we obtain that From ( 13) and the fact that h  ℓ ≡ 1, we have Let +1 is updated based on ( 14) and (15).Now (19) changes to

Mathematical Problems in Engineering
When  =  − 1, we get Solving the first order condition, we get that Putting ( 25) into ( 24), we get we have Solving the first order condition, we get and Repeating the procedures above, we can prove that ( 17) is the optimal adjustment strategy at time  and get At last, we can derive (18) by putting ( 17) into (13).Since ( 17) is irrelated to the given time index , the control strategy also solves the infinite-horizon problem.Then we finish the proof.
In practice, the steps to implement the proposed control algorithm are as follows.
Step 2. Collect the new observation   and compute the adjustment Δ  based on (17).
Note that Step 0 is an off-line procedure and Step 1 to Step 4 are on-line procedures.As the newly proposed control algorithm is specially designed for adjusting any ARIMA disturbances, we call it ARIMA controller in the rest of this paper.

Simulation Study
In this section, we will study the performance of the ARIMA controller under multiple scenarios through Monte Carlo simulations.For simplicity, we only focus on the system (1)-( 5) with   being an ARIMA(1, 1, 1) disturbance, although the ARIMA controller can be applied to any general ARIMA(, , ) disturbance.Without loss of generality, we set the parameters in the ARIMA(1, 1, 1) disturbance as  1 = 0.7,  1 = −0.3, and   = 1.For comparison, a  controller's performance is also evaluated.The  controller is widely used in feedback control, and it involves two separate constant parameters: the proportional and the integral values, denoted by   and   .The steps to implement a  controller can be found in [25].
Tuning a  controller could be a challenging task.There are many ways of tuning, such as Ziegler-Nichols tuning, lambda tuning, robust loop shaping, optimization methods, and others (see [26]).Among them, optimization methods are powerful and direct ways.In order to do fair comparisons with the ARIMA controller, we need to choose   and   to make the  controller have the best performance for controlling the specified ARIMA(1, 1, 1) disturbance.To the best of our knowledge, there are no closed-form expressions for   and   to optimally control a ARIMA(1, 1, 1) process; then experimental tuning of the  parameters has to be performed.The procedure of optimally choosing   and   in this paper is described in Appendix B. For the ARIMA(1, 1, 1) disturbance with  1 = 0.7 and  1 = −0.3,we get   = 1.223 and   = 0.190 according to the optimization procedure.
Four scenarios are examined in the following simulations.In Scenario 1, the effect of the measurement error and the adjustment error on the ARIMA controller and the  controller is investigated; in Scenario 2, the effect of the process initial bias  0 on the two controllers is explored; in Scenario 3, we study how the estimate of the process gain β affects the two controllers; in Scenario 4, we investigate both controllers' performance when the disturbance parameters are not estimated accurately.
We do 1000 replications for each case and run 100 steps for   in each replication.The first 100-run mean square error (MSE) of   is computed.The Average MSE (AMSE) of the 1000 replications is reported in Tables 1-4.Also the standard error in the AMSE (SEAMSE) is computed, which is where SDMSE is the standard deviation of mean square errors, and the number of replicates here is 1000.AMSE measures the performance of the controllers, and SEAMSE reflects the variability of the AMSE.The smaller the AMSE is, the better performance the controller has.

Effects of Measurement Error and Adjustment Error.
In order to focus on the effects of the measurement error   and the adjustment error   on the controllers, we set the initial process bias  0 = 0, that is, ĝ0 =   = 0, and set the estimate of the process gain β to the true value .We also assume all  the ARIMA(1, 1, 1) parameters can be accurately estimated, that is, φ1 =  1 and θ1 =  1 .Table 1 and Figure 1 show the performance of the two controllers affected by the measurement errors and adjustment errors.We set the values of   and   to be 0, 0.5, 1.0, 1.5 and 2.0, respectively, so there are 25 pairs of (  ,   ).For each pair of (  ,   ), we repeat the simulations.We can observe that for both controllers the AMSE of   increases when   or   increases.However, the AMSE of   using the ARIMA controller is always smaller than that using the  controller at a significant level for all the pairs of (  ,   ).

Effects of Initial Bias.
For varying sizes of ĝ0 and   in the prior distribution of the process initial bias, Table 2 and Figure 2 show the results of the AMSE of   using the ARIMA controller or the  controller for the ARIMA(1, 1, 1) disturbance.In order to focus on the effects of the initial bias, we set all the   and   to 0, β = , φ1 =  1 , and θ1 =  1 .
It can be seen that for both controllers, the AMSE increases with   when ĝ0 is fixed; and the AMSE increases with | ĝ0 | when   is fixed.For the same pair of ( ĝ0 ,   ), the AMSE of   using the ARIMA controller is always smaller than that using the  controller at a significant level.

Effect of Estimation Uncertainties in the Process Gain.
In order to focus on the effect of the estimate of the process gain brought to the controller's performance, we assume both The performance of the ARIMA controller and the  controller when the process initial bias varies.
measurement error and adjustment error are absent, that is,   =   = 0; the initial bias  0 is 0, that is, ĝ0 =   = 0; and all the ARIMA parameters are accurately estimated, that is, φ1 =  1 and θ1 =  1 .
Mathematical Problems in Engineering We set the process gain  to 2.5 and assume β ∈ [1.
. Table 3 and Figure 3 present the performance of the two controllers when β varies.It is shown that the ARIMA controller always outperforms the  controller for all the βs we examined at a significant level.Additionally, underestimated  (i.e., β/ < 1) will hurt both controllers' performance more than overestimated  (i.e., β/ > 1).It is shown that if the process gain  is underestimated to some degree, both controllers would completely fail.

Effect of Estimation Uncertainties in the Disturbance.
Again, in order to focus on the effect of the estimation uncertainties in the disturbance, we assume   =   = ĝ0 =   = 0 and β = .Suppose the disturbance parameters are estimated as φ1 = 0.7 and θ1 = −0.3.However, the disturbance's true parameters may not be equal to the estimates, that is,  1 ̸ = φ1 and/or  1 ̸ = θ1 .We choose 5 values of  1 and 5 values of  1 as the true parameters for the ARIMA(1, 1, 1) disturbance, so there are a total of 25 disturbance ARIMA(1, 1, 1) models that are studied to investigate both controllers' robustness.
Table 4 reports the results of the AMSE of   using the ARIMA controller and the  controller for different ARIMA(1, 1, 1) disturbances with varying  1 and  1 .It is shown that the ARIMA controller's performance is better than the  controller's in most of the cases at a significant level.Only when  1 = 0.95 is the  controller's performance better than the ARIMA controller's.This is because a ARIMA(1, 1, 1) process will become a ARIMA(0, 2, 1) process when  1 = 1.As this ARIMA controller is purposely designed to control a ARIMA(1, 1, 1) disturbance, its performance will be contaminated when  1 is close to 1.The results also imply that if a ARIMA(0, 2, 1) disturbance is misidentified to be a ARIMA(1, 1, 1) disturbance, the ARIMA controller would have a bad performance.
For illustration, Figure 4 further presents the contour plots for the AMSE as a function of  1 and  1 by using the ARIMA controller and the  controller, respectively.It can be generally observed that both controllers that are designed based on underestimated disturbance parameters (i.e., φ1 <  1 , θ1 <  1 ) will have worse performance than the ones designed based on overestimated disturbance parameters (i.e., φ1 >  1 , θ1 >  1 ).Specially, as can be seen from Figure 4(a), when  1 is not greater than 0.9, the AMSE of   by using the ARIMA controller changes very slowly with varying  1 and  1 .However, when  1 is greater than 0.9, the AMSE increases very quickly with  1 increasing.The parameter  1 has less effects than  1 on the performance of the ARIMA controller.In comparison with the ARIMA controller, the  controller's performance is affected by both  1 and  1 in almost the same degree as shown in Figure 4(b).In all the change area of  1 and  1 except the area  1 > 0.9, the  controller's performance is worse than the ARIMA controller's.However, the maximum AMSE (the worst case) used by the  controller in the whole change area of  1 and  1 is smaller than that used by the ARIMA controller.In this sense, we can say that the  controller's performance is more robust than the ARIMA controller's when uncertainties exist in the ARIMA parameters.

An Illustrative Example
The studies in the last section are carried out based on an ARIMA(1, 1, 1) disturbance.As aforementioned, the proposed ARIMA controller can be applied to any general ARIMA(, , ) disturbance.For illustration purpose, we assume a process as follows: and (1 − 0.2B)(1 − B)  = (1 − 0.1B − 0.41B 2 + 0.105B 3 )  ,   ∼ WN(0, 0.8 2 );   is the deviation from target.The disturbance model   is the same as one of the ARIMA(1, 1, 3) disturbances presented in [21].For the sake of simplicity, we further set   =   = 0,  0 = 0, and assume that all the disturbance parameters are known.That is, we ignore measurement errors, adjustment errors, initial bias uncertainties, and disturbance parameter uncertainties and only show the ARIMA controller's superiority in controlling higher order ARIMA disturbance models.
For the above ARIMA(1, 1, 3) disturbance, we can get  = max(1, 3) = 3 and So, in the offline procedure, the ARIMA controller's parameters are given as follows: The initial values R− 1 =  − 1 = 0. Using the same optimization tuning techniques described in Appendix B, we get   = 1.128 and   = −0.008.
We randomly draw one simulation and show the paths of the process output   in Figure 5 when the two control algorithms are implemented, respectively.The MSE of   implemented by the  controller is 0.819 while that implemented by the ARIMA controller is 0.646.The paths suggest that the ARIMA controller maintains the process output   closer to the target than the  controller in much more places in the first 100 simulated runs.

Concluding Remarks
In this paper, we have developed an optimal control algorithm, the ARIMA controller, for a linear system with a general ARIMA(, , ) disturbance, in the presence of measurement errors and adjustment errors together with a random initial bias.We theoretically prove that the ARIMA controller is optimal to both the finite-horizon and infinitehorizon problems.The performances of the ARIMA and  controllers have been evaluated and compared via Monte Carlo simulations under multiple scenarios.In almost all the analyzed scenarios, the ARIMA controller outperforms the  controller.Only when uncertainties exist in the ARIMA parameters does the  controller have more robust performance than the ARIMA controller.Our simulation studies also show that when designing the two controllers, underestimating the process parameters, including the process gain and the ARIMA parameters, will have more negative impact on the controllers' performance than overestimating the parameters.Although the steps to implement the  controller are simple, a tuning process is always needed to determine the  parameters   and   for each given disturbance.It is usually time-consuming.Conversely, the ARIMA controller can automatically determine its algorithm parameters for each given disturbance.
The ARIMA controller is a complement to the ARMA controller which is designed to optimally control any general ARMA(, ) disturbance.The ARMA controller can be used to control a large class of weakly stationary disturbances while the ARIMA controller can handle a large class of nonstationary or periodic disturbances.Such disturbances are common in rapid thermal processing, reactive ion etching, Iline lithography, and lapping processes in the semiconductor manufacturing.

Figure 1 :
Figure 1: The performance of the ARIMA controller and the  controller when both measurement error (  ) and adjustment error (  ) exist and vary.

Figure 3 :Figure 4 :
Figure3: The performance of the ARIMA controller and the  controller when the process gain is not accurately estimated.
The corresponding SEAMSE's are enclosed in the parentheses.
The corresponding SEAMSE's are enclosed in the parentheses.Neither measurement errors nor adjustment errors are considered in the simulations.

Table 3 :
AMSE with different estimates of the process gain.The true process gain  = 2.5.

Table 4 :
AMSE with uncertainties in the ARIMA parameters.
The corresponding SEAMSE's are included in the parentheses.