Neural Network-Based State Estimation for a Closed-Loop Control Strategy Applied to a Fed-Batch Bioreactor

1 Instituto de Ingenieŕıa Quı́mica, Universidad Nacional de San Juan, CONICET, Av. Lib. San Mart́ın Oeste, 1109 San Juan, Argentina 2Instituto de Automática, Universidad Nacional de San Juan, CONICET, Av. Lib. San Mart́ın Oeste, 1109 San Juan, Argentina 3Facultad Regional Santa Fe, Universidad Tecnológica Nacional, CONICET, Lavaisse 610, Santa Fe, Argentina 4Instituto de Desarrollo Tecnológico para la Industria Quı́mica (INTEC (UNL-CONICET)), Güemes, 3450 Santa Fe, Argentina


Introduction
In the last decade, several control strategies applied to different bioprocesses have been extensively investigated (e.g., [1][2][3][4][5]).In the particular case of fed-batch bioreactors, some nonlinear control strategies were developed to improve the process efficiency and the "batch-to-batch" reproducibility [6], as well as to deal with the unavoidable uncertainties typically present in bioprocesses [7].
Many articles have investigated the effects of model uncertainties on the performance of model-based nonlinear control strategies typically applied to bioprocesses.For example, the "yield coefficient of cell to glucose" in a bioprocess is a time-varying parameter often assumed as a constant.Also, the "specific growth rate" can be represented through the Monod-Haldane model [8], but it is possible that the model becomes inaccurate because of yeast growth inhibition due to an eventual presence of ethanol.In principle, three main types of uncertainties can be envisaged [9][10][11][12][13]: (1) uncertain and/or time-varying model parameters, (2) uncertain nonlinearities, and (3) unmodeled dynamics.Additionally, nonmodeled external perturbations can disturb the process.In general, model uncertainties and external perturbations can cause severe risks in bioreactors, such as leading to undesirable operating conditions [14][15][16].For instance, inadequate modeling of the inhibitory effect of a substrate can lead to a structural instability in the dynamical behavior of a bioprocess [15].
Adaptive control strategies are a valuable choice to control bioreactors due to their ability to compensate for parametric uncertainties.The Nussbaum gain method has been applied to control a bioprocess of unknown and time-varying control gain and parameters [17].A fed-batch bioreactor control with simultaneous dynamic identification of unknown state and parameters has been investigated through an algorithm based on the auxiliary model principle with adaptive controllers 2 Complexity [18].A control strategy was proposed to track a desired profile for the biomass of a yeast culture in a high-density fed-batch bioreactor [19].This last strategy was based on a model that included an uncertainty term estimated through an adaptive technique.Also, an adaptive nonlinear feedback controller has been designed to automatically estimate uncertain timevarying parameters in order to guarantee the closed-loop control of a waste water plant [20].
Fuzzy control systems have been successfully applied to several bioreactors.For instance, [12] considered an adaptive fuzzy-decentralized control for a system with unknown nonlinear uncertainties and dynamical uncertainties.The fuzzy system was used to estimate the nonlinear uncertainties.Also, a fuzzy controller for the alcoholic fermentation with Saccharomyces cerevisiae has been proposed [21].It combined the capability of handling uncertainties with the ability of predictive control to improve the plant performance, making use of a neural network model of the nonlinear system.Optimal control strategies have also been applied to bioprocesses [22,23].In order to compensate for process uncertainties, this type of controllers incorporates an additional term to the bioprocess model (typically, in the specific growth rate), which is estimated through an optimization technique.Alternative strategies for controlling bioreactors under model uncertainties are trajectory-based control [24], hybrid control [25], and model predictive control [26].Additionally, due to the inherent nature of a bioprocess, a control scheme based on a stochastic model seems to be adequate.The specific growth rate can be modeled as affected by a white noise representing environment uncertainties, and the mathematical model consists of a set of stochastic differential equations [27] (some applications can be consulted in [28,29]).The application of this approach onto a real plant is quite difficult due to its complex design and implementation.
Based on a linear algebra methodology, a control strategy has been applied for trajectory tracking in a bioprocess [30].The technique was used to calculate the control actions that ensure the tracking of independently determined optimal trajectories of some process variables.To this effect, a discrete approximation of the original process model must first be derived, and the control actions are then calculated through a least square problem.This methodology has been applied to a fed-batch bioprocess designed to produce a secreted protein, and it was originally modeled by Park and Ramirez [31].Optimal profiles of such bioprocess have also been published [32].
Many nonlinear control techniques of bioprocesses assume that the feedback variables can be measured online [33,34].However, such hypothesis is often unrealistic and therefore some process variables must be estimated through virtual sensors (or soft sensors).There are several kinds of virtual sensors published in the literature which exhibit different abilities to deal with model mismatches and perturbations.In this work, an artificial intelligence-based observer has been chosen because this type of estimator is especially recommended for highly nonlinear systems in the presence of uncertainties [35].
This paper aims at designing a controller for the multivariable tracking of optimal profiles when the process is affected by uncertainties in the dynamic model and/or in the model parameters.Unmeasured process variables are estimated through a RBF neural network.The proposed approach is evaluated onto a bioprocess originally modeled by [31], with the optimal profiles determined by [32] in absence of model errors.Brief descriptions of the fed-batch bioreactor and the controller derived in [30] are presented in Appendix A.
In comparison with [30], the proposed approach prevents the polynomial uncertainties from affecting the tracking control.To achieve this aim, an integrative term is incorporated in the linear algebra methodology originally proposed.Then, the conditions in order for a system of linear equations to have exact solution are analyzed.The control action is obtained by solving the linear system, even though the original process model is nonlinear.
The main contribution is that the original method proposed herein ensures the convergence to zero of the tracking errors under additive uncertainties.Another important contribution is the inclusion of RBF neural network estimators: the control issue is addressed considering more realistic conditions with nonmeasurable variables.Further, the stability analysis is made using Lyapunov theory in discrete time.
The article is organized as follows.In Section 2, the controller design methodology and the convergence to zero of the tracking errors are presented.Section 3 shows briefly the description of the RBF estimator.A method to tune the controller is presented in Section 4, along with four testing cases designed to check the effectiveness of the new control law.Main conclusions of the work are summarized in Section 5.

Controller Design
Derivation of the control strategy is based on the mathematical model presented in Appendix A. The effect of an additive uncertainty (E  ) is modeled by extending (A.4) as follows: Taking into account the fact that the mismatch might depend on both system states and inputs, we here consider a real plant described by z +1 = (z  , u  ).The additive uncertainty can be expressed as E  = (z  , u  ) − f(z  , u  ), where f(z  , u  ) is the discrete-time nonlinear system model.If z and u are assumed to be bounded and f is Lipschitz, then E  can be modeled as a bounded uncertainty [36][37][38].

Complexity 3
With the additive uncertainty of (1), the fed-batch bioreactor model of (A.3) becomes ] . ( The control action for the secreted protein concentration, , given by (A.4) of [30], is The discrete value of ,  +1 , is calculated by replacing (3) into the first row of (2): where   and  depend on   and  , , respectively.Thus, the tracking error of   is By combining (4) and ( 5), one obtains The Taylor interpolation of   at its desired value  is By replacing (7) If a similar procedure is followed for , , and , then the error equation becomes [30] Alternatively, in vector form, where h  is a nonlinearity that tends to zero when  , tends to zero.Equations (9a) and (9b) suggest a direct effect of the additive uncertainty on the tracking error.Therefore, the presence of an uncertainty avoids a convergence of tracking errors to zero (see Theorem 1 of [30]).

Single Integral Action.
In order to reduce the effect of E  , some integrators for the tracking errors will be introduced, depending on the time variation hypothesis of E  .We will assume that E  is unknown and each of its components is an -order polynomial.
Remark 1.The first-order difference of E  is defined as E  = E +1 − E  , the second-order difference is defined as and, as a rule, the -th order difference is defined as Remark 2. The -th difference of a  − 1-order polynomial is zero.
Consider first a constant uncertainty E  = const.Then, E  = E +1 − E  = 0.In this case, an integrator for each state variable will force the error to converge to zero.Call e() the continuous time error in the state vector.Then, the integral of the tracking error is defined as

Complexity
Hence, the control action (A.7) will be computed by assuming a new term in (A.5); that is, where K and k 1 are the proportional and integral design parameters, respectively.By adding one integrator (k 1 U 1,+1 ) to (9b) and using (10), one obtains Rewriting (12) for the next sample time, Combining ( 10) and ( 12) and substituting them into (13), Alternatively, Therefore, K and k 1 are chosen in order to ensure the stability of the linear system represented in the left-hand side of (15).
To achieve this condition, the roots of such polynomial (  ) must lie inside the unit circle.Then, e  → 0 when  → ∞; that is, the error tends to zero as long as uncertainties are constant.
Once the integral of the error has been added, the control action is calculated following the same design procedure based on linear algebra presented in [30].Therefore, the new control law   can be obtained through the least squares procedure as

Multiple Integral Actions.
Assume that the uncertainty can be modeled by a function with a second-order difference equal to zero; that is, (10), a double integrator is now introduced, and new variables U 1 and U 2 are defined: In this case, the control action (A.7) will include an additional term in (A.5) as follows: where K, k 1 , and k 2 are the proportional, integral, and double-integral design parameters, respectively.With a similar procedure to that used for deriving (15) and after adding two integrators (k 1 U 1,+1 and k 2 U 2,+1 ), the error dynamics can be expressed as As suggested by (19), under the assumption of a constant or linear varying uncertainty ( 2 E  = 0), the uncertainty has no influence on the error dynamics.As in the case of the single integrator, the controller parameters K, k 1 , and k 2 should be chosen in order to ensure the stability of the left-hand side of (19).
The previous derivation can be generalized for uncertainties that can be approximated with a  − 1-order polynomial; and, therefore, the influence of E  upon e  would be cancelled off by addition of  integrators.
The linear algebra-based procedure enables the calculation of the following control law:

Neural State Estimator
The proposed feedback control technique requires reliable estimates of several state variables.The state estimation error, x , is defined as where x is the vector of estimated state variables and x  = [ P , T , X ] is the vector of online measured states.The following neural estimator represents the nonlinear dynamic of the bioreactor described in (7): where   = [  ,   ,   ] is the input vector to the neural estimator,  * is the optimal weight vector,  is the neural approximation error,   is the RBF that represents each neuron in the hidden layers, subindex  indicates the neuron number of the RBF, and  is the maximum number of neurons.Every neuron of the hidden layer is modeled as where c  and   are two parameter vectors that represent the centers and widths of the RBF, respectively.
In the bioreactor under study, biomass () cannot be measured online and the protein concentrations (, ) cannot be measured at all; then a state estimator function based on ( 22) is determined as follows: From the difference between ( 22) and ( 24), the neural identification error can be described by where θ represents the neural weight vector estimation error and it is defined as In order to train the neural network for identification, an offline data set of (x, u) will be used.This training data will be obtained after following the same procedure presented in [39] and then applied in [40,41].A similar procedure was also used in [42].The learning rule utilized to train the neural network is demonstrated in the next theorem.
Theorem 3. The model of the fed-batch bioreactor (see (A.1) and ( 2)) can be approximated by the neural network (24) using a neuronal adjustment law defined by Proof.The demonstration of this theorem is presented in Appendix B.
As recommended by [35], the estimator was evaluated to test its performance and avoid biased estimates.Table 1 summarizes the implemented tests, and their results were evaluated through the average values of the mean squared errors (MSE) corresponding to , , and .The obtained results show an acceptable performance of the RBF neural network for estimating the nonmeasurable states.Note that none of the MSE values exceeds the thresholds proposed by [43], 1 × 10 −3 .Figure 1 shows the proposed closed-loop architecture.Note that the controller requires information on both measured and estimated variables.

Results and Discussion
All simulation examples are based on the bioprocess originally modeled by [31].The available optimal profiles for such process are also adopted [32].Equation (A.1) represents the fed-batch bioreactor model.The effectiveness of the new control law is tested through some simulation examples.First, the optimal controller parameters are tuned through a Monte Carlo experiment.Then, a sinusoidal perturbation is added to some model parameters.Also, the controller performance  under uncertain time-varying parameters is verified through a Monte Carlo Randomized Algorithm.In other examples, the functionality of the specific growth rate is changed.Finally, some parameters are contaminated by additive white noise.
As previously discussed, the behavior of the controlled system depends on parameters k, K 1 , . . ., K  , where  is the number of integrators.For  = 0, 1, 2, the controller parameters are determined through a Monte Carlo method, where the number of simulations is determined by application of Theorem 2 of [30].All simulations are implemented in Matlab6 (MathWorks, Inc.), with a sample time Δ = 0.01 h.For each controller (i.e., for each ), 1,000 simulations are run; and the controller parameters are randomly chosen, such that the roots (  ) of the linear systems defined in the righthand side of (9a) and (9b) and left-hand side of ( 15) and (19) verify   ∈ (0, 1).More specifically, for  = 0, the controller parameters are directly taken from [30].For  = 1, 1,000 new simulations are performed, and each controller parameter is chosen according to where the superscript  refers to each state variable and  1 and  2 are the roots of the linear system in the left-hand side of (15).For  = 2, 1,000 simulations are performed, and each controller parameter is chosen according to where  1 ,  2 , and  3 are the roots of the linear system in (19).Table 2 summarizes the results obtained with each controller.For comparison purposes, the performance of the proposed strategy is compared with that provided by a Parallel Two-Degree-of-Freedom PID Controller (2-DOF PID), which computes a weighted difference signal for each of the proportional, integral, and derivative actions according to the set point weights specified by the user [44].In this case, the culture cell density has been chosen as the controlled variable.The 2-DOF PID was tuned with the automatic tuning of the Simulink6 Control Design software that provides the proportional, integral, and derivative gains, as well as the filter coefficient.( The tracking error (  ) is defined as In what follows, we shall call C0, C1, and C2 the controllers that use  = 0,  = 1, and  = 2 integrators, respectively, and C3 the 2-DOF PID controller.Particularly note that C0 corresponds to the controller designed in [30].In spite of the assumed time-varying parameters, Figure 2(a) shows that C2 produces the minimum tracking error.According to Figure 2(b), both C2 and C1 are effective to reach the desired protein concentration profile, while C0 and C3 produce large errors.
In several research fields, probabilistic methods have been found to be useful for dealing with problems related to robustness of systems affected by uncertainties [46].Particularly, the Monte Carlo Randomized Algorithm has been used in many applications such as radioactive decay, power system generation, and traffic on roads [47].In the control area, Monte Carlo methods are effective tools for the analysis of probabilistically robust control schemes [30,46].A Monte Carlo simulation is here performed to demonstrate the effectiveness of C2 from a statistical point of view [48][49][50], when uncertainties are introduced by time-varying parameter.To this effect, 1,000 random values of   and  / are chosen in a range of ±25% of their nominal values.Figures 3(a responses are quite similar, thus suggesting that the timevarying parameter has not affected their performances.For comparison, the tracking error was evaluated through the following performance index : where  ref represents the number of points of the reference profile.Figures 3(b), 3(d), 3(f), and 3(h) confirm a better performance of C2 and C1 over C0 and C3.In fact,  is quite small for C2 and C1, even though 25% of uncertainty in the main bioreactor parameters was adopted.Since 1,000 simulations with random values of   and  / were carried out and the tracking errors remained bounded, the performance of C2 and C1 will be acceptable with 99% of probability whatever the parameter values are in a range of ±25% [46].

Uncertain Nonlinearities.
Uncertain nonlinearities are often present in many bioprocesses.For instance, the specific growth rate, , can suffer from important changes from batch to batch, and unfortunately such variability is impossible to be accurately modeled.There are a number of models in the literature for  [8], for example, the third expression of (A.2), for the bioprocess currently under study [31].To demonstrate the controller performance, an additional term can be included in the calculation of  as follows [18,51]: μ = (21.87+ 0.28 sin ())  ( + 0.4) ( + 62.5) .
Figure 4(a) compares the performances reached by C2, C1, C0, and C3 under the simulated uncertain nonlinearities in .The lower tracking errors correspond to C2. Figure 4(b) shows that only C2 and C1 are able to reach the maximum concentration of the total protein.

Unmodeled Dynamics.
The presence of white noise in  max leads to a mathematical model governed by a set of stochastic differential equations.In order to properly deal with this type of uncertainties, a kind of stochastic control approach should be used.As an alternative to the complex theory of stochastic control, a simpler deterministic strategy is here proposed.To this effect, note first that all parameters in (A.1) and (A.2) are truly affected by environmental noise, and consequently they always fluctuate at long times around some average values.For simplicity, we will consider that only  max is affected by randomness and changes according to the following expression [52,53]: where () is a standard Brownian motion defined on a complete probability space and  ≥ 0 represents the intensity of the noisy signal.
A white noise can be approximated through a bandlimited white noise (BLWN), which is simply implemented through a proper block of Simulink.Theoretically, a white  noise is uncorrelated; it exhibits a flat power spectral density, and therefore it is characterized by an infinite energy.The BLWN block can simulate the effect of white noise by using a random sequence with a correlation time much smaller than the shortest time constant of the system.The correlation time of the noise is the sample rate of the block [54] and can be estimated as where   is the correlation time,  max is the bandwidth of the system, and Δ max is the maximum sample time for which the system is stable.For the feedback control presented in this work, Δ max = 0.0145 h.Then, from (35),   = 0.0018 h.Many researchers have utilized this methodology to prove the robustness of the bioreactors controllers [19,[22][23][24]55].In the current simulations, a BLWN with a noise power of 0.9 and  = 0.5 is introduced along with the feedback signal.Thus, the maximum specific growth rate affected with BLWN is calculated as Figure 5(a) indicates that C2 and C1 adequately work in the presence of unmodeled dynamics.In fact, the tracking errors are quite small in comparison with C0 and C3. Figure 5(b) shows the profile tracking for each controller.Again, C2 and C1 are able to follow the reference profile, while C0 and C3 fail to reach the maximum protein concentration.

Conclusions
A new control law for tracking an optimal concentration profile in a fed-batch bioreactor has been presented.The proposed method allows controlling a nonlinear system in presence of unavoidable uncertainties in bioreactors: timevarying parameters, uncertain nonlinearities, and unmodeled dynamics.
In order to deal with these uncertainties, some integrators have been added to the original controller proposed in [30], and a Radial Basis Function (RBF) neural network was employed to estimate , , and .The integrators have been chosen based on the variation hypothesis of the modeling error (E  ): if a -grade polynomial can approximate E  , then  + 1 integrators will be added in order to avoid its influence on the tracking error (e  ).
Several tests were carried out to demonstrate the effectiveness of the proposed methodology.The controllers are tuned through a Monte Carlo experiment and under stability conditions.To evaluate the performance of the designed controllers, different uncertainties were simulated.It was proven that the proposed methodology successfully compensates the uncertainties, and the tracking errors decrease as long as the number of integrators increases.The performance of the proposed controller with two integrators (C2) is remarkable, and the complexity of the control algorithm is relatively small.Moreover, the addition of integrators does not increase significantly the calculation time.
In comparison to previous strategies that deal with similar control problems (e.g., [17,21,22,24,25,27]), the proposed controllers present the advantage of avoiding a stochastic modeling, as it is typically required to deal with systems where the parameters are contaminated by white noise.
The trade-off between the response speed and the stability is solved through a simple Monte Carlo experiment.In addition, the controller design is mathematically simple.Currently, the developed methodology for the controller design is being applied to another bioprocess.Besides, the inclusion of saturation of the control signals and sliding mode control schemes in the formulation of the problem will be addressed in future contributions.

Complexity
(A.1) can be rewritten in a discrete version using the Euler approximation: Let us denote by z  the state vector and by c  and d  the input vectors.Then, the state vector in the sample time ( + 1) can be expressed as ] ; The following replacements should be done in (A.3): where K is the controller parameter matrix and   ,   ,   ,   ∈ (0, 1).The Monte Carlo Randomized Algorithm was applied to select the controller parameter matrix.This procedure does not ensure reaching the global optimum.Nevertheless, based on Theorem 2 of [30], a large number of trials were made in order to guarantee a large confidence interval.Then, (A.3) can be rewritten as ] .
(A.6) Equation (A.6) has an exact solution when the culture glucose concentration is adopted as a "sacrificed variable,"   [30].Then, the control action   can be obtained through the least squares procedure: (A.7)

B. Proof of Theorem 3
Assumption B.1.The optimal approximation error  is bounded by a constant: ‖‖ ≤ .
Proof of Theorem 3. Define a candidate function of Lyapunov, with   being a positive definite function given by The difference in discrete time Δ  is adopted as follows: And Δ  is defined as  The increase in the model error is defined as

Figure 1 :
Figure 1: The proposed closed-loop control architecture: the linear algebra-based controller and the Radial Basis Function (RBF) neural network estimator.

(B. 8 ) 9 )
In (B.8), the increment of the model error of Δ x is unknown, and it can be approximated by the following equation:The partial derivative of the model error depends only on the weights of the neural network and can be rewritten as Δ x = − (  x(+1)  θ )  Δ θ .(B.10) Changing the values of the weights according to (B.6) and considering that  * is constant, ideal weight vectors are required only for analytical purpose.Now reorganizing Δ θ , it can be rewritten as Δ θ = θ − θ(−1) = ( *  − θ ) − ( *  − θ(−1) ) = − θ + θ(−1) = −Δ θ . (x  ,   )) .(B.12) Considering the value of the partial derivatives of the neural network (see (27)) and by substitution in (B.10), it yields Δ x = −x    (x  ,   )  (x  ,   ) .(B.13) is the state vector, c  and d  are the input vectors, Δ is the sample time,   is the control action, and E  is an additive uncertainty.In principle, E where z