Journal of Applied Mathematics

Volume 2012 (2012), Article ID 835319, 22 pages

doi:10.1155/2012/835319

Research Article

Delayed Stochastic Linear-Quadratic Control Problem and Related Applications

Li Chen,¹ Zhen Wu,² and Zhiyong Yu³

¹Department of Mathematics, China University of Mining and Technology, Beijing 100083, China

²School of Mathematics, Shandong University, Jinan 250100, China

³School of Economics, Shandong University, Jinan 250100, China

Received 25 May 2012; Accepted 30 July 2012

Academic Editor: Ying Hu

Copyright © 2012 Li Chen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

We discuss a quadratic criterion optimal control problem for stochastic linear system with delay in both state and control variables. This problem will lead to a kind of generalized forward-backward stochastic differential equations (FBSDEs) with Itô’s stochastic delay equations as forward equations and anticipated backward stochastic differential equations as backward equations. Especially, we present the optimal feedback regulator for the time delay system via a new type of Riccati equations and also apply to a population optimal control problem.

1. Introduction

The problem of optimal control for delayed stochastic system has received a lot of attention recently. One of the reasons is that there are many phenomena which have the nature of pastdependence that is, their behavior at time not only depends on the current situation, but also on their past history. Such kinds of mathematical models described by stochastic delay differential equations (SDDEs) are ubiquitous and have wide range of applications in physics, biology, engineering, economics, and finance (see Arriojas et al. [1], Mohammed [2, 3], and the references therein).

In control problem, a delay term may arise when there is a time lag between observation and regulation or the aftereffect of control; that is, there may be delay in state or control variables. Although many papers came out to discuss the delayed control problem, the analysis of systems with delay is fraught with many difficulties, not only for the infinite dimensional problem, but also for the absence of Itô's formula to deal with the delayed part of the trajectory. In order to surmount these difficulties, one can consider specific classes of systems with aftereffect, such as Øksendal and Sulem [4].

This paper is concerned with optimal control of a linear system with delay under a quadratic cost criteria, namely, the quadratic problem for stochastic linear control system with delay. It is well known that linear-quadratic (LQ) control is one of the most important classes of optimal control and the solution of this problem has many real-world applications. Deterministic LQ control problem with delay has been discussed in Alekal et al. [5], Basin et al. [6], and so forth. However, there is little results on stochastic LQ problem with delay since the difficulties in exploring the optimal feedback regulator, such as the Riccati equation, are very different from the case without delay.

In Peng and Yang [7], a new type of backward stochastic differential equations (BSDEs) was introduced, which is named anticipated BSDEs. The anticipated BSDE provides a new method to deal with optimal control problem with delay (see Chen and Wu [8]). In our paper, we study the delayed stochastic LQ problem by virtue of the anticipated BSDEs combined with the SDDEs.

In the next section, we introduce a kind of generalized FBSDE and give the existence and uniqueness result for its solution. With the help of the FBSDE, we find the optimal control of the stochastic LQ problem with delay in both state and control variables and the quadratic cost also involves delay terms.

It is very important to design an optimal feedback regulator for LQ problem in practice. Traditionally, a fundamental tool to obtain the state feedback is the Riccati equation. In Section 3, we introduced a new type of generalized Riccati equations and give the feedback regulator of delayed stochastic LQ problem. To the best of our knowledge, it is the first result on the optimal feedback control for the delayed stochastic LQ problem, where the state and control variables with delay are involved not only in the system, but also in the cost functional. In the last section, we apply our theoretical result to a population optimal control problem. Some technical proofs are put in Appendix.

2. Stochastic LQ Problem with Delay

We first introduce some notations. For any Euclidean space H, we denote by (resp. ) the scalar product (resp. norm) of H. Let be the Hilbert space consisting of all matrices with the inner product:

Here, the superscript denotes the transpose of vectors or matrices. Particularly, we denote by the set of all symmetric matrices.

Let be a standard -dimensional Brownian motion on a complete probability space . The information structure is given by a filtration , which is generated by and augmented by all the -null sets. We assume the dimension of Brownian motion just for the simplicity of notations. In fact, all the conclusions in this paper still hold true for the case that the dimension of Brownian motion . For any , the Euclidean spaces or sets of matrices, the following notations will be used throughout the paper:

In this section, we consider the following linear controlled system involving delays in both state variable and control variable:

where is a constant time delay, , are deterministic functions, and . , , is an -adapted square-integrable process taking values in . Let denote the set of stochastic processes of the form:

An element of is called an admissible control. By the classical results of SDDEs, we know that the system (2.3) has a unique solution for any admissible control . The solution of SDDE (2.3) is called the state trajectory corresponding to the control and is called an admissible pair.

The cost functional is given by

Here, , , and is an -measurable nonnegative symmetric bounded matrix. Moreover, we assume that, for any ,   and are nonnegative definite, is positive definite, and the inverse is also bounded.

The controller hopes to minimize the above cost functional by selecting an appropriate admissible control ; that is, the problem is to find such that

We call the problem above the linear-quadratic optimal control problem of delayed system, and we denote it by Problem (LQD). An admissible pair is called optimal for Problem (LQD) if achieves the infimum of .

The Problem (LQD) we introduced above is a general type of LQ problem for stochastic system with delay. Not only the state variable and control variable are involved with delays, but also the cost functional contains delay terms.

By introducing a new type of BSDEs-anticipated BSDEs, we desire to solve Problem (LQD).

Theorem 2.1. The control

is the unique optimal control of Problem (LQD). Here, is the solution of the following generalized FBSDE:

Remark 2.2. If we set

then the second equation in (2.8) can be rewritten as

Proof. Suppose is an arbitrary admissible pair of system (2.3), then

Since , by the time-shifting transformation and the initial conditions in (2.8), we derive that

Applying Itô's formula to , we have

Moreover, because and , we can rewrite (2.13) as follows:

Consequently, by the fact that and are nonnegative and is positive, we have

So letting

we have

that is, defined in (2.7) is the optimal control of Problem (LQD).

We will use the parallelogram rule to prove the uniqueness of the optimal control, and this method can also be seen in Wu [9]. We assume that and are both optimal controls, and the corresponding trajectories are and . It is easy to know the trajectories corresponding to are . Since is positive, and are nonnegative, we know that , and

Because of being positive, we have . We complete our proof.

Remark 2.3. The existence of the optimal control is equivalent to the existence of solution for (2.8), which is a kind of complex generalized FBSDEs. The proof of the existence and uniqueness of the solution for this kind of FBSDE are put in Appendix.

3. Feedback Regulator of Delayed System

It is well known that the feedback representation of optimal control is very useful in applications. And in the classical case, the optimal feedback control can be represented via the Riccati equation. But, for stochastic systems with delay, it is not easy to find the feedback control because of the dependence of the history. What should then be an appropriate “Riccati equation” corresponding to our LQ problem with delay?

In this section, we will pay attention to the feedback regulator of the general delayed LQ problem discussed in Section 2 and try to get the appropriate Riccati equation associated with Problem (LQD). We remark that all given coefficients of the problem are assumed to be deterministic from now on.

Let us start with the following results about anticipated BSDE.

Assume that for all , , where . And satisfies the following conditions.

(H3.1) There exists a constant , such that for all , , , , , we have

(H3.2).

Let , , be, respectively, solutions of the following two anticipated BSDEs:

We have the following lemma, which is a direct corollary of Theorem 2.2 in Yu [10].

Lemma 3.1. Assume that satisfy and , , , and

then

In order to get the feedback regulator, we introduce the following generalized matrix-valued Riccati equation system:

With the boundary conditions

Here, is some constant.

Then we have the following main result.

Theorem 3.2. Suppose that there exist matrix-valued deterministic processes satisfying the generalized Riccati equation system (3.5)–(3.8) with corresponding boundary conditions, and for system (2.3), , . Then the optimal feedback regulator for the delayed linear quadratic optimal Problem (LQD) is

with

And the optimal value function is

Proof. Let be the solution of Riccati equation system (3.5)–(3.8) and we set

Applying Itô's formula to , we have

with

Substituting (3.16) into (3.15), we have

where and

The following analysis shows that with the help of (3.7) and (3.8). In fact, taking (3.16) into the presentation of , we will have

By the conditions , , , and , , we derive

We can see that (3.7) and (3.8) ensure that

In other words,

So we use the following equation instead of (3.17):

Namely, introduced in (3.14) satisfies the above anticipated BSDE. From now on, we pay attention to the anticipated BSDEs (2.8) and (3.22).

Applying the similar method we used above, we derive that (3.5) and (3.6) will lead to the following:

Thus, we get by Lemma 3.1 immediately, and feedback representation of optimal control in Theorem 3.2 is proved.

Applying Itô's formula to in , it is easy to get (3.13).

Remark 3.3. (i) The condition is reasonable in practice: the condition implies that there is no delay on the time interval . That is to say there is no history of the system on , that is, .

(ii) The matrix-value differential equation (3.5)–(3.8) is a generalized type of Riccati equations which is different from the classical one. When there are no delays in the system, that is, , , our generalized anticipated Riccati equation will degenerate to the Riccati equation in Wu [9].

4. Application

The solvability of Riccati equation (3.5)–(3.8) is not easy to obtain in general. In this section, we derive the unique solvability of the problem for some special cases. As an application of our above results, we will consider a kind of stochastic LQ problem with delay arising from the population control model in this section. The optimal feedback control is given by the new type of Riccati equation and we can obtain its existence and uniqueness.

The following population growth model comes from Mohammed [3]:

where is the population at time , and are constant death rate and birth rate per capita, respectively. denotes the development period of each individual, and the diffusion part describes the migration of the population; that is, there is migration whose overall rate is distributed like .

Based on the population growth model, we consider the following population control problem:

Here, we change the original constant death and birth rate to be time-varying are deterministic bounded functions on and on . . And the control models the intensity of the spending on controlling the population. We take into account that there will be a time lag between control expenditure and the corresponding effect on the population level, so the system is involved with control delay. is a -dimensional B.M.

The objection is to minimize the following cost functional:

over the admissible control set . and are nonnegative- and positive-valued bounded functions on , respectively, and is a given constant. And all the coefficients in our model are equal to zero outside of the interval . About the cost functional, one can consider that we are going to control some pests' quantity. We want to pay less expenditure and make the quantity of pests on lower level.

The above population control problem is a special case of the Problem (LQD) we discussed in Sections 2 and 3. From Theorems 2.1 and 3.2, we have the following results.

Proposition 4.1. The control

is the unique optimal control. Here, is the solution of the following generalized FBSDE:

Proposition 4.2. The feedback control regulator of the population control problem is given by

where satisfies

Moreover, the above Riccati (4.7) admits a unique solution, so the presentation form of (4.6) exists and it is unique. Here, is a parameter satisfying , for all .

Proof. Obviously, the existence and uniqueness of presentation (4.6) are equivalent to Riccati (4.7) which has a unique solution, so we pay attention to (4.7).

Let us start with the second equation in (4.7). It is a classical Riccati-type differential equation with bounded coefficients, thence, by Theorem 7.7 of Chapter 6 in Yong and Zhou [11], we get that it admits a unique solution on and also we have with which is big enough positive constant.

Now, let us turn to the first equation in (4.7). It is anticipated. In order to derive its solvability, we discuss the equation backward step by step on the time interval , , .

For example, when , the equation is equivalent to the following:

Because can be solved in advance, we get that there exists a unique solution of (4.8) and over also by Theorem 7.7 of Chapter 6 in Yong and Zhou [11].

The discussions on the other time intervals are similar; we omit them. All in all, the Riccati equation (4.7) is solved backward in time from as an initial value problem and the solution is unique. This will lead to the existence and uniqueness of our feedback control immediately. The proof is completed.

Appendix

Let us pay attention to the generalized FBSDE (2.8). For the classical FBSDE, Hu, and Peng [12] and Peng, Wu [13] obtained the existence and uniqueness results under some monotonicity conditions. Yong [14] let the method in [12, 13] be systematic and called it the “continuation method." Then Wu [9, 15] discussed the applications of FBSDEs in LQ problem and maximum principle for optimal control problems of FBSDE systems. While if we substitute the optimal control (2.7) into (2.8), we notice that our FBSDE (2.8) is different from the classical one; it contains more delayed and anticipated part in the coefficients. Now, we try to analyze this generalized kind of FBSDEs.

For more general case, we consider the following FBSDE. (Just for simplicity, in this appendix we change the notations with writing the time variable as a subscript).

Consider the following:

Here are matrices, and are -adapted with appropriate dimensions.

We will use the notations:

where . We also impose the following monotonicity conditions:

for all , where and are given nonnegative constants with .

Then we have the following result.

Theorem A.1. Let and hold. Then there exists a unique solution satisfying the generalized FBSDE (A.1).

Proof. Uniqueness

let and be two solutions of (A.1). We set . And we apply Itô's formula to on :

Integrating and taking expectations on both sides of the above equation, we have

Combining , we obtain

This implies . Following and the uniqueness of solution for SDDE (see [2, 3]), we have . Consequently, . By the uniqueness of solution for BSDE, we have .

Existence. In order to prove the existence of the solution we first consider the following family of equations parameterized by :

where and are given processes in , . Obviously, the existence of solutions for (A.7) with implies our desired conclusion. When , we notice that (A.7) is in a decupled form, and we obtain the existence and uniqueness result from the classical SDE and BSDE theory. Detailedly, in this case, the second equation and the terminal condition construct a standard BSDE which admits a unique solution . Substituting into the first equation, we get a standard SDE and it admits a unique solution . Thence, we need to consider the following case:

Here, . We want to prove that the mapping defined by

is a contraction.

For the difference , we have the following result:

Here depends on the Lipschitz constants of and . If we select , where , then

with . For , using the estimate of SDDE, we derive

Similarly, for the difference of solutions , we have

by the anticipated BSDE estimates (see [7]). Here, the constant depends on the Lipschitz constants, and . Combining the estimates (A.11)–(A.13), we have

where the constant depends on and . So we choose , then for each , the mapping is a contraction; that is, for all the (A.7) for has a unique solution. We repeat this process times with . It then follows that (A.7) for has a unique solution.

Remark A.2. It is verified that the FBSDE (2.8) satisfies the assumptions and ; therefore, there exists a unique solution for FBSDE (2.8). Consequently, the optimal control of our Problem (LQD) exists and is unique.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (11101242, 11126214, 10921101, and 61174092), the National Science Fund for Distinguished Young Scholars of China (11125102), the Natural Science Foundation of Shandong Province, China, (ZR2010AQ004), the Independent Innovation Foundation of Shandong University (IIFSDU, no. 2010TS060), and Doctoral Fund of Education Ministry of China.

References

M. Arriojas, Y. Hu, S.-E. A. Mohammed, and G. Pap, “A delayed Black and Scholes formula,” Stochastic Analysis and Applications, vol. 25, no. 2, pp. 471–492, 2007.

S.-E. A. Mohammed, Stochastic Functional Differential Equations, vol. 99, Pitman, 1984.

S.-E. A. Mohammed, “Stochastic Differential Equations with Memory: Theory, Examples and Applications, Stochastic Analysis and Related Topics 6,” in The Geido Workshop, Progress in Probability, Birkhauser, 1996.

B. Øksendal and A. Sulem, “A maximum principle for optimal control of stochastic systems with delay, with applications to finance,” in Optimal Control and Partial Differential Equations, J. M. Menaldi, E. Rofman, and A. Sulem, Eds., pp. 64–79, ISO Press, Amsterdam, The Netherlands, 2000.

Y. Alekal, P. Brunovský, D. H. Chyung, and E. B. Lee, “The quadratic problem for systems with time delays,” Institute of Electrical and Electronics Engineers. Transactions on Automatic Control, vol. 16, no. 6, pp. 673–687, 1971.

M. Basin, J. Rodriguez-Gonzalez, and R. Martinez-Zuniga, “Optimal control for linear systems with time delay in control input based on the duality principle,” in Proceedings of the American Control Conference, vol. 3, pp. 2144–2148, 2003.

S. Peng and Z. Yang, “Anticipated backward stochastic differential equations,” The Annals of Probability, vol. 37, no. 3, pp. 877–902, 2009.

L. Chen and Z. Wu, “Maximum principle for the stochastic optimal control problem with delay and application,” Automatica, vol. 46, no. 6, pp. 1074–1080, 2010.

Z. Wu, “Forward-backward stochastic differential equations, linear quadratic stochastic optimal control and nonzero sum differential games,” Journal of Systems Science and Complexity, vol. 18, no. 2, pp. 179–192, 2005.

Z. Yu, “The stochastic maximum principle for optimal control problems of delay systems involving continuous and impulse controls,” Automatica, vol. 56, no. 6, pp. 1401–1406, 2012.

J. Yong and X. Y. Zhou, Stochastic Controls, vol. 43, Springer, New York, NY, USA, 1999.

Y. Hu and S. Peng, “Solution of forward-backward stochastic differential equations,” Probability Theory and Related Fields, vol. 103, no. 2, pp. 273–283, 1995.

S. Peng and Z. Wu, “Fully coupled forward-backward stochastic differential equations and applications to optimal control,” SIAM Journal on Control and Optimization, vol. 37, no. 3, pp. 825–843, 1999.

J. Yong, “Finding adapted solutions of forward-backward stochastic differential equations: method of continuation,” Probability Theory and Related Fields, vol. 107, no. 4, pp. 537–572, 1997.

Z. Wu, “Maximum principle for optimal control problem of fully coupled forward-backward stochastic systems,” Systems Science and Mathematical Sciences, vol. 11, no. 3, pp. 249–259, 1998.