Linearization of a Matrix Riccati Equation Associated to an Optimal Control Problem

The matrix Riccati equation that must be solved to obtain the solution to stochastic optimal control problems known as LQG homing is linearized for a class of processes. The results generalize a theorem proved by Whittle and the one-dimensional case already considered by the authors. A particular two-dimensional problem is solved explicitly.


Introduction
Let { ( ), ≥ 0} be a one-dimensional standard Brownian motion and consider the controlled diffusion process { ( ), ≥ 0} that satisfies the stochastic differential equation where (⋅) is the control variable and (⋅), (⋅), and V(⋅) > 0 are Borel measurable functions. The problem of finding the control * that minimizes the expected value of the cost function in which (⋅) > 0 is a Borel measurable function, ̸ = 0 is a real parameter, and ( ) is a first passage time defined by ( ) = inf { > 0 : ( ) ∉ ( 1 , 2 ) | (0) = } , (3) with 1 < < 2 , is a special case of what Whittle [1] (p. 289) termed LQG homing. This type of problem has applications in financial mathematics (see Makasu [2]). In Lefebvre and Zitouni [3], the authors considered an optimal landing problem. They also mentioned another possible application in which one wants to optimally control a dam.
In order to obtain * , we define the value function Using dynamic programming, one only has to determine the value of * ( ). We find that ( ) is such that 2 International Journal of Differential Equations It follows that * ( ) = − ( ) ( ) ( ) , so that The boundary conditions are Whittle has shown that if the relation holds for a positive constant , then the function satisfies the linear second-order ordinary differential equation subject to Equation (12) is actually the Kolmogorov backward equation satisfied by the moment-generating function (or the Laplace transform of the density function) of the random variable 0 ( ) that corresponds to ( ) but for the uncontrolled process obtained by setting [ ( )] ≡ 0 in (1). Moreover, the above boundary conditions are the appropriate ones. Thus, Whittle was able to sometimes transform the optimal control problem into a purely probabilistic problem.  (10) is obviously satisfied. Therefore, it is then always possible to linearize (8) in such a case. However, if two (or all) of these functions are not constant, we can say that it is a special case when (10) does hold. When only one of these three functions is not a constant, the relation cannot be satisfied. Next, notice that the optimal control is expressed in terms of the derivative ( ) := ( ) of the value function, which satisfies the Riccati equation However, in general, we do not have a condition that would enable us to determine the value of the arbitrary constant that appears in the solution of (14). Therefore, we must solve either the nonlinear second-order differential equation (8) or the Kolmogorov equation (12). In Lefebvre and Zitouni [3], the authors generalized Whittle's result. They showed that if (⋅) is different from zero, then the function ( ) defined through is a solution of the linear second-order ordinary differential equation They then gave a method that can be used to obtain an explicit expression for ( ), hence the optimal control * ( ). Now, Whittle actually considered LQG homing problems in dimensions. {X( ), ≥ 0} is then an -dimensional controlled diffusion process defined by where the noise matrix N is symmetric and positive definite. The cost function ( ) is replaced by The matrix Q is positive definite and where denotes the continuation region.

Remark 2.
In the general formulation, B is an × matrix, u is a (column) vector of dimension , and Q is an × matrix. Here, we assume that = .
The optimal control is given by and the value function (x) satisfies where x is the derivative of with respect to the vector x.
The equation is subject to International Journal of Differential Equations 3 in which denotes the boundary of the stopping region (the complement of continuation region ).
The relation that generalizes (10) and that must hold between the control matrices B and Q and the noise matrix N in order to be able to linearize the nonlinear partial differential equation (21) is the following: In practice, it is difficult to satisfy (exactly) the above relation for ≥ 2, especially if the matrices involved are not constant matrices. In fact, even in the case when the various matrices in (23) are indeed constant, it is rather rare that this relation is satisfied. Problems for which (23) holds must be symmetrical. For instance, an important particular case is the one when N, B, and Q are all proportional to the identity matrix of order (and f is identical to zero), so that we want to optimally control an -dimensional Brownian motion.
Because of the importance of the matrix Riccati equation in many applications, the problem of linearizing this equation has been considered by a number of authors. Grasselli and Tebaldi [4], in particular, proposed a method that enabled them to transform the matrix Riccati equation that appeared in their work into 2 linear equations; see also Gourieroux and Sufana [5].
The aim of this paper is to generalize the theorem proved by Whittle [1] and, at the same time, the results in Lefebvre and Zitouni [3]. In the next section, first the two-dimensional case will be presented. Then, the results will be extended to the -dimensional case. In Section 3, a particular twodimensional problem will be solved explicitly. Finally, we will end with a few concluding remarks in Section 4.

Linearization of the Matrix Riccati Equation in Two Dimensions
Let If the relation in (23) holds, then the matrix M is symmetric and invertible. To generalize Whittle's theorem, we will assume that M is indeed symmetric and invertible but not necessarily proportional to N. For simplicity, we will present the linearization technique that we propose in the case of two-dimensional controlled diffusion processes. Equation (17) and the cost function becomes The two standard Brownian motions are assumed to be independent. Next, let where we have assumed that . The optimal control is given by ) .
Moreover, the matrix Riccati equation satisfied by the derivative G( 1 , 2 ) of the value function ( 1 , 2 ) with respect to x and that we want to linearize is given by [see (21)] Proposition 3. Assume that the matrix M defined in (24) is symmetric and invertible. Then, the function ( 1 , 2 ) defined by the transformation satisfies the linear partial differential equation where Furthermore, for the transformation to be valid, there must exist functions ( 1 ) and ( 2 ) such that Proof. We compute where the function is defined in (33). Substituting these expressions into (29), we find that the differential equation But this relation is verified for all matrices B, Q, and N that satisfy the conditions mentioned above. Notice, however, that from (30) we deduce two expressions for the value function ( 1 , 2 ). For the transformation to be valid, these two expressions must of course be compatible, which yields (34).

Remarks 4. (i) When (23) holds, so that we can apply
Whittle's theorem, the matrix K is given by where I 2 denotes the identity matrix of order 2. The condition in (34) becomes that is, We can obviously choose ( 2 ) = ( 1 ) ≡ 0 , a constant. Therefore, this condition is always satisfied when Whittle's theorem can be used. Moreover, it is clearly more likely to satisfy (23) [and (34)] when all the matrices are constant.
(ii) The function ( 1 , 2 ) must be strictly positive. In one dimension, ( ) is indeed strictly positive since, making use of (15), it can be expressed as an exponential function. When (23) (in two dimensions) is satisfied, it is also easy to prove that ( 1 , 2 ) is strictly positive.
(iii) The simplest problems that can be considered are such that the two controlled processes defined by (25) are independent, so that 12 = 0, and all the matrices are constant, which implies that is also a constant, for , = 1, 2. The linear function then reduces to (iv) Proposition 3 does not give us the function G from which one deduces the optimal control. Similarly to Whittle's theorem, it rather simplifies the optimal control problem. Indeed, it is generally easier to solve a linear than a nonlinear differential equation.
Since the proof is a simple extension of that of Proposition 3, we can state the following corollary.

Corollary 5. In the -dimensional case, the matrix Riccati equation [see (21)]
is transformed into a linear partial differential equation for the function ( 1 , . . . , ) defined through where provided that the expressions that we deduce from (43) for the value function (x) are compatible.
Remarks 6. (i) For the sake of brevity, we did not give the linear equation satisfied by ( 1 , . . . , ), but it is a simple matter to derive it.
(ii) The larger is, the more difficult it should be to obtain compatible expressions for the value function. Nevertheless, the result is clearly an improvement over Whittle's theorem.
International Journal of Differential Equations 5 (iii) If we define instead ( 1 , . . . , ) through the equation with then we find that ( 1 , . . . , ) satisfies a linear partial differential equation if which does not always hold true. Therefore, the transformation that we used is more appropriate.

Explicit Solution to a Particular Problem
In this section, we will make use of Proposition 3 to help us solve a particular LQG homing problem in two dimensions. Assume that and let where with 0 < 1 + 2 < . We assume that is positive. We calculate ) .

(51)
Remark 7. It is important to notice that, in this particular problem, the relation in (23) does not hold, since the matrix K is not proportional to the identity matrix of order 2. Hence, we could not appeal to Whittle's theorem to linearize the differential equation satisfied by the value function. From Proposition 3, we deduce that the function ( 1 , 2 ) here satisfies the linear partial differential equation To solve this differential equation, we will use the method of similarity solutions. That is, we assume that the function ( 1 , 2 ) can actually be written as with := 1 + 2 . Equation (52) is then transformed into the second-order ordinary differential equation whose general solution can be written as so that Let us take = 3/2, for simplicity. We then deduce from the preceding equation that which implies that Similarly, we obtain that so that Since ( 1 , 2 ) depends on both 1 and 2 in our problem, we conclude that we must set ( 1 ) = ( 2 ) ≡ 0 . Next, making use of the boundary conditions ( 1 , 2 ) = 0 if 1 + 2 = 0 or , we can write that International Journal of Differential Equations If 0 = 0, we find that Hence, When 0 ̸ = 0, we deduce from (62) that Thus, we obtain the following expression for the value function: which is valid for 0 ≤ 1 + 2 ≤ . Now, remember that we do not need to determine ( 1 , 2 ) explicitly to obtain the optimal controls * 1 ( 1 , 2 ) and * 2 ( 1 , 2 ). Only ( 1 , 2 ) = ( 1 , 2 ) is needed. Here, we compute for any constant 0 (and any 1 ). It follows that the optimal controls are given by [see (20)] ) .

Conclusion
In this paper, we have generalized a theorem due to Whittle that sometimes enables us to linearize the matrix Riccati equation satisfied by the derivative of the value function in the so-called LQG homing problems. In Section 3, we were able to use our result to obtain the explicit solution to such a problem in two dimensions, for which Whittle's theorem did not apply. Solving this type of problem explicitly is usually very difficult. Furthermore, notice that the optimal control * ( 1 , 2 ) that we obtained depends on 1 and 2 , for = 1, 2, rather than being constant. Therefore, the solution was not evident at all.
One cannot expect to be able to linearize the matrix Riccati equation (21) in all cases. Whittle's theorem depends on a quite restrictive condition. Here, we showed that there are other cases than the ones to which Whittle's result applies. We saw in Proposition 3 that the transformation that we propose will linearize (21) (with = 2), provided that (34) is satisfied. Since, as we mentioned above, our transformation will work every time Whittle's theorem applies, we can at least claim that we have improved Whittle's result.
Apart from the case when we can transform the stochastic optimal control problem into a purely probabilistic problem, few particular cases have been solved so far in two or more dimensions. The authors (see Lefebvre and Zitouni [6]) treated the general case in one dimension. They used symmetry, when possible, to obtain the exact optimal control, and they proposed a technique that yields a very good approximation to this optimal control when symmetry arguments cannot be used. It would be interesting to generalize these results in the -dimensional case.
Finally, we could try to apply the technique developed by Grasselli and Tebaldi [4], as well as other techniques proposed by various authors, to linearize the matrix Riccati equation in the case of LQG homing problems.