General LQG Homing Problems in One Dimension

Optimal control problems for one-dimensional diffusion processes in the interval d1, d2 are considered. The aim is either tomaximize or tominimize the time spent by the controlled processes in d1, d2 . Exact solutions are obtained when the processes are symmetrical with respect to d∗ ∈ d1, d2 . Approximate solutions are derived in the asymmetrical case. The one-barrier cases are also treated. Examples are presented.


Introduction
Let {X t , t ≥ 0} be a one-dimensional controlled diffusion process defined by the stochastic differential equation where u • is the control variable, m • and v • > 0 are Borel measurable functions, b 0 / 0 and k ∈ {0, 1, . ..} are constants, and {B t , t ≥ 0} is a standard Brownian motion.The set of admissible controls consists of Borel measurable functions.
Remark 1.1.We assume that the solution of 1.1 exists for all t ∈ 0, ∞ and is weakly unique.
We define the first-passage time where x ∈ d 1 , d 2 .We want to find the control u * that minimizes the expected value of the cost function where q 0 > 0 and λ / 0 are constants.Notice that if λ is negative, then the optimizer wants to maximize the survival time of the controlled process in the interval d 1 , d 2 , taking the quadratic control costs into account.In general, there is a maximal value that the parameter λ can take.Otherwise, the expected reward becomes infinite.
When the relation holds for some positive constant α, using a theorem in Whittle 1, p. 289 , we can express the value function in terms of a mathematical expectation for the uncontrolled process obtained by setting u X t ≡ 0 in 1.1 .Actually, for the result to hold, ultimate entry of the uncontrolled process into the stopping set must be certain, which is not a restrictive condition in the case of onedimensional diffusion processes considered in finite intervals.
In practice, the theorem in Whittle 1 gives a transformation that enables us to linearize the differential equation satisfied by the function F x .
In Lefebvre 2 , using symmetry, the author was able to obtain an explicit and exact expression for the optimal control u * when {X t , t ≥ 0} is a one-dimensional controlled standard Brownian motion process so that m X t ≡ 0 and v X t ≡ 1 , d 2 −d 1 d and k 1.Notice that the relation in 1.4 does not hold in that case.The author assumed that the parameter λ in the cost function is negative, and he found the maximal value that this parameter can take.
Previously, Lefebvre 3 had computed the value of u * when k 0, but with the cost function rather than the function J x defined above.We cannot appeal to the theorem in Whittle 1 in that case either.However, the author expressed the function F x in terms of a mathematical expectation for an uncontrolled geometric Brownian motion.
In Section 2, we will generalize the results in Lefebvre 2 to one-dimensional diffusion processes for which the functions m X t and v X t are symmetrical with respect to zero and d 1 −d 2 .Important particular cases will be considered.
Next, in Section 3, we will treat the general symmetrical case when d 1 is not necessarily equal to −d 2 .In Section 4, we will consider processes for which the functions m X t and v X t are not symmetrical with respect to a certain d * ∈ d 1 , d 2 .An approximate solution will then be proposed.In Section 5, we will present possible extensions, including the case of a single barrier.Finally, we will make some concluding remarks in Section 6.

Optimal Control in the Symmetrical Case with
In this section, we take d 2 −d 1 d.Assuming that it exists and that it is twice differentiable, we find that the value function F x defined in 1.5 satisfies the dynamic programming equation inf where x X 0 .Differentiating with respect to u x and equating to zero, we deduce that the optimal control u * x can be expressed as follows: Substituting the optimal control into the dynamic programming equation 2.1 , we obtain that the function F x satisfies the nonlinear second-order ordinary differential equation subject to the boundary conditions Now, in general solving nonlinear second-order differential equations is not an easy task.As mentioned previously, when the relation in 1.4 holds, there exists a transformation that enables us to linearize 2.3 .Notice, however, that in order to obtain an explicit expression for the optimal control u * x , one only needs the derivative of the value function F x .Hence, if we can find a boundary condition in terms of F x , rather than the boundary conditions in 2.4 , then we could significantly simplify our problem, since we would only have to solve the first-order nonlinear Riccati differential equation: where

2.6
Proposition 2.1.Assume that the function m x is odd and that the function v x is even.Then the optimal control u * x is given by where G x satisfies 2.5 , subject to the condition Proof.The condition 2.8 follows from the fact that, by symmetry, when the parameter λ is positive, then 0 is the value of x for which the function F x has a maximum, whereas F x has a minimum at x 0 when λ is negative.Indeed, the origin is the worst resp., best position possible when the optimizer is trying to minimize resp., maximize , taking the quadratic control costs into account, the time spent by X t in the interval −d, d .Remarks 2.2.i The solution to 2.5 , subject to 2.8 , might not be unique.
ii Notice that the symmetrical case includes the one when m x is identical to 0, and v x is a constant, so that the uncontrolled process is a Wiener process with zero drift.
The previous proposition can be generalized as follows.
Corollary 2.3.If X k t is replaced by h X t in 1.1 , where h 2 x is even, and if the hypotheses in Proposition 2.1 are satisfied, then the optimal control u * x can be expressed as that satisfies the condition G 0 0.
We will now present an example for which we can determine the optimal control u * explicitly.
We consider the case when {X t , t ≥ 0} is a controlled Bessel process, so that q 0 X 2k t .

2.12
Hence, we can appeal to the theorem in Whittle 1 when k is equal to zero.We will treat the case when k > 0 instead.The differential equation that we must solve is We find that where J ν and Y ν are Bessel functions and c 0 is an arbitrary constant,

2.15
The expression above for the function G x is appropriate when the parameter λ is negative.However, when λ > 0, it is better to rewrite it as follows: where I ν and K ν are modified Bessel functions and

2.17
In order to determine the value of the constant c 0 , we will use the condition G 0 0. First, we consider the special case when k 1, θ 1 and λ is negative.Then, we have that ν 1/4 and where c b 0 −λ/ 2q 0 .

International Journal of Stochastic Analysis
Next, when ν / − 1, −2, . .., we have the formula see 5, p. 358 It follows, with ν 1/4 and ν −3/4, that Finally, making use of the limiting form of the function J ν z when z → 0 see 5, p. 360 : It follows that we must set the constant c 0 equal to 0, which implies that

2.23
We then deduce from 2.7 with k 1 that the optimal control is given by

2.24
This formula for the optimal control is the same as the one obtained by Lefebvre 2 .Now, in the general case, proceeding as previously we find that when λ < 0, the function G x defined in 2.14 is such that when x decreases to zero,

International Journal of Stochastic Analysis 7
This expression may be rewritten as follows: where c i is a constant, for i 1, 2, 3, 4. Multiplying the numerator and the denominator by x θ/2 , we obtain that 2.27 Hence, we deduce that if θ ∈ 1, 2 , we must set the constant c 2 equal to 0. This implies that c 0 0, so that the constant c 4 is equal to 0 as well.It follows that the function G x is given by

2.28
This expression is valid as long as the denominator is positive.This is tantamount to saying that the parameter λ, which represents the instantaneous reward given for survival in the interval −d, d , must not be too large.Finally, the optimal control is

2.29
Now, if θ ∈ 0, 1 , it turns out that lim x → 0 G x 0 for any constant c 0 , so that the solution is not unique.However, this does not entail that we can choose any c 0 .For instance, in the particular case when θ 1/2, k 1, λ −1, b 0 1 and q 0 1/2, we find that

2.30
If we choose c 0 0, as in the case when θ ∈ 1, 2 , then the expression for the optimal control is

2.31
One can check that if d 1/2, then u * x < 0 for 0 < |x| < 1/2, which is logical because the optimizer wants to maximize the survival time in −1/2, 1/2 .But if we let c 0 tend to infinity, the optimal control becomes which is strictly positive for 0 < |x| < 1/2.Thus, when the solution to 2.5 , 2.8 is not unique, one must use other arguments to find the optimal control.One can obviously check whether the expression obtained for the optimal control does indeed correspond to a minimum or a maximum in absolute value .In the particular case considered previously, if we let we find that this function satisfies all the conditions of the optimal control problem set up in Section 1 and leads to a valid expression for the optimal control.Next, if the parameter λ is positive, we deduce from the function G x in 2.16 that the optimal control is given by when θ ∈ 1, 2 .However, when θ ∈ 0, 1 , again we do not obtain a unique solution to 2.5 and 2.8 .Moreover, contrary to the case when λ is negative, there is no constraint on this parameter when it is positive.That is, we can give as large a penalty as we want for survival in the continuation region.

Optimal Control in the Symmetrical General Case
In this section, we assume that d 1 and d 2 are not necessary such that d 1 −d 2 .Moreover, we assume that there is a transformation Y t g X t of the stochastic process {X t , t ≥ 0} such that the functions m Y t and v Y t are symmetrical with respect to zero.Then the optimal control problem is reduced to the one presented in the previous section.
A simple example of such a situation is the case when {X t , t ≥ 0} is a onedimensional controlled standard Brownian motion and d 1 / − d 2 .Then one can simply define to obtain a controlled Brownian motion with zero drift and variance parameter σ 2 a 2 in the interval −d, d .We can then apply Proposition 2.1 to find the optimal control.A more interesting example is the following one: assume that the controlled process {X t , t ≥ 0} is defined by That is, {X t , t ≥ 0} is a controlled geometric Brownian motion.Since this process is strictly positive, we cannot have where d > 1.
Notice that the relation in 1.4 only holds in the case when k 1.To obtain the control that minimizes the expected value of the cost function defined in 1.3 , we will transform the geometric Brownian motion process into a Wiener process by setting Y t ln X t .

3.5
The infinitesimal parameters of the process {Y t , t ≥ 0} are given by see, e.g., 6, p. 64

3.6
Hence, we can write that Y t satisfies the stochastic differential equation That is, {Y t , t ≥ 0} is a controlled standard Brownian motion.Moreover, the random variable T x becomes where y ∈ − ln d , ln d .We can find the function G y , from which the optimal control u * is obtained at once, for any choice of k ∈ {0, 1, . ..}.We will present the solution in the case when k 0. Furthermore, we let λ −1, b 0 1 and q 0 1/2.We then must solve the nonlinear ordinary differential equation 3.9 The solution that satisfies the condition G 0 0 is Hence, from Corollary 2.3, we can state that the optimal control is given by u * −2e −y G y .

International Journal of Stochastic Analysis
In terms of the original process, we have that Since λ < 0, this solution is only valid as long as it remains finite.That is, because we chose the value of λ, the constant d must not be too large.The optimal control is plotted in Figure 1 when d 3/2.Notice that u * x is positive when x < 1 and negative when x > 1, which is logical because the optimizer wants to maximize the survival time in the interval 2/3, 3/2 .However, the optimal control is not symmetrical with respect to 1.

Approximate Optimal Control in the Asymmetrical Case
We will now consider the case when the infinitesimal parameters of the controlled process {X t , t ≥ 0} do not satisfy the hypotheses in Proposition 2.1.In order to obtain the optimal control without having to find the function F x explicitly, we need a condition on G x .If we could determine the value x 0 of x in the interval d 1 , d 2 for which the function G x has a maximum or a minimum, then we would set G x 0 0.An approximate solution can be obtained by finding the value of x that maximizes the expected value of the time it takes the uncontrolled process that corresponds to {X t , t ≥ 0} to leave the interval d 1 , d 2 .Let e x denote this expected value.This function satisfies the ordinary differential equation see 6, p. 220 v x e x m x e x −1.

4.1
The boundary conditions are obviously e d 1 e d 2 0.

4.2
We can state the following proposition.
Proposition 4.1.Let x 0 be the value of x that maximizes the function e x defined previously.The optimal control u * x is approximately given by 2.7 , where the function G x satisfies 2.5 , subject to the condition G x 0 0.
To illustrate this result, we will present an example for which we can find the exact optimal control.We will then be able to assess the quality of the approximation proposed previously.
Let {X t , t ≥ 0} be the controlled Wiener process with drift μ / 0 and variance parameter σ 2 defined by dX t μdt b 0 u X t dt σdB t .

4.3
Because the relation in 1.4 holds with and ultimate entry of the uncontrolled process into the set {d 1 , d 2 } is certain, we can indeed appeal to Whittle's theorem to obtain the control that minimizes the expected value of the cost function J x defined in 1.3 .Assume that the parameter λ is positive, so that the optimizer wants X t to leave the interval d 1 , d 2 as soon as possible.We deduce from Whittle's theorem that the value function F x can be expressed as follows: where in which τ x is the same as the random variable T x in 1.2 , but for the uncontrolled process {ξ t , t ≥ 0} defined by dξ t μdt σdB t .

4.7
It is a simple matter to find that where International Journal of Stochastic Analysis for i 1, 2, and We then obtain that the exact optimal control is given by Now, the function e x : E τ x satisfies the ordinary differential equation

4.12
The unique solution for which e d 1 e d 2 0 is

4.13
The value of x that maximizes e x is obtained by differentiation: Next, from 2.2 , the optimal control is given by where G x : F x satisfies the nonlinear differential equation We find that the solution of this equation is the following: The constant c 1 is uniquely determined from the condition G x 0 0. We have that

4.18
The expression that we obtain for the approximate optimal control by multiplying the function G x by −b 0 /q 0 is quite different from the exact optimal control.To compare the two solutions, we consider the special case when μ σ b 0 q 0 λ 1, d 1 0 and d 2 2. We then find that the constant Δ is equal to √ 3, and the value that maximizes G x is approximately x 0 0,7024.We plotted the two controls in Figure 2. Notice how close the two curves are.

Extensions
To complete this work, we will consider two possible extensions of the results presented.First, suppose that the random variable T x defined in 1.2 is replaced by That is, we want to solve a one-barrier, rather than a two-barrier problem.To do so, we can introduce a second barrier, at x d * .In general, it will be necessary to find a transformation Y t g X t for which the infinitesimal parameters of the uncontrolled process {Y 0 t , t ≥ 0} that corresponds to {Y t , t ≥ 0} satisfy the hypotheses in Proposition 2.1.If we can find such a transformation, then we can try to obtain the optimal control u * y for the transformed process.Finally, we must express the optimal control in terms of the original variable x and take the limit as

International Journal of Stochastic Analysis
Remark 5.1.If there is a natural boundary at the origin, for example, and if d * < d, then we would take the limit as d * decreases to zero.
We will now present an example where the technique described previously is used.We consider the controlled geometric Brownian motion defined in Section 3 by Hence, we can appeal to Proposition 2.1 to determine the optimal value of the control u y .Assume that k 1 in 5.2 .Then Whittle's theorem applies with The optimal control is given by u * x − b 0 q 0 xF x , 5.9 and, as in Section 4, the value function F x can be expressed as

5.12
The general solution of this equation can be written as

5.13
We assume that the parameter λ is positive.Then, we can write that lim x → ∞ M x 0.

5.14
It follows that we must choose the constant c 1 0 in the general solution.Finally, making use of the boundary condition M d 1, we obtain that from which we deduce that the optimal control is constant: If we do not appeal to Whittle's theorem, we must solve the nonlinear first-order differential equation see Section 3 G y 0, 5.17 subject to the boundary condition G 0 0. We find that

International Journal of Stochastic Analysis
It follows that In terms of the original variable x e y /a √ dd * e y , we can write that for any positive constant c 0 , we obtain that lim Thus, we retrieve the formula for the optimal control.Next, we will treat the case when k 0 in 5.2 , so that Whittle's theorem does not apply.The optimal control becomes u * x − b 0 q 0 G x .

5.23
In terms of the transformed variable y, we have that where G y is a solution of see Section 3 When λ is positive, the solution that satisfies the condition G 0 0 is

5.29
Finally, using the asymptotic expansions for large arguments of the functions I ν z and K ν z see 5, p. 377 , we find that lim which is the same optimal control as in the case when k 1.
Remarks 5.2.i If we take the limit as d * decreases to zero in u * x instead, then making use of the formulas see 5, p. 375 ii We can also try to solve the differential equation satisfied by the function G x directly.However, the solution that we are looking for must be such that G √ dd * 0, because x √ dd * is the value of the original variable x that corresponds to y 0.

International Journal of Stochastic Analysis
Now, in Corollary 2.3 we mentioned that Proposition 2.1 could be generalized by replacing X k t by h X t in 1.1 .Another extension of Proposition 2.1 is to generalize the cost function J x defined in 1.3 to where the function q • ≥ 0 is even.
To illustrate this result, we consider a particular controlled Ornstein-Uhlenbeck process defined by dX t −X t dt u X t dt dB t , 5.35 and we take

5.37
Then, we find that the optimal control is given by and that the function G x satisfies the nonlinear differential equation subject to the condition G 0 0. Next, by symmetry, we can write that G −x −G x and that u * −x −u * x .Hence, we can restrict ourselves to the interval 0, d .The solution of the differential equation that is such that G 0 0 is where "erf" is the error function.It follows that the optimal control is given by

Conclusion
We have shown that when the LQG homing problem that we want to solve possesses a certain symmetry, then it is not necessary to obtain the value function F x explicitly; only the derivative of F x is needed to determine the optimal control.Using this result, we were able to solve various problems for which Whittle's theorem does not apply.In Section 4, we proposed an approximate solution in the case when the infinitesimal parameters of the controlled processes are not symmetrical with respect to the origin.Many papers have been written on LQG homing problems, in particular by the first author see, e.g., 7 and recently by Makasu 8 .In most cases, the problems considered were only for one-dimensional processes, because to apply Whittle's theorem a certain relation must hold between the noise and control terms.This relation is generally not verified in two or more dimensions.Furthermore, even if the relation in question holds, we still must solve a nontrivial probability problem.More precisely, we need to evaluate the moment-generating function of a first-passage time.To do so, we must find the solution of a Kolmogorov backward equation that satisfies the appropriate boundary conditions.
Proceeding as we did in this paper, we could simplify, at least in the symmetrical case, the differential equation problem, even in more than one dimension.Therefore, we should be able to solve more realistic problems.Such problems will also have interesting applications.
Finally, in order to be able to treat real-life applications, we should try to find a way to solve problems that are not symmetrical and for which Whittle's theorem does not apply.This could be achieved by finding a transformation that linearizes the differential equations that we need to solve.
x ≤ 2. This function is plotted in Figure3.