LQGHoming in a Finite Time Interval

where q0 > 0 and k / = 0 are constants. We want to find the control u∗ that minimizes the expected value of J(x). This type of problem is a special case of the ones that Whittle [1, page 289] termed LQG homing. Notice that if the constant k is negative, then the optimizer tries to maximize the survival time of the process in the interval (0,d), taking the quadratic control costs into account. LQG homing problems have been treated by various authors; see Kuhn [2], Lefebvre [3], and Makasu [4]. Kuhn and Makasu used a risk-sensitive cost criterion (see also Whittle [5, page 222]). In the general formulation given by Whittle, {X(t), t ≥ 0} is an n-dimensional process and the random variable T(x) is the moment of first entry of the joint variable (X(t), t) into a stopping set D ⊂ Rn × (0,∞). However, in practice, it is very difficult to obtain explicit solutions to problems in two or more dimensions (except in special instances). Moreover, in the papers published so far on homing problems, the hitting time T(x) was defined only in terms of X(t). Here, we consider the case when the optimizer stops controlling the diffusion process at most at time t1. Using a theorem in Whittle [1], we can state that the optimal control u∗ can be expressed as follows: u∗ = σ 2 0 b0 G′(x) G(x) , (5)


Introduction
Let {X(t), t ≥ 0} be the one-dimensional controlled diffusion process defined by the stochastic differential equation where a(•) is a real function, u(t) is the control variable, b 0 / = 0, and σ 0 > 0 are constants and {B(t), t ≥ 0} is a standard Brownian motion.We define the first-passage time where x < d, and the random variable where t 1 > 0 is a constant.Next, we consider the cost criterion where q 0 > 0 and k / = 0 are constants.We want to find the control u * that minimizes the expected value of J(x).This type of problem is a special case of the ones that Whittle [1, page 289] termed LQG homing.Notice that if the constant k is negative, then the optimizer tries to maximize the survival time of the process in the interval (0, d), taking the quadratic control costs into account.LQG homing problems have been treated by various authors; see Kuhn [2], Lefebvre [3], and Makasu [4].Kuhn and Makasu used a risk-sensitive cost criterion (see also Whittle [5,page 222]).
In the general formulation given by Whittle, {X(t), t ≥ 0} is an n-dimensional process and the random variable T(x) is the moment of first entry of the joint variable (X(t), t) into a stopping set D ⊂ R n × (0, ∞).However, in practice, it is very difficult to obtain explicit solutions to problems in two or more dimensions (except in special instances).Moreover, in the papers published so far on homing problems, the hitting time T(x) was defined only in terms of X(t).Here, we consider the case when the optimizer stops controlling the diffusion process at most at time t 1 .
Using a theorem in Whittle [1], we can state that the optimal control u * can be expressed as follows: where In the above formula, τ(x) is a random variable defined by with and {ξ(t), t ≥ 0} is the uncontrolled process that satisfies the stochastic differential equation That is, τ(x) is the random variable that corresponds to T(x) for the diffusion process obtained by setting u(t) = 0 in (1).Hence, the optimal control problem is reduced to the computation of the mathematical expectation G(x).Actually, for this result to hold, we must have P[τ(x) < ∞] = 1.However, in our case this condition is trivially satisfied because τ(x) ≤ t 1 .
In Section 2, we will obtain an explicit solution u * in the case when a[X(t)] ≡ μ > 0, so that Notice that the uncontrolled process {ξ(t), t ≥ 0} is then a Wiener process with drift μ and diffusion parameter σ 2 0 .Furthermore, we will choose the constant k = −q 0 σ 2 0 /b 2 0 .With this choice, the mathematical expectation G(x) simplifies to (11)

Optimal Control of a Wiener Process
Let m 1 (x) denote the expected value of the first-passage time τ 1 (x).In the case of the Wiener process defined by the function m 1 (x) satisfies the ordinary differential equation and is such that m 1 (x) = 0 if x = d.We find (see Lefebvre [6, page 220]) that Therefore, in the case when t 1 tends to infinity, the function G(x) is given by It follows from ( 5) that Now, to obtain the expected value of the random variable τ(x), we can condition on τ 1 (x): We may write that (18) Moreover, because the conditional probability density function of τ 1 (x), given that τ 1 (x) ≤ t 1 , is given by we have: The function f τ1 (x) (t) is known to be (see Lefebvre [6, page 219]) Making use of this formula, we can obtain an explicit expression for the mathematical expectation E[τ(x)] and hence, for the optimal control u * .To illustrate the results, we computed (numerically) the optimal control when b 0 = q 0 = σ 0 = μ = 1 and d = t 1 = 5.Looking at Figure 1, we see that the optimal control u * tends to zero much faster than u * t1=∞ = 1/(x − 6) as x tends to −∞.However, for x close to the boundary at d = 5, the two functions are similar.
Next, to see the effect of the constant t 1 on the optimal control, we computed the value of u * when x = 0 and t 1 varies from 0 to 20.This value is compared to u * t1=∞ = −1/6 in Figure 2. When t 1 decreases to 0, so does u * , as it should be.For t 1 ≥ 15 (approximately), we have u * u * t1=∞ .Finally, in Figure 3, we show the optimal control u * when d = 15 and t 1 = 5.Because E[τ 1 (x)] = 15 − x (see above), when x ≤ 5 (approximately), it is very unlikely that the uncontrolled process will hit the boundary at d = 15 before time t 1 = 5.Therefore, the optimal control is close to 0. Notice that lim d → ∞ u * = 0 (for a finite value of x).Indeed, we can write that lim d → ∞ P[τ 1 (x) > t 1 ] = 1.Hence, we deduce from ( 17) that E[τ(x)] = t 1 , which implies that G(x) = 1+t 1 , and thus u * d=∞ = 0.

Conclusion
We have considered LQG homing problems for which the optimizer controls the diffusion process in the time interval (0, T(x)], where the random variable T(x) is smaller than or equal to a fixed constant t 1 .Moreover, the termination cost function was chosen so that the optimal control was expressed in terms of the expected value of a first-passage time for the corresponding uncontrolled process.An application of this type of problem is the following: suppose that X(t) denotes the wear of a machine.The optimizer wants to maximize the lifetime of the machine.However, it is natural to assume that the machine will be replaced after a certain time, even if it is still in working order, because it might become obsolete.
To obtain a more realistic model for the wear of a device, we could use a degenerate two-dimensional diffusion process, as in Lefebvre [7].The most difficult problem would then be to compute the probability density function of the first-passage time τ 1 .