Existence of Solutions of a Riccati Differential System from a General Cumulant Control Problem

We study a system of infinitely many Riccati equations that arise from a cumulant control problem, which is a generalization of regulator problems, risk-sensitive controls, minimal cost variance controls, and k-cumulant controls. We obtain estimates for the existence intervals of solutions of the system. In particular, new existence conditions are derived for solutions on the horizon of the cumulant control problem.


Introduction
Consider a linear control system and a quadratic cost function: where A t ∈ R n×n , B t ∈ R n×m , Q t ∈ S n , and R t ∈ S m are continuous matrix functions for t ∈ t 0 , t f , Q f ∈ S n S n is the set of n × n symmetric matrices., x t ∈ R n is the state with known initial state x 0 , u t ∈ R m the control, and w t ∈ R p a standard Wiener process.Because x is completely determined by the first equation in 1.1 in terms of u, the cost function J is only a function of u.

International Journal of Differential Equations
For θ ∈ R, denote by E e θJ the general expectation of the random variable e θJ .For i ≥ 1, let κ i d i dθ i ln E e θJ | θ 0 1.2 be the ith cumulant of the cost J u .Let {μ i } ∞ i 1 be a sequence of nonnegative real numbers.Consider the following combination of κ i : The cumulant control problem, considered in 1 , is to find a control u that minimizes the combined cumulant κ defined in 1.3 .This problem leads to the following system of infinitely many equations of Riccati type: where d/dt denotes the derivative to t, A t ∈ R n×n , B t ∈ R n×m , Q t ∈ S n , R t ∈ S m are as in 1.1 , and W t G t G T t ∈ S n .K t ∈ R m×n and H i t ∈ S n are the unknown matrix functions, which are required to be continuously differentiable for t ∈ t 0 , t f .For convenience, the time variable t is often suppressed.System 1.4 will be combined with the following equation 1.5 If k ≥ 1 is a given integer and μ i 0 for i > k, then and the cumulant problem is the classical regulator problem that minimizes E J .If k 2, then the cumulant problem is the minimal cost variance control considered in 4, 5 .Interested readers are referred to 1-6 for the investigations, generalizations, and applications of cumulant controls.
Another important cumulant control occurs when μ i θ i−1 for i ≥ 1.In this case, κ 1/θ ln E e θJ is precisely the cumulant generating function, and the cumulant problem is the risk-sensitive control; see, for example, 7 .In this case, 1.4 and 1.5 lead an equation for the matrix function P θ; t ∞ i 1 θ i−1 H i t : As shown in 1 , the solution H i t of 1.4 is related to P θ; t by the equation: and the equations in 1.4 for H i t can be obtained by differentiating 1.6 to θ at θ 0. For a feedback control u Kx with a given matrix function K, it was shown in 8 and 1, Theorem 2 that the ith cumulant κ i of J u has the following representation: where {H i } ∞ i 1 is the solution of 1.4 .Consequently 1.3 can be written as In 1 the cumulant control problem was restated as minimizing κ in 1.9 with K as a control, {H i } ∞ i 1 as a state, and 1.4 a state equation.Furthermore, the following result is proved in 1, Theorem 3 .Theorem 1.1.If the control u Kx is the optimal feedback control of 1.9 , then the solution {H i } ∞ i 1 of 1.4 and K must satisfy 1.5 .
By Theorem 1.1, it is necessary to solve 1.4 and 1.5 in order to find a solution of a cumulant control problem.Because of the nonlinearity of 1.4 and 1.5 in {H i } ∞ i 1 , a global solution may not exist on the whole horizon t 0 , t f of the cumulant problem.This can be illustrated by a scalar case of 1.6 .Suppose A 0, B G θ R Q Q f 1, then 1.6 becomes P P 2 1 0 and P t f 1.The solution is P t tan π/4 t f −t , which is defined on s, t f with s t f − π/4.So 1.6 has no solution unless t 0 > t f − π/4.
By the local existence theory of differential equations, the solutions K and {H i } ∞ i 1 of 1.4 and 1.5 exist on a maximal subinterval s, t f ⊂ t 0 , t f .Our interest is to give an estimate for this interval.In particular, we will obtain conditions that guarantee s t 0 .The idea of our approach is to show that the trace tr H of satisfies a scalar differential inequality: with some functions a, b, c on t 0 , t f .A key of the proof is Proposition 2.1 below.It follows that tr H is bounded by the solution of the Riccati equation: where z f tr Q f .Consequently, the existence interval of 1.12 gives an estimate for that of system 1.4 and 1.5 ; see Theorems 2.4 and 3.5 below.By a similar argument, we prove that the cumulant problem is well posed under appropriate conditions; see Theorems 2.3 and 3.4 below.
In 9 the norm of a solution of a coupled matrix Riccati equation was shown to satisfy a differential inequality similar to 1.11 .Consequently, specific sufficient conditions were derived for the existence of solutions of the Riccati equation in 9 .Estimates for maximal existence interval of a classical Riccati equation had been obtained in 10 in terms of upper and lower solutions.For the coupled Riccati equation associated with the minimal cost variance control, some implicit sufficient conditions had been given in 11 for the existence of a solution.In this paper, we use the trace tr H to bound the solution of system 1.4 and 1.5 , which generally leads to a better estimate for the existence interval.

Comparison Results for Traces
We start with an assumption and some preparations.In this paper we assume that

2.1
For the sequence μ {μ i } ∞ i 1 in 1.3 , we will assume that where Note that the assumption μ 1 1 is not essential.The assumption that μ i ≥ 0 for i ≥ 1 and Proposition 2.1 below imply that the matrix H in 1.4 is a positive semidefinite series.The requirement that ρ μ < ∞ imposes some growth condition for the sequence μ; see the proofs of Theorems 2.3 and 2.4.
Also note that if θ ρ μ < ∞, then μ i ≤ θμ i−1 and μ i ≤ θ i−1 for all i ≥ 1. Theorem 3.4 below shows that the cumulant control problem is well posed for any sequence μ with a small ρ μ .Some examples of ρ μ are as follows: We need the following properties of Proof.The formula of the ith cumulant κ i in 8 implies that κ i is nonnegative for all x 0 .It follows from the representation 1.9 that H i t 0 must be positive semidefinite.This argument continues to hold with t 0 replaced by any s ∈ t 0 , t f .Next we verify some properties related to matrix trace that are needed for our analysis.For A ∈ R n×n , denote by tr A n i 1 a ii the trace of A, and λ 1 A and λ n A the smallest and largest eigenvalue of A, respectively.
Proof.The properties in a are obvious by the definitions of trace and matrix multiplication.Some of the inequalities in b -d might be known, but the authors were not able to find proofs in existing literature.So we include our proofs of b -d below for readers' convenience.
To prove b , let T be a unitary matrix such that B T * DT where D is diagonal with where TAT * ii is the entry of TAT * at i, i .Let v i be the ith row of T , which is a unit vector.

International Journal of Differential Equations
To show c , use the symmetry of A, B, and C; we get that tr ABC ijk a ij b jk c ki ijk a ji c ik b kj tr ACB .Inequality 2.6 follows from part b and the fact that λ n A ≤ tr A since A ≥ 0. To show 2.7 , first note that tr ABC ≤ λ n AB tr C by 2.4 ; then it remains to show that

2.11
Since A, B ≥ 0, we have For 2.8 , using notations in the proof of b , we first get

2.12
Then 2.8 follows from the inequalities

2.13
Now we estimate the existence intervals of solutions of 1.4 and 1.5 .First, let K be given and {H i } ∞ i 1 be the solution of system 1.4 .We have the following result, which will be used in the proof of Theorem 3.5 Theorem 2.3.Suppose ρ μ < ∞ and K in 1.4 is given.Let a 1 , b 1 , and c 1 be functions on t 0 , t f satisfying has a solution on t 0 , t f , then system 1.4 has a solution

International Journal of Differential Equations 7
Proof. a Denote F A BK. Multiplying the equation in 1.4 for H i by μ i and sum over i, we obtain 2.17 Taking traces of both sides of 2.17 gives tr H j WH i−j .

2.18
Note that W GG T ≥ 0 and by Proposition 2.

2.20
Substituting 2.19 and 2.20 into 2.18 and using the definition of c 1 in 2.14 we get

2.21
b Suppose that 2.16 has a solution z * t on t 0 , t f .By local existence theory, system 1.4 has a solution {H i } ∞ i 1 on a maximal interval s, t f ⊂ t 0 , t f .By a , tr H satisfies inequality 2.15 ; that is tr H is a lower solution of 2.16 .By a comparison theorem of lower-upper solutions, tr H ≤ z * t on s, t f .Since series 1.10 is positive semidefinite, it follows that H and H i are all bounded and 1.10 is convergent on s, t f .Since {H i } ∞ i 1 satisfies system 1.4 , each H i is in fact continuously differentiable on s, t f .If s > t 0 , then the local existence theory implies that {H i } ∞ i 1 can be extended further to the left of s, a contradiction to the maximality of s, t f .Therefore s t 0 and 1.4 has a solution on t 0 , t f , which can be extended to t 0 , t f .Now consider system 1.4 and 1.5 .We have International Journal of Differential Equations Theorem 2.4.Denote R BR −1 B T .Let a 2 , b 2 , and c 2 be functions on t 0 , t f satisfying a Suppose K and {H i } ∞ i 1 are solutions of 1.4 and 1.5 on some s, t f .Then on s, t f , z tr H satisfies has a solution on t 0 , t f , then system 1.4 and 1.5 have solutions K and {H i } ∞ i 1 on t 0 , t f .Proof. a Substituting K −R −1 B T H into system 2.17 we get where R BR −1 B T .Taking traces of both sides of 2.25 gives tr H j WH i−j .

2.26
As in the proof of previous theorem, we have

2.27
Using the fact that H, R ≥ 0 and 2. i 1 are continuously differentiable on s, t f .If s > t 0 , then local existence theory implies that K and {H i } ∞ i 1 can be extended further to the left of s, a contradiction to the maximality of s, t f .Therefore s t 0 , and system 1.4 and 1.5 have solutions on t 0 , t f .

Well-Posedness and Sufficient Existence Conditions
In this section we will derive specific conditions that ensure that the scalar equations 2.16 and 2.24 have solutions on t 0 , t f .Consequently we will obtain sufficient conditions for the well-posedness of the cumulant control and the existence of solutions of 1.4 and 1.5 .First we consider an autonomous scalar equation where z is a polynomial with degree ≥ 2. Assume for some k ≥ 0 that h z has k distinct zeros Since h z is locally Lipschitz, the solution z t of 3.1 exists and is unique for every z f for t in a maximal interval, say σ, t f .If z f z i for some i 1, . . ., k, then z t z i for all t.If z f ∈ z i , z i 1 for some i 0, 2, . . ., k, then z t ∈ z i , z i 1 for t ∈ σ, t f .This implies that for t ∈ σ, t f , −z t h z t has the same sign as h z f .In particular, as t ∈ σ, t f decreases, z t is strictly increasing if h z f > 0 and decreasing if h z f < 0. Denote z f lim t → σ z t .Then The following is a well-known fact in stability theory of differential equations.
If h z i < 0, then z f z i for every z f ∈ z i−1 , z i 1 , where d/dz.Indeed, h z i < 0 implies that h z f < 0 for z f ∈ z i , z i 1 and h z f > 0 for z f ∈ z i−1 , z i .In either cases, z f z i by 3.2 .Consider 1.12 as an example.We have the following.
Proof.If a 0, then h z bz c, which has root z 1 −c/b.Since h z 1 b < 0, z f z 1 by 3.3 .This shows a .
Next we prove b and c .First assume a 1.
In either cases, we have z f ∞ by 3.2 .This finishes the proof of b and c when a 1.If a / 1, then consider y az, which satisfies y y 2 by ac 0 and y t f az f , and the conclusions follow from the special case just proved.
International Journal of Differential Equations Write 3.1 as dz/h z dt 0 and integrate it against t from t f to σ, then we get Note that if z f is finite, then z f must be a zero of h z .It follows that Applying Proposition 3.2 to 1.12 we obtain the following.Proposition 3.3.a If either a 0 and b < 0, or a z f − z 1 < 0 and Δ ≥ 0, then the solution of 1.12 exists on −∞, t f .
b If either Δ < 0 or Δ ≥ 0 and a z f − z 1 < 0, then the solution of 1.12 exists on a finite interval σ, t f with

3.6
Next assume Δ 0 and a z f − z 1 > 0. Then z f sgn a ∞ and az 2 bz c a z − z 1 2 , where z 1 −b/ 2a .We have International Journal of Differential Equations 11 Finally when Δ > 0 and z f sgn a ∞, we have az 2 bz c a z − z 1 z − z 2 and Now we show that the cumulant control problem is well posed by Theorem 2.3 and Proposition 3.3.Theorem 3.4.For any number L > 0 there is ρ 0 > 0 such that the series ∞ i 1 μ i κ i /i! in 1.3 converges for each matrix K and sequence μ with ||K|| ∞ < L and ρ μ < ρ 0 .
Proof.Suppose that K is a matrix function with ||K|| ∞ < L. Choose a 1 , b 1 , and c 1 as follows: 3.9 Note that c 1 ≤ tr Q ∞ L 2 R ∞ , which depends only on L. In addition, a 1 → 0 as ρ μ → 0. It follows that when ρ μ is sufficiently small, h z a 1 z 2 b 1 z c 1 has two real roots with z 2 → −c 1 /b 1 and z 1 → −∞ as ρ μ → 0. In particular, since z f ≥ 0, we have h z f > 0 and so z f ∞.Proposition 3.3 implies that In particular, 2.16 has a solution z t on t 0 , t f when ρ μ is sufficiently small.By Theorem 2.3 b , system 1.4 has a solution {H i } ∞ i 1 such that H converges.
Finally we apply Proposition 3.3 to 2.24 to give a sufficient existence condition for 1.4 and 1.5 and the cumulant control problem.Choose 3.12 Theorem 3.5.System 1.4 and 1.5 have solutions K and where z f is defined as in 3.2 with h z a 2 z 2 b 2 z c 2 .In particular, system 1.4 and 1.5 have solutions K and {H i } ∞ i 1 on t 0 , t f if one of the following holds.a a 2 −λ 1 R /n 2ρ tr W < 0. Proof.The general conclusion follows directly from Proposition 3.3 and Theorem 2.4.In the case a , h z 0 has two roots z 1 < 0 < z 2 .Since z f tr Q f ≥ 0, a 2 z f − z 1 < 0. In the case b , h z 0 has two solutions z 2 ≤ z 1 .So a 2 z f − z 1 < 0 also holds.The conclusion follows from Propositions 3.1 and 3.2 and Theorem 2.4.
Note that in Theorem 3.5 condition a holds if B has full rank i.e., λ R > 0 and ρ μ is sufficiently small, while condition b holds if the system in 1.1 is stable i.e., b 2 < 0 and the product ρ μ tr Q ∞ is relatively small.The cumulant control problem has an optimal control under each of these conditions.
As an existence theorem, Theorem 3.5 gives one of the very few existence results for a Riccati differential system of infinitely many equations.In terms of the cumulant controls that lead to the system 3 and 4 , Theorem 3.5 generalizes the corresponding results in 1, 5 for risk-sensitive controls where μ {1, θ, θ 2 , . ..} and in 2, 4 for finite cumulant controls where μ has only finite nonzero components .Numerical examples for risk-sensitive and finite cumulant controls satisfying the conditions in Theorem 3.5 may be found in 3-6 .

Conclusions
In general it is very difficult to determine the existence interval of a differential Riccati equation or system .By the approach in this paper, we can at least give an estimate for the existence interval of the Riccati system.Such an estimate leads to sufficient conditions for the existence of solutions to the Riccati system and the cumulant control problem.

3 . 5 Proof. 2 d 2
Part a directly follows from Proposition 3.1 a and Proposition 3.2 a b .For part b , first assume that Δ < 0. Then z f sgn a ∞ and az 2 bz c a z − z 1 , where d |Δ|/2|a|, z 1 −b/2a.So
1, H i , H ≥ 0 for t ∈ t f , t f .Proposition 2.2 a , b , and c imply that tr F T H HF 2 tr FH ≤ 2λ n F tr H b 1 z, tr H j WH i−j ≤ tr W tr H j tr H i−j .
must converge because h z has a degree ≥ 2. In summary, we have the following.Suppose h z is a polynomial of degree ≥ 2. Then a z f is finite if and only if the solution z t of 3.1 exists on −∞, t f .b z f ±∞ if and only the solution z t of 3.1 exists on a finite maximal interval σ, t f with length t f − σ