Some Results on Bellman Equations of Optimal Production Control in a Stochastic Manufacturing System

Many manufacturing enterprisers use a production inventory system to manage fluctuations in consumer demand for the product. Such a system consists of a manufacturing plant and a finished goods warehouse to store those products which are manufactured but not immediately sold. The advantages of having products in inventory are as follows: first, they are immediately available to meet demand; second, by using the warehouse to store excess production during low demand periods to be available for sale during high demand periods. This usually permits the use of a smaller manufacturing plant than would otherwise be necessary, and also reduces the difficulties of managing the system. We are concerned with the optimization problem to minimize the expected discounted cost control of production planning in a manufacturing systems with degenerate stochastic demand:


Introduction
Many manufacturing enterprisers use a production inventory system to manage fluctuations in consumer demand for the product.Such a system consists of a manufacturing plant and a finished goods warehouse to store those products which are manufactured but not immediately sold.The advantages of having products in inventory are as follows: first, they are immediately available to meet demand; second, by using the warehouse to store excess production during low demand periods to be available for sale during high demand periods.This usually permits the use of a smaller manufacturing plant than would otherwise be necessary, and also reduces the difficulties of managing the system.
We are concerned with the optimization problem to minimize the expected discounted cost control of production planning in a manufacturing systems with degenerate stochastic demand:

J p E
∞ 0 e −ρt h x t p 2 t dt 1.1 of production inventory control problem that is satisfied by the value function of this optimization problem.In Section 3 we have established the properties of the value function and have shown the existence of an unique solution associated with the reduced onedimensional Hamilton-Jacobi-Bellman HJB equation.Finally in Section 4 we present an application to production control of optimization problem 1.1 subject to 1.2 and 1.3 .

The Hamilton-Jacobi-Bellman Equation
Suppose u x, y, t : R n × R n × R → R is a function whose value is the minimum value of the objective function of the production inventory control problem for the manufacturing system given that we start it at time t in state x, and y.That is, where the value function u is finite valued and twice continuously differentiable on 0, ∞ .We initially assume that u x, y, t exists for all x, y, and t in the relevant ranges.Since 1.2 and 1.3 is a scalar equation, the subscript t here means only time t.Thus, x and y will not cause any confusion and, at the same time, will eliminate the need of writing many parentheses.Thus, dw t is a scalar.
To solve the problem defined by 1.1 , 1.2 , and 1.3 , let u x, y, t , known as the value function, be the expected value of the objective function 1.1 from t to infinity, when an optimal policy is followed from t to infinity, given x t x, y t y.

2.7
The exact meaning of these expressions comes from the theory of stochastic calculus; Arnold

2.9
Note that we have suppressed the arguments of the functions involved in 2.3 .
Canceling the term u on both sides of 2.9 , dividing the remainder by dt, and letting t → 0, we obtain the dynamic programming partial differential equation or Hamilton-Jacobi-Bellman equation where F * x is the Legendre transform of F x , that is, F * x min p>0 {p 2 px} −x 2 /4 and u x , u y , u xx , u yy are partial derivatives of u x, y, t with respect to x and y.

A Reduction to 1-Dimensional Case
In this subsection, the general two-dimensional HJB equation has been reduced to a onedimensional second-order differential equation.From the two-dimensional state space form one state x for inventory level and the other state y for demand rate , it has been reduced to one-dimensional form for z x/y the ratio of inventory to demand.
There exists a v ∈ C 0, ∞ such that u x, y y 2 v x/y , y > 0. Since u x yv x/y , u y 2yv x/y −xv x/y , u yy 2v x/y −2 x/y v x/y x/y 2 v x/y .Setting z x/y and substituting these in 2.10 , we have

2.12
Then the HJB equation 2.11 becomes where ρ −ρ σ 2 2A, A − A σ 2 , and F * z is the Legendre transform of F z , that is, The main feature of the HJB equation 2.13 is the vanishing of the coefficient of u xx for x 0 in partial differential equation terminology, then the equation is degenerate elliptic.Generally speaking, the difficulty stems from the degeneracy in the second-order term of the HJB equation 2.13 .

Value Function
Let us consider the minimum value of the payoff function is a function of this initial point.The value function can be defined as a function whose value is the minimum value of the objective function of the production inventory control problem 1.1 for the manufacturing system, that is,

2.14
The value function V z is a solution to the reduced one-dimensional HJB equation 2.

Stochastic Neoclassical Differential Equation for Dynamics of Inventory-Demand Ratio
As in the certainty optimal production control model, the dynamics of the state equation of inventory level 1.2 can be reduced to a one-dimensional process by working in intensive per capita variables.Define , per capita production.

2.16
To determine the stochastic differential for the inventory-demand ratio, z ≡ x/y, we apply It ô's lemma as follows:

2.18
From 1.3 , we have that dy 2 σ 2 y 2 dt.Substituting the above expressions into 2.18 , we have that the dynamics of z t to be the inventory-demand ratio at time t which evolves according to the stochastic neoclassical differential equation for demand 2.19

Riccati-Based Solution
This subsection deals with the Riccati-based solution of the reduced one-dimensional HJB equation 2.13 corresponding to the production inventory control problem 2.14 subject to 2.19 using the dynamic programming principle 17 .
To find the Riccati-based solution of HJB equation 2.13 , we refer to Da Prato 20 and Da Prato and Ichikawa 11 for the degenerate linear control problems related to Riccati equation in case of convex function like h z z 2 .By taking the derivative of 2.13 with respect to k and setting it to zero, we can minimize the expression inside the bracket of 2.13 i.e., F * v z min k≥−1 k 1 2 kv z with respect to k.This procedure yields

2.20
Substituting 2.20 into 2.13 yields the equation

Existence and Uniqueness
To solve the Bellman equation 2.13 let us consider this HJB equation associated with the discounted production control problem in the following form: where

3.2
We make the following assumptions: h : continuous function on R, h : non-negative, convex on R,

3.3
h satisfies the polynomial growth condition such that In order to ensure the integrability of J p , we assume that This condition 3.5 is needed for the integrability of z t or J p .Under 3.5 , we have Lemmas 3.1, 3.2, and Theorem 3.5, which ensures the finiteness of J p and hence the finiteness of an existence unique solution of HJB equation of v.
First we have established the properties of the value function of the optimal control problem.
Lemma 3.1.Under 3.5 and for each n ∈ N , there exists K > 0 such that 3.6 Proof.We have given its proof here to need the same kind of calculations in the future.By It ô's formula we have

3.13
Thus we get 3.9 with K > 0 independent of sufficiently small ρ.
Proof.For any > 0, there exist k, k ∈ P k such that where

3.15
We set 3.17 Hence, by convexity

3.18
Letting → 0, we get which completes the convexity of the value function v z .for some constant K > 0.Moreover, v admits a representation Proof.Since F * z min{ k 1 2 kz} is Lipschitz continuous, this follows from Bensoussan 21 in case of Assumption 3.4 except convexity.For the general case, we take a nondecreasing sequence h n ∈ C R convergent to h with 0 ≤ h n ≤ h.It is well known Bensoussan 21 that, for every n ∈ N , 3.1 has a unique solution v n for h n of the form in the class C 2 R of continuous functions vanishing at infinitely, where z t is a solution of 2.19 .

Journal of Probability and Statistics
To prove 3.29 , we recall 3.30 .Hence by 3.4 and Lemma 3.2 we have

3.32
This implies that v satisfies 3.29 .
To estimate v n on −s, s for s ∈ R , we remember the Taylor expansions of v ∈ C 2 R : Then wa can obtain the Landau-Kolmogorov inequality:

3.36
Now by 3.34 and 3.4 , we have sup

and hence
sup

3.39
By the Ascoli-Arzelà theorem, we have taking a subsequence if necessary.Passing to the limit, we can obtain 3.1 and 3.30 .Following the inequality 3.29 , we have E e − ρt v z t −→ 0 as t −→ ∞.

3.41
Hence by It ô's formula to e − ρt v z t , for convex function 19, page 219 , we have

3.42
By virtue of 3.1 and taking expectation on the both sides we have

3.43
Now by 3.41 , we obtain

An Application to Production Control
In this section we will study the production control problem to minimize the cost 2.14 over the class P k of all progressively measurable processes p t such that 0 ≤ p t ≤ k t and for the response z t to p t .
Let us consider the stochastic differential equation where β k z arg min F z , that is, We need to establish the following lemmas.

4.38
Therefore the optimal production cost control p * which minimizes the production control problem 1.1 subject to 1.2 and 1.3 .The proof is complete.

3 . 25 Now by 3 . 0 e
20 and taking expectation on the both sides, we obtain E τ − ρs ζg r z s ds e − ρτ g r z τ ≤ g r z − deduced 3.21 .The convexity of the value function V follows from the same line as Proposition 3.3 .Let z 0 t be the unique solution of dz 0 t Az 0 t dt − σz 0 t dw t ,

3 . 28 which implies 3 .Theorem 3 . 5 .
22 and satisfies 3.4 .Hence this completes the proof.Assume 3.3 , 3.4 , 3.5 .Then there exists a unique solution v ∈ C 2 R of 3.1 such that ρv z ≤ K 1 |z| n 3 , z ∈ R, 3.29 18, chapter 5 and Karatzas and Shreve 19 .For our purposes, it is sufficient to know the multiplication rules of the stochastic calculus: x dt h x dt • dt .
21known as the HJB equation.This is a partial differential equation which has a solution form Substituting 2.22 and 2.23 into 2.21 yields1 − ρa σ 2 a − a 2 2a A z 2 − 2az 0.2.24Since 2.24 must hold for any value of z, we must havea 2 − a 2 A σ 2 ρ − 1 ρs z s |z s | 2n−1 sgn z s dw s Now by 2.15 , 3.5 and taking expectation on the both sides, we obtain E e − ρt |z t | 2n ≤ |z| 2n 2nE Obviously, Z z is bounded above.Thus we can deduce 3.6 .
.44 which is similar to 3.30 and here the infimum is attained by the feedback law β v z with β θ arg min F θ .The proof is complete.
So, the uniqueness of 4.2 holds.Thus we conclude that 4.2 admits a unique strong solution z * t , Ikeda and Watanabe 22, Chapter 4, Theorem 1.1 , with E |z * t | 2n < ∞ .By 3.5 and It ô's formula, By 4.3 it is easily seen that zβ kv z ≤ k − 1 |z| if |z| ≥ a, for sufficiently large a > 0.By the same line as the proof of Lemma 3.1, we see that the right-hand side is bounded from above.This completes the proof.We first note by 4.3 that F k z min{ k 1 2 kz; k 1 ≥ 0}, and the minimum is attained by β k z .We apply It ô's formula for convex functions 19, page 219 to obtain Clearly F k z ≤ k 1 2 kz for every k ∈ P k .Again following the same construction of 3.45 and by the HJB equation 2.13 , we haveE e − ρt v z t ≥ v z − E So,the uniqueness of 4.21 holds.The proof is complete.By Lemma 4.3, we observe that p * t belongs to P. Now we apply It ô's formula −ρs σy s u y x * , y dw s .Again by the HJB equation 2.10 , we can obtain E e −ρt u x t , y t ≥ u x, y − E Theorem 4.4.Under 3.5 , the optimal production inventory cost control p * t is given by * ≤ J p .