Fluid Limits of Optimally Controlled Queueing Networks

We consider a class of queueing processes represented by a Skorokhod problem coupled with a controlled point process. Posing a discounted control problem for such processes, we show that the optimal value functions converge, in the fluid limit, to the value of an analogous deterministic control problem for fluid processes.


Introduction
The design of control policies is a major issue for the study of queueing networks.One general approach is to approximate the queueing process in continuous time and space using some functional limit theorem, and consider the optimal control problem for this (simpler) approximating process.Using functional central limit theorems leads to heavy traffic analysis; see for instance Kushner [1].Large deviations analysis is associated with the so-called "risk-sensitive" approach; see Dupuis et al. [2].We are concerned here with strong law type limits, which produce what are called fluid approximations.
The study of fluid limit processes has long been a useful tool for the analysis of queueing systems; see Chen and Yao [3].Numerous papers have considered the use of optimal controls for the limiting fluid processes as an approach to the design of controls for the "prelimit" queueing system.Avram et al. [4] is one of the first studies of this type.This approach was originally justified on heuristic grounds.Recent papers have looked more carefully at the connection between the limiting control problem and the queueing control problem.See, for instance, Meyn [5,6], and Bäuerle [7].
To be more specific, suppose X(t) is the original queueing process, which depends on some (stochastic) control u ω (•).Its fluid rescaling is X n (t) = (1/n) X(nt).The associated control is u n ω (t) = u ω (nt) and initial condition x n 0 = (1/n) x 0 .For each value of the scaling parameter n, we pose the problem of minimizing a generic discounted cost over all possible (stochastic) controls u n ω (•).In the limit as n → ∞ we will connect this to an analogous problem for a controlled (deterministic) fluid process x(t) (see (3.10) below): ( This is to be minimized over all (deterministic) control functions u(•).We will show (Theorem 5.1) that the minimum of J n converges to the minimum of J as n → ∞.This is essentially the same result as Bäuerle [7].We are able to give a very efficient proof by representing the processes X n (•) and x(•) using a Skorokhod problem in conjunction with controlled point processes.By appealing to general martingale representation results of Jacod [8] for jump processes, we can consider completely general stochastic controls.Bäuerle concentrated on controls of a particular type, called "tracking policies." Compared to [7], our hypotheses are more general in some regards and more restrictive in others.The benefit of our approach is the mathematical clarity of exhibiting all the results as some manifestation of weak convergence of probability measures, especially of probability measures on the space of relaxed controls.The principle ideas of this approach can all be found in Kushner's treatment of heavy traffic problems [1].Section 2 will describe the representation of X n (•) in terms of controlled point processes and a Skorokhod problem.A discussion of fluid limits in terms of convergence of relaxed controls is given in Section 3. Sections 2 and 3 will rely heavily on existing literature to bring us as quickly as possible to the control problems themselves.Section 4 develops some basic continuity properties of the discounted costs J n and J with respect to initial condition and control.Then, in Section 5, we prove the main result on convergence of the minimal values, and a corollary which characterizes all asymptotically optimal stochastic control sequences u n ω (•).Our technical hypotheses will be described as we encounter them along the way.
G. Pang and M. V. Day 3

Process description
An elementary example of the kind of network we consider is pictured in Figure 1.1.There is a finite set of queues, numbered i = 1,...,d.New customers can arrive in a subset A of them.(For the example of the figure, the appropriate subset is A= {1, 2}.) Arrivals occur according to (independent) Poisson processes with rates λ a j , j ∈ A. Each queue is assigned to one of several servers S m .For notational purposes, we let S m denote the set of queues i associated with it.(S 1 = {1, 2}, S 2 = {3} in the figure.)Each customer in queue i ∈ S m waits in line to receive the attention of server S m .When reaching the head of the line, it requires the server's attention for an exponentially distributed amount of time, with parameter λ s i .When that service is completed, it moves on to join queue i and awaits service by the server that queue i is assigned to.The value of i for a given i is determined by the network's specific routing.We use i = ∞ if type i customers exit the system after service.(In the example, 2 = 3 and 1 = 3 = ∞.)We insist that the queues be numbered so that i < i .This insures that each class of customer will exit the system after a fixed finite number of services.Thus, there are no loops in the network and the routing is predetermined.
Each server must distribute its effort among the queues assigned to it.This service allocation is described by a control vector u = (u 1 ,...,u d ) with u i ≥ 0, constrained by i∈Sm u i ≤ 1 for each m.Thus in one unit of time, the customer at the head of the line in queue i will receive u i time units of service.The set of admissible control values is therefore, In general, a control will be a U-valued stochastic process u ω (t).(The (•) ω serves as a reminder of dependence on ω ∈ Ω, the underlying probability space, to be identified below.)We want to allow u ω (t) to depend on all the service and arrival events that have transpired up to time t.In other words, X(t) and u ω (t) should be adapted to a common filtration Ᏺ t .However, the remaining amount of unserviced time for the customers is unknown to the controller at time t; it only knows the distributions of the arrival and service times and how much service time each customer has received so far.Let X(s) = (X 1 (s),...,X d (s)) be the vector of numbers of customers in the queues at time s.We can express X(s) as where x 0 = X(0) is the initial state, N a j (s), N s i (s) are counting processes which give the total numbers of arrivals and services of the various types which have occurred on the time interval (0,s], and δ a j , δ s i are event vectors which describe the discontinuity X(s) − X(s−) for each of the different types of arrivals and services.For a new arrival in queue 4 Journal of Applied Mathematics and Stochastic Analysis , and for service of a class i customer, then the +1 term is absent.)For a given scaling level n let t = s/n be the rescaled time variable.Let N n,a j (t) = N a j (nt), N n,s i (t) = N s i (nt) be the counting processes on this time scale.Then, we have (2.4) The difficulty with the representation (2.4) is that the N n,• i (t) depend on the control process which specifies their rates or "intensity measures," but additionally on the past realization of the N i because (regardless of the control) we need to "turn off " N n,s i when X n i (t) = 0 -it is not possible to serve a customer in an empty queue.This last feature is responsible for much of the difficulty in analyzing queueing systems.It imposes discontinuities in the dynamics as a function of the state.A Skorokhod problem formulation frees us of this difficulty, however.We can use controlled point processes N a j (t), N s i (t) to build a "free" queueing process Y n (t) as in (2.4), but without regard to this concern about serving an empty queue, and then follow it with the Skorokhod map Γ(•), which simply suppresses those jumps in N s i which occur when X n i (t) = 0. (We will see in the next subsection that there is no reason for N a j (t), N s i (t) to retain the n-dependence of N n,a j and N n,s i , hence its absence from the notation.)This gives us the following representation: (2.5) The next subsection will describe the controlled point processes, and the subsection following it will review the Skorokhod problem.We should note that not all queueing networks can be described this way.For a standard Skorokhod representation to be applicable, the routing i → i must be prescribed and deterministic.Fluid limit analysis is also possible if the routing is random: i → i with i chosen according to prescribed probabilities p ii ; see Chen and Mandelbaum [9] and Mandelbaum and Pats [10].The formulation of Bäuerle [7] allows the routing probabilities to depend on the control as well.The loss of that generality is a tradeoff for our otherwise more efficient approach.On the other hand, Skorokhod problem representations are possible for some problems with buffer capacity constraints, so our formulation provides the opportunity of generalization in that direction.
In the discussion above, we have viewed the initial position and control as those resulting from a given original X(t) by means of the rescaling: G. Pang and M. V. Day 5 But we are not really interested in following one original control u ω through this sequence of rescalings.We are interested in the convergence as n → ∞ of the minimal costs after optimizing at each scaling level.The control u n ω (•) which is optimal for scaling level n will not in general be a rescaled version of the optimal control u n+1 ω (•) at the next level.So from this point forward, the reader should consider the u n ω (•) to be any sequence of stochastic controls, with no assumption that they are rescaled versions of some common original control.They will all be chosen from the same set of progressively measurable U-valaued stochastic processes, so as we discuss below the construction of the processes (2.5), we will just work with a generic u ω (t).Then, as we consider convergence of the minial values, we will consider a sequence u n ω (t) of such controls, selected independently for each n in accord with the optimization problem.As regards the initial states x n 0 , the principal convergence result, Theroem 5.1, assumes that the (optimizied) X n all start at a common initial point: x n 0 = x (or convergent sequence of initial points: This means that we also want to discard the presumption that x n 0 are rescaled versions of some original initial point, and allow the x n 0 to be selected individually at each scaling level.

Free queueing processes and martingale properties.
For purposes of this section there is no need to distinguish between arrival and service events.We drop the superscripts (•) a and (•) s on λ i and N i (t), and simply enlarge the range of i to include both types of events: (We must also replace U by U × {1} |A| as the control space.)Thus the first equation of (2.5) becomes simply The central object here is the (multivariate) stochastic point process (We have omitted the stochastic reminder "(•) ω " here to make room for the coordinate index "(•) i .")The fluid scaling parameter n belongs in the intensity because the time scale t for X n (t) = (1/n)X(nt) is related to the original time scale s by nt = s.Later in the section, we identify the underlying probability space and state an existence result.Bremaud's treatment [11] describes the relationship between N i (t) and the intensities as a special case of marked point processes, using the mark space E = {1, ...,m}.Each component N i (t) is piecewise constant with increments of +1, characterized by the property that is a martingale for each vector of (bounded) predictable processes C i (s).(See [11, Chapter VIII D2 and C4] with H(s,k) = C k (s).)We note that this formulation precludes simultaneous jumps among the different N i .The interpretation of a marked point process is that each jump time τ n is associated with exactly one of the marks k ∈ E and only that component of the point process is incremented: (To allow simultaneous jumps, one would use a different mark space, E = {0, 1} m say, with an appropriately formulated transition measure.)This is consistent with our understanding that the arrival and service distributions are such that two such events occur simultaneously only with probability 0. We will address the existence of such N i (t) given a control process u ω (t) at the end of this section; but first we continue to describe the essential properties of the associated free queueing process Y n (t), constructed from N i (t) and prescribed event vectors δ i as in (2.6).It follows that is a (vector) martingale, null at 0. Here, v : which will play a prominent role in the fluid limit processes below.We note that v(u) is continuous and bounded.We will call M n the basic martingale for the free queueing process Y n with control u ω (t).We see that M n is the difference between Y n and a continuous fluid process It is significant that the factors of n cancel leaving no n-dependence in the v(u) term.However, M n does depend on the fluid scaling parameter n, as is apparent in the following result on its quadratic variation.
Lemma 2.1.The quadratic covariations of the components of M n are given by where δ i, j is the jth component of δ i : We have been careful to use angle brackets which, following the usual convention, distinguish the previsible quadratic covariation from the standard quadratic covariation [M n j ,M n k ](t).The later is discontinuous wherever M n (t) is; see [12,Theorem IV.36.6].The right-hand side of the expression in the lemma is obviously continuous, which makes it previsible, and thus the angle bracket process.Lemma 2.1 can be established in several ways; see Kushner [1, Section 2.4.2] for one development.Because keeping track of the proper role of the scaling parameter n can be subtle, we offer a brief independent proof.
Proof.Pick a pair of indices j, k; our goal is to show that G. Pang and M. V. Day 7 is a martingale.Since M n (t) is a process of finite variation, its square bracket process is simply the sum of the products of its jumps, see [12,IV (18.1)]: which makes the following a (local) martingale: For us, ΔM n j (s) = ΔY n j (s), and because there are no simultaneous jumps, it follows from our equation (2.6) for Y n that but this is the left-hand side of (2.7) with C i (s) = (1/n 2 )δ i, j δ i,k .It follows then that is also martingale.Combining (2.16) with (2.14), we see that (2.12) is a local martingale.
From the boundedness of the integral term in (2.12), it is relatively easy to apply Fatou's lemma to remove the "local." The importance of the quadratic variation is that it provides the key to passing to the fluid limit M n → 0 with respect to the uniform norm on compacts in probability as There is a constant C q (independent of the control), so that (2.17) (Here and throughout, | • | denotes the Euclidean norm on R d .) Proof.This is just Doob's inequality [13, Theorem II.70.1]: where the last inequality follows from Lemma 2.1 if we pick C q , so that 4d i λ i δ 2 i,k u i ≤ C q for each k = 1,...,d and all u ∈ U.
We now return to the issue of existence.If we are given u ω (t) defined on some filtered probability space, one might imagine various ways to construct from it a point process N(t) with the desired property (2.7).However, we want to allow the control u ω (t) to depend on the history Y n (s), 0 ≤ s < t of the queueing process X n = Γ(Y n ).If we build N(t), and then X n , after having fixed u ω , then we will have lost the dependence of u on Y n which we intended.The resolution of this dilemma is to prescribe both N(t) and u ω (t) in advance and then choose the probability measure to achieve (2.7).This is the martingale problem approach.We are fortunate that it has been adequately worked out by Jacod [8].The key is to take a rich enough underlying probability space.In particular, we take Ω to be the canonical space of paths for multivariate point processes: the generic ω ∈ Ω is ω = (α 1 (•),...,α m (•)), where each α i (t) is a right continuous, piecewise constant function with α i (0) = 0 and unit jumps.Define N(t,ω) to be the simple point evaluation: and take Ᏺ t to be the minimal or natural filtration: and Ᏺ = Ᏺ ∞ .The fundamental existence and uniqueness result of Jacod, [ ], applied in our context, is the following.
In other words, with both u ω (t) and N(t) defined in advance on Ω (thus preserving any desired dependence of u ω (t) on Y n (s), s ≤ t), we can choose the probability measure (uniquely) so that the correct distributional relationship between N(t) and u ω (t) (as expressed by (2.7)) does hold.Thus, u ω controls the distribution of N(•) by controlling the probability measure, not by changing the definition of the process itself.
We thus consider an admissible stochastic control to be any progressively measurable u ω (t) ∈ U defined on the canonical filtered probability space (Ω,{Ᏺ t }) of the theorem.Given a scaling parameter n ≥ 1, this determines a unique probability measure P n,uω so that the canonical N(t) = N(t,ω) is a stochastic point process with controlled intensities nλ i u i (t) as defined by (2.7).The free queueing process Y n (t) is now constructed as in (2.5).
We can now see the basis of our remark just above (2.5) that there was no need for n dependence in the counting processes of (2.5): the counting processes are always defined in the same canonical way (2.19) on Ω, regardless of n and regardless of the control.Only the probability measure P n,uω itself actually depends on n and u ω .

The Skorokhod mapping.
With N i (t) in hand and Y n (t) constructed as in (2.6), we need to produce X n (t) by selectively repressing those jumps in the N s i (t) which would correspond to serving empty queues.Reverting to separate indexing for arrivals j ∈ A and services the cumulative number of N s i jumps that have been suppressed up to time t.This will give us where I − Q is the matrix whose rows are the −δ s i , and The problem, given Y n (t), is to find X n (t) and K n (t) so that X n i (t) ≥ 0 and the K n i (t) are nondecreasing and increase only when X n i (t) = 0.This is the Skorokhod problem in the nonnegative orthant R d + as formulated by Harrison and Reiman [14].Although Harrison and Reiman only considered this for continuous "input" Y n (t), Dupuis and Ishii [15] generalized the problem to right continuous paths with left limits and more general convex domains G. We consider G = R d + exclusively here, but describe the general Skorokhod problem in the notation of [15].Given The essential well-posedness and continuity properties of the Skorokhod problem are summarized in the following result, a compilation of results of [15].D G denotes the space of all right continuous functions in R d + with left limits; see Section 3. Theorem 2.4.The Skorokhod problem as stated above has a unique solution for each right continuous ψ(•) with ψ(0) ∈ G bounded variation on each [0,T].Moreover, the Skorokhod map Γ is Lipschitz in the uniform topology.That is, there exists a constant C Γ so that for any two solution pairs (2.23) Γ has a unique extension to all ψ ∈ D G with ψ(0) ∈ R d + , which also satisfies (2.23).
Observe that if φ(•), η(•) solve the Skorokhod problem for ψ(•) then φ(• ∧ t 0 ), η(• ∧ t 0 ) solve the Skorokhod problem for ψ(• ∧ t 0 ).As a consequence, we find that sup (2.24) Several additional properties of φ = Γ(ψ) follow from (2.24).(i) If ψ(t) satisfies some growth estimate (linear for example), then so will φ(t), just with an additional factor of C Γ in the coefficients.(ii) If ψ(t) is right continuous with left limits, then so is φ(t).(iii) If ψ(t) is absolutely continuous, then so is φ(t).As noted, the Skorokhod problem can be posed for more general convex polygons G in place of G = R d + , subject to some technical properties [15].The use of more complicated G allows certain problems with finite buffer capacities to be modelled using (2.5).See [16] for instance.Although we are only considering G = R d + here, the point is that this approach can be generalized in that direction.

Weak convergence, relaxed controls, and fluid limits
Now that the issues of existence and representation have been addressed, we can consider convergence in the fluid limit n → ∞.This involves the notion of weak convergence of probability measures on a metric space at several levels.We appeal to Ethier and Kurtz [17] for the general theory.In brief, if (S,Ꮾ(S)) is a complete separable metric space with its Borel σ-algebra, let ᏼ(S) be the set of all probability measures on S. A sequence P n converges weakly in ᏼ(S), P n ⇒ P if for all bounded continuous Φ : This notion of convergence makes ᏼ(S) into another complete separable metric space.
A sequence P n of such measures is relatively compact if and only if it is tight: for every > 0 there is a compact K ⊆ S with P n (K) ≥ 1 − for all n.In particular, if P n is weakly convergent, then it is tight.Moreover, if S itself is compact, then every sequence is tight, and it follows that ᏼ(S) is also a compact metric space.
Our processes Y n (t) and X n (t), 0 ≤ t are right continuous processes with left limits, taking values in R d and G, respectively.The space(s) of such paths are typically denoted D. We will use the notations to denote the R d -valued and G-valued versions of this path space, respectively.The Skorokhod topology makes both of these complete, separable metric spaces.We will use ρ( G. Pang and M. V. Day 11 In other words, uniform convergence on compacts implies convergence in the Skorokhod topology. Weak convergence of X n or Y n is understood as weak convergence as above using the metric space D G or D R d , respectively.Thus, when we say that a queueing process X n (•) converges weakly to a fluid process x(•), X n (•) ⇒ x(•) (as we will in Theorem 3.4 below), we mean that their distributions converge weakly as probability measures on D G , in other words, for every bounded continuous Φ : D G → R. If these processes are obtained from the Skorokhod map applied to some free processes . By applying (2.23), we are led to whenever , where BV is the set of functions in R d of finite variation.Returning to (3.4), Φ • Γ is continuous so (3.4) follows from for all bounded continuous Ψ : Thus, to establish a weak limit for a sequence of X n with representations (2.5), corresponding to a sequence u n ω (•) of controls, it is enough to establish weak convergence of the free processes Y n .The decomposition (2.8), and the result that M n → 0 (Corollary 2.2) means the convergence boils down to that of the fluid components So the remaining ingredient is an appropriate topology on the space of controls.The above suggests that convergence of integrals of continuous functions again provides the right idea.This leads us naturally to the space of relaxed controls.An (individual) relaxed control is a measure ν defined on ([0,∞) × U,Ꮾ([0,∞) × U)) with the property that ν([0,T] × U) = T for all T. will denote the space of all such relaxed controls.A (deterministic) control function u(•) : [0,∞) → U (Borel measurable) determines a relaxed control ν ∈ according to for any measurable A ⊂ [0,∞) × U.The ν that arise in this way, from some deterministic u(t), will be called standard relaxed controls.Each (1/T)ν is a probability measure when restricted to [0,T] × U, and so can be considered with respect to the notion of weak convergence of such measures described above.By summing the associated metrics (×2 −N ) over T = N = 1,..., we obtain the usual topology of weak convergence on .(See Kushner and Dupuis [18] for a concise discussion and further references to the literature.)This means that a sequence converges ν n → ν in if and only if for each continuous f : [0,∞) × U → R with compact support, we have f dν n → f dν.Since ν({T} × U) = 0 ([0,T] × U is a a ν-continuity set in the terminology of [17]), this is equivalent to for each continuous f : [0,∞) × U → R and each 0 ≤ T < ∞.With this topology, and since our U is compact, is a compact metric space.Even though the standard controls do not account for all of , they are a dense subset.This fact is sometimes called the "chattering theorem." Theorem 3.1.The standard controls are dense in .
At this stage, we can collect a simple consequence of our discussion, which will be important for our fluid limit analysis.If x 0 ∈ G and ν ∈ , define the fluid process x x0,ν (•) by analogy with (2.5): (3.10) is jointly continuous with respect to x 0 ∈ G and ν ∈ .
Proof.Suppose x n 0 → x 0 in G and ν n → ν in .We want to show that x x n 0 ,ν n (•) → x x0,ν (•) in D G .By our discussion above, it suffices to show that y x n 0 ,ν n → y x0,ν uniformly on any [0, T].The convergence of ν n in the topology implies the convergence of y x n 0 ,ν n (t) to y x0,ν (t) for every t; see (3.9) and (3.10) .Since v(u) is bounded, the y x n 0 ,ν n are equicontinuous.This is enough to deduce uniform convergence of y x n 0 ,ν n (t) to y x0,ν (t) on [0,T].Next, consider an admissible stochastic control u ω (t).The relaxed representation (3.8) produces an -valued random variable (defined on (Ω,Ᏺ,P n,uω(•) )), which we will denote ν ω .As an -valued random variable, ν ω has a distribution Λ on .Since is a compact metric space, every sequence Λ n of probability measures on is tight and thus has a weakly convergent subsequence: Λ n ⇒ Λ.In other words, given any sequence ν n ω of stochastic relaxed controls (associated with a sequence of admissible stochastic controls u n ω ), there is a subsequence n and a probability measure Λ on , so that holds for all bounded continuous Φ : → R. The following lemma will be useful.
Λ weakly as probability measures on G × .This is well known; see Billingsley [19,Theorem 3.2].These observations lead us to the following basic fluid limit result, which will be the foundation of the convergence of the value functions in the next section.Theorem 3.4.Suppose that X n is the sequence of queueing processes corresponding to sequences x n 0 ∈ G of initial conditions and u n ω (•) of admissible stochastic controls.Let ν n ω be the stochastic relaxed controls determined by the u n ω (•) and Λ n their distributions in .Suppose x n 0 → x 0 in G and Λ n ⇒ Λ weakly in .Then, X n converges weakly in D G to the random process defined on (,Ꮾ(),Λ) by ν ∈ → x x0,ν according to (3.10).
Some clarification of notation is in order here.The process X n = Γ(Y n ) and its associated control process u n ω are defined on the probability space Ω of Theorem 2.3, and ω ∈ Ω denotes the generic "sample point."The associated probability measure on Ω is P n,u n ω .Thus, expressions such as E[Φ(Y n )] below and (4.3) of the next section are to be understood as expectations with respect to P n,u n ω of random variables defined on Ω. Likewise ν n ω is an -valued random variable, still defined on Ω with distribution determined by P n,u n ω .Although we might have written E P n,u n ω [•], we have followed the usual convention of using only E[•], considering it clear that the underlying probability space for Y n or X n must be what is intended.In the last line of the theorem the perspective changes, however.There we are viewing x x0,ν = Γ(y x0,ν ) as random processes with itself as the underlying probability space (not Ω) and ν ∈ as the generic "sample point."There no longer remains any dependence of x x0,ν or ν on ω ∈ Ω.If Λ is the probability measure on , we have used the notation E Λ [•] to emphasize this change in underlying probability space.
Proof.As explained above, it is enough to show weak convergence in D([0,∞)) of the free processes Y n to y x0,ν (with ν distributed according to Λ).Consider an arbitrary bounded continuous Φ defined on D R d .We need to show that (3.12) Our martingale representation of the free queueing process Y n can be written From Lemma 3.2, we know that Ψ : G × → R defined by Ψ(x 0 ,ν) = Φ(y x0,ν ) is bounded and continuous.It follows from Lemma 3.3 that δ x n 0 × Λ n ⇒ δ x0 × Λ and therefore In other words y x n 0 ,ν n ω ⇒ y x0,ν , where ν is distributed over according to Λ.Because of Corollary 2.2 and the domination ρ by the uniform norm as in (3.3), we know that M n ⇒ 0 in D([0,∞)).It follows from this that Y n → y x0,ν ; see [20,Lemma VI.3.31].

The control problem and continuity of J
So far we have just considered the processes themselves.As we turn our attention to the control problem, we need to make some hypotheses on the running cost function L. We assume that L : G × U → R is jointly continuous, and there exists a constant C L so that for all x, y ∈ G and all u ∈ U, L(x,u) − L(y,u) ≤ C L |x − y|. (4.1) In contrast to [7], no convexity, monotonicity, or non-negativity are needed.Notice that (4.1) makes {L(•, u) : u ∈ U} equicontinuous.Also, by fixing some reference y 0 ∈ G, the Lipschitz property of L implies a linear bound for the x-dependence of L, uniformly over u ∈ U: Reference [7] allows more general polynomial growth.That could be accommodated in our approach as well by extending Corollary 2.2 to higher order moments.We now state formally the two control problems under consideration.The discount rate γ > 0 is fixed throughout.First is the (fluid-scaled) stochastic control problem for scaling level n and initial position x n 0 ∈ G: minimize the discounted cost over admissible stochastic controls u ω (•).(As per the paragraph following Theorem 3.4, expectation is understood to be with respect to P n,uω .)The value function is Next is the fluid limit control problem.Here it is convenient to consider arbitrary relaxed controls rather than just standard controls.Recall the definitions (3.10).The problem is to minimize over all relaxed controls ν ∈ .The value function for the fluid limit control problem is Although we are minimizing over all relaxed controls, the infimum is the same if limited to standard controls.This follows since the standard controls are dense in and Lemma 4.1 below will show that J is continuous with respect to ν.

4.1.
Estimates.We will need some bounds to insure the finiteness of both J n (x n 0 ,u ω (•)) and J(x 0 ,ν).For a given stochastic control u ω (•), with relaxed representation ν ω , let G. Pang and M. V. Day 15 y x n 0 ,νω (t) denote the "fluid component" of Y n (t): (This is just the free fluid process of (3.10) using the randomized relaxed control ν ω .)By (2.8), where M n is the basic martingale of Lemma 2.1.Since U is compact, the velocity function (2.9) is bounded: Therefore, y x n 0 ,νω grows at most linearly: By (2.24) it follows that and therefore, for a new constant C, independent of control and initial position.Using Corollary 2.2, we deduce the bound

.14)
This implies that J n (x n 0 ,u ω (•)) is finite.Moreover, it follows from this bound that for any bounded set B ⊆ G and any > 0, there is a T < ∞ so that for all x n 0 ∈ B and all controls.This uniform approximation will be useful below.The same bounds apply to the controlled fluid processes (3.10): holds as above, and without the M * n term, we are led to holding for all u ∈ U.The finiteness of J(x 0 ,ν) and analogue of (4.15) follow likewise.
Proof.Suppose x n 0 → x 0 in G and ν n → ν in .Due to (4.15) for J, continuity of J(•,•) will follow if we show that for any T < ∞, but this follows from the equicontinuity of L(•,u) with respect to u and the convergence of x x n 0 ,ν n (t) → x x0,ν (t).Here is our key result on convergence of the costs.Theorem 4.2.Suppose that X n is the sequence of queueing processes corresponding to sequences x n 0 ∈ G of initial conditions and u n ω (•) of admissible stochastic controls.Let ν n ω be the stochastic relaxed controls determined by the u n ω (•) and Λ n their distributions in .Suppose x n 0 → x 0 in G and Λ n ⇒ Λ weakly in .Then,

.21)
Proof.First, by Lemmas 3.3 and 4.1, we know that So, to finish, we need to show that The bound provided by Corollary 2.2 makes the rest of the proof simple.

Convergence of values and asymptotic optimality
Now, we are ready to prove our main result.
Theorem 5.1.If x n 0 → x 0 is a convergent sequence of initial points in G, then V n (x n 0 ) −→ V x 0 . (5.1) Proof.The proof is in two parts.We first show that and then its counterpart (5.5).For each n, select a stochastic control u n ω (•) which is approximately optimal for V n (x n 0 ): V n (x n 0 ) ≤ J n (x n 0 ,u n ω (•)) ≤ (1/n) + V n (x n 0 ).By passing to a subsequence n , we can assume that liminf n→∞ V n x n 0 = lim n→∞ J n (x n 0 ,u n ω •) . (5.3) Again let ν n ω denote the stochastic relaxed controls determined by the u n ω (•) and Λ n their distributions on .There is a weakly convergent subsubsequence (which we will also index using n ): Λ n ⇒ Λ in ᏼ().Theorem 4.2 says that J n x n 0 ,u n ω (•) −→ E Λ J x 0 ,ν . (5.4) Clearly, V (x 0 ) ≤ E Λ [J(x 0 ,ν)].Thus, (5.2) follows.
The second half of the proof is to show that limsup n→∞ V n x n 0 ≤ V x 0 .
(5.5) Since, by Lemma 3.2, J(x 0 ,ν) is a continuous function of ν ∈ and the standard controls are dense in (Theorem 3.1), for a prescribed > 0, we can select an (individual) standard control ν with J(x 0 ,ν) nearly optimal: Since ν is a standard control, it comes from a (deterministic) measurable u(t) ∈ U.As a deterministic process this is progressive, hence admissible as a stochastic control.Let Y n be the free queueing process associated with this control u(t), initial position x n 0 , fluid scaling level n, and X n = Γ(Y n ) the associated queueing process.Theorem 4.2 implies that J n x n 0 ,u(•) −→ J x 0 ,ν .
We make the usual observation that since we have allowed x n 0 → x 0 in the theorem, instead of a fixed x n 0 = x 0 , we can conclude uniform convergence.The argument is elementary and omitted.
As a capstone, we have the following result which characterizes the asymptotically optimal policies for J n .(5.9) Proof.Theorem 5.1 implies the equivalence of (a) and (b).For any weakly convergent subsequence Λ n ⇒ Λ, we know from Theorem 5.1 that J n x 0 ,u n ω (•) −→ E Λ J x 0 ,ν .
(5.10) Thus, (a) is equivalent to saying E Λ J x 0 ,ν = V x 0 (5.11) for all weak limits Λ; but this is equivalent to (c).
5) and (3.10)), then it is sufficient to prove weak convergence of the free processes: Y n (•) ⇒ y(•).This is because the Skorokhod map is itself continuous with respect to the Skorokhod topology.In brief, the reason is as follows.The Skorokhod metric ρ(ψ 1 (•),ψ 2 (•)) is obtained by applying a monotone, continuous time shift s = λ(t) to one of the two functions, and then looking at the uniform norm of ψ 1 (•) − ψ 2 • λ(•).Such monotone time shifts pass directly through the Skorokhod problem: if .24) G. Pang and M. V.Day 17According to the estimate (4.15) of the preceding section, it is enough to prove this with the integral truncated toX n (t) − x x n 0 ,ν n ω (t) ≤ C Γ M * n (T).(4.25)Therefore, for 0 ≤ t ≤ T, we haveL X n (t),u n ω (t) − L x x n 0 ,ν n ω (t),u n ω (t) ≤ C L C Γ M *n (T).(4.26)