Optimal Control for Bufferbloat Queue Management Using Indirect Method with Parametric Optimization

. Because memory buffers become larger and cheaper, they have been put into network devices to reduce the number of loss packets and improve network performance. However, the consequences of large buffers are long queues at network bottlenecks and throughput saturation, which has been recently noticed in research community as bufferbloat phenomenon. To address such issues,inthisarticle,wedesignaforward-backwardoptimalcontrolqueuealgorithmbasedonanindirectapproachwithparametric optimization.Thecostfunctionwhichwewanttominimizerepresentsatrade-offbetweenqueuelengthandpacketlossrate performance.Throughtheintegrationofanindirectapproachwithparametricoptimization,ourproposalhasadvantagesof scalabilityandaccuracycomparedtodirectapproaches,whilestillmaintaininggoodthroughputandshorterqueuelengththan severalexistingqueuemanagementalgorithms.Allnumericalanalysis,simulationinns-2,andexperimentresultsareprovidedto solidifytheefficiencyofourproposal.Indetailedcomparisonstootherconventionalalgorithms,theproposedprocedurecanrun muchfasterthandirectcollocationmethodswhilemaintainingadesiredshortqueue( ≈ 40 packets in simulation and 80 (ms) in experiment test).


Introduction
Nowadays, modern computer networks are incredibly complex and we rely on them to transport huge quantities of data across the globe in seconds.Although this works well, there are some foreseen issues that need to be tackled.As bandwidth-heavy applications such as peer-to-peer networks and websites relying on user-generated content have become more prevalent, especially the relatively slow residential broadband links have been used at full capacity, and interruptions in connectivity have become more common [1].Recent research has shown that the culprit is the buffers built into network equipment [2].Accordingly, bufferbloat term has been used to describe related issues whenever these buffers misbehave to produce unnecessary latency [3].
In order to efficiently manage queues which are generated due to the bufferbloat phenomenon, active queue management (AQM) algorithms have been recommended to use in network equipment [4].Most of the existing approaches to AQM design exploit feedback control theory with the linearized TCP core model that was proposed by Hollot et al. [5].The well-known AQMs that monitor the average queue size and drops (or marks) packets based on statistical probabilities are Random Early Detection (RED) [6] or Random Early Marking (REM) [7].If the buffer is empty, all incoming packets are accepted.When the buffer is full, the probability has reached 1 and all incoming packets are dropped.When queue is growing, the probability grows according to a piecewise linear function and RED (or REM) drops (or marks) packets using the updated probability.The main drawback of RED or REM is that it is sensitive to network parameter changes and requires careful tuning of its parameters in order to provide optimal performance in any scenarios.Recently, in [8], a Proportional Integral Enhanced (PIE) controller as a lightweight AQM is proposed, without the need of per-packet extra processing.Such PItype controllers are known to provide queue control with zero offset (the mean queue length converges to the target value) but are consequently less stable and slower reacting.There have been several other control-theoretic solutions based on feedback fluid-flow model proposed in [9][10][11][12] to improve stable and robust control mechanisms, but none of them tackle the issue of system optimality in terms of minimizing both queue length and packet dropping rate performance and searching for an optimal control trajectory.
To address such issues, a promising design direction is to reformulate the network queuing problem as an optimal control queue (OCQ) problem [13], where the main state variable is queue length and the control variable is the actual input rate to form the queue [14] or packet dropping rate.Then one of the approaches to solve OCQ is that we can priorly discretize the governing ordinary differential equations (ODEs) and the integral terms in the cost function or constraint functions and thereby replace the infinite dimensional optimal control problem with a large nonlinear optimization problem (NOP).This is known as the direct method in the literature for solving OCQ.This approach is typically easier to use, especially for OCQ with a state equality or inequality constraints.The main difference among direct approaches is how to handle the constraints corresponding to the system dynamics.The three most common direct approaches are direct single shooting, direct multiple shooting, and direct collocation.Direct methods have been used with references in [15][16][17][18].
Alternatively, one can first form the optimality conditions, using the calculus of variations and Pontryagin minimum principle, and then solve the resulting boundary value problem.This is known as the indirect method for solving OCQ.The references present just a small sample of the work that discusses or applies indirect methods for the solution of optimal control problems [15,16,18,19].In rare cases, the solution can be obtained in closed form from the optimality conditions, but, in general, approximation methods are used to solve the problem numerically.The optimality conditions of these problems generally take the form of differential algebraic equations (DAEs) with boundary conditions (BCs).The approximate solution to the OCQ can be obtained by using a boundary value problem (BVP) solver.Perhaps the most popular methods are multiple shooting and collocation.More recently, a combination of direct and indirect methods was proposed leading to hybrid methods [16].
In this paper, we firstly derive methods to solve OCQ using both direct and indirect approaches.We show how to apply the direct collocation approach to solve OCQ problem which can be solved in popular optimization solvers such as JModelica [20] and GAlib [21].Indirect method with forward-backward for optimal control queue algorithm (FB-OCQ) is designed as an alternative to tackle it as well.Our key difference from existing works is that we provide a novel method to update the control step  by solving a parametric optimization subproblem.This method is scalable which means that we can expand for larger problems with more variables as well.The numerical results for both direct and indirect approaches are discussed in Section 5.1 to emphasize our choice of integrating an indirect method with parametric optimization for active queue management design, which is demonstrated more efficiently (i.e., faster reaction and more stable) than direct methods.Finally, we evaluate the proposed

Problem Formulation
We consider an optimal control model of queue management problem, named (OCQ), in [14] (Figure 1).The main idea is to minimize the cost function which implies a trade-off between queue length and dropping rate, subject to a dynamic constraint of queue length along time  0 to   as follows:

Numerical Methods for OCQ
3.1.Direct Method: Collocation Method.In this section, we present how to apply the direct collocation [18] for OCQ.In this method, we discretize the time interval The controls are chosen as piecewise linear interpolating functions between (  ) and ( +1 ) for   ≤  ≤  +1 as follows: The value of the control variables at the center  +1/2 is given by The piecewise linear interpolation is used to prepare for the possibility of discontinuous solutions in control.Similarly, we can derive the approximate of the total number of the packets  app () and ( +1/2 ) as above.The state variable () is approximated by a continuously differentiable and piecewise Hermite-Simpson cubic polynomial between (  ) and ( +1 ) on the interval   ≤  ≤  +1 of length ℎ  : where ( The value of the state variables at the center point  +1/2 of the cubic approximation is and the derivative is In addition, the chosen interpolating polynomial for the state and control variables must satisfy the midpoint conditions for the differential equations as follows: Equations (OCQ) can now be defined as a discretized problem as follows: subject to  ( app () ,  app () ,  app () , ) − ẋ app = 0, where  app ,  app , and  app are the approximations of the state, the control, and the total number of packets, constituting  in (9).This above discretization problem ( 9)-( 10) can be solved using the following: (i) JModelica, which is a package for simulation and optimization of Modelica models (for more details see [20]); (ii) GAlib, which is C++ library of genetic algorithm (for more details see [21]).

Indirect Method:
Forward-Backward Sweeping.We solve OCQ by using indirect method approach: (i) forming optimality conditions; (ii) solving BVP by First-Order Sweeping algorithm.
Let () be the adjoint variable.At time , let  * () denote the optimum controls and let  * () and  * () denote the state and adjoint evaluated at the optimum.Using Pontryagin minimum principle we get the following equations.
If  * () is optimal in (OCQ), then it must satisfy the minimum condition: Furthermore, we assume the following: (H1) the Hamiltonian is strictly convex with respect to control variable ; Note that () is determined in a unique way for each  since  is convex and  is convex.Now we consider the problem of minimizing the Hamiltonian with respect to : Usually in the literature,  is found explicitly as a function  = (, , ) and, after substituting it into the system, the problem reduces to the boundary value problem.

Theorem 1. Assume that conditions (H1)-(H3) hold and
problem has an optimal solution ( * ,  * ).Then for a given  there exists a finite discretization and an approximate solution ũ(),  ∈ [ 0 ,   ], such that Proof.Let  * be an optimal solution of problem.Then ( * ,  * ) satisfies the conditions: where and ( * ,  * ) satisfies the minimum principle: (a) First-Order Sweeping Method.We linearize the Hamiltonian around a reference solution () and obtain the following equation for variation (): With the strong Legendre-Clebsch condition, one can approximate the above equation as where which proves the assertion.
Remark 3. Parametric optimization also can be applied in finding nominal optimal control given in [22].It is easy to see that, at each iteration , the Hamiltonian function is a scalar function of  ∈  ⊂   and  ∈  = [ 0 ,   ]; that is, The latter states that û  () must be a minimizer of the following problem: which is a problem of parametric optimization as formulated in various papers from [23], where the independent variable  is now considered as unknown parameter  ∈  = [ 0 ,   ].We can also consider a case when the set of admissible controls is time-varying; that is,  = (),  ∈  = [ 0 ,   ].In this case, a general theory of parametric optimization is also applicable for finding the nominal optimal controls.

Numerical Results.
In this section, we provide some iteration results for both direct and indirect approaches as follows.
Indirect Method: FB-OCQ.The ODE was solved on equidistant discretization with 1000 discretization points.The optimal control and optimal state are depicted in Figure 3.We realize, by looking at the final control, that there exist some points where the set of active constraints changed due to the singularity; that is, the optimal control is of the bang-bang type with the possibility of a singular arc.To the meaning of such singular arc for queue management, it presents the sudden changes of input rate from 60 (packets/sec) to 35 (packets/sec) which accordingly result in the changes of buffer load (or queue length).In the  history, in Figure 3, the parameter  changed at the inner iterations because we do not have a descent direction, so we must divide  by  ∈ (0, 1) until we get a reduction in the cost function J = 6.81280316.
The processing time of central processing unit (CPU) for this algorithm is 0.04 s and the norm of the gradient of Direct Collocation Method: JModelica and GAlib Solvers.With  = 1000 and the number of interpolation points being 3, we tested the discretized problem ( 9)-( 10) by JModelica and GAlib solvers (see [20,21]) as follows.
(i) JModelica Solver.The cost function is reduced to J = 6.8678362 and the CPU processing time is 0.52 s.The optimal control and optimal state trajectories are obtained during JModelica running and illustrated in Figure 4.An advantage of this approach is that we do not need to derive the adjoint equations.
(ii) GAlib Solver.For this problem, we use the number of generations  gen = 2500, and the population size is 200.The optimal control and optimal state are depicted in Figure 5 obtained during the run of the GAlib.The cost function is reduced to J = 7.06744 during the run time of the program.The CPU processing time is 1.290 s.Although the genetic  algorithm has an advantage that it needs no derivatives or Hessian's information, the control functions produced by it are useless and the convergence is very slow in comparison to Algorithm 1 (FB-OCQ) and the one using JModelica solver.Table 1 summarizes and compares the three methods that we develop numerically.We conclude that Algorithm 1 is much faster than the other solvers and gives the best cost function.By using genetic library GAlib, the cost function does not reach the local solution as it claims in finding the global solution.The limitation of the indirect methods (FB-OCQ) is that one should derive the adjoint equations which are not easy to derive in some applications.

Simulation Results.
In this section, the performance of the obtained algorithm (FB-OCQ) is evaluated by comparing it with some popular AQMs including RED, REM, and PI.The credibility of results is confirmed using ns-2 simulator [24].We investigate a network topology with 40 sources, an intermediate router, and one destination (Figure 6).All sources simultaneously send packets to the destination through the router.Hence, a large queue is built up at the router or bottleneck point.Maximum buffer size of the router is 100 packets.All of the compared RED, REM, and PI algorithms are configured to obtain the desired queue length at 40 packets.Simulation lasts for 60 seconds and is repeated using a built-in random generator in ns-2 to obtain more credible results.Figure 7 shows dropping probability of FB-OCQ and RED algorithms.With the proposed square-root drop function, FB-OCQ drops queuing packets more aggressively than RED although dropping frequency is nearly the same.Maximum dropping ratio is 0.045 for FB-OCQ and 0.039 for RED.In fact, when the algorithm drops more packets, the queue stability will be increased but we will have to sacrifice the system throughput performance.We can see this trade-off in next results.
Figure 8 presents average queue length values measured at the bottleneck link from router to destination for different algorithms.Due to aggressive dropping, FB-OCQ maintains the shortest queue at 40 packets which is also the desired value.Only REM can obtain the same value but in a longer time, ≈50 seconds.PI, in fact, can achieve the same performance only if its parameters are well configured and be dynamically changed to different network scenarios.In Figure 8, we can see that PI performs not so well, even we set the desired queue length variable at 40 packets and exploit default parameters in ns-2 for PI controller.
Finally, we investigate throughput performance of proposed FB-OCQ algorithm.It is confirmed from Figure 9 that there exists a trade-off in the relationship between throughput and queue length/dropping probability.PI and REM achieve nearly the same throughput at 0.06 (Mbps) while FB-OCQ and RED obtain the better throughput performance at 0.08 (Mbps) value.In Figure 8, although REM can achieve the desired queue length 40 packets in this scenario, REM still maintains a large queue during simulation time from 0 (sec) up to 25 (sec).That reason leads to throughput performance of REM being lower than RED and FB-OCQ.Our proposed FB-OCQ drops aggressively the packets so that its throughput value (0.075904 Mbps) is slightly less than RED (0.082146 Mbps).

Experiment Results.
We examine the effectiveness of our state-of-the-art optimal control queue method in the Linux kernel (e.g., Ubuntu 16.04).We conduct experiment from a desktop computer through the Internet gateway to an outside server (Figure 10).To manage working queuing disciplines (qdiscs), we use the scheduler qdisc in the Linux kernel.The chosen server is a dedicated bufferbloat server, which is able to stand very high congestion due to many data flows at the same time.We exploit the Flent: FLExible Network Tester [25] and Realtime Response Under Load (RRUL) test [26] to evaluate our proposal.RRUL test puts a network under worst case conditions, reliably saturates the network link, and thus recreates bufferbloat phenomenon for queue algorithm testing.Simulation time duration is 60 seconds.
We compare latency under load test with queue being handled by different AQM schemes (DropTail, RED, and FB-OCQ) in turn.DropTail queuing method is by far the simplest approach to network router queue management.The router accepts and forwards all the packets that arrive as long as its buffer space is available for the next incoming packets.If a packet arrives and the queue is currently full, the incoming packet will be dropped.The sender then detects the packet lost event and shrinks its sending window.While it is the most widely used due to simplicity and relatively high efficiency, DropTail has some weakness such as the bad fairness sharing among TCP connections, and throughput and bottleneck link efficiency suffer severe degradation if congestion is getting worse.
RED [5,6] was presented with the objective of minimizing packet loss and queuing delay.Moreover, it can compensate the weakness of DropTail by avoiding global synchronization of TCP sources so that it improves fairness.To achieve these goals, RED utilizes two thresholds, min th and max th , and an exponentially weighted moving average (EWMA) formula to estimate average queue length [27].When the average queue length exceeds a predefined threshold, the link is implied to be in congested state and drop action is taken.A temporary increase in the queue length notifies the transient congestion, while an increase in the average queue length reflects long-lived congestion.Based on such information, RED router sends randomized feedback signals to the senders to make decision of decreasing their congestion windows.RED has good fairness among connections because of the feedback randomized mechanism [28].
Figure 11 presents latency ping results under RRUL test suite.Ping is a networking utility and operates by sending Internet Control Message Protocol (ICMP) echo request packets to the target server and waits for an ICMP echo reply.The program measures the round-trip time from transmission to reception, reporting errors and packet loss.Our proposed algorithm FB-OCQ achieves the lowest packet latency compared with the other two algorithms inside Linux () ,  () , ) subject to ẋ =  ( () ,  () , ) , 0 ≤  () ≤  () , (OCQ)where (, , ) = () + ( − ()) is cost function; ((), (), ) = () − /( + );  is weight on dropping rate;   is final time;  is service rate (bandwidth capacity); and  is parameter for different types of queuing model; for example, when  = 1, we obtain an M/M/1 queue.

Figure 10 :
Figure 10: Experiment test setup, from the test client to the test server.