Power-Aware Routing and Network Design with Bundled Links : Solutions and Analysis

The paper deeply analyzes a novel network-wide power management problem, called Power-Aware Routing and Network Design with Bundled Links (PARND-BL), which is able to take into account both the relationship between the power consumption and the traffic throughput of the nodes and to power off both the chassis and even the single Physical Interface Card (PIC) composing each link. The solutions of the PARND-BL model have been analyzed by taking into account different aspects associated with the actual applicability in real network scenarios: (i) the time for obtaining the solution, (ii) the deployed network topology and the resulting topology provided by the solution, (iii) the power behavior of the network elements, (iv) the traffic load, (v) the QoS requirement, and (vi) the number of paths to route each traffic demand. Among the most interesting and novel results, our analysis shows that the strategy of minimizing the number of powered-on network elements through the traffic consolidation does not always produce power savings, and the solution of this kind of problems, in some cases, can lead to spliting a single traffic demand into a high number of paths.


Introduction
Energy saving is one of the most important challenges of the twenty-first century for environmental and economical reasons.From an environmental perspective, due to the lack of diffusion and efficiency of renewable energy, the reduction of power consumption is important because the production of energy is directly related to the emission of carbon dioxide (CO 2 ), the main reason of global warming.From an economical point of view, the incremental growth of the energy price and of the power demand of emerging countries makes energy saving a key issue to contain the increase of energy cost in various sectors (industrial, commercial, and residential).Furthermore, we are witnessing an explosive growth in the use of Information and Communication Technologies (ICTs) equipment that is rapidly becoming a major consumer of energy.Indeed, according to the SMART 2020 study [1], CO 2 emissions from ICT are increasing at a rate of 6% per year, and with such a growth rate they could represent 12% of worldwide emissions by 2020.
In this scenario, the research community is studying a set of approaches for improving the energy efficiency of the future Internet.Detailed and up-to-date surveys on the different strategies for energy-efficient networking are presented in [2,3].These studies are based on the consideration that current networks are widely overprovisioned and network devices are not energy aware; that is, they always consume the same energy irrespective of their utilization.The relevance of these problems is further emphasized by the works of the Energy Management (EMAN) WG of IETF.The EMAN WG started with the basic objective of operating communication networks and other devices with a minimal amount of energy while still providing sufficient performance to meet service level objectives.The EMAN is defining the requirements for standard specifications in energy management, both monitoring functions and control functions [4].Network entities are subjected to energy management.Examples of these entities are a whole device or one component of a device, which is subject to individual energy monitoring, control, or both.
The research activities can be classified into two main areas: local strategy and network-wide strategy.In the first case, power saving functions are executed locally by an entity.The strategies consist in monitoring the entity usage and dynamically adapting its power consumption according to the required performance.As an example, the entity may switch to a sleep state when it is not in use or out of scheduled business hours [5][6][7].Local management has the main advantage that the power savings are obtained by means of measured data based on local observations, without the interaction with other network entities.The corresponding solutions lead to the design of power-aware network elements.The network-wide policies aim to achieve further energy savings by means of strategies that exploit the power consumption features of power-aware network elements.They require the information on power consumption and network load acquired through monitoring and a networkwide energy management system.
Starting from the seminal work on power-efficient network devices presented in [8], many efforts have been recently devoted to define strategies for reducing the power consumption of the whole network infrastructure.As an example, in [9] the authors present ElasticTree, a method for reducing the energy consumption in data center networks.In [10], the authors compare three different approaches to power off the line cards by reconfiguring the routing at the IP and/or WDM layer.Another topic of network-wide power management addresses is the protocol issues in Internet Service Provider networks.In [11], the authors present an energy-aware traffic engineering technique that reduces the energy consumption by considering sleeping of links and nodes as well as link rate adaptation.In [12], the authors propose a traffic engineering mechanism that minimizes the number of active line cards (or their ports) and explore the implementation issues by assuming networks that run both OSPF-like protocol and MPLS.Finally, in [13] the authors propose a distributed algorithm to dynamically adapt the working state of links to the current traffic load.Their solution belongs to an OSPF-like protocol.
The contribution of this paper is focused on the networkwide power optimization in the backbone network.In particular, the paper investigates a very general network-wide power management (NWPM) problem, called Power Aware Routing and Network Design with Bundled Links (PARND-BL) [14], which is able to take into account both the constant and the variable terms of the power consumption of network equipment.In particular, PARND-BL is able to consider both the energy behavior of the route processor, that is, the relationship between the power consumption and the traffic throughput of the nodes, and to power off both the chassis and even the single PIC composing each link.
In the case of large network topologies, PARND-BL cannot be solved to optimality in an acceptable time, that is, few hours (see [14]).Therefore, we propose a heuristic named "Heuristic for PARND-BL" (HPARND-BL), whose solutions are capable of guaranteeing interesting power savings results.The well-known heuristic Fast Greedy Heuristic (FGH) [15] has been considered as the benchmark heuristic.
The performance analysis takes into account different actual network scenarios and diverse metrics, such as the number of powered-on network elements, the CPU time for solving the problem, and the number of paths per flow of the solution.To the best of our knowledge, none of the previous works presenting models for the minimization of the network power consumption has evaluated the actual applicability of the obtained solution in terms of number of paths used to route each single traffic demand.The relevance of this analysis consists in the fact that the solutions of the power reduction problems could entail a huge number of paths per traffic demand.The management of a high number of paths per traffic demand can be so expensive in terms of signaling to reduce the appeal deriving from the power savings.Finally, the extensive simulation analysis gives insights into the performance of the considered NWPM approaches under diverse network parameters, such as the power characterization of network devices, the traffic load, the network topology, and the QoS requirements.
The paper is structured as follows.Section 2 presents the NWPM in more detail.Section 3 reviews the main literature on the subject.Sections 4 and 5 introduce the power-aware problem formulation and the related proposed heuristic, respectively, that are the main focus of the paper.Section 6 addresses some implementation issues.Section 7 reports the network scenarios considered for the simulations.Sections 8 and 9 present the simulation settings and discuss the computational results.Finally, the conclusions are drawn in Section 10.

Network-Wide Power Management (NWPM)
Generally speaking, NWPM consists in finding the design and the routing strategies that minimize the overall power consumption of a network by taking into account the power behavior of the network elements and the traffic demand.Therefore, three critical issues can be devised in NWPM: the time horizon of the traffic demands, the power behavior of the network devices, and the protocol architecture.

Time Horizon of the Traffic Demands.
Regarding this issue, we can observe that in actual deployments the traffic matrix presents significant changes mostly between two periods: peak and off-peak.Just relatively small variations are usually registered within each period (see Figure 1).Therefore, the time scale of the NWPM approaches could safely be based on peak and off-peak traffic periods only.In general, the choice of a smaller time scale implies the study of the trade-off between responsiveness to traffic variations and time to perform the algorithms and apply the obtained solution.In a deployment scenario, the network power consumption strategies could be recalculated following the slow and daily variation of traffic, that is, referring to time scale of 30 min.However, this approach leads to diverse Figure 1: Byte volume on a domestic link in each direction (north/south)-source [16].
reconfigurations per day.On the contrary, the choice of a time scale of 6/12 hours (with the coarse classification of peak and off-peak traffic periods) mitigates the eventual extra energy cost and additional latencies that devices may introduce when changing power state (e.g., energy spent to save/recover router state on/from disk, etc.).In conclusion, we assume that the duration of the two periods is day and night, respectively, and that the maximum acceptable time for computing an NWPM solution can be order of 6/12 hours.

Power Behavior of the Network Devices.
The energy characterization of the network devices determines what can be exploited to minimize the overall network consumption.In this work, we have used a general power consumption model of a router, which is composed of three main components [17]: (i) chassis; (ii) Physical Interface Cards (PICs); (iii) route processor.
The chassis can be powered off (i.e., it works in a lowpower mode); hence, the power consumption can be assumed to be constant if the chassis is powered on and zero otherwise.
To model the power characteristics of the PICs, we have to dissect the communication process in more detail.The energy for transferring a bit from a node  to V, can be ascribed to various components, such as the power consumed by the transmitting PIC in  and the receiving PIC in V. To make the treatment easier (and without loss of generality), when dealing with the traffic sent from  to V we will associate all the power consumption and the related capacity with the PIC at node .Obviously, in the reverse direction, we will associate them with the PIC at node V.It is worth noting that the network operators tend to deploy similar devices in their core networks.Thus, we can assume that the power consumption and the capacity of the PIC used to transmit along (, V) are equal to the ones of the PIC transmitting along (V, ).Similar to the chassis, the PICs can be powered off.Therefore, there is a constant, nonzero power consumption when the PIC is powered on, and a zero power consumption when the PIC is powered off.
Another aspect that will be included in the analysis is the fact that in modern core networks pairs of routers are typically connected by multiple PICs that form one logical bundled link [18].This technique is called link aggregation and is standardized by 802.1AX [19].Link aggregation is widely diffused because it allows to easily upgrade the link capacity by adding new PICs and reach link capacities bigger than those available even by using the fastest available technology.For example, a 40 Gb/s bundled link may be set up with four OC-192 PICs having a 10 Gb/s capacity each.Therefore, in our model, we assumed that links are composed of multiple PICs and that each PIC of the bundled links can be independently powered off.
In the route processor, the power consumption generally depends on the traffic load of the router in a nonlinear way.In [20], the authors present several possible behaviors: (i) linear-switch architectures like Batcher, Crossbar, and Fully Connected; (ii) logarithmic-equipment implementing hibernation techniques; (iii) cubic-equipment that uses energy saving methods such as dynamic voltage and dynamic frequency scaling (DVS-DFS); (iv) on-off -almost the totality of current network equipment that consumes always the maximum power in every load condition.

Protocol Architecture.
In modern broadband communication networks, in order to apply the calculated traffic engineering solution, we should refer to protocol architectures such as MPLS (Multiprotocol Label Switching) [21].
In this architecture, the paths, denoted as Label Switched Paths (LSP), are created and managed by means of signalling protocols (such as RSVP-TE, Resource Reservation Protocol for Traffic Engineering [22]), and they support the traffic demand for a given pair of nodes.A similar strategy can be implemented by means of emerging architecture paradigms such as the Software-Defined Networking (SDN).
For this reason, we propose a centralized architecture where a Network Control Unit (NCU) provides the LPSs (see Figure 2).Based on topology and traffic load information, the NCU selects the best routing and design strategy to reduce the overall network power consumption.

Related Work
We focus our attention on the network-wide power saving strategies, which are able to exploit the energetic characterization of the power-aware network devices.As presented in [14], the known strategies can be classified by taking into account the following four basic problems.
(i) Power-Aware Routing (PAR).Given the traffic demands associated with the nodes, PAR consists in determining the traffic routing strategy that minimizes the overall power consumption of the network by taking into account only the power absorbed by the route processor.
(ii) Power-Aware Network Design (PAND).Given the traffic demands associated with the nodes, PAND consists in determining the traffic routing strategy that minimizes the overall power consumption of the network just by powering off elements of the network.
Given the traffic demands associated with the nodes, PARND consists in determining the traffic routing strategy that minimizes the overall power consumption of the network by jointly considering the power absorbed by the route processor and the possibility of powering off the entire links and/or nodes.
(iv) Power-Aware Routing and Network Design with Bundled Links (PARND-BL).This problem extends PARND by exploiting also the possibility to power off single PIC of bundled links.
A first set of activities has been aimed at defining models for the PAND problem.In particular, [23] presents at first some measurements of power consumption of networking devices and then evaluates the total network consumption of a certain topology given the power footprint of each element.A standard Capacitated Multicommodity Minimum Cost Flow formulation is proposed in [24] for finding the minimum set of network elements to be powered on in order to guarantee the required service.However, the complexity of the model grows very quickly with the number of devices, making it very expensive to solve even for small networks.In [25], the authors present the analysis of the strategies to concentrate network traffic on a minimal subset of network resources in order to power off nodes and links while still guaranteeing full connectivity and fulfilling the maximum link utilization constraints.In addition, by taking into account strict constraints on the computational time, [25] provides efficient heuristics to find an approximated solution in an acceptable time also for large network scenarios.A variation of this class of problems is presented in [15], which describes a simple heuristic, called Fast Greedy Heuristic (FGH), to power off links when bundles of physical cables are present.In [26], the authors present new heuristics for this problem; the main characteristic of the new approach is the ability of routing each traffic demand by using only a single path.Since this paper addresses the more general PARND-BL problem, that allows multiple paths for each traffic demand, we have selected the FGH heuristic as the benchmark for evaluating our approach.
A second set of works has been focused on the definition and the solution of the PAR problem.In [20], the authors presented a study on the reduction of the overall power consumption that can be achieved when the energy profiles of the network devices are taken into account during routing and traffic-engineering operations.They consider rough linear approximations of the power consumption of the devices, and then use a linear programming solver (i.e., CPLEX [27]).However, since the power consumption of energyaware network devices is generally a nonlinear function of their workload, PAR is a non-linear multicommodity flow problem.Given this feature, the PAR problem should be modelled and solved by means of non-linear solvers such as Ipopt (Interior Point Optimizer) [28], a software package for large-scale non-linear optimization that implements an interior-point line-search filter method (as performed in this study, see Section 8).In [29], the authors have proposed a heuristic based on Dijkstra's algorithm and have deeply investigated the impact of the traffic load and the topology on the network performance.
A third set of works has been devoted to jointly consider the power aware routing and the network design problems (i.e., PARND).The problem is discussed in [30,31] under different technology assumptions.Different from our work, the authors of [30] assume that nodes and even links have a linear energy behaviour; instead the authors of [31] assume that nodes and even links have a staircase energy behaviour.Also, in [30], the authors take into account one network topology only, thus neglecting the effect that different network scenarios may have on the quality of the solution.
A heuristic to solve the PARND problem is proposed in [32], where the authors consider only the energy consumption of the links, which is again assumed to be linearly proportional to the traffic that flows through.The first preliminary ideas for modelling the non-linear behaviour of the routers, together with a heuristic algorithm, are suggested in [33], based on the indications in [17].Specifically, the proposed energy model is built from the sum of two constant addends associated with the power consumption of the chassis and the PICs and one variable (non-linear) element associated with the route processor.
On the other hand, to the best of our knowledge, none has deeply analyzed both the performance and the actual applicability of a general network-wide problem like PARND-BL, proposed in [14].

Problem Statement
Let us introduce the parameters and the notation used in the paper.The starting point of the analysis is a network modelled as a directed graph  = (, ), where  denotes the set of the vertices and  is the set of the edges, which model the network nodes and links, respectively.
The following parameters are assumed to be given in order to characterize the power consumption of the network elements: V is the power consumption of a PIC that transmits traffic from node  to node V (i.e., a PIC of link (, V)); (ii)   V is the power consumption concerning the chassis of the node V; (iii)  RP V,(V) is the power consumption concerning the route processing of the node V at the traffic throughput (V); as justified before, hereafter we shall assume that  RP V,(V) is a non-linear function.
Since each logical link is generally composed of a set of PICs, the overall power consumption for the traffic transmission on a link (, V) is equal to the number of powered-on PICs, in node  connected to V, multiplied by  PIC V .Hence, the maximum power consumption associated with a directional link (, V) is given by  V ⋅  PIC V , where  V is the number of deployed PICs forming the bundled link that connects  to V.
Concerning the traffic demand and the capacity of nodes and links, we define the following parameters: (i)  is the set of the origin-destination pairs of the traffic matrix; (ii)   is the traffic demand between the source node  and the destination ; V is the capacity of each PIC which composes the link (, V).
Three sets of variables are defined: V is the amount of   flowing through the link (, V); (ii)  V is the number of powered-on PICs composing the link (, V); (iii)  V ∈ {0, 1} is set to 1 when the node V is powered on and to 0 otherwise.
The traffic throughput of node V can then be defined as the total flows entering V plus the traffic originated from V, according to the following formula: For reducing the overall power consumption of a network with bundled links, we define the PARND-BL problem.This minimization problem considers both the possibility of routing the traffic demands by taking into account the relation between the power consumption of the route processor and the traffic load, and of powering off the nodes and the links, as well as even single PIC of bundled links.
The above introduced PARND-BL problem leads to a Mixed Integer Nonlinear Programming (MINLP) design and routing model, which can be formulated as follows: subject to the constraints Equations ( 3) are the classical flow conservation constraints; instead (4) provide the definition of the flow variables.Equations ( 5) and ( 6) are the node and the link capacity constraints extended to power off nodes and PICs, respectively.Moreover, (7) affirm that if a node is powered off, then all its incident links must be powered off.Finally, the constraints (8) and ( 9) guarantee that nodes and even PICs of bundled links can be powered off.
The FGH heuristic uses the Maximum Spare Capacity (MSC) problem as building block.Taking into account the notations introduced in the previous section, the MSC problem can be formulated as follows: minimize ∑ (,V)∈  V (10) subject to (3), ( 4), and where   V is the capacity of the link (, V) (in the case of a bundled link   V =  PIC V ⋅  V , where  V is the number of powered-on PICs).Obviously, (11) and ( 12) are the node and the link capacity constraints, respectively.
The initialization of the FGH heuristic consists in solving the MSC problem to obtain the flow  V assigned to each link.Then, for each link the maximal number of PICs is removed by taking into account that the link capacity constraints have to be still satisfied.
After the initialization, the link with the greatest spare capacity is identified; that is, FGH finds the (, V) for which arg max where   denotes the number of PICs remaining after the initialization.Then, FGH tries to remove one PIC (in each direction) from the link (, V) because after the removal, the excess traffic that needs to be rerouted is the smallest.With the PIC removed, FGH solves the MSC with the new link capacities to find the new distribution of the traffic.If the problem has a feasible solution, the PIC considered for removal is permanently removed; otherwise it is not removed and the corresponding link is marked as final; that is, no additional PICs will be removed from final edges (FEs).Then in the next iterations, the identification of the link with the greatest spare capacity is performed by ignoring all final links.The iterative procedure concludes when all links become final.
Focusing on the ability of powering off link/PICs, HPARND-BL mainly differs from FGH in two aspects: (i) at each iteration a PIC is removed if the overall power consumption is actually reduced, (ii) and the Power-Aware Routing (PAR) problem is used as a building block in place of MSC.Furthermore, HPARND-BL is able to power off nodes.Therefore, HPARND-BL can exploit the ability of the chassis to be powered off, and this is particularly relevant when the power consumption due to route processor is dependent on the node throughput.In this case, the power consumption due to route processor has to be explicitly addressed in the problem formulation, and just to power off links/PICs does not necessarily determine a reduction of the overall network consumption.
The considered PAR problem consists in determining the traffic routing strategy that permits to reduce the overall power consumption of the network by taking into account only the power consumption of the nodes concerning the route processor, that is,  RP V,(V) .It is formulated by means of the following non-linear multicommodity flow model: (14) subject to (3), ( 4), (5), and (6).
PAR problem has been solved by means of Ipopt [28].This software implements an exact approach which considers the actual power consumption of the nodes (i.e., without performing a linear approximation of the non-linear  RP V,(V) curve).
In Algorithm 1, the lines 4 ÷ 17 enable HPARND-BL to power off nodes; lines 18 ÷ 31 have been obtained by modifying FGH, and their purpose is to try to power off single PIC.Similar to the strategy used by FGH to power off PICs, at each iteration of HPARND-BL, a transit node, TN, (i.e., a node V ∈ TN if ∑ (,V)∈  V + ∑ (V,)∈  V = 0) with the greatest spare capacity is temporally removed, it is marked as final vertex (FV), and PAR is solved by considering the new topology.If the problem has a feasible solution and the overall power consumption is actually reduced, the selected node is permanently removed; otherwise it is not removed.In the next iterations, the selected node is the one with the greatest spare capacity that is not included in the final vertices.The iterative procedure in lines 4-17 ends when all nodes become final vertices.The same behavior can be observed by considering the iterative component described by lines 18-31.

Implementation Issues
In the following section, we present some implementation issues to enhance the computational efficiency of the approach, to take into account the Quality of Service (QoS), and to evaluate the applicability of the proposed solutions in a real network scenario.

Flow Variable Modification.
In order to reduce the computational effort, we indeed considered and implemented aggregated versions of the flow variables, where all the commodities having the same origin node are considered to be "the same kind of flow." We have modified the constraints (4) as follows: where  is the set of sources of the traffic demand.

Quality of Service (QoS).
In order to take into account the QoS in the considered problem, we have limited the link utilization by multiplying  PIC V by a factor  ∈ (0, 1) in the link capacity constraints (6) and (12).The value of , which represents the link utilization, has to be appropriately determined in order to guarantee the QoS (i.e., a limited delay).To this aim, we have modeled the transmission of the traffic on the link as a M/M/1 queue, where the service rate  is the link capacity.By considering that the minimum link capacity of the network is  PIC V (i.e., a link composed of a single PIC), the mean delay of a single hop can be estimated as 1/ PIC V ⋅ (1 − ).

Flow Decomposition.
The solution of the PARND-BL formulation and the related heuristic provides the set of flow variables.Therefore, for providing the LSPs, as discussed in Section 2, the NCU has to decompose the flows into sets of paths between the origin-destination pairs.Note that the splitting of a traffic demand into too much LSPs can deteriorate the performance of the networks, due to the overhead of the signalling protocols used to create and manage LSPs.Hence, the solution should limit to few units the number of paths used to support the traffic of each origindestination pair [34][35][36].
The Multidestination Flow Decomposition Heuristic (MD-FDH) is a new heuristic for decomposing a flow (onesource multiple destinations) into a set of paths that is based on a heuristic proposed in [35].
After initializing the ℎ  vector to zero for each destination node  ∈   from the source node  ∈ , the MD-FDH (see Algorithm 2) consists in defining the set of residual links,   , selecting the destination node, , with the minimum traffic demand from  to it, and calculating the shortest path, , from  to  in the residual topology,   .Then, the ℎ V is calculated as the minimum among the traffic demand,   , and all the flow variables belonging to .Moreover, the values of   and   V are coherently reduced, and the computed path is added to the ℎ .Finally, the ℎ  is increased.The iterations continue until all the flow variables are equal to zero.

Simulation Scenario
The simulation analysis is devoted to evaluate the impact of the diverse parameters of the network scenario on the performance of the described solutions.In particular, we take into account the network topology, the energy behavior of the network equipment, the traffic load, and the QoS requirements.

Network Topology.
The first scenario is an European core network topology obtained from the Simple Network Description Library [37]; in particular, we considered the file nobel-eu.The other network scenarios are backbone topologies obtained from the set of data collected during the Rocketfuel study [38].In particular, we considered the following topologies: Exodus (US), Ebone (EU), Abovenet (Australia), and Sprintlink (US), which correspond, respectively, to the datasets AS 3967, AS 1755, AS 6461, and AS 1239 of the Rocketfuel study.The last considered network scenario is a large Austrian core topology (ta2) taken from [37] and given by the Telekom Austria.
The nodes of all the tested networks are assumed to have the same energy behavior, and the links are supposed to be symmetric; that is,  PIC V =  PIC V ,  PIC V =  PIC V , and  V =  V ∀(, V) ∈ .This last assumption implies that all the networks are indeed composed of undirected links, which are modelled in terms of two directed links, one for each direction.
Table 1 shows the statistics of the considered networks, that is, the number of the nodes (# Nodes), the number of the undirected links (# Links), and the number of the nodes connected to the rest of the network with a single undirected link (# S.L.N.).

Energy Characterization of the Network Devices.
In all network scenarios, each node represents a core router.We assumed the use of the Juniper T1600 core router, which has a total throughput capacity of 1600 Gb/s and a maximum power consumption of 8352 W [39]. Thus, all the nodes of the networks have the same energy behavior.For each link, we assumed to use multiple SONET/SDH OC768c/STM256 PICs.Each one has a payload bandwidth of 38.486 Gb/s and a power consumption of 65.7 W [40].
The results reported in [23] show that the power consumption of a chassis is equal to about 200 W for all classes of routers; thus, for each node V, we set   V = 200 W.In summary, we have considered the following setting for the model parameters: where  MAX V is the maximum power consumption of node V. Concerning the power consumption component of the route processor, we focused our attention on a cubic curve and on a logarithmic curve (see Figure 3).In particular, the cubic curve models the energy behavior of network devices that use energy-saving techniques such as Dynamic Voltage and Dynamic Frequency Scaling (DVS-DFS), which permit energy consumption to scale with resource requirements.Instead, the logarithmic curve corresponds to the energy behavior of devices that implement hibernation techniques such as the low-power idle mechanism standardized by IEEE 802.3az, which is based on the idea that data have to be sent as fast as possible so that equipment can return quicker to the low-power idle state [41].Current routers do not implement such techniques and are very energy inefficient, but we assume that they could be implemented in the nextgeneration routers.
Based on the previously cited energy parameters of the routers, we have thus defined the power consumption concerning the route processing by considering cubic and logarithmic curves, respectively, as follows: ⋅ log 10 ( (V) + 1) ∀V ∈ . (17)

Traffic Load.
In the case of the nobel-eu topology, the traffic matrix has been obtained from the data file "Nobel-2 directed graph" downloaded from [37].The file contains the undirected traffic demand between each couple of nodes of the considered network scenario.To obtain the directed traffic demands, we have randomly split the demand between the two directions.The total amount of traffic demand is 1898 Gb/s, distributed among the 756 origin-destination pairs (i.e., the pairs (, ) ∈ ).The mean traffic demand of a powered-on node toward all the other nodes is about   = 67.8Gb/s.To obtain the traffic matrix for the other topologies, we set the mean value of the traffic demand from  to ; that is,   , to   /(|| − 1).Then, each element of the matrix   has been extracted from a uniform distribution: The choice of a traffic matrix where each node of the network is both source and destination of traffic would imply that only links (or PICs) could be powered off.To introduce the presence of transit nodes that can be powered off, for each network topology we have selected the 20% of nodes with the highest degree and deleted the traffic demands from/to them.At the end of these procedures, for each network topology, we obtained the reference traffic matrix, that is, TM  .To vary the traffic load, we multiplied the reference traffic matrix by a scalar .

Design of Link Capacity.
The dataset used for the network topologies does not provide information on the capacity of the links and, in particular, on the number of PICs composing each link.Hence, the maximum number of PICs per link has been computed as follows: where   = ∑ (,)∈    denotes the total flow on link (, ) when the Shortest Path Routing (SPR) is applied, while  = 0.5 is the overprovisioning factor (see [25]).In more details, for calculating the values    , we assumed that, for each traffic demand   of TM  , a single shortest path from  to  is computed.The weight of each link is assumed to be equal to 1. Therefore, a minimal cardinality path has been computed for each origin-destination pair (, ).For the tested symmetric networks, we set  V =  V for each undirected link (, V).
Table 2 shows the statistics about the number of PICs per link for the different topologies, that is, the maximum (Max) and the average (Avg) numbers of PICs per link and the number of links composed of a single PIC (# S.P.L.).

QoS Requirements.
Taking into account the formula of the mean delay of the M/M/1 model assumed in Section 6.2, and the value  PIC V = 38.486Gb/s, the choice of  = 0.95 leads to a mean delay of 0.52 ns, which is widely sufficient to guarantee a low end-to-end delay.

Performance Analysis: Reference Scenario
The first set of simulations is devoted to the performance analysis of the PARND-BL model and of the proposed heuristic in a fixed configuration of network parameters.In particular, all results refer to a simulation scenario characterized by  = 1 and  = 0.95.
The PARND-BL formulation has been solved by the mixed integer non-linear programming solver BONMIN 1.5.0 [42] (only for small instances) or by means of a linear approximation by using the IBM ILOG CPLEX Optimization Studio V12.3 [27], whereas the solution of the PAR model has been calculated by using the Ipopt version 3.10.0[28].
The linear approximation consists in approximating the curves representing the power consumption of the route processing, that is,  RP V,(V) , via a piecewise linear function composed of 20 segments, as in [29].The points on the curves of Figure 3 represent the limits of each segment of the piecewise approximation.When possible, we also found the optimal solution of the non-linear PARND-BL model by using BONMIN 1.5.0.The results have shown that the difference (in terms of power consumption) between the solutions is order of 0.01%.However, the computational time needed to BONMIN for determining the optimal solution is often two orders of magnitude higher than the time spent by CPLEX to calculate the solution of the linear approximation.Furthermore, in some cases, BONMIN produced no results after 160 hours of CPU time.As a consequence, in the remaining of the study, we shall consider only the linear approximation of PARND-BL.
The performance analysis has been carried out by taking into account the following indicators: (i) number of powered-on network elements; (ii) overall power consumption of the network; (iii) CPU time; (iv) number of flow paths per origin-destination pair.
It is relevant to note that, for a fair comparison, we considered FGH under the setting  = 0.95.All the approaches have been also compared to SPR that represents in fact the widely used approach in core networks, when no specific administrative or cost constraints are present.

Number of Powered-on Network Elements.
The number of powered-on network elements (i.e., nodes, links, and PICs) obtained applying the considered approaches in the case of cubic and logarithmic  RP V,(V) is shown in Tables 3 and  4, respectively.The N/A (not available) value in the tables indicates that the optimal solution has not been obtained after 160 hours of CPU time; this situation has manifested in the case of large topologies, such as ta2.In this case, only the heuristics provided results.It is worth mentioning that, in the case of FGH, the solution is not affected by the  RP V,(V) .Thus, the results of FGH are only reported in Table 3, which refers to the cubic  RP V,(V) .
The results reported in Table 3 point out that the ability of powering off entire nodes is not well exploited by HPARND-BL.Only in the case of Abovenet and Sprintlink, the heuristic leads to power off a single node.With respect to this heuristic, the solution of PARND-BL model permits to power off two nodes in nobel-eu, whereas the further energy savings of one node are obtained in Ebone and Sprintlink.All approaches power off more than the 10% of links; the maximum is obtained in the Abovenet topology, where the FGH strategy leads to power off about the 26% of links.The comparison of the heuristics shows that in the case of small topologies, FGH provides lightly better performance than HPARND-BL.On the contrary, HPARND-BL produces better results in more complex network topologies.It is worth noting that the power consumption due to the route processor can be notable, as shown in the ta2 topology where HPARND-BL powers off less links and PICs than FGH but produces higher power savings (as discussed in the next subsection).
The comparison of the results shown in Tables 3 and 4 highlights that, in the case of logarithmic  RP V,(V) , the PARND-BL model and HPARND-BL power off a higher number of nodes and links with respect to the cubic case, whereas the number of powered-on PICs increases in each topology.This outcome emphasizes the ability of the PARND-BL model and HPARND-BL to take into account the power behavior of the route processor, which leads to obtaining solutions that are more energy efficient than those given by FGH.
Furthermore, in the case of cubic  RP V,(V) the PARND-BL approaches lead to network topologies with a few number of powered-off nodes and links, but with "thin" links (i.e., a high number of powered-off PICs per link).Instead, in the logarithmic case, they lead to topologies with more poweredoff nodes and links but with "thick" links.
In order to evaluate the impact of the power consumption of the chassis on the performance, we carried out further simulations by setting   V = 4 kW.The results are summarized in Table 5.They point out that, in the cubic case (cub in the table), there are no significant changes in the medium-large topologies (such as Sprintlink or ta2), whereas in the case of small networks we can note a variation in the characteristics of the obtained solution.Indeed, we obtained solutions with an increase in the number of powered-off nodes and links, with "thick" links.This reaction is due to the fact that the high V reduces the relevance of the  RP V,(V) term in the objective function (2).On the contrary, in the logarithmic case (log in the table), the increase of   V does not appreciably vary the results.6 summarizes the results of the overall network power consumption obtained by the diverse solutions in the cubic case.These results highlight that the power savings of the power-aware approaches with respect to SPR are very high and vary from the 50% to the 65%.Obviously the best performance is achieved by the optimal solution of the PARND-BL model (when available).

Overall Power Consumption of the Network. Table
As shown in Table 6, for medium-large topology HPARND-BL generally provides a good approximation of the optimal solution of the PARND-BL model (when available) and it saves further 2 ÷ 3% of energy with respect to FGH in the ta2 topology.On the contrary, for some small topologies (i.e., Exodus and Ebone), FGH provides better performance than HPARND-BL.
In the case of the logarithmic  RP V,(V) , the results reported in Table 7 point out that the power savings with respect to SPR are lower (up to more than the 15%) than the cubic case, although the absolute energy savings (in W) are higher.Moreover, there is an increase of the power savings of PARND-BL model and of HPARND-BL with respect to FGH (77% in nobel-eu).Furthermore, comparing the results given by the PARND-BL model and HPARND-BL, we infer that there is a slight worsening of the performance of HPARND-BL (when the optimal solution is available).These results can be explained by taking into account the results presented in the previous subsection; that is, the number of network elements that these approaches is able to power off.
The results obtained with   V = 4 kW are summarized in Table 8.In the case of the cubic  RP V,(V) (cub in the table), we can observe that the power savings of the PARND-BL model with respect to SPR are reduced; the power savings are order of 10 ÷ 20%.Furthermore, HPARND-BL outperforms FGH in the power saving.This behavior can be explained by taking into account that in this scenario the high value of   V (almost the half of  MAX V ) reduces the range of variability of  RP V,(V) , and so the power savings that can be obtained by the first addendum of the objective function (2).A similar behavior can be observed with the logarithmic  RP V,(V) (log in the table), although in this case the loss in the power saving is lower with respect to the cubic case.

CPU Time.
We evaluated the CPU times required by the different approaches for calculating the energy-aware solutions.The measured data have been collected referring to a 3.07 GHz Intel 4-Core i7 CPU (with hyperthreading enabled).The obtained results are summarized in Tables 9  and 10, which refer to the cubic and the logarithmic  RP V,(V) , respectively.In the case of small topologies, Table 9 shows that the solution of the PARND-BL model can be obtained in a time widely inside the required time scale; this result makes the utilization of the heuristics useless when the time scale for the network reconfiguration is order of few minutes.On the contrary, in the case of large topologies, such as Sprintlink or ta2, the CPU time becomes unacceptable (see Section 2).In these scenarios, the time required by the HPARND-BL for calculating a power-aware solution is into the time scale of few hours.This time is obviously longer than that shown by the FGH approach since the proposed HPARND-BL is more and shown in Table 10, we can notice that the times needed to optimally solve the PARND-BL model increase.In particular, even for small-medium topologies, such as nobel-eu and Sprintlink, the PARND-BL solution cannot be obtained in an acceptable CPU time.In these scenarios, HPARND-BL is profitable, since it provides a solution in acceptable CPU times, and it usually leads to lower power consumptions than FGH.
The analysis of the scenario with   V = 4 kW does not evidence relevant differences with respect to the   V = 200 W case. Hence, the corresponding results are not reported.

Number of Paths per Traffic Demand.
A relevant indicator to be considered in actual networks is the number of paths, which are used to support the computed multicommodity flows, as indicated in Section 6.The results of the analysis from this perspective are summarized in Tables 11 and  12, which refer to the cubic and the logarithmic  RP V,(V) , respectively.Table 11 shows that the number of paths is very low (i.e., 2 or 3) for the optimal solution of the PARND-BL model.On the contrary, the heuristic approaches may cause an increment of the number of paths (up to 10 in FGH).From this perspective, HPARND-BL generally exhibits better performance than FGH.These maximum values cannot be acceptable in an actual MPLS-based network architecture, although the average number of paths per flow is low.
In the case of a logarithmic  RP V,(V) , the results shown in Table 12 reveal a decrease of the average and the maximum number of paths per demand in the solution returned by HPARND-BL.This behavior can be explained by observing that in the cubic case the resulting power-aware routing strategy is to spread the load across the whole network, whereas in the logarithmic case the routing strategy is to concentrate the load in a small portion of the network, as it is easily deducible by the comparison of the results presented in Tables 3 and 4.
The results obtained with   V = 4 kW are summarized in Table 13.Such results indicate a decrease in the number of paths per demand for both the PARND-BL solutions and for both the cubic and the logarithmic  RP V,(V) .This behavior can be easily associated with the decrease of the number of powered-on links in the scenario   V = 4 kW.

Performance Analysis: Impact of Network Parameters
In this section, we report the results of the simulation study aimed at evaluating the behavior of the considered solutions under different traffic load conditions and QoS requirements.As described in Section 7.3, the variation of the traffic load has been obtained by multiplying the reference traffic matrix, TM  , by a scalar .We considered the traffic conditions obtained by varying  of 0.1 in the range 0.7-2.0.The analysis concerning the QoS requirements was carried out by varying the maximum link utilization; the values of  are variation of 0.05 in the range 0.50-1.00.All simulation results refer to the cubic  RP V,(V) , since this represents the most critical scenario for the heuristics and the most interesting from the potential power saving perspective.9.1.Impact of Traffic Load.The results of this analysis are summarized in Figure 4, which shows the network power consumption of the considered approaches as a function of .We report only the results obtained with the Ebone and the ta2 scenarios, which represent small-medium and large topologies, respectively; the results for the other considered topologies are similar and hence are omitted for the sake of brevity.In general, the results highlight an increment of the gap between the PARND-BL approaches and FGH in almost all the topologies when the traffic demand increases.For example, in the Ebone topology (see Figure 4(a)) the power savings of HPARND-BL with respect to FGH, which were negative at  = 1.0, are more than the 2% at  = 2.0.Instead, in the ta2 topology (see Figure 4(b)), the power savings of HPARND-BL firstly increase and successively are quite stable, with a reduction of power consumption of about 14% at  = 1.5.This trend is due to the ability of HPARND-BL to be aware of the power consumption of the route processor.Obviously, the optimal solution of the PARND-BL model is not reported because it has not been obtained after 160 hours of CPU time.9.2.Impact of the Maximum Link Utilization.The results of the overall power consumption as a function of the maximum link utilization are summarized in Figure 5 for the Ebone and the ta2 topologies.The figure points out that the compared strategies have a similar behavior.In particular, in all cases we can observe a decrease in the power consumption as the maximum link utilization increases.This result can be easily explained by taking into account that the power-aware solutions try to optimize the utilization of the poweredon links/PICs.This action is performed while attempting to power off the network elements that actually lead to a reduction of the overall network power consumption without degrading the traffic transport service.The power saving of the PARND-BL approaches with respect to FGH just lightly increases when  increases.For example, for the ta2 topology the power saving varies from 1% to 5%.Furthermore, the curves depicted in Figure 5 indicate the impact of the network greening on the QoS requirements.Indeed, referring to Figure 5(a), we can see that by allowing a maximum load of  = 0.95 (corresponding to a mean delay of 0.52 ns) with respect to  = 0.50 (0.05 ns), we can obtain power savings of about 30%.

Conclusion
The paper deeply analyzed the PARND-BL problem, proposed the related heuristic HPARND-BL, and presented the performance evaluation carried out by taking into account diverse network parameters and metrics.The simulation results show that the power saving is more evident in the case of a cubic  RP V,(V) .In this case, the solutions of the PARND-BL model and HPARND-BL outperform FGH.Moreover, the power saving obtained with the PARND-BL approaches with respect to FGH increase in the case of a logarithmic  RP V,(V) .The analysis of the results indicates that the energy saving depends on the network size (in terms of links and nodes).In the case of large topologies, the solution of the PARND-BL model and HPARND-BL provide the best results, whereas for small topologies, FGH permits to achieve interesting power savings, near to the optimal solution obtained by solving the PARND-BL model.Anyway, in these cases, the solution of PARND-BL is the more profitable; that is, it has highest energy savings and lowest CPU times.
The analysis of the number of powered-on network elements obtained by the diverse solutions highlights that in some cases the strategy of powering off the network   elements is not always the best choice for saving network energy.As an example, by referring to the ta2 topology results shown in Tables 3 and 6, although the FGH solution produces the lowest number of powered-on links and PICs, the overall network power consumption is higher than in the solution computed by HPARND-BL.In general, FGH (and other similar approaches focused on powering off network elements) minimizes the number of powered-on network elements through the traffic consolidation, but this strategy is not always the best one.Indeed, depending on how the power consumption of the route processor varies with the node traffic throughput, in some cases the more efficient routing strategy can be the spreading of the traffic among the nodes, by avoiding powering off the maximum number of network elements.The introduction of  RP V,(V) in the objective function of the PARND-BL approaches permits to take into account this significant aspect and leads to having a solution that is able to adapt the NWPM strategy to the power behavior of the network devices.Indeed, the analysis of the solutions of the PARND-BL model showed that different energy behaviors of the route processor (logarithmic or cubic curves) lead to different routing strategies (traffic consolidation or load

Figure 5 :
Figure 5: Power consumption versus maximum link utilisation.

Table 1 :
Statistics of the network scenarios.

Table 2 :
Number of line cards per link.

Table 11 :
Number of paths per flow (Avg/Max)-cubic curve.