Joint Optimization of Energy Conservation and Migration Cost for Complex Systems in Edge Computing

School of Computer and So ware, Nanjing University of Information Science and Technology, Nanjing, China Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science and Technology, Nanjing, China Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science & Technology, Nanjing, China Department of Computer Science and Engineering, Michigan State University, MI, USA School of Information Science and Engineering, Qufu Normal University, Qufu, China


Introduction
In recent years, fossil fuels are adopted in most energy systems, which generate a lot of harmful substances and pollute the environment [1,2].In order to save fuel and reduce pollution to the environment, renewable energy gradually replaces the fossil energy.Nevertheless, renewable energy is intermittent and di cult to control.For example, provided that the wind stops, wind power generation is di cult to continue.With the intermittent nature of renewable energy, one type of renewable energy cannot meet the balance between supply and demand of current energy systems [3].erefore, it is of urgency to develop complex systems that integrate multiple renewable energy sources.Complex systems are systems of intelligent, adaptive subjects with a medium number of actions based on local information, which are de ned by attributes like feedback and adaptation.Considering the intermittent nature of renewable energy, a large number of mobile devices are used in complex systems.e mobile devices are arranged to collect real-time environmental parameters to provide reliable references for strategy formulation.
Although the environmental information is valuable, in view of the huge volume of data in complex systems, the resource consumption of mobile devices rises sharply [4].Constrained by numerous factors such as limited battery life and limited capacity, mobile devices gradually fail to meet the requirements of mobile services and application execution is of less e ciency in local mobile devices.erefore, it is suitable to o oad the computing applications or services to the cloud platform, considering that the resources in the cloud are accessed conveniently [5].In the cloud platform, physical resources are provided in the form of multiple virtual machines (VMs) and each mobile device corresponds to a cloud clone which is deployed on a VM.Users are able to access the con gurable computing resources shared pool in the cloud and use the required cloud clone to perform the o oaded mobile services [6].
Nevertheless, the transmission delay is unneglectable in terms of the long distance between mobile devices and the remote cloud.Besides, transmitting mobile services to the cloud consumes massive energy in complex systems, violating the concept of green cloud computing [7].Edge server, which is co-located with access point (AP), is proposed to be a substitute for the cloud.Mobile users access computing resources of edge servers through APs.Compared to the cloud, edge servers are in close proximity with mobile devices, reducing the data transmission delay [8].Furthermore, the virtualization technology is used in edge computing as well.By means of the virtualization technology, hardware resources in edge servers are decoupled from so ware and the same type of VM can run in di erent physical devices [9].Consequently, mobile so ware is enabled to execute on multiple hardware, increasing the utilization of hardware resources as well as achieving the energy conservation.
Despite the advantages, the adoption of edge servers in complex systems has several shortcomings.In fact, compared to remote cloud clusters, the computing resources of edge servers are limited.In densely populated areas, as the number of mobile devices increases dramatically, there is a high possibility that a mass of users intends to o oad the computation-intensive services or applications to edge servers nearby.Hence, the edge servers need to response the resource requirements from a large number of mobile devices.As edge servers cannot provision su cient computing resources for abundant services simultaneously, the computing resources of edge servers run out, resulting in the rejection of new service requests [10,11].
e applications whose requests are rejected are queued in the edge server until the previous application is completed and the needed resources are available again.Especially, provided that the waiting applications are latency-sensitive, unexpected consequences take place and the quality of experience (QoE) for users greatly drops o .us, it is necessary to o oad the queued service quests to the edge servers with unoccupied computing resources through VM migration.
Apart from the waiting time, energy consumption is a key criterion for resource management in complex systems.With the explosive growth of consumed energy for information and communication technology (ICT), numerous greenhouse gases are emitted into the atmosphere [12].us, energy conservation plays a signi cant role in the implementation of future wireless communication.Considering that edge computing is of great use in wireless communication, the energy consumption of edge computing is a research hotspot for better communication modes [13].In complex systems, the energy consumption of computation o oading includes two parts.e rst part is the energy used for information transmission across edge servers.With the virtualization technology, the transmission is in the form of VM migrations.at is to say, the rst part of energy consumption is the energy consumption of VM migrations.Moreover, the other part is the energy consumption of infrastructures, including the energy caused by data processing in edge servers and the energy caused by sending and receiving data in APs.As the con guration of each edge server is the same and, the energy consumption of data processing is the same for a service.With the consideration of time consumption and energy consumption, it is of utmost signi cance to develop a strategy to achieve the joint optimization of migration cost and energy consumption.
e main contributions of this paper are summarized as follows: (i) We present the basic concepts and de nitions to analyze the VM migration cost and the energy consumption of VM migrations in complex systems.(ii) We design a Balanced Resource Scheduling Method for trade-o s between VM migration cost and energy consumption of VM migrations for edge server management, named BRSM, to dynamically provision resource management in complex systems by live VM migration technique.(iii) We adopt nondominated sorting genetic algorithm III (NSGA-III) [14] to bring about the multi-objective optimization.Multi-criteria decision-making (MCDM) and simple additive weighting (SAW) are employed to select the optimal VM migration strategy.(iv) We conduct simulations to verify the e ciency of our proposed method BRSM.
e remainder of this paper is organized as follows.Section 2 introduces the basic concepts and the completed modeling as well as formulation.Section 3 elaborates the proposed dynamic resource management method.Section 4 shows the comparison analysis of the simulation experiments.Section 5 reviews the related work.Finally, Section 6 outlines the conclusions and future work.

System Model and Problem Formulation
In this section, basic concepts and de nitions for complex systems in edge computing are introduced.Besides, VM migration cost and energy consumption of VM migrations are also analyzed.Key terms and the descriptions are listed in Table 1.In complex systems, the experience of mobile users are promoted continuously and edge servers provide computing services which are in close proximity to mobile terminals, increasing the QoE for mobile users.ere is an AP which is located with an edge server, forming an edge computing device (ECD).Mobile users are able to access the resources in the edge server through the colocated AP.Assume that there are edge servers in complex systems, which are denoted as = 1 , 2 , . . ., .
Due to the loss of generality, there is a physical machine (PM) in each edge server to provide mobile services.As the virtualized technique plays a critical role in the resource management for infrastructure, it is appropriate to adopt the virtualized technique in complex systems.e physical resources in an edge server are provided in the form of virtual machines (VMs).By means of the virtualized technique, the physical resources in the edge server are abstracted as several resource units, which are known as VM instances, to accommodate mobile services.Aiming to measure the capacity of each edge server, let be the number of VM instances in the -th ( = {1, 2, . . ., }) edge server .Considering that there are separate access points, we assume that there are access points, denoted as = 1 , 2 , . . ., , and the -th ( = {1, 2, . . ., }) access point is denoted as .e mobile service set o oaded to VMs is denoted as = 1 , 2 , . . ., and represents the number of mobile services.For resource provisioning, the mobile service ( = {1, 2, . . ., }) in requires multiple VMs which are in the requirements of desired execution time.
De nition 1 (VM occupation requirement of s z ).A 3-tuple, which is denoted as = (nv z , os z , od z ), is used to de ne the VM occupation requirement of , , and are the number of VMs of , the occupied start time, the duration time respectively.
To avoid the unnecessary transmission delay, the VMs which host the same mobile service are deployed in the same edge server, decreasing the communication cost.

De nition 2 (packaged VM for ). Let
represent the packaged VM for the mobile service .e packaged VM contains all VMs to host the mobile service , whose number is .
Figure 1 presents an instance of mobile service o oading in the form of VM migration.e AP which is deployed next to the source edge server is named as AP sour and the AP deployed next to the destination edge server is named as AP dest .ere are several APs between the AP sour and AP dest , each of which is named as AP mid .In a VM migration strategy, the VM image and the dirty pages caused by migrations are transmitted from the AP sour to the AP dest across AP mid s.

Migration Cost Analysis of VM Migrations.
Considering that VM migration takes a lot of time and it is not always possible to be done, it is necessary to calculate the cost caused by the VM migrations during the whole execution period , .
e migration cost, which also represents VM downtime, consists of the access time of the log le and the switch time of the VMs [18].Suppose the transmission times of the memory image is ( ) during the migration process of from to ὔ.At the time instant , ( ) is a binary variable to judge whether is deployed on , which is de ned by , ὔ ( ) is a binary variable to judge whether is migrated from to ὔ, which is de ned by When the dirty page transfers at ℎ ( ) time, let ℎ ( ) be the access time of all log les which contains ℎ ( ) and ℎ ( ) [18].When needs to be migrated between the edge server and the AP which is located next to the edge server, the time consumption is de ned as of data processing is the same for a service whatever the o oading strategy is.erefore, the energy consumption of data processing is not calculated in this paper.At the time instant , the energy consumption of running VMs in edge servers is calculated by where is the energy consumption rate of each running VM.
e edge server keeps running until all VMs in this edge server have nished the hosted tasks.erefore, the running time of the -th edge server is calculated by e energy consumption of idle VMs in edge servers is calculated by where is the energy consumption rate of VMs in idle mode.
e basic energy consumption of edge servers is calculated by where is the basic energy consumption rate of .During the VM migrations, APs frequently receive and send data, contributing to the energy consumption in complex systems.The energy consumption rate of is calculated by ( 10) where ℎ ( ) is the size of dirty page produced by the migration of at ℎ ( ) time, and is the bandwidth between the edge server and its corresponding AP.
Suppose , ὔ is the total number of APs between and ὔ, including a AP sour , a AP dest and several AP mid s. e AP sour is located next to and the AP dest is located next to ὔ. us, the time consumption caused by the migration of from AP sour to AP dest is calculated by where is the bandwidth between APs.
e process of VM migration is divided to into multiple rounds.In the rst round, the VM image would be sent to the destination PM while a er that the dirty page produced in the previous round is sent to the destination PM in each subsequent round.erefore, the size of dirty page is calculated by where ( ) refers to the size of the image memory of and ( ) is the producing rate of the dirty page.e switch time of in the migration operation is calculated by where ( ) represents the switch time of the packaged VM .

De nition 3 (migration cost of a packaged VM in
). e migration cost represents the time consumption of a VM migration.When is migrated from to ὔ, the migration cost is calculated by e average migration cost is calculated by During the whole execution period , , the migration cost is calculated by

Energy Consumption Analysis of VM Migrations.
Apart from the VM migration cost, we consider the energy consumption in complex systems during the whole execution period , .
Generally, we calculate the energy consumption of edge servers and APs.With the adoption of virtualized technique, the energy consumption of edge servers mainly includes the energy consumption of VMs in edge servers.Considering that each edge server con guration is the same, the energy consumption ).e -th ( = {1, 2, . . ., }) allocation record, which is denoted as = , , , , , preserves the distribution for the packaged VM.In the -th allocation record, ae m , ac m , av m , as m and ad m represent the allocated mobile service, the occupied edge server, the employed VM, the requested start time and the duration time respectively.
According to the number of VM instances for a mobile service, the same number of allocation records are generated.By means of the allocation records, the idle space of each edge server is presented dynamically, considering that the allocation record combines the elapsed time of an edge server.
Algorithm 1 speci es the key idea for edge server idle space acquisition.
rough identifying the occupied VM instances, the key idea is to acquire the number of idle VM instances.We input the allocation record collection , the scheduling instant and the edge server .e output is the spare space is t,n for the edge server at .
On the other hand, since the mobile services are o oaded almost continuously, the load distribution of the edge servers changes dynamically.erefore, migrating the VMs from the overload edge servers to the underload edge servers is urgent.
Due to the real holding time of VMs, when a VM is migrated from one edge server to another, the original allocation record needs to be updated and revised.Moreover, the migration leads to a generation of new allocation records for the referred VM instance.In this new allocation record, the migration time is the start occupation time while the duration time is determined by the rest hosting time of the mobile service.
In Algorithm 2, the allocation record updating is specied.We input the scheduling instant , the processed objective edge server , the destination edge server dc and the destination VM dv. e output is the updated allocation record where , , , , respectively represents the baseline power of , the signal transmission power factor of , the signal transceiver power factor of , the tra c demand of and the link rate between and .
Based on the energy consumption rate of APs, the energy consumption of APs is calculated by Besides, the energy consumption of the switch operation of VMs is calculated by

De nition 4 (energy consumption of VM migrations).
e energy consumption of VM represents the consumed amount of energy a er a VM migration.e total energy consumption is calculated by During the whole execution period , , the energy con- sumption is calculated by 2.4.Problem De nition.From the foregoing, the migration cost of VMs and the energy consumption of VM migrations are analyzed and quanti ed.In this paper, we aim to achieve the goal of minimizing the migration cost presented in (9) and reducing the energy consumption presented in (18) while meeting the capacity constraints of ECDs.e problem is formulated by

BRSM: A Balanced Resource Scheduling Method
In this section, the occupation conditions of edge servers are detected and updated during the dynamic resource scheduling process.en NSGA-III is adopted to generate resource scheduling strategies to select.Finally, SAW and MCDM are employed to evaluate the resource scheduling strategies and select the appropriate resource scheduling strategy.

Edge Server Status Detecting and Updating.
As the formulation of the resource scheduling strategies largely depends on the load of edge servers, it is necessary to detect the resource usage of each edge server.In this section, the analysis of the allocation record is utilized to preserve the resource usage of edge servers.Assume that there are allocation records during the execution period, which are denoted as = 1 , 2 , . . ., .Complexity 6 adopted to solve the optimization problem while satisfying the constraints simultaneously.

Initialization.
In the initialization, the parameters of GA are rstly determined, including the population size , the maximum times of iteration , the probability of crossover and the probability of mutation .

Crossover and Mutation.
e crossover operation, which combines two chromosomes, is the single-point crossover in this paper.Figure 2 shows the crossover operation for two chromosomes in one schedule.In this instance, we select a crossover point and swap the genes which are around the selected crossover point.In particular, the crossover operation starts at 3 and the swap creates two new chromosomes.
Aiming to enhance the tness values, the mutation operation is taken to modify genes to create new chromosomes.Each gene has the same probability of being modi ed. Figure 3 illustrates an instance of the mutation operation.In this instance, the code of 3 is mutated from 6 to 1.

Selection Operation for the Next Generation.
For better solutions, the individuals are selected to generate the next population in this subsection.A er the operation of crossover and mutation, the size of population is 2H.e migration cost and the energy consumption of each scheduling strategy are calculated respectively to judge whether the scheduling strategy is optimal.According to tness values, the scheduling strategies, which are also known as solutions, are sorted by the usual domination principle.erefore, the nondominated fronts are successfully generated.
As the preparation is completed, the selection operation of chromosomes is conducted.Each time one solution is selected from the highest nondominated front until there are selected solutions.We de ne that the last added solution is in the -th nondominated front.Provided that all of the solutions in the -th nondominated front are selected, the selection operation n shes and the selected solutions form the next generation.is kind of selection is called primary selection.Whereas in reality, all solutions in the -th nondominated front are not be selected.As the solutions in the -th nondominated front are not all included, further selection needs to be conducted.Consider the number of selected solutions in the -th nondominated front is .We need a series of operations to ensure that the selected solutions are better.is kind of selection is called further selection.
First, we normalize the 2Z tness values of individuals in the population.We search the minimum values of the VM migration cost and the energy consumption, which are denoted as and .us, the tness values for the solutions are updated as collection .At the migration instant , the allocation record is updated for the objective mobile service (Lines 1-10).Besides, a new allocation record with the VM instance is produced at the destination edge server and the new allocation record is added to the allocation record collection (Lines 11-16).

Resource Scheduling Method Using NSGA-III.
As the VM instances are migrated from the overload edge server to the underload edge server, the resource scheduling problem is de ned as a multi-objective optimization problem to minimize the VM migration cost and the energy consumption of VM migrations.Compared with traditional genetic algorithms, NSGA-III reduces the complexity of the noninferior sorting genetic algorithm and expands the sampling space, thus greatly improving the convergence speed [14].Considering that NSGA-III solves multi-objective optimization problem e ciently, NSGA-III is employed to solve the optimization problem.

Encoding.
In this subsection, each packaged VM has a scheduling strategy and we encode for the VM scheduling strategies.In the genetic algorithm (GA), a gene represents a migration strategy of a packaged VM and a chromosome, which is composed of a set of genes, represents a hybrid migration strategy of VMs in the same schedule.e destination edge server of VM migrations is encoded, depending on the edge server set = 1 , 2 , . . ., .

Fitness Functions and Constraints.
A chromosome represents the o oading strategies of all packaged VMs which are in the same schedule.In the population, each chromosome con rms a solution in the aspect of the resource allocation optimization problem.erefore, tness functions emerge as standards to measure the superiority of possible solutions.In this paper, the tness functions include two categories: the VM migration cost and the energy consumption of VM migrations, presented respectively in ( 9) and (18).As shown in (19) and (20), the design intent of the method is to select an appropriate resource allocation strategy for trade-o s between the two tness functions as well as satisfying the potential constraints.With e ciency and e ectiveness, NSGA-III is According to the number of reference points that each solution associates with, the solutions in the -th nondominated front are sorted and one of the solutions is randomly selected.is process repeats until solutions are selected.
Algorithm 3 represents the selection operation for the next generation.In this algorithm, we input the -th generation population and a set of reference point while the output is the ( + 1)-th generation population +1 .In each iteration, the tness values of every solution are calculated (Lines 1-4).With the operation of crossover and mutation, there are 2H solutions to be chosen (Lines 5-6).en the 2H solutions are sorted according to the nondominated principle (Lines 7-9).Finally, by means of the reference points, solutions are selected as the next generation (Lines 10-17).e process of Algorithm3 would repeat until the maximum iteration.

Scheduling Strategy Based on SAW and MCDM.
In a population, there are chromosomes which are also called solutions.Each chromosome represents a feasible hybrid scheduling strategy.For the chromosomes, our designed method is to achieve the trade-o s between reducing the VM migration cost as well as optimizing the energy consumption of VM migrations.For the generated solutions, SAW and MCDM are employed to select the relatively optimal solutions.
As the criterions are divided into positive criterions and negative criterions, the migration cost is a negative criterion, which means that the lower the migration cost is, the better the solution is.Similarly, the energy consumption is also a negative criterion.
e migration cost is normalized as where max and min represent the maximum and minimum of the migration cost in the population respectively.Moreover, the energy consumption is normalized as where max and min represent the maximum and minimum of the energy consumption in the population respectively.ere is no feasible way how to minimize both latency and energy at the same time, it is rather a trade-o between these two objectives.Consequently, each of the two objectives is given a weight.rough di erent values of the two weights, the impact of each objective on the o oading strategy changes accordingly and the trade-o can be realized.Let 1 , 2 be the weight of the migration cost and the energy consumption respectively.For the comprehensive utility values evaluation of the solutions, the optimization of the VM migration cost Based on the tness values, the extreme values of the VM migration cost and the energy consumption of VM migrations, which are denoted as and , are calculated by where and is the weight vector of the two tness functions respectively.
We use each tness function as a measure of an axis.In the hyperplane, the intercept of each axis is calculated respectively.e 2Z tness values of individuals in the population are normalized as A er the normalization process, the two tness values are put in the domain [0, 1).en the distributed reference points are connected with selected solutions, which improves the diversity of solutions.With the normalization, the intercept of each axis is 1 and each axis is divided into parts.Besides, the number of the reference points is denoted as .To ensure that every solution is associated with a reference point, is Complexity 8 4.1.Simulation Setup.In the experiment, we select LENOVO TS250 as the edge server, which consists Intel Xeon-E3-1225V6, Quad-Processor clocked at 3.4 GHz and 4 GB of RAM.To conduct the experiment evaluation, 5 datasets of di erent scales are generated, whose number of mobile services are 1000, 2000, 3000, 4000, 5000 respectively.Additionally, the concrete parameter settings in the experiment are presented in Table 2 [18].Speci cally, datasets can be downloaded at https://pan.baidu.com/s/1coXNu8B5_YdXAWTBJSn6uQ.e VM migration cost and the energy consumption of VM migrations are the two criteria to judge whether the method BRSM is e cient.Aiming to represent the superiority of BRSM intuitively, we adopt comparative methods to conduct the experiment evaluation.e comparative methods are speci cally described as follows.
(i) First Fit (FF).In the order of starting addresses of edge servers, the resources in each edge server are checked and the mobile services are respectively assigned to the rst edge server encountered with enough resources.(ii) Best Fit (BF).e mobile services are o oaded to the edge server with enough resources respectively while the destination edge server has the least resources among all edge servers.and energy consumption of VM migrations are both taken into consideration.Since the two objectives have the same degree of in uence on the selection of the o oading strategy, the weights of the tness function are set equal in this paper.erefore, the utility value of the ℎ-th solution is calculated by Based on the utility value of the ℎ-th solution, the optimal solution, which is denoted as V(C), is calculated by At this point, the optimal solutions with maximum utility value have been selected from the population.

Method Overview.
In this paper, we aim to minimize the migration cost and the energy consumption of each service o oading strategy.e resource scheduling problem is quanti ed as a multi-objective optimization problem.NSGA-III is utilized to generate the resource scheduling strategies.First, we specify the real-time conditions of edge servers and build the resource allocation record collection.en the randomly generated resource scheduling strategies are encoded.Besides, the tness functions and potential constraints are proposed to judge each solution.In order to generate new solutions, the operation of crossover and mutation are taken, leading to a total of 2H solutions.For the next generation, the usual domination principle and the reference points are used to select the relatively appropriate solutions.Finally, in spite of SAW and MCDM, the optimal solutions are selected from the population.
Algorithm 4 shows the overview of the proposed method BRSM.e input includes the initialized population , the allocation record set and the maximum iteration times .
e output is the new allocation record set .In this algorithm, we rst update the running mobile service collection and obtain the spare space of edge servers (Lines 3-6).en we formulate the initial resource scheduling strategies randomly and conduct crossover and mutation operation to generate 2H solutions (Line 9).Aiming to select the appropriate solutions, we calculate the tness functions for each solution and do the selection operation according to Algorithm 3 (Lines 10-13).is process would repeat until the number of iterations reached .Based on the utility value of each solution, the alternative solutions are sorted and we select the optimal solution which has the highest utility value (Lines 16-20).e ow chart of selecting the optimal strategy is shown in Figure 4. Finally, the resource allocation records are updated and output (Line 21).

Experimental Results and Analysis
In this section, simulations and experiments are conducted to evaluate the performance of the proposed method BRSM.First, the parameter settings are introduced.en, the performance evaluation on BRSM is presented.
(1 ≤ ℎ ≤ ). of solutions at di erent mobile service scales is presented.

Input: X, A, I, S
As can be seen from the gures, regardless of the number of mobile services, there are always 3 solutions generated by BRSM.Among the generated solutions, the most balanced service o oading strategy is acquired, depending on the utility value in (30).e service o oading strategy with maximum utility value is selected as the most optimal o oading strategy.For instance, in Figure 4(a), the nal service o oading strategy is solution 2 as it has the highest utility value.

Evaluation on the Amount of Employed Edge Servers.
In the edge server-based WMAN, the number of employed edge servers is a key factor to measure the resource utilization of edge servers.erefore, it is necessary to evaluate the number of employed edge servers.Given the 5 di erent scale datasets employed in FF, BF, FFD, BFD, and BRSM, the number of employed edge servers di ers from each other a er VM migrations, which is illustrated in Figure 6.In Figure 6, it is obvious that FF and BF utilize the same number of edge servers.FFD as well as BFD also utilizes the same number of edge servers.However, BRSM employs fewer edge servers than the other four methods.

Evaluation on Resource Utilization.
Resource utilization of edge servers is of great signi cance, which is calculated by virtue of the number of running VMs in each edge server.High resource utilization represents the rational utilization of resources.In Figure 7, the comparison of average resource utilization by FF, BF, FFD, BFD, and BRSM is presented.FF and BF have the same resource utilization, while FFD and BFD likewise achieve the same utilization.Distinctly, compared with FF, BF, FFD and BFD, BRSM has the highest resource utilization value, proving the superiority of the proposed method BRSM in the respect of resource utilization.
In the process of dynamic VM migrations, the resource utilization of edge servers, which changes with time instants, is taken into consideration.To monitor resource usage in real time, we track the resource utilization of edge servers when there are 5000 mobile services during the execution period.Figure 8 illustrates the comparison of real-time resource utilization by FF, BF, FFD, BFD, and BRSM.According to the analysis, the resource utilization of edge servers exceeds 80% during the execution period.Besides, BRSM achieves better resource utilization than FF, BF, FFD, and BFD in almost every instant, guaranteeing the good performance of BRSM.

Evaluation on Migration Cost.
e migration cost represents the time consumption of VM migrations across edge servers.However, the migration cost is divided into two parts, which are the cost between AP and edge server as well as the cost between APs.
Moreover, the transmission time of dirty pages caused by last round of data transmission is considered.Figure 9 shows the comparison of migration cost by FF, BF, FFD, BFD, and BRSM, using 5 di erent scales datasets.It is intuitive that FF and BF own the same migration cost while FFD and BFD also own the same migration cost.Nevertheless, our proposed (iii) First Fit Decreasing (FFD).e mobile services are sorted in descending order according to the amount of required resources.en the rst mobile service is o oaded to the rst edge server encountered with enough resources.is procedure continues until all mobile services are o oaded to edge servers.(iv) Best Fit Decreasing (BFD).e mobile services are sorted in descending order according to the amount of required resources.en the rst mobile service is o oaded to the edge server which has the least resources among all edge servers but enough for the mobile service. is procedure is repeated until all mobile services are o oaded to edge servers.

Performance Evaluation of BRSM.
In this section, we evaluate the number of employed edge servers, the resource utilization of edge servers, the VM migration cost and the energy consumption of VM migrations.e corresponding evaluation results are shown as follows.

Evaluation on the Utility Value of the Solutions.
In the ve sub-gures in Figure 5, the comparison of the utility value  and BRSM are compared in Figure 10.Under the di erent scales of mobile services, the energy consumption of VM migrations are di erent.It is obvious that FF and BF has the same energy consumption during the VM migrations, and FFD, as well as BFD, also has the same energy consumption.Moreover, the energy consumption of BRSM is lower than the energy consumption of FF, BF, FFD, and BFD, which draws the conclusion that BRSM achieves better energy conservation than the 4 comparative methods.
method BRSM has the least migration cost when compared with the other 4 methods.mobile stations [25], QoE and operation cost of users movement [26], security problems of data transmission [27], load balance of the network [28].Zhou et al. [24] formulated a heterogeneous networks framework to support computing and content caching.Sheu et al. [25] proposed an advanced power saving mechanism named A-PSM, which uses the average packet inter-arrival time to adjust the sleep window to get command of power usage.Guan et al. [26] divided a metropolitan area into di erent clusters to minimize the sum of handovers between MEC regions, increasing the QoE and reducing the operation cost in WMAN.Baskaran et al. [27] presented an enhanced scheme compared with Privacy Key Management Protocol Version 2 (e-PKMv2).By feat of a new technique named Blind Key Distribution (BKD), the security and reliability problems could be improved.Jakimoski et al. [28] designed a vertical handover decision algorithm to optimize the load of WMAN, relying on the velocity of the mobile devices and load of the whole network.
Generally, the applications would be more computation-intensive as various requirements would be added to them for resource response [5,29].erefore, the transmission delay during the process of connecting the mobile users and servers is costly in complex systems [5,30,31].O oading tasks to edge servers is a suitable way to reduce the delay of applications.Edge servers are reliable server clusters and the physical resources of edge servers are provided in the form of di erent kinds of VMs.us, users could resort VMs to deal with computing tasks through establishing wireless communication with nearby edge servers.
Cui et al. [32] devised a so ware-de ned cooperative oloading model based on the regularities of distribution of users and used an online task scheduling algorithm to reasonably distribute energy among devices.Similarly, Chen et al. [33] studied the edge server deploy problem as well, aiming to minimize the access delay of users and decrease the total cost which is represented by the number of edge servers.To achieve the goals, heuristic and clustering algorithms were designed for reducing delay while an integer linear programming is formulated to lessen the used edge servers.Panigrahi et al. [34] proposed an energy e cient o oading approach to increase the resource utilization of edge servers for better energy usage.Analogously, Gai et al. [35] designed a dynamic energy-aware edge server-based mobile cloud computing model for energy saving according to dynamic edge servers (DCL)-based model.
When the scale of mobile users expands rapidly, they access the edge servers frequently and the resources run out rapidly.Consequently, the new requests from mobile devices could be rejected and a large number of users are queued.As the access delay caused by waiting is nonnegligible, negative e ects would be brought about and the QoE for users would drop dramatically [36,37].Di erent aspects of researches on edge servers are conducted to reduce the transmission delay [36][37][38].
Jia et al. [36] devised an algorithm to optimize the edge server placement problem in WMAN.e algorithm could assign mobile users to edge servers which have been placed to balance the workload.On the other hand, a novel model is

Related Work
With the adoption of fossil fuels, various problems such as air pollution and rapid resource consumption have aroused, jeopardizing the sustainable development of environment [1,15].To improve the environment, renewable energy is used as a substitute for fossil fuels [15,16].However, the supply of renewable energy depends on many uncontrollable natural factors, which is intermittent.Consequently, complex systems are utilized to integrate kinds of renewable energy to achieve continuous supply of stable energy [17,18].With the largescale data in complex systems, cloud computing emerges as an applied paradigm [19,20].e mobile devices transmit the data to the cloud platform while the long distance between mobile devices and the cloud leads to unneglectable transmission delay.us, the data are transmitted to the edge servers which are close to mobile devices, reducing the transmission time consumption [21][22][23].Edge computing has been studied thoroughly in many aspects, including virtual resource allocation strategy formulation [24], energy consumption of o oading requests, the proposed method allows the system to formulate appropriate service o oading strategies to relieve the load on the overload edge servers and realize e cient application execution.With the increasing service requests sent from mobile users, the cloud services are guaranteed not to be greatly in uenced through o oading the services in the overload edge servers to other edge servers which have idle computing resources and storage.First, the o oading problem is de ned as a multi-objective optimization problem to reduce the VM migration cost and achieve energy conservation during the o oading process.en, NSGA-III is adopted to deal with the multi-objective optimization problem.Finally, relative experiments are conducted to evaluate the performance of the proposed method BRSM.
In future work, the proposed method BRSM would be extended to the real-world scenario of WMAN.Besides, execution time limits are speci ed for computing tasks to identify a computation o oading scheme to reduce the migration cost as well as the energy consumption.

Conclusion and Future Work
e ability to provide multifarious cloud services is signi cant for the modern WMAN system.In this paper, a balanced resource scheduling method, named BRSM, is developed for trade-o s between VM migration cost and energy consumption for edge server management.In terms of the real-time load conditions of edge servers and the number of the

T 1 :
Key terms and descriptions.Terms Descriptions e number of edge servers e set of edge servers, = 1 , 2 , . . ., e -th edge server in e capacity of the -th edge server e number of APs e set of APs, = 1 , 2 , . . ., e -th AP in W e number of mobile services e set of mobile services, = 1 , 2 , . . ., e -th mobile service in e occupied start time of e duration time of e amount of VMs of e packaged VM to perform 2.1.Basic Concepts and De nitions.

F 1 :
An instance of VM migration in ECD-based complex systems.

F 4 :T 2 :
e ow chart of selecting the optimal strategy.Parameter settings.Parameter description Value e number of VMs in each edge server 11 e number of running VMs in each edge server [1, 7] e transmission rate between Aps 540 Mb/s e transmission rate between AP and edge server 1200 Mb/s e duration time of VM [1, 3] e transmission data of VM (G) [0.5, 0.8]

4. 2 . 5 .F 5 :F 6 :
Evaluation on Energy Consumption.As mentioned in section 2, the energy consumption of VM migrations is an important criterion to judge whether the o oading situation of the o oading strategy is suitable.e lower the energy consumption is, the more e cient the o oading strategy becomes.us, the energy consumption of FF, BF, FFD, BFD, Comparison of the utility value of solutions at di erent mobile service scales.Comparison of the number of employed edge servers by FF, BF, FFD, BFD and BRSM.Complexity

F 7 :F 8 :
Comparison of the average resource utilization of edge servers by FF, BF, FFD, BFD and BRSM.Comparison of real-time resource utilization of 5000 mobile services by FF, BF, FFD, BFD and BRSM.

F 10 :
Comparison of energy consumption by FF, BF, FFD, BFD and BRSM.

F 9 :
Comparison of migration cost by FF, BF, FFD, BFD and BRSM.
) min , min .ae M =ae x , ac M =dc, av M =dv, as M = t, ad M = −t A: Allocation record updating.