SDN-Enabled Communication Network Framework for Energy Internet

. To support distributed energy generators and improve energy utilization, energy Internet has attracted global research focus. In China, energy Internet has been proposed as an important issue of government and institutes. However, managing a large amount of distributed generators requires smart, low-latency, reliable, and safe networking infrastructure, which cannot be supported by traditional networks in power grids. In order to design and construct smart and flexible energy Internet, we proposed a software defined network framework with both microgrid cluster level and global grid level designed by a hierarchical manner, which will bringflexibility,efficiency,andreliabilityforpowergridnetworks.Finally,weevaluateandverifytheperformanceofthisframework intermsoflatency,reliability,andsecuritybyboththeoreticalanalysisandreal-worldexperiments.


Introduction
"Smart grid" applies communication and networking technologies into the power grid to allow for more efficient generation, transmission, distribution, and usage of energy [1], which lead to a sustainable energy generation and consumption.With the increasingly serious situation of energy shortage and environment pollution, DG (distributed generator) has been adopted by more and more countries as renewable and green energy resources.However, compared with the development of distributed renewable energy generations and long distance power transmission technologies like Ultrahigh Voltage (UHV), a series of advanced information technologies, such as smart metering and control, data fusion and mining, smart scheduling, and optimization, are also critical for power grid.Besides, the traditional unidirectional power grid turns into bidirectional power grid with the introduction of DGs.Hence, energy Internet brings an exciting prospect of the future energy utilization containing all phases of novel energy generation, storage, transmission, and distribution [2].
Technically, energy Internet brings three aspects of innovation for power grids.Firstly, more distributed renewable energy terminals, such as wind power and solar power, can be allowed to access the energy Internet through ubiquitous networks.They will considerably improve the proportion of renewable energy usage and reduce greenhouse gas emissions.Secondly, smart grid technologies, such as microgrid technologies, can just support effective energy utilization in local scope, while energy Internet brings about more effective energy utilization in both of small and large scopes by smart communication networks.Thirdly, energy Internet is able to protect the power grid from cascading failure.Cascading failure (also known as blackout) is notoriously known as one of the most devastating forces, which usually results in disastrous damage to modern societies.Independence and isolation are two main reasons for cascading failure.And energy Internet could resist the cascading failure in power grid by extensive interconnecting, which will improve the stability of power systems.
Therefore, energy Internet has attracted increasing attention of government and institutions, such as the US

Requirements and Related Works for Communication Networks
Smart grid has been studied for several years in China, and a series of achievements have been obtained for assurance of energy supply.To allow more distributed generators and increase energy efficiency, the current energy system needs to be improved with the aim of evolving to energy Internet.

State-of-the-Art Development of Energy Internet.
In 2009, the State Grid Corporation of China (SGCC) declared that they would start up the strong and smart grid plan in China and they would focus on four working areas, including strong grid, extensive interconnection, high intelligence, and open interaction.In 2010, "the 12th Five-Year Plan" launched by SGCC affirmed achievements on the upgrades of wildly interconnected power grid and transmission system, for example, pharos measurement units (PMUs), online security and stability, large-scale wind turbines, and solar photovoltaics monitoring.However, in the early 2010s, the power market in China was almost in the state of monopoly.To ensure the safety of power systems, power plants were deployed in a centralized way and the power transaction was almost operated in the planned model in which the electricity price was fixed without any adjustments neither in peak hours nor in trough hours.Limitation of the traditional communication network architecture leads to the passive consumption of power energy and low efficiency of energy utilization.However, there was no obvious change occurring in these years for this in spite of the implementation of peak-valley price in minority urban cities.Hence, there is still lots of work to do for energy Internet.
With the further development of the market economy, a series of sweeping reforms are coming in China, which will result in the further openness of energy market, and will allow more distributed energy generators to access power grids.In order to build energy Internet in China, the government encourages distributed power generation and liberalized electricity transaction, which is regarded as the milestone of the energy Internet development in China.

Requirements of Communication Network.
According to the current situation of energy system, Figure 1 illustrates the future vision and system framework of energy Internet in China.
Users are the terminal nodes of energy Internet.Microgrids are small self-management areas in local distribution grids, which can contribute to the utilization of renewable energy resources and improve the energy efficiency by reducing transmission loss.And the bulk power system is mainly composed of power generation systems, Ultrahigh Voltage backbone grids, responsible for the long distance power transmission to relieve the unbalance between power consumption and resource location in different regions, power distribution networks, end-users, and so forth.To guarantee the safety and stability of power grid, energy flows and information flows will be deeply integrated with each other under the support of ICT.
Though ICT has been already involved in traditional power grid to strengthen the intelligence of the power system, the communication capabilities in traditional power grid were very weak, especially in China.Compared with traditional power grid, energy Internet is characterized by the access of massive distributed DGs, highly intelligent control, highly effective energy utilization, flexible energy trading online, and series of novel business scenarios.As DGs have features like small capacity, high quantity, and fluctuant and uneven distribution, the reliable low-latency managements of

Related Work of Communication Network Framework.
Massive DGs will generate large amounts of metering, monitoring, and control data, which brings specific requirements of communication networks.IP-based networks have been considered as the infrastructure of communication networks of energy Internet [6].However, most current researches on energy Internet have just focused on the single issues in energy Internet, such as the SOFIA (Service-Oriented Information Centric Networking), which just focuses on serviceoriented application information transmission [7], and ICN (Information Centric Networking), which just emphasizes machine-to-machine data delivery [12].The grid communication system is divided into several parts in [13], including energy management system, distribution management Fortunately, in recent years, Software Defined Networking (SDN) has attracted much attention.It proposes a flexible, effective, and reliable network framework that abstracts the control plan from the packet forwarding hardware (data plane) to an external software controller [14,15].Therefore, SDN is perceived to have tremendous potential for utilization in the communication network of the energy Internet.

SDN-Enabled Hierarchical Communication Network Framework
As is shown in Figure 2, energy Internet is expected to involve a large number of distributed renewable energy generators  and improve the efficiency of energy utilization with the combination of self-consumption of power in small scope and power transmission in long distance.
As the basic cell of energy Internet, microgrid is a stable and independent microsystem responsible for small scope power consumption.Type I users could directly obtain power energy from Type II users in the same microgrid, if there is any redundant energy produced or stored by them.This small scope self-consumption model can not only improve the utilization efficiency of renewable energy, but also decrease the power transmission loss and cost.When this energy transaction occurs, massive state information, such as load state, generation state, and storage state of users, should be collected in real-time by the microgrid energy controller to strictly guarantee the real-time balance of power supply and demand.Hence, highly frequent interaction of information between communication terminals is typical of microgrid level.Meanwhile, the adjustment of energy supply-demand among adjacent microgrids will take place when it is beyond the self-balance abilities of only one microgrid.When this cross-domain transaction occurs, amounts of information among different microgrids will be generated.
In addition, long distance transmission by UHV (Ultrahigh Voltage) would also be needed in order to realize large scope balance of energy supply and demand, such as the West-East Electricity Transmission Project in China, which is significant for the optimization of energy allocation among different regions.In this case, it is important to insure the reliable transmission of scheduling information, control information, and monitoring information as well as service operation and management in wide area.
In order to enhance the efficiency of communication system, communication genre and information frequency should be concerned specially.Therefore, we propose an SDN-enabled hierarchical framework of communication network, which can be implemented by the following two parts.

Microgrid Cluster Level Communication Network Framework.
As is shown in Figure 3, several adjacent microgrids can constitute a local microgrid cluster.In each microgrid, there are several switches managed by a centralized local domain controller.The local domain Forwarding Information Base (FIB), containing all routing information inside the microgrid, can be established by the local domain controller.By forwarding this FIB message in local domain, the major communication requirements caused by small scope energy self-consumption can be met, which occupies the majority of energy and communication businesses in microgrid.Moreover, one or more switches in each microgrid are selected as edge switches to connect with those in other domains or the aggregation switch controlled by a centralized cluster controller.This cluster controller also manages those local domain controllers within its domain.Thus communication among different microgrids can be realized under the control of cluster controllers.In this cluster level framework, it has been adequately considered that the more frequent message interaction within local domains always occurs compared with that among multidomains, which is a typical feature of energy Internet communication and is usually caused by the energy self-consumption within local domains.This framework can tremendously contribute to the improvement of communication efficiency.

Global Grid Level Communication Network Framework.
As is shown in Figure 4

Characteristics of Proposed Framework.
As is shown in Figure 5, SDN-enabled communication network framework can be divided into four tiers, that is, data plane, control plane, orchestration tier, and application tier.Different tiers are responsible for different functions of the network.By decoupling data plane and control plane on the basis of SDN,  three characteristics of the aforementioned SDN-enabled network framework can be summarized as follows.
Flexibility.SDN allows network administrators to configure network functions flexibly by software.This is done by decoupling control plane from the data plane.Hence, through the north interface of SDN controller, flexible coordination of energy management and communication network management could be achieved.In addition, since the controller has a global view of the whole network topology, it can find the optimal forwarding path and control traffic flow according to different strategies, such as low-latency driven or security driven ones.
Efficiency.Owning to the highly frequent information interaction always occurring inside microgrid, most of message forwarding tasks can be accomplished according to the flow table distributed by the local domain controller.Only if it fails to provide necessary routing information will the message be delivered to the cluster controller.Compared with other SDN-enabled communication network frameworks, such as the flat controller model in [16], the proposed hierarchical framework can reduce communication delay and lessen the burden of cluster controllers effectively.
Reliability.Global control domain consisted of a series of distributed cluster controllers which can survive from the danger of single point failure.In addition, benefiting from the global view, when one or more communication links break down, infrastructure sharing from other links will take place under the control of the controller to achieve fast recovery, by which the reliability of communication network can be exactly strengthened.

Application Cases of SDN-Enabled Communication Network in Energy Internet
With the influence of the "Internet," energy Internet is also expected to be qualified for more complex and flexible businesses, which cannot be realized without the support of prominent communication network.Two most typical cases will be shown as follows, which can also provide credible evidence that SDN-enabled communication network is required in energy Internet.

Virtual Power Plants (VPPs).
At present, power industry is undergoing a period of marketization at home and abroad.
For instance, Chinese government has enacted the correlated policies to promote the marketization of power distribution businesses.It can break the monopoly of SGCC and CSG (China Southern Power Grid) in power transmission and distribution, which will create opportunities for the springing up of new power operators.However, those newborn operators have no enough abilities or no permission to build their own power grid infrastructure.Under this background, the VPP will become an effective approach to relieving their insufficiency of infrastructure.
The south protocol and north protocol can enable more flexible and dynamic configures of network resources.By the north interfaces of SDN-enabled framework, VPPs can logically abstract and integrate distributed utility resources in different microgrids, such as DGs, storage devices, and controllable loads, in order to realize the self-governed management and optimization towards local resources [17] without affecting each other.It can enable different virtual operators to share the same infrastructure but implement their own operations and businesses independently.

Energy E-Commerce. The primary prospection of energy
Internet is to realize energy sharing, which means loads can get supplies from those near DGs with rich power [13].For instance, when there is a pair of users having made agreement on energy trading via a third-party platform, the seller will send its power to the third-party platform in terms of the preceding agreement.Then the buyer can obtain the same amount of electricity from the third-party platform as depicted in Figure 6.This third-party platform driven energy e-commerce is regarded as another prospective scenario under the background of the marketization of power industry.Compared with traditional network, the programmability and openness of SDN can make it easier to achieve the management of network resources and the deployment of new businesses.
In order to illustrate how this SDN-enabled hierarchical architecture can successfully support these open businesses more clearly, we will give a case of intradomain or interdomain energy trading and then briefly analyze the deployment of flow rules and the orchestration of hierarchical controllers in this subsection.As is shown, Figure 7(a) displays the complete network architecture, while Figure 7(b) displays the logical links of data plane in global topology maintained by cluster controllers.And we note that in order to make the diagram more legible there are some unimportant links that have been omitted, such as links between the local controller and the common switch, as the deployment of flow rules and the orchestration of hierarchical controllers in SDN are not our main concern.Here, we will just give a simple analysis.Actually, there are some related excellent studies that have already been done, such as [18][19][20][21][22].
Firstly, as for the local topology, the local controller can periodically construct a LLDP (Link Layer Discovery Protocol) packet for each switch port.If there is any new link between two different switch ports found, a packet-in message will be triggered.Then the local controller receives this message and infers that there is a link existing between the two involved switches and makes a record.In this process, edge switches and interdomain links can also be found.In addition, local controller can take advantage of Dijkstrabased routing algorithms, or other algorithms, to calculate shortest paths [23] and generate the local FIB.As for global topology, to reduce the difficulties of calculating global paths, the size of global topology is required to be limited.Thus instead of providing all details of the local topology in each local domain, the global view only contains edge switches, aggregation switches, interdomain links, and logical intradomain links, as is shown in Figure 7(b).Up to now, there are several multidomain routing algorithms in SDN that have been well researched, including the calculation of logical intradomain routes, such as [24,25].Given the limited space, we will not give more details about these.We note that each local controller just maintains the detailed routing information within its local scope and cannot obtain the views of other microgrids, while all cluster controllers have consistent global logic views.And all what switches should do is to forward messages to the next hop switch or the default switch according to the flow table distributed by controllers.If there is no rule matched, it will ask local controller for route messages.
Secondly, in the real process of message forwarding, when frequent intramicrogrid energy trading occurs, which is the most common event in energy e-commerce, massive local information exchange requirements will be produced at the same time.And all switches are OpenFlow switches and can forward messages according to the flow table sent by the local controller with high speed.If there is a new flow arriving, the source switch will send a packet-in message to the local controller to require a new flow entry.In this process, rare interdomain information will be produced and nearly no access requests to cluster controllers will occur, while when energy trading among several adjacent microgrids in the same microgrid cluster occurs, large amounts of interdomain information exchange requirements will be generated.Once the source local controller infers that the message is an interdomain one and no local rules matched, it will send requests to the cluster controller.And the cluster controller then determines a global path and sends a reply to each local controller whose domain is passed through by the determined path.Each reply contains the information of each domain's ingress and egress switches or ports.Next, local  controller receives the route reply and expands this logical link to its corresponding physical path and installs rules along the involved switches within its scope.Furthermore, when large scope energy transmission occurs, such as interprovince power transmissions based on UHV, which just have 13 lines in China up to now, more than one cluster controller and corresponding aggregation switches will be involved.Because all cluster controllers have consistent global views, the global path can be calculated by the nearest cluster controller instead of asking routing information from all cluster controllers whose domain is passed through.In this process, aggregation switches controlled by cluster controllers directly will take part in the message forwarding among these far-distance microgrids to reduce end-to-end delay.Once there is an In summary, the openness and flexibility of SDN will make it behave better than traditional IP-based network in these complex business scenarios.The hierarchical framework brings higher efficiency and preferable global performance.These are the original motivations for our research on SDN-enabled communication network in energy Internet.

Performance Analysis of SDN-Enabled Communication Network Framework
According to the above analysis, obviously SDN-enabled network will play prominent roles in energy Internet on account of its openness and flexibility.Meanwhile it is necessary to do sufficient investigations on the performance of SDN-enabled network, including availability (low-latency) and reliability.

Availability.
The performance of communication network can be influenced by several factors of bandwidth, latency, channel quality, and so forth.Compared to the Internet, latency issues of communication network play more important roles in the protection and control of power system, especially for the crucial nodes, such as substations, energy control centers, and other vital devices.As is shown in Table 1, IEEE has classified the information types of power systems into several categories.To ensure the security of power systems, private networks, such as TDM + SDH, traditionally transmit higher priority businesses.Although latency issues are not the inherent characteristics of IP-based network, there is the tendency to be all-IP for the communication network in energy Internet owing to its characteristics of businesses, flexibility, and inexpensive cost [26].Fortunately, SDN-enabled communication network framework could reduce the latency of network to make it competent for power systems.Table 2 depicts the latency of an SDN-enabled network with 1 GB/s bandwidth and 200 km long links in the case of transmitting 1250-byte data [11].From the comparison of Tables 1 and 2, it is obvious to find that SDN-enabled networks could meet the latency requirements for either internal substation or external substation of power systems with reasonable strategies.There are two reasons contributing to the reduction of queue and cache latency.Firstly, compared with traditional IP-based networks, SDN-enabled networks greatly improve the speed of flow forwarding on account of the separation of control plane and data plane, which could considerably reduce processing latency.Secondly, with the global view, it is easier for the controller to deploy optimal strategies for congestion control and load balancing, which could greatly reduce the unpredictable latency produced by traffic burst and hence improve the stability and controllability of network latency.All these could be utilized to reduce the latency of communication network.
However, when forwarding new packets in an SDNenabled network, the interactions between switches and controllers would produce extra latency.Especially when massive mice flows occur, frequent requesting to controllers will increase the round-trip time.Fortunately, the hierarchical framework proposed in this paper can greatly ease the burdens of global controllers.Recent work [18] shows that the hierarchical controller mode could lead to more than 90% events being processed locally, which makes the workload of global controllers go down to nearly 1/300 of that in normal OpenFlow networks, and the round-trip time is reduced.
In addition, there are plenty of strategies investigated to reduce the latency of SDN-enabled networks further, such as CheetahFlow [27], DIFANE [28], DevoFlow [29], and FlowShadow [30].Mice flows account for a large portion of the total number of flows usually corresponding to the most important information in power grid, such as protection and control information.CheetahFlow is a forecasting scheme which can install flow entries to the switches proactively on the basis of historical communication records to reduce setup latency.This scheme is expert in reducing extra latency in SDN via forwarding most of the flows in terms of the advanced rules in data plane without asking for the decision of the controllers.When a mess of mice flows occur, the controller would be frequently invoked and the network performance would be apparently impaired.It applies SVM (Support Vector Machine) to identify frequent communication pairs with high accuracy.Then the forwarding rules with the shortest paths between frequent pairs will be installed in advance, which is of great significance in power systems due to the existence of numerous frequent and significant pairs, such as energy controllers and transformers.

Reliability.
Reliability is another challenge of IP-based networks, which is important for power systems.Fortunately, with the development of SDN, various flow optimization and congestion control technologies have been proposed and many simulations of latency for SDN have also been done [31].As depicted above, the distributed global controllers are conductive to avoid single point failures, and SDN-enabled fast recovery from failures also makes great contribution in avoiding congestion and packet loss.In general, the recovery process in communication network could be divided into two steps: the failure detection and network recovery.Firstly, credible failure detection is the basis of the whole recovery process.Multiple efficient protocols and schemes on failure detection have been established, such as BFD (Bidirectional Forwarding Detection) [32] and OpenFlow Fast Failover Group Table [33].The BFD protocol is successfully used in providing fast and low-cost failure detection for higher-plane control protocols, with a speed of millisecond level.The OpenFlow Fast Failover Group Table, supported in OpenFlow switches and independent of the controllers, could be configured to monitor the state of ports and interfaces, as well as switch forwarding actions accordingly.
Secondly, network recovery is usually complex and timeconsuming in both traditional IP-based network and SDNenabled networks.Plenty of work, contributing to the reduction of recovery latency, has been done up to now.Meanwhile multiple SDN-enabled fast recovery schemes have shown great potential.For instance, when failures are detected, the recovery process proposed in [33] could be divided into two steps: the switch-initiated recovery based on preconfigured backup path pairs to guarantee end-to-end connectivity and the controller-initiated recovery based on new optimal paths calculated by controllers.This fast failover scheme realizes the fastest recovery time of 3.3 ms through combining preconfigured backup paths and fast link failure detection in SDN, which shows an enormous improvement compared to present average recovery time varying from 28.2 to 48 ms.

Testbed and Experimental Results
The performance of flexibility brought by SDN is obvious; thus we verify the latency and reliability in this section.In our proposed hierarchical communication network framework, the global grid level communication network is composed of several microgrid cluster level communication networks.As the microgrid is the basic unit of energy Internet, in this section, we consider a single microgrid scenario where information terminals of energy devices communicate with each other through three OpenFlow switches managed by one controller.If the latency and reliability of the communication network are ensured, the performance of communication network in energy Internet could be ensured evidently.Therefore, a testbed based on communication network of microgrid is built and experimental results are presented.

Testbed Implementation.
A typical microgrid is implemented, which contains AC and DC loads, renewable energy resources, and energy storage devices.Meanwhile, based on the physical energy infrastructure, a SDN-enabled communication network which consists of a SDN domain controller, several OpenFlow switches, and a large number of information terminals is also established as shown in Figure 8.In this testbed, SDN controller Floodlight, which could provide a Web GUI for network operator, is implemented on a server which has physical links with switches.The information terminals of energy devices communicate with each other by sending ICMP packets.Three x86 PC modules installed with 64-bit Ubuntu 12.04.3,3.8.0-25-genericLinux Kernel and OpenvSwitch 2.3.0 are used as SDN switches.Each switch is equipped with 2-core 1.8 GHz Intel Celeron 1037U CPU, 2 G RAM, and 6 Intel PCI-E 1000 M network interface cards.To easily configure network parameters, all devices are set in the same network segment 192.168.1.x.When switches and information terminals get connected with controller, the state of network will be displayed on Web GUI.

Experimental Results
(1) Latency.First of all, the mode of OpenvSwitch is set to standalone, which means, without a controller, the switch could forward packets in a traditional way.Thus, as illustrated in Figure 9(a), the average RTT between two information terminals is measured and calculated, respectively, in proposed SDN-enabled framework and traditional communication network framework.If the SDN controller is shut down, the proposed SDN-enabled network framework will turn into traditional network framework.With the proposed framework, it is obvious that packets always have a lower latency less than 1 ms except for the first one.The reason is that the switch does not know how to forward packets by optimal routing path for the first time, until the controller discovers the links among switches and sends flow tables to the switches along the routing path, which introduces extra latency in this discovery and delivery process.Our results also coincide with the measurements in [27,29].Meanwhile, the latency of packets with the traditional network framework is a little bit higher than that with the proposed network framework.It is because each switch acts as a regular MAClearning switch when the controller cannot be contacted.In addition, compared with traditional network framework, the delay jitter of SDN-enabled network framework is lower.controller.When disturbance occurs in a communication link, the packets transmitted via this path will be forwarded by switches according to the flow table delivered from the controller.As depicted in Figure 9(b), when the link between switch 2 (with IP address 192.168.1.20)and switch 3 (with IP address 192.168.1.30)is disturbed, the communication link will recover fast and the RTT increases slightly with packets forwarded by switch 1 (with IP address 192.168.1.10).Such fast recovery mechanism is significant for the reliability of communication network in energy Internet, especially for the transmission of energy control and monitor information.

Conclusion and Future Work
In this paper, we have summarized the state-of-the-art development for energy Internet in China and analyzed the requirements such as flexibility, latency, and reliability for communication networks in energy Internet.Then related work of communication network frameworks in energy Internet is discussed.Furthermore, a novel SDN-enabled hierarchical communication network framework for energy Internet is proposed.At last, performances such as latency and reliability of this novel communication network framework are verified by theoretical analyses and experiments.Actually, this proposed network framework could be applied in not only the power grid of China, but also other countries with hierarchical grid.
In our future work, specific methods and strategies related to security issues in this SDN-based framework will be investigated to support the development of energy Internet.Besides the latency and reliability analyzed above, security is also significant for communication networks in energy Internet.With the vast DG's access, the authentication and data protection for users and equipment become more vital and complex than before.Meanwhile, the security problems in SDN have already attracted widespread attention.SDN-enabled security schemes characterized by openness and intercommunity could make more advantageous performances compared with traditional physical isolation schemes.The centralization of control plane makes it possible to guarantee the global consistent security.The controller can also provide more fine-grained security control via configuring global security approaches from the network view.Hence, the SDN-enabled security schemes would be an effective approach for data protection in energy Internet.
U lt ra h ig h v o lt a g e M u l t i s t e p s u b s t a t i o n Microgrid: B Power distribution network Mic rog rid : A M ic ro g ri d : C (b) Energy Internet

Figure 2 :
Figure 2: Differences between traditional power grid and energy Internet.Type I users (traditional users) represent those that just equal common loads, while Type II users (distributed energy users) denote those that possess abilities of power generation and storage besides consumption).

Figure 4 :
Figure 4: SDN-enabled global grid level communication network framework.

Figure 5 :
Figure 5: SDN-enabled communication network framework in energy Internet.

Figure 6 :
Figure 6: Multiple and flexible businesses based on SDN.
Global view of data plane maintained by cluster controllers

Figure 7 :
Figure 7: A case of SDN-enabled hierarchical architecture in energy e-commerce.