Open-source based testbed for multioperator 4G/5G infrastructure sharing in virtual environments

Fourth-Generation (4G) mobile networks are based on Long-Term Evolution (LTE) technologies and are being deployed worldwide, while research on further evolution towards the Fifth Generation (5G) has been recently initiated. 5G will be featured with advanced network infrastructure sharing capabilities among different operators. Therefore, an open-source implementation of 4G/5G networks with this capability is crucial to enable early research in this area. The main contribution of this paper is the design and implementation of such a 4G/5G open-source testbed to investigate multioperator infrastructure sharing capabilities executed in virtual architectures. The proposed design and implementation enable the virtualization and sharing of some of the components of the LTE architecture. A testbed has been implemented and validated with intensive empirical experiments conducted to validate the suitability of virtualizing LTE components in virtual infrastructures (i.e., infrastructures with multitenancy sharing capabilities). The impact of the proposed technologies can lead to significant saving of both capital and operational costs for mobile telecommunication operators.


Introduction
Mobile telecommunication operators (telcos), with a huge number of subscribers distributed in large-scale geographic areas, require the physical deployment of proprietary and expensive hardware to provide radio network coverage.They also aim to improve bandwidth and quality of experience of their subscribers to gain competitive advantage in the telecommunication market.
Along the last years, telcos have been using different kinds of Radio Access Networks (RANs).GSM (2G) [1] was developed to carry on real-time services with very low data rates.To reach higher data rates, a new access technology was developed, namely, GERAN/UTRAN (3G/UMTS) [2], where packet-switching technologies were introduced.The next standardization step forward has been E-UTRAN (4G/LTE) [3] end-to-end all-IP networks.The LTE RAN is mainly composed of Enhanced Node Bs (eNBs).An eNB can be logically and physically split into two different architectural components: one or more Remote Radio Heads (RRHs) and a Base Band Unit (BBU).This separation helps in reducing capital and operational costs and is known as Cloud-RAN (C-RAN) deployment.
In every step of the evolution of mobile networks, a significant investment has been required by telcos in order to deploy the new generation of mobile networks.This

Background and Related Work
2.1.LTE and C-RAN Architecture.Figure 1 shows an overview of the LTE architecture and its basic components.The RRH is in charge of transforming the signals between the air interface and the optical fiber.The air interface provides User(s) Equipment (UEs) with connectivity to both data and control planes.The BBU is in charge of the management of the different RRHs in order to provide interference management, air spectrum allocation.The Serving Gateway (SGW) and the Mobility Management Entity (MME) are in charge of data and control planes of the mobile networks, respectively, in order to provide UEs with access to the network.The Home Subscriber Server (HSS) is a database to store user information and network information, such as the International Mobile Subscriber Identity (IMSI) or mobile telephone number.The Policy and Charging Rules Function (PCRF) enables telcos to manage bandwidth allocation to users, for example, limiting the data rate for a UE in roaming.The Packet Data Network Gateway (PGW) is in charge of allowing roaming when the users are abroad and of giving an IP address to grant Internet access.Firmin [8] provided a comprehensive description on this topic.
After an exhaustive investigation, it has been identified that there is no open-source software for LTE architectures supporting any type of capability for sharing architectures released.The closest available options are NS-3 and OpenAir-Interface [9].In fact, this research work is reporting a oneyear implementation effort, and by the time this research was carried out, OpenAirInterface was not even yet published, so NS-3 was the only choice.C-RAN deployments are based on the usage of a centralized pool of BBUs serving a number of RRHs geographically distributed.Both architectural elements are connected using the Common Public Radio Interface (CPRI) [10].The centralized base band processing enables very tight coordination between cells to maximize radio capacity.The main advantage is to be able to manage this coordination in a centralized way rather than doing so via the external X2 [11] interfaces between base stations.As a consequence, quicker handovers can be achieved due to the less signaling messaging exchanged, which in turn gives the system more robustness against interference.C-RAN makes the antenna site installation easier and provides footprint reduction.It leads to shorter installation times and lowers renting cost, thus saving both operational and capital costs.A C-RAN architecture is shown in Figure 2 where two telcos are deployed, each one with its own C-RAN deployment.
Figure 2 shows the protocols used in 4G/5G networks: (i) X2 manages handovers when the users are switching to a different BBU.
(ii) CPRI carries the signaling between the RRH and the BBU.
(iii) S1-U is in charge of the implementation of the data path between the BBU and the SGW.
(iv) Gx is in charge of QoS policies.
(v) S1-AP is the protocol for the control of users in the network between the BBU and the MME.
(vi) S6a is used for authentication and authorization of users.
In order to exploit this novel C-RAN concept, a mechanism for sharing resources is critical.It will enable RRHs to share BBUs and telcos to share infrastructures to improve the efficiency in the usage of resources of the LTE architecture.

Existing LTE Sharing Alternatives.
Two main options are available to perform the sharing of LTE resources: passive and active sharing [12].In passive sharing, active coordination between telcos is not needed.Therefore, telcos equipment to share will be masts, power supplies, cabinets, towers, and so forth.Active sharing complements passive sharing with the sharing of active elements on the network such as BBUs.The radio spectrum can or cannot be shared depending on the kind of agreement decided between telcos.Infrastructure sharing in LTE can be achieved by different technologies.Multioperator Radio Access Network (MORAN) [13], Multioperator Core Network (MOCN) [13], and Gateway Core Network (GWCN) [14] are the main options.MORAN allows two or more telcos to share passive elements and may also share spectrum carrier and license, among other spectrum considerations.MOCN architectures typically imply the sharing of C-RAN.GWCN is another approach for sharing components of the LTE architecture where both C-RAN and MME components are shared between telcos.Figure 3 shows the different shared elements for the three approaches mentioned.
There are several studies related to LTE architecture and C-RAN.CloudIQ [15] works with OpenAirInterface focusing on the management of the backhaul LTE for C-RAN.It manages the sharing of homogeneous computations and performs the scheduling of resources from a set of base stations to show the advantages of using a resource pool.Bo et al. [16] carried out a research on the impact of the existing positioning technology in LTE and proposed two approaches based on the reference signal time difference (RSTD) in the Observed Time Difference of Arrival (OTDOA) together with UE receiving time subtracting transmitting time (UE Rx-Tx).Liu et al. [17] proposed locating users based on C-RAN using Time Difference of Arrival (TDOA) of Sounding Reference Signals (SRSs) from different antennas.He et al. [18] worked on the CPRI protocol to achieve a low-latency compression scheme of LTE downlink signal based on a clustering algorithm to reduce the required CPRI transmission bandwidth in C-RAN.Guo et al. [19] also focused on a low-latency base band signal compression algorithm to reduce bandwidth requirements in CPRI.However, neither network simulation/emulation nor software has been used to validate the claimed results, limiting the contributions only to theoretical approximations.Guangjie et al. [20] proposed a C-RAN BBU pool architecture with five key components: front-end RRH, BBU processing pool with cluster structure, control unit, raw data switch network, and traffic data switch network.Challenges such as computation resources and switch bandwidth can be utilized as needed, maximizing the usage of resources.However, again there is not any type of empirical validation based on software prototypes of the proposed work.It is very important in terms of QoS to have a well-balanced network where traffic flows with minimal congestion; hence, a load balance optimization approach has been provided by Jia et al. [21] using network flows in combination with the proposed novel algorithm.However, again the main drawback is that there is no empirical validation of the results since all the studies have been carried out based on theoretical formulas.There are some contributions focused on virtualized environments.For example, Chen et al. [22] investigated virtual resource allocation in small cell networks, developing an algorithm to face signaling overhead, outdated dynamics information, and scalability issues efficiently.He et al. [23] investigated the usage of Licensed Shared Access (LSA) on the downlink cell edge along with the Fractional Frequency Reuse scheme for resource allocation and analyzed the average capacity gain for bandwidth ratio.

Design and Implementation
The design and implementation of the architecture proposed enable MOCN LTE sharing capabilities over both physical and virtual infrastructures.The basic prototype of the components available in the LTE infrastructure has been reused from the LENA project of the NS-3.Mainly, both SGW and BBU components have been significantly extended to allow support for MOCN LTE sharing capabilities.Among others, a new admission control algorithm has been designed and implemented; a multiconnectivity interface to allow multiple operators assigned to the same SGW and a novel LTE-based routing protocol has been designed and implemented to determine which data path needs to be selected according to the telco the UEs belong to.These extensions are described in Sections 3.2 and 3.3.

Design Principles.
Figure 4 shows the proposed LTE MOCN sharing architecture, illustrated with the deployment of two different telcos.In the figure, telcos are presented with their respective complete network elements.It can be concluded that there are protocols that have to be enhanced in order to provide the MOCN LTE sharing capability.In fact, some extensions to the standard LTE architecture have been proposed, mainly in both BBU control and user planes.

BBU Control Plane Extension for Sharing Capabilities.
In the control plane, the S1-AP interface has been extended to provide BBUs with an MME selection protocol to determine where the MME is associated with the UE that plans to be attached to a BBU.To achieve this capability, when a new UE  This can lead to a security concern due to the fact that a UE belonging to Telco A is connected to the same (shared) BBU of Telco B and this UE could try to access the other telco that it does not belong to (i.e., Telco B).Therefore, an extended admission control mechanism has been designed and implemented in order to make sure that only authorized UEs are allowed to be attached to the BBU they belong to.

BBU Data Plane Extension for Sharing Capabilities.
In the user plane, the S1-U interface has been extended with a new switching protocol that helps the BBU to select the SGW used to deal with both uplink and downlink communications between UE and Internet (external servers).Every shared BBU has to deal with several S1-U connections.To manage the different connections, smart handling of the Tunnel Endpoint Identifier (TEID) has been used.This TEID is available in the GPRS Tunneling Protocol (GTP) established between BBU and SGW.TEID is a unique ID, which allows identifying a tunnel for a given UE.To handle many connections to different telcos, BBU has to be equipped with a switching mechanism.We hence propose to embed both MCC and MNC inside of the TEID field.This modification allows using this information in the data path to perform MCC/MNCbased switching of the data to the right S1-U connection.

Implementation.
To achieve compatibility with standards, several changes have been made over the data plane and control plane (S1-U and S1-AP) in the LENA software [25], an existing LTE-emulator tool that has already been integrated and distributed in NS-3.LENA has sufficient accuracy in the implementation of the LTE stack to run in the emulation mode.Our main contribution has been focused on extending LENA to provide MOCN capabilities and to validate it in virtual infrastructures.All the innovations proposed in this contribution have been fully implemented and validated.The LENA source code has been modified to create a release with MOCN capabilities in virtualized environments.The source code is available at https://sourceforge.net/p/aslte/ in order to enable the reader to reproduce all the experiments available herein.We have achieved keeping the level of accuracy enough to enable the emulation mode in our prototype (i.e., interacting with real LTE devices).The code has been significantly extended to provide all the functionalities claimed in this paper.It is worth mentioning that there is a helper class in the LENA code (point-to-point-epc-helper) that has been redesigned to do all the setting up to achieve the deployment of the communication between different machines and different telcos within the shared virtualized infrastructure.This is the entry point for the users of our LTE architecture with the MOCN capabilities.
Figure 5 shows an excerpt of the relevant NS-3 code, which has been extended to include such MME selection capabilities.
The different extensions made through the code to achieve the MOCN implementation can be summarized as follows: (i) S1-AP in control plane has been extended to allow the BBU to deal with more than one MME connection.
(ii) S1-U in data plane has been extended to perform switching to the SGW that the UE belongs to.
(iii) Different LENA classes have been modified, and the most important one is not only point-to-point-epchelper but also every other class that has a dependency on this former one.

Empirical Validation
This section describes the empirical analysis carried out over the proposed and prototype LTE MOCN sharing architecture to analyze the impact of virtualization on the performance of LTE MOCN.The idea is to analyze the suitability of this architecture for virtual infrastructures by providing a complete set of empirical results based on both physical and virtual infrastructures in order to understand how the novel data center workload affects the sharing capabilities of LTE architectures.The testbed description is presented in Section 4.1, followed by Section 4.2 where the scenarios executed are described.The metrics used for the analysis of the performance are discussed in Section 4.3, and then the evaluation results are presented and analyzed in Section 4.4.

Testbed Description.
Figure 6 shows the setup of the testbed used to carry out the empirical analysis.Two Super-Micro FT4-E2616 servers, with 1x Xeon E5-2630V2, 6 cores,   As an example, a scenario defined as UEs = 256, eNBs = 32, telcos = 2, and sending packet = 1 ms means that the scenario includes 256 UEs and 32 antennas.Each antenna has 8 UEs connected, which are those closer to the antenna.The UEs are stationary.There are two telcos (i.e., 2 MME, 2 SGW, and 2 PGW).Each antenna is connected by default to all the operators of the scenario.Moreover, the assignment of each of the UEs to a given telco is completely balanced.In the example, 128 UEs will belong to Telco 1 and the other 128 will belong to Telco 2. Each antenna will have four UEs from Telco 1 and four UEs from Telco 2. The workload is generated in both ways; that is, there is a UDP flow from UE to the remote host and another one from the remote host to UE.Each flow generates packets at a constant rate of 1 packet/ms.All the experiments were executed for two seconds.This means that, in this example, each UE will send 1000 packets/sec for two seconds.Each UDP packet has a Wireless Communications and Mobile Computing payload of 64 Kbytes; this size is always the same for all the experiments.The idea behind the experiments is to send packets from UEs to the remote host while exponentially varying the values of the different parameters saturate the network.The main purpose is to see how the increasing of these parameters as well as the sharing rate affects the overall performance.We have used three network saturation levels to analyze the behavior of the proposed MOCN architecture at different levels of saturation.Real-time emulation mode using TAP [28] interfaces and checksum computations has been enabled to take a higher level of accuracy for the results.

Evaluation Metrics.
Two metrics have been used for the analysis of the architecture: average UE transmission time and average throughput.The UE transmission time is defined as the time it takes for one UE to send all the packets in the scenario to the remote host and for the packets to be received by the destination (i.e., the end-to-end delay).The average UE transmission time is the average delay calculated for all the UEs in the executed test scenario.The individual throughput is calculated at the remote host UDP daemon for each of the UEs representing the number of bits received from that UE per second.It is calculated using the following formula: The average throughput is an averaged value of the calculated individual throughput per UE.

Evaluation Results.
Given the large number of executed scenarios (i.e., 576), it is impossible to render all of them in this paper.In addition, the analysis revealed that many of these scenarios exhibit a similar behavior.This section will, therefore, show the most interesting scenarios to illustrate relevant findings.These are summarized in Figure 7.
Figure 7 shows four different subfigures.The subfigures of Figure 7 ((b) and (d)) represent the averaged throughput achieved by the UE for a nonsaturated scenario (i.e., interleaving packet rate 100 ms), while subfigures (a) and (c) represent a midsaturated scenario with interleaving packet rate of 10 ms. Figure 7 has a fixed number of UEs of 64, 128, and 256 to analyze large deployments with the effect of ranging the number of BBUs and telcos sharing these BBUs.This is why there are different three-dimensional surfaces plotted in the subfigures.It is noted that Figures 7(a) and 7(b) are scenarios deployed in physical environments while Figures 7(c) and 7(d) are executions in virtual environments.This layout in Figure 7 can be used to analyze the impact of virtualization by comparing top-down figures and the impact of stressed workload by comparing the left-right figures.The Maximum Nominal Bandwidth for both nonsaturated and midsaturated scenarios is (10 or 100 packets/sec * 10560 packet size (bits) *  UEs) 27 Mbits/s and 270 Mbits/s, respectively.The most stressed scenario with 1 ms of interleaving packet (2.7 Gbits/s) leads to a high number of packet drops due to the fact that the hardware cannot handle this level of saturation in real time; this is why the figure has not been shown.Different axes in Figures 7(a)-7(d) represent, respectively, the number of telcos (PLMNs) sharing the BBUs deployed in the scenario, the number of BBUs deployed, and the average throughput received in the remote host.Firstly, it is noticed how Figure 7(a) (physical machine to physical machine) has a high peak up to 900 kbits/s whereas Figure 7(c) (VM to VM) has a high peak up to 600 kbits/s.Secondly, it can be observed that the most relevant aspect when comparing all the surfaces plotted in each of the figures is that the system is always stable up to ∼300 kbits/s regardless of the number of telcos sharing the BBUs.At higher rates, packet loss starts to increase significantly.Therefore, it can be considered as the maximum speedy throughput the system can handle.The high peak at 900 kbits/s per user (i.e., over 57 Mbits/s) is the empirical maximum validation due to limitations of the hardware used.It is noted that 256 full LTE stacks are being emulated in real time while sending and receiving traffic simultaneously.Fontes et al. [29] were able to reach a high peak of almost 20 Mbits/s due to the same reason, although in our case we are working with a full emulation of the physical layer radio air interface whereas they were only using Optimized Link State Routing (OSLR) [30] protocols.In the nonsaturated scenario, that is, Figures 7(b) and 7(d), the results show as well how the system becomes stable at ∼ 100 kbits/s (i.e., over 20 Mbits/s), which is the expected result for this workload with almost no packet drops.
Thirdly, it is noted how the throughput is decreasing when the number of BBUs is increased until they become stable.This is due to the fact that the emulation needs to handle more virtual NICs (and more complex scenarios) and thus the performance is being decreased.The results of the overhead associated with the execution of VMs for the midsaturated scenario (10 ms interleaving packet) are shown in Table 3 while the results for the nonsaturated scenario are shown in Table 4.
The overhead in the average of the performance associated with the execution of the MOCN LTE infrastructure in VMs could be set up in almost all the 576 scenarios analyzed close to 10%.This fact validates empirically the efficiency and feasibility of the proposed MOCN architecture running in virtual infrastructures.Another interesting aspect to analyze is how packets are lost when NS-3 interacts with real NIC interfaces in comparison with NS-3 interacting with virtual NICs.Table 5 shows that, in all the cases, the difference of packet losses between physical-to-physical machine communication and virtual-to-virtual machine communication is 0%, which is also a good result of the suitability of running emulated MOCN LTE in virtual infrastructures.
Finally, it has been observed how the number of operators sharing the LTE infrastructure almost does not affect the performance of the network.This is a clear insight into the

Conclusion
This paper has provided the first open-source design and implementation of an LTE architecture with MOCN infrastructure sharing capabilities and has validated its feasibility in virtual infrastructures.The design includes several extensions in both control plane and data plane to the protocols S1-AP and S1-U, respectively.The suitability of the implemented testbed has successfully demonstrated running such software in both physical and virtual infrastructures.The architecture has been validated by an extensive set of experiments.Scalability, overhead, and network saturation have been analyzed, providing very promising results.As to future work, there are some improvements to be made related to the LENA software.For example, it would be interesting to have an implementation of CPRI in LENA to be able to perform some tests in the future.Adding support to interact with real LTE hardware (antennas, BBU, and RRH) would be also a natural step.
The porting of these MOCN capabilities to the opensource software OpenAirInterface would be another future step to allow implementations with hardware air interfaces.It is also expected to design and implement other alternatives for sharing infrastructures in LTE, for example, the GWCN mode proposed by 3GPP.

Figure 3 :
Figure 3: Overview of different LTE sharing architectures.
Physical machine to physical machine communicationAverage throughput per UE (kbps) Figure 7: (a) 10 ms interleaving scenario running in physical machines.(b) 100 ms interleaving scenario running in physical machines.(c) 10 ms interleaving scenario running in virtual machines.(d) 100 ms interleaving scenario running in virtual machines.
[24] and Kadoch[24]investigated the Shared Commercial Radio with offloading, for Public Safety Network (PSN) over LTE Heterogeneous Networks (HetNets).They proposed a solution where the PSN users have access to the commercial radio network resources.Despite the existing published work in this field, it is noted that only a scarce number of publications contain any kind of empirical validation with prototypes and go beyond traditional theoretical hypothesis.
In fact, almost no validated results have been found even with more than 100 different references in the area analyzed.This is where this contribution provides a key differentiating point with respect to other work, by delivering an LTE architecture with MOCN capabilities suitable for virtual infrastructures and empirically validated by a prototype.
(and its VM B? associated) is used to emulate a remote host on the Internet.It is running a custom-made UDP daemon that accepts packets from the UEs and collects statistics (e.g., it obtains times and sizes of the packets received) to enable performance analysis.UEs are in Computer A as part of the LTE architecture and are communicating with the remote host in Computer B. There is one remote host per SGW (i.e., per telco).The usage of both physical and virtual machines with the same software deployment enables the analysis of four different testing environments as indicated in

Table 2 :
Fixed parameters used in NS-3.All UEs are geographically located in the same vertical plane.The distance between each UE is defined by this parameter.All eNBs are located 1 meter far in the vertical plane and all of them are equally spaced between themselves and also between UEs.
Maximum data rate for download in LTE interface.Upload link bandwidth 100 MBs Maximum data rate for upload in LTE interface.

Table 3 :
Throughput overhead for the saturated scenario (10 ms of interleaving packet rate).

Table 4 :
Throughput overhead for the nonsaturated scenario (100 ms of interleaving packet rate).
experiments.It is plotted in Line A where eight telcos sharing the same BBU provide the best performance in saturated scenarios.It is noted that surfaces represent scenarios with different numbers of UEs and the trend in the behavior of Line A is similar to all the surfaces.For midsaturated scenarios, depicted in Figures7(b) and 7(d), 16 telcos sharing

Table 5 :
Overhead in terms of packet loss.thesameBBUshow the best results for all the scenarios analyzed.These two values provide a clear hint on the range of an acceptable sharing factor between BBUs and telcos.The analysis of Line B unveils the best number of BBUs set up when the best sharing factor is assumed.There are eight BBUs and 16 BBUs for the saturated and midsaturated scenarios, respectively.Points P1 and P2 are the combinations showing the best throughput achieved (685.25 and 548.2 averaged kbits/s per UE).In a saturated scenario, Figures7(a) and 7(c) show a similar shape regardless of the number of users as Lines C, D, and E show that the throughput is increasing until the system reaches eight BBUs, and from that point, the system tends to stabilize regardless of the number of telcos.In a midsaturated scenario, Figures7(b) and 7(d) plot P3 and P4 which are the combinations showing the best throughput achieved (84.8 kbits/s).Lines C, D, and E in Figures7(b) and 7(d) show similar behaviors for different numbers of users and how the proposed architecture tends to stabilize after 16 BBUs have been reached regardless of the number of telcos sharing the BBUs.This analysis demonstrates the benefit of sharing BBUs between different telcos and provides empirical insights into ranging values to be used to achieve good performance settings.