Experimental Performance Evaluation of Multihop IEEE 802.15.4/4g/4e Smart Utility Networks in Outdoor Environment

This paper presents the experimental performance evaluation results of the IEEE 802.15.4/4g/4e Smart Utility Networks (SUN) in applications suited for outdoor environment. SUN is an advanced wireless communications network designed for reliable, low data rate, and low energy consumption networks for command-and-control applications like utility service, sensor network, and so on. IEEE 802.15.4g/4e is the international standard for SUN supported by multiple utility providers and product vendors. In this paper, a comprehensive field test was conducted by employing the implementation we have developed to evaluate the performance of the SUN devices based on IEEE 802.15.4/4g/4e standard. The output power of the implementation is 250mW for extended range, reducible to 20mW for short-range scalability and battery preservation. Results showed that in an outdoor line-of-sight environment, the achievable one-hop range of a 50 kbps SUN device was 450m. Next, in a non-line-of-sight environment involving typical residential concrete building, the communications could be established penetrating obstructions to reach above the 11th storey, reaching the performance degradation limits at the 20th storey. Next, the network of the SUN system was proven to be capable of supporting a typical multihop tree network in a dense populated building, meeting the required performance by the standard.


Wireless Communications and Mobile Computing
By using the developed prototype, field trials were conducted to evaluate the performance of the SUN system. The tests were conducted in Singapore, covering multiple scenarios that represent the typical environment most wireless communication networks may encounter. These scenarios involve open air line-of-sight (LOS) achievable range, outdoor nonline-of-sight (NLOS) achievable range, dense urban area multihop networks establishment, and range extension for suburban networks. The test results of the field trial were gathered and analyzed to determine if the SUN system is capable of meeting the requirements set forth by the use case scenarios in Section 3. The test deployment scenarios and network performance evaluation are described in Sections 6 and 7, respectively. Finally, detailed discussions relating the use case scenarios and the evaluation results are presented in Section 8.
One of the main challenges in this project is to enable the node-to-node communications to overcome different buildings and geographical terrains. As shown in [1], the SUN is primarily a field area network operating in outdoor scenarios, connecting the wide area network to the premises. In outdoor scenarios, there are bound to be natural and manmade obstacles which may compromise the effectiveness of the network. In these scenarios, a multihop system operating in the sub-GHz band is very efficient in increasing the efficiency and robustness of the network.
To the best knowledge of the authors, the evaluation of IEEE 802 standards SUN system, based on simulation and theoretical means, is minimal, let alone evaluation based on realistic field trials. Hence, the contribution of this paper is threefold: (a) provided system design and prototype implementation for SUN device; (b) conducted field tests for multiple outdoor test case scenarios; (c) verified the test results relating to practical SUN deployment in the real world. It is our strong belief that this paper should provide a viable reference for both the research and industrial community.

Related Works
This section lists the related studies on system design, performance evaluation, and interference management in the SUN area. Documents in [1,9] show the interrelated domains of smart grid/utility and outlining the communications entity in the field area network for SUN. In [2,10], we have recorded extensive and detailed performance evaluation on both the IEEE 802. 15.4g frequency shift keying (FSK) and orthogonal frequency division multiplexing (OFDM) PHY layers. Optimization and enhancement on system design were studied in [11][12][13][14]. In [11], energy efficient mechanism for SUN was explored to prolong battery lifespan, while, in [12,13], low duty systems and beaconless synchronization methods were investigated for the same reason. Effectiveness of a wireless mesh network is discussed in [15,16]. Besides system design, we have also conducted numerous studies on the impact of interference. In [17], the achievable communication range and tolerable interference range were investigated. In [18], the impact of homogeneous and heterogeneous interferer on SUN systems was evaluated. Then in [19,20], we have proposed methods to manage and mitigate the interference from different sources. To further our understanding of the SUN system behavior, besides theoretical and simulation works mentioned, we have diversified our studies into field trials with actual implementation, as recorded in this paper. Besides our field trial, there was also another implementation recorded in [21][22][23].

Smart Utility Networks Usage Model
SUN is the next-generation utility network equipped with communication technologies facilitating enhancements to optimize the consumption of resources. Examples of applications are Advanced Metering Infrastructure (AMI), Demand Response (DR), Distribution Automation (DA), and others, developed to enable efficient resource management [24,25]. Targeted utility services for these applications are typical, but not limited to power, water, natural gas, and sewage utility systems. By enabling two-way communications between the service provider and end-user terminals in these utility networks, advanced command-and-control features can be applied to optimize the resources while maintaining minimum cost, ultimately benefiting both utility providers and utility consumers.
A typical usage model of the Smart Utility Network (SUN) is illustrated in Figure 1. On the utility consumers' end, utility meters are equipped with communication modules that form a wireless network. The network is commonly deployed for meters and other sensors in residential areas, office, and industrial sites. In these areas, the network should be sufficient to provide coverage to different scenarios of building structure and distribution. In multistorey buildings for residential and office purposes, the network is expected to overcome the attenuation due to the walls in between nodes. At the ground level, the network should also be able to cover the distance across nodes in both line-ofsight (LOS) and non-line-of-sight (NLOS) environments. For nodes located at extended distances, multihop operations can be employed to provide additional coverage, particularly into suburban regions where the population is relatively sparse. The aforementioned network is the SUN network that facilitates communication among several thousands of nodes to exchange information. The SUN network is then extended all the way to the utility collection/control station via wide area network (WAN). The WAN may be wired or connected through wireless channel, depending on different system demands.
In order to provide the service described, SUN should possess capabilities to support the following use cases: (i) Use case 1: provide coverage that supports the typical distance among nodes deployed in LOS facilities on the ground. (ii) Use case 2: provide coverage that overcomes penetration of obstructions among NLOS entities in multiple-storey buildings. (iii) Use case 3: form a network that covers the field or neighborhood area networking system connecting multiple nodes in a dense environment. A radio terminal More than 10-year operation driven by the data is collected via low-power mesh/tree relaying among the radio devices battery, providing 150 m radio range. Meter (iv) Use case 4: form an extended network that expands the effective range to multiples of order of a one-hop signal into areas with sparse population.
These four use cases can be combined and derive a generic usage model for an outdoor networking system in an urban area with primarily buildings and roads. This matches well with the deployment scenario in a typical urban state, such as Singapore.  15.4e intends to support a wide range of industrial and commercial applications that require lower latency, higher robustness, and deterministic behaviors. To achieve this, 802.15.4e has specified a number of mechanisms targeting different application domains with different features [7]. On the other hand, IEEE 802.15.4e has also developed additional MAC functions to enhance the general capabilities, like low energy (LE), information element (IE), enhanced beacons (EB) and enhanced beacon requests (EBR), and so forth. This section introduces selected 802.15.4e MAC features that has been implemented in the test-bed devices.

4.1.
Regulation. The field trial was carried out in Singapore so the developed SUN devices must comply with the local shortrange device regulation which is in [26]. Three possible frequency bands in Singapore are allocated for applications like smart metering, that is, 866-869 MHz, 920-925 MHz, and 2.4000-2.4835 GHz. The maximum allowed output powers in the three bands are 10 mW ERP (Effective Radiated Power), 500 mW ERP, and 100 mW EIRP (Equivalent Isotropic Radiated Power), respectively.   Additionally, the spectrum mask of the data rate modes can be found in [6].

Data Rates.
The MR-FSK PHY supports multiple data rates for different frequency bands/countries as listed in Table 1, where the data rates with star marker are mandatory and others are optional. According to the SRD operating frequency band (920-925 MHz) in Singapore, the supported MR-FSK data rates are 50 kbps, 100 kbps, 150 kbps, 200 kbps, and 400 kbps in Singapore. Figure 2, one for MR-FSK without mode switch, the other for MR-FSK with mode switch. Mode switch is a mechanism by which MR-FSK PHY changes symbol rate and/or modulation scheme on a packet-by-packet basis. The PPDU format of MR-FSK PHY without mode switch is mandatory while the other is optional.

PPDU Format. MR-FSK PHY has two PPDU formats as shown in
The first part of a PPDU is synchronization header (SHR) which includes Preamble field and Start of Frame Delimiter (SFD) field. A Preamble field contains multiple repeated bit sequence of "01010101" for filtered 2-FSK and multiple repeated bit sequence of "0111 0111 0111 0111" for filtered 4-FSK. An SFD field contains a 16-bit sequence for filtered 2-FSK and a 32-bit sequence for filtered 4-FSK. The value of an SFD bit sequence indicates whether the PHY header and the PHY payload are coded or not.
The second part of a PPDU is PHY header (PHR) and it is different in the PPDU with mode switch and the PPDU without mode switch. The mode switch field indicates whether the PPDU is of mode switch or not. When it is zero, the PPDU is one without mode switch and the whole packet is transmitted at a single data rate and single modulation scheme. When it is one, the PPDU is one with mode switch and contains no PHY payload.
The FCS Type field indicates the length of the Frame Check Sequence (FCS). In the 802.15.4 baseline standard, the maximum packet length is of 127 bytes and the FCS length is of 2 bytes. In order to support IP applications in a better way, IEEE 802.15.4g extends the maximum packet length to 2047 bytes, and, correspondingly, adopts a 4-byte FCS as mandatory while keeping 2-byte FCS as optional. When value of an FCS Type field is zero, the FCS length is 4 bytes, and when it is one the FCS length is 2 bytes.
The data whitening (DW) field indicates whether data whitening mechanism is applied. Data whitening is optional in MR-FSK PHY.

Channel
Coding. Forward error correction (FEC) channel coding is optional and determined by the SFD value in MR-FSK PHY. It can be either systematic or nonsystematic encoder with a 1/2-rate and constraint length of four. Interleaving of code-bit must be implemented when the convolution code is nonsystematic; however, it is optional when the code is systematic.

Receiver Sensitivity.
Receiver sensitivity is the lowest input power for which the PER requirements are met. The 802.15.4 baseline standard requires a PER < 1% when one measures receiver sensitivity; however, considering the more challenging environment of neighborhood area network and much longer packet length, IEEE 802.15.4g requires a PER < 10% when receiver sensitivity is measured. Other measurement conditions include a PSDU length of 250 bytes and a  data rate of 50 kbps or higher. Equation (1) is used to calculate the required receiver sensitivity of a device employing MR-FSK PHY.
where 0 is −91 without FEC and −97 with FEC, 0 is 50 kbps, and is the bit rate in kbps. For data rate of 200 kbps, the receiver sensitivity is −85 dBm without channel coding, while it can be improved to −91 dBm when channel coding is used. The practical value of receiver sensitivity in this implementation is around −100 dBm, depending on the specific data rate mode. This is a value produced from repeated testing in the laboratory and field.  Figure 3. Two types of IEs are employed by IEEE 802. 15.4e, that is, header IEs and payload IEs. The former is part of the MAC header (MHR) and is used to process the frame itself, while the latter is a part of MAC payload dealing with next higher layer or service access point (SAP). IEEE 802.15.4g defined a SUN Device Capabilities IE for exchanging PHY capabilities of data rates, modulation scheme, operating frequencies, and so forth, among SUN devices.

Low-Energy Superframe.
To meet low-energy consumption requirement of applications such as SUN, three mechanisms have been defined in 802. 15.4e, that is, lowenergy (LE) superframe, coordinated sampled listening (CSL), and receiver initiated transmission (RIT). The lowenergy (LE) mechanisms are not specific to any particular application domain; they are generic for applications that trade low latency for low-energy consumption. The LE protocol allows devices to operate down to a fraction of 1% duty cycles while presenting an "always-on" illusion.
Low-energy superframe saves energy consumption of a SUN device by turning off most of or all of the periodic beacon transmissions and by extending inactive periods. When all beacon transmissions are turned off, a coordinator is still capable of transmitting beacons when required to do so. Figure 4 shows comparison between the legacy IEEE 802.15.4 superframe and the IEEE 802.15.4e LE superframe. A coordinator of the former transmits one beacon every beacon interval (BI), while, for the latter, it transmits a beacon every BIs, where can be specified when the network is set up. Moreover, the inactive period is extended by removing the contention-free period (CFP); therefore a SUN device can "sleep" longer to save more energy. In active periods, all SUN devices are awake and monitor data frames transmissions. When a data frame transmission is directed to a SUN device,  it enables its receiver even during the inactive period, while other SUN devices switch to an inactive state. Data frame transmission shall begin within an active period and be completed before the next active period.

EB and EBR.
The 802.15.4e enhanced beacon (EB) is extension of beacons of the baseline standard. An EB is constructed by specific IEs for targeted applications. The EB enables the application related information from higher layers to be periodically broadcast.
The 802.15.4e enhanced beacon request (EBR) is extension of beacon request command frame of the baseline standard. It specifies contents of a number of response filter IEs, by which only the neighbors meeting the filtering conditions are requested to response or only the wanted information is requested to be included in the response EBs. By achieving this, an 802.15.4e device may exchange selected MAC or PHY capability information with selected neighbors. This is beneficial for devices to establish communication with neighbor devices with specific MAC or PHY capabilities. Figure 5 shows a developed prototype device of a SUN based on IEEE 802.15.4g and 802.15.4e standards [8]. Parameters of the device are listed in Table 2. The operation band is the Japanese band of 920-928 MHz. The transmission power is 20 mW and is expandable to 250 mW, which is the allowed maximum transmission power in this operation band for license-exempt devices in Japan. The modulation scheme is 2GFSK so the implementation cost could be relatively low. The operation data rates are 50 kbps, 100 kbps, and 200 kbps, which can be configured in a flexible manner. It supports data packet with size up to 2047 octets. Using 2047 bytes packets enables most reading and applications to work with relatively long packet sizes without requiring packet fragmentation. Adaptive forward error correction (FEC) using a systematic code is implemented so that data transmission in a relatively harsh environment can be supported. At the MAC layer, both beacon and nonbeacon modes are supported and the beacon interval, active period, and inactive period can be configured. The SUN prototype device can be powered by three AA batteries or by AC power. The size of the end product is in the order of several-centimeters-cube. During the active period, the consumption current is 50 mA and in the inactive sleep state the consumption current is 2 mA. The interfaces of the prototype device include RS-232C, RS-485, U-LINE, and are expandable to others. The prototype is capable of multihop transmissions with the support of both tree and mesh topology. The developed prototype SUN device meets spectrum mask requirement of license-exempt operation in 920-928 MHz band in Japan.

Field Trial Test Scenarios
By employing the implementation described in Section 5, field trial tests were conducted in outdoor environments. The outdoor test scenarios are as follows:  (iii) Multiple-node network test along typical building corridor.
(iv) Multiple-node network test across different buildings.
The tested data rate modes were 50, 100, and 200 kbps. The transmission power was set to 20 mW.

Line-of-Sight Communication Range
Tests. This test was located in East Coast Park Singapore along a straight road and parallel to the coastline. The reason to select this location was to enable a line-of-sight communication between the devices. A data collector (DC) and a node were deployed from each other in line-of-sight in several distances. Both devices were placed 1 meter above the ground. The deployment scenario is shown in Figure 6(a), where DC-N is the distance between the DC to the node(s). The packet length was set to 250 octets with transmission interval of 5 s. Data collection was conducted for data rate modes 50 kbps, 100 kbps, and 200 kbps. For each data rate mode, several sample points for distance DC-N were taken and communication was established between the DC and the node. Next, data was collected into a laptop with a data logger and processing software. Based on the data, the achievable communication range, above which the DC was not able to receive any packets, was determined.

Non-Line-of-Sight Communication Range
Tests. This test was setup at Casa Clementi Housing Estate in Singapore. The housing estate consists of multiple blocks as high as 20 storeys. This test site provides a typical housing estate in Singapore with typical building structures and environment. The deployment scenario is shown in Figure 6  For each data rate mode, two sample points for the location of DC were taken and the network was established.

Tree Topology Network along
Corridor. This test was setup at Ayer Rajah Industrial Estate in Singapore. This area is filled with multiple-storey buildings for office and factory usage. The environment is a typical industrial estate in Singapore. The illustration of the deployment scenario can be found in Figure 7(a). The network was a tree topology with where DC is connected to N1 and N2, while N1 was connected to N3 and N4 and and N2 to N5. This formed a two-path tree topology network. For the 100 kbps tests, DC-N1 and DC-N2 were both 45 m, while for the 200 kbps tests, DC-N1 and DC-N2 were 16 m and 30 m, respectively.

Interbuilding Line Topology
Network. This test was setup at Ayer Rajah Industrial Estate in Singapore, across two building blocks 67 (level 4) and 71 (level 7), as shown in Figure 7(b). The deployment scenario was a line network topology with the following connectivity: N1-N5-N2-N6-N3-N7-N4-N8. There was obstruction in between the blocks by trees. The DC was located nearby N1 to N4 in block 71. Here, DC-N was approximately 100 m.

Performance Evaluation
Deploying the test scenario described in Section 6, data collection was conducted and analyzed. The following presents the results of the measurement activities. An important performance metric is the packet error rate (PER), expressed as the fraction of error packets to the total received packets in the receiver. It is worth noting that the required PER as per [6] is 10%. This value will be used as the reference in evaluating the performance throughout this study. Table 3 presents the measured data in the LOS communication range tests. The data consists of data rate modes 50 kbps, 100 kbps, and 200 kbps.

Performance in LOS Communication Range Tests.
A general observation was that, although [6] specified PER = 10% as the critical PER rate, in the field tests, it is difficult to obtain the distance whereby the PER becomes 10%. Instead, the common occurrence was that, above a certain distance (i.e., the achievable communication range), PER quickly increased from 0% significantly. From the tests, the achievable communication ranges for 50 kbps, 100 kbps, and 200 kbps were 450 m, 240 m, and 260 m, respectively. This is due to the fact that at a fixed power of 20 mW, higher data rates result in less energy per transmitted bit, thus degrading the performance. From Table 3, it was understood that PER increased as DC-N increased. This can be explained that more packets are received in error with the longer propagation path due to reduction power density over distance. Theoretically, a 20 mW device should be able to achieve the range up to 1 km. In this test, we explain that the shorter achievable range in LOS environment is due to the fact that it is not a perfect LOS. There are still obstructions and reflectors impeding the achievable range of the device.
For data rate mode 200 kbps, two test scenarios were employed, one along the road and one at the beach, both locations are LOS environment. PER for the link along the road was 19% while that at the beach was 2.2%. Considerable factors are the larger volume of signal noise created by vehicles on the road being more significant that the human traffic at the beach. Table 4 presents the measured data in the NLOS communication range tests. All parameter settings of the DC and node were the same. The location was setup at a typical housing estate in Singapore with flats up to 20 storeys. The DC was located placed at level 11 and level 20, with N1 placed at ground floor across an adjacent building. Additionally, N2 was placed connected to N1 to form a star topology network. The materials of construction for the buildings are cement and bricks.

Performance in NLOS Communication Range Tests.
The data was collected corresponding to different locations of the DC at different storeys. It was found that, at  Figure 6(b). In the same row, the upper value denotes DC location on the 11th storey L11 and the lower value denotes DC location on the 20th storey L20. Other findings showed that PER increases as the data rate, number of concrete walls in between, and distance increase. In general, a one-hop 20 mW network is capable of covering up to 11th storey vertically and across one block horizontally.

Performance in Tree Topology
Network Tests along Corridor. Table 5 presents the measured data of the tree topology network in along a corridor. Data rate modes 100 kbps and 200 kbps were used for this test. The test venue was a concrete building for office and industrial establishments. The network deployment scenario was tree topology where the DC was connected to two second-tier nodes, N1 and N2, while the third-tier was N3, N4, and N5. The distances among different nodes were described in Figure 7(a).
Based on the collected data, it was found that the network for both data rate modes 100 kbps and 200 kbps had no problem being deployed in the test site. Performance of almost no error packets can be achieved with such a setup. This indicates that for a typical office/industrial building, the network capability is sufficient to support a full range coverage. Table 6 presents the test data of the interbuilding line topology network. In this setup, the signal was bounced to and from across two buildings. The DC, N1 through N4 were placed in the same building, while N5 through N8 were placed at another building 100 meters apart. The buildings were concrete office/industrial facilities with tree obstructing in between. The illustration was given in Figure 7(b). There was a total of eight tiers of hierarchical structure in the line topology network.  The measured data showed only 1 or 2 error from the total received packets. This indicated that the PER was well below the required PER in [6].

Results Analysis
This section provides analysis based on the data gathered in the test described in Section 7 and how the test results correspond to the requirements in the use cases listed in Section 3.
Use case 1 requires the support of a typical range between nodes in an open area with LOS environment. From the test results in Section 7.1, the achievable ranges for SUN 50 kbps, 100 kbps, and 200 kbps data rate modes were 450 m, 240 m, and 260 m. It was observed that the supportable range is capable of facilitating the use case featuring the connectivity among nodes in an open air. The distance between two buildings in a typical city such as Singapore does not go beyond 450 m. This indicates that the 50 kbps mode is sufficient to connect most buildings. Additionally, for buildings separated in closer proximity within 200 m, higher data rates can be employed to establish a faster utility network.
Use case 2 requires the achievable typical range between nodes in an NLOS environment. Section 7.2 showed that for a typical deployment scenario, such as a residential multistorey building in Singapore, the achievable range for SUN 50 kbps, 100 kbps, and 200 kbps data rate modes is at least 11th storey, that is, vertically 35 m above the ground. Beyond the 11th storey, the PER of the system decreases gradually. At the 20th storey, the PER is found to exceed the required performance value of 10%. This indicates that for a 20-storey building in Singapore, a two-hop network is sufficient to support the utility network, up to 200 kbps. This data rate is sufficient to cater the demands for most utility command-and-control features.
Use case 3 highlights the requirement of multihop network establishment in a dense urban environment. Based on the results in Section 7.3, it is evident that the network is capable of enabling the network establishment and communication within an office/industrial premise. The network was deployed along an outdoor corridor, most of which involved NLOS signal propagation among nodes from various corners of the premise. The results revealed that in a typical office/industrial building spanning across approximately 100 m in length the SUN multihop tree network was capable of recording performance meeting the PER requirements of 10% for both 100 kbps and 200 kbps data rate modes. With reasonable extrapolation, the 50 kbps mode which is relatively more robust should also be able to deliver the same, if not better performance.
Use case 4 outlines the system requirements for communication networks with extended range to cater scenarios such as expansion to outer regions of the core network area. Results from Section 7.4 showed that the SUN network is able to establish and maintain an eight-hop network across buildings, penetrating through obstruction such as trees. The network is able to achieve required PER of 10% while extending the range up to eight fold. Similarly, the network is able to support the 100 kbps and 200 kbps data rate modes. This indicates that, for a particular region outside a dense area, range extension through increasing the number of hops can be employed to address the traffic demands of that region alone. This feature is especially useful in extending the range from a dense urban area to suburban areas.
All the four use cases can be combined to generically model the typical deployment scenario in an urban city. This field trial has covered most of the anticipated network deployment in such settings. Therefore, the results of this field trial may be extended and reused to analyze performance in similar areas.

Conclusion
This paper has presented the field trial results of the IEEE 802.15.4/4g/4e SUN in applications suited for outdoor environment. It was concluded that the system is able to meet the requirements set forth by the practical application scenarios. This study has verified that the SUN system can be applied in real world environments to facilitate the communications of applications of similar requirements such as utility networks, machine-to-machine networks, and sensor networks. Future works involve more tests in other environments including indoor locations.

Disclosure
At the time of publication, Dr. Ming-Tuo Zhou is affiliated with Key Lab of Wireless Sensor Network and Communication, Chinese Academy of Science.

Conflicts of Interest
The authors declare that they have no conflicts of interest.