A Combining Method for Wireless Protocol Conformance Testing: A Empirical Case

Ensuring the consistency of protocol implementation and protocol specification is the basic premise of wireless communication. With the application of wireless communication in more and more fields, the wireless communication environment becomes more and more complex, and the fault coverage of wireless protocol conformance testing is also facing more and more challenges. To solve this problem, this paper uses Finite State Machine (FSM) as a formal description tool for wireless protocols and presents a combining test method based on two test methods with complementary characteristics in the test technologies based on structural coverage and state identification. Then, the paper evaluates the effectiveness of the method based on 14 empirical cases. The experimental results show that the fault coverage of each empirical case can be effectively improved to 100% when the average test cost is only increased by 17.99%.


Introduction
Wireless protocol is the rule that must be obeyed by the communication entities and processes in wireless communication [1]. It is also the basic guarantee to realize the stability of wireless communication and the competitive advantage in the low-power wireless communication market [2,3]. Conformance testing, performance testing, interoperability testing, and robustness testing are all essential to verify the correctness of wireless protocol implementation [4]. Conformance testing is the basis of the other three kinds of testing, which is used to ensure the consistency of protocol implementation and protocol specification [1,5,6]. Finite State Machine (FSM) [7][8][9][10] is an effective formal modeling tool [11,12] in wireless protocol conformance testing since the operations implemented on the entities in wireless protocols (such as button click or window input in GUI and message request in Web service) can be mapped to the inputs in FSMs, and the responses to the operations according to the wireless protocols (such as message response in GUI or web service) can be mapped to the outputs in FSMs [13][14][15][16]. To facilitate the description, we abstract these events into alphabetic or numeric symbols in the examples and experiments of this paper. ZigBee is an emerging short-range, low-rate, and low-power wireless network technology. Taking the connection and release process of MAC layer in ZigBee protocol as an example [17], the interaction process between communication entities is shown in Figure 1(a) and the corresponding FSM is shown in Figure 1(b). The relationships between specific events in ZigBee protocol and abstract symbol definitions in FSM are described in Tables 1, 2, and 3, respectively. In this way, the problem of judging whether the implementation of ZigBee protocol is consistent with the ZigBee protocol specification can be transformed into the problem of judging whether the implementation of Zig-Bee protocol is consistent with the specification FSM of Zig-Bee protocol. Specifically, a test set is generated based on the specification FSM and the test set is applied to the wireless protocol implementation. When the actual output is consistent with the expected output, the implementation is consistent with the protocol. Otherwise, it means that there is a fault in the implementation.
Both the transition-pair coverage (TPC) method [18][19][20] based on structure coverage and the unique input output (UIO) method [7,13] based on state identification can be used in wireless protocol conformance testing. However, neither TPC method nor UIO method can achieve 100% fault coverage. To meet the high reliability requirements of wire-less protocol conformance testing, this paper presents a combined method based on the complementary characteristics of the TPC method and UIO method. And the combined method is studied by 14 established empirical cases including ZigBee protocol and the WM2RP protocol both of which are widely used in wireless communication. The contribution of this paper is mainly as follows: (1) The fault coverage of wireless protocol conformance testing is effectively improved (2) Since the experimental objects are all FSM empirical cases designed and generated by testers according to professional experience and domain knowledge, the experimental data and results have more practical significance, which can also promote the application of FSM conformance testing in industry The remainder of the paper is organized as follows. Section 2 introduces the concepts and terms about FSM. Section 3 presents a combined method for wireless protocol conformance testing. Section 4 describes the case study and evaluates the experimental results. Section 5 overviews the

Preliminaries
An FSM is formally defined as a 6-tuple M = ðI, O, S, s 0 , δ, λÞ [21,22] where (i) I and O are finite and nonempty sets of input symbols and output symbols, respectively (ii) S is a finite and nonempty set of states (iii) s 0 ∈ S represents the initial state of M (iv) δ : S × I ⟶ S denotes the state transition function According to this definition, when an input symbol i is delivered to the current state s, M moves to state s ′ = δðs, iÞ with an output produced by λðs, iÞ. A transition t is defined by a tuple ðs, s ′ , i/oÞ where s is the start state of t, i is an input of s, o = λðs, iÞ is the associated output, and s ′ = δðs, iÞ is the end state of t.
An FSM M is initially connected if for any state s there is at least one input sequence from s 0 to s, i.e., ∀s ∈ S, ∃α ∈ I * · ðδðs 0 , αÞ = sÞ (where I * is the set of finite sequences of input symbols).
If s ≠ s ′ , α ∈ I * · ðλðs, αÞ ≠ λðs ′ , αÞÞ, then s and s ′ are a pair of distinguishable states, and α is called a separating sequence of s and s ′ . If all the states in FSM M are pairwise distinguishable, i.e., ∀s, s ′ ∈ S, s ≠ s ′ , ∃α ∈ I * · ðλðs, αÞ ≠ λðs ′ , αÞÞ, then M is said to be minimal or reduced.
An FSM M is deterministic if at any state s and for any input i, there is at most one transition leading to the next state. Otherwise, M is nondeterministic.
An FSM is not initially connected means there are some states that cannot be reached from the initial state. In other words, the states that cannot be reached from the initial state are meaningless to the FSM. Therefore, an FSM that is not initially connected can be converted to an initially connected FSM by deleting the states that cannot be reached from the initial state. An FSM with equivalent states can be transformed into a reduced FSM by joining equivalent states. A nondeterministic FSM can be converted to a deterministic FSM by combining states, i.e., raising the level of abstraction.
Since all the above definitions can be satisfied, only initially connected, reduced and deterministic FSMs are considered in this paper. In Figure 2, M 1 is an initially connected, reduced, and deterministic FSM where S = fs 1 , s 2 , s 3 g, I = fx, yg, O = f0, 1g and s 1 highlighted in bold defaults to the initial state.
The connection of input sequences in FSM is represented by the juxtaposition of input sequences; that is, if α 1 , α 2 ∈ I * , then α 1 α 2 represents the connection of α 1 and α 2 . If β = α 1 α 2 , then α 1 is the prefix of β, expressed as α 1 ≤ β. If α 1 ≠ ε (ε is an empty string), then α 1 is the true prefix of β, expressed as α 1 < β. Given a set of input sequences T, pref ðTÞ represents the set of all prefixes of all sequences in T; that is, pref ðTÞ = fαj∃β ∈ T and α ≤ βg; if T = pref ðTÞ then, T is a set of prefix closures.
In this work, we assume that FSM M has a reset operation r which can bring M to its initial state from any state without any output. A string x 1 x 2 , x k∈ I * is said to be a defined input sequence at s ∈ S if there exists s 1 , ⋯, s k, s k+1 , where s 1 = s, such that ðs i , x i Þ ∈ ðS × IÞ and δðs i , x i Þ = s i+1 for all i = 1, ⋯, k. We use Ω M ðsÞ to denote the set of all defined input sequences at state s and Ω M as an abbreviation for Ω M ðs 0 Þ, i.e., the input sequences defined at the initial state of M, and hence for M itself. A test case of M is an input sequence α ∈ Ω M starting with r. A test suite of M is a finite set of test cases, and there are not two test cases α and β such that α < β. The number of symbols of a sequence α is represented by |α | . For instance, given a test case ryyy for M 1 , the length of ryyy is 4, since it contains four symbols, that is, |ryyy | = 4. This notation is extended to a test suite T; i.e., |T | means the sum of all the length of sequences in T. For instance, given a test suite T = fryx, ryyy, ryyx, rxg for M 1 , the length of T is 13, i.e., |T | = | fryx, ryyy, ryyx, rxg | = 13.
An initialization fault occurs when FSM starts from an incorrect initial state s i (s i ≠s 0 ). An output fault occurs when the correct output of a transition (s, s ′ , i/o) is changed, and a transition (s, s′, i/o′) in which the output is different from the expected one is generated. An end state fault occurs when the end state of a transition (s, s′, i/o) is changed, and a transition (s, s″, i/o) in which the end state is different from the expected one is generated. Figure 3 demonstrates the three types of faults based on M 1 . There is an initialization fault starting from an incorrect initial state s 2 in Figure 3(a). There is an output fault in Figure 3(b). The transition (s 3 , s 1 , y/0) is changed to the transition (s 3 , s 1 , y/1) with a wrong output. There is an end state fault in Figure 3(c). The transition (s 1 , s 1 , x/0) is changed to the transition (s 1 , s 2 , x/0) with a wrong end state. The FSM in Figure 4(a) is a wireless protocol specification, and the FSM in Figure 4(b) is an implementation, where the dotted line represents a transition with an end state fault. {rcbbdb, rcbbecda, rcbbecbbac} is the test set of TPC method The FSM in Figure 5(a) is a wireless protocol specification, and the FSM in Figure 5(b) is an implementation, where the dotted line also represents a transition with an end state fault. {rba, rbb, raaa, raab, rabaa, rabab, rabba, rabbb} is the test set of the TPC method generated based on Figure 5(a), and {raaa, rbba, rabba, rabaaa} is the test set of the UIO method generated based on Figure 5(a).

TPC-UIO Test Method
Test results indicate the following: (1) The test set of the TPC method can test the fault in Figure 5(b) but cannot test the fault in Figure 4(b) (2) The test set of the UIO method can test the fault in Figure 4(b) but cannot test the fault in Figure 5(b) It can be concluded from the above examples that there are obvious differences in the performance of TPC method and UIO method in different examples. Specifically, the two methods show strong complementarity. When the number of transitions outgoing from the correct end state and the wrong end state is equal, furthermore, the input/output of the transitions outgoing from the correct end state and the wrong end state are all the same, fault coverage ability of UIO method is superior to TPC method. When there are differences about the number of transitions outgoing from the correct end state and the wrong end state or there are differences about the input/output of transitions outgoing from the correct end state and the wrong end state and the UIO sequence of the correct end state is also available in the wrong end state, fault coverage ability of the TPC method is superior to that of the UIO method.

Test
Generation of TPC-UIO Method. Based on the above analysis, we introduce a combined TPC-UIO test method to realize the complementarity of the TPC method and UIO method.
Initialization fault Output fault End state fault There must be a state covering set in initially connected FSM, and an empty sequence ε is included to reach the initial state s 0 . In addition, only state covering set of prefix closure is considered in this paper. As described in Algorithm 1, given an FSM with n states {s 1 , s 2 , ⋯, s n } in which s 1 is the initial state and a transition tree T with only a root node s 1 , state covering set SCðMÞ is first initialized to ϕ. Then, for each transition starting with s 1 , a branch annotated with the input/output of the transition is added in T. Next, for each transition starting with a leaf node appears for the first time in T, a branch annotated with the input/output of the transition is added in T iteratively until there is no leaf node appearing for the first time that can be found. So far, the complete transition tree is constructed. Finally, for each state s i ð1 ≤ i ≤ nÞ, an input/output sequence β i from s 1 to s i that appears for the first time in T is obtained and β i is thrown to SCðMÞ.
According to Algorithm 1, the transition tree of M 1 is shown in Figure 6, and fε, y/1, ðy/1Þðy/1Þg is a state covering set obtained from the transition tree.

TPC Test Set Generation
Definition 2 (adjacent transition). If the end state of a transition t 1 =(s 1 , s, i 1 /o 1 ) is the start state of a transition t 2 =(s, s 2 , i 2 /o 2 ), t 1 and t 2 are a pair of adjacent transition.
The test set of TPC method covers each pair of adjacent transitions in FSM at least once. As shown in Algorithm 2, given an FSM with n states {s 1 , s 2 , ⋯, s n } in which s 1 is the initial state and a state covering set SCðMÞ = fα 1 , α 2 , ⋯, α n }, TPC test set TPCðMÞ is first initialized to ϕ. Then, for each state s i ð1 ≤ i ≤ nÞ, each transition sequence β of length 2 starting from s i is obtained and α i β is thrown to TPCðMÞ.

UIO Generation
Definition 3 (UIO sequence). An input/output sequence β is a UIO sequence of state s i if and only if for any state s j ∈ Sðs j ≠ s i Þ, λðs i , β in Þ ≠ λðs j , β in Þ, where β in is the input part of β.
Definition 4 (uncertainty of the start states). Given an FSM M and an input sequence α, a partition πðαÞ of the state set S can be obtained according to the output generated by each state for α. For any state s i and s j , they will be in the same partition block if and only if they cannot be distinguished by α, i.e., λ ðs i , αÞ = λðs j , αÞ, and this partition is called the uncertainty of the start states.

Wireless Communications and Mobile Computing
∈ πðαÞ, and this partition is called the uncertainty of the current states.
In this way, the scope of the state before and after the input α can be learned according to the output corresponding to α. The idea of state partition can be applied to the construction of successor tree.
Definition 6 (successor tree). The successor tree of an FSM is a tree showing the behavior of the FSM at any state and upon any input sequence.
For each possible input sequence, there is a path starting from the root node in the successor tree and each node in the successor tree is annotated with the uncertainty of start states as well as the uncertainty of current states.
The successor tree of M 1 is shown in Figure 7. When x is delivered to the set of states {s 1 , s 2 , s 3 }, according to the different output, the partition blocks of start state uncertainty are 0fs 1 , s 3 } and 1{s 2 }, respectively, where 0 and 1 represent the output generated by the state in the partition block. The corresponding partition blocks of current state uncertainty are {s 1 , s 3 } and {s 2 }, respectively. When x is again delivered to the current state uncertainty {s 1 , s 3 } and {s 2 } according to the different output, the partition blocks of start state uncertainty are 00{s 1 , s 3 } and 11{s 2 }, respectively. The corresponding partition blocks of current state uncertainty are {s 1 , s 3 } and {s 2 }. Since the set {s 1 , s 3 } can never be further divided by x, this branch can terminate the decomposition about input x. Next, y is delivered to the partition blocks of the current state uncertainty {s 1 , s 3 } and{s 2 }; according to the differ-ent output, the partition blocks of start state uncertainty are 01{s 1 }, 00{s 3 }, and 11{s 2 }, respectively. The corresponding current state uncertainty partition blocks are {s 2 }, {s 1 }, and {s 3 }, respectively. Since there is only one element in each partition block, the division of this branch ends here. In the same way, we can construct other branches in the successor tree until a successor tree meeting the requirement is generated.
In this paper, UIO sequence is obtained by successor tree. Given an FSM and its successor tree, if there is only one state s i in a partition block of the start state uncertainty, we will get a UIO sequence of s i . Here, we only consider the shortest UIO sequence; that is, the UIO sequence we first obtained in the top-down search of the successor tree. As shown in Figure 7, the UIO sequence of s 1 is ðx/0Þðy/1Þ, ðy/1Þðx/1Þ, or ðy/1Þðy/1Þ, the UIO sequence of s 2 is x/1, and the UIO sequence of s 3 is y/0.

Test Set Generation of TPC-UIO Method
Definition 7 (TPC-UIO method). The test set of the TPC-UIO method should satisfy two conditions. (1) Cover each pair of adjacent transition in an FSM at least once. (2) Identify the end state of each pair of adjacent transition using the shortest UIO sequence.   Table 4. |S | , |I | , and |T | denote the number of states, the number of inputs, and the number of transitions, respectively.

Mutation Generation.
In this paper, we use mutations of the specification to model faults in the implementation. Mutation operators [30] are introduced to obtain the three most typical faults in FSM conformance testing.
Definition 8 (change initial state (CIS)). This operator represents the initialization fault: change initial state s 0 to s k ðs k ≠ s 0 Þ such that the implementation starts from an incorrect initial state, that is, CISðMÞ = fN = ðI, O, S, s k , δ, λÞ | s k ∈ S · s k ≠ s 0 g.
Definition 9 (change output (CO)). This operator represents the output fault such that an unexpected output is produced in an implementation. Formally, COðMÞ = fN = ðI, O, S, s 0 , δ, ΛÞj∃x ∈ I, s i ∈ S · Λðs i , xÞ ≠ λðs i , xÞg.  7 Wireless Communications and Mobile Computing average number of the test cases for the UIO method is 9.93, the average number of the test cases for the TCP method is 32.07, and the average number of the test cases for the TPC-UIO method is 33.14. Figure 9 indicates that the number of the test cases for TPC-UIO method is more than or equal to that for the TPC method and UIO method for each empirical case. Compared with the TPC method, the average increase of the number of the test cases of the TPC-UIO method is 6.43%  (iii) Consider both the length of the test set and the number of the test cases. In order to quantitatively analyze the cost considering both the length of the test set and the number of the test cases, L is used to denote the length of the test set and the cost generated by the length of the test sets is expressed as a function C 1 = f ðLÞ, and N is used to denote the number of the test cases and the cost generated by the number of the test cases is expressed as a function C 2 = gðNÞ. The total test cost considering both the length of the test set and the number of the test cases can be expressed as C = C 1 +C 2 . Assuming that the test cost of a test set containing only one test case of length 1 is 1, then f ð1Þ + gð1Þ = 1 ð f ð1Þ < 1, gð1Þ < 1Þ. Assuming α = f ð1Þ, then gð1Þ = 1 − α, f ðLÞ = α × L, gðNÞ = ð1 − αÞ × N, and C = α × ∑ N i=1 L i + ð1 − αÞ × N where L i represents the length of the ith test case, so that the cost of each test method can be quantified by assigning a value to α. Generally, the impact of the length of the test set is greater than the impact of the number of the test cases, so it is usually α > 0:5. Figure 10 shows the relationship among the comprehensive test costs of each empirical case under different test methods when α is assigned 0.6, 0.7, 0.8, and 0.9, respectively.
From the examples of different values of α, it can be seen that when α takes different values greater than 0.5, the comprehensive test cost of each empirical case under different test methods changes, and the larger the value of α, the greater the test cost. However, the relative relationship among test costs of different empirical cases has not changed. The comprehensive test cost of the TPC-UIO method is always larger than that of the TPC method and UIO method for each empirical case. Compared with the TPC method, the average increase of the comprehensive test cost of the TPC-UIO method is 23.22%. Table 6, the TPC method, UIO method, and TPC-UIO method can achieve 100% fault coverage for initialization fault and output fault. Neither the TPC method nor the UIO method can reach 100% fault coverage for end state fault. The TPC-UIO method can effectively improve the fault coverage rate and achieve 100% fault coverage.

Comparison of Fault Coverage. As shown in
Although the differences of data in the experiment seem to be subtle, these differences are still worthy of attention.
(1) When the fault domain is very large, even a small increase of fault coverage represents a large number of faults. (2) In a safety-critical environment, there is a tiny optimization even if an optimization of only one fault has high practical value. It is noted that the end state faults in the experiment are generated randomly. In fact, the higher the proportion of the analyzed end state faults in Section 4, the better the fault coverage of the combined method.

Related Work
Conformance testing based on FSM has always been an active research direction in software testing [31][32][33][34][35][36][37][38][39] Results show that significant improvements in the effectiveness of coverage-based tests usually occurred as coverage increased from 90% to 100% and the complementarity of the two methods in their effectiveness was proved. Briand et al. [42] extended the round trip path criterion in the white-box test with the classification partition in the blackbox test and obtained better fault detection performance. This paper also analyzes the complementary characteristic of different test methods and improves the fault coverage at the expense of an acceptable increase of test cost, so as to meet the high requirements of fault coverage in wireless communications.

Conclusion and Future Work
There are some differences about the test requirements of the TPC method and UIO method in wireless protocol conformance testing. The TPC method focuses on covering all consecutive operations of length two between entities or processes in wireless communication. UIO method focuses on identifying each state of entities or processes and confirming the input and output information between any two states in wireless communication. In order to improve the effectiveness of wireless protocol testing, a combined method is presented and the method is evaluated by 14 established empirical cases. A unified method based on state covering set is used to generate test sets for each test method and three types of classic faults are generated by mutating the specification. Fault coverage has been effectively improved by the combining method. The experimental results show that the fault coverage of each empirical case can be improved to 100% when the average test cost is only increased by 17.99%. Since what we considered is the specifications modeled by FSMs and the faults generated based on FSMs, although the fault coverage has been effectively improved, these results cannot be popularized to the specifications modeled by other formal models or faults generated based on code.
In a future work, we will try to enrich the experimental data with more publicly available FSM empirical cases. To better describe the behavior characteristics of wireless protocol testing, the future work will also consider extended semantic FSMs to improve the expression ability of the model.

Data Availability
The data used to support the findings of this study are available from the corresponding author upon request.

Conflicts of Interest
None of the authors have any conflicts of interest.