Formal Specification Based Automatic Test Generation for Embedded Network Systems

Embedded systems have become increasingly connected and communicate with each other, forming large-scaled and complicated network systems. To make their design and testing more reliable and robust, this paper proposes a formal specification language called SENS and a SENS-based automatic test generation tool called TGSENS. Our approach is summarized as follows: (1) A user describes requirements of target embedded network systems by logical property-based constraints using SENS. (2) Given SENS specifications, test cases are automatically generated using a SAT-based solver. Filtering mechanisms to select efficient test cases are also available in our tool. (3) In addition, given a testing goal by the user, test sequences are automatically extracted from exhaustive test cases.We’ve implemented our approach and conducted several experiments on practical case studies.Through the experiments, we confirmed the efficiency of our approach in design and test generation of real embedded air-conditioning network systems.


Introduction
An embedded network system (ENS) is a system consisting of a number of embedded units that communicate with each other.Nowadays, ENSs are growing rapidly and playing an important role in our daily life, such as in-vehicle networking systems, home appliance networks, and building energy management systems.On the other hand, because of the increasing complexity of functionality and growing network scales, as well as the increasing numbers and types of embedded units in ENSs, development of the ENSs has been becoming more and more difficult.Particularly, design and test generation are ones of the most important and cost-consuming phases in reliable ENS development, but many companies still rely on the traditional techniques where system requirements are described in a natural language and test cases are generated based on human experience.However, design and test by manpower may not sufficiently cover enormous numbers of combination to be considered especially for complicated ENSs.
During the last decade, numerous formal methods have been proposed to improve software quality.Especially, test generation is an attractive application for mechanized formal methods; the importance of good test cases is universally recognized and so is the high cost of generating them by hand.One of these methods, the model-based test generation, has been extensively studied (e.g., see [1][2][3][4]).One of the advantages of this technique is that each behavior of the system model is described directly, and then test cases can be automatically generated from the models.Note, however, that if test cases are generated from incomplete specifications and models, they suffer from the problem of lacking and low quality of test cases.Therefore, formal specification based modeling and test generation for large-scale ENSs are still challenges.
This paper presents our works gained in a joint project of AIST, DaiKin Industries, and Kyoto Institue of TechnoIogy.The ultimate goal of our project is to develop a robust and reliable automatic test generation tool suitable for a development process of air-conditioning network systems developed by Daikin .(We refer to the company as Company D in the rest of this paper.)The target system consists of a number of embedded controllers and indoor/outdoor air-conditioning equipment for building management.For this project, we have taken an approach of automatic test generation based on a formal specification.In the following, we present three main contributions toward our overall goal.
(i) Development of a Formal Specification Language.First, we have developed a specification language for ENSs called SENS (Specification language for Embedded Network Systems).SENS is designed under the concept of test-oriented, object-oriented, modular, and lightweight description of ENSs.In SENS system, the requirements are specified using logical constraints.This logical property rigorously determines a set of state transitions that satisfy all constraints and thus helps to get the specification without lacking and inconsistency and clearly obtain an advantage over a manpower design of system and test.
(ii) Development of a Test Case Generator.Second, we have developed a tool called TGSENS (Test Generator based on SENS) which automatically generates high quality test cases from a given specification written in SENS.Our tool can find distinct and enormous test cases quickly that satisfy complex specifications using a SAT solver as a core engine.Our tool can also perform filtering to select efficient test cases.
In a feasibility study using air-conditioning systems of Company D, given a specification for a certain function of the system, our tool took only about 12 seconds to generate 77,700 test cases.We expect that our tool will provide both quality improvement and cost saving for testing target ENSs.
(iii) Development of a Test Sequence Generator.Third, we have also developed a tool in TGSENS to generate test sequences that satisfy a given testing goal by a user.In an experiment using components of the target air-conditioning systems, our tool generated 156 test sequences in 8 minutes for a given testing goal such that the sequence contains 2 milestone states whose distance is less than 3 state transitions.We confirmed that our tool is helpful for test engineers to design test scenarios efficiently.
The rest of this paper is organized as follows.In the next section, we briefly introduce the overview of our work and our tool.Section 3 describes the features, syntax, and semantics of the SENS language.In Section 4, we propose the method for generating test cases and sequences from SENS specifications.Section 5 presents experimental results when applying our tool to the air-conditioning systems of Company D. Section 6 discusses related works.Finally, we conclude our work and mention future works in Section 7.

Overview of Our Work
Our target system is an embedded network system (ENS).An ENS can be considered as a concurrent system consisting of multiple subsystems with global environments, for example, embedded indoor and outdoor equipment and controllers in an air-conditioning network system as illustrated in Figure 1.Each subsystem in an ENS is an event-triggered system whose events are communications with other subsystems and whose operations are constrained by conditions related to its environment and time.
The behaviors of an ENS are specified in a hierarchical way: an ENS consists of subsystems, a subsystem is constructed from its functions, and each function is constructed from a set of functional requirements, as described in Figure 1.To enhance the reliability and efficiency of design and testing ENSs, we first focus on designing and modeling a formal specification language for ENSs, called SENS, which supports the hierarchy of ENSs and is convenient for our test generation purpose.We next achieve an automatic test generation approach based on SENS, called TGSENS. Figure 2 illustrates the overview of our tool that consists of the following 5 steps.
(1) Specifying Requirements in SENS.A user describes system requirements of the target ENS using SENS.
For the air-conditioning systems of Company D, we have translated original specification documents written in a natural language to formal specifications in SENS.
(2) Translating SENS Specifications to CNF.The SENS translator first performs the syntax and type checking for the given SENS specification.Next, the translator automatically transforms logical requirements of the SENS specification to CNF (conjunctive normal form) formulas according to our translation rules. (

SENS Specification
In this section, we present a compact formal language called SENS to specify requirements of embedded network systems (ENSs).In Section 3.1, we present features of SENS.In Sections 3.2 and 3.3, we briefly describe the syntax and semantics of SENS.

Features of SENS.
The SENS specification is basically a set of requirements represented by logical formulas.SENS is based on a quantifier-free predicate modal logic with a next operator where predicate variables are classified into environments, systems, communication channels, and timer variables.SENS has the following features.

Property-Based.
Property-based logical formulas represent requirements which constrain system behaviors.Since a part of the specification less affects the entire specification, it is easy to modify and reuse SENS specifications.A requirement is described in a traditional Hoare triple-like way: where  (resp., ) denotes a logical constraint holding in a prestate (resp., poststate) and an V (resp., ) represents a message receiving (resp., sending) between subsystems.This 3-tuple requirement needs to hold for all transitions of the system model, and we call this 3-tuple requirement a transition requirement.

Test-Oriented.
For a testing purpose, specifying tested values and bounds of variables is available in the variable declaration.This supports a test-oriented design and an efficient test generation from SENS specifications.

Object-Oriented.
A group of subsystems is represented as a parameterized class, and each subsystem is defined as an instance of the class in the main class representing the whole system.This representation leads to an efficient description of a large number of subsystems in an ENS.
3.1.4.Modular.Specifications of an ENS, a subsystem, and a function are broken down into specifications of subsystems, functions, and transition requirements, respectively.This modular construction makes it easy to specify a large-scaled and complicated ENS piece by piece and provide a reusability of specifications.This also allows a step-wise test generation from SENS specifications for unit testing, integration testing, and system testing.

Lightweight Description of Time
Constraints.We consider a timer as a component of a subsystem and then specify time constraints using events and actions with the timer.SENS provides a timer variable which is declared with a timevalue, a time-unit, and a condition formula for counting up the timer.

Syntax of SENS.
Here, we give the brief explanation of SENS description using the following example embedded system.
Example 1.Consider an illustrating system in Figure 3.There are 2 kinds of equipment, namely, classes  and .Let us consider a simple ENS consisting of 3 subsystems  1 ,  1 , and  2 , where  1 is an instance of  and  1 and  2 are instances of .A subsystem of class  sends a message  after 2 minutes from starting its operation.Each subsystem of class  turns on its lamp when receiving message .This system can be described in the SENS specification of Pseudocode 1.
The SENS declarations consist of the following declarations (refer to Appendix A for the full syntax of SENS).
(i) Main Class Declaration.File <main.cls>specifies the main class.The main class represents the whole system consisting of subsystems and the environment.In the file <main.cls> of Pseudocode 1, the classes  and  are imported in line 1, and subsystems  1 ,  1 , and  2 are instantiated in lines 6-7.Environment variables are declared in the Var part.Data types, environment variables, and channels declared in the main class are referred in all class declarations.Invariants (logical expressions) for the entire system are declared in the main class.In the example <main.cls>,global data type onoff status is declared in lines 3-4.
(ii) Class Declaration.Files <A.cls> and <B.cls> of Pseudocode 1 specify classes  and , respectively. class other than the main class specifies a class of subsystems.Local data types, variables, channels, and timers for the subsystem are declared in the class declaration.In <A.cls>, local variable op and timer t are declared in lines 6-9, and a function of the subsystem is declared in lines 11-14.For data types, variables, channels, and timers externally declared and internally used, the files to declare them are imported with the Import statement and their names are described in the Extern part.The main class file is imported  in lines 5-6.Using the test declaration beginning with the keywordtestedwith, one can define the specific values of variables to be considered in a testing phase.For example, consider the following variable declaration: testedwith {-2, 0..5 (0.5)}; variable v1 is defined as follows: its type is double, its range is −2 ≤ v1 ≤ 5, its initial value 0, and its tested values are −2 and a number from 0 to 5 in increments of 0.5.In our semantics of SENS, each variable of a subsystem takes one value, that is, an element of its domain.If the declaration of a variable has a test declaration, the domain is defined by the test declaration.Otherwise, the domain is defined by the data type of the variable.
(iv) Channel and Timer Declarations.Channels are used to describe communications between subsystems.A channel is declared with its name and message type.In lines 10-11 of <B.cls>, a channel ch for receiving message m is declared.A timer is declared with its name, time-value, unit, and conditions (logical formula) for counting up.When the condition is omitted, the condition is always true.For example, consider the following timer declarations: Timer t1 is declared to count up 5 minutes.Timer t2 is declared to count up 5 hours, but the counting is performed only when the value of v is dry or cool.
The event is a message receiving (channel-name?expr) or timeout (timer-name?ring).Similarly, the action is a message sending (channel-name!expr), timer starting (timername!start), or timer reset (timer-name!clear).When specifying a communication between subsystems, a message sending (action) is described using both a recipient's object name and a channel name, while a message receiving (event) is described using only a channel name.
(v) Function and Requirement Declarations.Functions are declared in the class declaration or in individual function declaration files.In the function declaration, constants, data types, variables, and timers can be defined locally.A function contains a set of requirements.
A requirement is an expression representing an invariant or a 3-tuple of a precondition, an event, and a postcondition or/with/without actions.(More precisely, the third part of the transition requirement is a postcondition or an action or actions or a postcondition with an action or a postcondition with actions.)We call the 3-tuple requirement a transition requirement.The BNF description for a transition requirement is given in Algorithm 1 (Refer to Appendix A.4).
The precondition and postcondition are logical expressions.In the transition requirement declaration, the event part can contain no more than one event (it can also be null which is a symbol showing that no event occurs), while the action part can contain multiple actions.We assume that a transition requirement contains no more than one event and actions related to the same timer.
Expressions are constructed with constants, variables, and operators.Expressions have logical comparison and arithmetic operators that general programming languages such as C have.In addition, expressions in SENS have the logical implication operator (->) and the previous state operator ( ∼ ).Especially, expression ∼ v represents the previous value of v.

Semantics of SENS.
A SENS specification describes requirements of an ENS consisting of subsystems communicating each other.We model the system specified by SENS using a labeled transition system.Figure 4 illustrates our system modeling where time constraints in SENS are modeled using our timer models that will be shown later.First, each instance corresponding to a subsystem in the given specification is modeled.When the class corresponding to the subsystem contains timer variables, we compose the subsystem model and its timer models.Next, an entire ENS is modeled as a parallel composition of all subsystems.The transition system modeled from the SENS specification is used for generating test cases and test sequences.
In Section 3.3.1,we give the modeling of timers.In Section 3.3.2,we give the modeling of a subsystem and explain the composition of the subsystem and the timer.In Section 3.3.3,we explain the composition of subsystems.

Modeling of Timers.
A timer is specified in the timer declaration with its timer name, number (time), unit, and conditions for counting up in SENS.We consider two kinds of models  1 and  2 for timers expressed in Figures 5(a) and 5(b).We can use simpler timer model  2 for state elimination, while we can use timer model  1 for considering the relation of a time length between timers (refer to Section 3.3.2.).
In models  1 and  2 , states ready and over, respectively, represent states in which timer   is ready and counted over.In model  1 , state "ℎ" (0 ≤ ℎ ≤ max) represents a state in which timer   has counted up ℎ times.In model  2 , we reduce states 0, . .., max to state run representing a state in which timer   is running, for the state elimination.
In both models, each state is changed to state ready when receiving message clear.State ready is changed to state 0 in  1 (resp., state run in  2 ) when receiving message start.State max in  1 (resp., state run in  2 ) is changed to stateover when sending message ring.We define that transitions labeled with ! and c occur only if the condition for the timer holds.If the condition is not described in the timer declaration, the condition is considered to be "true".
If there are multiple conditions , , . . ., , we define transitions as represented in Figure 5(c).In the figure, transitions labeled with ?clear are eliminated for simplicity.In the figure, the first, second, and third transitions labeled with c occur when conditions , , and , respectively, hold.

Modeling of Subsystems.
A subsystem is specified in the class declaration in SENS.For a given specification, we model the behavior of each subsystem as a labeled transition system in two steps.First, the behavior of a subsystem   except timers is modeled as a labeled transition system (  ).Next, the behavior of a subsystem including timers is modeled as   (  ) which is a composition of (  ) and models of the timers.
Definition 3 (subsystem model).We model the behavior of a subsystem   , except for its timers, as  We model the behavior of a subsystem   with its timers  1 ,  2 , . . .,   as an interleaving composition of the subsystem model and timer models as follows: where (  ) denote the subsystem model explained in Definition 3 and (  ) denotes the model for timer   (1 ≤  ≤ ) explained in Section 3.3.1.We inductively define the above model composition, that is,

System
Model.Now we explain the model for the entire system.Assume that the entire system consists of  subsystems.We define the model for the system as the parallel synchronized composition of labeled transition systems   (  ) representing all subsystems   (1 ≤  ≤ ).The composition of two subsystems is defined in Definition 20 and the entire system model   ( 1 ) ‖ ⋅ ⋅ ⋅ ‖   (  ) is obtained by an inductive composition.(When a transition of a subsystem has multiple message sending, we assume that its composition is synchronously performed in our modeling.)The composition rule is defined in the definition of Appendix B.3.
Example 6.Consider the example specifications of subsystems   and   [1] in Pseudocode 1. Figure 8 shows an example composition of transitions in   (  ) and   (  [1]).

Automatic Test Generation from the SENS Specification
In this section, we present the methods for a given SENS specification to automatically generate test cases and test sequences in Sections 4.1 and 4.2, respectively.

Test Case Generation
4.1.1.Concept.In our approach, we find all test cases by using a SAT solver.A SAT solver is a tool to solve the SAT (satisfiability) problem which determines whether there is a value assignment to satisfy a given Boolean formula in CNF (conjunctive normal form).Although this problem is an NP-complete problem, there have been very high-speed SAT solvers (e.g., MiniSAT [5]) for practical sized problems.Using a SAT solver, we can find distinct and enormous test cases quickly that satisfy SENS specifications.

Test Cases.
Given a SENS specification that describes a target system, every transition of the system model satisfies all requirements in the given specification, as mentioned in the previous section.Therefore, we can consider that each transition  of the system model corresponds to a one-step test case.That is, the prestate (resp., events) of  is a valid input state (resp., system inputs) of a test case and the poststate (resp., action) of  is a valid output state (resp., system output) of the test case.We formally define a test case as follows.
Definition 7 (test case).Consider a given specification S. A test case is a tuple, (, V, , ) , corresponding to a transition of the system model  such that  ⊨ S. In other words, a test case is an assignment that makes every transition requirement of S hold.
Note that, if the given specification is for a subsystem module, subsystems, and an entire ENS, derived test cases are, respectively, used for unit testing, integration testing, and system testing.

Translation of SENS to CNF.
For generating test cases, we first translate all the requirements in the given SENS specification to a CNF formula which becomes an input of a SAT solver.Since the SENS specification is a set of logical constraint formulas, we can translate it to CNF according to the semantics of SENS.
There are 3 main steps of the translation procedure as follows.First, we introduce fresh propositional variables to construct the state space of the system model.Second, transition requirements and invariants in the SENS specification are translated into logical formulas.Third, we collect all of the constraints presented in logical formulas.The constraints are connected with conjunction and the result is converted to CNF. (Every propositional logic formula can be converted into an equivalent formula, that is, in CNF.However, by naive algorithms where the transformation is based on rules of De Morgan's laws and the distributive law, the size of CNF can be exponentially increased.To get a compact CNF formula, we used the Tseitin-transformation [6] after translating SENS to a propositional logic formula.)Due to space limitation, the details of all translation rules are given in Appendix C.

Generating Test Cases.
Once we obtain the CNF formula from the given SENS specification, we apply a SAT solver to the CNF formula to find an assignment.As mentioned above, the assignment that makes requirements of the SENS specification true corresponds to a test case.
Algorithm 2 describes our method for generating test cases.We use a SAT solver iteratively to find all possible assignments and convert them to test cases that cover all possible transitions of the system model.

4.1.5.
Filtering.The proposed method can generate all possible test cases with 100% coverage for requirements of a given specification.On the other hand, the number of complete test cases can be very huge.In order to extract efficient test cases, we also give several filtering mechanisms in our method.In the current version of our tool, a user can select test cases satisfying a precondition of transition requirements, test cases containing a communication label, and test cases whose timer label isring, which means the test cases related to a time constraint.
Example 8. Consider again the example specification of subsystem   in Pseudocode 1.Each transition of the model for system   in Figure 7 corresponds to a test case.Especially, (( ∼ op=off,op=on), 0, {t!start}, ( ∼ op=on,op=on)) correspond to a test case satisfying a precondition of the transition requirement < ∼ op==off&&op==on:null:t!start> in line 13 of the specification.On the other hand, (( ∼ op=on,op=on),{t?ring}, {C B [1].ch!m,C B [2].ch!m}, ( ∼ op=on,op=off)) correspond to a test case containing timer label ring and communication labels.
As shown in Table 3, the number of all test cases for   is 176.Among them, test cases satisfying a precondition are 10, which is less than 6% of the entire test cases.Moreover, among them, 2 test cases can be extracted, respectively, with a communication label and timer label ring.

Test Sequences and Testing Goals.
In our method, we generate test sequences from two inputs: given a SENS specification and a testing goal.First, we define a test sequence as follows.
Definition 9 (test sequence).A test sequence is a sequence of test cases, whose adjacent output and input are the same.This means that a test sequence is a sequence, where   =  +1 , for  = 1, . . .,  − 1.
Next, a "testing goal" is a key feature that helps to generate appropriate test sequences.This also allows us to reduce the size of space and time for searching test sequences.In the current version of our method, a user can specify the following 3 features as a testing goal: "key variables" for a testing purpose, a list of "milestone" states forming a backbone of the sequence, and a "maximum length" between two adjacent milestones.We formally define a testing goal as follows.
Definition 10 (testing goals).Given a SENS specification, a testing goal  = ⟨, , ⟩, where  is a set of key variables,  is a list of milestone states, and  is a list of maximum lengths such that (1)  "key variable" is a variable that we focus on testing in the given specification; (2)  "milestone" is a state that we want each test sequence to pass through; Data: A SENS specification Result: A Set of test cases (1) begin (2) T ranslatethegivenSENS specification to a logical formula  in conjunctive normal form (CNF) based on translation rules in Appendix C; (3) Input to a SAT solver; (4) if is satisfiable then (5) The SAT solver outputs a solution  (value assignment) corresponding to one test case; (6) Add the solution  to a set of solutions; (7) L e t ←  ∧ ¬ and return Step 2 to search for another solution; ( 8) else (9) The SAT solver declares  is unsatisfiable; (10) Then there are no more solutions; (11) Translate previous solutions to test cases and stop; (12) return The set of test cases; Algorithm 2: Our algorithm of test case generation.
(3)  "maximum length, " max of a run from a milestone  ∈  to the next milestone   ∈ , denoted by  ≤max   →   , is the maximum number of transitions needed for the run.
We say that a test sequence ⟨( 1 ,  1 ,  1 ,  1 ), . . ., (  ,   ,   ,   )⟩ satisfies a testing goal  = ⟨, = ⟨ 1 , . . .,   ⟩, ⟩, if and only if (1) the first input state  1 and the last output state   of the test sequence, respectively, are equal to the first milestone  1 and the last milestone   , (2) all other milestone states exist in the input states of the test sequence in order, and (3) the length between the milestone states in the test sequence is less than the maximum length defined in .In addition, value assignments only for the key variables appear in the generated test sequences satisfying the testing goal.
Example 11.If the list of milestones is ⟨ 1 ,  2 ,  3 ⟩, then we can specify the maximum length in total as the where max 1 is the maximum of transitions which can be taken to go through from the milestone  1 to the milestone  2 , and max 2 is the maximum of transitions which can be taken to go through from the milestone  2 to the milestone  3 .

Generating Test Sequences.
Our approach for test sequence generation has three main steps.First, from a given SENS specification, we first generate all test cases by our method in Algorithm 2.Then, the obtained test cases correspond to all transitions of the target system model.
Second, we synthesize all the transitions into a transition system (graph).Each state of this transition system is a tuple of variable assignments.Each edge is a transition relation from a prestate to a poststate, and a label of each edge is an event that makes the transition occur.The transition system of the specification is stored in a file, while the testing goal is stored in a separate file.
Third, given the transition system and the testing goal in two files, we compute all possible desired test sequences that satisfy the testing goal; that is, we search the set of sequences starting from a given milestone state to another milestone state whose distance is less than the given maximum length by tree searching algorithms including exhaustive BFS (breath first search) and DFS (depth first search) algorithms.Each resulting test sequence is treated as a test scenario.In our test sequence generation, we reduce the searching state space of the system model by an equivalence partitioning based on the key variables of the given testing goal and search the sequences containing milestone states.
In our TGSENS, test sequences can be generated from both the original transition system and the filtered test cases.In the test sequence generation from the original transition system, the obtained test sequences obviously keep the transition relation of the original system model.In the test sequence generation from the filtered test cases, test sequences including transitions eliminated by filtering are not generated.However, since the filtering is defined by formal rules, the effect by filtering is clear for users who recognize that test sequences with filtering options will be obtained.For this reason, we believe that it does not affect the main results of our research, in other words, generations of reliable test cases/sequences.

Implementation and Experiments
5.1.Implementation.We implemented our approach of SENS-based automatic test generation in the tool called TGSENS.The tool TGSENS consists of the following four parts as illustrated in Figure 2. (1) The first part translates the specification written in SENS into a CNF formula.(2) The second part searches and collects all assignments that satisfy the CNF formula by using a SAT solver as its core engine.(3) The third part translates the obtained assignments into test cases which are expressed with variables used in the original SENS specification.(4) The forth part searches the test sequences from the output file of the third part and given testing goals.The resulting test sequences can be viewed by two ways: a textual format or a graphical figure using a dot file viewer.
Currently, for the function level (corresponding to the function declaration in SENS) and the subsystem level (corresponding to the class declaration in SENS), we have completed the implementation of the first part in TGSENS but not yet for the system level.That means that the implementation for the composition of subsystems for the entire system level is our future work.Hence, the current version of our tool can perform the test generation for unit testing.
Our tool was implemented on Windows XP for both ×86 and ×64 environments.The first, third, and forth parts were implemented with a Java language.The second part was implemented based on MiniSAT [5] with C++.The size of the implemented programs is totally about 20,000 lines, respectively, 16, 000, 3, 000, 500, and 340 lines for each part.
We also implemented the user interface (in Japanese) of TGSENS given in Figure 9.Given an input SENS specification, our tool has execution options to generate output files of exhaustive test cases, filtered test cases, and test sequences under a given testing goal.TGSENS also has a function to check a reachability of states and generate path sequences among the input states.

Experiments: Specification in SENS.
We applied SENS to specify system requirements of real air-conditioning systems by Company D. In general, there are specifications with different levels.In Company D, specifications are divided into "instruction manuals, " "functional specifications, " and "detailed specifications, " and each specification is broken into functions.For example, as to specifications of certain indoor equipment, an instruction manual consists of about 10 functions with 30 pages, and a functional specification and a detailed specification, respectively, consist of about 80 functions with 400 pages.
We translated several specifications in a natural language with tables and figures to SENS specifications.The target specifications were for two modules  ℎ and   in an instruction manual and one function   in a detailed specification of an indoor equipment, one function   in a functional specification of an air-cleaning system, and three functions   ,   , and   in a functional specification of a controller.
For all specifications, this translation needed 1 or 2 days by one person who is one of our SENS developers.Almost all (70-80%) of the translation time was for understanding the real specifications and confirming whether the SENS specifications are valid with industrial engineers, while the rest of the time (several hours) was for the actual description.In addition, we held a one-day seminar for industrial engineers to explain how to describe the SENS specifications and we  confirmed that after the one-day seminar, the engineers were easily able to use SENS to specify embedded systems by themselves.
Example 12.We show a part of an example SENS specification for function   in Pseudocode 2.   is a mode change function for an air-cleaning system by Company D. The original specification of this function was written in Japanese with  4 sized 4 pages and includes 5 tables and 4 transition diagrams.We transformed this document into about 100 lines of a SENS specification.The SENS specification consists of 8 variables (including 6 global variables, 1 local variable, and 1 channel), 20 invariants, and 8 transition specifications.Lines 16-18 specify invariants and lines 20-22 specify transition specifications.All 8 variables have discrete values and the details of variables and their ranges are given in Table 1.
Table 2 shows the scale of the SENS specifications for function  ℎ -  : the number of lines except comments and blank lines, the number of declared variables and channels, and the number of requirements including invariants and transition requirements.As to the specification of   , the size of requirements was relatively long since variables of the array type with a big size are used.In the future, our SENS description could be more compact by preparing for-statement or while-statement descriptions.For comparison, we also show the number of pages of the original specifications in a natural language in Table 2.
Although we, here, showed the experimental results only for air-conditioning systems, we also have other case studies applying SENS to specify embedded systems such as an electric pot and an automatic dispenser.For instance, we applied SENS to describe a requirement specification for an electric pot (http://www.sessame.jp/workinggroup/WorkingGroup2/POT Specification.htm), which has been openly used as an example embedded system by SESSAME (http://www.sessame.jp)(Society of Embedded Software Skill Acquisition for Managers and Engineers) of Japan.The original requirement specification was written in Japanese including state transition diagrams with 14 pages and consisting of 14 functions.We described approximately 1,100 lines of the SENS specifications for all the functions, which include 28 variables, 5 timers, and 7 channels in the class specification and totally 386 requirements for all function specifications.Through our case studies, we confirmed that SENS is efficiently applicable to not only the air-conditioning systems but also other general embedded systems.
In addition, by describing the specifications in SENS, we found several undefined and under-specified requirements in the original specifications for the target air-conditioning system.Through the feasibility study, we could confirm both the description ability and facility of SENS.Engineers in Company D plan to use our tool in real development processes.We anticipate that using SENS increases the quality of specifications by preventing the undefinition, the ambiguity, and the inconsistency in the specifications.

Experiments: Test Generation.
In this section, we describe several case studies of generating test cases and test sequences from given example specifications <A.cls> and <B.cls> in Pseudocode 1 and   ,   ,   , and   , using our tool TGSENS.  is a specification whose variable range is reduced from that of   for testing.The experiment was performed in the following environments: Intel Core 2 Duo P8600 2.4 GHz, 3.0 GB Memory, Microsoft Windows XP SP3, and Sun Java Runtime Environment 1.6.024.Table 3 describes the result of applying TGSENS to test case generation.In the table, we show the size of SENS specifications, the size of CNF, the number of test cases, and the execution times to generate test cases for each part of TGSENS.For the size of a SENS specification, we represent the number of states and the number of transition labels in the table.Here, the number of states indicates the one of the state space constructed by all possible variable assignments in the specification.The number of transition labels indicates the number of labels in the system model constructed from the specification.
Example 13.Consider the case of   .The number of possible states (including states not satisfying the specification) for   is 6 × 4 × 7 × 5 × 3 3 = 68,040 from Table 1.The number of transition labels is 2 as shown in Table 3.Then the number of all possible state transitions (those that do not satisfy the specification are included) is 68,040 × 2 × 68,040 ≈ 9.3 × 10 9 .From such a large number of transitions, TGSENS figured out 77,700 transitions satisfying the given specification as test cases within 12 seconds.For instance, as for the filtering option of including a communication label, TGSENS selected 8,100 test cases (≈10.5% of all the test cases) within 1 second.
For   (resp.,   ), TGSENS took only about 26 (resp., 1) minutes to generate approximately 900,000 (resp., 160,000) test cases.For example, the specification of   has 13 variables including 3 previous state variables and those value ranges are 3, 5, 5, 4, 5, 5, 2, 3 5 , 3 5 , 3, 3, 3, and 3.Then, the number of states for   is 2 × 3 15 × 4 × 5 4 = 71,744,535,000 ≈ 7.2 × 10 10 .On the other hand, the numbers of states for   and   are 45,000 and 68,040, respectively.  has a feature of including variables whose range are large, like 3 5 , and thus the number of states is excessively large compared to those of other specifications.The execution result of   indicates that the specification has 71,744,535,000 ≈ 7.2 × 10 10 states and 30 transition labels.Then, the number of all possible state transitions for   is 71,744,535,000 × 30 × 71,744,535,000 ≈ 1.5 × 10 23 .Since the number of state transitions for   is much larger than those for   and   , the CNF size for   is much larger than those for   and   .Accordingly,   needed much execution time to search test cases.Although, with such a huge amount of state space, it is very impossible to manually check all cases by human, TGSENS can correctly do that and takes only about 26 minutes.
We next describe the result of applying TGSENS to test sequence generation.Table 4 shows the results using the SENS specification for function   with 3 example testing goals, G1-G3.In the table, for each testing goal, we show the number of milestone states, the maximum lengths between milestones, the number of key variables of the given testing goal, the number of test sequences, and the execution time to generate test sequences.
Example 14. Pseudocode 3 shows the input file to specify the testing goal G3 in TGSENS.We can also set the testing goal directly in the GUI interface of TGSENS shown in Figure 9(b).The testing goal G3 specifies the following: (i) key variables (declared in lines 25-28): three variables Mode A, Mode B, and AV in the specifi cation.
When the specification for   and the testing goal G3 are given, a transition system of this function is first obtained from the result of test case generation.Then, TGSENS automatically searches desired test sequences satisfying the testing goal from the transition system.Pseudocode 4 shows a part of the generated test sequences.The resulting transition system for   has 1,800 states and 77,700 transitions.From the transition system, TGSENS figured out 28 (resp., 32) sequences from  1 to  2 (resp.,  2 to  3 ) and thus totally 896 (=28 × 32) test sequences satisfying the testing goal only in 3 minutes.As for the modeling, many techniques have been proposed, even if we focus on formal modeling only.Models are described based on contracts, abstract data types, process algebras, labeled transition systems, data flow, and so on.These concepts are implemented using specification languages in tools like VDM [7], Z [10], B [11], Spec# [12], and JML [13].With most modeling languages, a user has to specify the systems' behaviors individually.

Related Work
Test case generations from models have been proposed in several ways.Here, test cases include arguments to execute specific functions, pairs of possible inputs and expected outputs, or event sequences.Test cases are searched by tracing executions of models, selecting random values with filtering, or generating candidates with some algorithms.
Here, we describe previous works on model-based testing.Spec Explorer [9] generates test cases by state exploration based on multiple strategies from contract-like specifications Spec# on a Microsoft.NET framework.In TGV [8] by INRIA, as a part of the CADP (Construction and Analysis of Distributed Processes) package, test graphs are generated from labeled transition systems.Uppaal-Tron [14] generates test sequences for real-time systems using the model checker UPPAAL, and an industrial case study of Uppaal-Tron targeted an electronic refrigerator controller.T-VEC [15] is a coverage-based test generation tool of Simulink/MathWorks which has been used to identify a fault in the Mars Polar Lander.
Some model-based test generators use constraints solving to generate test cases.In [16], a test generator based on autofocus is proposed.Autofocus [17] is a graphical tool for model-based development of distributed systems.By translating a specification into a constraint logic program, this generator can deal with all the possible execution traces of the model.TestEra [18] is a test generation framework for Java programs.By considering pre-/postconditions for executing methods as constraints, TestEra generates test cases with SAT-solvers.Exhaustive generation is available through certain options.
Most previous studies targeted the testing of programs or protocols.Although TGV has case studies for smallsized embedded applications, there are only a few studies on the large-scale embedded network systems targeted in our proposed project.Different from other works, we have proposed the original specification language, SENS, focusing on property-based specifications and the tool TGSENS to automatically generate test cases and sequences focusing on testing goals for embedded network systems.

Conclusion
In this paper, we have proposed a method to automatically generate test cases and sequences based on the specification of embedded networked systems (ENSs).We first proposed a formal specification language SENS whose features are (1) property-based, (2) test-oriented, (3) object-oriented, (4) modular description, and (5) lightweight description of time constraints.By the "property-based" feature such that requirements are property-based constraints holding in the entire ENS, logical properties for the system specification are clear and modifications and reuse of the specifications are easy since a part of specifications less affect the entire specifications.Thus, SENS has an advantage on the efficient description and reuse of specifications for large and complicated ENSs.The "object-oriented" and "modular" feature enables SENS to structurally describe requirement specifications of ENSs consisting of a number of embedded subsystems/equipment.The "lightweight" feature on description of time constraints leads to readily specify embedded systems sensitive to time requirements.In addition, the "testoriented" feature is helpful for automatic test generation from SENS specifications.To the best of our knowledge, there has been no other formal specification language having all the above features for ENSs.Using SENS, we can efficiently describe the requirements of ENSs.
We next proposed the methods for automatic test generation in the tool TGSENS whose features are summarized as follows: (1) TGSENS takes specifications of an ENS written in SENS as its input and automatically generates test cases and test sequences; (2) the test case generation problem is translated into a satisfiability problem, and test cases are generated by iteratively using a SAT solver; (3) the test sequences are also automatically obtained based on a testing goal, that is, key variables, milestone states, and a desired maximum length of test sequences, given by a user.We finally applied TGSENS to generate tests for the real airconditioning systems, and our tool rapidly succeeded in generating about 900,000 test cases in 30 minutes and extract 28 test sequences in 3 minutes.
We expect that using TGSENS can improve the quality of test cases and thus enhance the quality of ENSs.Previously in industrial fields, test cases have often been designed in a manual way.Such the test design depending on testing engineers' skills seldom covers all the requirements of the complicated ENSs.On the other hand, TGSENS can automatically and mechanically generate test cases from given formal SENS specifications.Both the description of formal specifications and the automatic test generation lead to improving the quality of test cases.Actually, we found several undefined and under-specified requirements by describing the SENS specifications for real industrial embedded systems as noted in Section 5.2.In addition, TGSENS can generate test cases with 100% coverage for requirements of the given SENS specification by an exhaustive search using a SAT-based solver.TGSENS also supports feasible test generation by userspecified test filtering and test sequence generation.
The successive work of our project is the improvement of our test generator in order to deal with the composition of subsystems.One of the most important aspects of test generation based on formal specifications and models for industrial systems is "scalability." To overcome this problem, we plan to tackle a higher-speed and more scalable test generation using parallel computing.We also plan to find a constructive way of generating test suites of the whole system from test suites of subsystems as a future work.We also plan to apply the model checking technique to generate more targetspecific test cases and sequences.

B.1. Modeling of Subsystems
States For (  ).Let    = {V 1 , . . ., V  } denote a set of all variables used in the SENS specification for the subsystem   .That is,    is a set of all variables declared in the variable declaration of the class declaration for   , and the extern declaration of the class-file-declaration for   .For each variable V  (1 ≤  ≤ ), let (V  ) denote a set of all values of V  declared in the specification.When considering a test model, let (V  ) be a set of all values in the test declaration for V  .
In addition, we also define a set of variables with previous state operator "∼.
)  satisfies all conditions specified as invariants in the requirement declaration of the class declaration for   and the invariant declaration.
Hereafter we denote  ⊨ (V  =   ) if the value for V  is assigned to   at state .Otherwise  ⊭ (V  =   ) and thus  ⊨ (V  ̸ =   ) since each variable is assigned to one value at the same time.
Labels For (  ).Let (  ) be a set of all events and actions declared in the class declaration and the class-file-declaration for subsystem   .There are three kinds of labels: input labels corresponding events, output labels corresponding actions, and label .where in() denotes a set of input labels in .
Each  (∈ Σ   ) is a set of labels in a transition, and we call  a transition label.Note that each transition label can contain at most one input label and multiple output labels by the syntax of SENS.Transition label 0 in Σ   corresponds to .
Transitions For (  ).Consider a transition requirement ⟨ :  : , ⟩ where  is precondition,  is event,  is   C.2. Timer Declaration.We translate the timer declaration step-by-step considering the three parts: a subsystem side, a timer side, and a composition of the subsystem and the timers.First, on the subsystem side, we prepare four kinds of propositional variables as follows: (i) SubCountUp: a counting up event occurring in a subsystem is prepared; there exists only one variable of this kind for each subsystem; (ii) t?ring: a subsystem receives a counting over event from its timer t; (iii) t!clear: a subsystem does an action by sending a reset message to its timer t; (iv) t!start: a subsystem does an action by sending a startup message to its timer t.
We assume that no more than one event or action related to the same timer t occurs at the same time.We represent this constraint as follows: Next, on the timer side, we introduce auxiliary propositional variables for two events, an action, a counting-up event, and the states of each timer.The propositional variables prepared for timer t are as follows: (i) t?clear/t?start:for reset/start-up event from the subsystem, (ii) t!ring: an action of sending the counting over message, (iii) t?CountUp: a counting-up event of the timer, (iv) t#ready/t#run/t#over: the current timer state, (v) @t#ready/@t#run/@t#over: the next timer state.
Finally, we describe constraints of composition between a subsystem and timers.
(i) Events and actions of a subsystem are associated with actions and events of each timer t: (a) t!clear ⇔ t?clear; (b) t!start ⇔ t?start; (c) t?ring ⇔ t!ring.
(ii) For timers of each subsystem, all the timers satisfying their counting up conditions count up at the same time: (a) (SubCountUp ∧ op# on → @op# on) ∧(SubCountUp ∧ op# off → @op# off).

C.3. Channel Declaration.
For events through a channel, we prepare propositional variables.A variable denoted by c?m shows that the event of receiving message m through channel c occurs.In addition, for each subsystem, prepare one propositional variable representing that no event occurs (denoted by null#true).
No more than one event occurs at the same time in a subsystem where we consider the counting over of a timer and the counting up of a subsystem as events.We need to represent this constraint of event as logical formulas by using prepared variables.For example, consider the subsystem <A.cls> in Pseudocode 1.The subsystem has two events, a timer-over event and a counting up the timer.So, the constraint of events in <A.cls> is translated as follows: (i) t?ring ∨ SubCountUp ∨ null#true ; (ii) (¬t?ring ∨ ¬SubCountUp) ∧(¬t?ring ∨ ¬null#true) ∧(¬SubCountUp ∨ ¬null#true).
For actions as well as events, we prepare propositional variables.A variable denoted by obj.c!m shows that a subsystem takes the action of sending message m to a district subsystem obj througha receiver's channel c.
There is a constraint that a subsystem cannot take any action, if its counting-up event occurs.For example, consider the subsystem <A.cls>.Since it has two actions, the constraint of actions is expressed by the following formula: (i) (¬arg1.ch!m∨¬SubCountUp) ∧(¬arg2.ch!m∨¬SubCountUp).
C.4.Requirements.Basically, in the translation of requirements of a SENS specification, we only have to replace the requirements with logical formulas using the propositional variables prepared above.A requirement represented as a logical expression in SENS should be satisfied both in the current state and in the next state.Therefore, we translate the requirement into two logical formulas: one for the current state and the other for the next state.
The transition requirement of SENS specifications, "P: e: (Q, a), " represents a logical implication: "(P∧e) → (Q∧a)." The postcondition Q is a constraint for the next state, so we translate the condition into a logical formula using propositional variables for the next state.
All the variables of SENS including variables of numerical types take only a finite number of values.We start explaining the translation using the following examples, Var a: int [0,2]; b: int [0,2]; (i) comparison between a variable of SENS and a numerical value: (a) the equation a==0 is denoted by a propositional variable a#0, and the inequation a!=0 is denoted by a literal ¬a#0; (b) the inequality a<2 is translated to a logical formula a#0 ∨a#1.
(ii) comparison between variables: For general numerical comparison, we translate the requirement inductively.In particular, let exp 1 ∼ exp 2 be a general numerical comparison in SENS, where ∼∈ {≤, ≥ , =, ̸ = }, and exp 1 , exp 2 are expressions.We transform this requirement to logical formula as follows.First, we introduce two auxiliary variables V 1 and V 2 ; these variables represent exp 1 and exp 2 , respectively.The requirement exp 1 ∼ exp 2 is reduced to V 1 ∼ V 2 .Second, based on the domain of

Figure 1 :Figure 2 :
Figure 1: An air-conditioning system and its SENS specification.

Figure 4 :
Figure 4: A model composition for the entire ENS.

Figure 6 :
Figure 6: An example model for system   .

Figure 7 :
Figure 7: An example model for system   with a timer.
The main interface for generating test cases.Test sequenceGenerating a graphical resultSetting for a testing goal (b) The interface for generating test sequences.

Figure 10 :
Figure 10: A transition composition of subsystems.
Data types are int and double with ranges, and enumerated data types are defined in the Data part.In the Data part, a type is defined as a set of values.In SENS, only finite and discrete values are used.In the example<main.cls>,a global data type, onoff status, which has two elements on and off, is declared in lines 3-4.In <B.cls>, a local data type, m type, which has the only element m, is declared specification, the set of states    , the set of labels Σ   , and the set of transitions    are obtained by Definitions 15, 16, and 17 of Appendix B.1.
[2]mple 4. Consider the example specification of subsystem   in Pseudocode 1. Figure6illustrates the subsystem model for   which satisfies the specification.Given SENS specification for class , the states are ( ∼ op=off, op=off), ( ∼ op=off, op=on), ( ∼ op=on,op=on), and ( ∼ op=on, op=off) from variable op and previous state variable ∼ op used in the specification by Definition 15.By Definition 16, the labels are t!start, t?ring, B[1].ch!m, and B[2].ch!m with the events and actions used in the specification.Each transition in (  ) satisfies the two conditions in Definition 17.For example,  →   with  = ( ∼ op=off,op=off) and   = ( ∼ op=on,op=off) cannot be a transition of (  ), since it does not hold the condition for previous state variables; that is,  ⊨ (op=on), if and only if   ⊨ ( ∼ op=on) by Definition 17.
Sequence Generation 4.2.1.Concept.As mentioned in Section 4.1, our test generator can generate all possible test cases for a given SENS specification.Since the number of complete test cases can be very huge, it is hard to run tests by inputting all the test cases separately.Therefore, it is practical to extract test sequences and focus on specific variables interested for testing.To deal with this problem, we propose a test sequence generator for appropriate testing goals given by users.

Table 1 :
Variables in the example specification for function   .
where  1 is a state in which Mode A=Auto,  2 is a state in which Mode A=Off, and  3 is a state in which Mode A=Auto.

Table 2 :
Scales of example SENS specifications.

Table 3 :
Experiment results for test case generation.

Table 4 :
Experiment results for test sequence generation.