^{1}

^{1}

^{2}

^{1}

^{1}

^{2}

Model checking and conformance testing play an important role in software system design and implementation. From the view of integrating model checking and conformance testing into a tightly coupled validation approach, this paper presents a novel approach to detect latent errors in software implementation. The latent errors can be classified into two kinds, one is called as Unnecessary Implementation Trace, and the other is called as Neglected Implementation Trace. The method complements the incompleteness of security properties for software model checking. More accurate models are characterized to leverage the effectiveness of the model-based software verification and testing combined method.

In software engineering practices, model-based software development and analysis methods receive extensive attention [

However, only applying model checking followed by conformance testing is not fully satisfactory. The essential reason is that model checking focuses on an accurate system formal model, not considering the system implementation, while conformance testing focuses on checking whether the system implementation behaves as the model specified, not considering whether the key properties are totally tested [

Three kinds of effects for the traditional software validation method.

In order to integrate model checking and conformance testing into a tightly coupled validation approach, two kinds of studies have been done. First, most studies in the literature focus on fixing the “Under Tested” problem [

To demonstrate the motivation of our work more clearly, we make further analysis first. Without loss of generality, we adopt an instance model checking method based on the Input-Output Labeled Transition System (IOLTS) models [

The paper is organized as follows. Firstly, certain preliminaries and related work are discussed in Section

In this section, we first introduce the formal definition of the IOLTS model and the basic ideas about the IOCO relation-based conformance testing method. Then, we discuss several related and important studies on how to integrate model checking and conformance testing technologies in recent literature.

An IOLTS model is actually an LTS model with explicitly specified input and output actions [

An IOLTS model is a four-tuple IOLTS =

For an IOLTS model

Traces

An IOLTS model for the specification of the AVM software is presented in Figure

The specification and implementation IOLTS models for the AVM software.

The IOCO relation-based testing approach has well-defined theoretical foundation [

Its intuitive idea is to compare output, that produced after executing trace

As for related studies about integrating model checking and conformance testing technologies in the literature, most of them focus on fixing the “Under Tested” problem, which is mentioned in the above section. That is, key properties are involved in the test case generation methods, so executing such set of test cases could guarantee that all key properties are tested. The VERTECS research team at INRIA [

Based on the discussion in Section

The iterative refinements in the integrated verification and testing method.

In this paper, the LIED method is regarded as an integrated software validation method. It is essentially well compatible with the traditional model checking and conformance testing procedures. That is, the LIED method is developed as a complementary method for detecting latent implementation errors, not for replacing the traditional model checking and conformance testing. So, the LIED tends to be an accelerant, because its central merit is to construct more accurate system formal models, which are quite helpful to promote the effectiveness of model checking and the model-based conformance testing. We could perform model checking, the conformance testing, and the LIED process iteratively and complementarily. Consequently, these validation methods work collaboratively to make software design and implementation more effective and more efficient.

As discussing the motivation of this paper in Section

Before presenting the detailed LIED method, we start with formally defining the specific kinds of latent implementation errors that we want to detect and repair in this paper. The concept of Trace in the IOLTS modeling (refer to Definition

For an IOLTS MS (software specification model) and an IOLTS MI (software implementation model), where

UIT (MI)

The obvTraces operator restricts the original traces

Clearly, in this case, if we do not consider a specific key property against this unnecessary implementation error in the model checking phase, the verification does pass. After then, the IOCO conformance testing will also pass, because none of test cases, which are generated from verified system model, are capable of detecting such implementation error. Consequently, though an unnecessary implementation error does exist in the system implementation, both model checking and conformance testing do not detect this fatal error, and this kind of errors indeed should be repaired.

For an IOLTS MS (software specification model) and an IOLTS MI (software implementation model), where

NIT (MI)

To sum up, we aim to detect and repair two kinds of latent implementation errors, that is, the unnecessary implementation errors and the neglected implementation errors, which may not be detected using the traditional model checking and IOCO conformance testing combined method. They have the same fatal effects, but they result from different causes. Therefore, in the LIED method, we utilize the unified method to detect these two kinds of latent implementation errors, but fix them using respective methods.

As we discussed in Section

The central goal of our LIED method is to find out some unnecessary implementation errors or neglected implementation errors, where no evident clues are provided by the system models. So, designing the LIED method has two necessary preconditions. First, we need to check the system implementation. Second, we need to check as many key properties as possible against the system implementations. Consequently, we adopt model checking as the basic technology and apply it directly into the system implementations. That is, the central idea of our LIED method is to explore model checking towards the actual software implementation to check whether some kinds of latent implementation errors exist and then utilize the counterexamples, which illustrate the exception behavior executions, as guidance to improve the system models and implementations.

The framework of the LIED method is shown in Figure

The framework of the LIED method.

(a)

(b)

(c)

The LIED method is developed for detecting latent implementation errors, and it is well compatible with traditional model checking and conformance testing procedures. The major advantages of our LIED method lie in two aspects. First, it complements the incompleteness of the key properties for the software validation. More importantly, it benefits constructing a more accurate system formal model to promote the effectiveness of the model-based software verification and testing sequentially combined method. Specifically, the original model checking could be performed more completely, because the key properties and the system models are both improved, and the conformance testing could be performed more precisely, because the improvement of the system models results in better test cases, which are generated with higher accuracy and stronger error-detecting capability. Consequently, the LIED method improves the “Under Verified” scenario as expected.

In order to detect more latent implementation errors proactively and focus on more necessary functional behaviors for a software system, we perform analogy enumeration based on the original set of key properties, which are constructed in the traditional model checking phase. According to the survey of patterns in property specifications [

The analogy set specifications for five property patterns.

Pattern name | Property specification | Analogy set specification | Explanations |
---|---|---|---|

Absence | LTL: |
LTL: |
OS: action |

| |||

Existence | LTL: |
LTL: |
OS: action |

| |||

Universality | LTL: |
LTL: |
OS: action |

| |||

Precedence | LTL: |
LTL: |
OS: the occurrence of action |

Response | LTL: |
LTL: |
OS: the occurrence of action |

As we want to detect more latent implementation errors proactively, the above analogy set for properties is used from two aspects in the LIED method. On one hand, if one specific property is verified in original model checking process, the properties of its analogy set should be paid more attention and correspondingly checked against the system implementations. On the other hand, taking the cause-effect relation into account, the absence, existence, and universality properties could be classified as a group, while the precedence and response properties as another group. So, if one specific property is verified in the original model checking process, the other kind of properties and its analogy set properties should be also considered to have a check against the system implementations. In this way, we complement the incompleteness of key security properties for the software model checking, and more importantly, we have more opportunities to find out the unnecessary implementation errors or the neglected implementation errors.

According to the framework of our LIED method in Figure

First, we present the formal definition of the counterexample from the LTS point of view, and it could be concretized with corresponding syntax towards different model checkers.

A Unified Counterexample is a kind of Trace of an IOLTS model

Intuitively speaking, a counterexample is a specific execution of a software system, that is, a trace of detailed behaviors. The model

Based on the counterexample and the software model, a Counterexample-Guided Synchronous Simulation (CGSS) algorithm is proposed as Algorithm

Given

(1)

(2) ct could be specified as:

(3)

/* omit all internal actions in the counterexample trace */

ct_new =

/* based on ct _new trace to perform synchronous simulation with MS model */

/* the external trace of counterexample acts the same as the trace in MS, exit to report non-empty set */

/* omit considering internal actions and go for the next iteration */

/* the same external action leads to one synchronous simulation step, and go for the next iteration */

/* the different external action leads to failure simulation, and exit to report empty set */

} // end of for each pair

} // end of procedure

If the unnecessary implementation errors are detected, the system model should be improved by adding additional parts to deal with the UIE errors that demonstrated by the counterexamples, and, the system implementations should be fixed by taking out extra program codes. Similarly, if the neglected implementation errors are detected, the system model should be modified against corresponding parts to handle the NIE errors that demonstrated by the counterexamples, and, the system implementation should be fixed by revising the inaccurate program codes.

To elaborate the feasibility and effectiveness of our LIED method, an AVM software system is analyzed using this method as a representative in this section. The Copper model checker is adopted. The IOLTS model for this AVM system is presented in Figure

In traditional model checking phase, we consider a requirement that if a coin is inserted, the milk or coffee is released. We formulate this requirement into an LTL property with precedence format. Besides, its analogy set is generated, which checks the scenario that the milk or coffee is released without inserting a coin. Original property is formulated as

Copper supports temporal logic claims expressed in State/Event Linear Temporal Logic (SE-LTL). The syntax of SE-LTL is similar to that of LTL, except that the atomic formulas are either actions or expressions involving program variables. Therefore, the analogy property could be formulated as follows:

The program for the AVM system is processed into the AVM.pp file and the above property is specified into the AVM.spec file. Then, the model checking towards the program is executed using the following command.

copper

The result of this LTL model checking is “conformance relation does not exist !! specification ExamProp is invalid….” Besides, a counterexample is produced correspondingly. As follows, a detailed UCE trace

In this trace,

We put above UCE trace

The improved IOLTS models for the AVM software.

Through the above exemplified execution of our LIED method towards the AVM system, its feasibility and effectiveness are demonstrated. That is, certain latent implementation errors are detected, and more importantly, the system models and implementations are well improved.

To validate the functional accuracy for a software system only applying model checking followed by conformance testing may not detect some latent implementation errors, that is, the unnecessary implementation errors and the neglected implementation errors. In this paper, the LIED method is proposed to detect such latent implementation errors proactively. Based on the analogy set of key properties, the LIED method applies model checking directly into the actual software implementation to check whether some latent implementation errors exist and utilize the counterexamples, which illustrate the exception behavior executions as intuitive guidance to improve the system models and system implementations respectively.

The LIED method is essentially well compatible with the traditional model checking and model-based conformance testing procedures. It could be applied as an effective complementary method for detecting latent implementation errors, but not for replacing the traditional model checking and conformance testing. The major advantages of our LIED method could be concluded from two aspects. First, it efficaciously complements the incompleteness of the key security properties for the software validation process. Second, it helps to construct more accurate system formal models to promote the effectiveness of model checking and model-based conformance testing, that is, the original model checking could be performed more completely because the key properties and the system models are both improved, and conformance testing could be performed more precisely because the improvement of the system models result in generating test cases with higher accuracy and stronger capability of detecting the implementation errors. In a word, the LIED method tends to be a well accelerant for better model checking and conformance testing iterative executions, where the “Under Verified” scenario is improved as expected, and consequently, these software validation methods work collaboratively to make software design and implementation more effective and more efficient. In the future, the LIED method will try to work in more complex and practical systems [

This work was supported by the National Natural Science Foundation of China (61262082, 61262017, 61163011, and 60973147), the Key Project of Chinese Ministry of Education (212025), the Inner Mongolia Science Foundation for Distinguished Young Scholars (2012JQ03), the Introduction Foundation for High-Level Talents of Inner Mongolia University, the Doctoral Fund of Ministry of Education of China