A Multistep Extending Truncation Method towards Model Construction of Infinite-State Markov Chains

Themodel checking of Infinite-State Continuous TimeMarkov Chains will inevitably encounter the state explosion problem when constructing the CTMCs model; our method is to get a truncated model of the infinite one; to get a sufficient truncated model to meet the model checking of Continuous Stochastic Logic based system properties, we propose a multistep extending advanced truncationmethod towardsmodel construction of CTMCs and implement it in the INFAMYmodel checker; the experiment results show that our method is effective.


Introduction
Continuous Time Markov Chains (CTMCs) have been used in various areas of research as a formalism; so far, the model checking of CTMCs has been a hot research topic in computer science research communities.Some algorithms and implementations have been shown in several papers and tools [1][2][3][4][5][6][7].However, our research aims to the model checking problem of the Infinite-State CTMCs, which means that the states of the CTMCs in our interest can be infinite.The papers [8][9][10][11] and the tool INFAMY [7] are with the same interest, which are also our research basis.Due to the explosion of states of CTMCs, our approach is based on a truncated CTMC model, which is determined by the exploration on the fly [9,10], which means that the depth of model is computed dynamically by the exploration of states.
The truncation process is involved with reachability analysis [9,10]; that is, the transient probability is computed with the exploration of the model.The transient probability is carried out by uniformization method [10].Together with the transient analysis when constructing the model, computation is very heavy, so to get a sufficient truncated model to meet the requirement of related Continuous Stochastic Logic (CSL) property or certain precision as fast as possible is our destination.We introduce a multistep extending advanced truncation method to meet this end, and the experimental results show our method is effective.
The main contents of the paper are organized as follows.Section 2 introduces the truncation based reachability analysis; Section 3 introduces an advanced truncation algorithm and do experiments.Section 4 proposes some multistep extending solutions and experiments.Section 5 talks about our result and next job.

Truncation Based Reachability Analysis
2.1.Finite Truncations.Let C = (, R, ) be a CTMC, where  is a countable set of states, R : ( ×  → R ≥0 ) is the rate matrix, and  :  → 2  is a labeling function.
First, we introduce some paths and probabilistic measures from [9].A (timed) infinite path is an infinite sequence  =  1  1  2  2 . . .satisfying R(  ,  +1 ) > 0, and   ∈ R ≥0 for all  = 1, 2, . ... For the path  and  ∈ , let [] =   denote the ( + 1)th state, and let (, ) =   denote the time spent in   .For   ∈ R ≥0 , let @ denote [] such that  is the smallest index with  ≤ ∑  =0   .For C, let path C ∞ denote the set of all paths, and let path C ∞ () denote the set of all paths starting from .For state  ∈ , a probability measure, denoted by Pr C  , on the set Pr C  () can be defined.A finite path is a finite sequence  =  1  1  2  2 . . .  for  > 0 satisfying R(  ,  +1 ) > 0 and   ∈ R ≥0 for  = 1, 2, . . .,  − 1.Let len() =  − 1 denote the length of the path, first() =  1 denote the first state, and last() =   denote the last state of the path.Let path C  denote the set of all finite paths.We omit the superscript C if it is clear from context.Next, we introduce the notion of depth.Let  0 be a finite subset of  with depth 0; that is, () = 0 for  ∈  0 .For now, one may think of  0 as being equal to the support of the initial distribution .However,  0 can be an arbitrary finite set.This will allow us not only to deal with the initial distribution, but also to compute truncation depths for nested CSL formulas.The depth of state  corresponds to the minimal distance from the set  0 .Definition 1.For C and  0 ⊂ , the depth function  :  → N is defined by Observe that   0 () = 0 for all  ∈  0 .The subscript is omitted if  0 is clear from the context.Intuitively, () corresponds to the minimal length of any finite path starting from  0 and ending in .
We consider a partition of the state space  = ⋃ ∈N   , where   := { ∈  | () = } is the set of states with depth .We say that the set   is the layer with depth  and call its elements layer- states.Assume that ⊥∈  is a special state not in  and furthermore ⊥ is also an atomic proposition not in .For  ∈ N, let  > ⊂  denote the set of states with depth greater than ; that is,  > = { | () > }.
The -truncation of an infinite CTMC is illustrated in Figure 1.Intuitively, the transition matrix is restricted to the truncated state space |  , and ⊥ is the distinguished absorbing state, which, by construction, is only reachable from states with depth .In state ⊥ only the atomic proposition ⊥ holds, which indicates that the system is in state ⊥.Since we consider finitely branching CTMCs, not surprisingly, the -truncated CTMC C|  is always finite.The absorbing state ⊥ has been introduced to abstract  > .We assume that ⊥∉ Sat(Φ) for any state formula Φ.We consider the probability of reaching the absorbing state ⊥ in the -truncated CTMC C|  , that is, ⃗  C|  (, )(⊥).For mere notational convenience, we extend ⃗  C|  to states of C with depth higher than  : ⃗  C|  (, )(  ) = 0 for all   ∈  with (  ) > .For a fixed , we define the forward rate fr  () of a state  ∈  ≤ within C|  .For  ∈  ≤ \   , it is the sum of the rates that go into the next layer fr  () = R(,  ()+1 ), and, for  ∈   , it is the sum fr  () = R(,  > ) of the rates entering states in   .

The Logic CSL.
The logic we consider is CSL without steady-state operator and unbounded until operator [10].Let = [,   ] be an interval with ,   ∈ R ≥0 and  ≤   .Let  ∈ [0, 1] and ⊴∈ {≤, <, >, ≥}.The syntax of state formula Φ and path formulas  is The semantics of the state formulas and path formula are precisely defined in [10].In this paper, the state formula like the kind of P ⊴ () will be more focused on; it means that starting some state, the probability of the system that satisfies a path formula  would meet the rational relation ⊴  or not, so the first solution to this problem is to compute this probability, so this is reduced to transient analysis, which is our main interest, to improve the efficiency of transient analysis for the whole model while extending the border states layer by layer.
The truncated model in our method is given by dynamically exploring from the initial state(s); once we want to add a layer on it, we need to compute the reachability probability from the initial(s) to the current layer states which we want to add.As shown in Figure 1,  0 is the initial states set, and each of  1 ,  2 , . . .,   is explored dynamically, so we get a depth truncated model of the infinite one.⊥ is the absorbing state, which is the border states set, for which the sum of the reachability probabilities from the initial state(s) is less than ;  is the precision of the result, which can be 10 −6 , 10 −9 , or 10 −12 , and so forth, which can be set under INFAMY model checker as a circumstance variable for some certain need.
The reachability probability is carried out by the uniformization method to the CTMC, which is a relatively fast end while (14) compute P  (◊ [0,] ) (15) end while (16) end procedure Algorithm 1: An advanced truncation algorithm towards construction of CTMC.method to get the transient probability at a certain bound time  at some state, which is denoted as P(◊ [0,] ).However, if the state explosion situation is very serious, the time to construct the model layer by layer would be of much cost.So we introduce an advanced truncation algorithm to explore the states, this method can improve the efficiency of the model constructing and model checking, and then at Section 4, we further propose some multistep extending solutions, which are implemented based on INFAMY model checker, the experiment results show that these solutions can help to improve the efficiency.

An Advanced Truncation Algorithm and Experiments
The truncation process is implemented by extending the states layer by layer from the initial states, and all the new states need to be transient analyzed and then to be extended further no matter how small the probability is.For that the precision of the result is under some certain value, so some states with relatively nearly no contribution to the result can be omitted when extending; thus, an advanced truncation algorithm is introduced.It is different from the finite state projection (FSP) [9,10] and layered chain and uniform chain method [9,10].The algorithm is shown in Algorithm 1.
The algorithm aims to stop the extending of less important (small probability) states and proceed to the extending of much important (large probability) states; the less or more is determined by the state reduction policy; as shown in Figure 2, line 7, P   (◊ [0,]   0 ) ≤ P   (◊ [0,]   1 ) ≤ P   (◊ [0,]   2 ) ≤ ⋅ ⋅ ⋅ ≤ P   (◊ [0,]    ) and ∑ P   (◊ [0,]    ) < .The policy means that the states in the border states set, which has been sorted upward,   0 ,   1 ,   2 , . . .,    , will be excluded from the extending states set, for the sum of them is just exactly less than the precision.So with this policy, the number of states to be transient analyzed will be smaller to the FSP and forwardlayered based model.
We consider the dependability of a fault-tolerant workstation cluster which is directly taken from case studies of [7]. Figure 2 depicts a dependable cluster of workstations.The cluster consists of two subclusters, which, in turn, contain  workstations connected via a central switch.The two switches are connected via a backbone.Each component of the system can break down and is then fixed by a single repair unit responsible for the entire system.Hereby, the quality of service (QoS) constraint minimum requires at least  ( < ) workstations to be operational, where  = ⌊(3/4)⌋.Workstations have to be connected via switches.If in each subcluster the number of operational workstations is smaller than , the backbone is required to be operational to provide the required service.We consider the property.P =?◊[0, ]¬Minimum.This probability means that the QoS drops below minimum quality within  time unit.
For the property, we compare PRISM [6], FSP method and layered method of INFAMY.The results are given in Tables 1, 2, and 3.Because the resulting probabilities are very small in some cases, we use a precision of 10 −6 here, for the computation of the truncation point.Results for INFAMY are given for the layered chain, FSP, and advanced configurations, respectively.The uniform chain configuration is omitted, as it is always dominated by the layered chain configuration.PRISM implements three different engines: a sparse-matrix and two symbolic engines.We used the sparse-matrix engine  as it was the fastest one.The results are shown in Tables 1, 2, and 3.
The experiment conditions are shown as follows.
Guest Machine: it includes Virtual Machine Software: VirtualBox 4.1.14r77440 for mac; Virtual OS: Linux ubuntu12.04LTS 32Bit; Processor: 2.2 GHz Intel Core 2 Duo T7500; Storage: 512 MB From Tables 1, 2, and 3, we can see that, for  ≤ 20, FSP based INFAMY is faster, but for  ≤ 30,  ≤ 50, INFAMY model checker needs more time; this is because the transient analysis when constructing the model needs more computing.This is also the result of [11].For the advanced method of the context, as in Tables 1, 2, and 3, comparing advanced with FSP, we can get that, under advanced based method, the depth of model is much deeper, and the states number is smaller; for the time costing, for  ≤ 20,  ≤ 30, the costing is less reduced, but for  ≤ 50, the costing is greater; this result is reasonable, for the exploration policy is essentially undeterministically efficient for different models; for the current case, when  ≤ 50, the current states number is very large, even with the reduction policy, with no contribution for it any more.We need to take other techniques to tackle this situation.Thus, we propose a multistep extending solution.See Section 4.

Multistep Extending Solutions and Experiments
The mutistep extending solution aims to reduce the extending of less important (small probability) states and enhance the extending of more important (large probability) states; for the latter states, we can, for example, extend two or more steps per extending, and for the former states, we can, for example, extend one step per extending, as we know that transient probabilities will be computed once again before the states were added to the border states, so, if we extend two or more steps, transient analysis at the intermediate states will be omitted; thus, time on the model construction will be reduced; then we can make the model much faster to converge to the absorbing state.We continue with the upper case study.As shown in Algorithm 1, Line 11,  = {() |  ∈ } is to extend the model from set  and get the border states set .This means that the advanced method in the upper section is extending one step per extending.Now, we design some multistep extending solutions: (1) two-step extending solution, that is,  = {(()) |  ∈ }; (2) three-step extending solution, that is,  = {((())) |  ∈ }; (3) synthesis solution 1: a synthesis extending solution separates set  to three parts, as set  is a sorted set, which is sorted upward by the probabilities, so we can separate  as  1 ,  2 , and  3 , and the sizes of each part are the same.Then we can get The experiments data are shown in Table 4. From the data in Table 4, we can see that under different extending solutions, the times needed are different, synthesis solution 2 performs better, when  ≤ 50, and the time (including model constructing and model checking) has been reduced to 1000 s.This means that for this case, the step per extending should be relatively small; for synthesis solution 2, half small states extend one step per-extending, and other half large states extend two steps per extending.And for this, we continue to propose a solution to revise synthesis solution 2, named Solution D the policy is as follows: For the states  0 ,  1 , . . .,   in set , the gap between the largest and smallest probabilities: ProbGap = Prob(  )−Prob( 0 ).We can separate set  to two parts  1 and  2 , such that.(iii)  =  1 ∪  2 .
The experiment results are as shown in Table 4. From Table 4, we can see that current solution D performs better on this case.

Conclusion
The multistep extending advanced truncation method can improve the efficiency of model construction of Infinite-State CTMCs; this is because the transient probabilities of states which have been jumped have not been computed, so to some extent this method is effective; however, which solution performs better needs to be experimented; there is no general solution that fits well for all cases.The efficiency is determined by the iterations when computing the transient probability.Less iteration is more efficient.However, this approach is essentially a linear approach to improve the efficiency; when the outsider state gets explosion, this approach will be less effective; just as in our case study, this approach can be used effectively to improve the model checking efficiency at a relatively small time bound .For future work, we need to consider other techniques to tackle the state explosion problem on model checking of CTMCs.And other works like [12][13][14][15] can also be considered.

Table 1 :
Experiments of forward-layered method of INFAMY.

Table 2 :
Experiments of advanced method of INFAMY.

Table 3 :
Experiments of PRISM and FSP method of INFAMY.

Table 4 :
Experiment data under different extending solutions.=  1 ∪  2 ∪  3 ;(4) synthesis solution 2 separates set  to two parts, so we can separate  to  1 and  2 , and the sizes of the two parts are the same.Then we can get