MPE Mathematical Problems in Engineering 1563-5147 1024-123X Hindawi Publishing Corporation 167453 10.1155/2012/167453 167453 Research Article A Numerical Algorithm on the Computation of the Stationary Distribution of a Discrete Time Homogenous Finite Markov Chain Zhao Di 1 Li Hongyi 1 Su Donglin 2 Wu Zheng-Guang 1 LMIB School of Mathematics and System Science Beihang University Beijing 100191 China buaa.edu.cn 2 Beihang Institute of EMC Technology Beihang University Beijing 100191 China buaa.edu.cn 2012 16 7 2012 2012 08 01 2012 14 03 2012 14 03 2012 2012 Copyright © 2012 Di Zhao et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

The transition matrix, which characterizes a discrete time homogeneous Markov chain, is a stochastic matrix. A stochastic matrix is a special nonnegative matrix with each row summing up to 1. In this paper, we focus on the computation of the stationary distribution of a transition matrix from the viewpoint of the Perron vector of a nonnegative matrix, based on which an algorithm for the stationary distribution is proposed. The algorithm can also be used to compute the Perron root and the corresponding Perron vector of any nonnegative irreducible matrix. Furthermore, a numerical example is given to demonstrate the validity of the algorithm.

1. Introduction and Preliminaries

Throughout this paper, the following notations and definitions are used. A matrix A=(aij)m×n is called nonnegative (positive), if all aij0 (aij>0), denoted by A0 (A>0). Similarly, a vector x=(x1,,xn)T is called nonnegative (positive) and denoted by x0 (x>0), if all xi0 (xi>0). Let B=(bij)m×n, we denote AB (A>B), if A-B0 (>0), that is, aijbij (aij>bij) for all i=1,2,,m,j=1,2,,n.

For a square matrix An×n with eigenvalues λ1,,λn,ρ(A)=max{|λj|} is called the spectral radius of A. If A0 is irreducible, there exists a unique eigenvector x=(x1,,xn)T>0 such that Ax=ρ(A)x and x1=|x1|++|xn|=1. In this case, we say that ρ(A) is the Perron root of A and x is the Perron vector .

We consider a discrete-time Markov chain X={X(n):n=0,1,} with a finite state space 𝒮={i1,,in}. Among ergodic processes, homogeneous Markov chains with finite state space are particularly interesting examples. Such processes satisfy the Markov property, which states that their future behavior, conditional to the past and present, depends only on the present. Precisely, for all t+, h>0, and for all sequences 0t1tr=t,i1,,ir𝒮 and ij𝒮, (1.1)P(X(t+h)=ijX(t)=ir,X(tr-1)=ir-1,,X(t1)=i1)=P(X(h)=ijX(0)=ir).

The behavior of such a process is characterized by an n×n matrix M called the transition matrix .

Its stationary distribution π, which is also its asymptotic distribution, is a vector satisfying the following. (1.2)πTM=πT,j=1nπj=1, that is, (1.3)πTM=πT,π>0,πTe=1, where e is the column vector of all ones.

It has been established that it is possible to represent all possible uses of a software system as a Markov chain . This model is called a Markov chain usage model. In a usage model, states of use (such as state “Document Loaded” in a model representing use of a word processing system) are represented by states in the Markov chain. Transitions between states of use (such as moving from state “Document Loaded” to “No Document Loaded” when the user closes a document in a word processing system) are represented by state transitions between the appropriate states in the Markov chain. Transitions between states of use have associated probabilities which represent the probability of making each transition. A usage model may be created based on information taken from functional specifications, usage specifications, and test objectives.

Considering the problem of software reliability, we represent a software system Sf with n states of use {s1,,sn} by a homogeneous discrete Markov chain {X(n):n=0,1,} (the corresponding transition matrix is M). We denote the initial state probability distribution π(0)=(π1(0),,πn(0))T, where πi(0)=P(X(0)=si). Then (π(k))T=(π(k-1))TM, where π(k) stands for the state probability distribution at time k. Let μi (i=1,,n) be the probability when the software fails at state si. The reliability of Sf at time k can be defined as R(k)=1-i=1nμiπi(k). After a long time running, the state distribution of system Sf will tend to the stationary distribution π=(π1,,πn)T. Then, the terminating reliability R=1-i=1nμiπi, with which we can evaluate the quality of a software system. By decreasing the μi of state si with the largest probability πi in the stationary distribution π, we can also enhance the reliability of Sf efficiently with limited resources.

A nonnegative matrix A=(aij)n×n is called a row-stochastic matrix (or a stochastic matrix for short) if j=1naij=1 for all i=1,,n, that is, Ae=e.

From the well-known Perron-Frobenius theorem, it can be easily deduced that the Perron root of a stochastic matrix A equals 1.

Obviously, the transition matrix M of a discrete-time homogeneous Markov chain is a stochastic matrix. From (1.3), we have MTπ=π. That is to say, the stationary distribution π is also an eigenvector of MT associated to 1. Since MT and M have the same eigenvalues, π is the Perron root of MT, that is, the solution to MTπ=π. As for the computational aspects of π, many approaches have been presented (e.g., see ) based on the Gaussian elimination, direct projection and so on. In this paper, from the viewpoint of the Perron root which has not been discussed, we propose an algorithm for the stationary distribution π considering that the computation of π is equivalent to the computation of the Perron vector of M, which not only can compute the stationary distribution, but also could be used to compute the Perron root and the corresponding Perron vector of any nonnegative irreducible matrix (noting that the stationary distribution is the Perron vector of the transition matrix, which is a special nonnegative matrix).

This paper is organized as follows. In the next section, we propose some lemmas and preliminary results. In Section 3, we prove the convergent theorem and give some facts. In Section 4, we propose an algorithm for the stationary distribution together with a demonstrating numerical example.

2. Some Lemmas

In this section, we present some lemmas which will be used in the proof of the main results. The following facts can be found in [1, 11, 12].

Definition 2.1 (see [<xref ref-type="bibr" rid="B1">1</xref>]).

Let An×n be a nonnegative matrix. If Am>0 for some integer m1, one says that A is primitive.

It is known that any primitive matrix must be irreducible . We will use the following important facts which can be found in [1, 12].

Theorem A (see [<xref ref-type="bibr" rid="B1">1</xref>]).

Let A=An×n0, then A is irreducible if and only if (I+A)n-1>0, where I is the unit matrix.

Theorem B (Perron-Frobenius (see [<xref ref-type="bibr" rid="B1">1</xref>])).

Let A=An×n0 be irreducible. Then,

ρ(A)>0,

ρ(A) is an eigenvalue of A,

There exists a vector x>0 such that Ax=ρ(A)x,

ρ(A) is a simple eigen value of A.

This theorem guarantees the eigenspace of ρ(A) is one-dimensional. That is, Ay=ρ(A)y implies y=kx. And there exists an unique positive vector x>0 whose components sum to 1 such that Ax=ρ(A)x. This x is called the Perron vector .

For the Perron root of nonnegative matrices, many algorithms and bounds estimations have been proposed (see in [13, 14]). In this paper, we will describe the Perron root by using the following Collatz-Wielandt functions [11, 12].

Definition 2.2 (see [<xref ref-type="bibr" rid="B11">11</xref>]).

Let A=(aij)n×n be nonnegative, define (2.1)fA(x)=min(Ax)jxj,gA(x)=max(Ax)jxj, for any positive vector x=(x1,,xn)T>0.

f A ( x ) and gA(x) are both continuous at any x>0.

Lemma 2.3.

Let A=An×n be nonnegative and irreducible. Then, for any x>0, fA(x) and gA(x) satisfy the following:

AxfA(x)x,AxgA(x)x,andfA(x)A1,

fA(tx)=fA(x),gA(tx)=gA(x)(t>0),

Ax>tx gives fA(x)>t; Ax<tx gives gA(x)<t,

If B0 is irreducible and AB=BA, let y=Bx, then fA(y)fA(x) and gA(y)gA(x).

Proof.

(1)–(3) are clearly true (see ). For (4), by AxfA(x)x, it follows that Ay=B(Ax)BfA(x)x=fA(x)y. This gives fA(y)fA(x). Similarly, gA(y)gA(x).

Lemma 2.4 (see [<xref ref-type="bibr" rid="B1">1</xref>]).

If B0 is primitive (Bm>0 for some m1), then (2.2)limk[ρ(B)-1B]k=L>0,BL=ρ(B)L.

3. Main Results

In this section, we will present the main results.

Theorem 3.1.

If A=An×n0 is irreducible and B=Bn×n0 is primitive such that AB=BA. Let x(0)=(a1,,an)T>0. Define for k=1,2,, (3.1)y(k)=Bx(k-1),x(k)=y(k)y(k)1. Then,

limkx(k)=x>0, and Ax=ρ(A)x with x1=1,

limkfA(x(k))=limkgA(x(k))=ρ(A),

fA(x(0))fA(x(1))ρ(A)gA(x(1))gA(x(0)).

Proof.

By (3.1), we can write x(k)=Bkx(0)/bk (for some bk>0). This means for k=1,2,(3.2)Bkx(0)bk1=x(k)1=1.

By Lemma 2.4, (3.3)limk[Bρ(B)]k=L>0,BL=ρ(B)L. Equations (3.2) and (3.3) imply that (3.4)limkbkρ(B)k=Lx(0)1,limkx(k)=limkBkx(0)bk=Lx(0)Lx(0)1. By putting x=Lx(0)/Lx(0)1, it is clear that x>0 (with x1=1) and (3.5)Bx=BLx(0)Lx(0)1=ρ(B)Lx(0)Lx(0)1=ρ(B)x. Since AB=BA, we get B(Ax)=ρ(B)Ax. The Perron-Frobenius theorem (Theorem B) guarantees that ρ(B) is a simple eigenvalue of B. So, B(Ax)=ρ(B)Ax gives that Ax=λx(x>0), which implies that λ=ρ(A) and Ax=ρ(A)x. On the other hand, by Definition 2.2, Ax=ρ(A)x(x>0) gives fA(x)=ρ(A)=gA(x). By limkx(k)=x(x>0), we conclude that (3.6)limkfA(x(k))=fA(x)=ρ(A),limkgA(x(k))=gA(x)=ρ(A). By Lemma 2.3 and (3.1), we have for k=1,2,(3.7)fA(x(k-1))fA(x(k)),gA(x(k))gA(x(k-1)). So, {fA(x(k))} and {gA(x(k))} are both monotonic convergent sequences. This proves (c), completing the proof.

Remark 3.2.

From the proof, we know ABx=ρ(A)ρ(B)x (x>0), and ρ(AB)=ρ(A)ρ(B).

For an n×n irreducible matrix A0, since (bI+A)n-1>0(b>0), B=(bI+A)m are primitive for m=1,2,. Clearly, BA=AB, we have the following.

Corollary 3.3.

If A=An×n0 is irreducible and let B=(bI+A)m (for fixed m1 and b>0). Let x(0)=(a1,,an)T>0. For all k=1,2,, define (3.8)y(k)=Bx(k),x(k)=y(k)y(k)1. Then,

limkx(k)=x>0,Ax=ρ(A)x,(x1=1),

limkfA(x(k))=limkgA(x(k))=ρ(A),

fA(x(0))fA(x(1))ρ(A)gA(x(1))gA(x(0)).

For a positive matrix A>0, all the matrices B=Am(m=1,2,) are primitive. The following is obvious.

Corollary 3.4.

If A=An×n>0 and B=Am (for m1). Let x(0)=(a1,,an)T be a positive vector. Define (3.9)y(k)=Bx(k),x(k)=y(k)y(k)1k=1,2,. Then,

limkx(k)=x>0,Ax=ρ(A)x,(x1=1),

limkfA(x(k))=limkgA(x(k))=ρ(A),

fA(x(0))fA(x(1))ρ(A)gA(x(1))gA(x(0)).

By (3.1), let λ(k)=(1/2)(fA(x(k))+gA(x(k))) for (k=1,2,), one has the following.

Corollary 3.5.

If gA(x(k))-fA(x(k))<ε(ε>0), then |ρ(A)-λ(k)|<(1/2)ε.

Proof.

From Theorem 3.1, it follows that (3.10)|ρ(A)-λ(k)|12|ρ(A)-fA(x(k))|+12|ρ(A)-gA(x(k))|=12(ρ(A)-fA(x(k)))+12(gA(x(k))-ρ(A))=12(gA(x(k))-fA(x(k)))<12ε.

If A0 is irreducible, it is obvious that B=bI+A(b>0) is primitive, and ρ(A)=ρ(B)-b. So, we have the following.

Corollary 3.6.

If A0 is irreducible, for any b>0, let B=bI+A. Using the sequences {fB(x(k))} and {gB(x(k))}. Then, (3.11)limkfB(x(k))=limkgB(x(k))=ρ(B);limkx(k)=x>0,(x1=1);ρ(A)=ρ(B)-b.

4. An Algorithm and a Numerical Example

In this section, we propose a numerical algorithm to compute the stationary distribution of a discrete time homogeneous finite Markov chain.

Algorithm 4.1 (to compute the stationary distribution <inline-formula><mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" id="M215"><mml:mrow><mml:mi>π</mml:mi></mml:mrow></mml:math></inline-formula>).

Step 1. Giving a transition matrix M of a discrete time homogeneous finite Markov chain, a calculation precision ε>0. Choosing parameters: a positive real number b>0 and an integer m. Setting the initial iterative vector π(0)=(1,1,,1)T, B=(bI+MT)m, k=1.

Step 2. Computing π(k) from π(k-1): (4.1)y(k)=Bπ(k-1),π(k)=y(k)y(k)1(k=1,2,).

Step 3. Compute fMT(π(k)) and gMT(π(k)): (4.2)fMT(π(k))=min1in(MTπ(k))iπi(k),gMT(π(k))=max1in(MTπ(k))iπi(k).

Step 4. If gMT(π(k))-fMT(π(k))<ε, go to Step 5. Otherwise setting k:=k+1, go back to Step 2.

Step 5. Let λ=(1/2)(fMT(π(k))+gMT(π(k))). Then λ is the approximation of the Perron root of MT, and the corresponding π(k) is the approximation of the stationary distribution of M.

Remark 4.2.

From Theorem 3.1, the convergence of Algorithm 4.1 is obvious.

We next give a numerical example.

Example 4.3.

For a given finite Markov Chain, with the corresponding transition matrix as the following: (4.3)M=[0.200.30.100.400.10.200.50.20.5000.100.400.30.20.20.10.20.30.1000.40.20.200.30.500], finding its approximating stationary distribution with calculation precision ε=10-6.

By choosing the initial iterative vector x(0)=(1,1,1,1,1,1)T, parameters m=3,b=1, that is, B=(I+MT)3, and applying Algorithm 4.1, the approximating Perron root and Perron vector are obtained after 12 iterations: (4.4)π(12)=(0.2115999,0.0758786,0.18776110,0.1956869,0.0958466,0.2332268)T,λ(12)=1.0000001. The iteration results are listed in Table 1.

Iteration results of Example 4.3 by Algorithm 4.1.

k y ( k ) π ( k ) f M T ( π ( k ) ) g M T ( π ( k ) )
1 9.8480005 0.2051667 0.8266823 1.0792825
4.2060003 0.0876250
8.2880001 0.1726667
8.1380005 0.1695417
6.8360000 0.1424167
10.6840000 0.2225833

2 1.7046144 0.2130768 0.9024122 1.0330924
0.6064956 0.0758119
1.4543651 0.1817956
1.4877499 0.1859687
0.8997051 0.1124631
1.8470705 0.2308838

3 1.7026380 0.2128298 0.9654347 1.0107353
0.6010615 0.0751327
1.4877805 0.1859726
1.5420293 0.1927537
0.8042196 0.1005275
1.8622712 0.2327839

4 1.6968167 0.2121021 0.9906667 1.0030047
0.6042080 0.0755260
1.4985833 0.1873229
1.4985833 0.1949336
0.7754812 0.0969351
1.8654419 0.2331802

5 1.6940787 0.2117598 0.9979988 1.0010511
0.6060677 0.0757585
1.5014149 0.1876768
1.5642191 0.1955274
0.7683222 0.0960403
1.8658978 0.2332372

6 1.6931415 0.2116427 0.9997041 1.0003293
0.6067572 0.0758447
1.5020157 0.1877520
1.5653062 0.1956633
0.7668935 0.0958617
1.8658856 0.2332357

7 1.6928755 0.2116094 0.9999617 1.0000869
0.6069643 0.0758705
1.5021036 0.1877629
1.5654967 0.1956871
0.7667158 0.0958395
1.8658446 0.2332305

8 1.6928117 0.2116015 0.9999923 1.0000242
0.6070167 0.0758771
1.5021019 0.1877628
1.5655103 0.1956888
0.7667347 0.0958418
1.8658243 0.2332281

9 1.6927999 0.2116000 0.9999974 1.0000122
0.6070276 0.0758785
1.5020945 0.1877618
1.5655029 0.1956879
0.7667580 0.0958448
1.8658174 0.2332272

10 1.6927987 0.2115998 0.9999989 1.0000043
0.6070291 0.0758786
1.5020909 0.1877613
1.5654980 0.1956872
0.7667685 0.0958460
1.8658154 0.2332269

11 1.6927989 0.2115999 0.9999996 1.0000013
0.6070290 0.0758786
1.5020894 0.1877612
1.5654960 0.1956870
0.7667719 0.0958465
1.8658147 0.2332269

12 1.6927993 0.2115999 0.9999998 1.0000004
0.6070290 0.0758786
1.5020891 0.1877611
1.5654955 0.1956869
0.7667729 0.0958466
1.8658148 0.2332268
Acknowledgment

The project was supported by the National Natural Science Foundation of China (Grant no. 60831001).

Horn R. A. Johnson C. R. Matrix Analysis 1985 Cambridge, UK Cambridge University Press xiii+561 832183 ZBL0591.68012 Regnault P. Estimation using plug-in of the stationary distribution and Shannon entropy of continuous time Markov processes Journal of Statistical Planning and Inference 2011 141 8 2711 2725 2787739 10.1016/j.jspi.2011.02.022 ZBL1213.62009 Poore J. H. Mills H. D. Mutchler D. Planning and certifying software system reliability IEEE Software 1993 88 99 Poore J. H. Trmmell C. J. Application of statistical science to testing and evaluating software intensive systems Statostics, Testing and Defense Acquisition 1998 Washington, DC, USA National Academy Press Whittaker J. A. Thomason M. G. Markov chain model for statistical software testing IEEE Transactions on Software Engineering 1994 20 10 812 824 2-s2.0-0028517307 10.1109/32.328991 Benzi M. A direct projection method for Markov chains Linear Algebra and Its Applications 2004 386 27 49 10.1016/j.laa.2003.12.019 2066606 ZBL1062.65011 Heyman D. P. Reeves A. Numerical solution of linear equations arising in Markov chain models ORSA Journal on Computing 1989 1 52 60 Marek I. Iterative aggregation/disaggregation methods for computing some characteristics of Markov chains. II. Fast convergence Applied Numerical Mathematics 2003 45 1 11 28 10.1016/S0168-9274(02)00232-5 1975125 ZBL1025.65012 Neumann M. Xu J. On the stability of the computation of the stationary probabilities of Markov chains using Perron complements Numerical Linear Algebra with Applications 2003 10 7 603 618 2030626 10.1002/nla.339 ZBL1071.65504 Paige C. C. Styan G. P. H. Wachter P. G. Computation of the stationary distribution of a Markov chain Journal of Statistical Computation and Simulation 1975 4 173 186 Minc H. Nonnegative Matrices 1988 New York, NY, USA John Wiley & Sons xiv+206 Wiley-Interscience Series in Discrete Mathematics and Optimization 932967 Varga R. S. Matrix Iterative Analysis 2000 27 Berlin, Germany Springer x+358 Springer Series in Computational Mathematics 1753713 Duan F. Zhang K. An algorithm of diagonal transformation for Perron root of nonnegative irreducible matrices Applied Mathematics and Computation 2006 175 1 762 772 2216377 10.1016/j.amc.2005.07.055 ZBL1095.65030 Li H. Y. Zhao D. Dai F. On the spectral radius of a nonnegative centrosymmetric matrix Applied Mathematics and Computation 2012 218 4962 4966