A Stochastic Model for the HIV / AIDS Dynamic Evolution

This paper analyses the HIV/AIDS dynamic evolution as defined by CD4 levels, from a macroscopic point of view, by means of homogeneous semi-Markov stochastic processes. A large number of results have been obtained including the following conditional probabilities: an infected patient will be in state j after a time t given that she/he entered at time 0 (starting time) in state i; that she/he will survive up to a time t, given the starting state; that she/he will continue to remain in the starting state up to time t; that she/he reach stage j of the disease in the next transition, if the previous state was i and no state change occurred up to time t. The immunological states considered are based on CD4 counts and our data refer to patients selected from a series of 766 HIV-positive intravenous drug users.


Introduction
In this paper the homogeneous semi-Markov reliability stochastic model is proposed as a useful tool for predicting the evolution of the human immunodeficiency virus (HIV) infection and the probability of an infected patient's survival.This model, when compared to the most common epidemiologic data analyses, has the following advantages: (i) not only is the randomness in the different states in which the infection can evolve into considered, but also the randomness of the time elapsed in each state; (ii) all the states are interrelated, therefore any improvements are also considered; (iii) a large number of disease states can be considered; (iv) fewer and less rigid working hypotheses are needed; (v) only raw data obtained from observations are needed, with no strong assumptions about any standard probability functions regarding the random variables analysed; (vi) the conclusions are simply based on a list of all computed probabilities derived directly from raw data.Semi-Markov processes were defined in the fifties independently of each other by Levy [1] and Smith [2].A detailed theoretical analysis of semi-Markov processes was produced in Howard [3,4].Since then, they have been applied in a number of scientific fields including: engineering applications (systems reliability) [3][4][5][6], finance [7], insurance, actuarial and demographic sciences [6,8,9].Semi-Markov models have also been employed in the field of biomedicine, for example, in applications to prevent, screen, and design cancer prevention trials, in Davidov [10], and Davidov and Zelen [11], respectively.
Moreover, many papers relating to HIV infection, have been written such as Lagakos et al. [12], Satten and Sternberg [13], Sternberg and Satten [14] and Sweeting et al. [15].Foucher et al. [16] also considered various patients based on their ages by means of a parametric approach.Joly and Commenges [17] reduced the instability of nonparametric models but introduced some strong assumptions in order to estimate a posteriori intensity functions by penalizing the log-likelihood.Apart from [16], in all the papers quoted, the model solvability is related to the possibility that a patient might move through the states following the same direction.Our data has shown that there are no negligible probabilities of recovering from the disease, and so, in our dynamic analysis the unidirectionality hypothesis for the transitions among the states was not considered.
Physicians consider that the HIV fully satisfies few and weak working hypotheses needed.Data refer to subjects selected from a series of 766 HIV-positive intravenous drug users screened at different Italian clinics in the period from October 1988 to December 1996.The cohort characteristics were described in [23].The computation is done by means of Mathematica software designed and written by some of the authors.

Homogeneous semi-Markov process
In this part, the homogeneous semi-Markov process (HSMP) will be defined and the notation will be as given in [24].
In the SMP environment, two random variables run simultaneously: X n with state space S = {S 1 ,...,S m } represents the state at the nth transition.In the health care environment, the elements of S represent all the possible stages in which the disease may show level of seriousness.T n , with state space equal to R, represents the time of the nth transition.In this way, we cannot only consider the randomness of the states but also the randomness of the time elapsed in each state.The process (X n ,T n ) is assumed to be a homogeneous Markovian renewal process, see [25].

Giuseppe Di Biase et al. 3
The kernel Q = [Q i j (t)] associated with the process is defined as follows: Thus, (Pyke [26]) where P = [p i j ] is the transition matrix of the embedded Markov chain in the process.Furthermore, it is necessary to introduce the probability that the process will leave state i in a time t as Obviously, It is now possible to define the distribution function of the waiting time in each state i, given that the state successively occupied is known, Obviously, the related probabilities can be obtained by means of the following formula: (2.7) The main difference between a continuous time Markov process and a semi-Markov process lies in the distribution functions G i j (t).In a Markov environment this function must be a negative exponential function.On the other hand, in the semi-Markov case, the distribution functions G i j (t) can be of any type.This means that the transition intensity can be decreasing or increasing.
If we apply the semi-Markov model in the health care environment, we can consider, by means of the G i j (t), the problem given by the duration of the time spent inside one of the possible disease states.Now the HSMP Z = (Z(t), t ∈ R) can be defined.It represents, for each waiting time, the state occupied by the process Z(t) = X N(t) , where N(t) = max n : T n ≤ t . (2.8) The transition probabilities are defined in the following way: They are obtained by solving the following evolution equations: where δ i j represents the Kronecker symbol.
The first addendum of formula (2.10) gives the probability that the system does not undergo transitions up to time t given that it was in state i at an initial time 0. In predicting the HIV/AIDS evolution model, it represents the probability that the infected patient does not shift to any new stage in a time t.In the second addendum, Qij (ϑ) is the derivative at a time ϑ of Q iβ (ϑ) and it represents the probability that the system remained in a state i up to the time ϑ and that it shifted to state β exactly at a time ϑ.After the transition, the system will shift to state j following one of all the possible trajectories from state β to state j within a time t − ϑ.In our application, it means that up to a time ϑ an infected subject remains in the state i.At the time ϑ, the patient moves into a new stage β and then reaches state j following one of the possible trajectories in some time t − ϑ.

A description of HSMP numerical solution.
In a previous paper, Corradi et al. [27] proved that it is easy to find the numerical solution of (2.10) by means of quadrature method.Moreover, they proved that the numerical solution of the process converges to the discrete time HSMP (DTHSMP).
Furthermore, in the same paper, it was proved that the DTHSMP process tends to be continuous if the discretization interval tends to 0. The discretization of formula (2.10) leads to the following infinite countable linear system: where h represents the discretization step (2.12) For more information on discretization see [28].Relation (2.11) can be written in the following matrix form: Giuseppe Di Biase et al. 5 If h = 1, we have: (2.14) The following theorems have been proved in [27].
Theorem From all these results it follows that the solution of an SMP can be obtained by means of the DTSMP.Furthermore, we are interested in solving the problem in a finite time span.The solution can be found by means of a simple recursive method.
As a first step, (2.13) for t = 0 gives

15)
Knowing Φ h (0), it is possible to compute Φ h (h).Knowing these two matrices, it is possible to compute Φ h (2h) and so on.

Homogeneous semi-Markov reliability model
There are several semi-Markov models in reliability theory, see for example, Osaki [29] and more recently Limnios and Oprisan [5].
Let us consider a reliability system S that may be at any given time t in one of the states of I = {1, ...,m}.The stochastic process of the successive states of S is Z = {Z(t), t ≥ 0}.The state set is partitioned into sets U and D in the following way: The subset U contains all "good" states in which the system is working and the subset D contains all "bad" states in which the system is not working properly or has failed.The typical indicators used in reliability theory are the following: (i) the reliability function R giving the probability that the system was always working from time 0 to a time t: (ii) the point-wise availability function A giving the probability that the system is working at a time t whatever happens in (0,t]: (iii) the maintainability function M giving the probability that the system will leave the set D within the time t being in D at time 0: It has been shown in [5] that these three probabilities can be computed in the following way if the process is a homogeneous semi-Markov process with kernel Q.
(i) The point-wise availability function A i given that Z(0) = i: (ii) the reliability function R i given that Z(0) = i.
To compute these probabilities, all the states of the subset D must be changed into absorbing states.R i (t) is given by solving the evolution equation of HSMP with the embedded Markov chain with p i j = δ i j if i ∈ D. The resulting formula is where φ i j is the solution of (2.10) with all the states in D that are absorbing.
(iii) The maintainability function M i given that Z(0) = i.
In this case, all the states of the subset U must be changed into absorbing states.M i (t) is given by solving the evolution equation of HSMP with the embedded Markov chain with p i j = δ i j if i ∈ U.The resulting formula is where φ i j (t) is the solution of (2.10) with all the states in U that are absorbing.

Application of the model to the HIV/AIDS dynamic evolution
The acquired immunodeficiency syndrome (AIDS) is caused by the human immunodeficiency virus (HIV), a virus belonging to the lentivirus subgroup of retroviruses [30,31].The hallmark of the HIV infection is the progressive depletion of a class of lymphocytes named CD4+ or helper lymphocytes which play a pivotal regulatory role in the immune response to infections and tumours.The immune suppression resulting from the CD4+ decline leads to high susceptibility to opportunistic infections and possibly unusual tumours.Without appropriate antiretroviral treatment, AIDS is almost uniformly lethal [30,31].
The natural history of HIV infection is characterized by a phase of latency that can last for several years, and evolves through consecutive steps [32] defined on the basis of CD4+ lymphocyte count and constitutional symptoms [33] with full blown AIDS representing the final stage of the disease [34].The time spent in each stage of the disease is not predictable on the basis of clinical and immunological parameters.
HIV is transmitted primarily by sexual contact, syringe sharing amongst intravenous drug users, blood and blood products not properly screened.From an epidemiological point of view, the disease has spread worldwide.It is currently estimated that the total number cases of HIV infections is some 39.5 million, with a peak in the sub-Saharan African continent, and East Asian countries [35].Physicians believe that the fundamental hypothesis needed in order to apply the model in HIV/AIDS environment is satisfied.Indeed, as quoted in [36] the relation (2.6) is nearer to reality, that is, in the absence of treatment, the future of the patient depends only on the present state but not on all previous history.
Followup took T = 87 months (from October 1989 to December 1996).The retrospective study concerned a cohort of K = 766 HIV-positive intravenous drug users.Database fields were completed by means of a number of biological and clinical parameters obtained from 2488 medical examinations.In order to predict the HIV/AIDS evolution, we employed the following immunological states related to CD4+ count plus an absorbing state (the death of the patient): state I (CD4 > 500 × 10 6 cells/L), state II (350 < CD4 ≤ 500), state III (200 < CD4 ≤ 350), state IV (CD4 ≤ 200), and state D (absorbing state).We assume, therefore, that the HIV/AIDS infection shifts between five different degrees of seriousness.This choice was justified by the CDC immunological classification [33], and taking into account the recommendations of the DHHS (Department of Human and Health Services) for the initiation of antiretroviral therapy [37].
All that led to the following set of states: The first four states are working states (good states) and the last one is the only bad state.This is represented in the following two subsets: In this case, the maintainability function M does not make sense because the default state D is absorbing and once an infected patient had entered this state it was no longer possible to leave it.Furthermore, the fact that the only bad state is an absorbing state implies that the availability function A corresponds to the reliability function R. Another important result that can be obtained by means of the semi-Markov approach is the distribution function of the subject's death conditioned to the state held at time 0.
In the health care environment, the reliability model is substantially simplified.In fact, to obtain all the results that are relevant to our study it suffices to solve the system (2.11) numerically only once since φ = φ i j (t) = φ i j (t).
In order to obtain the claimed results, we need to estimate the semi-Markov kernel Firstly, we introduce the following symbols: (i) K is the number of independent trajectories in our data set; (ii) X r n is the state at nth transition of the rth subject; (iii) T r n is the time in which the rth subject makes the nth transition; (iv) N r = N r (T) = sup{n ∈ N : T r n ≤ T} is the total number of transitions held by the rth subject; is the number of visits of the rth subject to the state i; r=1 N r i is the total number of visits of all subjects to the state i.Then we consider the empirical kernel estimator defined in [21] by In [21] it was proved that the empirical kernel estimator is uniformly strongly consistent and, properly centralized and normalized, it converges to the normal random variable.
Owing to lack of space, we do not show the kernel estimates, but we can make them available upon request.We report, in Table 4.1, the frequencies of the transitions between the states and, consequently, in Table 4.2, the estimates of the embedded Markov chain.
Obviously the obtained estimates Q i j (t,K) are used as input to estimate all the relevant variables listed in Section 5.

Numerical results
After solving the evolution equations of the semi-Markov model with kernel Q, an extensive amount of information useful to a phisician can be obtained, including the following.
(1) φ i j (t), that represents, for each t, for each j ∈ {I, II, III, IV, D}, and for each i ∈ {I, II, III, IV} the probabilities of being in a state j after a time t given that she/he entered at time 0 (starting time) in the state i.In Figure 5.1, there is a graphical representation of such conditional probabilities.For the sake of brevity, only the values corresponding to lapses of sixteen months and up to month 88 are reported.They are all, however, available on request.It seems superfluous to underline the medical relevance of such computed probabilities.For example, if an HIV infected patient is in the third stage of the disease, with 21% risk, after 52 months he will be in the fourth stage (see Figure 5.1, Month 52).
(2) R i (t) = A i (t) = j∈U φ i j (t), that represents the conditional probabilities, given the starting state, that an infected patient will survive up to a time t.R i (t) gives a physician vital information.In Figure 5.2, four curves, which depend on the starting state of the subject, have been computed.For example, if we look at the lowest curve we can read R i=IV (42) = 0.8 and we may conclude that, with a probability equal to 0.8, an infected patient that was in state IV will not die after 42 months.
(3) 1 − H i (t) represents the conditional probabilities of staying in the starting state until month t.In Figure 5.3 these conditional probabilities have been computed depending on the starting state.For example, if an HIV-infected patient comes under study at the fourth stage of the disease, with 40% risk, after 24 months he will still be in the fourth stage.
Before giving another result of current interest for epidemiologic purposes that can be obtained in an SMP environment, the concept of the first transition after time t must be introduced.More precisely, it is supposed that a subject at time 0 was in state i and it is known that with probability (1 − H i (t)) he does not shift from state i.Under these hypotheses, it is possible to know the probability of the next transition is to state j.This probability will be denoted by ϕ i j (t).In terms of formulas it means the following: ( This probability can be estimated by means of the following formula: After definition (5.1) by means of SMP, it is possible to obtain the following result.(4) ϕ i j (t) represents the estimated probability of developing stage j of the disease at the next transition if the previous state was i and no change of state occurred up to time t.In this way, in the case we studied, if the patient does not shift for a time t from state i, the probability of him being dead in the next transition can be computed easily.In Figure 5.4, a graphical representation of the probabilities of the first transition after a time t is shown.As for φ i j (t), only the values corresponding to lapses of sixteen months are reported.They are all, however, available on request.A physician might read the probability of moving into state j of the disease (for each j ∈ {I, II, III, IV, D}) at the next transition if the previous state occupied was i (for each i ∈ {I, II, III, IV}) and no change occurred up to month t (for each t).

Concluding remarks
In this paper we have presented an HSMP approach to the dynamic evolution of the Human Immunodeficiency Virus Infection, as defined by CD4+ levels, and the probabilities of an infected patient's survival.By means of this approach, we cannot only consider randomness in the possible stages of seriousness which the disease may show but also the randomness of the duration of the waiting time in each state.The model starts from the idea that the disease evolution problem can be considered a special type of reliability problem and this idea allows the application of some semi-Markov reliability results to a healthcare environment.
We would like to point out that this paper does not show all the potential of the semi-Markov environment.Indeed, by means of the backward recurrence time process it is possible to assess different transition probabilities as a function of the duration inside the states.Moreover, it is possible to attach a reward structure to the process that allows the possibility of doing a cost analysis that considers, for example, the cost of antiretroviral treatment and/or other social costs related to the dynamic evolution of the HIV infection.These features will be the object of future research.

Figure 4 . 1 .
Figure 4.1.The model of the immunological stages a HIV/AIDS infected patient can go into.

Figure 4 . 1
Figure 4.1 represents the graph model.It shows all the immunological states an HIV/AIDS infected patient can undergo.All the states, apart from D, are interrelated, and also improvements are taken into account.It is also possible that an examination will show that a patient's state has not changed.The first four states are working states (good states) and the last one is the only bad state.This is represented in the following two subsets:

Table 4 .
1. Transition frequencies matrix of the followed-up cohort and estimates of the transition matrix.

Table 4 .
2. Estimates of the transition matrix of the embedded Markov chain.Figure 5.1.Conditional probabilities of being in state j after a month t given the starting state i.The starting states are in the axis categories.
Figure 5.2.Survival conditional probabilities up to month t given the starting state.Figure 5.3.Stay on conditional probability in the starting state at least for a time t.