A New Software Reliability Growth Model: Multigeneration Faults and a Power-Law Testing-Effort Function

Software reliability growth models (SRGMs) based on a nonhomogeneous Poisson process (NHPP) are widely used to describe the stochastic failure behavior and assess the reliability of software systems. For these models, the testing-effort effect and the fault interdependency play significant roles. Considering a power-law function of testing effort and the interdependency of multigeneration faults, we propose a modified SRGM to reconsider the reliability of open source software (OSS) systems and then to validate the model’s performance using several real-world data. Our empirical experiments show that the model well fits the failure data and presents a high-level prediction capability. We also formally examine the optimal policy of software release, considering both the testing cost and the reliability requirement. By conducting sensitivity analysis, we find that if the testing-effort effect or the fault interdependency was ignored, the best time to release software would be seriously delayed and more resources would be misplaced in testing the software.


Introduction
Software systems are thought to run successfully when they perform without any failure during a certain period of time.However, in practice, failures are frequently observed and a perfect operational process cannot always be guaranteed due to a variety of reasons.Software reliability, which is defined as the probability of failure-free operation of a software program for a specified time interval, is an essential consideration in the processes of software development and a critical factor in measurement of software quality.In fact, accurately modeling the software reliability growth process and predicting its possible trend are of high values for software developers.For these reasons, a bunch of software reliability growth models (SRGMs), most of which are based on a nonhomogeneous Poisson process (NHPP), have been constructed [1][2][3][4][5].Unlike closed-source software, open source software's (OSS) knowledge base is not bounded to a particular organization.As a result, reliability measurement is important and complicated for OSS, such as Linux operating systems, Mozilla browser, and Apache web server.Due to their unique properties, a parallel of SRGMs has been purposefully developed to quantify the stochastic failure behavior of OSS [6,7].
One key factor of software reliability growth modeling is the testing-effort effect [8][9][10][11][12][13][14].Huang and Kuo [15] incorporated a logistic testing-effort function into software reliability modeling and showed that it provides powerful prediction capability based on real failure data.Huang [16] considered a generalized logistic testing-effort function and changepoint parameters in software reliability modeling and some practical data were employed to fit the new models.Ahmad et al. [17] incorporated the exponentiated Weibull testingeffort function into inflection S-shaped software reliability growth models.Peng et al. [18] established a flexible and general software reliability model considering both testingeffort function and imperfect debugging.
In these models, assuming independence among successive software runs may not be appropriate [19][20][21].For instance, Goševa-Popstojanova and Trivedi [22] introduced a software reliability model based on Markov renewal processes that covers interdependence among successive software runs.The Compound-Poisson software reliability model presented 2 Mathematical Problems in Engineering by Sahinoglu [23] considered multiple failures that occur simultaneously.Singh et al. [24] proposed new SRGMs that assume the presence of two types of faults in the software: leading and dependent.Leading faults can be removed when a failure is observed, but dependent faults are masked by leading faults and can be removed only after the corresponding leading faults are removed.
We also notice that a bundle of prior papers have examined the problems of optimal release time under different criteria.One stream studies this issue by minimizing the total cost subject to a constraint on software reliability.For example, Huang and Lin [25] provided a series of theoretical findings regarding when a software version should be released and when fault dependency and debugging time lag are considered.In addition, Ahmad et al. [26] analyzed the optimal release timing in an SRGM with Burr-type X testing-effort functions.Another stream adopts the multiattribute utility theory (MAUT) to decide the release time.For instance, Singh et al. [27] built an SRGM based on the Yamada two-stage model and focused on two different attributes to investigate the release time.Pachauri et al. [28] studied an SRGM incorporating a generalized modified Weibull testingeffort function in an imperfect debugging environment and the optimal release policy is examined using both genetic algorithm (GA) and MAUT.
In this work, we propose a new model that incorporates a power-law testing-effort function and three generations of interdependent errors.Specifically, the detection process for the first-generation errors is independent, and the detection rate is proportional to the instantaneous testing-effort expenditure and the mean of remaining first-generation errors in the system.By contrast, the detection process of the second-generation errors depends on the first generation and the detection rate is proportional to not only the instantaneous testing-effort expenditure and the mean of remaining second-generation errors but also the ratio of removed first-generation errors.Similarly, the detection process of the third-generation errors relies on the secondgeneration errors and the detection rate is proportional to the instantaneous testing-effort expenditure, the mean of remaining third-generation errors, and the ratio of removed second-generation errors.
Model parameters are estimated by the nonlinear least square estimation method, which is realized by minimizing the sum of squares of the deviations between estimated values and true observations.Numerical experiments are carried out based on three versions of Apache.The estimation results show that the proposed model well fits the real observations and performs better than the traditional Goel-Okumoto model and the Yamada delayed S-shaped model.Comprehensive comparisons in terms of prediction capabilities among various models are conducted to reveal that our model can predict the error occurrence process more accurately than the benchmarks.We also analyze the optimal release policy and the minimal cost for the software development managers, based on which insightful suggestions are provided.Besides, we conduct sensitivity analysis and find that resources would be misplaced if either the testing-effort effect or the error interdependency is overlooked by the software testing team.
Unlike the benchmarks, our results suggest that the strategy of an earlier release time can substantially reduce the cost.

Basic Assumptions.
To describe the stochastic failure behavior of software systems, we define {(),  ≥ 0} as a counting process that represents the cumulative number of failures by time .We use the mean value function (MVF) () to denote the expected cumulative number of detected failures in the time period (0, ].That is, where [⋅] means the expectation operator and (  ) is called failure intensity function which indicates the instantaneous fault-detection rate at time   .Specifically, for an NHPP-based SRGM, it is assumed that () follows a Poisson distribution with a mean value function ().That is, the model can be mathematically characterized as where Poim(, ()) is the Poisson probability mass function (pmf) with mean () [29,30].Generally, one can obtain different NHPP models by taking different mean value functions.
In this paper, to characterize the stochastic behavior of the software failure process and obtain the expression for the mean value function, we make the following assumptions: (i) The software system is subject to failures at random times caused by errors remaining in the system.(ii) The error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process.(iii) Each time a failure occurs, the error causing it is immediately removed and no new errors are introduced.(iv) The faults existing in the software are of three generations (G1, G2, and G3) and each generation is modeled by a different growth curve.(v) The mean number of G1 errors detected in the time interval (,  + Δ) is proportional to the instantaneous testing-effort expenditure and the mean number of remaining G1 errors in the system.(vi) The mean number of G2 errors detected in the time interval (,  + Δ) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G2 errors in the system, and the ratio of removed G1 errors.(vii) The mean number of G3 errors detected in the time interval (,  + Δ) is proportional to the instantaneous testing-effort expenditure, the mean number of remaining G3 errors in the system, and the ratio of removed G2 errors.
(viii) Fault-detection/removal rate is a power-law function of testing time for all three generations of errors.
The first four assumptions in the above list are the building blocks in most prior software reliability growth models and the others are specific in our model.To facilitate our later analysis, we summarize the notations used throughout this paper in Notations section.
2.2.Proposed Software Reliability Growth Model.In general, an implemented software system is tested to detect and correct software errors in the development process.During the testing phase, software errors remaining in the system are discovered and corrected by test personnel.Based on the aforementioned assumptions, the detection rate must be proportional to the instantaneous testing-effort expenditure.Particularly, unlike traditional closed-source software, in open source software, instructions are executed by various concurrent users and each release of open source software can attract an increasing number of volunteers in the software development and/or code modification procedure.Hence, it is reasonable to assume the detection/correction rate as a power-law function of testing time [19,24].That is, the instantaneous testing-effort expenditure at time , (), takes the following form: where  and  are parameters to be estimated by real-world data sets.As such, we can quickly obtain the cumulative amount of testing-effort expenditures by time ; that is, As stated above, it is essential to consider fault interdependency when modeling software reliability growth processes.Specifically, in this work we propose three generations of interdependent errors.That is, the detection of the secondgeneration errors relies on the first generation, and the detection of the third-generation errors depends on the second generation.Moreover, the second-generation errors are detectable only if the first-generation errors have been removed, and the third-generation errors are detectable only if the second-generation errors have been removed.The flow chart is shown in Figure 1 to illustrate the error detecting process in a software system.
Next, we sequentially examine the mean value function for each generation of errors: (i) For the first-generation errors in the software system, the number of detected errors is proportional to the number of remaining errors and the instantaneous testing-effort expenditure.Thus, the mean value function for G1 errors satisfies the following differential equation: where  1 is the initial content of G1 errors in the system and  1 () represents the expected number of G1 errors detected in time (0, ].Specifically,  1 () is assumed to be a bounded nondecreasing function of  with the initial condition  1 (0) = 0. Solving the above differential equation yields (ii) The mean value function for G2 errors satisfies the following differential equation: where  2 is the initial content of G2 errors in the system and  2 () stands for the expected number of G2 errors detected in time (0, ].Specifically,  2 () is also assumed to be a bounded nondecreasing function of  with the initial condition  2 (0) = 0. Solving the above differential equation yields (iii) Similarly, the mean value function for G3 errors satisfies the following differential equation: where  3 is the initial content of G3 errors in the system and  3 () means the expected number of G3 errors detected in time (0, ].Specifically,  3 () is assumed to be a bounded nondecreasing function of  with the initial condition  3 (0) = 0. Solving the above differential equation leads to After this, we formulate the overall mean value function () that consists of mean value functions for three generations of errors.Assume that  =  1 +  2 +  3 is the expected total number of faults that will be eventually detected and  1 ,  2 , and  3 are corresponding ratios for each generation of errors at the beginning of testing.By noting that As such, () is a bounded nondecreasing function of , with (0) = 0 and (∞) = .
Then, based on the aforesaid mean value function, the failure intensity function for the proposed model, with testing-effort, can be explicitly given as We further note that when  = 0,  1 = 1, and  2 =  3 = 0, the original Goel-Okumoto model can be realized [31]; that is, which is one of our benchmarks for the comparative analysis.
In this case, errors are assumed to be independent and the testing-effort function keeps constant, that is, .Besides, the mean value function () is a concave function with respect to .Moreover, when  = 0, we have () = ; namely, the instantaneous testing-effort expenditure at time  reduces to a constant.In this case, we have and we call it No- model for brevity.
In addition, when  1 = 1 and  2 =  3 = 0, the error interdependency is ignored and only one generation of errors exists in the software system.As a result, we have We call it Only-G1 model for exposition.
As another benchmark, hereby we introduce an S-shaped model, which incorporates an increasing detection rate function and well explains "learning effect" of the testing team.Essentially, the mean value function is stated as follows: It is also called Yamada delayed S-shaped model [32,33] in literature.For this model, the detection rate first increases and then decreases with .

Estimation Method.
The nonlinear least square estimation (NLSE) method is a widely used estimation technique, especially for small/medium sized data sets [34].We will use this technique to estimate parameters in models raised in this paper.More specifically, estimation is done by minimizing the sum of squared residuals, which are the squares of the differences between estimated values and true observations.The objective function for the minimization problem is stated as follows: where   is the th observation,  is the total number of observations,   is the time in which the th error occurs, and (  ) is the corresponding estimated/theoretical value.Specifically, in this paper, (  ) can be from ( 11), ( 13), ( 14), (15), or (16).They correspond to our proposed model, the Goel-Okumoto model, the No- model, the Only-G1 model, and the Yamada delayed S-shaped, respectively.Differentiating  2 with respect to each unknown parameter and setting the partial derivatives to zero, we can obtain the estimated values [34,35].Specifically, for our proposed model, we have Using numerical methods to solve the set of simultaneous equations above yields the estimation for unknown parameters in our proposed model.

Metrics for Model Comparison.
With estimated parameters, we can conduct comparative analysis among models based on the following metrics: (1) Goodness of Fit ( 2 ): the value of  2 determines how well the curve fits the real data [19,36].It is defined as follows: It measures the percentage of the total variance about the mean accounted for the fitted curve.It ranges from 0 to 1.
The larger  2 is, the better the model fits data.In this work, the model is considered to provide a good fit if  2 is above 0.95.
(2) Mean of Square Error (MSE): the value of MSE measures the average of squares of the differences between the predicted values and the actual values [9].It is defined as follows: Generally, the closer to zero the MSE value is, the better the model fits the data.
(3) Theil Statistic (TS): the value of TS indicates the average percentage deviation with respect to the actual failure data during the selected time period [37].It is defined as follows: A smaller TS value means a smaller deviation and a better prediction capability.In this work, the model is considered to provide a high-level prediction if the TS value is below 10%.(4) Relative Error (RE): the value of RE measures the relative difference between the predicted value and the real observations [25,38].The definition can be stated as follows:

Estimation Results and Model
Comparison.The parameter estimation can help characterize properties of theoretical models, determine the number of errors that have already been detected/corrected in the testing process, and predict the number of errors that will be eventually encountered by users.Specifically, by minimizing the objective function in (17), we can obtain the estimated results for each model.
(i) For Apache 2.0.35, the estimated results are presented in Table 2, with â = 74.8102,b = 0.0432, p1 = 0.5905, p2 = 0.2838, and k = 0.3554 for our proposed model.Then, the ratio of G1 errors to all types of errors is approximately 59%, the ratio of G2 errors is around 28%, and the leftover are G3 errors.Note that the errors consist of not only independent errors (G1), although G2 and G3 account much less than G1.Through incorporating such interdependency among generations of errors, we are able to strengthen the understanding towards composition of software system.Besides, we find that the estimated value of  is obviously larger than 0. Hence, not considering the effect of testing effort in the analysis of the reliability growth process may lead to significantly biased results.
Illustrations for the theoretical values versus their observations are shown in Figure 2. We find that the theoretical values are highly consistent with the real-world observations.We can further examine the failure intensity function ().Figure 3 shows that the failure intensity function is a singlepeaked curve; that is, the error detection rate first increases and then decreases with time.It in turn implies that our proposed model falls into the S-shaped category for the SRGMs [34].
We observe that our proposed model well fits the realworld software failure data.This can be seen from the high value of  2 , that is, 0.9954, which is very close to 1.By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No- model, and the Only-G1 model,  2 values are the highest in our proposed model.This conclusion can also be drawn by comparing MSE values   among the three models; our proposed model provides the lowest MSE values (2.0979).From this aspect, the model in our paper best fits the observed data about Apache 2.0.35.
Furthermore, the prediction capability in our model is very high.The low TS value for Apache 2.0.35,0.0252, is much lower than 10% and well confirms a high prediction capability for our model.Compared with the benchmarks, our model can provide significantly lower TS values and higher predication capacity to the failure of Apache system.Lastly, the relative error in prediction using this data set is computed and results are plotted in Figure 4. We observe that the relative error approaches 0 as   approaches   and the relative error curve is usually within ±5%.
(ii) For Apache 2.0.36, the estimated results are presented in Table 3, with â = 49.4122,b = 0.0045, p1 = 0.4031, p2 = 0.2855, and k = 0.8967 for our proposed model.Three generations of errors account for 40%, 29%, and 31%, respectively.Obviously, G2 and G3 significantly contribute to the error composition, which counts for 60% in total.We also find that the estimated value of  is much larger than 0 implying that the testing-effort effect is necessary.
Then, we can obtain theoretical values in our proposed model.Illustrations for the theoretical values versus their observations are presented in Figure 5, which shows that the theoretical values are consistent with the real-world data.In Figure 6, we present the failure intensity function () based on the estimated values.Similar to Apache 2.3.35, the hump-shaped curve shows that this function possesses an increasing-then-decreasing trend.
We find that our proposed model also well fits the realworld software failure data for Apache 2.0.36.This can be seen from the high value of  2 , 0.9806.It is also much higher   Furthermore, the TS value for Apache 2.0.36 is as low as 0.0452, which is much lower than 10% and well confirms a high prediction capability of our model.Compared with the benchmarks, our model shows a lower value of TS and a higher level of predication capability.Lastly, the relative error in prediction for this data set is computed, as shown in Figure 7.We observe that the relative error approaches 0 as   approaches   and the curve is usually within ±5%.
(iii) For Apache 2.0.39, the estimated results are presented in Table 4, with â = 57.1499,b = 0.0300, p1 = 0.5299, p2 = 0.3820, and k = 0.1940 for our proposed model.The ratios of three generations of errors become about 53%, 38%, and 9%, respectively.Then, the ratio for G1 errors is approximately 59%, the ratio for G2 errors is around 28%, and the leftover are G3 errors.We also find that the estimated value of  is significantly greater than 0. Thus, ignoring the effect of testing effort in the analysis of the software reliability growth curve would lead to significant bias.Comparison between the fitted curve and the observed failure data is illustrated in Figure 8, which indicates that the theoretical values and the real-world observations are highly consistent.In Figure 9, we present the failure intensity function based on the estimated values.Obviously, it is also a single-peaked curve: the error detection rate first increases and then decreases with time.
The high value of  2 , 0.9937, suggests that our proposed model well fits the real-world software failure data for this version.By comparing with the Goel-Okumoto model, the Yamada delayed S-shaped model, the No- model, and the Only-G1 model,  2 values are the highest in our proposed model.This conclusion can also be drawn by comparing MSE values among the models; our proposed model provides the lowest MSE value (0.5314).Therefore, our proposed model best fits the observed data about Apache 2.0.39.
Furthermore, the TS value for Apache 2.0.39 is 0.0144.It is much lower than 10% and well confirms a high prediction capability for our model.Compared with the benchmarks, TS values in our model are significantly lower, and prediction capacity of the failure occurrence is also much greater.Lastly, the relative error in prediction for this data set is computed and results are plotted in Figure 10.We observe that the relative error approaches 0 as   approaches   and the error curve is usually within ±5%.In summary, all of three tests using different versions of Apache failure data sets suggest that our proposed model well fits the real data and presents better performance than the benchmarks in error occurrence prediction.Besides, both the testing effort and the error interdependency effects play an important role in the parameter estimation.

Software Release Policy
The theoretical modeling can also determine whether the software is ready to release to users or how much more effort should be spent in further testing.In this section, we formally examine the software developer's best release time and deliver sensitivity analysis regarding how the test effort effect and error interdependency affect the optimal time and the cost.

Optimal Release Time.
When the failure process is modeled by a nonhomogeneous Poisson process with a mean value function, (), a popular cost structure [15,25,27,39] is applied as where  is the release time of the software to be determined.In addition,  1 is the expected cost of removing a fault during the testing phase,  2 is the expected cost of removing a fault during the operation phase, and  3 is the expected cost per unit time of testing.Specifically, the expected cost of removing a fault during the operation phase is higher than that during the testing phase; that is,  2 >  1 .Overall, the above cost function is the summation of the testing cost, including the failure cost during testing and actual cost of testing, and the cost of failure in the field.
Proof of Proposition 1.Note that the objective function is as follows: The first-order condition yields which indicates   ( * ) =  3 /( 2 −  1 ).
It is also reasonable to quit testing as soon as the number of remaining faults in the system is less than a prescribed portion of total faults.Hence, a simple criterion for software release is given [8,25]: which is the type-I reliability definition.Alternatively, given that the testing or the operation has been conducted up to time , the probability that a software failure will not occur in the time interval (,  + ), with  ≥ 0, can be given by the following conditional probability [3,40]: which is the type-II reliability definition.In most situations, the project managers need to strike a balance between the cost and the reliability level.That is, on one hand, the reliability level should be at least as high as a desired level set beforehand; on the other hand, given the target level of software reliability , the cost minimization problem over the release time  should be imposed.Considering this tradeoff yields  1 () ≥  (28) for the type-I reliability definition or for the type-II reliability definition.As an illustrative example, Figure 11 presents how the software reliability and the total cost change with time.In this example, we use the data set with Apache 2.0.36 and take  = 5,  1 = 20$,  2 = 150$, and  3 = 15$.Suppose that the reliability must be higher than 0.95; that is,  = 0.95.Then, the reliability constraint can be satisfied only if  ≥  1 = 45.99 for the type-I reliability definition (or  ≥  2 = 66.95 for the type-II reliability definition).When only the total cost is considered, we find that the cost is minimized at   = 54.37.Furthermore, when both criteria (reliability and cost) are taken into account, we find that  * = 54.37 for the type-I reliability definition (or  * = 66.95 for the type-II reliability definition).
In addition, for Apache 2.0.

Sensitivity Analysis.
As mentioned above, we have incorporated both the testing-effort and error interdependency effects into our proposed model.In this part, we intend to examine what if the testing effort, the error interdependency, or both of them are ignored in decision-making of software release.The first case corresponds to the No- model ( 14), the second case corresponds to the Only-G1 model (15), and the third case is just the Goel-Okumoto model (13).
For each model, we shall examine the optimal release time and the corresponding minimal cost when a desired reliability level is set beforehand.The data set with Apache 2.0.36 is used to illustrate our findings.Under the type-I reliability definition for reliability measurement, the optimal release time and the minimal cost as a function of  are presented in Figures 12 and 13, respectively.Similarly, under the type-II reliability definition, the optimal release time and the minimal cost as a function of  are shown in Figures 14  and 15, respectively.Other parameters take the same values as analysis in Section 4.1; that is,  = 5,  1 = 20$,  2 = 150$, and  3 = 15$.
Under either type-I or type-II reliability definition, both the optimal release time and the total cost would be mistakenly estimated if either testing effort or fault interdependency is ignored.Unlike these two cases, our proposed model always suggests an earlier release time and a lower testing cost.
As an example, we compare the Goel-Okumoto model and our proposed model when the type-I reliability definition is adopted and  is set as 95%.The Goel-Okumoto model indicates the best release time to be 76.56 while that for our proposed model is 54.3.Hence, the testing lasts much longer if the Goel-Okumoto model is used to decide the stopping time.Similarly, using the Goel-Okumoto model yields the minimal cost as 2529$ while that for our proposed model is just 1896$, implying that a decision based on our model can save the software developers as much as 25% in cost.
Alternatively, when the type-II reliability definition is adopted, the Goel-Okumoto model predicts the best release time as 132.2 while that for our proposed model is 66.75, which suggests that the testing duration in our model is doubled compared with traditional Goel-Okumoto model.Similarly, according to the Goel-Okumoto model, the minimal cost is 3083$ while that for our proposed model is merely 2000$.Thus, a decision based on our model can save the software developers as much as 35% in terms of the cost.

Concluding Remarks
In the development process, a variety of tests need to be conducted to ensure the reliability of the software system.In this work, we propose a framework for software reliability modeling that simultaneously captures effects of testing effort and interdependence between error generations.In order to evaluate our proposed model, we compare its performance with other existing models by doing experiments with actual software failure data, that is, three versions of Apache.The nonlinear least square estimation technique is applied for parameter estimation, and values of  2 , mean of square error, Theil Statistic, and relative error are used for model comparison in terms of fitting quality and prediction capability.The numerical analysis from all three Apache versions demonstrates that the proposed model fits the real-world data very well and exhibits a high level of prediction capability.
Our theoretical results can help software project managers assess the most cost-effective time to stop software testing and to release the product.Specifically, we have formally examined the release timing and provided useful managerial insights based on the Apache data sets.Moreover, our proposed model recommends an earlier release time and a reduced cost compared with the Goel-Okumoto model and the Yamada delayed S-shaped model.Therefore, our modeling essentially enhances accuracy of the understanding towards the software reliability growth process and the best time to release a software version.

Figure 1 :
Figure 1: A flow chart for the error testing process when multigeneration faults exist in a software system.

Figure 3 :
Figure 3: Failure intensity function with respect to time, with estimated values based on Apache 2.0.35.

Figure 5 :
Figure 5: Failure intensity function with respect to time, with estimated values based on Apache 2.0.36.

Figure 7 :Figure 8 :
Figure 7: Failure intensity function with respect to time, with estimated values based on Apache 2.0.39.

Figure 11 :
Figure 11: The reliability and the cost as a function of time for Apache 2.0.36.Solid line: cost; dash-dot line: type-I definition of reliability; dash line: type-II definition of reliability; dot line: target level of reliability.

Figure 12 :Figure 13 :
Figure 12: Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36.The type-I reliability definition is adopted for such analysis.Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No- model; dash line: the Goel-Okumoto model.

Figure 14 :Figure 15 :
Figure 14: Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36.The type-II reliability definition is adopted for such analysis.Solid line: our proposed mode; dot line: the Only-G1 model; dash-dot line: the No- model; dash line: the Goel-Okumoto model.

Table 1 :
Detected faults (days from release and number of errors) for three versions of Apache, 2.0.35/2.0.36/2.0.39.Note that some failure data is not shown due to page limit.For example, as no faults are detected on the 42nd day in Apache 2.0.35,no failure data is shown on this day.

)
Specifically, assuming that we have observed   failures by the end of test at time   , the failure data up to time   (  ≤   ) will be used to estimate parameters in the models.Substituting the estimated parameters in the mean value function yields the estimation of the number of failures ( ) by   .The estimation is then compared with the actual number   .Such procedure is repeated for various values of   .Positive values of error indicate overestimation and negative values indicate underestimation.
35, the reliability level is above 0.95 only if  ≥  1 = 30.91 if the type-I reliability definition is