Software reliability growth models (SRGMs) based on a nonhomogeneous Poisson process (NHPP) are widely used to describe the stochastic failure behavior and assess the reliability of software systems. For these models, the testing-effort effect and the fault interdependency play significant roles. Considering a power-law function of testing effort and the interdependency of multigeneration faults, we propose a modified SRGM to reconsider the reliability of open source software (OSS) systems and then to validate the model’s performance using several real-world data. Our empirical experiments show that the model well fits the failure data and presents a high-level prediction capability. We also formally examine the optimal policy of software release, considering both the testing cost and the reliability requirement. By conducting sensitivity analysis, we find that if the testing-effort effect or the fault interdependency was ignored, the best time to release software would be seriously delayed and more resources would be misplaced in testing the software.
Software systems are thought to run successfully when they perform without any failure during a certain period of time. However, in practice, failures are frequently observed and a perfect operational process cannot always be guaranteed due to a variety of reasons. Software reliability, which is defined as the probability of failure-free operation of a software program for a specified time interval, is an essential consideration in the processes of software development and a critical factor in measurement of software quality. In fact, accurately modeling the software reliability growth process and predicting its possible trend are of high values for software developers. For these reasons, a bunch of software reliability growth models (SRGMs), most of which are based on a nonhomogeneous Poisson process (NHPP), have been constructed [
One key factor of software reliability growth modeling is the testing-effort effect [
In these models, assuming independence among successive software runs may not be appropriate [
We also notice that a bundle of prior papers have examined the problems of optimal release time under different criteria. One stream studies this issue by minimizing the total cost subject to a constraint on software reliability. For example, Huang and Lin [
In this work, we propose a new model that incorporates a power-law testing-effort function and three generations of interdependent errors. Specifically, the detection process for the first-generation errors is independent, and the detection rate is proportional to the instantaneous testing-effort expenditure and the mean of remaining first-generation errors in the system. By contrast, the detection process of the second-generation errors depends on the first generation and the detection rate is proportional to not only the instantaneous testing-effort expenditure and the mean of remaining second-generation errors but also the ratio of removed first-generation errors. Similarly, the detection process of the third-generation errors relies on the second-generation errors and the detection rate is proportional to the instantaneous testing-effort expenditure, the mean of remaining third-generation errors, and the ratio of removed second-generation errors.
Model parameters are estimated by the nonlinear least square estimation method, which is realized by minimizing the sum of squares of the deviations between estimated values and true observations. Numerical experiments are carried out based on three versions of Apache. The estimation results show that the proposed model well fits the real observations and performs better than the traditional Goel-Okumoto model and the Yamada delayed S-shaped model. Comprehensive comparisons in terms of prediction capabilities among various models are conducted to reveal that our model can predict the error occurrence process more accurately than the benchmarks. We also analyze the optimal release policy and the minimal cost for the software development managers, based on which insightful suggestions are provided. Besides, we conduct sensitivity analysis and find that resources would be misplaced if either the testing-effort effect or the error interdependency is overlooked by the software testing team. Unlike the benchmarks, our results suggest that the strategy of an earlier release time can substantially reduce the cost.
To describe the stochastic failure behavior of software systems, we define
In this paper, to characterize the stochastic behavior of the software failure process and obtain the expression for the mean value function, we make the following assumptions: The software system is subject to failures at random times caused by errors remaining in the system. The error detection phenomenon in software testing is modeled by a nonhomogeneous Poisson process. Each time a failure occurs, the error causing it is immediately removed and no new errors are introduced. The faults existing in the software are of three generations The mean number of The mean number of The mean number of Fault-detection/removal rate is a power-law function of testing time for all three generations of errors.
The first four assumptions in the above list are the building blocks in most prior software reliability growth models and the others are specific in our model. To facilitate our later analysis, we summarize the notations used throughout this paper in Notations section.
In general, an implemented software system is tested to detect and correct software errors in the development process. During the testing phase, software errors remaining in the system are discovered and corrected by test personnel. Based on the aforementioned assumptions, the detection rate must be proportional to the instantaneous testing-effort expenditure. Particularly, unlike traditional closed-source software, in open source software, instructions are executed by various concurrent users and each release of open source software can attract an increasing number of volunteers in the software development and/or code modification procedure. Hence, it is reasonable to assume the detection/correction rate as a power-law function of testing time [
As stated above, it is essential to consider fault interdependency when modeling software reliability growth processes. Specifically, in this work we propose three generations of interdependent errors. That is, the detection of the second-generation errors relies on the first generation, and the detection of the third-generation errors depends on the second generation. Moreover, the second-generation errors are detectable only if the first-generation errors have been removed, and the third-generation errors are detectable only if the second-generation errors have been removed. The flow chart is shown in Figure
A flow chart for the error testing process when multigeneration faults exist in a software system.
Next, we sequentially examine the mean value function for each generation of errors:
(i) For the first-generation errors in the software system, the number of detected errors is proportional to the number of remaining errors and the instantaneous testing-effort expenditure. Thus, the mean value function for
(ii) The mean value function for
(iii) Similarly, the mean value function for
After this, we formulate the overall mean value function
Then, based on the aforesaid mean value function, the failure intensity function for the proposed model, with testing-effort, can be explicitly given as
We further note that when
Moreover, when
In addition, when
As another benchmark, hereby we introduce an S-shaped model, which incorporates an increasing detection rate function and well explains “learning effect” of the testing team. Essentially, the mean value function is stated as follows:
To compare the proposed model against traditional models for reliability assessment, numerical examples are provided based on real-world data. Specifically, the well-known Apache project, the most used web server software that is developed and maintained by an open community of developers, is selected in this paper for illustration. This project has large and well-organized communities: a great number of developers have the right to update and change files freely. In particular, testing data about three versions of Apache, 2.0.35/2.0.36/2.0.39, are used. The failure data is presented in Table
Detected faults (days from release and number of errors) for three versions of Apache, 2.0.35/2.0.36/2.0.39. Note that some failure data is not shown due to page limit. For example, as no faults are detected on the 42nd day in Apache 2.0.35, no failure data is shown on this day.
Apache 2.0.35 | Apache 2.0.36 | Apache 2.0.39 | |||
---|---|---|---|---|---|
Days from release | Detected errors | Days from release | Detected errors | Days from release | Detected errors |
1 | 5 | 1 | 2 | 1 | 1 |
2 | 5 | 2 | 5 | 2 | 2 |
3 | 4 | 3 | 1 | 3 | 2 |
4 | 1 | 4 | 1 | 4 | 3 |
5 | 2 | 5 | 1 | 5 | 3 |
6 | 4 | 7 | 2 | 7 | 2 |
7 | 6 | 8 | 1 | 8 | 1 |
8 | 2 | 9 | 1 | 9 | 1 |
10 | 1 | 10 | 3 | 10 | 1 |
11 | 8 | 12 | 2 | 11 | 1 |
12 | 5 | 13 | 1 | 15 | 3 |
13 | 2 | 15 | 2 | 16 | 2 |
14 | 2 | 17 | 1 | 17 | 3 |
15 | 1 | 18 | 2 | 18 | 1 |
17 | 2 | 21 | 1 | 19 | 1 |
18 | 3 | 25 | 1 | 22 | 3 |
19 | 4 | 27 | 1 | 23 | 1 |
20 | 1 | 29 | 2 | 24 | 1 |
21 | 4 | 30 | 2 | 25 | 2 |
23 | 2 | 31 | 3 | 26 | 1 |
24 | 1 | 32 | 1 | 28 | 1 |
25 | 1 | 33 | 3 | 29 | 1 |
26 | 2 | 34 | 1 | 30 | 2 |
27 | 1 | 35 | 3 | 31 | 1 |
28 | 2 | 38 | 3 | 32 | 1 |
31 | 1 | 40 | 1 | 35 | 3 |
34 | 1 | 43 | 1 | 38 | 1 |
43 | 1 | 44 | 1 | 39 | 1 |
103 | 1 | 42 | 1 | ||
43 | 1 | ||||
49 | 3 | ||||
50 | 1 | ||||
51 | 1 | ||||
57 | 1 | ||||
66 | 1 | ||||
70 | 1 | ||||
81 | 1 | ||||
164 | 1 |
The nonlinear least square estimation (NLSE) method is a widely used estimation technique, especially for small/medium sized data sets [
Differentiating
With estimated parameters, we can conduct comparative analysis among models based on the following metrics:
(
(
(
(
The parameter estimation can help characterize properties of theoretical models, determine the number of errors that have already been detected/corrected in the testing process, and predict the number of errors that will be eventually encountered by users. Specifically, by minimizing the objective function in (
(i) For Apache 2.0.35, the estimated results are presented in Table
Optimal estimated values: Apache 2.0.35.
| | | | | | | MSE | TS | |
---|---|---|---|---|---|---|---|---|---|
Goel-Okumoto model | | | — | — | — | — | | | |
Yamada delayed S-shaped model | 74.2741 | 0.1539 | — | — | — | — | 0.9830 | 7.7033 | 0.0482 |
No- | 76.4154 | 0.1087 | 0.1593 | 0.7293 | | — | 0.9947 | 2.3912 | 0.0269 |
Only-G1 model | 77.1477 | 0.0438 | — | — | — | 0.2323 | 0.9925 | 3.4129 | 0.0321 |
Proposed model | | | | | | | | | |
Illustrations for the theoretical values versus their observations are shown in Figure
Predicted values and real-world observations for Apache 2.0.35. Solid line: predicted values; dot: observations.
Failure intensity function with respect to time, with estimated values based on Apache 2.0.35.
We observe that our proposed model well fits the real-world software failure data. This can be seen from the high value of
Furthermore, the prediction capability in our model is very high. The low TS value for Apache 2.0.35, 0.0252, is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, our model can provide significantly lower TS values and higher predication capacity to the failure of Apache system. Lastly, the relative error in prediction using this data set is computed and results are plotted in Figure
Predicted values and real-world observations for Apache 2.0.36. Solid line: predicted values; dot: observations.
(ii) For Apache 2.0.36, the estimated results are presented in Table
Optimal estimated values: Apache 2.0.36.
| | | | | | | MSE | TS | |
---|---|---|---|---|---|---|---|---|---|
Goel-Okumoto model | | | — | — | — | — | | | |
Yamada delayed S-shaped model | 49.8964 | 0.0896 | — | — | — | — | 0.9550 | 8.3927 | 0.0689 |
No- | 50.4507 | 0.0669 | 0.0000 | 0.8065 | | — | 0.9706 | 5.4946 | 0.0557 |
Only-G1 model | 50.1765 | 0.0196 | — | — | — | 0.3207 | 0.9643 | 6.6635 | 0.0614 |
Proposed model | | | | | | | | | |
Then, we can obtain theoretical values in our proposed model. Illustrations for the theoretical values versus their observations are presented in Figure
Failure intensity function with respect to time, with estimated values based on Apache 2.0.36.
Predicted values and real-world observations for Apache 2.0.39. Solid line: predicted values; dot: observations.
We find that our proposed model also well fits the real-world software failure data for Apache 2.0.36. This can be seen from the high value of
Furthermore, the TS value for Apache 2.0.36 is as low as 0.0452, which is much lower than 10% and well confirms a high prediction capability of our model. Compared with the benchmarks, our model shows a lower value of TS and a higher level of predication capability. Lastly, the relative error in prediction for this data set is computed, as shown in Figure
Failure intensity function with respect to time, with estimated values based on Apache 2.0.39.
(iii) For Apache 2.0.39, the estimated results are presented in Table
Optimal estimated values: Apache 2.0.39.
| | | | | | | MSE | TS | |
---|---|---|---|---|---|---|---|---|---|
Goel-Okumoto model | | | — | — | — | — | | | |
Yamada delayed S-shaped model | 56.8974 | 0.0805 | — | — | — | — | 0.9879 | 2.4045 | 0.0306 |
No- | 57.3279 | 0.0577 | 0.2414 | 0.6741 | | — | 0.9969 | 0.6133 | 0.0154 |
Only-G1 model | 57.3266 | 0.0204 | — | — | — | 0.2453 | 0.9957 | 0.8562 | 0.0182 |
Proposed model | | | | | | | | | |
Comparison between the fitted curve and the observed failure data is illustrated in Figure
Relative error for Apache 2.0.35.
Relative error for Apache 2.0.36.
The high value of
Furthermore, the TS value for Apache 2.0.39 is 0.0144. It is much lower than 10% and well confirms a high prediction capability for our model. Compared with the benchmarks, TS values in our model are significantly lower, and prediction capacity of the failure occurrence is also much greater. Lastly, the relative error in prediction for this data set is computed and results are plotted in Figure
Relative error for Apache 2.0.39.
In summary, all of three tests using different versions of Apache failure data sets suggest that our proposed model well fits the real data and presents better performance than the benchmarks in error occurrence prediction. Besides, both the testing effort and the error interdependency effects play an important role in the parameter estimation.
The theoretical modeling can also determine whether the software is ready to release to users or how much more effort should be spent in further testing. In this section, we formally examine the software developer’s best release time and deliver sensitivity analysis regarding how the test effort effect and error interdependency affect the optimal time and the cost.
When the failure process is modeled by a nonhomogeneous Poisson process with a mean value function,
If only the cost is considered, the optimal release time satisfies
Note that the objective function is as follows:
It is also reasonable to quit testing as soon as the number of remaining faults in the system is less than a prescribed portion of total faults. Hence, a simple criterion for software release is given [
Alternatively, given that the testing or the operation has been conducted up to time
In most situations, the project managers need to strike a balance between the cost and the reliability level. That is, on one hand, the reliability level should be at least as high as a desired level set beforehand; on the other hand, given the target level of software reliability
As an illustrative example, Figure
The reliability and the cost as a function of time for Apache 2.0.36. Solid line: cost; dash-dot line: type-I definition of reliability; dash line: type-II definition of reliability; dot line: target level of reliability.
In addition, for Apache 2.0.35, the reliability level is above 0.95 only if
Similarly, for Apache 2.0.39, the reliability level is higher than 0.95 only if
As mentioned above, we have incorporated both the testing-effort and error interdependency effects into our proposed model. In this part, we intend to examine what if the testing effort, the error interdependency, or both of them are ignored in decision-making of software release. The first case corresponds to the No-
For each model, we shall examine the optimal release time and the corresponding minimal cost when a desired reliability level is set beforehand. The data set with Apache 2.0.36 is used to illustrate our findings. Under the type-I reliability definition for reliability measurement, the optimal release time and the minimal cost as a function of
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-I reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-
Optimal release time as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-
Total cost as a function of desired reliability level, with estimated values based on Apache 2.0.36. The type-II reliability definition is adopted for such analysis. Solid line: our proposed mode; dot line: the Only-
Under either type-I or type-II reliability definition, both the optimal release time and the total cost would be mistakenly estimated if either testing effort or fault interdependency is ignored. Unlike these two cases, our proposed model always suggests an earlier release time and a lower testing cost.
As an example, we compare the Goel-Okumoto model and our proposed model when the type-I reliability definition is adopted and
Alternatively, when the type-II reliability definition is adopted, the Goel-Okumoto model predicts the best release time as 132.2 while that for our proposed model is 66.75, which suggests that the testing duration in our model is doubled compared with traditional Goel-Okumoto model. Similarly, according to the Goel-Okumoto model, the minimal cost is 3083$ while that for our proposed model is merely 2000$. Thus, a decision based on our model can save the software developers as much as 35% in terms of the cost.
In the development process, a variety of tests need to be conducted to ensure the reliability of the software system. In this work, we propose a framework for software reliability modeling that simultaneously captures effects of testing effort and interdependence between error generations. In order to evaluate our proposed model, we compare its performance with other existing models by doing experiments with actual software failure data, that is, three versions of Apache. The nonlinear least square estimation technique is applied for parameter estimation, and values of
Our theoretical results can help software project managers assess the most cost-effective time to stop software testing and to release the product. Specifically, we have formally examined the release timing and provided useful managerial insights based on the Apache data sets. Moreover, our proposed model recommends an earlier release time and a reduced cost compared with the Goel-Okumoto model and the Yamada delayed S-shaped model. Therefore, our modeling essentially enhances accuracy of the understanding towards the software reliability growth process and the best time to release a software version.
The
Testing time
Mean value function of the
Mean value function for all three generations of faults
The
Failure intensity function
Cumulative number of software errors detected by testing time
Initial content of all three generations of faults
Proportion of generation
Initial content of generation
Coefficient for the testing-effort function
Power of the testing-effort function
Testing-effort expenditure at time
Cumulative testing-effort expenditure by time
Metric for goodness of fit
Mean of square error
Theil Statistic
Relative error
Total testing cost
Type-I reliability definition of software systems
Type-II reliability definition of software systems
Target level of reliability
Testing cost coefficient.
The authors declare that they have no competing interests.