In general, the software-testing time may be measured by two kinds of time scales: calendar time and test execution time. In this paper, we develop two-dimensional software reliability models with two-time measures and incorporate both of them to assess the software reliability with higher accuracy. Since the resulting software defect models are based on the familiar nonhomogeneous Poisson processes with two time scales, which are the natural extensions of one-dimensional software defect models, it is possible to treat the time data both simultaneously and effectively. We investigate the dependence of test-execution time as a testing effort on the software reliability assessment and validate quantitatively the software defect models with two-time scales. We also consider an optimization problem when to stop the software testing in terms of two-time measurements.
The reliable software plays a central role to develop the dependable and high assurance computer-based systems. Since the debugging cycle times of software are often reduced due to smaller release time requirements, the accurate estimation of software reliability tends to be more important day by day, especially, in the earlier testing phase. Software defect models (SDMs) are used to measure the quantitative software reliability and to control the software testing process effectively. Since the software reliability is defined as the probability that software failures caused by faults do not occur for a specified time period, the time evolution of failure occurrence process (fault detection process in software testing) should be modeled by any stochastic counting process. In fact, during the last three decades since the seminal contribution by Jelinski and Moranda [
When one counts the number of software faults detected in the testing phase, the cumulative number of faults is described by a generalized pure birth process with time and/or state dependent transition rate [
The above modeling frameworks may unify the NHPP-based SDMs, but never refer to how to use them effectively in practice. For instance, the software testing time may be usually measured by two kinds of time scales: calendar time (hour, day, week, etc.) and test-execution time (CPU second, CPU minute, CPU hour, etc). Musa et al. [
In this paper, we summarize two-dimensional SDMs with two-time measures and incorporate both of them to assess the quantitative software reliability with higher accuracy. Ishii and Dohi [
We make the following assumptions. The software fault detected in testing phase is instantly fixed and removed. The number of initial fault contents in the program, The
We call the above c.d.f.
In addition, suppose that the initial number of software faults
Configuration of one-dimensional software debugging.
On the other hand, when the cumulative test-execution time
It is common to assume that the cumulative test-execution time is regarded as a function of calendar time, that is,
Let
To cope with two-time scales (
In this paper, we focus on two simplest cases where Marshall-Olkin distribution:
Bivariate Weibull distribution:
where
Configuration of two-dimensional software debugging.
For the initial fault contents
It is reasonable to assume that the test-execution time is proportional to the number of test cases. In the two-dimensional SDM, if either
On the other hand, for the bivariate Weibull distribution in (
In the seminal paper [
Since the logarithmic function is monotonically increasing, the maximum likelihood estimate of the parameter
Of our next concern is the formulation of the optimal software release problem (e.g.,[ software release time measured by calendar time cumulative test-execution time measured by CPU time software-testing cost per unit calendar time software-testing cost per unit execution time software-fault correction/removal cost per unit fault in testing phase software-fault correction/removal cost per unit fault in operational phase lifetime of software product (given constant).
During the testing phase, two kinds of testing costs proportional to the calendar time and the test-execution time, respectively, are considered, where
For the expected total software cost during the software life cycle, the optimal software release policy is defined by the optimal pair
Unimodality of the expected total software cost function.
In this section, we consider two real data sets collected in the actual software development projects for the real time command and control systems [
Data set DS no. 1.
|
|
|
|
|
|
---|---|---|---|---|---|
1 | 0.053 | 1 | 10 | 16.200 | 42 |
2 | 0.115 | 2 | 11 | 17.100 | 42 |
3 | 0.273 | 4 | 12 | 18.600 | 46 |
4 | 0.354 | 5 | 13 | 20.600 | 47 |
5 | 1.400 | 13 | 14 | 21.800 | 47 |
6 | 3.150 | 22 | 15 | 23.000 | 49 |
7 | 6.110 | 28 | 16 | 25.200 | 51 |
8 | 11.080 | 35 | 17 | 32.800 | 54 |
9 | 11.500 | 39 |
Data set DS no. 2.
|
|
|
|
|
|
---|---|---|---|---|---|
1 | 0.358 | 8 | 9 | 4.580 | 48 |
2 | 0.690 | 14 | 10 | 6.620 | 50 |
3 | 0.945 | 17 | 11 | 9.980 | 51 |
4 | 1.270 | 17 | 12 | 11.300 | 51 |
5 | 1.670 | 28 | 13 | 13.700 | 51 |
6 | 2.210 | 35 | 14 | 15.300 | 53 |
7 | 2.610 | 35 | 15 | 17.700 | 53 |
8 | 3.760 | 41 | 16 | 18.500 | 53 |
Using these two data sets, we derive the maximum likelihood estimate of the parameter
Goodness-of-fit test with DS no. 1.
100% | MO | BW | GO | GL | LT | WT |
---|---|---|---|---|---|---|
LLF | −45.266 | −33.740 | −41.468 | −41.247 | −43.385 | −38.518 |
AIC | 98.532 | 79.480 | 86.936 | 88.493 | 90.769 | 81.036 |
RSS | 71.160 | 9.216 | 25.513 | 44.988 | 27.078 | 15.730 |
| ||||||
75% | MO | BW | GO | GL | LT | WT |
| ||||||
LLF | −33.334 | −25.132 | −31.540 | −24.431 | −26.493 | −24.842 |
AIC | 74.668 | 62.263 | 67.080 | 54.861 | 56.986 | 53.684 |
RSS | 30.450 | 2.503 | 24.673 | 2.605 | 6.632 | 1.784 |
| ||||||
50% | MO | BW | GO | GW | LT | WT |
| ||||||
LLF | −13.358 | −12.394 | −22.547 | −15.423 | −14.748 | −16.038 |
AIC | 34.716 | 36.787 | 49.095 | 36.846 | 33.497 | 36.077 |
RSS | 1.887 | 0.311 | 50.024 | 1.928 | 1.400 | 1.142 |
| ||||||
25% | MO | BW | GO | GL | LT | WT |
| ||||||
LLF | −4.327 | −4.382 | −4.577 | −4.504 | −4.444 | −4.412 |
AIC | 16.653 | 20.765 | 13.155 | 15.009 | 12.889 | 12.824 |
RSS | 0.010 | 0.164 | 0.094 | 0.043 | 0.028 | 0.044 |
Goodness-of-fit test with DS no. 2.
100% | MO | BW | GO | GL | LT | WT |
---|---|---|---|---|---|---|
LLF | −48.967 | −34.314 | −39.951 | −39.016 | −37.885 | −42.403 |
AIC | 105.933 | 80.629 | 83.901 | 84.032 | 79.770 | 88.807 |
RSS | 47.519 | 3.626 | 12.010 | 7.868 | 6.158 | 9.100 |
| ||||||
75% | MO | BW | GO | GL | LT | WT |
| ||||||
LLF | −29.559 | −28.191 | −34.645 | −34.570 | −32.730 | −38.166 |
AIC | 67.117 | 68.383 | 73.291 | 75.140 | 69.459 | 80.331 |
RSS | 4.664 | 3.558 | 9.203 | 8.623 | 6.390 | 11.790 |
| ||||||
50% | MO | BW | GO | GL | LT | WT |
| ||||||
LLF | −22.231 | −22.083 | −25.327 | −24.996 | −25.606 | −27.059 |
AIC | 52.463 | 56.167 | 54.653 | 55.992 | 55.211 | 58.118 |
RSS | 4.314 | 4.708 | 6.252 | 6.264 | 6.626 | 8.731 |
| ||||||
25% | MO | BW | GO | GL | LT | WT |
| ||||||
LLF | −7.307 | −5.493 | −6.808 | −5.866 | −6.895 | −7.192 |
AIC | 22.614 | 22.985 | 17.615 | 17.732 | 17.789 | 18.384 |
RSS | 0.751 | 0.261 | 0.700 | 0.141 | 0.745 | 0.970 |
Estimates of cumulative number of detected software faults (DS no. 1).
MO
BW
One-dimensional SRMs
Estimates of cumulative number of detected software faults (DS no. 2).
MO
BW
One-dimensional SRMs
Next, we predict the behavior of the number of software faults detected in the future. Two possible cases are considered: in case 1, the model parameters are estimated at the observation point
Prediction error in case 1 with DS no. 1.
75% | MO | BW | GO | GL | LT | WT |
---|---|---|---|---|---|---|
Given | 42.218 | 1.945 | 71.224 | 9.731 | 16.877 | 10.880 |
Regression | 7.911 | 18.681 | ||||
| ||||||
50% | MO | BW | GO | GL | LT | WT |
| ||||||
Given | 99.002 | 103.981 | 159.551 | 209.869 | 119.644 | 35.762 |
Regression | 120.303 | 120.310 | ||||
| ||||||
25% | MO | BW | GO | GL | LT | WT |
| ||||||
Given | 645.622 | 1293.560 | 724.258 | 1186.030 | 1277.290 | 1158.760 |
Regression | 1333.690 | 1333.450 |
Prediction error in case 2 with DS no. 1.
MO | BW | GL | |
---|---|---|---|
75% | 30.661 | 2.154 | 4.889 |
50% | 97.368 | 119.921 | 111.164 |
25% | 651.142 | 1333.160 | 911.021 |
Prediction error in case 1 with DS no. 2.
75% | MO | BW | GO | GL | LT | WT |
---|---|---|---|---|---|---|
Given | 2.431 | 0.656 | 10.252 | 5.825 |
1.656 |
0.863 |
Regression | 2.994 | 0.691 | ||||
| ||||||
50% | MO | BW | GO | GL | LT | WT |
| ||||||
Given | 9.812 | 34.861 | 77.538 | 114.273 |
3.240 |
4.330 |
Regression | 92.790 | 107.750 | ||||
| ||||||
25% | MO | BW | GO | GL | LT | WT |
| ||||||
Given | 811.770 | 895.649 | 842.364 | 894.170 | 824.662 | 795.296 |
Regression | 895.753 | 677.371 |
Prediction error in case 2 with DS no. 2.
MO | BW | GL | |
---|---|---|---|
75% | 2.431 | 1.844 | 2.431 |
50% | 9.812 | 34.861 | 9.812 |
25% | 811.770 | 895.890 | 811.770 |
Next, we assess the quantitative software reliability as the probability that no software failure occurs during a specified time period, where the software reliability is defined by
Estimating software reliability with DS no. 1.
Estimating software reliability with DS no. 2.
Finally, we investigate the dependence of cost parameters characterizing the software testing process on the optimal software release policy (Tables
Dependence of cost ratio
|
|
|
---|---|---|
1/1 | (23.284, 91.379) | 226.999 |
5/1 | (17.097, 91.155) | 304.576 |
10/1 | (14.432, 90.881) | 382.635 |
20/1 | (11.766, 90.333) | 512.098 |
50/1 | ( 8.240, 88.640) | 804.229 |
100/1 | ( 5.566, 85.542) | 1141.750 |
200/1 | ( 2.872, 77.581) | 1548.600 |
300/1 | ( 1.225, 63.745) | 1749.270 |
Dependence of cost ration
|
|
|
---|---|---|
1/80 | (11.098, 119.890) | 1724.020 |
1/50 | ( 9.253, 109.094) | 1529.560 |
1/30 | ( 7.219, 96.576) | 1315.360 |
1/20 | ( 5.566, 85.542) | 1141.750 |
1/15 | ( 4.357, 76.585) | 1015.440 |
1/10 | ( 2.539, 60.175) | 829.632 |
1/5 | ( 0.000, 0.000) | 437.726 |
1/1 | ( 0.000, 0.000) | 87.545 |
Dependence of cost ratio
|
|
|
---|---|---|
1/1 | (12.280, 139.110) | 1133.400 |
5/1 | ( 2.462, 136.827) | 1153.190 |
10/1 | ( 1.232, 135.845) | 1161.730 |
20/1 | ( 0.616, 134.864) | 1170.270 |
50/1 | ( 0.247, 133.569) | 1181.570 |
100/1 | ( 0.123, 132.591) | 1190.130 |
200/1 | ( 0.062, 131.615) | 1198.680 |
300/1 | ( 0.041, 131.044) | 1203.690 |
Dependence of cost ratio
|
|
|
---|---|---|
1/80 | (0.567, 245.157) | 3944.060 |
1/50 | (0.344, 202.974) | 2593.110 |
1/30 | (0.196, 161.844) | 1668.020 |
1/20 | (0.123, 132.591) | 1190.130 |
1/15 | (0.088, 113.591) | 943.667 |
1/10 | (0.052, 89.176) | 687.952 |
1/5 | (0.019, 53.380) | 412.309 |
1/1 | (0.000, 0.000) | 120.758 |
In this paper, we have developed two-dimensional software reliability models based on bivariate fault-detection time distributions related with calendar time and test execution time and compared their goodness-of-fit and prediction performances with real project data. The numerical examples have suggested that our new SDMs provide better performance than the existing NHPP-based SDMs in many cases.
In the future, we will apply the same technique to the case with many factors influencing the cumulative number of failure occurrences. For instance, when the code coverage is considered as Grottke [
The present research was partially supported by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid for Scientific Research (C), Grant no. 23510171 (2011–2013). This work is an extended version of our conference paper [