Modifications in software can affect some functionality that had been working until that point. In order to detect such a problem, the ideal solution would be testing the whole system once again, but there may be insufficient time or resources for this approach. An alternative solution is to order the test cases so that the most beneficial tests are executed first, in such a way only a subset of the test cases can be executed with little lost of effectiveness. Such a technique is known as regression test case prioritization. In this paper, we propose the use of the Reactive GRASP metaheuristic to prioritize test cases. We also compare this metaheuristic with other search-based algorithms previously described in literature. Five programs were used in the experiments. The experimental results demonstrated good coverage performance with some time overhead for the proposed technique. It also demonstrated a high stability of the results generated by the proposed approach.

More than often, when a system is modified, the modifications may affect some functionality that had been working until that point in time. Due to the unpredictability of the effects that such modifications may cause to the system’s functionalities, it is recommended to test the system, as a whole or partially, once again every time a modification takes place. This is commonly known as regression testing. Its purpose is to guarantee that the software modifications have not affected the functions that were working previously.

A test case is a set of tests performed in a sequence and related to a test objective [

According to Myers [

The regression test case prioritization problem is closely related to the regression test case selection problem. The Regression Test Case Selection problem can be directly modeled as a set covering problem, which is a well-known NP-Hard problem [

To order the test cases, it is necessary to consider a base comparison measure. A straightforward measure to evaluate a test case would be based on APFD (Average of the Percentage of Faults Detected). Higher APFD numbers mean faster fault detection rates [

As an example, consider a test suite

Greedy algorithms have been employed in many researches regarding test case prioritization, in order to find an optimal ordering [

Metaheuristic search techniques are algorithms that may find optimal or near optimal solutions to optimization problems [

The application of genetic algorithms, an evolutionary metaheuristic, has been shown to be effective for regression test case prioritization [

The remaining of this paper is organized as follows: Section

This section reports the use of search-based prioritization approaches and metaheuristics. Some algorithms implemented in [

The works below employed search-based prioritization approaches, such as greedy- and metaheuristic-based solutions.

Elbaum et al. [

Yoo and Harman [

Li et al. [

This section describes some algorithms which have been used frequently in literature to deal with the test case prioritization problem. The performance of them will be compared to that of the approach proposed later on this paper.

The Greedy Algorithm performs in the following way: all candidate test cases are ordered by their coverage. Then, the test case with the highest percentage of coverage is then added to an initially empty solution. Next, the test case with the second highest percentage is added, and so on, until all test cases have been added.

For example, let APBC be the coverage criterion, and let a partial solution contain two test cases that cover 100 blocks of code. Suppose there are two other test cases that can be added to the solution. The first one covers 80 blocks, but 50 of these were already covered by the current solution. Then, this solution covers 80% of the blocks, but the actual added coverage of this test case is of 30% of coverage (30 blocks). The second test case covers 40 blocks of code, but none of these blocks was covered by the current solution. This means that this solution covers 40% of the blocks. The Greedy algorithm would select the first test case, because it has greater percentage of block coverage overall.

The Additional Greedy algorithm adds a locally optimal test case to a partial test suite. Starting from an empty solution, the algorithm follows these steps: for each iteration, the algorithm adds the test case which gives the major coverage gain to the partial solution.

Let us use the same example from Section

Genetic algorithm is a type of Evolutionary Algorithm which has been employed extensively to solve optimization problems [

In the genetic algorithm proposed by Li et al. [

The crossover algorithm follows the ordering chromosome crossover style adopted by Antoniol [

Simulated annealing is a generalization of a Monte Carlo method. Its name comes from annealing in metallurgy, where a melt, initially disordered at high temperature, is slowly cooled, with the purpose of obtaining a more organized system (a local optimum solution). The system approaches a frozen ground state with

This section is intended to present a novel approach for test case prioritization based on the Reactive GRASP metaheuristic.

Metaheuristics are general search algorithms that find a good solution, sometimes optimal, to

GRASP (Greedy Randomized Adaptative Search Procedures) is a metaheuristic with two phases: construction and local search [

GRASP’s phases.

In the construction phase, a feasible solution is built by applying some Greedy algorithm. The greedy strategy used in GRASP is to add to an initially empty solution one element at a time. This algorithm tends to find a local optimum. Therefore, in order to avoid this local best, GRASP uses a randomization greedy strategy as follows. The Restrict Candidate List (RCL) stores the possible elements which can be added at each step in this construction phase. The element to be added is picked randomly from this list. RCL is associated with a parameter named

In the local search phase, the aim is to find the best solution in the current solution neighborhood. Indeed, a local search is executed in order to replace the current solution by the local optimum in its neighborhood. After this process, this local optimum is compared with the best local optimum solution found in earlier iterations. If the local optimum just found is better, then this is set to be the best solution already found. Otherwise, there is no replacement.

As can be easily seen, the performance of the GRASP algorithm will strongly depend on the choice of the parameter

Initially, all

The pseudocode below, in Algorithm

with

The first step initializes the probabilities associated with the choice of each

Initially, all probabilities are assigned to

For each iteration,

procedure select_

The pseudocode in Algorithm

test cases from the pool of test cases;

The

procedure update_

After the construction phase, a local search phase is executed in order to improve the generated solution. This phase is important to avoid the problems mentioned by Rothermel [

(5) return

Let

In order to evaluate the performance of the proposed approach, a series of empirical tests was executed. More specifically, the experiments were designed to answer the following question.

How does the Reactive GRASP approach compare—in terms of coverage and time performances—to other search-based algorithms, including Greedy algorithm, Additional Greedy algorithm, genetic algorithm, and Simulated Annealing?

In addition to this result, the experiments can confirm results previously described in literature, including the performance of the Greedy algorithm.

Four small programs (print_tokens, print_tokens2, schedule, and schedule2) and a larger program (space) were used in the tests. These programs were assembled by researchers at Siemens Corporate Research [

Programs used in the Evaluation.

Program | LoC | Blocks | Decisions | Test Pool Size |
---|---|---|---|---|

Print_tokens | 726 | 126 | 123 | 4,130 |

Print_tokens2 | 570 | 103 | 154 | 4,115 |

Schedule | 412 | 46 | 56 | 2,650 |

Schedule2 | 374 | 53 | 74 | 2,710 |

Space | 9,564 | 869 | 1,068 | 13,585 |

Besides Reactive GRASP, other search algorithms have also been implemented, in order to compare their effectiveness. They are Greedy algorithm, Additional Greedy algorithm, genetic algorithm, and simulated annealing. These algorithms were implemented exactly as described in Section

For the simulated annealing approach, the initial temperature was set to a random number between 20 and 99. For each iteration, the new temperature is given by the following steps:

In the experiments, we considered the three coverage criteria described earlier (APBC, APDC, and APSBC). In addition, we considered different percentages of the pool of test cases. For example, if the percentage is 5%, 5% of test cases are randomly chosen from the pool to compare the performance of the algorithms. We tested with 1%, 2%, 3%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 100% for the four small programs and 1%, 5%, 10%, 20%, 30%, 40%, and 50% for space. Each algorithm was executed 10 times for the four small programs and 1 time for the space program, for each coverage criterion and each percentage.

The pools of test cases used in the experiments were collected from SEBASE [

All experiments were performed on Ubuntu Linux workstations with kernel 2.6.22-14, a Core Duo processor, and 1 GB of main memory. The programs used in the experiment were implemented using the Java programing language.

The results are presented in Tables

Results of Coverage Criteria (Average of 10 Executions), Program Print-tokens.

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 96.6591 | 98.235 | 96.6893 | 96.1242 | 97.808 |

2% | 98.3209 | 99.2101 | 98.3954 | 98.3113 | 99.0552 |

3% | 98.6763 | 99.5519 | 98.5483 | 98.5553 | 99.3612 |

5% | 98.5054 | 99.6909 | 98.8988 | 98.9896 | 99.5046 |

10% | 98.2116 | 99.8527 | 99.2378 | 99.3898 | 99.7659 |

20% | 98.266 | 99.9317 | 99.2378 | 99.6414 | 99.8793 |

30% | 98.3855 | 99.9568 | 99.6603 | 99.6879 | 99.9204 |

40% | 98.3948 | 99.9675 | 99.7829 | 99.736 | 99.9457 |

50% | 98.4064 | 99.9747 | 99.8321 | 99.8213 | 99.9627 |

60% | 98.4097 | 99.979 | 99.8666 | 99.8473 | 99.9622 |

70% | 98.4133 | 99.9818 | 99.8538 | 99.8698 | 99.9724 |

80% | 98.4145 | 99.9841 | 99.8803 | 99.8657 | 99.9768 |

90% | 98.4169 | 99.9859 | 99.9013 | 99.8958 | 99.9783 |

100% | 98.418 | 99.9873 | 99.9001 | 99.8895 | 99.9775 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 96.7692 | 98.3836 | 96.9125 | 95.9213 | 98.1204 |

2% | 98.0184 | 99.1429 | 97.9792 | 98.2299 | 98.8529 |

3% | 98.5569 | 99.4499 | 98.3785 | 98.0762 | 99.2886 |

5% | 98.4898 | 99.6971 | 98.7105 | 98.7513 | 99.4631 |

10% | 98.1375 | 99.8462 | 98.8659 | 99.1759 | 99.697 |

20% | 98.2486 | 99.928 | 99.3886 | 99.5111 | 99.8668 |

30% | 98.3131 | 99.952 | 99.587 | 99.6955 | 99.9061 |

40% | 98.3388 | 98.3388 | 99.7137 | 99.7505 | 99.9237 |

50% | 98.3437 | 99.9712 | 99.7305 | 99.78 | 99.9386 |

60% | 98.358 | 99.9766 | 99.817 | 99.8235 | 99.959 |

70% | 98.3633 | 99.9799 | 99.8109 | 99.7979 | 99.9543 |

80% | 98.3651 | 99.9821 | 99.8631 | 99.8447 | 99.9663 |

90% | 98.4169 | 99.9859 | 99.9013 | 99.8541 | 99.9783 |

100% | 98.418 | 99.9873 | 99.9001 | 99.869 | 99.9775 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.2989 | 98.3561 | 97.0141 | 97.251 | 98.0439 |

2% | 97.7834 | 99.2557 | 98.0175 | 98.576 | 98.9675 |

3% | 98.0255 | 99.4632 | 98.5163 | 98.5633 | 99.2356 |

5% | 97.8912 | 99.6826 | 98.5167 | 99.0268 | 99.4431 |

10% | 97.8137 | 99.8534 | 99.1497 | 99.3131 | 99.681 |

20% | 98.0009 | 99.9264 | 99.5024 | 99.5551 | 99.8554 |

30% | 98.0551 | 99.954 | 99.6815 | 99.7151 | 99.9079 |

40% | 98.0661 | 99.9656 | 99.7342 | 99.7677 | 99.9296 |

50% | 98.0705 | 99.9724 | 99.8123 | 99.8108 | 99.9464 |

60% | 98.0756 | 99.9773 | 99.8348 | 99.8456 | 99.9598 |

70% | 98.0887 | 99.9805 | 99.8641 | 99.8633 | 99.9704 |

80% | 98.088 | 99.9831 | 99.89 | 99.8649 | 99.9682 |

90% | 98.0924 | 99.985 | 99.9026 | 99.8819 | 99.9709 |

100% | 98.0943 | 99.9865 | 99.8998 | 99.8897 | 99.977 |

Results of Coverage Criteria (Average of 10 Executions), Program Print-tokens2.

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.233 | 98.3518 | 97.6629 | 98.042 | 98.1576 |

2% | 98.3869 | 99.2665 | 98.6723 | 98.8302 | 99.208 |

3% | 97.9525 | 99.5122 | 98.8576 | 99.1817 | 99.3274 |

5% | 98.1407 | 99.711 | 99.2379 | 99.3382 | 99.5932 |

10% | 98.131 | 99.8564 | 99.5558 | 99.6731 | 99.7994 |

20% | 98.01 | 99.9293 | 99.7894 | 99.8015 | 99.8689 |

30% | 98.0309 | 99.9535 | 99.8269 | 99.839 | 99.9239 |

40% | 98.0462 | 99.9656 | 99.8602 | 99.8957 | 99.9495 |

50% | 98.0569 | 99.9727 | 99.9166 | 99.9106 | 99.9653 |

60% | 98.0589 | 99.977 | 99.9165 | 99.9269 | 99.9689 |

70% | 98.0611 | 99.9805 | 99.9264 | 99.9236 | 99.9756 |

80% | 98.0632 | 99.9828 | 99.9383 | 99.9261 | 99.9778 |

90% | 98.0663 | 99.9849 | 99.9543 | 99.9385 | 99.9796 |

100% | 98.0671 | 99.9864 | 99.9562 | 99.9434 | 99.9811 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.05 | 98.3055 | 97.2108 | 97.6375 | 98.0859 |

2% | 98.6637 | 99.2489 | 98.4368 | 98.5244 | 99.0987 |

3% | 98.5798 | 99.5496 | 98.8814 | 99.0411 | 99.4344 |

5% | 98.5903 | 99.7371 | 99.3676 | 99.2289 | 99.6772 |

10% | 98.5673 | 99.8628 | 99.5183 | 99.6528 | 99.8118 |

20% | 98.6351 | 99.9353 | 99.7939 | 99.8084 | 99.913 |

30% | 98.6747 | 99.9593 | 99.8615 | 99.8405 | 99.9482 |

40% | 98.6837 | 99.9692 | 99.8836 | 99.8802 | 99.9556 |

50% | 96.1134 | 99.9552 | 99.8269 | 99.8992 | 99.9318 |

60% | 98.6948 | 99.9795 | 99.9181 | 99.9109 | 99.9751 |

70% | 98.6964 | 99.9826 | 99.9358 | 99.9302 | 99.9768 |

80% | 98.6985 | 99.9848 | 99.9478 | 99.931 | 99.9788 |

90% | 98.0663 | 99.9849 | 99.9543 | 99.9409 | 99.9796 |

100% | 98.0671 | 99.9864 | 99.9562 | 99.9424 | 99.9811 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.4458 | 98.3742 | 97.8234 | 97.453 | 98.2804 |

2% | 98.7611 | 99.2552 | 98.755 | 98.5444 | 99.0653 |

3% | 98.3634 | 99.4745 | 98.9385 | 98.9165 | 99.3279 |

5% | 97.8694 | 99.6856 | 99.1507 | 99.3858 | 99.5327 |

10% | 98.0271 | 99.8494 | 99.5268 | 99.6258 | 99.7906 |

20% | 98.1264 | 99.927 | 99.7455 | 99.7283 | 99.9086 |

30% | 97.9467 | 99.9518 | 99.8533 | 99.8297 | 99.9328 |

40% | 97.9653 | 99.9645 | 99.8833 | 99.864 | 99.9564 |

50% | 97.9762 | 99.9717 | 99.9126 | 99.8891 | 99.9584 |

60% | 97.9792 | 99.9768 | 99.9162 | 99.905 | 99.9644 |

70% | 97.9851 | 99.9803 | 99.9265 | 99.9156 | 99.9708 |

80% | 97.9854 | 99.9827 | 99.9399 | 99.9187 | 99.9759 |

90% | 97.9877 | 99.9847 | 99.9399 | 99.9288 | 99.9789 |

100% | 97.9894 | 99.9863 | 99.9477 | 99.9262 | 99.9791 |

Results of Coverage Criteria (Average of 10 Executions), Program Schedule.

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 96.6505 | 98.2286 | 98.2286 | 97.9387 | 98.2286 |

2% | 96.5053 | 99.0237 | 98.9499 | 98.8596 | 99.0073 |

3% | 96.451 | 99.3315 | 99.2336 | 99.1955 | 99.2445 |

5% | 95.6489 | 99.5652 | 99.2481 | 99.4066 | 99.3233 |

10% | 95.2551 | 99.767 | 99.5586 | 99.6455 | 99.7013 |

20% | 95.9548 | 99.8884 | 99.7604 | 99.7497 | 99.8589 |

30% | 95.8225 | 99.9224 | 99.8219 | 99.8442 | 99.8918 |

40% | 96.0783 | 99.9429 | 99.8995 | 99.8982 | 99.9163 |

50% | 96.3159 | 99.9553 | 99.9051 | 99.899 | 99.9396 |

60% | 96.9283 | 99.9644 | 99.918 | 99.9156 | 99.9546 |

70% | 97.0744 | 99.9695 | 99.9235 | 99.9322 | 99.9643 |

80% | 97.0955 | 99.9733 | 99.9464 | 99.9411 | 99.9649 |

90% | 97.1171 | 99.9763 | 99.9474 | 99.946 | 99.9704 |

100% | 97.0495 | 99.9786 | 99.9573 | 99.9454 | 99.7013 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 96.3492 | 98.2671 | 98.0952 | 98.0092 | 98.2407 |

2% | 95.9838 | 99.0566 | 98.7129 | 98.7364 | 98.9218 |

3% | 95.933 | 99.3303 | 98.9107 | 98.9575 | 99.2589 |

5% | 95.1047 | 99.5327 | 98.6224 | 99.0856 | 99.2427 |

10% | 94.8611 | 99.7668 | 99.2237 | 99.3422 | 99.6159 |

20% | 94.75 | 99.8739 | 99.4858 | 99.6266 | 99.7749 |

30% | 95.3616 | 99.9241 | 99.7047 | 99.7181 | 99.8757 |

40% | 95.3396 | 99.9413 | 99.7944 | 99.7871 | 99.9144 |

50% | 96.1134 | 99.9552 | 99.8269 | 99.8515 | 99.9318 |

60% | 96.3241 | 99.9627 | 99.852 | 99.8541 | 99.9416 |

70% | 96.5465 | 99.968 | 99.8673 | 99.8927 | 99.9586 |

80% | 96.9312 | 99.9722 | 99.88 | 99.8868 | 99.9553 |

90% | 97.1171 | 99.9763 | 99.9474 | 99.9077 | 99.9704 |

100% | 97.0495 | 99.9786 | 99.9573 | 99.9118 | 99.9701 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 96.747 | 98.1768 | 98.0792 | 98.1911 | 98.1596 |

2% | 97.0323 | 99.039 | 98.8108 | 98.8664 | 99.0273 |

3% | 96.937 | 99.3284 | 99.1366 | 99.1927 | 99.2257 |

5% | 96.3181 | 99.5751 | 99.2731 | 99.4252 | 99.4398 |

10% | 96.1091 | 99.782 | 99.452 | 99.6635 | 99.6428 |

20% | 96.9909 | 99.8945 | 99.7965 | 99.8168 | 99.8693 |

30% | 97.2931 | 99.9307 | 99.8703 | 99.8683 | 99.9112 |

40% | 97.0724 | 99.9471 | 99.9003 | 99.8983 | 99.9358 |

50% | 97.4288 | 99.9584 | 99.9214 | 99.9146 | 99.9445 |

60% | 97.4015 | 99.9653 | 99.932 | 99.9281 | 99.9594 |

70% | 97.6458 | 99.9707 | 99.9374 | 99.931 | 99.9653 |

80% | 97.8832 | 99.9748 | 99.9399 | 99.9273 | 99.9722 |

90% | 97.8907 | 99.9777 | 99.9496 | 99.9471 | 99.9653 |

100% | 97.8901 | 99.9799 | 99.9627 | 99.9494 | 99.978 |

Results of Coverage Criteria (Average of 10 Executions), Program Schedule2.

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.2708 | 98.1199 | 98.066 | 98.167 | 98.1064 |

2% | 98.2538 | 99.0566 | 98.9605 | 99.0325 | 99.036 |

3% | 98.6447 | 99.3764 | 99.3304 | 99.3464 | 99.3534 |

5% | 99.4678 | 99.6184 | 99.5851 | 99.5879 | 99.6184 |

10% | 98.2116 | 99.8527 | 99.2378 | 99.7869 | 99.7659 |

20% | 99.9056 | 99.907 | 99.8952 | 99.893 | 99.907 |

30% | 99.9385 | 99.9385 | 99.9348 | 99.9267 | 99.9385 |

40% | 99.9538 | 99.9538 | 99.9476 | 99.9418 | 99.9538 |

50% | 99.963 | 99.963 | 99.9586 | 99.9535 | 99.963 |

60% | 99.9692 | 99.9692 | 99.9676 | 99.9612 | 99.9692 |

70% | 99.9736 | 99.9736 | 99.9702 | 99.9584 | 99.9736 |

80% | 99.9769 | 99.9769 | 99.972 | 99.9641 | 99.9769 |

90% | 99.9794 | 99.9794 | 99.9779 | 99.9735 | 99.9794 |

100% | 99.9815 | 99.9815 | 99.9796 | 99.9701 | 99.9815 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 95.6563 | 98.3687 | 97.9922 | 98.1129 | 98.3301 |

2% | 96.1375 | 98.9533 | 98.2113 | 98.5404 | 98.8501 |

3% | 95.5965 | 99.3111 | 98.4344 | 98.9122 | 99.0936 |

5% | 97.6887 | 99.6164 | 99.058 | 99.2189 | 99.4773 |

10% | 97.1277 | 99.7985 | 99.4385 | 99.4873 | 99.7057 |

20% | 97.2249 | 99.9027 | 99.7033 | 99.7575 | 99.8713 |

30% | 97.2647 | 99.9352 | 99.8177 | 99.8224 | 99.9126 |

40% | 97.2726 | 99.9513 | 99.8145 | 99.8673 | 99.9144 |

50% | 97.2823 | 99.9712 | 99.8745 | 99.8907 | 99.9411 |

60% | 97.2869 | 99.9676 | 99.8827 | 99.9143 | 99.9584 |

70% | 97.2981 | 99.9722 | 99.915 | 99.9013 | 99.9595 |

80% | 97.3005 | 99.9756 | 99.9311 | 99.915 | 99.9695 |

90% | 99.9794 | 99.9794 | 99.9779 | 99.9304 | 99.9794 |

100% | 99.9815 | 99.9815 | 99.9796 | 99.9297 | 99.9815 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 97.7116 | 98.2883 | 98.1984 | 98.0316 | 98.2777 |

2% | 97.4612 | 99.1097 | 98.9346 | 98.6235 | 99.0208 |

3% | 97.1499 | 99.336 | 98.9259 | 99.0397 | 99.1481 |

5% | 97.7227 | 99.6029 | 99.3066 | 99.428 | 99.5114 |

10% | 98.3422 | 99.8072 | 99.6104 | 99.6295 | 99.734 |

20% | 98.4317 | 99.9014 | 99.7765 | 99.7866 | 99.8815 |

30% | 98.474 | 99.9363 | 99.8543 | 99.8455 | 99.9074 |

40% | 98.4861 | 99.9525 | 99.8892 | 99.8833 | 99.9441 |

50% | 98.4988 | 99.962 | 99.9159 | 99.9055 | 99.9568 |

60% | 98.5041 | 99.9683 | 99.9251 | 99.9123 | 99.9626 |

70% | 98.5109 | 99.9728 | 99.9345 | 99.9278 | 99.9663 |

80% | 98.512 | 99.9762 | 99.9429 | 99.9291 | 99.9725 |

90% | 98.5166 | 99.9788 | 99.9549 | 99.9463 | 99.9757 |

100% | 98.521 | 99.981 | 99.9583 | 99.9453 | 99.9783 |

Coverage Significance and Time Mean Difference, Small Programs.

Algorithm ( | Algorithm ( | Mean Coverage Difference (%) | Coverage Difference | Time Mean Difference |

Significance ( | (s) | |||

Greedy Algorithm | Additional Greedy Algorithm | 0.0000 | ||

Genetic Algorithm | 0.0000 | |||

Simulated Annealing | 0.0000 | |||

Reactive GRASP | 0.0000 | |||

Additional Greedy | Greedy Algorithm | 1.9041 | 0.0000 | |

Algorithm | Genetic Algorithm | 0.0818 | 0.0000 | |

Simulated Annealing | 0.0790 | 0.0000 | 1.4274 | |

Reactive GRASP | 0.0103 | 0.1876 | ||

Genetic Algorithm | Greedy Algorithm | 1.8223 | 0.0000 | 8.4361 |

Additional Greedy Algorithm | 0.0000 | 6.9452 | ||

Simulated Annealing | 0.4918 | 8.3727 | ||

Reactive GRASP | 0.0000 | |||

Simulated Annealing | Greedy Algorithm | 1.8250 | 0.0000 | 0.0634 |

Additional Greedy Algorithm | 0.0000 | |||

Genetic Algorithm | 0.0026 | 0.4918 | ||

Reactive GRASP | 0.0000 | |||

Reactive GRASP | Greedy Algorithm | 1.8938 | 0.0000 | 100.0312 |

Additional Greedy Algorithm | 0.1876 | 98.5403 | ||

Genetic Algorithm | 0.0715 | 0.0000 | 91.5951 | |

Simulated Annealing | 0.0688 | 0.0000 | 99.9679 |

Weighted Average for the Metrics, Small Programs.

Coverage Criterion | Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP |
---|---|---|---|---|---|

Block Coverage | 98.2858 | 99.8825 | 99.8863 | 99.9335 | |

Decision Coverage | 97.8119 | 99.9276 | 99.8406 | 99.8417 | |

Statement Coverage | 98.0328 | 99.8743 | 99.8706 | 99.9417 |

Difference in Performance between the Best and Worst Criteria, Small Programs.

Greedy Algorithm | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |

Algorithm | |||||

Difference in performance | 0.4739 | 0.0302 | 0.0419 | 0.0446 | 0.0082 |

between the best and worst criteria |

Average for Each Algorithm (All Metrics), Small Programs.

Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |
---|---|---|---|---|---|

Final Average | 98.0435 | 99.9476 | 99.8658 | 99.8662 | 99.9373 |

Standard Deviation of the Effectiveness for the Four Algorithms, Small Programs.

Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |
---|---|---|---|---|---|

Standard Deviation | 0.002371 | 0.000172 | 0.000222 | 0.000226 | 0.000041 |

Summary of Results, Small Programs.

Algorithm | Coverage Performance | Execution Time | Observations |
---|---|---|---|

Greedy Algorithm | The worst performance | Fast | |

Additional Greedy Algorithm | Best performance of all | Fast | |

Genetic Algorithm | Fourth best performance | Medium | It generated a better coverage only once. |

Simulated Annealing | Third best performance | Fast | No significant difference to genetic algorithm. |

Reactive GRASP | Second best performance | Slow | No significant difference to Additional Greedy Algorithm. |

Results of Coverage Criteria (1 Execution), Program Space.

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 87.4115 | 96.4804 | 92.6728 | 91.4603 | 95.6961 |

5% | 85.8751 | 98.5599 | 94.8614 | 94.9912 | 98.0514 |

10% | 85.5473 | 99.1579 | 95.9604 | 96.7242 | 98.6774 |

20% | 86.5724 | 99.6063 | 98.0118 | 97.991 | 99.4235 |

30% | 86.9639 | 99.7423 | 98.5998 | 98.6937 | 99.6431 |

40% | 87.3629 | 99.811 | 98.9844 | 98.9004 | 99.7339 |

50% | 87.8269 | 99.842 | 99.1271 | 99.216 | 99.7755 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 88.753 | 96.9865 | 91.6811 | 92.0529 | 96.4502 |

5% | 85.5131 | 98.553 | 93.6639 | 94.9256 | 97.8443 |

10% | 86.9345 | 99.1999 | 95.9172 | 96.6152 | 98.358 |

20% | 87.9909 | 99.6074 | 98.0217 | 97.7348 | 99.2446 |

30% | 88.4008 | 99.7464 | 98.4662 | 98.5373 | 99.3256 |

40% | 88.6799 | 99.8074 | 98.9283 | 98.8599 | 99.7149 |

50% | 88.6635 | 99.8476 | 99.0786 | 98.84 | 99.7469 |

TSSp | Greedy | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP |

1% | 92.8619 | 97.7642 | 94.3287 | 93.5957 | 97.0516 |

5% | 90.9306 | 99.1171 | 95.7946 | 96.4218 | 98.4031 |

10% | 91.3637 | 99.5086 | 97.5863 | 97.7154 | 99.3172 |

20% | 91.7803 | 99.7598 | 98.6129 | 98.6336 | 99.6214 |

30% | 92.1344 | 99.8473 | 99.0048 | 99.2151 | 99.6555 |

40% | 92.1866 | 99.8859 | 99.3106 | 99.2963 | 99.8365 |

50% | 92.2787 | 99.9117 | 99.4053 | 99.4852 | 99.8517 |

Coverage Significance and Time Mean Difference, Program Space.

Algorithm | Algorithm | Mean Coverage | Coverage Difference | Time Mean |

Difference (%) | Significance | Difference (s) | ||

( | ||||

Greedy Algorithm | Additional Greedy Algorithm | 0.0000 | ||

Genetic Algorithm | 0.0000 | |||

Simulated Annealing | 0.0000 | |||

Reactive GRASP | 0.0000 | |||

Additional Greedy Algorithm | Greedy Algorithm | 10.5391 | 0.0000 | |

Genetic Algorithm | 1.1354 | 0.0000 | ||

Simulated Annealing | 1.0931 | 0.0000 | 11.303 | |

Reactive GRASP | 0.1752 | 0.0613 | ||

Genetic Algorithm | Greedy Algorithm | 9.4036 | 0.0000 | 495.608 |

Additional Greedy Algorithm | 0.0000 | 478.965 | ||

Simulated Annealing | 0.4418 | 490.268 | ||

Reactive GRASP | 0.0000 | |||

Simulated Annealing | Greedy Algorithm | 9.4459 | 0.0000 | 5.339 |

Additional Greedy Algorithm | 0.0000 | |||

Genetic Algorithm | 0.0423 | 0.4418 | ||

Reactive GRASP | 0.0000 | |||

Reactive GRASP | Greedy Algorithm | 10.3639 | 0.0000 | |

Additional Greedy Algorithm | 0.0613 | 36,922.945 | ||

Simulated Annealing | 0.9180 | 0.0000 | 3,6934.249 | |

Genetic Algorithm | 0.9602 | 0.0000 | 36,443.980 |

Weighted Average for the Metrics, Program Space.

Coverage Criterion | Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP |
---|---|---|---|---|---|

Block Coverage | 87.1697 | 98.4650 | 98.53273 | 99.5424 | |

Decision Coverage | 88.3197 | 98.3631 | 98.33361 | 99.4221 | |

Statement Coverage | 92.0653 | 98.9375 | 99.02625 | 99.6819 |

Difference in Performance between the Best and the Worst Criteria, Program Space.

Greedy Algorithm | Additional Greedy | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |

Algorithm | |||||

Difference in performance | 4.8956 | 0.1300 | 0.5744 | 0.6926 | 0.2598 |

between the best and worst criteria |

Average for Each Algorithm (All Metrics), Program Space.

Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |
---|---|---|---|---|---|

Final Average | 89.1849 | 99.7240 | 98.5885 | 98.6308 | 99.5488 |

Standard Deviation of the Effectiveness for the Four Algorithms, Program Space.

Greedy Algorithm | Additional Greedy Algorithm | Genetic Algorithm | Simulated Annealing | Reactive GRASP | |
---|---|---|---|---|---|

Standard Deviation | 0.025599 | 0.000730 | 0.003065 | 0.003566 | 0.001300 |

Summary of Results, Program Space.

Algorithm | Coverage Performance | Execution Time | Observations |
---|---|---|---|

Greedy Algorithm | The worst performance. | Fast | |

Additional Greedy Algorithm | Best performance of all. | Fast | |

Genetic Algorithm | Fourth best performance. | Medium | |

Simulated Annealing | Third best performance. | Fast | No significant difference to genetic algorithm. |

Reactive GRASP | Second best performance | Slow | No significant difference to Additional Greedy Algorithm. |

APBC (Average Percentage Block Coverage), Comparison among Algorithms for Small Programs.

Tables

Analyzing the results obtained from the experiments, which are detailed in Tables

The Reactive GRASP algorithm had the second best performance. This approach also significantly outperformed the Greedy algorithm, the genetic algorithm, and simulated annealing, considering the coverage results. When compared to the Additional Greedy algorithm, there were no significant differences in terms of coverage. Comparing the metaheuristic-based approaches, the better performance obtained by the Reactive GRASP algorithm over genetic algorithm and simulated annealing was clear.

In 168 experiments, the genetic algorithm generated a better coverage only once (block criterion, the schedule program, and 100% of tests being considered). The two algorithms tied also once. For all other tests, the Reactive GRASP outperformed the genetic algorithm. The genetic algorithm approach performed the fourth best in our evaluation. In Li et al. [

Figures

APDC (Average Percentage Decision Coverage), Comparison among Algorithms for Small Programs.

APSC (Average Percentage Statement Coverage), Comparison among Algorithms for Small Programs.

For better visualization, consider Figures

Weighted Average for the Metrics (Comparison among the Metrics), Small Programs.

Weighted Average for the Metrics (Comparison among the Algorithms), Small Programs.

Weighted Average for the 3 More Efficient Algorithms (Comparison among the Metrics), Small Programs.

Weighted Average for the 3 More Efficient Algorithms (Comparison among the Algorithms), Small Programs.

Average for Each Algorithm (All Metrics), Small Programs.

To investigate the statistical significance, we used

We can also notice in Table

Another conclusion that can be drawn from the graphs is that the performance of the Reactive GRASP algorithm has remained similar for all metrics used, while Additional Greedy algorithm was a slightly different behavior for each metric.

Table

Table

The standard deviation shown in Table

The results for space program were similar to results for the four small programs. The Reactive GRASP algorithm had the second best performance. Additional Greedy algorithm, genetic algorithm, simulated annealing, and Reactive GRASP algorithms significantly outperformed the Greedy algorithm. Comparing both metaheuristic-based approaches, the better performance obtained by the Reactive GRASP algorithm over the genetic algorithm and simulated annealing is clear.

The Reactive GRASP algorithm was followed by genetic algorithm approach, which performed the fourth best in our evaluation. The third best evaluation was obtained by simulated annealing.

Figures

APBC (Average Percentage Block Coverage), Comparison among Algorithms for Program Space.

APDC (Average Percentage Decision Coverage), Comparison among Algorithms for Program Space.

APSC (Average Percentage Statement Coverage), Comparison among Algorithms for Program Space.

For better visualization, consider Figures

Weighted Average for the Metrics (Comparison among the Metrics), Program Space.

Weighted Average for the Metrics (Comparison among the Algorithms), Program Space.

Weighted Average for the 3 More Efficient Algorithms (Comparison among the Metrics), Program Space.

Weighted Average for the 3 More Efficient Algorithms (Comparison among the Algorithms, Program Space.

Final Average for Each Algorithm, Program Space.

The

These results qualify the Reactive GRASP algorithm as a good global coverage solution for the prioritization test case problem.

It is also important to mention that the results were consistently similar across coverage criteria. This fact had already been reported by Li et al. [

Note that we have tried from 1% to 100% of test cases for each program and criterion for the four small programs, and the performances of all algorithms remained unaltered. This demonstrated that the ability of the five algorithms discussed here is not deeply related to the number of test cases required to order.

In terms of time, as expected, the use of global approaches, such as both metaheuristic-based algorithms evaluated here, adds an overhead to the process. Considering time efficiency, one can see from Tables

Tables

Regression testing is an important component of any software development process. Test Case Prioritization is intended to avoid the execution of all test cases every time a change is made to the system. Modeled as an optimization problem, this prioritization problem can be solved with well-known search-based approaches, including metaheuristics.

This paper proposed the use of the Reactive GRASP metaheuristic for the regression test case prioritization problem and compared its performance with other solutions previously reported in literature. Since the Reactive GRASP algorithm performed significantly better

Our experimental results confirmed also the previous results reported in literature regarding the good performance of the Additional Greedy algorithm. However, some results point out to some interesting characteristics of the Reactive GRASP solution. First, the coverage performance was not significantly worse when compared to that of the Additional Greedy algorithm. In addition, the proposed solution had a more stable behavior when compared to all other solutions. Next, GRASP can be set to work with as many or as little time as available.

As future work, we will evaluate the Reactive GRASP with different number of iterations. This will elucidate whether its good performance was due to its intelligent search heuristics or its computational effort. Finally, other metaheuristics will be considered, including Tabu Search and VNS, among others.