To improve the technology of unmanned ground vehicles, it is necessary to conduct a proper evaluation on various technologies. Previous evaluation methods are mainly based on completion of the task; this may mislead most of teams of unmanned ground vehicles using a conservative strategy during the evaluation. In this paper, a new evaluation method is proposed. Based on typical working conditions including intersection, car-following, and obstacle-avoiding, the new evaluation indicator system is established, and the entropy-cost function method is applied to the comprehensive evaluation of unmanned ground vehicles. As reported in a numerical example, the proposed evaluation method can get a quantitative result that authentically reflects the intelligent behavior level of unmanned ground vehicles.
Based on the preestablished evaluation system, the technology of intelligent vehicles abroad has developed rapidly in recent years. Obviously an excellent evaluation system can guide participants to improve the performance of intelligent vehicles in evaluation/test. For example, none unmanned vehicles finished the entire race in the US intelligent vehicle competitions DARPA 2004 [
The 2nd DARPA Grand Challenge simply used the “the number of finishing the races” and “the total number of gates through” to rank the teams [
Most of existing foreign evaluation methods of domestic and abroad intelligent vehicle competitions used the mission-driven evaluation approach, which has an obvious shortcoming, leading many teams to adopt a conservative strategy [
This paper concentrates on proposing a novel evaluation method for intelligent vehicles that is based on information entropy and cost function. The information entropy checks all evaluation indicators’ weight from information amount aspect to handle uncertainty problems in evaluation/test; the cost function checks the intelligent level of each of abilities to get specific evaluation score. The evaluation method also concludes a new rigorous evaluation indicator system based on typical working conditions; each typical working condition of intelligent vehicles is subdivided into different physical indicators to reflect real situation. Not only the completion time of indicators but also all the details are taken into account, even down to the completion quality and veracity of the various secondary indicators. The new evaluation method refines three aspects of evaluation process: evaluation indicator system establishment, indicators’ weight arrangement, and evaluation score classification to guarantee the objectivity, comprehensiveness, and scientificalness of evaluation results. The logic and construction of the paper are shown in Figure
The logic and construction of the paper.
The final evaluation results can show the intelligent level of unmanned vehicles and its weakness and then guide the participating vehicles to move in the right direction and goals of high-tech development.
As described above, most of the practical evaluation activities were mission-driven, which resulted in the incomprehensive evaluation indicator system. Thus the evaluation results were partial. Most participants could get higher score just because of their own research superiorities. It is not fair and objective. That means a proper selection of evaluation indicators is an important part in the evaluation.
Evaluation of unmanned vehicle intelligent behavior is a multilevel comprehensive evaluation problem. Considering the characteristics of unmanned vehicles’ data which are scattered and the advantages and disadvantages of traditional evaluation indicators now, expert opinions and analysis of typical working conditions are selected; this can not only make full use of the experts’ cognitive knowledge of unmanned vehicles but also avoid missing important indicators. The result is relatively accurate with the rigorous indicators selection process (see Figure
Foundation of the evaluation indicator system.
The paper mainly takes two factors into consideration: objective one and subjective one, which are given in Figure
The typical working conditions are summarized from many intelligent vehicle competitions. In each of intelligent vehicle competitions, the participants will encounter three main working conditions; those working conditions almost conclude all of intelligent driving behaviors in evaluation/test. According to the typical working conditions including intersection, car-following [
The evaluation system of unmanned ground vehicles’ intelligent behavior based on typical working conditions.
Evaluation objective | Typical intelligent behaviors | Evaluation auxiliary indicators |
---|---|---|
Multi-indicator evaluation system for unmanned vehicles | Intersection behavior | Parking precision |
Obstacle-avoiding behavior | Early warning | |
Car-following behavior | Stimulation |
The indicator called “Parking precision” tests the ability to park the vehicle at the right place and the right time when facing intersection. “Restart ability” tests the ability to identify complex intersection conditions and participate in the traffic intelligently. “Speed capability” reflects the performance on the speed controlling, acceleration, and deceleration. “Braking deceleration” tests braking quality. “Early warning” tests video detection system and the ability to identify different obstacles. “Avoidance in right angle” tests whether the vehicle can use less space to avoid the obstacles. “Path replanning” can guarantee the vehicle will not abort the original destination. “Stimulation” tests the ability to calculate the fore car’s real-time speed, thus taking actions timely. “Safe distance” evaluates the stationary and the following model of the intelligent vehicle [
This paper introduces knowledge of information theory to determine the weight distribution parameters. In 1948, in the paper “A Mathematical Theory of Communication,” Shannon used Probability and Statistics approach to the study of communication systems, revealing that the object of communication system is information and then proposed the concept of entropy through describing the information quantitatively.
Based on Shannon’s theory, the recipient cannot predict the message that will be received; therefore, uncertainty exists objectively because of the message’s randomness. Delivery of messages is a process to eliminate the uncertainty of information, a process from uncertainty to certainty. Uncertainty in information theory is a kind of inherent property existing in working process, so the concept of entropy can be applied to the evaluation of intelligent behaviors of unmanned ground vehicles. Information entropy can determine the key indicators in terms of the amount of information to calculate the indicator weight parameters, and the entropy of information can be calculated as follows:
To judge all factors with 1~9 scaling method to determine each indicator’s importance, experts’ suggestion is summarized to construct the judgment matrix
According to the linear proportional relationship, the decision matrix
Calculation of
The real parameters that reflect indicator weights are the values of information utility; the values can be calculated by the following formula:
The importance weight of all factors of this layer corresponding to the above layer of a certain factor that can be calculated using the result of all single hierarchical sorts at the same hierarchy is as follows:
According to the 2007 DARPA Urban Challenge’s low-speed, low-density traffic environment issues, scholars at Carnegie Mellon University proposed a kind of robust highway autonomous driving technology that is combined with the cost evaluation method [
Taking the indicator “Path replanning” as a research object, unmanned vehicles need to consider the process cost control that means more planning time and more useless operating range deserve more penalty and higher process cost. The cost can be calculated by the following formula:
To make sure of the comprehensive consideration for all indicators and the full use of evaluation information, the paper adopts a weighted average algorithm to get final evaluation results by decomposing calculation. As for typical intelligent behaviors, the paper presents each cost function as
According to the evaluation indicators for unmanned vehicle intelligent behavior comprehensive evaluation model, the judgment matrix of each level is constructed following the 1~9 scaling methods by expert group composed by research members in related fields.
Typical working conditions’ behavior weight is as follows:
Affiliated indicators weight is as follows:
Then the level of indicators’ weight coefficients is determined, and the evaluation results will be more objective because of different treatments of indicators.
The weight of each indicator in the evaluation system has been calculated in Section
Since the data are rough, the score can just estimate a general intelligent level of unmanned vehicles; two teams with almost same score must make further comparison about more details. The rest of the indicators’ cost value can be calculated identically, and the first class indicators’ cost is
The score of each indicator is given based on the comparison between optimal performances and minimum acceptable performances from manned driving conditions roughly; the gap will be divided into several score ranks (see Figure
Calculation of each indicator’s score.
The method for calculating score is a transition from manned driving to driverless. As unmanned vehicles’ development direction is human-like capacity in driving, the method is accordingly beneficial for technical updating.
Also taking the indicator “Path replanning” as the research object, some data in “Future Challenge 2012” competition are shown in Table
The “Path replanning” data.
The competing teams | A | B | C | D | E | F |
---|---|---|---|---|---|---|
Planning time (s) | 13 | 18 | 10 | 13 | 11 | 15 |
Operating range (m) | 8.3 | 7.4 | 9.2 | 6.5 | 10.9 | None |
Cost value | 40 | 60 | 20 | 20 | 60 | 100 |
Table
Analyzing the data of 10 indicators of team A in the competition, “Parking precision” is corresponding to parking time and the distance between the front of the vehicle and the stop line; “Restart ability,” restart time and acceleration; “Speed capability,” horsepower; “Braking deceleration,” deceleration and braking distance; “Early warning,” lead time and accuracy rate; “Avoidance in right angle,” offset distance and maximum offset angle; “Stimulation,” time gap in gear shift; “Safe distance,” reasonable distance; “Speed optimization,” optimal speed and adjustment time. The indicators’ data and the corresponding score are listed in Table
The data and score of team A in the competition.
Indicators |
|
|
|
|
|
|
|
|
|
|
---|---|---|---|---|---|---|---|---|---|---|
Data-1 | 0.7 | 4.5 | 59 | 12.3 | 30 | 1.3 | 13 | 7 | 3.6 | 36 |
Unit-1 | m | s | h | m | s | m | s | s | M | km/h |
Data-2 | 13 | 1.5 | 2 | 89.3 | 53 | 8.3 | 18 | |||
Unit-2 | s | m/s2 | m/s2 | % | ° | m | s | |||
Cost | 20 | 20 | 40 | 20 | 20 | 60 | 40 | 80 | 20 | 40 |
Calculating the data from Table
The total cost is
The result shows that team A is at 2nd (20–40) level in the competition.
In this paper, the evaluation of unmanned ground vehicles is studied. Based on the typical working conditions of unmanned ground vehicles, a multilevel indicators evaluation system is established. Because the uncertainty is intrinsic property of each evaluation process, information entropy is applied to quantify the weight of each indicator, and each factor matches different weight coefficients to highlight the importance of the evaluation factor. Then entropy-cost function evaluation method is proposed to evaluate team A’s unmanned vehicle in “Future Challenge 2012” competition. From the quantitative results, the teams can learn the vehicle’s intelligent level generally and find their technical shortcomings in some specific indicators; thus the team will get the right development direction.
The first issue to be developed is more detailed quantification of indicators’ score. The technology development is from manned driving to driverless, but manned driving behaviors are also difficult to quantify. Therefore, the cost function should be reformed for more precise cost value through both qualitative and quantitative manned driving empirical data.
The authors declare that there is no conflict of interests regarding the publication of this paper.
This project was supported by the National Natural Science Foundation of China (no. 90920304 and no. 91120010) and the Fundamental Research Foundation of Beijing Institute of Technology (no. 20120342026).