A Comprehensive Decision-Making Approach Based on Hierarchical Attribute Model for Information Fusion Algorithms ’ Performance Evaluation

Aiming at the problem of fusion algorithm performance evaluation inmultiradar information fusion system, firstly the hierarchical attributemodel of track relevance performance evaluationmodel is established based on the structural model and functionalmodel and quantization methods of evaluation indicators are given; secondly a combination weighting method is proposed to determine the weights of evaluation indicators, in which the objective and subjective weights are separately determined by criteria importance through intercriteria correlation (CRITIC) and trapezoidal fuzzy scale analytic hierarchy process (AHP), and then experience factor is introduced to obtain the combination weight; at last the improved technique for order preference by similarity to ideal solution (TOPSIS) replacing Euclidean distance with Kullback-Leibler divergence (KLD) is used to sort the weighted indicator value of the evaluation object. An example is given to illustrate the correctness and feasibility of the proposed method.


Introduction
Cyber-physical system (CPS) was listed as an important research project in USA competitiveness plan released by American Academy of Sciences in February 2006.CPS can be understood as a high-performance network intelligent information system based on embedded devices [1][2][3][4][5][6][7][8].Through the highly integration and interaction of a series of calculation units and physical objects under network environment, the capacity of information processing, realtime communication, precise remote control, and component self-coordination is improved.So it is a hybrid autonomous system which is heterogeneous and multidimensional in time and space.
As an application of CPS in military field, command, control, communication, and intelligence (C 3 I) system has been developed and improved all over the world to complete specific strategic and tactical missions in nearly 20 years.How we analyze and process a large amount of information collected by various ways to make the right decisions in C 3 I system has become a new and important research branch of data processing [3,[5][6][7][8][9].Multiradar information fusion system (mRIFS) is a typical example of C 3 I system.In mRIFS, a multilevel and multifaceted processing for the multiple radars data is executed, and the measurement information provided by a number of radars is analyzed, processed, and integrated to produce the desired synthesis information [10][11][12].
Through the information integration, the expanded time coverage area, and the increased confidence level, ambiguity can be reduced and spatial resolution and reliability can be improved.A number of distinctive algorithms of information fusion technology have been formed and developed.The K-nearest neighbor (KNN) algorithm is widely used in information fusion and other related respects [13].Hayat and Khan analyzed the performance of various machine learning mechanisms for discriminating outer membrane proteins (OMPs) such as genetic programming, K-nearest neighbor, and fuzzy K-nearest neighbor (fuzzy K-NN) in conjunction with discrete methods such as amino acid composition, amphiphilic pseudo amino acid composition, split amino acid composition (SAAC), and hybrid versions of these methods and observed that fuzzy K-NN using SAAC basedfeatures makes it quite effective in discriminating OMPs after the simulation [14].Zhang and Wang investigated the combined feedback-feedforward tracking control problem for networked control systems (NCSs) under the discrete-time framework and proposed the controller design method by solving a nonlinear trace minimization problem.Finally, they gave an example on the control of a helicopter to illustrate the proposed design approach [15].Tomašev et al. went a step further by embracing the soft approach and proposed several fuzzy measures for K-nearest neighbor classification, all based on hubness, which express fuzziness of elements appearing in K-neighborhoods of other points [16].Zhang et al. investigated the vehicle lateral dynamics stabilization problem to enhance vehicle handling by considering timevarying longitudinal velocity.They described the longitudinal velocity by a polytope with finite vertices and proposed a novel technique to reduce the number of vertices [17].Ran and Zhang studied a data fusion algorithm based on track fuzzy membership.Carlo simulation shows that the proposed algorithm is superior to weighted track association algorithm, and tracking system can obtain the target fusion track that has the same tracking performance of simple fusion (SF) algorithm [18].Moosavian et al. surveyed a new scheme for fault diagnosis of main journal-bearings of internal combustion (IC) engine based on power spectral density (PSD) technique and two classifiers, namely, K-nearest neighbor (KNN) and artificial neural network (ANN) [19].Zhang et al. proposed a mode-dependent estimator with a fast sampling frequency such that the estimation can track the signal to be estimated with the nonuniformly sampled outputs.They obtained the corresponding stochastic estimation error system by using an augmentation technique and derived the linearmatrix-inequality-based sufficient conditions by studying the stochastic stability and the energy-to-peak performance of the estimation error system [20].Zhang and Wang investigated H∞ the filtering problem of discrete-time Takagi-Sugeno (T-S) fuzzy systems in a network environment and presented a tunnel-diode circuit in a network environment to show the effectiveness and the advantage of the proposed design approach [21].Huang and Wu proposed an algorithm of weighted covariance for centralized asynchronous fusion based on Kalman (WCCAF), which made linear minimum mean square error (LMMSE) the criterion, and considered the relevance of measurement noise [22].Zhang et al. exploited the sliding mode control problem for a fluid power electrohydraulic actuator (EHA) system [23].Zhang et al. investigated the problem of robust static output feedback (SOF) control for networked control systems (NCSs) subject to network-induced delays and missing data [24].
For the very extensive fusion object and different application requirement of all military departments, the demand for comprehensive decision-making approach for information fusion algorithms' performance evaluation is ubiquitous [25].The experiments for a variety of fusion algorithms and evaluation of the experimental results have a considerable significance to find the most suitable fusion algorithm for the current data source condition.Through the fusion algorithms' performance evaluation, the processing capability of fusion algorithm under different conditions of data source can be shown by the evaluation results.

Multiradar Information Fusion System: An Application of Cyber-Physical Systems in Military C 3 I Field
A typical air traffic control (ATC) distributed mRIFS has a structural model as is shown in Figure 1.All radar stations complete the tracking process of detected target locally.After converting the original point track to local single radar track, the data will be sent to ATC control center through network to form multiradar track of all targets and will be sent to front display to provide a variety of control applications [7][8][9].The flight plan, line, path, chart and meteorological information, and other related flight data are stored in system database.
Additionally, the single radar data and multiradar fusion data are put in storage real time.
The most authoritative functional model of data fusion systems (DFS) is proposed by USA, and its simplified form is shown in Figure 2 with C 3 I system as an example.According to the function modules supported by comprehensive information process, the whole process is divided into two phases: in the first phase, integrate the data providing target state parameter estimation and classification information and enrich and update the database; in the second phase, based on the first phase and supported by the knowledge base, comprehensively infer the semantic information [4,12].The information fusion of mRIFS functional model is generally divided into five levels, namely, detection-level fusion, location-level fusion, attribute-level fusion, situation evaluation, and threat assessment.

Hierarchical Attribute Model
Comprehensive decision-making is a conversion process which changes the multiple attributes system from highdimensional to low-dimensional, in which it is required that decision-making results can reflect the information of original indicators as much as possible.Aiming at the actual needs of military electronic information system software's development, we carry out the research of comprehensive decision-making approach for information fusion algorithms' performance evaluation.On the basis of the analysis of performance indicators for mRIFS fusion algorithm, the hierarchical attribute model (HAM) of track relevance performance is built.

The Structure of Hierarchical Attribute Model.
As is shown in Figure 3, the target of fusion algorithm performance evaluation (FAPE) can be divided into three standards, namely, data source quality standard (DSQ), track quality standard (TQ), and track association quality standard (TAQ).In this paper, we only analyze and study the eight indicators of TAQ, namely, non-common target ratio (nCTR), state position estimation error (SPEE), state speed estimation error (SSEE), correct relevant rate (CRR), error relevant rate (ERR), leakage relevant rate (LRR), correct separation rate (CSR), and track  repeat factor (TRF). nCTR, SPEE, and SSEE belong to the substandard of local node information consistency (LNIC); CRR, ERR, and LRR belong to the substandard of relevant rate (RR); CSR and TRF belong to the substandard of fusion degree (FD).

Indicator Definition and Its Quantization.
Assuming that the number of integration centers and radars is 1 and , respectively, the outcome track number of the radars  is   .In the case of two radars  and , their outcome track sets are   and   , which can form    ×    relevant information ( If there are a couple of radars  and ,  Figure 4: The procedure of the comprehensive decision-making approach.

Comprehensive Decision-Making Approach
According to the HAM, the comprehensive decision-making approach is proposed based on combination weight and TOPSIS improved by Kullback-Leibler divergence.When the performance evaluation is achieved, it can provide support for the optimization selection of mRIFS fusion algorithms.The determination of indicator weight is one of the key steps of comprehensive decision-making.Objective weighting and subjective weighting are two usual types of the determination methods.The latter is decision-makers' subjective judgment of the importance degree of indicator's influence on the target, which is the expression of decisionmakers' preference and is usually determined by AHP.Then the method of TOPSIS is usually used to sequence the objects in the weighted normalization decision matrix.As is shown in Figure 4, this paper has proposed a comprehensive decision-making approach of combination weighting and TOPSIS improved by Kullback-Leibler divergence based on the related research.

The Objective Weight.
The objective weight is an abstract representation of the inherent relationship among all evaluation objects, which is determined by entropy method [26], standard deviation method [27], criteria importance through intercriteria correlation (CRITIC) [28], and so forth.
In this paper, there are  indicators and  evaluation objects, and every evaluation object represents the result of an experiment.So the indicator decision-making matrix is defined as  = ( , ) × , in which  , represents the value of evaluation object  on the indicator V  (0 ≤  ≤ , 0 ≤  ≤ ).To eliminate the effects of different dimensions between indicators, take the following standardized processing to obtain the standardized indicator decision-making matrix.
(iii) For the indicator which is better if it is more close to its mid-value, where ( (1) Entropy Method.Entropy is a measure of the uncertainty of system state [29].Assuming the system has  different states and the appearance probability of each state is   ( = 1, 2, . . ., ), the entropy of the system is  = − ∑  =1   ln   (0 ≤   ≤ 1, ∑  =1   = 1).When  statuses have the same appearance probability of   = 1/, the entropy is maximum; that is, max  = ln .
For the  indicators, the entropy value of indicator ) according to the entropy theory.As   gets bigger, the differences among the values of all evaluation objects on V  become smaller.It is generally believed that the indicator V  is more important if the differences among the values of all objects on V  are bigger.Using entropy method, the objective weight of the indicator V  is (2) Standard Deviation Method.The principles of standard deviation method and entropy method are similar.If the standard deviation value of an indicator is bigger, the variation degree of the indicator is higher.So this indicator can play a more important role in the comprehensive decision-making, and its weight should be bigger.Using standard deviation method, the objective weight of the indicator V  is where is the standard deviation of the indicator V  .
(3) CRITIC.The idea of CRITIC is to determine the objective weight based on two concepts: the variability of an indicator and the conflict among indicators.The variability of the indicator V  represents the indicator value difference among all evaluation objects with the form of the standard deviation   .If   is bigger, the value difference of V  among all evaluation objects is greater.On the basis of the correlation between two indicators, the conflict among indicators can be expressed as is the correlation coefficient between the indicators V  and V  and  , ,  , are the arithmetic average value of V  and V  .If there is a strong positive correlation between two indicators, the conflict of them is low.The information volume of V  is determined through synthesis consideration of the variability of V  and the conflict between V  and other indicators.If an indicator has a larger information volume, it is more important.Using CRITIC, the objective weight of the indicator V  is where   =     is the information volume of V  .

The Subjective Weight.
Traditionally, the exact number is generally used in decision-makers' judgments to express the relative importance between two elements of the AHP judgment matrix.Human beings are often with ambiguity, and using the "little, " "clear, " "strong, " and other vague concepts to express their purpose is more rational.Therefore, based on the integration of the trapezoidal fuzzy number and scale method, the indicators' subjective weights are determined through the trapezoid fuzzy scale AHP.
The natural number (1, 2, . . ., ) can be converted to trapezoidal fuzzy number k in turn based on the membership function of trapezoid fuzzy number, so that it is When evaluation matrix is built by decision-makers' fuzzy comment, the -level comparative scale method is often used to determine the element value of evaluation matrix, and  is usually odd.To some extent, this method can reflect ambiguity of decision-makers but still cannot more "aptly" reflect the actual thinking.Therefore, with a division of two trapezoidal fuzzy numbers to improve the n-level comparative scale according to formula (7), we have the trapezoidal fuzzy scale value R as formula (8).The comments of the 9-level comparative scale are "extremely inferior, " "strongly inferior, " "significantly inferior, " "slightly inferior, " "equal, " "slightly superior, " "significantly superior, " "strongly superior, " and "extremely superior", and their trapezoidal fuzzy scale values are 1/ 9, 2/ 8, . . ., 9/ 1.Therefore, The subjective weight is determined as follows.
Step 4. Assuming that TRPEM shown in Figure 3 has  levels, repeat the above steps top-down.

The Combination Weight.
According to the objective weight vector and subjective weight vector, the experience factor is introduced in the method, so the combination weight is Here, the experience factor represents the decisionmakers' preferences for the objective and subjective weights.If decision-makers think that the objective weight is more important than the subjective weight, select 0.5 < Exp < 1; otherwise select 0 < Exp < 0.5.If decision-makers think that both of them are equally important, select Exp = 0.5.The weighted normalized decision-making matrix  is obtained,  = (ℎ , ) × , ℎ , =   ⋅  , .

Improved TOPSIS by Kullback-Leibler Divergence.
In a limited number of evaluation objects, make a set of optimal indicator values and a set of the worst indicator values as the positive ideal point and negative ideal point, respectively, and sort evaluation objects with their proximity to the ideal point.This is the basic idea of TOPSIS.In traditional TOPSIS, Euclidean distance between the evaluation object and the ideal point is used to indicate the proximity, which cannot indicate the proximity of evaluation objects on the vertical line of the two ideal points and fully reflect the merits of all objects.Corresponding solutions mainly include improving the closeness by the cosine, replacing Euclidean distance with the vertical plane distance or vector projection, and partial sequence alignment, but these methods have some deficiencies.
Therefore, replacing Euclidean distance with Kullback-Leibler divergence between evaluation object and the ideal points is reasonable, which coincides with the basic sort principles of TOPSIS, and can solve the problem of the above improved methods.So calculate the Kullback-Leibler divergence closeness between each evaluation object and the ideal points successively and get the final evaluation results by sorting them in the descending order [16,17].

Case Study
Experiment conditions were as follows: (i) flight target type: airplane XX-50; (ii) radar type: radar XX-713; (iii) detection resolution: distance of 500 m, direction of 0.55 ∘ , and height of 0.83 m; (iv) random error: distance of 500 m, direction of 0.3 ∘ , and height of 0.
The first radar's longitude is set to 120 ∘ 0  26  and the latitude 25 ∘ 0  32  .The second radar's longitude is set to 119 ∘ 0  26  and the latitude 24 ∘ 0  32  .In the experiment, the number of flight targets is from 3 to 13, and the tracks include horizontal-flight and cross flight.As the data source, both the data collected by the two radars and the real flight data are put into the information integration testing laboratory platform.
The key of multitarget track is data association.For the association of trace points generated by radar data, the square gate and elliptical gate are used as the tracking gate, and the nearest neighbor data association (NNDA) and probabilistic data association (PDA) are used as the association algorithm.
After being associated with trace points, the tracking information is put into the track fusion module for track association and fusion, and the fusion result is also the object to be analyzed in this paper.KNN is used for track association, and SF is used for track fusion.
The indicators can be divided into three types as follows.
(i) The efficiency type indicators include CRR (in theory, its minimum value is 0 and maximum value is 1) and CSR (in theory, its minimum value is 0 and maximum value is 1).
(ii) The cost type indicators include nCTR (in theory, its minimum value is 0 and maximum value is the number of targets), SPEE (in theory, its minimum value is 0 and maximum value is 2 km), SSEE (in theory, its minimum value is 0 and maximum value is 40 m/s), ERR (in theory, its minimum value is 0 and maximum value is 1), and LRR (in theory, its minimum value is 0 and maximum value is 1).
(iii) The indicator which is better if it is more close to its mid-value includes TRF (in theory, its minimum value is 0.7 and maximum value is 1.3).
The original experimental data is shown in Table 1.
After standardized processing, the standardized indicator decision-making matrix can be obtained as .0000 0.3714 0.4722 1.0000 1.0000 1.0000 1.0000 0.9775 1.0000 0.8864 0.7981 1.0000 1.0000 1.0000 1.0000 0.9775 1.0000 0.6500 0.4250 1.0000 1.0000 1.0000 1.0000 0.9775 1.0000 0.8387 0.9140 1.0000 1.0000 1.0000 1.0000 0.9775 1.0000 1.0000 0.6800 1.0000 1.0000 1.0000 1.0000 0.9775 1.0000 0.5571 0.4198 0.8300 1.0000 0.0024 1.0000 0.8525 0.0018 0.4333 0.7727 0.7500 1.0000 0.0016 1.0000 0.9053 1.0000 0.5571 1.0000 0.6700 0.0014 0.0012 0.7800 0.9775 0.0014 0.5000 0.5862 0.7800 0.0014 0.0018 0.7800 0.9775 0.0016 0.3545 0.5075 0.7200 0.0033 0.0014 0.9100 0.8270 5.1.The Objective Weight.Using entropy method, standard deviation method, and CRITIC, respectively, we can obtain three objective weight vectors shown in Table 2. Through the comparison of them in Figure 5, it is found that the assignment of objective weight determined by entropy method tends towards equilibrium, and the assignments of objective weight determined by standard deviation method and CRITIC are in general close.Entropy method, which reduces the competition of different indicators and makes the assignment of objective weight balanced, cannot fully reflect the objective weight assignment.Standard deviation method and CRITIC can reflect more competition  information than entropy method.Taking both the variability of an indicator and the conflict between it and other indicators into consideration, CRITIC is more reasonable than standard deviation method.The analysis result is consistent with the above definition of these methods in Section 5.1.So we select CRITIC to determine the objective weight.The objective weight vector is  = [0.21320.0952 0.1671 0.0496 0.2119 0.1823 0.0493 0.0315]  .

The Subjective Weight.
Due to space limitations, we take the substandard level for an example using trapezoidal fuzzy scale AHP.LNIC, RR, and FD belong to the standard level node TAQ.
Step 1. Assuming that there are three decision-makers, the fuzzy reciprocal judgment matrices are  Step 2. After calculation, all of them can pass the consistency check and can be used to calculate the subjective weights.
Step 4. Repeating the above steps, the subjective weights of all nodes relative to the upper node are obtained as is shown in Figure 6.
The evaluation results of the ten experiments are shown in Table 3. Obviously, the fusion results of the 2nd, 4th, and 5th experiment are relatively satisfactory in the ten experiments, and the 8th, 9th, and 10th experiment have a lower evaluation score.
The target numbers of the ten experiments are 3, 3, 3, 4, 4, 5, 8, 9, 11, and 13, respectively.The last three experiments have more targets than others.Evaluation results show that the processing capability of fusion algorithm (KNN and SF) is weak for the condition of more than 10 targets.This helps to select optimized fusion algorithms to suit the current status of data source, such as selecting the modified K-nearest neighbor algorithm (MKNN) for track correlation and weighted covariance fusion algorithm (WCF) for track fusion.

Conclusions
The evaluation results of information fusion algorithms' performance are of great significance for the selection of fusion algorithms.Aiming at this problem, we get the following.3.5374 0.0080 0.0023  10  3.3903 0.0022 0.0007 (1) The structural model and functional model were analyzed and studied to establish the HAM of track relevance performance, and then the formal description and detailed quantization methods of HAM's indicators are given.
(2) To determine the weights of HAM's indicators, a combination weighting method based on subjective and objective weights is proposed.
The exact number is used to express decision-makers' subjective judgment in traditional AHP, which cannot accurately reflect the ambiguity of the human mind.Using the trapezoidal fuzzy numbers to represent the proportion scale is more reasonable, so the method of trapezoid fuzzy scale AHP is proposed to determine the subjective weights.The determining methods of the objective weight include standard deviation and evidence distance.We select the entropy method to determine the objective weight.Finally, the experience factor is introduced to obtain the combination weight, which can solve the information loss problem of single weighting and make the weight more close to actual result.
(3) The improved TOPSIS replacing Euclidean distance with Kullback-Leibler divergence is used to sort the weighted indicator value of the evaluation object, which can solve the problems of traditional TOPSIS, vector projection TOPSIS, and partial order TOPSIS.The method is easy to be programmed.
The case at last shows that the approach is simple and easy to do, and a scientific and reasonable result of information fusion algorithms' performance evaluation is obtained, which can provide support for us to select more optimized fusion algorithms of mRIFS.The outcome track set of the radars    :

Notations
The outcome track set of the radars     : The element number of the sets      ×    : The number of relevant information pairs formed by   and     : The set of relevant information pairs composed by the same target   : The set of information pairs correctly judged to be relevant by relevant processing   : The set of information pairs correctly judged to be irrelevant by relevant processing   : The set of relevant information pairs composed by different targets   : The set of information pairs correctly judged to be irrelevant by relevant processing   : The set of information pairs mistakenly judged to be relevant by relevant processing nCTR , : The non-common target ratio of the two radars  and   ,, : The state position estimation error of the track  in   ∪    ,, : The state speed estimation error of the track  in   ∪    , : The state position estimation error of a couple of radars  and   , : The state speed estimation error of a couple of radars  and    : The correct relevant rate (the rate of information pairs which are formed by the same objective and correctly judged to be relevant)   : The error relevant rate (the rate of information pairs which are formed by different objectives and mistakenly judged to be relevant)   : The set of partial error relevant tracks   : The partial error relevance rate   : The set of partial error transfer tracks   : Thepartialerrortransferrate   : The set of error switch tracks   : Theerrorswitchrate   : The leakage relevant rate (the rate of information pairs which are formed by the same objective and mistakenly judged to be irrelevant)   : The correct separation rate (the rate of information pairs which are formed by different objectives and correctly judged to be irrelevant)   : The track repeat factor  , : The value of evaluation object  on the indicator V  (0 ≤  ≤ , 0 ≤  ≤ )  , : Thestandardizedformof , : The entropy of an uncertainty system   : The entropy value of indicator V   V  : The objective weight of the indicator V  : The objective weight vector   : The standard deviation of the indicator V    : The conflict information of the indicator V    : The correlation coefficient between the indicators V  and V    : The information volume of the indicator V  k: The trapezoidal fuzzy number R: The trapezoidal fuzzy scale value   : A node of the hierarchical attribute model x()  : The importance degree of   relative to   given by the decision-maker  X() : The fuzzy reciprocal judgment matrix of the decision-maker   () : Thesimpleformof X() ker( x()  ): The kern of x()  F  : The fuzzy evaluation value of relative weight of    F  (): Themembershipfunctionof F  : The optimism factor  (−1) : Therelativeweightvectorof −1 elements in level  − 1 for the total target  ()   : Therelativeweightvectorof  elements in level  to the element  in level  − 1  () : The relative subjective weight vector of the elements in level  to the total target Exp: The experience factor (the decision-makers' preferences for the objective and subjective weights) : The combination weight vector : Theweightednormalized decision-making matrix : The difference degree of two -dimensional uncertain systems Ψ  and Ψ    : The evaluation object (the row  of )  + : The positive ideal point of   − : The negative ideal point of   +  : The Kullback-Leibler divergence from evaluation object   to the positive ideal point  −  : The Kullback-Leibler divergence from evaluation object   to the negative ideal point KC  : The Kullback-Leibler divergence closeness between   and the ideal points.

Figure 5 :
Figure 5: The subjective weights of all nodes relative to the upper node.
: Th eo u t c o m et r a c kn u m b e ro ft h er a d a r s   : is the element number of the sets   .We can get the following relationship equations:    =    +    ,    =    +    , and    ×    =    +    , in which   indicates the set of relevant information pairs composed by the same target;   indicates the set of information pairs correctly judged to be relevant by relevant processing and   indicates the set of information pairs correctly judged to be irrelevant by relevant processing;   indicates the set of relevant information pairs composed by different targets;    ∪  −    ∩  )/   ∪  .
pairs; that  and  are the longitude and latitude, while  and  are the speed in longitude and the speed in latitude, so the track state estimation vector can be represented as[   ].For the track  in   ∪   , its SPEE and SSEE are The partial error transfer rate is   =    /   , in which   is the set of partial error transfer tracks.(iii)If the first half of  is relevant to the true track of the objective  and the second half is relevant to the true track of the objective ,  ̸ = , it means error switch.The error switch rate is   =    /   , in which   is the set of error switch tracks.This includes leakage relevant rate (LRR).It means the rate of information pairs which are formed by the same objective and mistakenly judged to be irrelevant,   =    /   .This includes correct separation rate (CSR).It means the rate of information pairs which are formed by different objectives and correctly judged to be irrelevant,   =    /   .Definition 7.This includes track repeat factor (TRF).If the number of tracks outcomes by integration center is    and the number of true tracks is    , the TRF is   =    /   .
, = (∑    ∩  =1  ,, )/   ∩  ,  , = (∑    ∩  =1  ,, )/   ∩  .Definition 3.This includes correct relevant rate (CRR).It means the rate of information pairs which are formed by the same objective and correctly judged to be relevant,   =    /   .Definition 4. This includes error relevant rate (ERR).It means the rate of information pairs which are formed by different objectives and mistakenly judged to be relevant,   =    /   .Assuming that the track  ∈ (1, 2, . . .,    ), we get the following.(i) If part of  is relevant to the true track of the objective  and the other is falsely relevant, it means partial error relevance.The partial error relevance rate is   =    /   , in which   is the set of partial error relevant tracks.(ii) If part of  is relevant to the true track of the objective  and the other is relevant to the true track of the objective ,  ̸ = , it means partial error transfer.Survey s (s > 1) . The difference degree of two -dimensional uncertain systems Ψ  and Ψ  can be measured with KLD; that is,  = ∑ It can be proved that  ≥ 0 and  = 0 when and only when Ψ  = Ψ  .Definition 8.The Kullback-Leibler divergence from evaluation object   to the positive ideal point is  +

Table 1 :
The original experimental data (FTN: actual flight target number).

Table 2 :
Comparison of the objective weights determined by entropy method, standard deviation method, and CRITIC.

Table 3 :
The evaluation results of the ten experiments.