Journal of Applied Mathematics

Volume 2014 (2014), Article ID 145285, 9 pages

http://dx.doi.org/10.1155/2014/145285

Research Article

A New Uncertainty Evaluation Method and Its Application in Evaluating Software Quality

Jiqiang Chen,¹ Litao Ma,¹ Chao Wang,² Hong Zhang,¹ and Jie Wan³

¹School of Science, Hebei University of Engineering, Handan 056038, China

²School of Economics and Management, Hebei University of Engineering, Handan 056038, China

³School of Energy Science and Engineering, Harbin Institute of Technology, Harbin 150001, China

Received 6 February 2014; Accepted 28 May 2014; Published 2 July 2014

Academic Editor: Chao Yan

Copyright © 2014 Jiqiang Chen et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract.

Uncertainty theory is a branch of axiomatic mathematics dealing with experts’ belief degree. Considering the uncertainty with experts’ belief degree in the evaluation system and the different roles which different indices play in evaluating the overall goal with a hierarchical structure, a new comprehensive evaluation method is constructed based on uncertainty theory. First, index scores and weights of indices are described by uncertain variables and evaluation grades are described by uncertain sets. Second, weights of indices with respect to the overall goal are introduced. Third, a new uncertainty comprehensive evaluation method is constructed and proved to be a generalization of the weighted average method. Finally, an application is developed in evaluating software quality, which shows the effectiveness of the new method.

1. Introduction

Due to human language and technology difficulties, it is difficult to provide an effect and objective evaluation for a system. Therefore, many scholars attempt to establish new mathematical methods to make evaluation results consistent with actual situations. Accordingly, Saaty [1] proposed the analytic hierarchy process (AHP), a method to address multicriteria decision analysis with quantitative and qualitative information. In AHP, human judgments are represented as exact numbers. However, some decision-makers may be reluctant or unable to assign exact numbers to comparison judgments because some evaluation criteria are subjective and qualitative. Therefore, Wu and Tsai [2] used both AHP and decision-making trial and evaluation laboratory methods to evaluate the criteria in autospare parts industry in Taiwan. Kumar et al. [3] presented a new general procedure to construct the membership and nonmembership functions of the fuzzy reliability using time-dependent intuitionistic fuzzy set. By using the finite Markov chain imbedding approach, Zhao and Cui [4] presented a unified formula with the product of matrices for evaluating the system state distribution for generalized multistate -out-of-: systems. Chen et al. [5] provided an evaluation method for enterprisers making investment decisions under hybrid cloud environment using grey system theory. Lee et al. [6] proposed a systematic approach to evaluation of new service concepts by integrating the merit of group analytic hierarchy process in modeling multicriteria decision-making problems. Geng et al. [7] presented a new integrated design concept evaluation approach based on vague sets in order to provide a method for complicated multicriteria decision-making problem under uncertain environments. Aiming to evaluate the government and the monopolist about the consumer’s taste, literature [8] was devoted to the characterization and quantitative representation of imprecise and vague uncertainties and measures of information produced by sources of the considered type. Kramosil [9] introduced the possibilistic variants of both the minimax (the worst case) and the Bayesian optimization principles and applied them in decision-making under uncertainty processed. Using finite-time control and backstepping control approaches, Li et al. [10] proposed a new robust adaptive synchronization scheme to make the synchronization errors of the systems with parameter uncertainties zero in a finite time. Lan et al. [11] presented a bilevel fuzzy principal-agent model for optimal nonlinear taxation problems with asymmetric information and so on.

The above methods address imprecise information, such as human language or experts’ degree of belief using fuzzy set theory (see Zadeh [12]), vague set theory, and grey system theory. However, for the evaluation system, the observed data are often not adequate and we have no choice but to invite some domain experts to evaluate the belief degree that an index belongs to an evaluation grade. In this situation, many surveys show that this imprecise information behaves like neither fuzziness nor randomness (see Liu [13] and Liu [14]). And it was showed by Kahneman and Tversky [15] that human beings usually overweight unlikely events. This fact makes the personal belief degree have much larger variance than the frequency.

In this case, Liu [13] proposed uncertainty theory to deal with belief degree, and Liu [14] refined uncertainty theory. Nowadays, uncertainty theory has become a branch of axiomatic mathematics. The first fundamental concept in uncertainty theory is the uncertain measure, used to measure the degree of belief in an event. The second concept is the uncertain variable, used to represent quantities with imprecise information (e.g., the exact value of oil field reserve). The third concept is uncertainty distribution, which is used to describe uncertain variables. Uncertainty theory has been applied to many areas. Liu [16] established a theory and practice of uncertain programming, Liu [17] applied uncertainty theory to risk analysis and reliability analysis, Liu [18] studied hybrid logic and uncertain logic, Liu [19] proposed inference rule with applications to uncertain control, and Liu [20] studied uncertain process with applications to inference risk model. To explore the recent developments in uncertainty theory, readers may consult Liu [21].

Subsequently, Liu [22] described the weights of indices and the score values of indices with uncertain variables and proposed a comprehensive evaluation method based on uncertainty theory. However, in some certain kinds of assessment domains, we find that different bottom indices play different roles in the evaluation of the overall goal. For example, suppose that and are two students whose four features are shown in Table 1.

Table 1: Four features of students and .

Students
Gender

(male = 1, female = 0)
Age

(years)
Body height

(cm)
Body weight

(kg)

1
20
180
80

1
20
170
75

Therefore, the feature vectors of and are and . Because and are with the same “Gender” and “Age,” we cannot identify them from the two features. In other words, “Gender” and “Age” do not take effect at all in the identification. Furthermore, because the “Body Height” of and is 180 and 170, respectively, they can be identified by “Body Height.” Of course, they can also be identified by “Body Weight.” In a word, the four features play different roles in the identification of and . For this reason, a weight for each bottom index with respect to the overall goal is introduced to show the different roles of different bottom indices. And without loss of generality, when different bottom indices play the same role in other assessment domains, weights of indices with respect to the overall goal are equal. Considering these reasons, a new evaluation method is proposed based on uncertainty theory.

The structure of this paper is as follows. Section 2 provides some relevant concepts about uncertainty theory. Section 3 establishes a new uncertainty evaluation method based on uncertain sets and uncertain variables. Section 4 gives an application of the proposed method in evaluating software quality, and the conclusions are presented in Section 5.

2. Preliminaries

In this section, we provide some useful definitions of uncertainty theory.

Let be a nonempty set, let be a -algebra on , and let be a set of real numbers. Each element is called an event. The uncertain measure , defined on , was proposed by Liu [13] as follows.

Definition 1 (see Liu [13]). The set function is called an uncertain measure if it satisfies the following.

Axiom 1 (normality axiom). Consider .

Axiom 2 (duality axiom). Consider for any event .

Axiom 3 (subadditivity axiom). For every countable sequence of events , we have

The triplet is called an uncertainty space. Let be uncertainty spaces for . Write

Then the product uncertain measure on the product -algebra is defined by the following product axiom [23].

Axiom 4 (product axiom). Let be uncertainty spaces for . The product uncertain measure is an uncertain measure satisfying

where are arbitrarily chosen events from for , respectively.

Remark 2. Uncertain measure is interpreted as the personal belief degree (not frequency) of an uncertain event that may occur. It depends on the personal knowledge concerning the event. The uncertain measure will change if the state of knowledge changes.

Definition 3 (see Liu [13]). An uncertain variable is a measurable function from an uncertainty space to the set of real numbers. That is, for any Borel set of real numbers, the set

is an event.

Definition 4 (see Liu [14]). Let be a real-valued measurable function, and let be uncertain variables on . Then is an uncertain variable defined by

Definition 5 (see Liu [14]). The uncertainty distribution of an uncertain variable is defined by

for any real number (see Figure 1).

Figure 1: An uncertainty distribution.

Definition 6 (see Liu [14]). An uncertain variable is called zigzag if it has a zigzag uncertainty distribution

denoted by , where , , are real numbers with .

Definition 7 (see Liu [23]). The uncertain variables are said to be independent if

for any Borel sets of real numbers.

Theorem 8 (see Liu [21]). Assume that and are independent zigzag uncertain variables and , respectively. Then the sum is also a zigzag uncertain variable ; that is,

The product of a zigzag uncertain variable and a scalar number is also a zigzag uncertain variable ; that is,

Definition 9 (see Liu [13]). Let be an uncertain variable. Then the expected value of is defined by

provided that at least one of the two integrals is finite.

Remark 10. Expected value is the average value of uncertain variable in the sense of uncertain measure and represents the size of uncertain variable.

Example 11 (see Liu [21]). The zigzag uncertain variable has an expected value

Theorem 12 (see Liu [14]). Let and be independent uncertain variables with finite expected values. Then for any real numbers and , one has

An uncertain set is a set-valued function on an uncertainty space that attempts to model “unsharp concepts,” which are essentially sets but their boundaries are not sharply described (because of the ambiguity of human language), such as “young” and “tall.” A formal definition is given as follows.

Definition 13 (see Liu [19]). An uncertain set is a measurable function from an uncertainty space to a collection of sets of real numbers. For any Borel set of real numbers, that is, for any Borel set of real numbers, both of

are events.

Definition 14 (see Liu [24]). An uncertain set is said to have a membership function if for any Borel set of real numbers one has

The above equations will be called measure inversion formulas.

Remark 15. Let be a set of real numbers. When an uncertain set has a membership function on , we immediately have

Liu [24] proved that a real-valued function is a membership function if and only if (see Figure 2).

Figure 2: A membership function.

Example 16 (see Liu [21]). By a triangular uncertain set we mean the uncertain set fully determined by the triplet of crisp numbers with , whose membership function is

Definition 17 (see Liu [24]). Let be an uncertain set with a membership function . Then the set-valued function

is called the inverse membership function of .

Definition 18 (see Liu [25]). The uncertain sets are said to be independent if for any Borel sets one has

where are arbitrarily chosen from ,  , respectively.

Theorem 19 (see Liu [24]). Let and be independent uncertain sets with membership functions and , respectively. Then their union has a membership function

Theorem 20 (see Liu [24]). Let and be independent uncertain sets with membership functions and , respectively. Then their intersection has a membership function

Theorem 21 (see Liu [24]). Let be an uncertain set with membership function . Then its complement has a membership function

Theorem 22 (see Liu [24]). Let be independent uncertain sets with inverse membership functions , respectively. If the function is a measurable function, then

is an uncertain set with inverse membership function

Definition 23 (see Liu [19]). Let be a nonempty uncertain set. Then the expected value of is defined by

provided that at least one of the two integrals is finite.

Please note that represents “ is imaginarily included in ” and represents “ is imaginarily included in .”

Example 24 (see Liu [21]). The triangular uncertain set has an expected value

3. Uncertainty Evaluation Method

When making a comprehensive evaluation, factors influencing the grade of the overall goal should be considered. The index system is often represented by a three-layer hierarchical structure with the overall goal, the second layer, and the bottom layer (namely, the factors influencing the overall goal’s scaling). Experts always intend to show their own opinions and expectations of each evaluation index, so the evaluation results represent human uncertainty and belief degree. Therefore, the score value and weight of each evaluation index are represented by uncertain variables, and evaluation grades are represented by uncertain sets. Therefore, a new evaluation method based on uncertain variables and uncertain sets is proposed.

3.1. Establishment of Hierarchical Index Structure

Based on evaluation criteria, a hierarchical index structure is established by experts as in Figure 3. There are indices in the second level and indices immediately below index (), where . Let the index (in the bottom level) set be .

Figure 3: A simple hierarchical index structure.

3.2. Weights of Indices

Suppose that there are experts in the evaluation group and the degrees of importance are represented by uncertain variables . Each expert gives a degree of importance for each index. For index (; ), experts select , experts select experts select , where . Weight of index is represented by

According to Definition 3, weight is an uncertain variable and can be described by its uncertainty distribution .

3.3. Grade Vectors of Bottom Indices

The score values from experts can be represented by uncertain variables. Let the evaluation grades (e.g., Poor, Fair,…, Excellent) be represented by uncertain sets , with membership functions , respectively. Let be the index score of index . Then, the grade vector of index is represented by

where () is the membership degree to which index belongs to grade .

The next step is to construct a method to realize the transformation from the grade vector of the bottom index to the grade vector of the overall goal .

3.4. Transformation Method

The transformation method can be obtained with the following three steps.

Step 1. To determine the importance of index in the grading of the overall goal, the weight of index with respect to the overall goal is introduced beginning with the following formulas:

where is the weight of index with respect to . From the above formulas, we have

Step 2. Because shows the degree of the effective information offered by in the evaluation of , we calculate

Step 3. By weight of index with respect to , we have

From the above formulas, we have

Thus, we obtain the grade vector of index in the second level, where indicates the degree to which belongs to evaluation grade .

The grade vector of the overall goal can be obtained in the same way.

Remark 25. From the property of , it is obtained that the larger the is, the more important the role plays in the evaluation of is. If , then the evaluation grade of can be determined only by index . If , then index does not play any role in determining the grade of .

Theorem 26. The weighted-average method is a special case of the above method.

Proof. For each row vector , if there is one component for some and the rest four components are 0, then , , and . Thus, by formulas

we have

That is to say, the weighted-average method is a special case of the above method.

3.5. Identification

Sometimes, the evaluation scales are comparable (e.g., Poor, Fair,…, Excellent scale; “Fair” is better than “Poor”), and a partial order “≻ ” can be defined according to the actual situation. If evaluation scale is better than , we denote ; otherwise we denote . Of course, whether is better than depends on the actual situation. To address the ordered division (e.g., Poor, Fair,…, Excellent), Cheng [26] proposed the confidence degree principle.

Confidence Degree Principle (Cheng [26]). Let “≻ ” be a partial order, let be an ordered division, and let () be the confidence level. If ,

and if ,

and then belongs to with at least the confidence level .

4. An Application in the Evaluation of Software Quality

In this section, an application of the proposed method in evaluation of software quality is given.

4.1. Evaluation System and Some Data

A problem of evaluating software quality was discussed in Li [27]. Next, we apply the proposed method to this evaluation problem, and the evaluation system is shown as in Table 2. Based on the evaluation criteria, the experts provided the scores of the bottom indices in Column 5 of Table 2. The evaluation grades are “Poor,” “Fair,” “Good,” “Very good,” and “Excellent.” They are represented by uncertain sets with membership functions , respectively, where membership functions

are given by experts according to their personal knowledge and actual situation of the evaluation. According to the scores given by experts (Column 5 in Table 2) and the above membership functions, the grade vectors (Column 4 in Table 2) can be obtained (see Section 4.3).

Table 2: Evaluation system of software quality.

The overall goal
Second-level indices and weights
Bottom indices and weights
Grade vectors
Scores

Software quality
Install section
Install difficulty

Size after installation

Memory occupation

Shortcuts creation

General structure
Complete module

Appropriate grade

Reasonable module partition

Business integration level

Data import and extraction function

Security mechanism
Permissions allocation

Operating log

Data backup and restoration

Data maintenance

Program and data processing
Reliability

Efficiency

Maintainability

Portability

Code maintenance
Definable code

Gradable code

Proper code maintenance

Operating performance
Friendly interface

Easy-to-use

Online help

Multitasking switching

4.2. Weights of Indices

There are ten experts in the evaluation group, and there are five degrees of importance (namely, “most important,” “more important,” “important,” “less important,” and “unimportant”), which are represented by uncertain variables with the same uncertainty distribution . Taking indices , , , , for examples, the evaluation results from the experts are shown as in Table 3. Weight of index is an uncertain variable . By , , and Theorem 8, we have . The vector is obtained by the 99-table, and the values of are shown in Table 4.

Table 3: Evaluation results given by experts.

Bottom indices
Most important
More important
Important
Less important
Unimportant

Install difficulty
1
2
3
2
2

Size after installation
2
1
3
2
2

Memory occupation
2
2
3
1
2

Shortcuts creation
2
3
2
2
1

Table 4: The 99-table of weights for indices .

Weights
0.01
0.02

0.99

0.54
0.55

0.84

0.54
0.55

0.84

0.54
0.55

0.84

0.54
0.55

0.84

By Example 11 and Theorem 12, the expected value of is . Normalizing the expected value, the weights of indices , , , are . The weights of the other indices can be obtained in the same way (the numbers in Column 3 of Table 2).

4.3. Grade Vectors of Bottom Indices

Taking the grade vector of index as an example, the score value of index is 2.50 and the membership functions of evaluation grades are , respectively. Therefore, the grade vector of index is

Similarly, the other grade vectors of the bottom indices can be obtained by the scores of the bottom indices (Column 5 in Table 2).

4.4. Grade Vector of Software Quality

In this subsection, the transformation algorithm established above is used to realize the transformation from the grade vectors of the bottom indices to the grade vector of software quality.

4.4.1. Grade Vectors of Indices in the Second Level

We take the calculation of grade vector of index (install section) as an example.

There are four indices, , , , , immediately below , and the grade-transition matrix of is

With the weights of indices , , , , we can obtain

Similarly, we can obtain the grade vectors

4.4.2. Grade Vector of Software Quality

With the grade vectors of indices , we can obtain the grade-transition matrix of as

Similar to calculation (1), the grade vector of the overall goal is obtained as

4.5. Identification

In this example, the confidence level of belonging to (Very good) is no less than 0.53 () and the confidence level of belonging to (Good) is no less than 0.96 (). Comparatively, it is more reasonable to define in grade . Therefore, the quality of this software is “Very good.”

4.6. Comparative Analysis

In Li [27], the fuzzy max-min method is used in the software quality evaluation, and we calculate the results via the fuzzy weighted-average method with the same data in Table 2. The results are shown in Table 5. Table 5 shows that the proposed method and fuzzy max-min method produced the same result (i.e., the evaluation result is Very good) and the evaluation result from fuzzy weighted method is Good. And also there is some difference in the membership degree of each grade. Furthermore, the literatures [28, 29] discussed the algorithms for maximizing the soft margin, which show that the larger the difference between two adjacent grades is, the stronger the classification ability of the method is. Therefore, we analyzed the differences between two adjacent grades in Table 6. From Table 6, we can see that there are three differences (between and , and , and and ) and the proposed method is larger than the other two methods. Thus, the proposed method has a stronger classification ability in the evaluation of software quality.

Table 5: Comparison of the results with the other two methods.

Grades
Poor
Fair
Good
Very good
Excellent

The proposed method
0
0.0356
0.4319
0.4903
0.0422

Fuzzy weighted-average method
0
0.1051
0.4507
0.4053
0.0389

Fuzzy max-min method
0
0.0822
0.4165
0.4697
0.0516

Table 6: The differences between two adjacent grades.

Differences
and
and
and
and

The proposed method
0.0356
0.3963
0.0584
0.4481

Fuzzy weighted-average method
0.1051
0.3456
0.0454
0.3664

Fuzzy max-min method
0.0822
0.3343
0.0532
0.4181

5. Conclusions

In this paper, weights for bottom indices with respect to the overall goal in an evaluation system are introduced and a new uncertainty evaluation method is proposed based on uncertain sets and uncertain variables. This method generalizes the weighted average method, and it is applied in the evaluation of software quality. Comparative analysis with other two methods shows that the proposed method has a stronger classification ability in the evaluation of software quality. More importantly, the proposed method in this paper can also be used in other evaluation systems with a hierarchical structure. Therefore, more applications of the proposed method can be underdeveloped further.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (no. 61073121), the Natural Science Foundation of Hebei Province of China (nos. F2012402037, A2012201033), and the Natural Science Foundation of Hebei Education Department (no. Q2012046). The authors also thank anonymous reviewers for their constructive comments and suggestions and acknowledge the English language editing by Elseviers WebShop.

References

T. Saaty, “Modeling unstructured decision problems—the theory of analytical hierarchies,” Mathematics and Computers in Simulation, vol. 20, no. 3, pp. 147–158, 1978.

H. H. Wu and Y. N. Tsai, “An integrated approach of AHP and DEMATEL methods in evaluating the criteria of auto spare parts industry,” International Journal of Systems Science, vol. 43, no. 11, pp. 2114–2124, 2012.

M. Kumar, S. P. Yadav, and S. Kumar, “Fuzzy system reliability evaluation using time-dependent intuitionistic fuzzy set,” International Journal of Systems Science, vol. 44, no. 1, pp. 50–66, 2013.

X. Zhao and L. Cui, “Reliability evaluation of generalised multi-state k-out-of-n systems based on {FMCI} approach,” International Journal of Systems Science, vol. 41, no. 12, pp. 1437–1443, 2010.

D. Chen, M. Fu, X. Jiang, and D. Song, “The investment decision-making index system and the grey comprehensive evaluation method under hybrid cloud,” Advances in Computer Science and Information Engineering, vol. 169, no. 2, pp. 171–176, 2012.

C. Lee, H. Lee, H. Seol, and Y. Park, “Evaluation of new service concepts using rough set theory and group analytic hierarchy process,” Expert Systems with Applications, vol. 39, no. 3, pp. 3404–3412, 2012.

X. Geng, X. Chu, and Z. Zhang, “A new integrated design concept evaluation approach based on vague sets,” Expert Systems with Applications, vol. 37, no. 9, pp. 6629–6638, 2010.

M. Mareš, “Information measures and uncertainty of particular symbols,” Kybernetika, vol. 47, no. 1, pp. 144–163, 2011.

I. Kramosil, “Decision-making under uncertainty processed by lattice-valued possibilistic measures,” Kybernetika, vol. 42, no. 6, pp. 629–646, 2006.

H. Y. Li, Y. A. Hu, and R. Q. Wang, “Adaptive finite-time synchronization of cross-strict feedback hyperchaotic systems with parameter uncertainties,” Kybernetika, vol. 49, no. 4, pp. 554–567, 2013.

Y. Lan, R. Zhao, and W. Tang, “A bilevel fuzzy principal-agent model for optimal nonlinear taxation problems,” Fuzzy Optimization and Decision Making, vol. 10, no. 3, pp. 211–232, 2011.

L. A. Zadeh, “Fuzzy sets,” Information and Computation, vol. 8, pp. 338–353, 1965.

B. D. Liu, Uncertainty Theory, Springer, Berlin, Germany, 2007.

B. D. Liu, Uncertainty Theory: A Branch of Mathematics for Modeling Human Uncertainty, Springer, Berlin, Germany, 2010.

D. Kahneman and A. Tversky, “Prospect theory: an a nalysis of decision under risk,” Econometrica, vol. 47, no. 2, pp. 263–292, 1979.

B. D. Liu, Theory and Practice of Uncertain Programming, Springer, Berlin, Germany, 2009.

B. D. Liu, “Uncertain risk analysis and uncertain reliability analysis,” Journal of Uncertain Systems, vol. 4, pp. 163–170, 2010.

B. D. Liu, “Uncertain logic for modeling human language,” Journal of Uncertain Systems, vol. 5, no. 1, pp. 3–20, 2011.

B. D. Liu, “Uncertain set theory and uncertain inference rule with application to uncertain control,” Journal of Uncertain Systems, vol. 4, pp. 83–98, 2010.

B. D. Liu, “Extreme value theorems of uncertain process with application to insurance risk model,” Soft Computing, vol. 17, no. 4, pp. 549–556, 2013.

B. D. Liu, Uncertainty Theory, 4th edition, 2013, http://orsc.edu.cn/liu/ut.pdf.

J. J. Liu, “Uncertain comprehensive evaluation method,” Journal of Information and Computational Science, vol. 8, no. 2, pp. 336–344, 2011.

B. D. Liu, “Some research problems in uncertainty theory,” Journal of Uncertain Systems, vol. 3, pp. 3–10, 2009.

B. D. Liu, “Membership functions and operational law of uncertain sets,” Fuzzy Optimization and Decision Making, vol. 11, no. 4, pp. 387–410, 2012.

B. D. Liu, “A new definition of independence of uncertain sets,” Fuzzy Optimization and Decision Making, vol. 12, no. 4, pp. 451–461, 2013.

Q. S. Cheng, “A model of property identification and its application,” Acta Scientiarum Naturalium Universitatis Pekinensis, vol. 33, pp. 12–20, 1997.

L. Li and X. Han, “Multilevel fuzzy integrated evaluation of software quality,” Journal of Harbin Institute of Technology, vol. 35, no. 7, pp. 812–819, 2003.

N. Cristianini and J. S. Taylor, An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods, Cambridge University Press, New York, NY, USA, 2000.

J. Shawe-Taylor and N. Cristianini, “On the generalization of soft margin algorithms,” IEEE Transactions on Information Theory, vol. 48, no. 10, pp. 2721–2735, 2002.