The Current Status of Usability Studies of Information Technologies in China: A Systematic Study

Objectives. To systematically review and analyze the current status and characteristics of usability studies in China in the field of information technology in general and in the field of healthcare in particular. Methods. We performed a quantitative literature analysis in three major Chinese academic databases and one English language database using Chinese search terms equivalent to the concept of usability. Results. Six hundred forty-seven publications were selected for analysis. We found that in China the literature on usability in the field of information technology began in 1994 and increased thereafter. The usability definitions from ISO 9241-11:1998 and Nielsen (1993) have been widely recognized and cited. Authors who have published several publications are rare. Fourteen journals have a publishing rate over 1%. Only nine publications about HIT were identified. Discussions. China's usability research started relatively late. There is a lack of organized research teams and dedicated usability journals. High-impact theoretical studies are scarce. On the application side, no original and systematic research frameworks have been developed. The understanding and definition of usability is not well synchronized with international norms. Besides, usability research in HIT is rare. Conclusions. More human and material resources need to be invested in China's usability research, particularly in HIT.


Introduction
Usability is essential for the effective, efficient, and safe design, use, and learning of information technology. Research and application of usability have received significant attention by scientists, designers, and industry professionals in Western countries where there are active study populations, comprehensive theories, methods, practices, practical results, and mature industrial and professional organizations. In the field of health information technology (HIT), usability research has been identified as an important cognitive challenge for the adoption and meaningful use of HIT by the Office of National Coordinator for Health Information Technology (ONC), which is part of the U.S. Department of Health and Human Services (DHHS), and has become an active area for research, design, and practice in HIT [1][2][3][4][5][6][7][8]. Methods of usability evaluation have been demonstrated to improve the design and utilization of clinical information systems [9][10][11]. Usability, under the name of "safety enhanced design, " has become a requirement of Stage 2 meaningful use requirement for electronic health records (EHR) in the United States [12]. In China, with 30 years of rapid economic development, China's investment in social development and scientific research has been increasing dramatically. Given this context, one question we would like to answer is as follows. What is the current status of usability research in China? In this paper, through systematically reviewing the literature published in China and applying a combined qualitative and quantitative approach, we present the current status and existing problems of usability research and practice in China's information technology field.  This problem is compounded when translating these terms into Chinese because the words often have multiple meanings. After carefully reviewing the literature in August 2012, we decided to conduct a literature review using the Chinese equivalents of the following search terms: "usefulness, " "usability, " "user experience, " "user satisfaction, " and "usercentered design. " We used the advance search functions of following three mainstream databases in China: China National Knowledge Infrastructure (cnki.net), China Science and Technology Periodical Database (cqvip.com), and Wanfang Electronic Journal Database (wanfangdata.com.cn). We only searched the "title" and "keyword" fields in the literature published from the time period between 1980 and 2012.

English Publication Search
Strategy. Although our research was to review the literature originating from China using Chinese academic databases, we used the English language database, EBSCO, because it had the advanced search function that allowed us to select papers that originated from China. We searched using the advanced functionality provided by EBSCO, in the fields of "title" and "abstract" with any of the search terms "usability, " "user experience, " "usercentered design, " "user satisfaction, " "customer satisfaction, " "user interface, " "UCD, " and "China" and limited the results to those originating in China. (2) for duplicate entries, excluding those without complete information; and (3) excluding those without full text.

Data Extraction and Statistical
Processing. The information summary sheet was designed to extract data from the selected literature. Information extracted included author, article title, year, journals, subcategory types for theoretical research, subcategory type for empirical studies, definition of usability, evaluation objects in empirical study, subcategory type of evaluation research in network application, and evaluation method used in empirical study. See Figure 1 for the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow chart.

Literature Search Results.
Through searching of the literature for thirty-two years in the three databases and EBSCO, we obtained a total of 9814 publications. After applying our inclusion and exclusion criteria, 617 Chinese and 30 English publications were included in our analysis and accounted for 6.6% of the total retrieved literature. Reasons for the high exclusion rate are as follows.
(1) Repeated retrievals: it is common that the same article may be retrieved in all three databases.
(2) Study fields involved in the retrievals are unrelated. For example, topics on architecture, transportation, and so on rather than "information technology" were excluded.
(3) Limitation of search function by certain database: for instance, the Wanfang database returned 6411 items, far more than 1347 and 1698 items from VIP and CNKI databases. The reason is that Wanfang is  unique in the way that it does not support whole word search; rather, it will return all results containing each composite word of a complete word. In Chinese usability is composed of three "words. " (4) Excluded nontechnical items: for example, some items retrieved were advertisements released by companies for new products or news published on nontechnical magazines. The inclusion and exclusion process and the results are listed in Table 1.

Type Distribution of the Literature.
In the 647 items incorporated into our study, 564 are periodical publications and account for 87.2% of the total items, 52 are conference proceedings and account for 8.0%, 30 are theses/dissertations and account for 4.6%, and one was a book chapter accounting for 0.2%.

Author Distribution of the Literature.
Among the 1190 authors (including the second and the third coauthors) that contributed to the selected research publications, the following 10 authors listed in Table 2 published the most publications.

Journal Distribution of Published
Literature. Journals or conference proceedings in the 647 publications (top 14 with the highest quantity) are listed in Table 3.

Distribution of Keywords Involved in the Literature.
There are a total of 2229 keywords in the 647 publications. The keywords with the top 8 occurrence frequencies are listed in Table 4.

Types of Domestic Usability Research in Information
Technology Field. Usability research in China may be roughly divided into two categories. The first category is the study about usability theories; there are a total of 395 publications Note: the publication quantity of other journals or conference proceedings is less than 1% and is not listed here. * aka "Peking University core journal" refers to the classification by Peking University Library on Chinese academic journals, published every 3-4 years and currently widely recognized by the Chinese academia. Publications in core journals are viewed with relative high academic levels and this is an important part of the academic evaluation system in China. accounting for 61% of the total retrieved publications. The other category includes empirical studies of usability, of which qualitative research methods were primarily utilized. The theoretical study mainly includes the following three aspects in contents: (a) usability history of development, influencing factors and problems encountered in usability (in different branch fields); (b) usability evaluation methods (in different branch fields); and (c) usability design principles and design concepts (in different branch fields).
Empirical studies related to usability accounted for 39% (i.e., 252) of the retrieved publications. Those publications related to the integration of usability during software or technology development account for 17% (i.e., 42) of the total retrieved publications. Most studies (83% or 210 total) focused on usability evaluations in specific target areas through the selected evaluation methods. The evaluation methods mostly applied are the combination of qualitative and quantitative studies.
The categorization of theoretical studies is shown in Figure 3. Classification of empirical studies is shown in Figure 4.

Understanding of Chinese Researchers on the Concept of
Usability. Based on the retrieved publications, the usability concepts in Table 5 are more commonly recognized by Chineseresearchers.  Specific definitions are as follows.
(i) In the China national standard GB/T16260-2006 "Software engineering products quality, " usability is defined as "the ability of a software product to be understood, studied, used and as well as [sic] the ability to attract users in a particular use environment. " (ii) In the usability definition given by Nielsen in 1993 [13], usability includes 5 aspects, which are learnability, efficiency, memorability, errors, and satisfaction, respectively. (iii) In the ISO 9241-11:1998 [14] "Ergonomic requirements for office work with visual display terminals (VDTs), " usability is defined as "extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. " (iv) In the ISO/IEC 9126-1:2001 [15] "Software products evaluation-quality properties and operation directions, " usability is defined as "the capability of the software product to be understood, learned and liked by the user, when used under specified conditions. " (v) In recent years, with the advancement of usability research, new terminology emerges constantly. In the 647 publications, the "user experience, user satisfaction, and user-centered design" are also mentioned; among them, "user experience" is mentioned 118 times, "user satisfaction" 25 times, and "user-centered design" 12 times.

Specific Types of Network Study Targets in Empirical Studies for Usability Evaluation.
Through analyzing the retrieved publications, we found that the Chinese empirical studies on usability focused mainly on website application (138 publications; see Figure 5 for the breakdown of website types).
Regarding the trend of evaluation objects, we can see from Table 6 that usability evaluation studies in web application still have a dominant position in recent years. However, usability evaluation studies on mobile Internet (including mobile phones) and applications did not increase as would be expected with the current rapid growth of mobile technology.

Types of Study Methods in Empirical Studies for Usability
Evaluation. Through analyzing the retrieved publications, we found that the usability evaluation methods used mainly include questionnaires, usability tests, heuristic evaluation, usability guidelines (such as MUG: Microsoft Usability Guideline), statistical analyses through system logs, cognitive walkthrough, behavior analyses, observation and interviews, eye movement analyses, distance of information-state transition (DIT), and other methods. The specific applications of these methods in the empirical studies are listed in Table 7.  In usability evaluation research, some studies may use two or more evaluation methods to obtain a greater understanding of the system's usability. Table 8 provides a breakdown of the number of research methods used in each article.

Studies in the Field of Healthcare Information Technologies.
Of special note is that among the 647 publications only nine are about usability as it relates to HIT (specific information about the literatures is listed in Table 10).  late and the history is not long, but it is attracting more attention.

Discussion
(2) From the data on the authors and the publishing journals, we conclude that the targets of usability research in China's information technology field are relatively scattered and no coordinated usability efforts, such as research institutes or centers, have been established.
The most prolific author with the largest number of publications has only published 12 publications, and the top 10 authors with the most publications account for 11% (including the coauthors) of the total publications printed. The fields of the main publications 32% 36%

32%
Library websites E-commerce website Others Of course, these journals are respectively included in the "China Journal Citation Report" by the China Science and Technology Information Institute and the "Chinese core journals" of the Peking University, which are believed to have a good reputation. In the United States, there are many pioneers and authorities in the field of usability, such as Jakob Nielsen, who have produced many publications, but China has yet to have such established and prolific experts on these topics. What is more, besides specific peer reviewed journals for usability, such as "Journal of Usability Studies", there are more than 50 types of journals/magazines which are directly associated with usability.
(3) The data on study types show that theoretical studies on usability account for 61% of the total publications, including 27% about usability evaluation methods. Through analyzing the retrieved literature, we found that theoretical studies typically combine usability with specific branch fields (such as study of usability in web application) in terms of usability influencing factors, usability problems, usability design principles, and usability evaluation methods. However, the theoretical studies typically do not have significant original proposals or innovative methods and frameworks. At present, major advancements in theoretical and methodological studies are lacking in China. (4) From the data on the understanding on usability concepts, we found that the Nielsen [13] and ISO 9241-11:1998 [14] definitions account for 66.1% of the publications, showing that these two definitions are widely recognized by Chinese researchers. However, the above two definitions were proposed in earlier years. Based on the newest usability definition provided in ISO 25010:2011, "System and software quality model", usability is not only a property about product quality but also a property about quality in use of the project (comprising effectiveness, efficiency, and satisfaction) [16]. Yet, this newest definition is not mentioned in any current Chinese usability literature, so, to a certain extent, this may demonstrate that Chinese researchers are falling behind on international usability research. A new definition of usability was just proposed last year by Zhang and Walji [5], with the intent to unify all the variations of usability definitions, concepts, and applications under a single theoretical framework. (5) From the data on study targets, we can see that most Chinese usability studies are combined with evaluation studies and the evaluation objects are mainly focused on Internet applications. In evaluation studies, 66% are usability studies in Internet applications; this is partially because usability studies on Internet started relatively early in western countries. Many usability experts in web applications have proposed various website usability evaluation methods and practice guidelines. For example, in the US, Story argues that a website developer should follow 10 usability principles when the site is designed [17]. The Northwest Alliance for Computational Science and Engineering (NACES) formulated common website usability guidelines for website design, webpage design, and navigation help [18]. Borges et al. from University of Puerto Rico also proposed 16 usability principles for web design and proved the effectiveness of these principles by experiments [19]. Nielsen, a pioneer in usability research, conducted many important studies on usability of websites, addressing theories, methods, practice, and other aspects of usability [13,20,21]. Meanwhile, with the development of digital multimedia technology and wireless network technology, evaluation objects in usability evaluation studies are also changing gradually. (6) From the data on usability methods we can see that evaluation methods are divided into two categories: usability testing and questionnaires. They account for 58.2% of all evaluation methods. In most cases, a single method is applied in usability evaluation and this 8 BioMed Research International accounts for 63.8% of the evaluation methods. The data on combination of different evaluation methods are listed in Table 9. Obviously, usability testing and questionnaires, as two prominent methods, appear to play an important role in usability research. Although each evaluation method has its own use conditions, combining multiple methods may evaluate usability from a more comprehensive perspective. Combination of multiple methods will likely become a development trend for use of evaluation methods in the future.

Study Limitations
This paper studies the current status and characteristics of usability research in China's information technology field by using the systematic review and a quantitative literature analysis. Limitations of the study are listed as follows.
(1) Usability research is an interdisciplinary field and researchers in different disciplines often use different terminologies. To minimize any effects of overrepresentation, we used many different keywords such as "usability, " "user satisfaction, " "user experience, " and "user-centered design" to query the related literature. (2) Three major databases are used in our study and we searched via the field of keywords. However, keywords in Chinese publications do not have corresponding vocabulary similar to Mesh, and all keywords in Chinese publications are manually added by the author. We had to assume that if the authors had considered the subject of the article to be mainly about usability, they would have used one of the above related usability terms in the keywords, especially given the frequent references to Nielsen [13] and ISO 9241-11:1998 [14]. (3) This paper is limited to the information technology field, but typical publications may not explicitly use the keywords such as information technology in the titles or keywords; thus, we cannot enter "information technology" in the search query. We could only manually screen publications about information technology after all usability related publications are retrieved; this process needs more manual efforts. (4) Finally, this study focuses on the academic literature only; thus, the results obtained here are not inclusive. Furthermore, as usability is also an application intensive discipline, it is possible that usability related efforts are more active in industrial society than in academic domain reflected from this research.