e-Health Systems quality management is an expensive and hard process that entails performing several tasks such as analysis, evaluation, and quality control. Furthermore, the development of an e-Health System involves great responsibility since people’s health and quality of life depend on the system and services offered. The focus of the following study is to identify the gap in Quality Characteristics for e-Health Systems, by detecting not only which are the most studied, but also which are the most used Quality Characteristics these Systems include. A strategic study is driven in this paper by a Systematic Literature Review so as to identify Quality Characteristics in e-Health. Such study makes information and communication technology organizations reflect and act strategically to manage quality in e-Health Systems efficiently and effectively. As a result, this paper proposes the bases of a Quality Model and focuses on a set of Quality Characteristics to enable e-Health Systems quality management. Thus, we can conclude that this paper contributes to implementing knowledge with regard to the mission and view of e-Health (Systems) quality management and helps understand how current researches evaluate quality in e-Health Systems.
As far as e-Health definition is concerned, Eysenbach [
As far as e-Health Systems quality evaluation work is concerned, Goletsis and Chletsos [
By the same token, Greenhalgh and Russell [
ISO 27799:2008 [
ICT service management which encompasses ITSM (IT service management) or ICT services as such [
ITIL [
The main goal of this study is to establish a set of Properties and Quality Characteristics to manage quality in different e-Health Systems under some specific criteria. The paper is organized into the following sections. After this introduction and general analysis of the situation, Section
The objective of this strategic study is to assist in how to design, develop, and implement e-Health Systems. It tends to ensure that ICT organizations, which are responsible for implementing these systems, attain operational effectiveness and supply distinctive services to the e-Health Systems stakeholder. Its ultimate goal is to make the ICT organization think and act in a strategic manner.
An e-Health System is figured out in the present work as a relatively recent term denoting healthcare practice supported by electronic processes and communication. However, it can also coin different meanings: some people argue that it is exchanged with Health informatics with a broader definition covering electronic/digital processes in Health, while others use it in the narrower sense of healthcare practice by means of the Internet.
This way, ICT organizations of e-Health Systems have to meet stakeholders’ needs. Nonetheless, ICT organizations must guarantee stakeholders that, despite these Properties, a set of Quality Characteristics such as Usability, Functionality, or Maintainability must also be covered. To sum up, it is worth stating that the real value of a system depends on these two factors: Properties and Quality Characteristics.
There are many reasons for conducting a Systematic Literature Review (SLR). The most common ones for this study focus on the identification of gaps in e-Health Systems quality management in order to suggest areas for further investigation and to provide a background, so that new research activities related to e-Health quality issues must be appropriately set. Specifically, we aim to determine the most studied Quality Characteristics in e-Health Systems so as to identify any gap in this area.
Due to the fact that defining Quality Characteristics involves a large initial effort, the description domain in our approach is identified through a SLR and the review method is based on the research protocol. This section defines the search strategy, sources, studies selection, and selection execution.
The guidelines for the systematic review stated in this work follow the protocol defined by Kitchenham [
The focus of this paper is detecting not only which are the most studied, but also which are the most used Quality Characteristics of e-Health Systems. A review protocol, which is specified in Table
The review protocol.
Background | An e-Health System is figured out in the present work as a relatively recent term denoting healthcare practice supported by electronic processes and communication. However, it can also coin different meanings: some people argue that it is exchanged with Health informatics with a broader definition covering electronic/digital processes in Health, while others use it in the narrower sense of healthcare practice by means of the Internet. ICT organizations must guarantee stakeholders that a set of Quality Characteristics such as Usability, Functionality, or Maintainability must also be covered. |
|
|
Research questions | What are the most studied and most used Quality Characteristics of e-Health Systems? |
|
|
Strategy | Sources: Google Scholar, Scopus, Mendeley, Science Direct, ISI Web of Knowledge, ACM Digital Library, CiteSeerX, or the IEEE digital library. |
|
|
Study selection criteria | All kinds of articles (conferences and journal articles, patents, websites, conference proceedings, doctoral dissertations, Open Access material, and some others) related to the quality evaluation of e-Health Systems, published from 2006. Three search types of logical criterion in the search field: |
|
|
Study selection procedures | The article contains any kind of evaluation or assessment about any e-Health Systems. |
|
|
Data extraction strategy | The data extracted from each paper will be as follows: |
The review protocol is a critical element in any systematic review. All authors review and criticize the protocol in order to find out the appropriate one. In addition, as Kitchenham et al. [ the search strings are appropriately derived from the research questions; the data to be extracted will properly address the research questions; the data analysis procedure is appropriate to answer the research questions.
Once all planning phase goals have been achieved, the review process leads to the review phase itself, which consists in finding and evaluating whether many primary studies associated with the research questions are adequate and relevant enough to be possible sources for further analysis. The primary studies are searched in the aforementioned databases by means of the keywords represented in Table
Firstly, a search with keywords of Table The primary study must have been published from 2006. In the context of this work, this exclusion criterion is considered realistic and acceptable. Therefore, the considered primary studies must be within these last years in order to infer practical conclusions. The paper must focus on the quality evaluation of e-Health Systems. The paper must have been published in any international journal or conference proceedings, book, or book chapter of interest.
This section aims to choose the sources to complete searches for primary studies. First of all, we carried out preliminary searches with the intention of both identifying existing systematic reviews and assessing the volume of potentially relevant studies, so that in this case we found the following relevant initiatives: Asoh and Rivers [
Thirdly, a new selection is discarded by means of a fast reading of each primary study. First of all, the title theme of the primary study must be linked to the topic of this work. Once this condition is satisfied and cataloguing this primary study as promising, the introduction and abstract must mention the goals of the research question posed in this section.
The selection criterion to evaluate study sources will be availability, not only for consulting articles on the Internet or the digital library of the University of Seville, which contains e-books and has access to other resources such as Google Scholar, Scopus, Mendeley, Science Direct, ISI Web of Knowledge, ACM Digital Library, CiteSeerX, or the IEEE digital library, but also for looking for other tools or mechanisms through keywords and literature from companies, books, journals, and conferences, written by experts in the field.
The process so as to take into consideration a paper within our research was as follows: firstly, we considered the Title and Abstract, Keywords, Content, and finally Conclusions of the paper. Most papers were included in that type of search, since the concept word in the Title let us know that the paper focused on that Quality Characteristic concept. Besides, when a paper deals with Quality, it refers to a general concept of quality involving a set of Quality Characteristics. Such particular Quality Characteristics were noticed after an in-depth reading of the paper.
In addition, we had some problems with some search engines, such as Mendeley search engine, which did not work with long-term conditions in this kind of search and was not suitable for Search Types 2 and 3. Looking for Abstract, Keyword, or Title, in Google Scholar search engine, was not possible; thus, only Scopus stood as the eligible search engine for this surf.
Quality management in e-Health Systems can be regarded as an improvement in healthcare quality, prevention from medical errors, reduction of paperwork, reduction of healthcare costs, expansion of access to affordable care, or implementation of administrative efficiencies. On the one hand, Properties describe the environment or context under study or the needs that ICT organizations must satisfy. On the other hand, Quality Characteristics are the quality aspects that ICT organizations have to guarantee users while utilizing all these Properties. So, these Quality Characteristics are influenced by Properties. For instance, an e-Health System provides a set of tools and services that ICT organizations have to cover to stakeholders’ Properties. Moreover, ICT organizations have to guarantee stakeholders that these Properties will be covered together with a set of Quality Characteristics of e-Health Systems, such as Interoperability, Security, Usability, or Accessibility. Hence, the Quality Model consists of a set of Properties, Quality Characteristics, and the relationships among them, and basically, it supports the basis for quality management. It may be defined as “conformance to requirements” and/or “fitness of use.” Furthermore, if they focus on their business, ICT organizations can manage e-Health Systems quality efficiently and effectively.
Then, we carried out the procedures to select the studies in order to obtain articles to verify whether the studies fit both the inclusion and the exclusion criteria. Most papers were obtained from Google Scholar, Mendeley, Scopus, and Science Direct. Some consideration for each Search Type is explained below.
In order to group and characterize Quality Characteristics, concept mapping techniques have been carried out. A concept map is a type of graphic organizer used to help people organize and represent knowledge of a subject. Concept maps begin with a main idea (or concept) and then branch out to show how that main idea can be broken down into specific topics. So, a concept map was used to build the characterization of Quality Characteristics during the SLR as shown Figure
Method to group Quality Characteristics.
This figure shows the process to build the characterization. First of all, we plan the review; then, the review process is carried out. Then, the results are collected and analyzed by the concept mapping techniques and finally, the results are reflected in the characterization of Quality Characteristics. This is a continual cycle of improvements. In consequence, all concepts that were found in the SLR were considered in the concept map to characterize Quality Characteristics. We have also considered ISO 25000 [
Concept maps are typically hierarchical, with the subordinate concepts stemming from the main concept or idea. This type of graphic organizer, however, always allows change and new concepts to be added.
In this section, we are going to do a specification of dissemination mechanisms and formatting of the main report. It must be mentioned that the Quality Characteristics for e-Health Systems can be grouped in two categories: External/Internal Quality and Quality in Use, the latter depending on the former.
On the contrary, as far as features of Properties definition are concerned, e-Health Systems can encompass a range of services or systems that are at the edge of medicine/healthcare and information technology, including the following.
Table
Number of results of Search Types 1, 2, and 3, respectively, for Quality Characteristics.
Usability | Functionality | Accessibility | Reliability | |
---|---|---|---|---|
Search Type 1 | 11 | 134 | 9 | 41 |
Search Type 2 | 59 | 723 | 93 | 173 |
Search Type 3 | 2.114 | 19.788 | 2.213 | 6.515 |
|
||||
Efficiency | Continuity | Maintainability | Portability | |
|
||||
Search Type 1 | 17 | 1 | 0 | 0 |
Search Type 2 | 0 | 28 | 5 | 12 |
Search Type 3 | 3.623 | 1.177 | 1.308 | 1.115 |
Normalized value of the results obtained in Search Types 1, 2, and 3, respectively, for Quality Characteristics.
Importance | Usability | Functionality | Accessibility | Reliability |
---|---|---|---|---|
60% | 0.05164319 | 0.62910798 | 0.04225352 | 0.19248826 |
30% | 0.05397987 | 0.66148216 | 0.08508692 | 0.15827996 |
10% | 0.05585594 | 0.52275449 | 0.05846249 | 0.17211166 |
|
|
|
|
|
|
||||
Importance | Efficiency | Continuity | Maintainability | Portability |
|
||||
60% | 0.07981221 | 0.00469484 | 0 | 0 |
30% | 0.00000000 | 0.02561757 | 0.00457457 | 0.01097896 |
10% | 0.09570271 | 0.03110250 | 0.03455442 | 0.02945579 |
|
|
|
|
As far as results are concerned, the black, light grey, and grey lines in Figure
Internal/External Quality.
The most studied Quality Characteristic by far is Functionality, as shown in Figure
Most papers dealing with Trust describe users’ ability to get confidence in e-Health Systems and they combine this concept with Reliability, which is defined as the ability of a system or component to perform its required functions under stated conditions for a specified period of time. In this sense, they are related because a Reliable system inspires users with Trust. Reliability is implicit in the standards implantation like Compliance, since such standards favor this Quality Characteristic. It is really troublesome to find papers based on availability (Quality Subcharacteristic of Reliability and Quality Characteristic in ITIL) of e-Health Systems. We guess that it undergoes the implicit idea that these systems have to be available 24/7. Nevertheless, techniques, processes, and systems responsible for this situation have to be defined in order to assure this service. Table
Number of results of Search Types 1, 2, and 3, respectively, for Functionality.
Security | Interoperability | Accuracy | Suitability | Compliance | |
---|---|---|---|---|---|
Search Type 1 | 93 | 38 | 0 | 1 | 2 |
Search Type 2 | 340 | 158 | 99 | 44 | 82 |
Search Type 3 | 6.328 | 3.330 | 3.994 | 2.302 | 3.834 |
Normalized value of the results of Search Types 1, 2, and 3, respectively, for Functionality.
Importance | Security | Interoperability | Accuracy | Suitability | Compliance |
---|---|---|---|---|---|
60% | 0.69402985 | 0.28358209 | 0.00000000 | 0.00746269 | 0.01492537 |
30% | 0.47026279 | 0.21853389 | 0.13692946 | 0.06085754 | 0.11341632 |
10% | 0.31980662 | 0.16828381 | 0.20183950 | 0.11631629 | 0.19375379 |
|
|
|
|
|
The black, light grey, and grey lines in Figure
Quality subcharacteristics of functionality.
Accuracy, Compliance, and Suitability reach higher values than Interoperability, but lower than Security, in Search Type 3. This means that these words cope with a more general meaning of Quality in papers. Then, Security is the most studied Quality Characteristic of Functionality. As far as Security is concerned, most studies focus on Privacy and fewer papers exist dealing with Security focused on Integrity. Another aspect to take into account is the fact that Security is usually linked to Reliability in works dealing with quality. In regard to Interoperability, it must be mentioned that lots of works point out semantic interoperability.
Figure
Table
Number of results of Search Types 1, 2, and 3, respectively, for Quality in Use.
Safety | Effectiveness | Satisfaction | Productivity | |
---|---|---|---|---|
Search Type 1 | 26 | 3 | 6 | 0 |
Search Type 2 | 127 | 96 | 91 | 4 |
Search Type 3 | 4.909 | 6.172 | 2.997 | 1.554 |
Normalized value of the results of Search Types 1, 2, and 3, respectively, for Functionality.
Importance | Safety | Effectiveness | Satisfaction | Productivity |
---|---|---|---|---|
60% | 0.74285714 | 0.08571429 | 0.17142857 | 0.00000000 |
30% | 0.39937107 | 0.30188679 | 0.28616352 | 0.01257862 |
10% | 0.31402738 | 0.39482663 | 0.19170896 | 0.09943703 |
|
|
|
|
Actually, a shortage of Safety has been identified in e-Health Systems. It has to be noted that Safety should be a point of reference among all Quality Characteristics in e-Health Systems, but the fact still remains that academic people do not envisage this Quality Characteristic because there is lack of literature on this topic, in utter contrast to the literature available to other Quality Characteristics.
The black, light grey, and grey lines in Figure
Quality in use.
Regarding Quality in Use, the Quality Characteristic of Effectiveness is the most used in e-Health Systems. Safety is the most studied Quality Characteristic, as represented in Figure
QuEF [
The phases above have been defined together with processes and artifacts to fulfill the complete Quality Model life cycle. The purpose of QuEF is to converge towards a continuous automatic quality improvement by means of generating Checklists and documentation as well as automatic evaluations and plans, so that quality can be controlled and implemented and, in turn, effort and time are automatically reduced.
This paper lets people be aware of the mission and view of e-Health quality management and helps them understand how current studies evaluate e-Health Systems quality. This contribution focuses on detecting the most studied and used Quality Characteristics on e-Health Systems by means of a Systematic Literature Review. Therefore, a complete portfolio of Quality Characteristics to evaluate these Systems is proposed.
The objective of a future research is to reach a centralized and shared consensus on the Quality Model with clear goals agreed by all ICT organizations so as to implement e-Health Systems. This agreed Quality Model benefits the convergence towards a standardization and quality continuous improvement of e-Health Systems efficiently and effectively. In consequence, we propose QuEF, a framework to manage quality based on the Quality Model life cycle and define its extension for the e-Health Systems quality management. These Quality Characteristics are just part of the Quality Model.
In our view, the key is to give users what they really need (Properties) and, under Quality Characteristics, what they expect. Everything is embodied in the Quality Model, and ICT organizations know what they have to take into consideration. We guess that the use of QuEF would enhance e-Health Systems quality. Therefore, the QuEF can improve e-Health Systems efficiency in turn, making a more widespread use of those systems, since this evaluation helps anyone understand both the strengths and weaknesses of e-Health Systems.
The future and new goal will be twofold. On the one hand, we have to respond to questions such as the following: Have these Quality Characteristics been correctly studied? Are these Quality Characteristics aligned with those that users really demand? On the other hand, we must carry out a Systematic Literature Review for Properties to complete the Quality Model and characterize e-Health Systems, with the aim of knowing which are the most studied Properties in e-Health Systems and suggesting a set of Features of Properties for such purpose. Besides, another important aspect will be to define the importance of Properties and Quality Characteristics and to what extent the former has influence on the latter. Consequently, we will work on Group Decision Making process in order to reach a consensus on the Quality Model.
The authors declare that there is no conflict of interests regarding the publication of this paper.
This research has been supported by the MeGUS project (TIN2013-46928-C3-3-R) of the Spanish Ministry of Science and Innovation, Spain.