Quality of Electronic Health Records – Coverage of Potential Information Weaknesses by Major EHR Quality Seals

of Potential Information Weaknesses by Major EHR Quality Seals Samrend Saboor1*+, Alexander Hoerbst2+, Elske Ammenwerth1 1Institute for Health Information Systems, UMIT University for Health Sciences, Medical Informatics, and Technology, Eduard-Wallnoefer-Zentrum 1, Hall in Tirol, Austria 2Research Division eHealth and Telemedicine, UMIT University for Health Sciences, Medical Informatics, and Technology, Eduard-Wallnoefer-Zentrum 1, Hall in Tirol, Austria


INTRODUCTION
Medicine is faced more than ever with a constant increase in available medical knowledge not least through an increasing mechanization and heavy use of IT in core medical fields.In order to guarantee an optimal treatment of patients, healthcare professionals, among other things, need to be provided with all of the necessary information along the process of treatment.This is especially difficult as the treatment of a patient often involves different specialists at different departments or even at different organizations.Therefore, providing optimal information implies putting together all of the pieces of information originating from various sources and systems.
The concept of Electronic Health Record (EHR) is dedicated to this task.It sets out to close the gap among institution-specific data and a comprehensive, probably lifelong, collection of a patient's health and healthcare data [1] that support information exchange among healthcare providers.As the EHR takes a crucial role in the provision and collection of information during current and future treatment, it is of utmost importance that EHRs comply with rigid quality standards.
Quality in general and also with regard to EHRs is a multidimensional construct.In order to guarantee high quality systems, different aspects have to be considered; this includes, among others, the functions of a system, the software itself, or the information.
In recent years, several quality/standardization initiatives with regard to EHRs have evolved, focusing on different aspects of quality, such as Integrating the Healthcare Enterprise (IHE) that focuses on technical interoperability or HL7, which is directed towards messaging.There are also two initiatives, the Certification Commission for Healthcare Information Technology (CCHIT) and EuroRec that certify EHRs as a whole based on functional requirements.
As already stated, the quality of EHRs is a multidimensional construct.Therefore, a holistic consideration of EHRs and their attributes is necessary in order to guarantee high quality systems.CCHIT and EuroRec, which are the most prominent certification approaches for EHRs in the US and Europe, are focused on system functionality and its certification.Hence, a fundamental question arises as to how far a functional view on system quality is adequate to sufficiently prove the quality of an EHR and, therefore, cover problems/deficiencies related to the exchange and handling of information as this is a central prerequisite for EHRs.

OBJECTIVES
The current work aims at analyzing the approaches from CCHIT and EuroRec with regard to their implicit coverage of the information (exchange) dimension within their functional certification criteria based on a previously developed categorization of information (exchange) [2].Subsequently, the approaches of CCHIT and EuroRec are briefly introduced.

CCHIT AND EUROREC -QUALITY CERTIFICATION APPROACHES
This section starts by introducing the CCHIT, which is primarily relevant to the United States.Also introduced is its European equivalent EuroRec, which is focused on the European market.

366
Quality of Electronic Health Records -Coverage of Potential Information Weaknesses by Major EHR Quality Seals

Certification Commission for Healthcare Information Technology (CCHIT)
The CCHIT [3] was founded in 2004 as a Non-profit organization by a group of organizations, including the American Health Information Management Association (AHIMA), the Healthcare Information and Management Systems Society (HIMSS), and the National Alliance for Health Information Technology (Alliance).Other partners have since joined the CCHIT, such as the American Academy of Pediatrics (AAP) and the American College of Physicians (ACP).In 2005, the CCHIT was commissioned by the U.S. Department of Health and Human Services (HHS) to establish quality criteria and a certification process for EHRs.The process was not only targeted towards the establishment of a certification framework but also to influence the harmonization process of different standards.Therefore, CCHIT closely cooperates with different standardization organizations.The aim of CCHIT is to establish efficient, trustworthy, and sustainable mechanisms for the certification of applications within healthcare to increase the robustness and interoperability of these applications [4].
The importance of the CCHIT in the US increased recently after the American Recovery and Reinvestment Act (ARRA) [5] was passed in 2009.This comprehensive economic stimulus package aims at stabilizing the US economy during and after the economic downturn.Several domains are covered in this act, including the healthcare sector.Within the healthcare sector, an estimated $32 billion is allocated for the adoption and meaningful use of EHRs within the next 10 years.CCHIT recently announced that they would apply to be accredited as an official certification body for the ARRA certification of EHR systems.CCHIT has already started a preliminary program [6] that aims at the certification of an EHR system with regard to the ARRA requirements.
In addition, CCHIT offers several other certifications for EHRs.Currently, they offer certifications for inpatient, ambulatory, and emergency department EHRs, for ePrescribing, and Health Information Exchange.These certifications are focused on the functionality, security, and interoperability of systems.The requirements contained are hierarchically structured and primarily taken from the existing standards.Approximately 70% of all requirements refer to functionalities [7].Table 1 shows some examples for different functional criteria.
By the end of 2010, there were approximately 210 systems certified by the CCHIT, of which 186 are ambulatory EHR systems.

Provider Information
The system shall provide the ability to capture and maintain, as discrete data elements, the principal provider responsible for the care of an individual patient.

Allergy Information
The system shall provide the ability to display an allergy list, including the date of entry.

Order Sets
The system shall provide the ability to incorporate multiple choices of medications or other types of orders within an order set for clinician's selection.

European Institute for Health Records (EuroRec)
The EuroRec Institute [8], founded in 2003, is a non-profit umbrella organization based on a network of nationally organized, independent ProRec centers.Apart from their organizational independence, all ProRec centers share the same goals and rely on the same fundamental principles.At the beginning of 2011, fifteen national ProRec centers have already been established and an additional seven are currently applying to be accredited by EuroRec.The aim of EuroRec and the affiliated ProRec centers is to foster and support the development and implementation of high-quality EHRs in Europe and to offer related services to the industry, health professionals, and decision makers.[8] EuroRec has so far developed a repository of candidate certification criteria.The criteria contained can be divided into 'Fine Grained Statements' and 'Good Practice Requirements'.'Fine Grained Statements' are basic, prime statements that can directly be referred to the sources.'Good Practice Requirements' are in general composed from one or more 'Fine Grained Statements' and represent self-contained entities with regard to a specific topic/content.See Table 2 for an example of the criteria.
All of the statements contained in the repository are categorized by different indices, including multiple indexing of single statements to increase the likelihood of finding relevant statements in a certain context.The repository currently contains 1,612 Fine Grained Statements and 178 Good Practice Statements [9].
Out of the repository, EuroRec has composed a set of basic, mainly functional criteria and put them together in quality seals.EuroRec offers two seals so far, the EuroRec 2008-2009 and the EuroRec 2010-2011 seals.Each seal comprises a certain amount of basic criteria that have to be fulfilled.The 2008-2009 seal comprises 20 criteria, whereas the 2010-2011 seal comprises 54 criteria [9].

CATEGORIZATION OF COMMUNICATION PROBLEMS 4.1. The Categorization -What It Is and Why It Is Necessary
The categorization of communication problems is a concise and structured collection of problems that commonly occur during the electronic transmission of information objects within integrated hospital information systems (HIS) -here, the addition of 368 Quality of Electronic Health Records -Coverage of Potential Information Weaknesses by Major EHR Quality Seals The system enables the author of a version of a health item to assign a level of access.
The system enables to assign a default level of access for each user.
The system enables to assign a different level of access for reading or writing.
The system assigns a default level of access to each version of a health item, depending on the nature of the health item and/or the access level of the author.The access level can be assigned by the author themselves.The access level assigned to a user and/or health item can be different for reading or writing operation.
"integrated" in turn addresses the fact that current HIS comprise multiple specialized electronic sub-systems and application systems (e.g., [10]) that utilize different communication standards (e.g., Digital Imaging and Communications in Medicine, DICOM [11] and Health Level 7, HL7 [12])).The categorization of communication errors was developed by our team [2].Its purpose is to contribute to a better comprehension of the nature of problems within electronic communication.Therefore, it specifies for each problem the respective reasons and recommendations for its prevention.Such a comprehension is essential since electronic communication has become increasingly important in recent decadesin addition to the general importance of correct and adequate communication for processes in healthcare (e.g., [13]).Among others, Haux et al. provide the variety and amount of data that are produced in the context of patient treatment [14].Another main reason is seen in the shift from the centralized to decentralized organization of healthcare institutions.

Overview over the Development of the Categorization
In order to be of practical use, the categorization was developed on the basis of the experiences of integration experts from the healthcare domain.Specifically, the following fields of this domain were of interest: standardization bodies, hospital information system management as well as commercial project management, product management, system administration, and software development.The experiences were acquired through an approach that combined the subsuming qualitative content analysis [15] in the relevant publications from PubMed and problem-centered expert interviews (e.g., [16]).The latter was meant to ensure the completeness and validity of the categorization.
In total, 426 out of 4,188 relevant literature references were identified and analyzed until a saturation of new communication errors was reached.The subsequent interviews involved 17 out of 42 experts who were identified as qualified before.A more detailed explanation of the development process (including acquisition, aggregation, structuring, and evaluation) can be found in [2].

The Content and Structure of the Categorization
The final categorization contains in its current version 81 common problems that are related to electronic communication in integrated HIS.In total, 229 reason or recommendation entries were identified and related to the respective problem.The categorization is organized in a hierarchy of five levels.
The following is a small example of one categorization entry.For reasons of simplicity, only the sections that pertain to the problem, reasons, and recommendation are considered here.For instance, the problem entry "Incompatible value representation of data attributes" is associated with the reasons "Conversion error in communication standard -lack in DICOM-standard regarding combination of attribute length and value representation/value multiplicity" or "Incompatible character sets".One recommendation for the avoidance of this problem is "Avoid the usage of attributes with implicit value representation".The complete first four levels of the categorization can be found in section 6.3.

Related work
To our knowledge there is no similar work that concisely summarizes common communication problems while also providing the underlying reasons.Even the IHE Technical Framework does not do so although it is considered best practice of hospital application system integration.Other contributions mainly focus on single aspects and are just aimed at sharing the respective experiences; thus it is up to the reader to identify the essential details and to match the described setting with their own situation.For instance, Lian et al. describe the experiences that they gained during their IHE project [17].Blado et al. deal with the possible causes of discrepancies between those databases that lead to inconsistencies [18].In contrast, Kuzmak et al. write about wrongly selected work list entries and the consequences [19].König et al. describe the shortcomings in the standard definitions and as reasons for possible integration difficulties [20].In the categorization, such references were analyzed during the acquisition step.The categorization thus can be regarded as an intersection of relevant experiences (see also section 4.2).

METHODS
The quality frameworks, CCHIT and EuroRec, were compared to the categorization of communication errors in order to determine what types of communication problems are anticipated by CCHIT and EuroRec.This comparison was conducted in the following steps:

Preparations
First, the relevant parts of the CCHIT-and EuroRec-framework were identified.The criteria for their identification can generally be summarized as follows:

•
Statements that concern the general functionality of EHRs (e.g., user login or data encryption or printing).

•
Statements concerning specific functionality regarding information exchange (e.g., data export or data conversion).

•
Statements that target specific functions within a certain domain were not considered (e.g., coding of medication or clinical alerts).
Wherever necessary, the contents of single entries were additionally abstracted in order to elaborate their main meaning; this allows for an easier comparison with the categorization of communication errors.For instance, the EuroRec entry "The system shall include the information that is necessary to identify each patient, including the first name, surname, gender, and date of birth" was abstracted to "Unique identification (of entity) requires the existence of unambiguous identifiers".The abstraction of the entries was based on techniques of qualitative content analysis as described in, e.g., [15,21].

Individual Comparison and Consensus -Building on Matchings
The actual comparison was conducted independently by two reviewers in order to enhance objectivity.Each of them iterated entry-wise through the CCHIT and EuroRec quality frameworks (specifically, those parts that were declared as relevant to the 370 Quality of Electronic Health Records -Coverage of Potential Information Weaknesses by Major EHR Quality Seals preparation step) and tried to find adequate matches in the reason and recommendation sections of the communication error classification.To increase consistency in the judgment of the reviewers ex ante, basic rules for the comparison were developed according to [22].All of the results of this matching were noted separately in a matching-list.Each entry of this list contained the respective CCHIT or EuroRec entry, the matching reason or recommendation entry and also the communication problem that the reason or recommendation entry is assigned to within the classification.For instance, a matchinglist entry contains the following content: the abstracted EuroRec entry is "Unique identification (of information entities) requires existence of unambiguous identifiers".This entry is matched to the recommendation "A.I.2.2.E3: requires [Unambiguous identification of each information object instance]".This recommendation entry is assigned to the communication problem "Values of important attributes are missing" within the problem categorization.
In cases no matching reason or recommendation could be found in the classification, the entry in the matching-list contained only the EuroRec or CCHIT entry identification.After the completion of the independent comparison, the reviewers merged their individual results into one common matching-list.Wherever necessary, differences in the matching were discussed in order to find a mutual agreement.

Identification of the Missing Quality Aspects
Based on the merged matching list, it was possible to determine what communication problems are anticipated in the quality frameworks CCHIT and EuroRec.For this purpose, the communication problems of the entries in the merged matching list were compared to the problem entries in the categorization of communication problems.

RESULTS
The results presented in the following sections (6.1.and 6.2.) are a summary of the detailed results obtained from this study.Due to the limitations in space, it is not possible to provide the detailed matching of the quality requirements with the assigned reasons for communication errors.The full matching of the requirements with the problem reasons may be obtained from the authors on request.
The following sections presents the results of the pair-wise comparison between the categorization of communication problems on the one side and the quality frameworks CCHIT and EuroRec on the other side: In section 6.1, the communication problems that are anticipated by the CCHIT-framework are presented.Subsequently, the communication problems that are anticipated by the EuroRec-framework are listed in section 6.2.The results in sections 6.1 and 6.2 show the outcome of the comparison of the two quality frameworks with the categorization of communication problems on the most detailed level.Thus, they help to understand the similarities between the content of the two approaches (i.e., frameworks and categorization).However, the complementary view (i.e., what problems are not covered by the respective quality framework) is of even greater interest.This can only be answered in light of the whole categorization, i.e., when the findings contained in tables 3 and 4 are correlated with the whole categorization.The results of this kind of correlation can be found in section 6.3, which provides an overview of the problems and problem classes that the CCHIT and EuroRec framework do not anticipate.

Results of Comparison: Communication Problems Anticipated by the CCHIT-Framework
Table 3 summarizes the communication problems that are anticipated by CCHIT quality-framework: The first column from the left (i.e., "CCHIT ID & entry") names the entries of the CCHIT-framework.In case the entry was abstracted, the second column (i.e., "Abstraction") contains the abstraction of the CCHIT entry that was necessary for a better comparison.The third column (i.e., "Recommendation/Reason") contains one of the matching reasons or recommendation entries from the problem categorization.The fourth column (i.e., "Problem") names the concrete communication problem.The fifth column (i.e., "Problem class") names the problem classes that are groupings of similar concrete problems.

Results of Comparison: Communication Problems Anticipated by the EuroRec-Framework
Table 4 summarizes the communication problems that are anticipated by the EuroRec quality-framework.It is structured similar to Table 3.

Classes and Problems Covered and Not Covered by the EuroRec and CCHIT Frameworks
Table 5 shows the categorization of communication errors.Because of the large numbers of the reasons and requirements, the entries are not included in the table.Instead, the first four levels of the categorization are specified (see in the headers of Table 5).
As in the previous two sections (see Table 3 and Table 4), the problem classes that group similar problems are also listed.Due to the large number of entries, Table 5 also includes the administrative levels "Aspect" and "Detailed Aspect"; the "Aspect" level is the most abstract and is meant as a rough differentiation as to whether a communication problem concerns the information objects, their administration, or their transfer between application systems.The "Detailed Aspect" level differentiates the rough categorization of the first level according to more concrete aspects (e.g., the aspect "(Series of) Information objects" is differentiated into "Content" and "Acquisition and import") and contains 10 entries.The "Problem class" level groups similar concrete problems (e.g., the entry "Content" on the second level contains amongst others the classes "Wrong details in data" and "Missing data"), and includes 28 problem classes.On the fourth level (i.e., "Problem"), the concrete problems are listed, 81 in total.
Table 5 further clarifies which communication problems are covered and not covered by the quality frameworks of EuroRec and CCHIT.In the two right columns ("CCHIT" and "EuroRec"), "X" indicates entries of the problems categorization covered by CCHIT and/or EuroRec.Complementarily, this provides an overview over the communication problems that are not anticipated:

CCHIT ID & entry (Examples of the requirements that match the categorization entries [3])
IP 04.09 -The system shall provide the ability to maintain a coded list of problems.
IP 16.09 -The system shall provide the ability to document medication administration using a positive ID technology to confirm right patient, right medication, right dose, right time, and right route.FN 07.06 -The system shall provide the ability to specify medication order details including dose, route, frequency, and comments.Dose, route, and frequency must be captured and maintained as discrete data.
IP 08.30 -The system shall provide the ability to electronically communicate the order to the receiving departmental system.

Abstraction
Usage of coded elements.

Existence of attributes
for the unique identification of information objects.
There are attributes that are relevant/ mandatory (for the respective information object). n.a.

Requirement/Reason
Requirement

Problem class (Grouping of similar problems)
Wrong details in dataproblems with wrong content.
Missing data -problems with incomplete information objects or incomplete series of information objects.
Poor quality of image or rest of content -problems with bad imaging quality or bad quality of the content of information objects that is not stored in tags or attributes.

DISCUSSION
Quality of EHRs is multidimensional and, therefore, has to be considered from various perspectives/views.This is especially important in the context of EHR quality seals or EHR system certifications as they certify the quality of a system as a whole.The most prominent approaches, the one offered by CCHIT and the one under development by EuroRec, are based on a functional approach to quality.The current manuscript aims at the analysis of these two approaches from the perspective of information exchange.It investigates the perception of weaknesses in information exchange by functional requirements as has been employed in the two approaches.This discussion section first summarizes the main results from this study, and then discusses the methods, the study limitations, potential benefits, and use of the results.

Summary and Discussion of the Main Results
The results of this study clearly indicate that both the approach of CCHIT and EuroRec are focused on the approval of the functional criteria of EHRs, specifically the clinical functionalities, such as medication or allergies, etc.This is also reflected in the categories of information weaknesses that could be matched.The majority of the information weaknesses concern content issues such as errors in data entry or missing data.Hardly any more common information weaknesses (e.g., application, network, or transcription errors) are covered.But even with regard to content-related problems, neither CCHIT nor EuroRec is comprehensive.Both approaches do not or only in parts cover the problem of semantic interoperability or data incompatibilities including aspects such as ambiguity of content or incompatible value representation of data attributes.In addition to different groups of functional criteria, CCHIT certifications also contain two other categories, security and interoperability requirements.EuroRec certification also contains these two categories of requirements.Again, both approaches only cover these groups of requirements roughly and in the majority of the observed cases in relation to clinical functionalities.Several essential data security issues could not be observed in neither of the approaches.Problems regarding availability of data or access time, for example, are not taken into account.The same is true for potential storage problems and causes for loss of data (e.g., inappropriate access, deletion by accident, wrong data assignment or storage failure), which are not covered in detail.

Journal of
Problems regarding the application software itself, e.g., stability, reliability, and communication in the transfer of data, are not or only implicitly -through other requirements -covered by CCHIT and EuroRec.In general, non-functional requirements are to a great extent neglected.
The approach of CCHIT can be regarded as more advanced than EuroRec.EuroRec differs from CCHIT both in the level of details of certification processes and to the extent to which the requirements for an EHR are covered in their certifications.Possible reasons for this may be the shorter existence of EuroRec certifications or the role that EuroRec plays in European certification.CCHIT acts as an official body for certification within the US, whereas EuroRec so far has no official mandate.It is, therefore, more difficult to align a greater set of requirements with all the partnering institutions/countries.Consequently, EuroRec aimed at establishing a basic, commonly agreed set of criteria rather than a comprehensive set of criteria.

Discussion of the Methods
Prior to the matching of the two approaches with the classification of information weaknesses, the requirements contained in these two approaches were abstracted.Although the abstraction of requirements was only done when necessary and to the extent to allow matching with information weaknesses, such a qualitative approach always harbors the risk of generating a false match by abstracting requirements in a wrong or too general way.To avoid false matches as much as possible, the abstraction of statements also has been done by two independent coders.The same is true for the matching of the requirements, although it has been done by two independent coders who afterwards discussed and reached consensus in their results; there is always some risk of subjectivity in judgment.What may also have influenced the results of this study is the classification of information weaknesses used to analyze the two quality certification approaches.The completeness and validity of this classification have a major influence.Therefore, the classification is based on a sound methodology and was developed using an extensive literature analysis and was also subject to a thorough judgment through domain experts.Details about the classification and the methodology that it is based on can be found in Saboor 2009 [2].

Study Limitations
In order to be able to provide a comprehensive judgment of the analyzed approaches with regard to their coverage of possible information exchange problems, it would 384 Quality of Electronic Health Records -Coverage of Potential Information Weaknesses by Major EHR Quality Seals additionally be necessary to align the results from the current study with results from other studies and with different approaches covering problems of information exchange.To the best of our knowledge, there is currently no other study that analyses EHR quality certification approaches with regard to problems in information exchange.A corresponding comparison or discussion is, therefore, not possible at the moment.There is also no other comprehensive and universal classification of weaknesses in information exchange available that would allow for a comparison of the current results.All of the other approaches available are targeted towards specific domains or cover subgroups of possible information weaknesses (see e.g., [2,23]).

Potential Benefits and Use of the Results
The study discusses the coverage of potential weaknesses in information exchange within current functional quality certification approaches and, therefore, makes an important contribution to the establishment of a comprehensive view on the quality of EHRs.
The study contributes to the existing quality certifications by raising awareness for new requirements resulting from the field of information exchange, by confirming existing functional requirements, and indicating potential gaps in existing functional requirements.The results may also be used as a basis for further improvement of existing certification approaches, or during the process of requirements engineered to prospectively avoid or identify potential quality deficits.
The results indicate that interoperability, either on a technical or semantic level, is only covered very roughly by both approaches.As interoperability in the exchange of data is a key feature of EHRs, this seems one of the major drawbacks with regard to the current certification approaches.CCHIT and EuroRec strongly focus on clinical functionality but often lack consideration of these functions within the scope of transinstitutional exchange of data.Several European projects, such as epSOS (Smart Open Services for European Patients) [24] or HITCH (Healthcare Interoperability Testing and Conformance Harmonisation) [25], as well as initiatives such as IHE [26] and HL7 cover these matters either directly or indirectly.It is essential that such approaches/concepts are integrated sustainably and constantly in the existing or even future certification approaches.

CONCLUSION
The quality of EHRs is dependent on a great variety of requirements from different functional domains involved in these systems and varied views on quality.Potential buyers and users expect EHRs to offer a high level of quality.This is especially true when these systems are awarded with a quality seal indicating a proven level of overall quality.
The current study aims at improving the coverage of the existing quality certification approaches by observing their requirements from the perspective of weaknesses in information exchange.The results of this study clearly indicate that both the approach of CCHIT and EuroRec are focused on the approval of the functional criteria of EHRs, specifically the clinical functionalities, such as medication or allergies, etc. Hardly any Journal of Healthcare Engineering • Vol. 2 • No. 3 • 2011

III. 2 )
in the communication of information objects (errors related to the forwarding or processing of information objects) Missing or incomplete processing rules Stagnant processing Communication disruption or Incomplete communication Communication with wrong partner No communication possible III.1.d)Establishment of communication connections (errors related to the establishment of communication connections) No communication establishment Requested system denies access or connection III.1.e)Incompatible communication interfaces (errors related to the communication interfaces of application systems) Too many different communication interfaces III.1.f)Leaking availability of systems (errors related to the availability of application systems) System is (temporarily) down Blocked system III.1.g)Unstable software (errors related to the implementation of the application systems (software)) Network (errors related to the (physical) network infrastructure) III.2.a) Errors in infrastructure (errors related to the network) Malfunctions of the network Network infrastructure is not uniform Insufficient electronic communication III.2.b) Insufficient capacities (errors related to capabilities of the network) Insufficient network bandwidth

III. 4 )
Security (errors related to the secure communication of information objects) III.4.a)Unsecure communication (errors related to securing communication links) Violation of privacy policy Unsecure communication of critical contents