Automating Quality Metrics in the Era of Electronic Medical Records: Digital Signatures for Ventilator Bundle Compliance

Ventilator-associated events (VAEs) are associated with increased risk of poor outcomes, including death. Bundle practices including thromboembolism prophylaxis, stress ulcer prophylaxis, oral care, and daily sedation breaks and spontaneous breathing trials aim to reduce rates of VAEs and are endorsed as quality metrics in the intensive care units. We sought to create electronic search algorithms (digital signatures) to evaluate compliance with ventilator bundle components as the first step in a larger project evaluating the ventilator bundle effect on VAE. We developed digital signatures of bundle compliance using a retrospective cohort of 542 ICU patients from 2010 for derivation and validation and testing of signature accuracy from a cohort of random 100 patients from 2012. Accuracy was evaluated against manual chart review. Overall, digital signatures performed well, with median sensitivity of 100% (range, 94.4%–100%) and median specificity of 100% (range, 100%–99.8%). Automated ascertainment from electronic medical records accurately assesses ventilator bundle compliance and can be used for quality reporting and research in VAE.


Introduction
Patients who receive mechanical ventilation are at high risk of complications and poor outcomes including death [1]. To effectively manage these high risk patients, providers are encouraged to put in place best practice "bundles" addressing the use of deep vein thrombosis (DVT) prophylaxis, peptic ulcer prophylaxis, oral hygiene, elevation of the head of the bed, daily sedation holiday, and daily spontaneous breathing trial [2]. The ventilator bundle has formed the backbone of many quality improvement efforts and metrics for intensive care units, though its impact on patient outcomes remains uncertain [3]. In 2011, CDC/NHSN proposed a new approach to surveillance including a broader range of ventilator complications termed ventilator-associated events (VAEs) [4]. We sought to investigate if compliance with ventilator bundle practices effectively reduces the risk of the broader set of VAEs and evaluate the relative contribution of each bundle element to patient outcomes. In order to accomplish this, we needed to develop a reliable strategy for assessing bundle compliance for a large number of patients in an efficient manner.
Manual chart review is the "gold standard" of retrospective studies. However, it is time-consuming, inaccurate, resource intense, and not feasible for large sample sizes. The recent development of information technology and the widespread use of electronic medical record (EMR) systems [5]   signature also can be translated into a real time automated algorithms or "sniffers, " where the same rules that were used to retrospectively search charts electronically can give real time or near real time reports and alerts to improve patient care [6]. This study aimed to develop and validate digital signatures for each part of the ventilator bundle, including DVT prophylaxis, peptic ulcer prophylaxis, oral care, head of bed elevation, and sedation breaks.

Materials and Methods
We designed this study as a retrospective study with both derivation and validation cohorts ascertained from intensive care unit patients. The Mayo Clinic Institutional Review Board approved the study as minimal risk research with waived informed consent.

Study Population.
We used a retrospective cohort of 1000 randomly selected patients who were admitted to the intensive care unit (ICU) for at least two consecutive days during 2010 to form our derivation cohort. Of these, 542 met our study inclusion criteria including two consecutive days of mechanical ventilation and research authorization. Our derivation cohort included both ventilated and nonventilated patients to ensure we would have an adequate number of both "true positive" and "true negative" compliance for each element of the bundle while adjusting our search strategy. We then validated the electronic data extraction strategy in an independent cohort of 100 randomly selected patients who were mechanically ventilated for at least two consecutive days in 2012. The purpose of the selection of mechanically ventilated patients from two different years was to better assess the performance of the strategy. Patients aged < 18 year or without research authorization were excluded.

Electronic Data Extraction.
To develop the electronic data extraction strategy, we utilized data from a custom integrative relational research database that contains a near real-time copy of clinical and administrative data from the electronic medical record (EMR). The Multidisciplinary Epidemiology and Translational Research in Intensive Care (METRIC) datamart accumulates pertinent vital signs, fluid input/output, and medication administration record data within an average of 15 minutes from its entry into the EMR and serves as the main data repository for data rule development. More detailed structures and contents have been previously published [7].
For each bundle element, we iteratively improved the accuracy of our electronic query using the derivation cohort ( Figure 1: flow chart). In all iterations, we calculated and analyzed sensitivity and specificity compared to the reference standard and examined discordant pairs for data which could be used to improve the electronic search accuracy. Once we achieved acceptable sensitivity and specificity, we validated our queries in another independent cohort and calculated final sensitivity and specificity of our digital signatures. The final electronic queries for each ventilator compliance bundle were presented in Table 1.

Reference Standard.
The reference standard was defined as the agreement between manual and electronic data extraction. A trained investigator (LH), who was blinded to electronic data extraction result, performed comprehensive medical record review to identify the presence or absence of each component of ventilator compliance bundle according to predefined definition (Table 1) between 00:00 to 23:59 on ICU day 2 in the derivation cohort and mechanical ventilator day 2 in the validation cohort. In case there was a disagreement between manual and electronic data extraction, a third independent investigator (JCO), who was blinded Table 1: Bundle components and definitions. The "medical definition" refers to the objective of the bundle element. The "EMR definition" is how we operationalized this for our digital signature. The EMR section used refers to portions of the patient chart searched with the digital signature for the bundle element.

Ventilator compliance bundle element
Medical definition EMR definition EMR section used

DVT prophylaxis
The presence of an appropriate anticoagulant within 24-hour period The systemic administration of one of the following medications within 24 hours regardless of dosage use: Medication administration record, fluid data

Peptic ulcer prophylaxis
The presence of an appropriate acid-inhibitory drug or sucralfate within 24-hour period The systemic administration of one of the following medications within 24 hours regardless of dosage use:   to both results, would make the final adjudication; this definition has been previously used [8].

Statistical Analyses.
We summarized clinical characteristics of derivation and validation cohorts using mean ± SD for continuous variables and using counts with percentages for categorical variables. We calculated sensitivity and specificity of each electronic data extraction based on the comparison of the test result and reference standard in the two cohorts. The 95% confidence intervals were calculated using an exact test for proportions. JMP statistical software (version 9.0, SAS Institute Inc.©) was used for all data analysis and randomization.

Results and Discussion
The derivation subset included a total of 542 ICU patients randomly selected from January 2010 to December 2010. The validation subset included a total of 100 randomly selected patients from January 2012 to December 2012. There were no differences in age, gender, and race between the two groups. The demographic characteristics and baseline of the derivation and validation subset are summarized in Table 2.
The sensitivities of five ventilator bundle components were from 92% to 100% in the derivation subset in our final iteration. The specificities ranged from 50% to 99.8% after modification. Elevation of the head of the bed was the bundle element that could not be improved to an adequate sensitivity or specificity because of variable and inconsistent charting. We thus decided not to validate this query and did not test in our validation cohort. When examining the validation cohort, the sensitivities of our digital signatures ranged from 94.4% to 100%, and specificity was 100% for each (Table 3).
Manual chart review was slow, requiring our reviewers to access two or more programs to abstract the relevant data from the EMR, taking an average of 10 minutes/patient. We achieved comparable results with electronic data abstraction, BioMed Research International 5 which will allow us to scan compliance of thousands of patients in a reasonable time frame for the second part of our study, an assessment of ventilator bundle compliance on the risk of developing a VAE.
With the widespread adoption of EMRs, the digital signature is an increasingly attractive alternative to manual chart review. Digital signatures have several advantages. First, they are more efficient, making larger-scale cohort studies practical without significant personnel or time expenditure. Second, in developing them, we can look for markers of specific activities that correlate with actual patient outcomes and thus mitigate some types of reporting bias. For example, our DVT prophylaxis signature looks for times where one of the commonly used agents is actually administered, as opposed to asking staff to fill out a checkbox saying that "DVT prophylaxis has been addressed. " More broadly, this kind of search allows automated searching beyond simple billing codes and administrative data, which are notoriously variable in accuracy [9][10][11][12].
Finally, digital signatures have the potential to be translated into real-time electronic search algorithms, or "sniffers, " to provide near real time data. For example, the same rules that we used to develop our peptic ulcer prophylaxis signature could provide real time data on compliance, use and misuse. Sniffers are increasingly prevalent, though a recent systematic review highlighted issues with variable performance and accuracy owing in part to inadequate validation [13]. As we noted in our effort to derive and validate a signature for head of bed elevation, variability in documentation practice may limit the ability to derive a clinically useful digital signature; however, an emerging automatic documentation technology could help overcome this limitation.
An interesting feature we noted in our validation cohort was higher diagnostic performance than in our derivation cohort. As our derivation cohort was what we used to derive the search, we expected to be "overfitted" to that set and lose both sensitivity and specificity as we moved to another cohort. However, we instead noted improvement. This probably owes to improvements made in the ICU datamart's accuracy over time, as our derivation cohort was from archived data in 2010, and validation used the same rules in 2012. We noted better agreement between datamart and EMR data in the more recent set and thus better improvement with our rules-based signatures.
With reasonable search algorithms, this allows us to move forward and evaluate the efficacy of specific ventilator bundle elements in preventing VAE. A previous study at our institution using pre-and postbundle implementation measures found no effect, but that study was an ecological design and was not able to evaluate individual patient bundle compliance [3]. With these signatures, we will be able to give a higher resolution evaluation of the effect of the ventilator bundle. We can also work towards developing real time compliance monitoring of the ventilator bundle for both quality improvement purposes, aiming to indirectly improve care and reduce costs with passive monitoring of valueadding practices.
Our study also has several limitations. First, as noted above, we are limited by what is electronically documented and the accuracy of initial inputs. Second, preferred medications and formularies differ between hospitals, and while our digital signature may be a starting point for other hospitals attempting something similar, calibration and validation would be necessary to generalize this elsewhere. Finally, the single-center, academic nature of our institution could raise the concern of referral bias and further limit generalizability of our approach.

Conclusion
The digital signatures used to extract and screen the usage of ventilator-associated pneumonia bundle elements were both sensitive and specific for DVT prophylaxis, peptic ulcer prophylaxis, daily sedation break, and oral care. We were not able to derive a similarly useful signature for head of bed elevation. These signatures have acceptable sensitivity and specificity for use in our larger study of the impact of the ventilator bundle on risk of VAE.