use of electronic data and existing screening tools to identify clinically significant obstructive sleep apnea

1Cumming School of Medicine; 2Department of Community Health Sciences; 3Division of Respirology, Department of Medicine, University of Calgary, Calgary, Alberta Correspondence: Dr Willis H Tsai, Rockyview General Hospital, 7007 14th Street Southwest, Calgary, Alberta T2V 1P9. Telephone 403-943-3000, e-mail tsai@ucalgary.ca Obstructive sleep apnea (OSA) affects at least 24% of men and 9% of women, and is associated with negative cardiovascular and metabolic health outcomes, increased risk of motor vehicle collisions, poor quality of life and increased medical costs (1-8). OSA is normally defined by the respiratory disturbance index (RDI) as determined by polysomnography or ambulatory monitoring. However, the RDI does not correlate well with sleep apnea symptoms or treatment outcomes (9,10). In the Wisconsin Sleep Cohort Study, approximately one-fifth of patients with OSA reported sleepiness (1). Moreover, adverse health outcomes related to OSA, such as cardiovascular disease or diabetes, may occur independently of symptoms, particularly in those with severe OSA (3,6,7). Current clinical guidelines recommend treatment of OSA to improve symptoms or to mitigate the risk of adverse health outcomes in patients with severe disease (11,12). A number of clinical prediction rules for OSA have been described, including the adjusted neck circumference score (ANC), cricomental space, elbow sign, Berlin Questionnaire, and the STOP and STOP-Bang questionnaires (13-18). These tools were validated for identifying OSA as defined by RDI. However, no prediction rules exist that identify patients with clinically significant OSA; that is, symptomatic or severe OSA. Moreover, many of the clinical prediction rules have not been evaluated as part of an online screening process. An automated online process for identifying patients with clinically significant OSA (ie, those who might benefit from treatment) would be useful for triaging patients to the appropriate health care provider, improving clinical management and identifying patients for research purposes. The primary objective of the present study was to develop diagnostic algorithms for identifying patients with symptomatic or severe OSA. The secondary objective was to validate these algorithms, as well as adaptations of existing rules, when collected in an electronic format.

Centre Sleep Centre (Calgary, Alberta) between January 1, 2009 and January 1, 2011 were identified.The Foothills Medical Centre Sleep Centre is the only tertiary referral centre for Calgary (a catchment area of approximately 1.3 million people).All referred patients are required to fill out an online questionnaire at the time of referral.

Determination of the primary diagnosis
A primary diagnosis of clinically significant OSA superseded other secondary nonrespiratory diagnoses.OSA was defined as an RDI ≥5/h.OSA severity was defined as mild (RDI 5/h to 14.9/h), moderate (RDI 15/h to 29.9/h) or severe (RDI ≥30/h).
Two American Board of Sleep Medicine-or American Board of Internal Medicine-certified sleep physicians independently reviewed all patient charts and assigned a primary sleep diagnosis to each patient in the cohort, as well as determining whether OSA was clinically significant.If a diagnosis was not agreed on independently, the disagreement was noted and the patient was excluded from the analysis.The sleep physicians performing the chart review were blinded to the results from the online questionnaire.
Clinically significant OSA was defined as symptomatic or severe OSA.Symptomatic OSA was based on the treating physician's impression as documented in the patient's chart.This impression took into account the Epworth Sleepiness Scale, adherence to treatment and the patient's reported perception of benefit.Patients with a severe OSA (RDI ≥30/h) were considered to have clinically significant OSA (regardless of symptoms or comorbid conditions), with the rationale being that severe OSA is associated with comorbid disease for which treatment is recommended by clinical guidelines (11,12).The decision to pursue further diagnostic testing and treatment was at the discretion of the treating physicians.

Ambulatory monitoring
The RDI was determined from level III sleep diagnostic testing (20).The Remmers Sleep Recorder (SagaTech Electronics Ltd, Canada) is an ambulatory monitor that measures snoring, oxygen saturation, respiratory airflow (by monitoring nasal pressure) and body position.The RDI is derived from automated shape analysis based on falls and recovery of digitally recorded oxygen saturation using a 4% desaturation threshold.The flow signal was used for quality assurance on manual review by a sleep physician.

Electronic data elements
The online questionnaire is comprised of 108 questions, which provide a comprehensive overview of a patient's demographics, anthropometrics, snoring history, daytime function and medical history, as well as sleep schedule, behaviour and complaints.
Several questions pertain to heart disease, hypertension and diabetes.These diseases were identified based on positive responses to binary yes/no questions, or self-report in the free-text medical history.Heart disease was defined as a self-reported history of angina, heart failure, heart attack or coronary artery bypass surgery.Hypertension was defined according to self-reported history of hypertension or high blood pressure.Diabetes was defined as self-reported diabetes.
The STOP and STOP-Bang questionnaires consist of four and eight yes/no questions, respectively.The questions in the present study were adapted from the STOP-Bang questions, but wording was not identical (Appendix 1) (18).

Statistical analysis
The distribution of the primary sleep diagnoses among patients referred for clinical assessment/sleep diagnostic testing within the study timeframe was calculated.The agreement between physician-assigned diagnoses was assessed by the kappa statistic.Descriptive statistics were used to compare the clinical and demographic characteristics of patients stratified according to the presence or absence of clinically significant OSA.Univariate logistic regression was then used to identify predictive variables from the online questionnaire, using the presence of clinically significant OSA as the dependent variable.A full model was constructed from the univariate predictors and reduced by stepwise regression.ROC curves and box plots were constructed.Cutpoints were selected by visual inspection, taking into account ROC area under the curve.
Using identified predictive variables, several diagnostic algorithms were constructed.The sensitivity, specificity, positive predictive value and negative predictive value were calculated to assess the performance of the various diagnostic algorithms in predicting clinically significant OSA.A sensitivity analysis was performed by repeating the modelling process to predict mild, moderate and severe OSA as defined by the RDI.The performance of the STOP and STOP-Bang questionnaires, and the ANC score were also assessed to identify clinically significant OSA (13,18).Last, the ability of the univariate predictors to identify OSA (as defined by ambulatory monitoring cut-offs) was examined.
All analyses were performed using Stata 9.0 statistical software (Stata Corporation, USA).Data are presented as mean (± SD) unless otherwise indicated.For all statistical tests, P<0.05 was considered to be statistically significant.

Ethics
The University of Calgary Conjoint Health Research Ethics Board approved the study.

Patient characteristics and diagnostic agreement
Between January 1, 2009 and January 1, 2011, a total of 1426 patients completed an online sleep questionnaire.Of these, one patient did not complete the questionnaire correctly and 202 patients refused consent to participate.An additional 16 patients were excluded due to disagreement between reviewers on their primary diagnosis, leaving 1207 patients in the baseline cohort.However, a total of 166 patients did not have an ambulatory monitoring test, leaving a final cohort size of 1041 patients for analysis (Figure 1).
Clinically significant OSA was the most common primary diagnosis in the cohort (46%), followed by insomnia (28%).Table 1 shows the distribution of primary sleep diagnoses.Patient characteristics are summarized in Table 2.The mean age of all patients was 45±2 years, 57% of all patients were men, and the mean body mass index (BMI) was 30.6±7.6 kg/m 2 .Patients with clinically significant OSA were significantly older, had higher BMI and larger neck circumferences than those without.Within the entire cohort, 21.4%, 31.3% and 6.3% of patients self-reported heart disease, hypertension and diabetes, respectively.
The reviewing physicians agreed on 98.7% of diagnoses (1207 of 1223).The kappa statistic indicated that the agreement between the two reviewing physicians was high (0.98±0.016).

Predictors of clinically significant OSA
A self-reported history of snoring, witnessed apneas, history of heart disease, nocturnal choking sensation, total sleep time >4 h, ANC ≥48 cm and BMI were significant predictors of a diagnosis of symptomatic or severe OSA (Table 3).Self-reported hypertension, diabetes and an Epworth Sleepiness Scale score >10 did not significantly contribute to the model (P>0.05)but were included in the algorithm analysis based on previous associations with OSA in the literature.

Diagnostic performance of decision rules
The ability of diagnostic algorithms to identify symptomatic or severe OSA is summarized in Table 4.A self-reported history of snoring, total sleep time >4 h and a BMI >25 kg/m 2 were associated with sensitivities >90%, although specificity was poor.No single univariate predictor had simultaneously high sensitivity and specificity.
Multivariate prediction algorithms increased either sensitivity or specificity (as compared with univariate predictors), but at the expense of reducing specificity or sensitivity, respectively.For example, snoring or a BMI >25 kg/m 2 was very sensitive but not specific at identifying clinically significant OSA (sensitivity 99.6%, specificity 12.6%), as was a history of snoring or choking (sensitivity 98.0%, specificity 20.7).Conversely, the combination of self-reported heart disease and a BMI >30 kg/m 2 had low sensitivity but was highly specific at identifying clinically significant OSA (sensitivity 17.1%, specificity 92.3%).No combination of univariate or multivariate predictors from the questionnaire was both highly specific or highly sensitive (Table 4).Moreover, diagnostic performance was not increased by increasing the number of predictive variables.
In general, the RDI was very specific at identifying patients with clinically significant disease particularly for moderate or severe OSA (specificity ≥95%).However, while objective testing alone could identify patients with OSA, it was relatively insensitive with respect to clinically relevant disease (sensitivity <35%).Combining a clinical predictor with level III sleep diagnostic testing failed to improve the diagnostic performance with respect to clinically significant disease.
In contrast, existing prediction rules such as the STOP and STOP-Bang, as derived from our modified clinical questionnaire, provided high sensitivities (85% and 91%, respectively) and intermediate specificities (52% and 46%, respectively) when identifying clinically significant OSA.The STOP-Bang had lower sensitivity than simpler multivariate predictors, but had slightly higher specificity.The performance of the ANC was similar to level III sleep diagnostic testing in that it had high specificity (86%) and lower sensitivity (39%).
In keeping with the previous literature, clinical predictors including the STOP-Bang were sensitive (>90%) at identifying patients with OSA (as defined by RDI on diagnostic testing); however, specificity was poor (<40%) (Table 5).Moreover, when administered as an online questionnaire, the derived STOP-Bang questionnaire exhibited similar diagnostic performance to that described in the original paper (Appendix 2) (17).

DISCUSSION
Diagnostic algorithms derived from an online questionnaire can be highly sensitive (>95%) or moderately specific (<75%) at identifying patients with clinically significant OSA, but not both.In contrast, the objectively determined RDI is highly specific (specificity ≥95%) when identifying clinically relevant disease, but suffers from low sensitivity (<35%).Our results suggest that when screening for clinically significant OSA, a combination of both clinical variables and objective testing is required.An automated process involving an online questionnaire and level III sleep diagnostic testing has similar diagnostic performance to previously reported screening strategies.
The present is the first study to evaluate electronically acquired data including existing screening tools (modified for online use) to identify clinically relevant OSA, defined as symptomatic or severe disease and   for which treatment is recommended.Most studies have focused on identifying OSA as defined by the RDI.In our study, both the STOP-Bang and ANC also identify OSA (as defined by RDI) with high sensitivity or high specificity, but not both simultaneously.When used for determining clinically relevant OSA, clinical variables such as a history of snoring, witnessed apneas or a BMI >25 kg/m 2 favour high sensitivity rather than specificity.Given that treatment is recommended in patients with symptomatic or severe OSA, the ability to identify this subgroup of patients with OSA would provide more timely care.
Using online and electronic data to screen for OSA has the potential to improve clinical efficiency, namely by providing an automated screening strategy.In a previous article (19), we outlined a similar approach for identifying patients with insomnia.In combination, a screening strategy that identifies patients with OSA or insomnia would help to direct patients to the most appropriate provider for assessment and management.Furthermore, if there is reasonable confidence in a diagnosis of OSA or insomnia based on this screening strategy, patients could be directed to non-sleep physician providers for initiation of continuous positive airway pressure (CPAP) or cognitive behavioural therapy, respectively.
Regardless of the decision rules employed, modification for online use did not impact diagnostic performance as compared with the original self-administered (or administered) questionnaires.The specific questions varied slightly among screening tools; however, diagnostic performance is similar.Recently, Chung et al (21) demonstrated that the diagnostic performance of STOP-Bang could be improved by alternating the scoring model.However, simultaneously high sensitivities and specificities could not be achieved (21).
As expected, the RDI alone was an imperfect predictor of clinically relevant disease.In a previous study, Whitelaw et al (22) demonstrated that when clinicians are provided with an RDI or apnea-hypopnea index, the ability to predict improvement in quality of life or CPAP compliance is poor (60%).In contrast, we have demonstrated that the RDI (at moderate or greater severity) is very good at identifying patients with clinically relevant disease (specificity ≥95%); however, sensitivity is poor (<35%).Despite slightly different outcomes of interest, both studies suggest that an objective measure alone (RDI) is insufficient for predicting clinically relevant outcomes.
Recently, Pereira et al (23) found that adding clinical data from questionnaires did not increase the discriminant ability of level III testing alone to identify OSA (as defined by the apnea-hypopnea index).Similarly, we found that combining clinical predictors with the RDI did not improve diagnostic accuracy for predicting clinically significant OSA.While the specificity remained high whether RDI and clinical predictors were used alone or in combination, sensitivity was decreased compared with using clinical variables alone.These results suggest that when screening for clinically relevant disease, a two-step process incorporating clinical variables (to maximize sensitivity) and objective testing (to maximize specificity) may be beneficial and warrants further  We also assumed a single primary sleep diagnosis for all patients in this analysis.While secondary sleep diagnoses were coded, treatment decisions were typically based on the primary diagnosis.It is not uncommon for patients to have overlapping sleep disorders and comorbidities.However, we assumed a single primary diagnosis because patients with multiple sleep diagnoses would likely require assessment by a sleep physician rather than benefiting from an automated screening strategy.
Furthermore, the use of ambulatory monitoring rather than polysomnography to determine RDI may have also been perceived as a limitation.However, the Remmers Sleep Recorder has been validated against polysomnography both in terms of bias and clinical management of OSA (24,25).Given a sensitivity of 98% (using a case definition of an RDI >15/h), it is unlikely that the Remmers Sleep Recorder would have missed many patients with OSA (25).Moreover, polysomnography was performed in 30% (n=75) of patients without OSA on ambulatory monitoring (n=276).However, polysomnographic evaluation was at the discretion of the treating physician based on their post-ambulatory monitoring suspicion of OSA.
Finally, the size of our existing questionnaire is too cumbersome for real-world use.It consisted of 108 questions, including data elements derived from STOP-Bang, ANC, insomnia severity index and the Patient Health Questionnare.However, the present study indicates that the number of questions can be markedly reduced.With respect to identifying clinically significant OSA, use of existing decision rules or simple two-variable decision algorithms offer similar diagnostic performance.A similar approach can be used for diagnosing insomnia (19).
Historically, screening tools have focused on prediction of OSA (as defined by RDI), or in terms of predicting treatment compliance.Moreover, these have frequently been paper questionnaires administered at the time of assessment.We propose an automated two-step diagnostic process for identifying clinically relevant OSA, using an online clinical questionnaire to identify patients who should then go on to further objective testing.
Given limited access to sleep clinicians, as well as challenges associated with improving CPAP compliance, an automated process for identifying patients with clinically relevant disease could be beneficial in terms of patient access and triage, resource allocation and timeliness of care.

CONCLUSIONS
When used in an online setting, simple two-variable algorithms or existing decision rules such as STOP-Bang or ANC (adapted for the online setting) can identify symptomatic or severe OSA (ie, clinically significant OSA) with high sensitivity or specificity, but not both.An automated two-step process combining online collection of clinical variables (maximizing sensitivity) and objective testing (maximizing specificity) may improve diagnostic accuracy.

TAbLe 4 Diagnostic performance of predictive algorithms derived from an online questionnaire at identifying clinically significant obstructive sleep apnea
Data presented as %.NPV Negative predictive value; PPV Positive predictive value; RDI Respiratory disturbance index; Sens Sensitivity; Spec Specificity