Identifying patients with physician-diagnosed asthma in health administrative databases

1The Hospital for Sick Children; 2University of Toronto; 3Institute for Clinical Evaluative Sciences, Toronto, Ontario; 4National Jewish Medical and Research Center, Denver, Colorado, USA Correspondence: Dr Andrea S Gershon, Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5. Telephone 416-480-4055 ext 3619, fax 416-480-6048, e-mail andrea.gershon@ices.on.ca Asthma is a common chronic condition that imposes a heavy and expensive burden on individuals, populations and health care systems. Previous population-based studies evaluating the burden of asthma have depended on cross-sectional surveys and cohort studies in specific populations (1). Although these sources of data are useful, they are only able to provide an estimate at one point in time, and are often unable to provide comparison estimates for smaller regional areas or patient groups. A prospective clinical disease registry of all individuals with asthma would be ideal; however, the resources and costs required to operate such a registry would be prohibitive. Health administrative (HA) databases provide a significantly less expensive, practical, and unbiased method of measuring and surveying asthma in a population. In regions with a universal, single-payer health care system such as the province of Ontario, where health care workers essentially submit standardized data to HA databases, they can be particularly useful. The Ontario Asthma Surveillance Information System was created to initiate and maintain a population-based longitudinal surveillance system for asthma based on HA data. Such a program, however, cannot succeed without first confirming that the HA identification of asthma is valid. In Ontario, physicians provide diagnostic data for which there are no incentives associated with accuracy (2). HA definitions of asthma in children have been previously validated; however, due to differences in presentation, diagnosis and treatment of asthma, and the presence of more and different comorbidity in adults, it cannot be assumed that these definitions also apply to adults (3,4). Therefore, to find a valid HA definition to identify physician-diagnosed asthma in adults, a case verification study was conducted (5). orIgInal artIcle


Subjects
Patients with and without asthma were randomly selected from primary care physician (PCP) practices that were randomly selected from all PCP practices across Ontario -a Canadian province with a population of approximately 12 million.The sampling frame included all PCPs identified in the Canadian Medical Directory Database who had treated 30 or more asthma and 30 or more chronic obstructive pulmonary disease (COPD) patients in 2003.A letter was mailed and a follow-up telephone call made to the selected PCPs, inviting them to participate in the study.PCPs were excluded if they did not have an office-based practice, did not have the time or space to accommodate a chart abstractor working in their office or did not use an electronic billing system.If a PCP did not respond, refused or was excluded from participation, a replacement PCP was randomly selected and invited.
In each recruited PCP practice, the electronic billing system was used to create a sampling frame that included all the charts of patients 19 to 80 years of age who were currently residing in Ontario and who were seen by the PCP in the previous five years.Population-based samples of individuals are often used to validate diseases in the population; however, to evaluate the ability of HA data to distinguish between asthma and COPD, and other related respiratory conditions, a more conservative cohort that included higher proportions of these patients than was present in the general population was used (3,6).Therefore, the study population consisted of equal numbers of patients who had asthma, COPD, asthma-related conditions and other conditions according to the their PCP's electronic billing system.Patients with asthma were included to ensure that the HA definition could truly identify individuals with asthma.Patients with asthma-related conditions and COPD were included to ensure that the algorithm could distinguish between asthma and clinically similar conditions, and because patients with occult clinically significant asthma in the population were likely to be found in these groups.Patients not specifically identified as having 'other' conditions not included in any of the other categories were included as respiratory healthy controls.To simplify subject selection and ensure that these patients did not have respiratory disease, this final category was limited to patients with hypertension or musculoskeletal problems (Table 1).Of note, it was possible for an individual to be eligible for inclusion in an asthma and a nonasthma category.These patients were included in the most recent category that they were identified to be in by their PCP.
To obtain a study population with equal numbers of asthma patients, COPD, asthma-related and 'other' diagnoses, all of the patients in each PCP practice were subdivided into these four categories.Ten charts were then randomly selected from each of the four categories and abstracted by a trained abstractor.The patient's entire chart was reviewed and a standardized abstraction form completed.Also abstracted was the unique health insurance number given to all individuals in Ontario.On completion of the abstractions, the abstracted data were reviewed for completeness.
Based on earlier studies validating asthma in HA databases and expected sensitivities and specificities (3,7), an approximate sample size of 150 individuals with asthma was determined to be required to obtain 95% CIs with a precision of plus or minus 12%.
determining reference standard diagnoses Two respirologists, who were blinded to the diagnostic category but not to the objectives of the study, independently reviewed all of the abstracted patient charts from all four categories to identify the cases as asthma or nonasthma.The identification of an asthma diagnosis was determined according to predefined criteria based on the Canadian Asthma Consensus Guidelines (8).Cases in which there was disagreement were discussed and an attempt to reach consensus was made.If consensus could not be reached, a third respirologist was consulted to be the 'tie-breaker'.Without pulmonary function tests being performed on every patient, this reference standard may not be considered the most rigorous available; however, it surpasses the standard by which the majority of patients with asthma are diagnosed (ie, by a lone PCP) and, therefore, was considered acceptable (9).It was believed to be particularly suitable because it was being used to validate an HA definition of physician-diagnosed asthma for use in the general population where most asthma is diagnosed by PCPs (10).Of note, to facilitate analysis, reviewers were asked to 'force' patients into asthma or nonasthma groups although, in some situations, it may have been appropriate for them to have been in both.

Constructing hA claim algorithms
Reference standard diagnoses of physician-diagnosed asthma were linked to patients' HA records via their unique health card number.Using fields from two HA databases, predefined HA definitions (eg, two ambulatory care visits and/or one hospitalization for asthma) that were hypothesized to accurately identify physician-diagnosed asthma were constructed and tested against this reference standard.The Ontario Health Insurance Plan (OHIP) physician services claims database, which contains information on outpatient claims for all Ontario residents (including claims for physician visits, laboratory tests and diagnostic imaging), was used.Physicians are reimbursed through OHIP by submitting claims for medical services provided.A disease code is provided as part of a claim.The inpatient database used was the Canadian Institute for Health Information discharge abstract database, which contains clinical administrative data relating to the hospital services provided to patients by all hospital facilities in Ontario.This includes diagnostic codes for each hospital visit from the International Classification of Diseases -9th or 10th Revision, depending on the year.Before 2002, it was possible to have up to 16 diagnostic codes; as of 2002, it has been possible to enter 25 diagnostic codes.If any of these diagnostic codes was for asthma, regardless of what the other codes were for, the hospitalization was used toward the HA definition.Collection of hospital discharge abstract data for submission to Canadian Institute for Health Information is mandatory for all hospitals in Canada.

Analyses
Demographic characteristics and clinical profiles were compared between the asthma and nonasthma groups using the Student's two-tailed t test for continuous variables and the c 2 test for categorical variables.Agreement between panel reviewers regarding asthma diagnosis was analyzed using the kappa statistic.The accuracy of the predefined HA definitions compared with the reference standard diagnoses were determined using the concepts of diagnostic test evaluation.Ninety-five per cent CIs for the test characteristics, where appropriate, were also calculated using the binomial distribution.Recognizing that PCPs could have had pertinent patient information that was not documented in the patient chart, and that would not have been available for the expert panel to base their reference standard diagnosis on, the same analysis was repeated using the PCP's chart abstract as the reference standard.In this case, if any mention of asthma was made in the chart or if a patient received treatment that was consistent with asthma treatment (but did not necessarily have a formal diagnosis of asthma), they were considered to have asthma as per this reference standard.All statistical analyses were performed using SAS version 9.1 (SAS Institute Inc, USA).

ethics committee approval
Ethics committee approval was obtained through The Hospital for Sick Children (Toronto, Ontario).

Recruitment of PCPs and chart abstraction
A total of 5821 PCPs saw 30 or more asthma and 30 or more COPD patients in 2003.Of these, 81 PCPs were contacted and invited to participate in the study.Forty responded -a crude response rate of 49.3%.Of those who responded, 22 (55%) chose not to participate and five were deemed to be ineligible because they did not have an office-based practice or did not have space for a chart abstractor to work.This left 13 PCPs who agreed and were eligible to participate in the study.From the 13 participating clinics, data from the charts of 518 patients were abstracted.

Inter-rater agreement between panel reviewers
The abstracted charts were reviewed by the expert panel to determine whether the patients had asthma.The overall agreement between reviewers expressed as a kappa and controlling for site was 0.79 (95% CI 0.73 to 0.85) (Table 2).Of the 518 charts reviewed, there were 56 with disagreements that were resolved by a tie-breaker.

Study population characteristics
According to the reference standard panel diagnosis, 160 patients (31%) in the study population had physician-diagnosed asthma.Of note, more than 20% of these patients were not originally diagnosed with asthma by their PCP.The patients for whom the PCP and expert panel disagreed did not significantly differ

dISCuSSIon
For surveillance purposes, if one must trade off sensitivity for specificity or vice versa, a more sensitive definition of disease is more desirable because it ensures that fewer affected individuals are missed (11).In the present case-verification study of HA definitions to identify physician-diagnosed asthma for surveillance purposes, two algorithms were shown to have both higher sensitivity and specificity when either an expert panel diagnosis or PCP designation of asthma was used as the reference standard.A second consideration when choosing an HA definition is the time period needed to satisfy it, because one that requires less time to identify cases will be more responsive.
Given our results and these considerations, we chose the algorithm of two or more ambulatory claims, and/or one or more hospital claim(s) for asthma in two years for the Ontario Asthma Surveillance Information System surveillance protocol.
Once an HA definition that meets one's needs has been chosen, it can then be used to construct a cohort of all individuals with physician-diagnosed asthma.This can, in turn, be used for many purposes (eg, to determine the prevalence of physician-diagnosed asthma, the health services used by individuals with physician-diagnosed asthma and the direct health care cost of physician-diagnosed asthma care).Such information is invaluable to health policy makers, stakeholders, clinicians and members of the public wanting to understand the impact of asthma on the population.Some of the HA definitions were noted to have a relatively low positive predictive value (PPV).This was because the HA definition identified individuals as having physician-diagnosed asthma, but the interpretation of the abstracted chart by the expert panel and/or the PCP chart did not.The main reason we hypothesize these individuals were not identified as having asthma was because full information regarding their respiratory status was not always recorded in their chart.There are several reasons why this may have been the case.For example, an individual may have had another physician (not their PCP) following their asthma or their asthma may have been diagnosed before becoming a patient of their PCP.In support of our hypothesis, further review of the histories of the 84 individuals who were found to be 'falsely' identified by our surveillance HA algorithm of two or more ambulatory claims and/or one hospitalization in two years, showed that 59 (75%) of them saw two or more doctors for their asthma and 17 (20%) had not made any asthma claims in the previous five years (data not shown).In support of some of these 'false-positive' cases actually being true positive cases, we found that nine (11%) of these individuals had been hospitalized for asthma and that 66 (79%) had three or more ambulatory care visits for asthma (data not shown).
Case verification using chart abstraction is a common study design used in health services research to validate case definitions used in HA databases.Previously, our group used this approach to validate a single ambulatory care claim for asthma in children (3), with an overall observed sensitivity of 91.4% and specificity of 82.9%.In the province of Quebec, a documented diagnosis of asthma in a PCP chart was used as a gold standard to validate asthma diagnosis in the provincial medical services databases.In individuals seen at one academic family practice clinic, two or more primary care outpatient  (12).In Manitoba, physician claims in the Manitoba Population Health Research Data Repository were compared with survey data regarding the prevalence of asthma and the agreement was found to be moderate (13).Our study expands on these previous studies by validating physician-diagnosed asthma in a wide variety of primary care practices in Ontario.Because individuals in Ontario cannot see a specialist without a referral from a PCP, we are confident that the present study considers and is generalizable to most, if not all, physician-diagnosed asthma in the province.
In addition to validating HA definitions against an expert panel reference standard, they were also validated against the PCP chart.The results differed very little, which further confirms the test characteristics the HA studied as well as the validity of using two or more ambulatory claims and/or one or more hospital claim for asthma in two years to build a surveillance cohort.
The strengths of the present study are its generalizability, its ability to validate an HA algorithm against two clinically relevant standards including one based on expert panel opinion, and the inclusion of comparison groups of patients with diseases that are similar to asthma.
The main limitation of the present study was that, because a true population sample was not used, the estimates of test characteristics were only approximations of the true population test characteristics.There are a few reasons why one may suspect some of the measurements to be higher or lower than those found.For example, using a study sample population heavily weighted toward conditions that resembled asthma may have led to an underestimate of the test characteristics.In addition, the expert panel that forced eligible patients who were in both the asthma and nonasthma categories in the original patient selection, into only one of these designations may have also led to an underestimate of the test characteristics.Alternatively, identifying most patients with asthma, COPD and other respiratory conditions (but not all) through patient records in which OHIP billing claim diagnoses used in the HA definition also came from, may have overestimated the test characteristics.Thus, compromises were made to validate an HA definition that could distinguish between asthma and other common similar respiratory conditions, and that was reflective of real-life clinical practice.Such compromises, which are often used in studies evaluating diagnostic tests, mean that the results are only estimates of the true test properties and they do not change the relative accuracy of the HA definitions studied (3,6).

TABLE 2 Overall and site-specific inter-rater agreement between panel reviewers
Test characteristics of hA claim algorithmsTables4 and 5summarize the diagnostic test characteristics of the predefined HA definitions compared with the reference standard panel diagnosis and PCP chart diagnosis, respectively.According to the PCP diagnosis, 196 (37.8%) of the 518 patients had physician-diagnosed asthma.