Diagnosing asthma : The fit between survey and administrative database

1Department of Internal Medicine, 2Manitoba Centre for Health Policy, 3Department of Community Health Sciences, University of Manitoba, Winnipeg, Manitoba Correspondence and reprints: Dr Jure Manfreda, Respiratory Hospital, 810 Sherbrook Street, Winnipeg, Manitoba R3A 1R8. Telephone 204-787-2338, fax 204-787-2420, e-mail manfred@ms.umanitoba.ca L Huzel, LL Roos, NR Anthonisen, J Manfreda. Diagnosing asthma: The fit between survey and administrative database. Can Respir J 2002;9(6):407-412.

A sthma has emerged as an important respiratory and public health problem in Canada.Population-based studies of asthma are hampered by the lack of a clear definition of the disease and its variable course.Standard methods for population studies of asthma include surveying population samples using questionnaires and examining people in laboratories.These procedures are extremely expensive (in terms of both finances and manpower), time consuming and limited to relatively small samples.It would be helpful if, at least for some purposes, these procedures could be replaced by cheaper techniques with adequate validity.Administrative databases such as the Manitoba Population Health Repository (1) offer such an option.While the validity of the Manitoba data has been studied and found acceptable both generally (1) and in a number of specific areas (2), this has not yet been done for asthma.Suitable asthma surveys to compare with administrative data have been lacking.
As part of a Canadian multicentre study (3) to estimate the prevalence of asthma in 1993 to 1994, a sample of the general population in Winnipeg, Manitoba was surveyed.The objective of this analysis is to determine the agreement between prevalence estimates obtained from the population survey and the prevalence of asthma in the same individuals as obtained from the Manitoba Health database.

METHODS
The methods of sample selection and examining procedures have been described in detail (3).In brief, a sample of 18,000 randomly selected telephone numbers for the City of Winnipeg was obtained.From March 1993 to May 1994, interviewers systematically called these numbers, and following an appropriate introduction, they determined whether the number was residential and whether the household contained eligible individuals -those aged 20 to 44 years.If nobody answered, calls were repeated up to seven times, at varying times on varying days.If more than one household member was eligible, the interviewer used a predetermined sampling scheme to select one subject.The subject's name and postal address were recorded.This procedure identified 3402 age-eligible individuals.
The selected individuals were sent a questionnaire with a cover letter explaining the purpose of the study as a survey of general respiratory health and not specifically of asthma.After three weeks, nonresponders were sent a second questionnaire; they were phoned to encourage them to complete it, either by mail or over the telephone.Responses were obtained from 2981 of the selected individuals (84.8%).
Questions were based on the European Community Respiratory Health Survey questionnaire (3)(4)(5).The present comparison focused on responses to two questions: "Have you had an attack of asthma in the last 12 months?"and "Are you currently taking any medication (including inhalers, aerosols or tablets) for asthma?".Answers to questions regarding asthma symptoms in the last 12 months were also analyzed: wheezing or whistling in the chest, being wakened by a feeling of tightness in chest, attacks of shortness of breath or coughing.Finally, respondents were questioned about a diagnosis of asthma by a physician at any time in their lives.
The Manitoba Population Health Research Repository (1) contains records of all physician contacts in hospitals and in offices for the total population of Manitoba.A comprehensive population registry facilitates record linkage using a scrambled personal health identification number (PHIN), unique for each individual.To maintain the anonymity and confidentiality of survey participants, Manitoba Health used the survey identifiers of name, birth date and address to find the PHIN for 2479 of the survey participants (83.2%).Manitoba Health removed personal identifiers from the survey file and scrambled the PHIN to permit linkage to the Research Repository.
The day of completing the questionnaire was considered to be the reference day for determining the respondent's age and the five-year interval before the questionnaire.For this period, divided into one-year segments, all physician contacts for the following International Classification of Diseases, ninth revision (ICD-9) (6) codes were identified: asthma (ICD-9 code 493), bronchitis not otherwise specified (ICD-9 code 490), chronic bronchitis (ICD-9 code 491), emphysema (ICD-9 code 492) and chronic airflow obstruction (ICD-9 code 496).

Statistical analysis
Kappa coefficients (7,8) were calculated to determine the agreement between survey responses and physician contacts documented in the database for periods from one to five years before the survey.Sensitivity and specificity of the administrative data were estimated considering response to the questionnaire as the gold standard.
The study was approved by the Ethics Board of the University of Manitoba and the Access and Confidentiality Committee of Manitoba Health (Winnipeg, Manitoba).

RESULTS
Of the 2479 subjects with complete information from both the survey and the administrative data, 169 (6.8%) reported an asthma attack in the previous 12 months and 172 (6.9%) were on asthma medication at that time.Two hundred fourteen (8.6%) reported either having an asthma attack or taking asthma medication, and 127 (5.1%) reported both having an asthma attack and taking asthma medication.A sample of those subjects using medication reported that they took inhaled bronchodilators (87.5%) and anti-inflammatory drugs (34.4%).The vast majority of those who had had an asthma attack or who were using medication also reported that their asthma had been confirmed in the past by a physician.This percentage was 87.0% for those reporting an asthma attack, 81.4% for those reporting the use of medication, 79.0% for those reporting either and 92.1% for those reporting both.One hundred subjects (4.0%) had a physician claim for asthma in the year before the survey.This number increased to 131 (5.3%) for the two years before, 154 (6.2%) for the three years before, 169 (6.8%) for the four years before and 187 (7.5%) for the five years before the survey.
Table 1 cross-tabulates responses from the questionnaire with physician claims.The agreement, indicated by kappa coefficients, between physician visits for asthma during one year before the survey and responses to survey questions about asthma attacks and the use of medication was moderate (κ=0.45 to 0.50) (9).Agreement improved substantially if two years of physician contacts were compared with questionnaire data (κ=0.60);considering additional years produced little further improvement.The kappa coefficients were similar for asthma attacks and use of medication, and for those reporting either or both of these (Table 1).
Table 1 also shows sensitivity and specificity of the administrative data in identifying those who had had an asthma attack in the previous 12 months or who were taking asthma medication at that time.The poor sensitivity for physician contacts within one year before completing the questionnaire (37.9% for asthma attacks and 41.3% for the use of medication) improved as additional years of physician contacts were considered.Approximately two-thirds of those patients who had suffered from an asthma attack in the previous 12 months or who were taking asthma medication had a record of seeing a physician for asthma within five years before completing the questionnaire.On the other hand, specificity was over 98% for the first year and decreased slightly with time.
Two groups of subjects were responsible for a less than perfect agreement between the questionnaire and physician claims data.The first group were subjects seen by a physician for asthma (according to physician claims) who did not report an asthma attack in the previous 12 months or asthma medication use.Table 1 shows that 98 of 100 such subjects had had some asthma symptoms in the preceding 12 months, although only 64 and 71 subjects reported an asthma attack or the use of medication, respectively.Thus, a physician visit for asthma seemed to be justified by the presence of asthma symptoms.Even those who had visited a physician two to five years before the survey was taken had a more than 90% probability of asthma symptoms on survey.Of those who had not visited a physician for asthma, approximately one-half had experienced some asthma symptoms.The sensitivity of physician claims for identifying subjects with any asthma symptoms was very poor (7.8%).On the other hand, the absence of asthma symptoms on the questionnaire almost excluded the probability of being seen by a physician for asthma.
The second group responsible for a less than perfect agreement between the questionnaire and physician claims data were subjects who had had an asthma attack or were taking medication in the previous 12 months but were not seen by a physician for asthma.These subjects were checked to see whether they were seen by a physician for a respiratory diagnosis that may overlap with asthma: bronchitis not otherwise specified, chronic bronchitis, emphysema or chronic airflow obstruction.Table 2 shows that the percentage of subjects who had had an asthma attack and/or who were taking asthma medication, and who were seen by a physician for asthma increased as additional years of physician contact were considered.In addition, approximately 10% to 15% of these subjects were seen for other respiratory diagnoses.Thus, over the five years before the survey, approximately 80% of those reporting asthma attacks or the use of asthma medication were seen by a physician for asthma or an overlapping diagnosis.Among 2479 subjects in the analysis, 1031 (41.6%) were men and 1448 (58.4%) were women.The agreement between responses to the questionnaire and physician claims was similar for both sexes.Subjects were divided into current smokers (n=756, 30.5%), ex-smokers (n=671, 27.1%) and never-smokers (n=1051, 42.4%).Again, the agreement did not differ significantly by smoking groups.Excluding those whose asthma had not been confirmed by a physician did not improve either the kappa coefficient or sensitivity.
Table 3 compares asthma prevalence estimated by questionnaire with that obtained using physician contacts one to five years before the survey.Using physician contacts one year before the survey underestimated the prevalence of asthma determined by the questionnaire.Including additional years of physician claims improved the agreement between the two estimates of asthma prevalence.However, the kappa coefficients of Table 1 did not improve as longer time periods were used.Extending the time considered in the administrative data did not pick up the same subjects identified in the survey.

DISCUSSION
Overall, the level of agreement between questionnaire responses and physician claims data is moderate (9).In comparison with studying one year, studying two years of database contacts increased the agreement with survey results.Studying more than two years of physician contact had no effect on the level of agreement, although the sensitivity of the database to identify people with positive responses on the questionnaire was higher if longer periods of time of physician contacts were considered.However, even if five years of physician contacts were studied, the claims identified only 63.3% of people reporting an asthma attack in the 12 months before the survey, 65.1% of people reporting being on medication, 62.0% of those reporting either an attack or using medication, and 70.1% of those reporting both an attack and using medication.Sex and smoking did not significantly affect the kappa coefficient.
A similar comparison has been made between data from a survey of factors related to cardiovascular diseases and  three years of physician claims from the same administrative database (2,10).The best agreement was for diabetes (κ=0.72), and the second best agreement was for hypertension (κ=0.59), a result similar to ours.The level of agreement we found for asthma was considerably better than for other cardiovascular conditions, including stroke and myocardial infarction.
Our results are unlikely to have been affected by the inability to link more than 83.2% of participants in the survey to physician claims.Linked and nonlinked individuals did not differ with respect to sex, age or the frequency of symptoms reported.The incomplete linkage may be due to strict requirements for matching and to the fact that the study population was relatively young (20 to 44 years) and mobile; different addresses may well have been given in the survey and in the administrative data.Almost all subjects (94.7%) in the analysis were covered by the Manitoba Health Insurance Plan throughout the five-year period.Restricting the analysis to individuals covered over the full five years would probably have improved the agreement only slightly.
Several reasons may underlie the only moderate agreement found in the present study.First, the survey and the administrative data record different aspects of asthma.The physician claims note whether the subjects went to see a physician for asthma in the previous 12 months, while the survey asked if the subjects had experienced symptoms and/or were using asthma medications.People with asthma attacks or using medications may be expected to see a physician at least annually.However, in most instances, the asthma was not severe.A sample of survey respondents (n=518) were questioned in greater detail about their disease when examined in the laboratory.Only 55.0% of participants who had had an asthma attack or who were using medication in the previous 12 months reported visiting a doctor because of a breathing problem or shortness of breath in the preceding 12 months.Apparently, a substantial fraction of people who reported symptoms suggestive of asthma did not feel sick enough to seek medical care.
Asthma is difficult to define; physicians may use ICD-9 codes for nonspecified bronchitis, chronic bronchitis, emphysema or chronic airflow obstruction instead of asthma when seeing a patient (11).Further, 20% to 50% of those reporting asthma attacks or medication use who had not visited a physician for asthma in the five years before the survey (false-negative) did see a physician for another, potentially overlapping respiratory diagnosis (bronchitis, chronic airflow obstruction).Still, approximately 20% of subjects reporting asthma attacks or medication use did not visit a physician for any respiratory diagnosis within the five years before the survey.
In the absence of a generally accepted definition of asthma (12), physicians appear to have problems diagnosing the disease and explaining it to their patients.Patients with similar symptoms can be labelled as asthmatic or can be given another respiratory diagnosis.Some patients labelled as asthmatic at a physician contact were apparently not aware of it.Not all patients with symptoms go to see a physician, particularly if their asthma is not severe.Although the vast majority of people with asthma attacks or medication use in the previous year reported confirmation of their asthma by a physician during their lifetime, many had not seen a physician for a respiratory problem in the previous five years.Some contacts with physicians for asthma were probably lost, because only one diagnosis could be listed as the reason for a physician visit.Patients may have seen their physicians for other health problems; their asthma could have been diagnosed and even treated with medication without appearing on the claim for payment.We do not know how often this happened, but in the age group studied, comorbidities were likely relatively few.
Thus, the estimates of prevalence derived from the survey and the administrative data include two overlapping groups of people.In each, the diagnosis of asthma seems justifiable, but the agreement between the survey and the data is only moderate to substantial.To put this in perspective, a number of clinical conditions show measures of diagnostic agreement (kappas) between two physicians of about 0.6 (13).Moreover, the 'physician'-diagnosed asthma prevalence underestimates prevalence estimates based on reported asthma symptoms (Table 3).On the other hand, both estimates of prevalence are useful, although for different purposes.Health care utilization estimates obtained from administrative databases are particularly useful for studying trends over time (10,14), an objective very difficult to achieve with survey methodology.The need for information on the prevalence of common conditions has never been greater, as Regional Health Authorities in Canada, and probably elsewhere, assume planning and budgetary responsibilities for health care.
Because the study population was restricted to the ages of 20 to 44 years, conclusions may not be generalizable to those younger or older.The diagnosis of asthma may be more difficult and controversial in other age groups.Because of this, there may be, although not necessarily, less agreement between records in administrative databases and self-reports by surveyed subjects in these age groups.

TABLE 1 Agreement between responses to the questionnaire and the database Responses to the questionnaire Physician Asthma attack Use of medication Attack or medication Attack and medication Any asthma symptom
*Asthma attack, current use of medication, wheezing, being awakened by tightness in chest, attacks of shortness of breath or coughing

TABLE 2 Comparison of responses to the questionnaire with physician's claim for asthma or nonasthma respiratory cause (resp m) Physican Responses to the questionnaire contact within Asthma attack Use of medication Attack or medication Attack and medication Any asthma symptom*
Asthma attack, current use of medication, wheezing, being awakened by tightness in chest, attacks of shortness of breath or coughing; *† Bronchitis not otherwise specified (International Classification of Diseases, ninth revision [ICD-9] code 490), chronic bronchitis (ICD-9 code 491), emphysema (ICD-9 code 492), chronic airflow obstruction (ICD-9 code 496); ‡ No physician claim for either asthma or resp m