A randomized survey of the preference of gastroenterologists for a Cochrane review versus a traditional narrative review

University of Western Ontario and London Health Sciences Centre, London, Ontario Correspondence and reprints: Dr John WD McDonald, Room 5OF12, University Campus, London Health Sciences Centre, London, Ontario N6A 5A5. Telephone 519-663-3550, fax 519-663-3232, e-mail john.mcdonald@lhsc.on.ca Received for publication June 18, 2001. Accepted September 24, 2001 JWD McDonald, J Mahon, K Zarnke, B Feagan, L Simms, W Tucker. A randomized survey of the preference of gastroenterologists for a Cochrane review versus a traditional narrative review. Can J Gastroenterol 2002;16(1):17-21.

voir page suivante C linicians frequently rely on reviews of primary research for guidance in making decisions about therapy.Review articles of health care interventions take two main forms -traditional narrative articles and systematic reviews.The merits of each have been discussed (1)(2)(3).Cook et al (4) pointed out that systematic reviews focus on the effects of a specific intervention, while narrative reviews can have broader purposes.Systematic reviews, including reviews published in The Cochrane Library (1996 to 2000, Update Software Ltd, Oxford, United Kingdom), attempt to avoid the weaknesses of traditional narrative reviews, such as biased recommendations and failure to consider all relevant evidence (5,6).Reviews published in The Cochrane Library are also maintained with regular updates.At least some systematic reviews appear to have had a powerful impact on medical practice (7,8).
Many clinicians continue to rely on traditional narrative reviews published in conventional journals, possibly due to a lack of familiarity with or availability of systematic reviews, including Cochrane reviews, or to a preference for selecting and consulting a narrative review written by an acknowledged expert and published in a conventional medical journal.The preference of clinicians between the two types of reviews has not been studied.Therefore, the responses of Canadian gastroenterologists to two reviews regarding the use of azathioprine for chronic Crohn's disease -a systematic Cochrane review (9) and a traditional narrative review (10) -were compared.A previous study (11) found that an unexpectedly small fraction of patients with chronic Crohn's disease had ever received azathioprine therapy, despite data that indicate its usefulness (12).The primary objective of the present study was to determine whether the systematic Cochrane review was perceived as being more useful than the narrative review for guiding a decision regarding therapy.The secondary objective was to determine whether identification of authorship and journal source influences the rating by physicians of the usefulness of a traditional review.

MATERIALS AND METHODS
Potential participants were practising gastroenterologists identified from the Canadian Association of Gastroenterology membership list.It is estimated that approxi-mately two-thirds of Canadian Gastroenterologists are members of this association.When a questionnaire was mailed in error to a person not in practice or was returned only partially completed, it was excluded from the analysis.The names of potentially eligible persons were listed alphabetically and numbered sequentially.By using a random number generator, these persons were divided into three groups and selected in a 1:1:1 ratio to receive: • the Cochrane review (not identified as a Cochrane review, and with authors concealed) of the use of azathioprine for induction of remission in Crohn's disease (9); • the traditional narrative review of therapy (including azathioprine) for inflammatory bowel disease (10), with the journal source and author concealed; or • the same narrative review but with journal source and authorship identified.
A letter of encouragement to participate was sent by the executive director of the Crohn's and Colitis Foundation of Canada a week before the mailing, but no incentives were offered to complete the questionnaire.Confidential coded questionnaires were mailed to 423 eligible persons (Tables 1,2).A follow-up mailing was sent to nonrespondents two months later.The allocation code was maintained in a computer accessed only by the individual who performed the mailings and later by the person who performed the statistical analysis, and was concealed from all other investigators until the primary analysis had been performed.
To identify a narrative review for comparison with the Cochrane review, a MEDLINE search restricted to English language publications for the years 1996 to the preparation

Surveys mailed 591
Duplicates, incorrect address, not in practice 168

Mailed to eligible recipients 423
Surveys returned 264 (62.4%) Surveys with outcome data date of the survey material in 1998 was conducted using the terms 'Crohn', 'drug therapy' and 'review'.The initial search yielded 56 items.However, perusal of the titles revealed that only 12 were relevant.The remaining titles were then examined to determine which was most likely to be selected by gastroenterologists seeking information in the 'real world'.Six of the 11 titles were eliminated because they had been published in journals that the investigators considered to be unlikely to be consulted by the gastroenterologists who were potential participants in the study.The remaining six reviews were examined; one was identified by consensus of the investigators as most likely to be selected by practising gastroenterologists because it was published in a highly respected, universally available journal and was written by an acknowledged authority.The extent to which it dealt with azathioprine was similar to or greater than that in any of the narrative reviews that were eliminated on the grounds of authorship or journal source.
The selected review also met the needs of the study because its length approximated that of the Cochrane review; considerable differences in length may have led to a difference in response rates for the two types of reviews or to biased responses.

Clinical scenario
The following clinical scenario was presented to participants as a focus for evaluation of the usefulness of the reviews.
A 32-year-old male patient has had Crohn's disease for the past four years.He has been treated with intermittent tapered courses of prednisone beginning at 40 mg daily.In the past year, he has been on prednisone approximately half the time and has felt well with this treatment, but his symptoms recur when the dose is reduced below 20 mg daily.

Outcomes
The primary outcome measure was the respondent's rating of the usefulness of the review material for guiding a decision about whether to recommend azathioprine for the patient described in the clinical scenario.The measurement tool was a 100 mm visual analogue scale (VAS) ranging from 0 (completely useless) to 100 (extremely useful).
Pretesting the VAS with a sample of 10 clinicians established the variance on the VAS response and that differences in responses of less than 10 mm would not be considered clinically important.Participants were asked to indicate whether the review was useful as a guide to therapy and whether they believed that azathioprine was indicated for the patient described in the case scenario by answering direct 'yes' or 'no' questions.

Masking
All recipients of reviews were advised in a covering letter that two types of review material were being compared but were not told that the comparison involved traditional nar-rative and Cochrane systematic reviews.The reviews were copied using a digital scanner, and then formatted and reprinted so that they were of similar appearance and length (Cochrane review approximately eight pages of text, four pages of references and five pages of tables; narrative review approximately seven pages of text, five pages of references and four pages of tables.The reviews prepared in this format are available on request.)

Statistical analysis
After accounting for nonresponders (assumed to be 50%), a sample of 378 clinicians was required to test the primary hypothesis at 80% power (two-tailed alpha error 5%) that there would be no more than a 10% difference in the perception of the usefulness of the reviews.Continuous data (VAS) were analyzed by ANOVA using SPSS (SPSS Inc, USA).The differences between mean values were compared in a post hoc analysis using Tukey's test for multiple comparisons.Dichotomous data were analyzed by χ 2 analysis.The possible influence of demographic characteristics of respondents was assessed using logistic regression analysis.

RESULTS
Surveys were initially mailed to all people on the membership list of the Canadian Association of Gastroenterology.However, a considerable number of people were listed more than once with more than one address, could not be located because of incorrect addresses, or were not in practice.The response rate and characteristics of respondents are shown in Tables 1 and 2, respectively.The overall response rate from eligible recipients was 62.4%.The individual response rates from recipients of each of the three types of reviews were very similar.Of the original 591 targeted names, responses were eventually received from 86 (32.6%) recipi- ents of the traditional 'concealed' review, 92 (33.7%) recipients of the traditional 'identified' review and 86 (32.6%) recipients of the Cochrane review.There was no evidence of bias between early (86% of total) and late responders when compared by ANOVA.The mean score for early respondents (mailing 1) was 59.6.The mean score for late responders was 56.0.ANOVA showed no statistically significant difference (F=1.019;P<0.314).Demographic data on the nonresponders were not obtained; therefore, the possibility of selection bias influencing the results cannot be excluded completely.However, the randomization process and the similarity of response rates from recipients of the three types of reviews are not consistent with a biased response.There were no significant differences among the groups with respect to variables that could have been associated with a preference, including age, practice setting and training in critical appraisal of the medical literature.
The rating of usefulness assigned by clinicians who received the Cochrane review was significantly higher than the rating assigned by those who received the narrative review in either format (Figure 1).The difference between mean ratings of the Cochrane review and the narrative review with authorship and journal source concealed was 21.3 mm (95% CI 14.5 to 28 mm) on the 100 mm VAS.The difference between mean ratings of the Cochrane review and the narrative review with authorship and journal source identified was 16.6 mm (95% CI 8.6 to 24.5 mm).Regression analysis did not show that the higher rating assigned by recipients of the Cochrane review was influenced by training in critical appraisal of medical literature, more recent graduation from medical school or practice location (community versus university based).
A significantly higher proportion of clinicians who received the Cochrane review considered it to be a useful guide to whether to use azathioprine therapy compared with the two groups who received the narrative review (Table 3).Concealment or identification of authors and journal source did not significantly affect the evaluation of the narrative review, although there was a trend toward both a higher rating for the identified review and a higher proportion of clinicians responding that the review was useful.There was no difference in the proportion of respondents who stated that azathioprine was indicated for the patient described in the clinical scenario (narrative review with author concealed 97.5%, traditional review with author identified 97.8%, Cochrane review 98.8%).

DISCUSSION
Our results show that a systematic Cochrane review was considered by Canadian gastroenterologists to be more useful than a traditional narrative review as a guide to specific therapeutic decision making.The latter is clearly the purpose of Cochrane reviews, and these reviews are focused and structured to provide this type of information.In our study, the question posed by the clinical scenario was sharply focused, and the Cochrane review by its very nature is also focused, while the narrative review selected for comparison is much broader in its scope.This difference is a source of potential bias toward the Cochrane review and may lead to questions about the fairness of the comparison.However, other characteristics of the narrative review, such as the perspective it may offer, the more familiar format or other considerations, could result in a bias in the opposite direction.Whatever biases exist in favour of a Cochrane or narrative review, the latter remains the established alternative to a Cochrane review.It is, therefore, reasonable that the narrative review be the benchmark against which to compare the value of a Cochrane review.We located a relevant narrative review that we thought was most likely to be consulted in the 'real world'.This comparison is analogous to a randomized trial comparing a new intervention with a traditional therapy for Crohn's disease.The Cochrane review may be considered to be analogous to a monoclonal antibody specifically engineered for possible use in inflammatory bowel disease, while the narrative  review may be analogous to conventional steroid therapy.
The narrative review we chose was judged most likely to be consulted by gastroenterologists.Accordingly, the study may be considered to be an appropriate comparison of a 'real life', widely used intervention with a new intervention.We specifically asked respondents to evaluate the usefulness of the reviews as a guide to therapeutic decision making and have no information on the value they would place on the narrative review for another purpose, such as maintaining a general fund of knowledge.Survey research using a questionnaire has inherent weaknesses.However, the response rate in our study was relatively high, and it appears that responders were broadly representative of gastroenterologists practising in Canada.The validity of the observed difference in rating of the reviews is strengthened by the study design.The probability of biased responses was reduced by the use of random allocation and by the provision of only a single type of review to each participant.Participants were unaware of the nature of the other type of review that was being evaluated.Care was taken to conceal the origin of the reviews, and they were formatted to ensure that they were comparable in appearance and length.
The difference between the ratings of the 'identified' and 'concealed' traditional narrative reviews showed a trend toward higher rating of the former, although this difference was not statistically significant.Whether identification of authorship and journal source introduces bias into the perception of utility requires a study larger than ours, which did not have adequate statistical power to identify or exclude differences as small as those observed.
Although the generalizability of our results requires further study, we believe that they may be widely applicable because the responses were not dependent on age, site of practice or training in critical appraisal of medical literature.
Over 97% of participants believed that azathioprine was indicated for the patient with chronic Crohn's disease described in the case scenario; this result contrasts with previous information concerning actual prescribing practices.When evaluating patients with chronic Crohn's disease for inclusion in a clinical trial of methotrexate therapy, we found that an unexpectedly small fraction of patients with chronic disease had ever received azathioprine (11).Although we did not address the question of whether Cochrane reviews have a greater power than narrative reviews to influence prescribing practices, this question is clearly relevant and should be explored in a future study.There is evidence that circulation of printed materials as a sole strategy has little power to influence prescribing practices (13).However, the availability to clinicians of complete, current and concise information is likely to be one critical component of achieving optimal prescribing practices.Perhaps Cochrane reviews used in conjunction with other strategies, such as presentations by opinion leaders (13), will prove to be effective.Our study indicates that Cochrane reviews may be preferred over conventional narrative material for this purpose.We believe that Cochrane reviews should be more widely available to practising clinicians, and that the availability and the strengths of these reviews should be more widely publicized.We have provided evidence that they are likely to be welcomed by clinicians seeking advice for clinical decision making and preferred over traditional narrative review materials for this purpose.

Figure 1 )
Figure 1) Rating of usefulness of reviews on 100 mm visual analogue scale.Higher scores indicate greater usefulness.Columns are mean values assigned by groups receiving the specified reviews.Error bars are one standard error of the mean