Systematic reviews (SRs) that fail to search non-English databases may miss relevant studies and cause selection bias. The bias may be particularly severe in SRs of traditional Chinese medicine (TCM) as most randomized controlled trials (RCT) in TCM are published and accessible only in Chinese. In this study we investigated how often Chinese databases were not searched in SRs of TCM, how many trials were missed, and whether a bias may occur if Chinese databases were not searched. We searched 5 databases in English and 3 in Chinese for RCTs of Chinese herbal medicine for coronary artery disease and found that 96.64% (115/119) eligible studies could be identified only from Chinese databases. In a random sample of 80 Cochrane reviews on TCM, we found that Chinese databases were only searched in 43 or 53.75%, in which almost all the included studies were identified from Chinese databases. We also compared SRs of the same topic and found that they may draw a different conclusion if Chinese databases were not searched. In conclusion, an overwhelmingly high percentage of eligible trials on TCM could only be identified in Chinese databases. Reviewers in TCM are suggested to search Chinese databases to reduce potential selection bias.
Systematic reviews (SRs) provide authoritative, summary of evidence for informing medical decisions. A good SR should identify and include all studies regardless their place of conduct and year and language of publication that meet pre-defined eligibility criteria so as to reduce selection biases [
Selection bias is a major threat to the validity of SRs and often caused by language restriction in literature search as many review teams find it difficult to access non-English language databases and subsequently fail to include relevant studies. It is known that trials with statistical significant results are more likely to be published in English [
An analysis [
As many trials in traditional Chinese medicine (TCM) may be published only in Chinese and available in Chinese databases, a failure to search databases in Chinese may miss many relevant studies and cause biases in SRs. How often have SRs of TCM failed to search Chinese databases? How many trials could be missed if Chinese databases are not searched? How would the missed trials affect the overall combined effect of a SR? We conducted this study to shed light on these questions.
This study has three components. In this part, we estimated and compared the number of RCTs of Chinese herbal medicine (CHM) for treating coronary artery disease (CAD) identified from international databases with that identified in databases in Chinese.
Five major international databases (MEDLINE, EMBASE, CINAHL, CENTRAL, and AMED) and 3 major Chinese databases (Chinese Biomedical Database, Chinese Medical Current Contents and Taiwan Periodical Literature Database) were searched from their inception to July 2010 to identify RCTs of CHM for treating CAD. Regarding CHM, the search included general terms and phrases for TCM and CHM and specific names of propriety Chinese herbal medicines and individual herbs that are commonly used for treating CAD. We compiled an exhaustive list of relevant terms and phrases by carefully studying relevant texts such as systematic reviews, narrative reviews, RCTS, and textbooks and by consulting TCM practitioners in the relevant field. We also used highly sensitive search strategies for RCTs and for CAD and finally limited to human studies only. Our detailed search strategies were attached in Appendix 1 In the Supplementary Material available online at
A study was considered eligible if (1) it is a RCT in study design, (2) it used CHM as the tested treatment compared with no treatment or a placebo treatment and often as an add-on treatment on top of routine western medicine treatment given to patients in all comparison groups, and (3) participants are adults of any age or ethnic origin with a CAD including angina and myocardial infarction. CHM is defined as any Chinese herbal preparations that contain at least one herb that is included in the latest version of the Chinese Pharmacopeia [
Finally, we estimated and compared the overall efficacy of CHM in reducing the risk of angina in patients with unstable angina as there were trials available in both international and Chinese databases. A funnel plot was drawn to show how studies identified in international databases may differ from those in Chinese databases.
In this part, we identified and compared 2 independent SRs of RTCs that addressed the same clinical question, but one searched Chinese databases and the other did not. By “the same clinical question,” it means that the RCTs have the same PICO or are common in the test TCM therapy, patients, control treatment, and clinical outcomes. We compared the two reviews in the number of RCTs identified by sources, combined results, and conclusions to see how likely the result would differ if Chinese databases were not searched.
We randomly selected 80 systematic reviews on TCM from the Cochrane Database of Systematic Reviews (CDSR) and investigated how often Chinese databases were searched in systematic reviews of TCM and the ratio of RCTs identified between international and Chinese databases.
We searched the January Issue of the 2013 CDSR for SRs on TCM by using Chinese medicine, Chinese herbal medicine; and acupuncture. Eighty SRs were sampled at random from 625 Cochrane SRs which are likely to include trials that have at least one arm of TCM therapy such as herbal medicine, acupuncture, and other forms of TCM. Detailed search strategies are attached in Appendix 2. If a study can be found in international databases, it will be considered “identified from international databases” regardless whether or not it can be found in Chinese databases.
We estimated the number and percentage of (1) SRs that searched both international and Chinese databases, (2) SRs that included a Chinese author, (3) RCTs that were identified from Chinese databases, and (4) SRs that tended to support for an efficacy of TCM treatment.
Statistical Package for the Social Sciences 18.0 (SPSS18.0), STATA 11 and Review Manager 5 were used for statistical analyses.
Fifteen thousand eight hundred and sixty-six citations were retrieved from the electronic databases searched, with 10,856 from Chinese databases and 5,010 from international databases. After exclusion of duplicates, 12,666 citations were retained for further screening. Another 10,006 were considered irrelevant and excluded purely according to the title and abstract. Scrutiny of the full texts was conducted for the remaining 2,660 citations. At last, a total of 119 papers met the eligibility criteria and were included in this analysis. Details of the literature search and studies identified in each step are shown in Figure
Flow chart of literature search and study selection for systematic reviews of Chinese herbal medicine for treatment of coronary artery disease.
96.64% (115/119, 95% confidence interval
Twelve eligible trials were found that used CHM to treat angina patients for preventing further attacks of angina, with 1 identified from international databases [
Details of the results are shown in Figure
Meta-analysis of trials of traditional Chinese medicine for treatment of unstable angina to prevent reattack of angina, according to databases from which RCTs were identified.
Funnel plot of RCTs of traditional Chinese medicine for treatment of unstable angina according to databases from which the studies were identified (the solid dot represents the trial identified from international databases, and the black dots represent trials from Chinese databases).
Two independent SRs [
The review by McCarney and colleagues [
Comparison of 2 systematic reviews on acupuncture for chronic asthma.
McCarney et al., 2004 [ |
Yu et al., 2010 [ | |
---|---|---|
Journal | Cochrane Database of Systematic Reviews | Chinese Acupuncture and Moxibustion |
Time period for included studies | Till August 2008 | January 2000–October 2009 |
Chinese databases searched | 0 | 3 |
International databases searched | 3 | 2 |
Number of RCTs identified | 12 | 22 |
Number and % of RCTs identified from Chinese databases | 0 (0.00%) | 17 (77.27%) |
Total number of patients | 350 | 3058 |
Publication bias | Unable to assess due to a limited number of included studies on each outcome | No clear publication bias for outcomes including total improvement rate, FEV1, and PEFR, unable to assess publication bias on FVC due to a small number of included studies |
Main results | No statistically significant or clinically relevant effects were found for acupuncture when compared to sham acupuncture | The acupuncture group showed a greater total improvement rate and significantly improved PEFR, FVC, and FEV1/FVC as compared to its control |
Conclusion | No enough evidence for making any conclusions on the value of acupuncture in the treatment of asthma | Acupuncture can significantly improve the overall improvement, PEFR, FVC, and FEV1/FVC |
Of the 80 SRs of TCM trials randomly selected from the CDSR, 43 (53.75% with 95% CI = 42.82%~64.68%) searched Chinese databases and 37 (46.25%) did not. Of the 43 SRs that search Chinese database, 11 (25.58%) were on acupuncture, 2 (4.65%) were on massage, and 30 (69.77%) were on herbal medicine or other forms of TCM. Of the 37 SRs that did not searched Chinese databases, 6 (16.22%) were on acupuncture, 4 (10.81%) on massage and/or aromatherapy, and 27 (72.97%) on herbal medicine or other forms of TCM. No statistical difference was found in the types of treatment between SRs that searched Chinese database and those that did not (
The average number of RCTs included was similar in the SRs that searched Chinese databases and those that did not. Almost all (
Comparison of SRs in TCM that searched Chinese databases with those that did not.
Whether Chinese databases were searched |
|
||
---|---|---|---|
Yes | No | ||
Number of SRs | 43 | 37 | |
SRs that included Chinese authors | 41 (95.35%) | 4 (10.81%) | <0.001 |
Median number of included studies (range) | 8.00 (0–75) | 6.00 (0–36) | 0.58 |
Median number of studies from Chinese databases (range) | 8 (0–74) | 0 | <0.001 |
Number (%) of SRs tended to support for an efficacy of TCM treatment | 15 (34.88%) | 12 (32.43%) | 0.82 |
% of studies from Chinese databases (range) | 100.00% (0.00–100.00%) | 0 | <0.001 |
Comprehensive literature search is essential for a systematic review [
Language restriction is very common in systematic reviews although to various degrees. A review [
It is uncertain whether failure to search non-English databases would affect the overall result and conclusion. For example, a review [
It is very likely that the language bias is dependent on the topic of the review and that the direction of bias is unpredictable. For example, a review [
In addition, failure to include relevant studies in languages rather in English may not only bias the result but also reduce the statistical precision of the overall estimate and reduce the possibility of meaningful subgroup analyses which may result in important findings on factors that may affect the efficacy of treatment.
Some may argue that trials published in languages other than English was of methodological quality lower than those in English and therefore reviews do not need to include these low quality trials. This is a relevant argument against search of non-English databases but may not generally hold. For example, a review [
These findings suggest that relevant trials published in different languages should all be included in a systematic review. As discussed above, the relative quality of Chinese trials and English trials is not precisely predictable, although often trials in Chinese may have a lower quality than those in English. There is no good reason to believe that studies published in non-English languages are always of low quality. Thus, studies which are deemed low quality should be excluded only after they have been identified and assessed in their quality.
Our study also showed that 89.19% of the SRs that did not search Chinese databases did not have Chinese coauthors, suggesting that lack of Chinese collaborators is probably one of the major reasons why Chinese databases had not been searched in many SRs in TCM. Given the high percentage of trials which can be found by searching Chinese databases, an organized approach to searching Chinese databases would be highly desirable for those who wish to conduct SRs in TCM. Before such a service is created, we suggest reviewers for identifying Chinese colleagues to collaborate on TCM reviews, possibly through the contact of the Chinese Cochrane Centre, Hong Kong Cochrane Branch, and other active groups in conducting TCM reviews in China.
In conclusion, an overwhelmingly high percentage of eligible trials on TCM can be identified only through searching Chinese databases. However, some 50% of systematic reviews in TCM failed to search Chinese databases, which may lead to a bias of substantive degree and of unpredictable direction. In order to reduce this language bias, researchers are suggested to include Chinese speaking colleagues as collaborators from the very beginning if they wish to conduct reviews in TCM.
This paper was funded by the Chinese Medicine Department of Hong Kong Hospital Authority (CUHK Project Code: 7050098) and Hong Kong Cochrane Branch (CUHK Project Code: 6901965).