A bibliometric analysis of digestive health research in Canada

1Research Excellence Metrics Ltd, Burlington; 2Department of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario Correspondence: Désirée Tuitt, Research Excellence Metrics, 2000 Appleby Line, Suite 226, Burlington, Ontario L7L 7H7. E-mail info@researchexcellencemetrics.com Received for publication May 31, 2011. Accepted September 14, 2011 With dwindling availability of funding for medical research, there has been increased scrutiny of the productivity of individual researchers as part of the evaluation of their grant applications. Simply counting the number of publications from a researcher as the primary measure of productivity has given way to other metrics (1), particularly since the 1970s, when records of the number of times that research articles were cited started to be enumerated. Fersht (2) argued that the number of times a paper is cited is the best metric of its quality. Journals are also assessed on the basis of citation data. The ‘journal impact factor’ is the most commonly used metric, representing the average number of citations in a year for the papers published in the journal during the previous two-year period (2). However, the journal impact factor does not provide information about the impact of individual papers, or about the quality of research performed by authors of those papers. Indeed, journal impact factors have been used – inappropriately – to judge the quality of individual researchers. In recent years, a new metric has been developed that focuses on the ‘influence’ of a journal, rather than its ‘impact’. The Eigenfactor uses a system to rank journals that is very similar to that used by Google (Google, USA) to rank the influence of websites (2). With respect to the rating of individual researchers, enumeration of the number of times that a researcher’s articles are cited is regarded as one of the best indexes of impact that a researcher has in their particular field (2,3). Weaknesses of this approach include the fact that a middle author on a paper would receive the same ‘score’ as the first or last author, whereas the contribution of the former would generally be considered to have been less than the latter two (particularly in the case of publications with many authors). The ‘Hirsch Factor’ (also referred to as the ‘H factor’) is a score developed by Jorge Hirsch (4) to provide an index of a researcher’s impact on their field (1). A researcher’s Hirsch Factor is the greatest number of publications they have that have each been cited at least that number of times. Thus, a researcher with 10 publications each with 10 or more citations would have a Hirsch factor of 10. A researcher with 50 publications cited 50 or more times each would have an Hirsch factor of 50. Hirsch opined that a physicist who had held an academic position for 20 years and had an Hirsch factor of 20 or greater would be considered a ‘success’, while a Hirsch Factor of 40 would be indicative of “outstanding scientists likely to be found only at the major research laboratories” (cited in reference 1). These thresholds would be different for different disciplines (most likely higher for biomedical sciences) (1). One of the weaknesses of the Hirsch factor metric is that a researcher can have a good Hirsch factor, such as 20, but relatively few citations (ie, as few as 400), while another researcher could have the same Hirsch factor with thousands of citations. Thus, with the same impact factor, the second researcher would likely have had much more impact on their field than the former. Recently, we adapted the evaluation of citations to include some additional indexes of productivity to yield a more informative assessment of the ‘influence’ of a researcher in their field. This includes incorporation of the Hirsch factor, the total number of citations, and the number of publications with more than 100 citations to create an ‘Influence factor’. originAl ArtiCle

Measurement of the impact and influence of medical/scientific journals, and of individual researchers has become more widely practiced in recent decades.This is driven, in part, by the increased availability of data regarding citations of research articles, and by increased competition for research funding.Digestive disease research has been identified as a particularly strong discipline in Canada.The authors collected quantitative data on the impact and influence of Canadian digestive health research.The present study involved an analysis of the research impact (Hirsch factor) and research influence (Influence factor) of 106 digestive health researchers in Canada.Rankings of the top 25 researchers on the basis of the two metrics were dominated by the larger research groups at the University of Toronto (Toronto, Ontario), McMaster University (Hamilton, Ontario), and the Universities of Calgary (Calgary, Alberta) and Alberta (Edmonton, Alberta), but with representation by other research groups at the Universities of Manitoba (Winnipeg, Manitoba), Western Ontario (London, Ontario) and McGill University (Montreal, Quebec).Female and male researchers had similar scores for the two metrics, as did basic scientists versus clinical investigators.Strategic recruitment, particularly of established investigators, can have a major impact on the ranking of research groups.Comparing these metrics over different time frames can provide insights into the vulnerabilities and strengths of research groups.Key Words: Canada; Digestive disease; Digestive health; Impact; Influence L'analyse bibliométrique de la recherche en santé digestive au Canada La mesure de l'impact et de l'influence des revues médicales et scientifiques ainsi que des chercheurs eux-mêmes s'est généralisée ces dernières décennies.Ce phénomène est motivé en partie par l'accessibilité croissante aux données liées aux citations d'articles de recherche et par une concurrence accrue à l'égard du financement de la recherche.La recherche sur les maladies digestives est perçue comme une discipline particulièrement bien implantée au Canada.Les auteurs ont rassemblé des données quantitatives sur l'impact et l'influence de la recherche canadienne sur les maladies digestives.La présente étude a comporté une analyse de l'impact (facteur de Hirsch) et de l'influence (facteur d'influence) des recherches de 106 chercheurs en maladies digestives au Canada.Les 25 premiers chercheurs, classés selon deux mesures, étaient dominés par les gros groupes de recherche universitaire, provenant de l'université de Toronto (de Toronto, en Ontario), de l'université McMaster (de Hamilton, en Ontario), de l'université de Calgary (de Calgary, en Alberta) et de l'université de l'Alberta (d'Edmonton, en Alberta), mais comprenaient également d'autres groupes de recherche de l'université du Manitoba (de Winnipeg, au Manitoba), de l'université Western Ontario (de London, en Ontario) et de l'Université McMaster (de Montréal, au Québec).Les chercheurs de sexe féminin et masculin obtenaient des indices similaires à l'égard des deux mesures, tout comme les chercheurs fondamentaux et les chercheurs cliniques.Un recrutement stratégique, surtout de chercheurs établis, peut avoir un effet marqué sur le classement des groupes de recherche.La comparaison de ces mesures à différents moments peut donner un aperçu des vulnérabilités et des forces des groupes de recherche.

Timeframe
The analysis was initially performed using articles published from 1976 until the end of 2010 (the full database of ISI Web of Science).For each researcher the Hirsch Factor (H), total number of citations (C), and the number of publications with more than 100 citations (P) were determined.The Influence factor (IF) was then calculated for each researcher based on these data, using the following formula: Because more junior investigators would be at a significant disadvantage in this analysis (simply because they have had fewer years to generate publications, and fewer years for those publications to be cited), a second analysis was performed using only the articles that were published in the years 2000 through 2010 (referred to as the 'recent publications' database).Most of the analyses presented hereafter focus on the recent publications database; the data it provides are a more accurate representation of the current performance of research groups than the data generated from the full database.

ResuLTs
Top-ranked researchers Rankings were created based on the analysis of the Hirsch factor and Influence factor for the 106 digestive health researchers.Table 1 shows the top 25 ranked researchers for the two metrics using the 'full publications' database.On this and other data tables, the identity of individual researchers is not provided; instead, they are represented by letters (A, B, C, etc.).While there are similarities in the rankings between the two metrics (23 researcher's names appear in the top 25 rankings for both the Hirsch and Influence factors), there are also some notable differences.Influence factor is calculated using the Hirsch factor along with two other measures of research influence: total number of citations and number of publications with greater than 100 citations.These added elements can differentiate researchers that are not differentiated by Hirsch factor alone.For example, in Table 1, 'researcher N' and 'researcher O' have the same impact factor (41), but the Influence factor for 'researcher N' is 19 points greater than that of 'researcher O' because of a greater number of total citations and of publications with more than 100 citations each.
Table 2 shows the rankings created using the two metrics with the 'recent publications' database.There are significant changes in the top 25 from those shown in Table 1: seven different researchers in the Hirsch factor rankings and seven different researchers in the Influence factor rankings.
Focusing on the 'recent publications' database and the Influence factor analysis, one can see that the larger research groups dominate the top rankings (13 researchers from the University of Toronto and three each from McMaster, the Universities of Calgary and Alberta), with the remaining three positions divided among McGill University and the Universites of Western Ontario and Manitoba.

sex and training
Female researchers comprised approximately 22% of the researchers in the analysis, and 20% to 28% of the top 25 rankings (depending on which timeframe and metric that was examined).It is noteworthy that the University of Toronto dominates when it comes to female researchers that place in the top 25 rankings, accounting for one-half of that university's representation in the top 25, and consistent with female researchers representing half of the total number of Toronto researchers.Indeed, when female researchers were separately ranked for Influence factor, Toronto researchers occupied nine of the top 10 positions.Female researchers comprised 33% of the basic scientists in the database, but only 16% of the clinical investigators in the database.
Whether a researcher was trained as a basic scientist or a clinical investigator did not appear to be a substantive factor in terms of how well they fared in these rankings.For example, for the recent publications database and Influence factor metric, basic scientists occupy nine of the top 25 rankings (36%), which is slightly less than their percentage representation (39%) in the full database of 106 researchers.
When one examines the rankings of all 106 researchers in the database (focusing on the 2000 to 2010 timeframe), no sex-related differences in Hirsch factor (mean of 16.9 versus 18.1 for female versus male, respectively) or Influence factor (mean of 20.8 versus 21.8 for female versus male, respectively) were evident.Clinical investigators have slightly higher Hirsch factors (mean of 18.8 versus 16.5) and Influence factors (mean of 22.7 versus 19.6) than basic scientists.Figure 2 summarizes a similar analysis, but focused on 'recent publications' (2000 to 2010).While somewhat similar to the results presented in Figure 1, there are some significant shifts when the analysis is focused on the more recent publication/citation activity, which gives a more accurate representation of the current influence and impact of each researcher and research group.McGill University joins several other groups in having mean Hirsch and Influence factors above the mean of the entire database of 106 researchers.For both metrics, McMaster University and the Universities of Manitoba, Toronto and Western Ontario are also above average.Particularly noteworthy are the scores for Toronto.A majority of the Toronto researchers rank well above average.Also noteworthy is the relatively low rankings for both metrics for the groups from Queen's University and the University of Sherbrooke.

shifting impact and influence
To further investigate the shifts in impact and influence of research groups from the full timeframe database to the recent publications database, the ratios (recent:full) for each researcher between the two metrics for the two databases was examined.A high ratio would be seen for a researcher who has been much more productive/influential in the past decade than in the period preceding the past decade.This would obviously be the case for all researchers who did not publish before 2000 (there were 10 such researchers with a ratio of 1).On the other hand, researchers who have reduced their research activity in the past decade would have a lower ratio (eg, some senior researchers who took on major administrative duties had ratios of less than 0.25).The mean ratios for each group were then calculated and are summarized in Table 3.The rankings only differed slightly for the two metrics.The groups with the highest ratios -indicating higher impact/influence of publications in the past decade versus the previous period -also tended to be smaller groups (British Columbia, McGill and Manitoba [and perhaps could be regarded as 'emerging' or 'maturing' groups]).Larger groups such as Toronto, Calgary, McMaster and Queen's, who may be replacing older faculty on a more regular basis than the smaller groups, dominated the next level in the rankings.Toward the lower end of the rankings were groups in which recruitment has been very limited (in number and/or restricted to junior researchers) and/or dominated by senior researchers whose research output has declined significantly since 2000 versus the pre-2000 period.

DIsCussIoN
The present study provides an overview of the digestive health research community in Canada, evaluated for the impact and influence of their research.The study was focused on researchers who devote a substantial portion of their research efforts to digestive health research.The results demonstrate that research excellence, as defined by the impact and influence metrics that were used, is concentrated to some extent in some of the larger research groups (eg, Toronto and McMaster) but is also represented in many centres across the country, irrespective of the size of the research groups.Bibliometric analysis of Canadian digestive health research has not previously been reported, although Canada has been identified as a leader in this field of research (6).Indeed, there appears to be only one previous published analysis of a research discipline in Canada: an evaluation of research publications by Canadian nurses in 2010 (3).In another study, research output related to primary care was compared for several countries, including Canada (7), and several discipline-specific bibliometric studies of other countries have been published (8)(9)(10).
The database of 108 Canadian digestive health researchers included 41 basic scientists.There were negligible differences between the basic scientist and the clinical investigators in terms of mean scores for impact and influence.However, it should be noted that, other than excluding researchers who were in their first three years of a university appointment, no attempt was made to determine whether the level of seniority of basic scientists versus clinical investigators was similar.The same is true in terms of our examination of impact and influence of the two sexes: no differences were observed.Female researchers were represented in the top rankings at a rate proportional to the percentage representation in the complete database.One notable observation with respect to sex was the high representation of female researchers at the University of Toronto (13 of a total of 26 researchers), and their strong performance relative to female researchers across the country (nine of the top 10 female researchers in terms of Influence factor were from the Toronto group).It was also notable that female researchers were approximately twice as prevalent in the basic science pool as in the clinical investigator pool.
The analysis of the highest-ranking researchers also highlighted the impact that recruitment of established investigators can have on the overall ranking of a research group.Many of the top-ranked researchers had joined their current university within the past decade.Indeed, the strength of the female contingent of researchers at the University of Toronto can be partly attributed to effective strategic recruitment.Three of the very highly ranked female researchers were recruited to Toronto after they had established their research programs elsewhere.
Comparing the Hirsch factor and Influence factor for research groups using the recent versus full database of publications may provide insights to the direction a group is heading in terms of growth or decline.A low ratio in this analysis (Table 3) suggests that a group's impact or influence is declining, while the opposite is true for a high ratio.These data should be interpreted with greater caution when considering small research groups because a decrease in productivity of just one researcher can significantly shift the ratio lower.However, with the larger groups, one can see that groups with ratios much lower than the mean for the entire database are failing to remain competitive (in terms of research impact and influence) with the other groups.This could be reflective of more senior researchers in these groups reducing their research effort (such as to undertake administrative tasks).It could also be related to groups having a significant constituent of young investigators, who may not have yet reached their peak research output.In contrast, the larger groups with above-average ratios appear to have maintained their impact and influence through ongoing renewal of their faculty, bringing in new researchers (junior or established) as the impact/influence of more senior researchers wanes.
The present analysis of the Canadian digestive health research community does have limitations.For example, a limitation of all such analyses is that a publication with many authors gives 'credit', in the form of citations, to each of the authors equally.This can be particularly the case regarding large clinical trials, where there may be dozens of authors.We have attempted to mitigate this limitation by utilizing the Influence factor, which incorporates other measures besides the Hirsch factor.However, those measures are also based on citations.The present analysis used the ISI Web of Science as a source for citation data.Alternative databases are available, such as Google Scholar, but have been reported to be inconsistent in terms of accuracy and less often updated than Web of Science (11).
The current analysis is a 'snapshot' of this research community as of the end of 2010.The citation data are continuously changing and, therefore, the top 25 rankings will change with time.However, these data do provide important insights on Canadian digestive health research including the lack of marked differences in research performance across sexes or by basic versus clinical training.The data also suggest the importance, in terms of building and sustaining research groups, of ongoing recruitment, be it of new or established investigators.

Influence of recruitment of
established researchers A research group's representation in the top 25 ranking can be markedly influenced by the recruitment of established investigators.Six of the top 25-ranked researchers (by Influence factor; recent publications database [Table2]) were recruited to their research groups within the past decade: researchers A, W, X, Y, Z and CC.Of course, retention of top-performing researchers is also very important in terms of maintaining research strength.Impact and influence of Canadian research groupsFigure 1 provides an overview of Hirsch factors and Influence factors for Canadian digestive health researchers for publications between 1976 and 2010 (data for research groups with less than three members are not displayed in this figure).In these figures, the red lines represent the mean scores for the entire database.The results are quite consistent with the top 25 rankings, with two of the larger research groups (University of Toronto and McMaster University) and two of the smaller research groups (Western Ontario and Manitoba) having mean scores above the overall mean.

Figure 1 )Figure 2 )
Figure 1) Hirsch factor and Influence factor for Canadian digestive health researchers according to their university affiliations.Data were generated using all publications by the researchers between 1976 and 2010.Each dot represents a score for an individual researcher.The red line indicates the mean score for the entire group of 106 researchers.Horizontal black bars represent the mean for each university.Research groups with less than three members are not included in the figure

Table 1 Top-ranking Canadian digestive health researchers for Hirsch factor and Influence factor (publications from 1976 to 2010)
In cases in which multiple researchers have the same Hirsch factor or Influence factor, they are ranked based on the total number of citations

Table 2 Top-ranking Canadian digestive health researchers for Hirsch factor and Influence factor (publications from 2000 to 2010)
In cases in which more than one researcher has the same Hirsch factor or Influence factor, they are ranked based on the total number of citations

Table 3 Ratios of Hirsch factor and Influence factor for publications between 2000 and 2010 versus 1976 to 2010 for Canadian digestive health research groups
n refers to the number of researchers at each institution (only research groups with three or more members were included in the analysis).The ratio of Hirsch factor for publications between 2000 and 2010 to that for publications between 1976 and 2010 was calculated.Ratios were similarly calculated for Influence factor.A ratio of 0.500 indicates that one-half of a group's total Hirsch factor or Influence factor has been generated from publications between 2000 and 2010.For the entire data set, the average Hirsch factor ratio was 0.685, and the average Influence factor ratio was 0.629 *