Evaluation of funding gastroenterology research in Canada illustrates the beneficial role of partnerships

An evaluation of the first 10 years of the CAG/CIHR/Partners Program, conducted in 2002, concluded that it was meeting the

objective of building capacity in gastroenterology research in Canada (1).The aim of the present updated evaluation, conducted in 2012 using data from 2000 to 2008, was to critically assess the contributions to research and training in gastroenterology in Canada generated through the CAG/CIHR/Partners Program.The specific objectives of the present evaluation were to measure the impact of funding on recipient career paths, and assess whether funding increased gastroenterology research output and impact in Canada.

MEtHodS
All recipients of Fellowship awards, Operating grants and Career Transition awards who received support from 2000 to 2008 were surveyed to assess the impact of the Program on their careers and to evaluate the impact of the Program on building capacity in gastroenterology research in Canada.The CAG prepared the survey in consultation with the CIHR Institute of Nutrition, Metabolism, and Diabetes (CIHR-INMD), and the CIHR Evaluation Unit.Using contact information for the grant and award recipients originating from the CAG membership database, the CAG circulated the survey to 160 recipients.Respondents predominantly completed and submitted their surveys electronically, although some written responses were also received.One hundred forty-seven (92%) of 160 recipients responded to the survey.
CIHR staff conducted a search of the CIHR Funding Decisions database to determine subsequent success in obtaining CIHR funding for grants and awards as a nominated principal investigator.To determine the impact of the supported research, the CAG and CIHR commissioned the Observatoire des sciences et des technologies (OST, Montreal, Quebec) to conduct a bibliometric analysis of publications identified by survey respondents as directly resulting from CAG/ CIHR/Partners funding.Of 693 publications submitted by survey respondents, 523 were included in this analysis.OST excluded 170 from the analysis because of incomplete data, duplication, lack of new scientific knowledge (eg, book reviews and editorial material) or noncoverage within the OST database.
OST conducted analyses to determine both the average of relative citations (ARC) and the average of relative impact factor (ARIF) (2-4).The ARC measures the frequency with which the research findings are cited by other researchers, while the ARIF indicates the prestige of the journal in which the research is published.OST conducted additional analysis on the five scientific subfields in which articles were most frequently published: gastroenterology, immunology, microbiology, pharmacology and physiology.
In addition to the ARC and ARIF, OST analyzed the rate of international collaboration among grant and award recipients.International collaboration is an indicator of the relative intensity of scientific collaboration between countries.An article was considered to be an international collaboration when it contained international addresses as well as Canadian addresses.

RESultS
From Participation by the 14 partners varied, with seven partners contributing to the Program for ≥8 years, and five partners contributing to the Program for one year.Collectively, the partners provided $8.7 million (50%) of Program funding between 2000 and 2008.Over the same period, CIHR invested $8.4 million (48%), through 1:1 matching funds.The remaining funds were provided by the CAG, universities or research institutes.In addition to the financial support of partners, substantial in-kind administrative contributions were made by both CIHR and the CAG, including recruitment and management of partners and program administration by the CAG, and peer review and grant administration by CIHR.
Although not specifically targeted by the Program, clinician scientists received strong support through the CAG/CIHR/Partners Program from 2000 to 2008.Data regarding professional designation was available in the CAG database for 158 of the 160 survey recipients.These data indicated that 56% of recipients held a PhD and the remaining 44% held an MD degree.
To assess whether the Program was successful in building research capacity, as defined by a continued career in a research environment, past recipients identified their current work position and whether they continued to perform research in their present role.Including current trainees, 76% of 147 survey respondents currently hold a research position (Figure 1A).Of funding recipients no longer working in a research environment, slightly more than 15% hold positions as clinical teachers in an academic setting or as clinicians in community practice.
Because Fellowship award recipients represent the group that is most vulnerable to leaving the research environment at the end of their award, additional data were generated to assess whether receipt of a Fellowship award resulted in a continued career in a research   1B).Of Fellowship award recipients not currently in a research position, 18% hold positions as clinical teachers or as clinicians in community practice.
To determine whether the Program succeeded in increasing gastroenterology research capacity in Canada, current work location of funding recipients were assessed.Of the 150 recipients whose current location was known through either their response to the survey or the CAG database, 71% are situated in Canada (Figure 2A).Of the recipients who responded to the survey, 54% are currently in Canada and hold a research position.Of the 122 Fellowship award recipients whose current location is known, 66% are based in Canada (Figure 2B).Another indication of the continuation of the recipients' research careers in Canada is their success in subsequently obtaining CIHR grants and awards funding.The 160 recipients surveyed subsequently held a total of 120 CIHR grants and awards.
To address the second objective of the present evaluation, the research output of the funding recipients was assessed.Survey respondents indicated that 545 research articles, 130 review articles, 35 book chapters and 11 patents resulted directly from Program funding (Figure 3).
To determine the impact of the research supported by the Program, a bibliometric analysis on the publications that survey respondents identified as resulting directly from Program funding was performed.ARC analysis showed that Program-supported articles received substantially more citations than the world average for articles published in the same five subfields of research.The overall ARC for Programsupported articles was 1.73, compared with 1.00 for the rest of the world and 1.18 for Canada (Figure 4A), indicating that Programsupported articles had greater scientific impact than other publications by peers both in Canada and around the world.
ARIF scores indicate that Program-supported researchers published in journals with higher impact compared with other Canadian researchers and their peers in the rest of the world (Figure 4B).
Analysis of author addresses on Program-supported publications revealed that approximately 42% of the publications were the product of international collaborations.This is consistent with the international collaboration rate of publications by Canadian researchers within the five subfields of research, and it indicates potential for increased dissemination and translation of research results.

diSCuSSion
The results of the present evaluation demonstrate that CAG/CIHR/ Partners Program funding has a significant positive impact on gastroenterology research in Canada and on recipient careers.CAG/CIHR/Partners Program funding recipients produced an impressive number of publications and patents.Bibliometric analysis indicates that publications by Program-funded researchers were cited significantly more often than publications by peers around the world and in Canada, and that Program-funded publications were significantly more likely to appear in prestigious journals than publications by peers around the world and in Canada.Together, these data indicate that Program-funded researchers consistently produce excellent research.It is important to note that Canada is home to some of the most impactful gastroenterology research in the world.According to The State of Science and Technology in Canada, 2012 report, the ARC rank for Canadian citations in the subfields of gastroenterology and hepatology was second in the world.While it is tempting to conclude that the high Canadian ARC ranking may be due, in part, to funding support from the CAG/CIHR/Partners Program, our methodology does not permit attribution.
Previous evaluations on three partnered capacity-building pro- A potential limitation of the present study was the possibility of error in the self-reporting of research outputs.Publications were reviewed to reduce the risk that they could be attributed to other sources of funding concurrently held by the respondents.However, there was no verification of self-reported patents or book chapters.It should also be noted that it was necessary to use self-reported publications to perform program-focused bibliometric analyses, which represents a departure from traditional bibliometric citation analyses.Finally, because the survey did not ask Fellowship award recipients to report on subsequent peer-reviewed funding, such as national health charities, other federal granting agencies and international organizations, such data were not available for the present analysis.
An important aspect of the success of this program has been the continued partnership between CIHR, the CAG, voluntary health organizations and partners in industry.The contributions made by all partners helped build a solid and productive platform of support for a talented pool of new investigators.This allowed for continued support of the gastroenterology research community across Canada, which has resulted in increased research capacity and the generation of new knowledge with potential to impact greatly on the gastrointestinal health of Canadians.
2000 to 2008, the CAG/CIHR/Partners Program was financially supported annually by CIHR through the Research and Development Program and CIHR-INMD, 13 industry partners and the Crohn's and Colitis Foundation of Canada (Box 1).These partnerships resulted in an investment of >$17.4 million to support 160 awards and grants, including 22 operating grants (including New Investigator grants), seven Career Transition awards, and 131 Fellowship awards between 2000 and 2008.Annual funding for the Program varied between $1.25 million to $2.6 million, with more than $13.8 million invested in Fellowship awards over the nine-year period.

Figure 1 )
Figure 1) a Of Canadian Association of Gastroenterology/Canadian Institiutes of Health Research/Partners Program funding recipients that received funding between 2000 and 2008, 76% continue to work in research positions (including trainee positions).B Seventy-two per cent of Fellowship Award recipients continued to work in a research position (including trainee positions)

Figure 2 )Figure 3 )Figure 4 )a
Figure 2) a Of Canadian Association of Gastroenterology/Canadian Institiutes of Health Research/Partners Program funding recipients that received funding between 2000 and 2008, 71% currently work in Canada.B Sixty-six per cent of Fellowship Award recipients currently work in Canada grams, the Kidney Research Scientist Core Education and National Training (KRESCENT) Program, the Focus on Stroke Program and the Children's Digestive Health and Nutrition Foundation (CDHNF) Young Investigator Development Awards, suggest that these programs positively impact researcher retention.The KRESCENT program, which is a partnership of CIHR-INMD, Kidney Foundation of Canada, and the Canadian Society of Nephrology, supports Fellowships and New Investigator awards.In 2010, an evaluation of the KRESCENT program reported that 88% of KRESCENT Fellowship recipients subsequently secured an assistant professor position (5).Focus on Stroke, which was led by the Heart and Stroke Foundation of Canada in partnership with CIHR and industry, supported doctoral research awards, postdoctoral Fellowship awards and New Investigator awards.A 2012 study of Focus on Stroke revealed that 69% of funding recipients continued to work at a university or college as a researcher, academic, scientist or trainee (personal communication, C Evans, Heart and Stroke Foundation of Canada).The CDHNF Young Investigator Development Awards, which are led by CDHNF, in partnership with the North American Society for Pediatric Gastroenterology, Hepatology and Nutrition (NASPGHAN) and industry, fund Young Investigator and Career Transition awards.A 2010 evaluation showed that up to 94% of CDHNF grant recipients continued to work in a research position (6).The CAG/CIHR/Partners Program evaluation similarly indicates strong capacity building in gastroenterology research, with 72% of the 119 fellowship recipients continuing to work in a research position.