Evaluation of an internal review process for grants and manuscripts in the Canadian Critical Care Trials Group l’évaluation d’un processus de révision interne des subventions et des manuscrits au sein du Canadian Critical Care Trials Group

a highly favourable evaluation of existing internal review process. The present evaluation has assisted in understanding and improving the current internal review process.

for authors (one survey for authors of grants, and one survey for authors of manuscripts) and two 13-item questionnaires for reviewers (one survey for reviewers of grants, and one survey for reviewers of manuscripts).The present study was approved by the University of British Columbia/Providence Health Care Research Ethics Board (Vancouver, British Columbia).Two authors (PD, KB) generated and reduced items for possible inclusion in the questionnaires.Likert scale (8), multiple choice and free-text response formats were used.All surveys were published for completion on SurveyMonkey (Online Appendixes, go to www.pulsus.com).
Each pair of questionnaires (for grants and manuscripts) administered to authors (first round) and reviewers (second round) was pretested (9).Twelve CCCTG members (eight for the authors' questionnaire and four for the reviewers' questionnaire) assisted with questionnaire pretesting.

Questionnaire administration
Authors of grants (n=29) and manuscripts (n=22) that were submitted for internal review between January 2009 and September 2011 were first invited to complete a nine-item questionnaire to address authors' perspectives of the peer review process.Subsequently, reviewers of grants (n=47) and manuscripts (n=44) over the same time period were invited to complete a 13-item questionnaire to address reviewers' perspectives of the review process.If more than one grant or manuscript was submitted or reviewed by a given author or reviewer, they were asked to consider the review of all grants and/or manuscripts in their responses to each of the author or reviewer questionnaires.Both author and reviewer questionnaires were accompanied by a cover letter bearing separate links to the grant and manuscript questionnaires; these letters were sent by e-mail approximately six months apart.A single e-mail reminder was sent to authors and reviewers approximately one month after the initial invitation to complete the questionnaires.
Descriptive statistics and free-text comments were tabulated from the aggregate responses to all questionnaires.

Authors of grants and manuscripts
Twenty-one of 29 (72%) grant authors and 16 of 22 (73%) manuscript authors responded.Most respondents were somewhat or very satisfied with the turnaround time and quality of the review, and with the review process overall (Table 1).These internal reviews by members of the CCCTG replaced the need for institutional internal peer review, where required, for 33% of grant and 75% of manuscript authors.Twenty-eight percent and 48% of grant and manuscript authors, respectively, reported that new scientific issues not identified by the CCCTG review process were identified by reviewers for grant agencies or journals.Of 20 grant authors who completed a question about grant success, 13 (65%) acknowledged that one or more of the grants that were reviewed were funded.Of 16 manuscript authors who completed a parallel question pertaining to manuscript success, 11 (69%) stated that one or more of the manuscripts that were reviewed were accepted for publication.
Changes made to grant applications based on these reviews were predominantly editorial and involved the background, rationale, significance/relevance and the methods/protocol sections.Less frequent changes to grant applications included the budget/budget justification, knowledge translation/exchange, changes in organization, statistical analysis/sample size calculation and objectives but never involved description of the research team (Figure 1).Changes made to manuscripts based on these reviews were primarily editorial and most frequently addressed the discussion section.Less frequent changes to manuscripts included the introduction/rationale, results, methods/statistical analysis sections, and overall changes in organization (Figure 2).
In response to open-ended questions about what worked well and what could be improved about the review process, authors of grants and manuscripts praised the rapid turnaround time, experience/ expertise of the reviewers and thoroughness of the reviews.Some respondents also suggested pairing of a senior reviewer with a junior reviewer, and posing specific questions or having an explicit template for the review process (data not shown).

Reviewers of grants and manuscripts
Twenty-one of 47 (45%) grant reviewers and 32 of 44 (73%) manuscript reviewers responded.Approximately one-half (11 of 21 [52%]) of the grant reviewers reported reviewing one to three grants per year for other agencies.In contrast, most manuscript reviewers reported reviewing either nine to 12 (10 of 32 [31%]) or >12 (six of 32 [19%]) manuscripts per year for journals.Most respondents reported a good to

Figure 1) Changes made to grants by authors based on the internal reviews (percent of respondents). Responses to question, "Which changes did you make to your grant(s) based on the reviews you received? (Please check ALL that apply)"
Figure 2) Changes made to manuscripts by authors based on the internal reviews (percent of respondents).Responses to question, "Which changes did you make to your manuscript(s) based on the reviews you received?(Please check ALL that apply)" Data presented as % of respondents excellent overall impression of the grant and manuscript review process, good fit between their expertise/interests and the grants they were asked to review, and ample time to review grants and manuscripts (Table 2).Although most respondents agreed with the current 'open' or nonblinded review process, more grant than manuscript reviewers preferred a structured review format.Most reviewers would have liked to have seen their co-reviewers review and to have received constructive, written feedback pertaining to the quality of their reviews from the Grants and Manuscripts Committee.Of reviewers who responded to a question about preferences, most grant (10 of 15 [75%]) and nearly one-half of manuscript (13 of 27 [48%]) reviewers did not express a preference to review only grants and manuscripts pertaining to topics within their areas of interest and expertise.Most grant reviewers believed that their role was to enhance the clarity of key messages, improve or shape key scientific arguments, and make overarching comments regarding design and/or interpretation.In addition, some grant reviewers believed that their role was to frame the study question, to review the consistency of the data presented, and to aid in document organization and description of methodology.Only a few grant reviewers believed that their role was to provide additional references, optimize presentation of tables and figures, or wordsmith.Two additional grant reviewers provided other responses highlighting a sense of altruism in helping in whatever manner possible, and a desire to replicate the grant or manuscript review process in other contexts (Figure 3).
Similar to grant reviewers, most manuscript reviewers believed that their role was to enhance the clarity of key messages, improve/ shape the key scientific arguments, review the consistency of the data presented, and make overarching comments regarding design and/or interpretation.Some manuscript reviewers believed that their role was to help optimize the presentation of tables and figures, document organization and description of methodology.Only a few manuscript reviewers believed that their role was to aid in framing the study question, wordsmith or to provide additional references.Other responses emphasized altruism in the review process, and highlighted the importance of identifying grammatical and typographical errors, and editing (Figure 4).
In response to open-ended questions about what worked well and what could be improved about the review process, reviewers praised the collegial and timely process, but preferred a structured template for reviews that included specific questions to reviewers from authors.In addition, they recommended a 'gratitude' report to reviewers that indicated how many reviews individuals had performed (data not shown).

disCussion
The grant and manuscript review process of the CCCTG is highly regarded by authors and reviewers, is useful in modifying grants and manuscripts, and is associated with good success rates for authors.Most author respondents were somewhat or very satisfied with the turnaround time, quality of the review and the review process overall.Changes made to grant applications based on the review were predominantly editorial and involved the background, rationale, significance/relevance and the methods/protocol sections.Most reviewer respondents reported a good to excellent overall impression of the grant and manuscript review process, and a good fit between their expertise/interests and the grants they were asked to review.Most respondents agreed with the current 'open' or nonblinded review process.More grant reviewers than manuscript reviewers preferred a structured review format.Most reviewers would have liked to have seen their coreviewers' review and to have received constructive written feedback about the quality of their reviews.Grant reviewers reported that their role was to enhance the clarity of key messages, improve or shape key scientific arguments, and provide overarching comments regarding design and interpretation.Similarly, manuscript reviewers reported that their role was to enhance the clarity of key messages, shape the key scientific arguments, review the consistency of the data presented, and make overarching comments regarding design and interpretation.Opportunities for improvement include a more systematic and explicit process in which authors pose specific questions to the reviewers for their input, use of a structured review format, feedback on the reviewed documents and access to coreviewers' comments.The review of grants and manuscripts is performed extensively as part of the scientific paradigm; however, this process has several limitations.For example, manuscripts are more likely to be recommended for publication if they describe a positive outcome, compared with no difference between experimental groups (1).In addition, inter-rater reliability among reviewers is low (2,3).Use of reporting guidelines in the review of manuscripts is associated with improvement of manuscripts, but authors adhere more to suggestions from conventional reviews than to the reviews related to these guidelines (4).One approach to reducing inter-reviewer differences in review of grants is use of a panel score; however, a panel discussion does not improve the reliability of a panel score (6).Although there are different perspectives about the value of blinding a review -authors prefer unblinded reviews and reviewers prefer blinded reviews (7) -quality of reviews and manuscript scores are independent of blinding of the review (5,7).Alternatives to traditional peer review including ranking instead of reviewing, bidding to review instead of being asked to review, and peer evaluation through social media have been proposed as potential solutions to some of these limitations (10).
We are not aware of any published evaluations of an internal process such as the one used in the CCCTG.This process was developed to help authors develop and strengthen their grants and manuscripts and to improve the success rate in funding and publication, respectively.Although we have no data from the period that preceded this process, the strongly positive feedback that we received from this evaluation suggests that the process is working well.Nevertheless, we learned that the process could be improved by pairing junior and senior investigators, and content experts and methodologists as reviewers and by making the review process more systematic and explicit.In our survey, both authors and reviewers suggested that authors should pose specific questions to reviewers so that the reviewers can focus on what the authors need most.This improvement is similar to research studios (11) in which an investigator presents their proposal and specific questions to an in-person panel of experts.The studio process has also been rated very favourably (11).
Strengths of the present study include the explicit approach to survey development, inclusion of all authors and reviewers over a two-to three-year period as potential respondents, and high response rates.Limitations include the relatively small number of total possible respondents and that some respondents completed both questionnaires, as both authors and reviewers.Our findings may not be generalizable to other research consortia that have different organizational cultures.In addition, we did not use a previously developed review quality instrument (12) because we intentionally developed our survey items based on input from members of the CCCTG.Coincidentally, many of the items that we developed were similar to the items on the review quality instrument (12).

suMMARy
We report a highly favourable evaluation of an existing internal review process for grants and manuscripts.The responses and constructive feedback have enabled authors to pose questions to reviewers, fostered development of structured templates for future grant and manuscript reviews, and are expected to inform additional changes to the review process in the near future.

Figure 3 )Figure 4 )
Figure 3) Perceived roles according to grant reviewers (percent of respondents).Responses to question, "Which of the following do you believe are the roles of the Canadian Critical Care Trials Group grant reviewers (Please check ALL that apply)"