Abstract
Background In this retrospective study, we investigated the outcomes (funded/not funded) and factors related to the funding of resubmitted applications to the Canadian Institutes of Health Research (CIHR) Open Operating Grants Competition and Project Grant Competition between 2000 and 2022.
Method and Findings The primary outcome was the proportion of resubmissions and new applications that were funded. Using a random forest model, we explored the importance of variables related to the success of resubmissions. A higher proportion of resubmissions (∼23%) were funded compared to new submissions (∼12%). The most important variables related to resubmission success were the rank (%) and score (/5) given to the preceding (initial) application and the number of the number of CIHR-funded grants where the PI was a named team member.
Conclusion Resubmitting applications to the CIHR Project Grant Competition was beneficial, particularly for projects that were previously highly ranked and received high scores. These results may offer guidance for researchers who are deciding whether to resubmit rejected applications.
Introduction
Few biomedical grant applications are successful [1]. When an application is rejected, applicants are typically encouraged to revise and resubmit their applications [2,3], an expensive and time-consuming task [4,5]. Given the low chances of success, researchers may question whether it is worth resubmitting the grant application [3]. Here, we describe the outcomes of resubmitted applications to the Canadian Institutes of Health Research (CIHR) Open Grants competitions.
In the U.S., resubmitted applications to the National Institute of Health (NIH) programs are more successful than new applications [2,6], though success rates remain modest [2]. Researchers require clear actionable information to help them decide whether to resubmit a rejected grant application [7], without which they may face multiple rejections from the same competition [8]. For NIH competitions, quantitative results (e.g. scores, ranking) from the peer review of the rejected application are related to whether someone decides to resubmit [9,10] and the success of the resubmission [11]. There is little data for other programs and countries, and it is not clear whether data from the NIH can be generalized to other funding systems [12]. Other factors such as applicants’ sex/gender, their previous funding success, and reviewer and review panel characteristics may also be related to whether a grant is funded [13,14]. Some of these factors may be related to how CIHR project grants are assessed [14]. However, to date, the factors related to the success of resubmissions to the CIHR project grant competition have not been explored.
Biomedical science and its benefits to society depend on successful applications for public research funding [15]. Substantial resources are wasted on repeated unsuccessful applications [4]. To help researchers decide whether to resubmit their rejected grant application, this exploratory study analyzed two decades of resubmission data from the CIHR Open Operating Grant and Project Grant competitions. The aims were to i) identify success rates of resubmitted CIHR operating and project grant applications and ii) explore the factors related to resubmission success.
Method
This is a retrospective cohort study of observational data collected by CIHR between 2010 and 2022. The methods are reported in accordance with the relevant requirements of the Minimum Information about Clinical Artificial Intelligence Modeling (MI-CLAIM) checklist [16]. A completed checklist can be accessed online on the Open Science Framework (https://osf.io/pw45z/). The Behavioural Research Ethics Board at the University of British Columbia approved the study (H23-00719; March 31st, 2023).
Context
The CIHR Open Grants competition–which currently accounts for approximately $750 million of CIHR’s $1.3 billion annual funding budget–is open to independent researchers at any career stage who seek funding to support their proposed fundamental or applied health-related research. The Open Grants competition comprises both the Open Operating Grants and the Project Grant competitions, data from which were extracted for 2010 to 2015 and 2017 to 2022, respectively. Each Peer Review Committee ranks the applications it considers in a competition, and an approach intended to account for varying ways that grant applications are scored by the approximately 60 different Peer Review Committees in each competition. The application rank, and not its score, dictates funding decisions.
How CIHR handles resubmitted grant applications
An applicant may submit a previously unfunded application in a subsequent Open Grant competition round. There is currently no limit on the number of times an unsuccessful application can be resubmitted. Although applicants can provide a response to the previous application’s feedback, CIHR instructs peer review committee members to consider a resubmitted grant application as a new application (i.e. relative to all others in the current competition) and states that “…addressing previous reviews does NOT guarantee that the application will be better positioned to be funded…”. A resubmitted application can be reviewed by a different peer review committee than the previously unfunded application and committees do not have access to the previous version of the resubmitted application, though members are asked to read and evaluate the applicant’s response to the previous review. [17]
Study design and dataset
Applications submitted to the Open Grant competition were surveyed to identify those that were new applications, unsuccessful, and followed by resubmission to the same program by the same Principal Investigator (PI). Data were extracted from 50138 applications to the CIHR Open Operating Grant and Project Grant competitions. Applications submitted to the 2016 Open Grant competitions were not included in this study because different reviewing and adjudication methodologies were used at that time. For the random forest model (see Data analysis, below), a randomly selected subset (20% of the data) was held back for model testing (“Test dataset”), and the remaining 80% of data were used for model training (“Training dataset”).
Consent
Every researcher who submitted an application to the CIHR Project Grant Competition consented to the CIHR Policy on ‘Use of Personal Information’. CIHR maintained a record of all grant applications and the assessment records (including information from the Canadian Common Curriculum Vitae) of researchers who applied for funding. Our study included an objective of quality assurance and quality improvement of CIHR’s programs and thus fell under Article 2.5 of Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans.
Data analysis
We performed an exploratory analysis [18] to 1): Describe the proportion of new and resubmitted applications that were funded and 2) Identify the importance of variables related to the outcome (funded/not funded) of resubmissions using a random forest model. The primary outcome was the proportion of successful resubmitted and new grant applications (%).
Data wrangling
The PI’s sex was hypothesized to be a significant explanatory variable [13]. Applications where the PI had not self-identified their sex (n = 37) were excluded. Applications were identified as resubmissions if they included a response to a previous review and could be paired with a preceding unfunded new application submitted by the same PI. The unit of analysis was application pairs. Resubmissions that could not be paired with a previously unfunded application (n = 9310) were excluded from analyses.
Variable importance
We used a 10-fold cross-validated random forest model to identify the importance of candidate variables related to the outcome of resubmissions. Random forest is an ensemble learning model that combines multiple decision trees, makes few assumptions about distributions and inter-relationships between variables and can describe the relative importance of explanatory variables [19,20]. Plots showing the results of hyperparameter tuning are shown in the Supplementary Material, Figure S4. The parameter values selected maximized model sensitivity (true positive classification) and specificity (true negative classification) (mtry = 3, nodesize = 1, ntrees = 1000).
The relative importance of each explanatory variable was calculated using the mean decrease of Gini impurity (a method for assessing how effectively the explanatory variables split the data based on the primary outcome) aggregated across the folds. The primary aim of the analysis was explanation not prediction [21]. For completeness, the prediction accuracy of the model was established using the unseen (Test) dataset.
Explanatory variables
The secondary outcome was the Gini importance of each of the candidate explanatory variables related to resubmission success. Candidate explanatory variables were selected from the factors related to grant resubmissions identified in our recent scoping review [12], other work highlighting possible biases in grant peer review selection [13,14], and from factors related to resubmissions in other funding competitions [9]). Two variable categories were identified: i) characteristics of the applicant (the named Principal Investigator (PI)) and ii) factors related to peer review. Variables included in the final model were: self-identified applicant sex (male, female), previous total CIHR funding awarded in CAD$M to all projects where the PI was a named team member (‘PI $ Funding’), the number of CIHR-funded projects that contributed to ‘PI $ Funding’ at the time of application (‘PI # grants’), whether the applicant had chosen
English or French as the application language, whether the resubmitted application was reviewed by the same peer review committee (true, false), whether the resubmitted application was reviewed by one or more of the same peer reviewers within the peer review committee (true, false), and the score (‘Previous Score, /5’) and ranking (‘Previous % Rank’) of the preceding application.
Results
Proportion of successful applications
The dataset included 40791 applications, 26142 of which were new applications and 14649 of which were resubmissions. During the study period, 3034 (∼12%) new applications and 3415 (∼23%) resubmissions were funded. Figure 1 shows density plots and median values for the scores and rankings for both new and resubmitted applications.
Variables related to resubmission success
Table 1 shows descriptive statistics for the explanatory variables related to the Resubmission outcome (funded vs. unfunded) in both the Training and Test datasets. Relationships between candidate variables are shown in the supplementary material, Figures S1-S3. See ‘Explanatory variables’ above for a description of the variables entered into the model.
The Gini variable importance data (from the training dataset) is shown in Figure 2. The percent rank assigned during peer review to the previous submission was the most important explanatory variable, followed by the previously assigned score (Figure 3). Application language (English, French) was the least important feature of the model.
Prediction performance
The confusion matrix for the performance of the model in the Test dataset is shown in the Supplementary Material, Figure S5. The model classification accuracy was 0.76. As would be expected with such large class imbalances, the model was more accurate at predicting which resubmitted applications would be unsuccessful (specificity = 0.92) compared to which would be successful (sensitivity = 0.22).
Discussion
Our results add to the literature on biomedical grant funding and peer review in two ways. First, we show that resubmitted applications to the Canadian Institutes of Health Research Project Grant Competition were funded more often and ranked and scored higher than new applications. Second, we show that the peer review ranking of the previous application was the most important variable related to whether a resubmitted application was funded. Applicant sex, application language and peer review continuity were the least important. Applicants who are considering resubmitting an unsuccessful application should feel encouraged by improved peer review outcomes for resubmissions, and may wish to consider the ranking of their previous application when deciding whether to resubmit a grant application to the CIHR Project Grant competition.
Matching data from the U.S. NIH [2,6], here we found a greater proportion of resubmitted applications were funded than new applications, and on average, resubmissions received a higher rank and score. We suggest three potential drivers of improved outcomes: First, it is possible that those who chose to resubmit were those who could adequately respond to the peer review feedback. Second, because CIHR treats resubmissions as new applications and instructs reviewers to compare them only to their present cohort, those who resubmit may be those whose application was more likely to be funded anyway (i.e. the application may be good but not quite good enough compared to the previous cohort it was initially judged against). Finally, a proportion of applicants whose applications received unfavourable reviews may choose not to resubmit, increasing the proportion of resubmissions which are likely to be funded.
Unlike journal peer review, providing applicant feedback is a lower priority of grant peer review committees [1,22]. However, applicants are recommended to, and do, use feedback to help with grant resubmission [7,10,23]. High-quality feedback helps researchers decide whether to submit their application [7]; low-quality feedback can confuse applicants [24] and may lead to multiple unsuccessful resubmissions [8]. CIHR has clear guidance for reviewers to promote high-quality reviews [25], which may have contributed to the improved success of resubmissions seen here. Unlike journal peer review [26], there has been little scientific evaluation of the impact of grant peer review feedback quality (but see also Derrick et al. 2023 [7]) perhaps due to the continuing opacity of many grant peer review systems, despite increasing calls for transparency [27–29]. Given the substantial time, effort and financial costs to applicants and society of grant applications, revisions and resubmissions [5,8,30], more research is warranted on how reviewer feedback impacts the volume, quality and success of subsequent grant applications.
Here we show that the most important factor related to the outcome of a resubmitted CIHR Open Grant was the rank given to the previous application. While many applicants may have assumed that the result of the previous peer review was related to the binary outcome of a resubmitted application, this is direct evidence of the relationship. The importance of rank compared to score may surprise some applicants, especially given the relevance of previous scores for resubmissions in other funding systems [9–11]. For some applicants, knowledge of the relationship between the previous applications’ rank and resubmission success may help them make an informed decision about whether to resubmit a grant application. We found that self-reported applicant sex was not an important factor related to resubmission outcome, mirroring the results of recent re-examinations of gender/sex bias in grant peer review [31,32].
The number of CIHR-funded projects the PI had been awarded was the third most important factor, after previous application rank and score. This result could be interpreted in at least two ways. The first is that grant writing is a skill [33], and one might assume that researchers may become more skilled in responding to reviewer comments with experience, especially with successful grants (Guyer et al., 2021). An alternative interpretation is that the result reflects the ‘Matthew effect’ in grant funding wherein previous success begets future success. A small group of previously successful researchers, rewarded more often on that basis, would threaten the assumed meritocracy of research funding systems [35]. Our observational data do not allow us to disentangle these explanations, though we speculate that both may be true. Recent innovations in grant peer review including ‘funding lotteries’ [36] and double-blind peer review processes [37] have been designed to reduce inherent bias in grant selection and peer review. Future research should examine whether the relationship between past success and the outcome of resubmissions still exists in applications to competitions that use these mechanisms designed to reduce bias.
This is an exploratory cross-sectional study, which precludes causal inferences about the relationship between the applicant and peer review characteristics, and grant resubmission success. We echo previous calls for further examination of grant peer review systems, including randomized controlled trials, to examine the causal factors that influence funding success [29,38,39]. We were unable to study the influence of many oft-reported biases in grant systems. For example, racial disparity in grant peer review and awards is well documented. However, because these data were not routinely collected by CIHR during the timeframe under consideration, we were unable to include self-identified race or ethnicity as factors in this analysis.
Conclusion
Resubmitted applications to the CIHR Project Grant competition were, on average, funded more often and ranked higher than new submissions. The most important factor related to whether a resubmission was funded was the percent rank assigned to the previous unfunded application. Resubmission may be worthwhile, as long as the initial application was well reviewed and applicants can adequately respond to reviewer feedback. These data help increase the transparency of grant peer review and strengthen recent calls for increased scientific analysis of scientific funding systems [38–41].
Code and data availability
Upon publication, the notebook containing the analysis code will be available on the Open Science Framework (https://osf.io/pw45z/). The data used in this analysis are held by CIHR and are not publicly available due to privacy and legal restrictions. Researchers wishing to obtain access to these data need to contact the Vice-President of Research Programs-Operations at CIHR (christian.baron@cihr-irsc.gc.ca) to obtain approval to access de-identified data on operating grant funding program applications submitted between 2010 and 2022. Data for unfunded applications cannot be shared.
Competing Interests
Authors AM, MH, and RS are CIHR employees. At the time of submission, author KK was the scientific director of the CIHR Institute of Musculoskeletal Health and Arthritis (CIHR-IMHA).
Funding disclosure
This work was funded in part by the CIHR Research Operating Grant (Scientific Directors) held by Karim Khan.