Research funding: past performance is a stronger predictor of future scientific output than reviewer scores

Abstract

Scientific grants are awarded almost exclusively on the basis of an independent peer review of a proposal submitted by the principal investigator (PI). The writing and reviewing of these applications consumes a significant amount of researchers' time. Here, we perform a large-scale performance evaluation of review-based grant allocation via analysis of the grant proposals submitted to the Hungarian Scientific Research Fund. In total, 42,905 scored review reports prepared for 13,303 proposals submitted between 2006 and 2015 were analyzed. The publication and citation characteristics of the PIs were obtained from the Hungarian Scientific Work Archive (www.mtmt.hu). Each publication was assigned to its respective SCImago Journal Rank category, and only publications in the first quarter (Q1) were considered. Citation, H-index and publication data were derived for each analyzed year for each researcher. Of all proposals, 3455 were funded (26%). PIs with a funded proposal had significantly more Q1 articles and first/last authored Q1 articles (1.91 vs. 1.30, p<1e-16 and 0.82 vs 0.53, p<1e-16, respectively). Of the successful applications, those involving international collaborations and extended budget had higher publication output. Applicant age, grant duration, and submission year were not correlated with publication performance. Reviewer scores displayed a minor association (corr.coeff = 0.08-011) with the number of Q1 publications. International reviewers were significantly less efficient than national reviewers (p = 0.021). A strong correlation with output was observed for the scientometric characteristics of the applying PI at the time of submission, including H-index (corr.coeff = 0.45-0.54), independent citation (corr.coeff. = 0.46-0.62), and yearly average Q1 articles (corr.coeff = 0.63-0.79, p<1e-16). Similar correlations were observed for nonfunded applicants. We performed a comprehensive evaluation of review-based resource allocation efficiency in basic research funding. Evidence suggests that the past scientometric performance of the principal investigator is the best predictor of future output. © 2020 The Authors.