Big names or big ideas: Do peer-review panels select the best science proposals?

Published on Apr 24, 2015in Science41.04
· DOI :10.1126/science.aaa0185
Danielle Li7
Estimated H-index: 7
(Harvard University),
Leila Agha4
Estimated H-index: 4
(BU: Boston University)
This paper examines the success of peer-review panels in predicting the future quality of proposed research. We construct new data to track publication, citation, and patenting outcomes associated with more than 130,000 research project (R01) grants funded by the U.S. National Institutes of Health from 1980 to 2008. We find that better peer-review scores are consistently associated with better research outcomes and that this relationship persists even when we include detailed controls for an investigator’s publication history, grant history, institutional affiliations, career stage, and degree types. A one–standard deviation worse peer-review score among awarded grants is associated with 15% fewer citations, 7% fewer publications, 19% fewer high-impact publications, and 14% fewer follow-on patents.
  • References (19)
  • Citations (60)
Bruce M. Albertsll70
Estimated H-index: 70
(UCSF: University of California, San Francisco),
Marc W. Kirschner142
Estimated H-index: 142
(Harvard University)
+ 1 AuthorsHarold E. Varmus110
Estimated H-index: 110
The long-held but erroneous assumption of never-ending rapid growth in biomedical science has created an unsustainable hypercompetitive system that is discouraging even the most outstanding prospective students from entering our profession—and making it difficult for seasoned investigators to produce their best work. This is a recipe for long-term decline, and the problems cannot be solved with simplistic approaches. Instead, it is time to confront the dangers at hand and rethink some fundamenta...
Published on Feb 14, 2014in Circulation Research15.86
Narasimhan Danthi7
Estimated H-index: 7
(NIH: National Institutes of Health),
Colin O. Wu47
Estimated H-index: 47
(NIH: National Institutes of Health)
+ 1 AuthorsMichael S. Lauer83
Estimated H-index: 83
(NIH: National Institutes of Health)
Rationale:Funding decisions for cardiovascular R01 grant applications at the National Heart, Lung, and Blood Institute (NHLBI) largely hinge on percentile rankings. It is not known whether this approach enables the highest impact science. Objective:Our aim was to conduct an observational analysis of percentile rankings and bibliometric outcomes for a contemporary set of funded NHLBI cardiovascular R01 grants. Methods and Results:We identified 1492 investigator-initiated de novo R01 grant applica...
Published on Jul 1, 2011in Research Policy5.42
Brian A. Jacob35
Estimated H-index: 35
(UM: University of Michigan),
Lars Lefgren19
Estimated H-index: 19
(BYU: Brigham Young University)
In this paper, we estimate the impact of receiving an NIH postdoctoral training grant on subsequent publications and citations. Our sample consists of all applications for NIH postdoctoral training grants (unsuccessful as well as successful) from 1980 to 2000. Both ordinary least squares and regression discontinuity estimates show that receipt of an NIH postdoctoral fellowship leads to about one additional publication over the next five years, which reflects a 20% increase in research productivi...
Published on Jan 1, 2011in National Bureau of Economic Research
Pierre Azoulay17
Estimated H-index: 17
(MIT: Massachusetts Institute of Technology),
Joshua Graff Zivin32
Estimated H-index: 32
(NBER: National Bureau of Economic Research),
Bhaven N. Sampat30
Estimated H-index: 30
(Columbia University)
Are scientific knowledge flows embodied in individuals, or "in the air"? To answer this question, we measure the effect of labor mobility in a sample of 9,483 elite academic life scientists on the citation trajectories associated with individual articles (resp. patents) published (resp. granted) before the scientist moved to a new institution. We find that article-to-article citations from the scientific community at the superstar's origin location are barely affected by their departure. In cont...
Published on Jan 13, 2010in JAMA51.27
E.R. Dorsey47
Estimated H-index: 47
(UR: University of Rochester),
Jason de Roulet4
Estimated H-index: 4
(Case Western Reserve University)
+ 6 AuthorsHamilton Moses17
Estimated H-index: 17
(Johns Hopkins University)
Context With the exception of the American Recovery and Reinvestment Act, funding support for biomedical research in the United States has slowed after a decade of doubling. However, the extent and scope of slowing are largely unknown. Objective To quantify funding of biomedical research in the United States from 2003 to 2008. Design Publicly available data were used to quantify funding from government (federal, state, and local), private, and industry sources. Regression models were used to com...
Published on Dec 22, 2008
William R. Kerr33
Estimated H-index: 33
(Harvard University)
The ethnic composition of US scientists and engineers is undergoing a significant transformation. This study applies an ethnic-name database to individual patent records granted by the United States Patent and Trademark Office to document these trends with greater detail than previously available. Most notably, the contributions of Chinese and Indian scientists to US technology formation increase dramatically in the 1990s, before noticeably leveling off after 2000 and declining in the case of In...
Published on Aug 1, 2008in The Review of Economics and Statistics3.64
William R. Kerr33
Estimated H-index: 33
(Harvard University)
This study explores the importance of knowledge transfer for international technology diffusion by examining ethnic scientific and entrepreneurial communities in the US and their ties to their home countries. US ethnic research communities are quantified by applying an ethnic-name database to individual patent records. International patent citations con.rm knowledge diffuses through ethnic networks, and manufacturing output in foreign countries increases with an elasticity of 0.1-0.3 to stronger...
Published on Oct 1, 2006in Scientometrics2.77
Julie Callaert10
Estimated H-index: 10
(Katholieke Universiteit Leuven),
Bart Van Looy26
Estimated H-index: 26
(Katholieke Universiteit Leuven)
+ 2 AuthorsBart Thijs23
Estimated H-index: 23
(Katholieke Universiteit Leuven)
The recent developments towards more systemic conceptualizations of innovation dynamics and related policies highlight the need for indicators that mirror the dynamics involved. In this contribution, we assess the role that 'non-patent references', found in patent documents, can play in this respect. After examining the occurrence of these references in the USPTO and EPO patent systems, their precise nature is delineated by means of a content analysis of two samples of nonpatent references (n=10...
Published on Apr 1, 2004in The American Economic Review4.10
Jonah E. Rockoff26
Estimated H-index: 26
(Harvard University)
Teacher quality is widely believed to be important for education, despite little evidence that teachers' credentials matter for student achievement. To accurately measure variation in achievement due to teachers' characteristics-both observable and unobservable-it is essential to identify teacher fixed effects. Unlike previous studies, I use panel data to estimate teacher fixed effects while controlling for fixed student characteristics and classroom specific variables. I find large and statisti...
Cited By60
Published on Feb 1, 2020in Research Policy5.42
Margit Osterloh25
Estimated H-index: 25
(UZH: University of Zurich),
Bruno S. Frey88
Estimated H-index: 88
(UZH: University of Zurich)
Abstract Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles. We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem. Three extant suggestions for reforming perfo...
Published on 2019in Artificial Intelligence4.48
Haris Aziz21
Estimated H-index: 21
(Technical University of Berlin),
Omer Lev8
Estimated H-index: 8
(BGU: Ben-Gurion University of the Negev)
+ 2 AuthorsToby Walsh51
Estimated H-index: 51
(Technical University of Berlin)
Abstract Peer reviews, evaluations, and selections are a fundamental aspect of modern science. Funding bodies the world over employ experts to review and select the best proposals from those submitted for funding. The problem of peer selection, however, is much more general: a professional society may want to give a subset of its members awards based on the opinions of all members; an instructor for a Massive Open Online Course (MOOC) or an online course may want to crowdsource grading; or a mar...
Published on Sep 1, 2019in BMJ Open2.38
Maryaline Catillon (Harvard University)
Objective To measure the frequency of adequate methods, inadequate methods and poor reporting in published randomised controlled trials (RCTs) and test potential factors associated with adequacy of methods and reporting. Design Retrospective analysis of RCTs included in Cochrane reviews. Time series describes the proportion of RCTs using adequate methods, inadequate methods and poor reporting. A multinomial logit model tests potential factors associated with methods and reporting, including fund...
Published on Jun 18, 2019in Journal of Evolutionary Biology2.54
Stephanie Meirmans6
Estimated H-index: 6
(UvA: University of Amsterdam),
Roger K. Butlin61
Estimated H-index: 61
(University of Sheffield)
+ 6 AuthorsMaurine Neiman20
Estimated H-index: 20
(UI: University of Iowa)
Published on May 17, 2019in Scientometrics2.77
To better understand the added-value of the academic evaluation process, this paper studies the relationship between scores given by 105 evaluators to 1900 doctoral candidates who received a scholarship and their outcomes 10 years after the competition. I first find that a one point increase in total score is associated with a 1.4 percentage point (2.1% of a s.e.) increase in the probability of completing a Ph.D. in 5 years, with a 1.0 percentage point (2.1% of a s.e.) increase in the probabilit...
Published on Mar 4, 2019in Publications
Francesco Pomponi11
Estimated H-index: 11
Bernardino D’Amico3
Estimated H-index: 3
Tom Rye18
Estimated H-index: 18
Scientific publishing is experiencing unprecedented growth in terms of outputs across all fields. Inevitably this creates pressure throughout the system on a number of entities. One key element is represented by peer-reviewers, whose demand increases at an even higher pace than that of publications, since more than one reviewer per paper is needed and not all papers that get reviewed get published. The relatively recent Publons platform allows for unprecedented insight into the usual ‘blindness’...
Published on Mar 1, 2019in Revista Espanola De Cardiologia5.13
Luis Rodríguez Padial14
Estimated H-index: 14
Ignacio Fernández Lozano11
Estimated H-index: 11
+ 4 AuthorsAndrés Íñiguez Romo7
Estimated H-index: 7
Abstract Introduction and objectives The Spanish Society of Cardiology/Spanish Heart Foundation (SEC/FEC) annually awards grants for cardiovascular research projects. Our objective was to analyze the trend in these investments and their resulting scientific production from 2007 to 2012. Methods A search of the publications funded by the SEC/FEC was carried out, according to the following inclusion criteria: publication in a journal indexed in MEDLINE or EMBASE, publication date after the grant, ...
Published on Mar 1, 2019in Ecology and Evolution2.42
Charles W. Fox51
Estimated H-index: 51
(UK: University of Kentucky),
C. E. Timothy Paine20
Estimated H-index: 20
(UNE: University of New England (Australia))
Published on Mar 1, 2019in Journal of data science
Ivo D. Dinov42
Estimated H-index: 42
(UM: University of Michigan)
Data science is a bridge discipline connecting fundamental science, applied disciplines, and the arts. The demand for novel data science methods is well established. However, there is much less agreement on the core aspects of representation, modeling, and analytics that involve huge and heterogeneous datasets. The scientific community needs to build consensus about data science education and training curricula, including the necessary entry matriculation prerequisites and the expected learning ...
View next paperChance and consensus in peer review