Big names or big ideas: Do peer-review panels select the best science proposals?

Published on Apr 24, 2015in Science41.037
· DOI :10.1126/science.aaa0185
Danielle Li7
Estimated H-index: 7
(Harvard University),
Leila Agha4
Estimated H-index: 4
(BU: Boston University)
This paper examines the success of peer-review panels in predicting the future quality of proposed research. We construct new data to track publication, citation, and patenting outcomes associated with more than 130,000 research project (R01) grants funded by the U.S. National Institutes of Health from 1980 to 2008. We find that better peer-review scores are consistently associated with better research outcomes and that this relationship persists even when we include detailed controls for an investigator’s publication history, grant history, institutional affiliations, career stage, and degree types. A one–standard deviation worse peer-review score among awarded grants is associated with 15% fewer citations, 7% fewer publications, 19% fewer high-impact publications, and 14% fewer follow-on patents.
  • References (19)
  • Citations (68)
📖 Papers frequently viewed together
2 Authors (Stephen Cole, Ga Simon)
425 Citations
287 Citations
25 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#2Richard R. NelsonH-Index: 86
99 Citations
#1Bruce M. Albertsll (UCSF: University of California, San Francisco)H-Index: 72
#2Marc W. Kirschner (Harvard University)H-Index: 144
Last. Harold E. VarmusH-Index: 110
view all 4 authors...
The long-held but erroneous assumption of never-ending rapid growth in biomedical science has created an unsustainable hypercompetitive system that is discouraging even the most outstanding prospective students from entering our profession—and making it difficult for seasoned investigators to produce their best work. This is a recipe for long-term decline, and the problems cannot be solved with simplistic approaches. Instead, it is time to confront the dangers at hand and rethink some fundamenta...
383 CitationsSource
#1Narasimhan Danthi (NIH: National Institutes of Health)H-Index: 7
#2Colin O. Wu (NIH: National Institutes of Health)H-Index: 47
Last. Michael S. Lauer (NIH: National Institutes of Health)H-Index: 87
view all 4 authors...
47 CitationsSource
#1Pierre Azoulay (NBER: National Bureau of Economic Research)H-Index: 18
#2Toby E. Stuart (University of California, Berkeley)H-Index: 34
Last. Yanbo Wang (BU: Boston University)H-Index: 8
view all 3 authors...
In a market context, a status effect occurs when actors are accorded differential recognition for their efforts depending on their location in a status ordering, holding constant the quality of these efforts. In practice, because it is very difficult to measure quality, this ceteris paribus proviso often precludes convincing empirical assessments of the magnitude of status effects. We address this problem by examining the impact of a major status-conferring prize that shifts actors' positions in...
94 CitationsSource
#1Brian A. Jacob (UM: University of Michigan)H-Index: 37
#2Lars Lefgren (BYU: Brigham Young University)H-Index: 19
In this paper, we estimate the impact of receiving an NIH grant on subsequent publications and citations. Our sample consists of all applications (unsuccessful as well as successful) to the NIH from 1980 to 2000 for postdoctoral training grants (F32s) and standard research grants (R01s). Both OLS and regression discontinuity estimates show that receipt of either an NIH postdoctoral fellowship or research grant leads to about one additional publication over the next five years. The estimates repr...
170 CitationsSource
#1Brian A. Jacob (UM: University of Michigan)H-Index: 37
#2Lars Lefgren (BYU: Brigham Young University)H-Index: 19
In this paper, we estimate the impact of receiving an NIH postdoctoral training grant on subsequent publications and citations. Our sample consists of all applications for NIH postdoctoral training grants (unsuccessful as well as successful) from 1980 to 2000. Both ordinary least squares and regression discontinuity estimates show that receipt of an NIH postdoctoral fellowship leads to about one additional publication over the next five years, which reflects a 20% increase in research productivi...
49 CitationsSource
#1Pierre Azoulay (MIT: Massachusetts Institute of Technology)H-Index: 18
#2Joshua Graff Zivin (NBER: National Bureau of Economic Research)H-Index: 32
Last. Bhaven N. Sampat (Columbia University)H-Index: 29
view all 3 authors...
Are scientific knowledge flows embodied in individuals, or "in the air"? To answer this question, we measure the effect of labor mobility in a sample of 9,483 elite academic life scientists on the citation trajectories associated with individual articles (resp. patents) published (resp. granted) before the scientist moved to a new institution. We find that article-to-article citations from the scientific community at the superstar's origin location are barely affected by their departure. In cont...
46 Citations
#1E.R. Dorsey (UR: University of Rochester)H-Index: 48
#2Jason de Roulet (Case Western Reserve University)H-Index: 4
Last. Hamilton Moses (Johns Hopkins University)H-Index: 17
view all 9 authors...
Context With the exception of the American Recovery and Reinvestment Act, funding support for biomedical research in the United States has slowed after a decade of doubling. However, the extent and scope of slowing are largely unknown. Objective To quantify funding of biomedical research in the United States from 2003 to 2008. Design Publicly available data were used to quantify funding from government (federal, state, and local), private, and industry sources. Regression models were used to com...
230 CitationsSource
#1William R. Kerr (Harvard University)H-Index: 35
The ethnic composition of US scientists and engineers is undergoing a significant transformation. This study applies an ethnic-name database to individual patent records granted by the United States Patent and Trademark Office to document these trends with greater detail than previously available. Most notably, the contributions of Chinese and Indian scientists to US technology formation increase dramatically in the 1990s, before noticeably leveling off after 2000 and declining in the case of In...
44 CitationsSource
This study explores the importance of knowledge transfer for international technology diffusion by examining ethnic scientific and entrepreneurial communities in the US and their ties to their home countries. US ethnic research communities are quantified by applying an ethnic-name database to individual patent records. International patent citations con.rm knowledge diffuses through ethnic networks, and manufacturing output in foreign countries increases with an elasticity of 0.1-0.3 to stronger...
312 CitationsSource
Cited By68
#1Gupeng Zhang (CAS: Chinese Academy of Sciences)H-Index: 7
#2Libin Xiong (CAS: Chinese Academy of Sciences)
Last. Hongbo Duan (CAS: Chinese Academy of Sciences)H-Index: 9
view all 5 authors...
#1Joan Farre-Mensa (Harvard University)H-Index: 7
#2Deepak Hegde (NYU: New York University)H-Index: 14
Last. Alexander Ljungqvist (Research Institute of Industrial Economics)H-Index: 36
view all 3 authors...
Motivated by concerns that the patent system is hindering innovation, particularly for small inventors, this study investigates the bright side of patents. We examine whether patents help startups grow and succeed using detailed micro data on all patent applications filed by startups at the U.S. Patent and Trademark Office (USPTO) since 2001 and approved or rejected before 2014. We leverage the fact that patent applications are assigned quasi-randomly to USPTO examiners and instrument for the pr...
15 CitationsSource
Abstract The scientific community is engaged in an active debate on the value of its peer-review system. Does peer review actually serve the role we envision for it—that of helping government agencies predict what ideas have the best chance of contributing to scientific advancement? Many federal agencies use a two-step review process that includes programmatic discretion in selecting awards. This process allows us to determine whether success in a future independent scientific-research career is...
#1Erin E. Butler (Dartmouth College)H-Index: 10
#2Nathaniel J. Dominy (Dartmouth College)H-Index: 36
Abstract Background Fifty years ago, the groundbreaking British sketch seriesMonty Python’s Flying Circus premiered on BBC One and forever changed the world of comedy. The humour transcended mere absurdity by poking a subversive finger in the eye of buttoned-up British society. Here, we commemorate this cultural milestone and simultaneously call attention to an emerging concept in the health sciences, termed simplified peer review. The union of these disparate subjects motivates a formal gait an...
#1Margit Osterloh (UZH: University of Zurich)H-Index: 28
#2Bruno S. Frey (UZH: University of Zurich)H-Index: 91
Abstract Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles. We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem. Three extant suggestions for reforming perfo...
5 CitationsSource
AbstractThe dimensions of mental health status (MHS), all those that may influence MHS, and the measures to represent these dimensions must come to be agreed upon and all of them included in every study of psychotherapy effectiveness in order for these studies’ results to be properly comparable, decisive, and cumulative. This is essential for eventually achieving a psychotherapy effectiveness research (PER) that is adequate for informing individual case psychotherapy practice rather than only re...
#1Yarden Katz (Harvard University)H-Index: 13
#2Ulrich Matter (HSG: University of St. Gallen)H-Index: 3
Academic scientists and research institutes are increasingly being evaluated using digital metrics, from bibliometrics to patent counts. These metrics are often framed, by science policy analysts, economists of science as well as funding agencies, as objective and universal proxies for scientific worth, potential, and productivity. In biomedical science, where there is stiff competition for grants from the National Institutes of Health (NIH), metrics are sold as a less arbitrary way to allocate ...
Abstract Introduction and objectives The Spanish Society of Cardiology/Spanish Heart Foundation (SEC/FEC) annually awards grants for cardiovascular research projects. Our objective was to analyze the trend in these investments and their resulting scientific production from 2007 to 2012. Methods A search of the publications funded by the SEC/FEC was carried out, according to the following inclusion criteria: publication in a journal indexed in MEDLINE or EMBASE, publication date after the grant, ...
Resumen Introduccion y objetivos La Sociedad Espanola de Cardiologia/Fundacion Espanola del Corazon (SEC/FEC) realiza convocatorias anuales de becas para proyectos de investigacion cardiovascular. El objetivo es analizar la evolucion de estas inversiones y la produccion cientifica derivada en el periodo 2007-2012. Metodos Se ha realizado una busqueda de las publicaciones financiadas por SEC/FEC, segun los siguientes criterios de inclusion: publicacion en revista indexada en MEDLINE o EMBASE, fec...
#1Travis Hoppe (NIH: National Institutes of Health)H-Index: 6
#2Aviva Litovitz (NIH: National Institutes of Health)H-Index: 1
Last. George M. Santangelo (NIH: National Institutes of Health)H-Index: 3
view all 11 authors...
Despite efforts to promote diversity in the biomedical workforce, there remains a lower rate of funding of National Institutes of Health R01 applications submitted by African-American/black (AA/B) scientists relative to white scientists. To identify underlying causes of this funding gap, we analyzed six stages of the application process from 2011 to 2015 and found that disparate outcomes arise at three of the six: decision to discuss, impact score assignment, and a previously unstudied stage, to...
5 CitationsSource