On the Predictability of Future Impact in Science

Published on Nov 1, 2013in Scientific Reports4.011
· DOI :10.1038/srep03052
Orion Penner11
Estimated H-index: 11
Raj Kumar Pan23
Estimated H-index: 23
+ 2 AuthorsSanto Fortunato44
Estimated H-index: 44
Correctly assessing a scientist's past research impact and potential for future impact is key in recruitment decisions and other evaluation processes. While a candidate's future impact is the main concern for these decisions, most measures only quantify the impact of previous work. Recently, it has been argued that linear regression models are capable of predicting a scientist's future impact. By applying that future impact model to 762 careers drawn from three disciplines: physics, biology, and mathematics, we identify a number of subtle, but critical, flaws in current models. Specifically, cumulative non-decreasing measures like the h-index contain intrinsic autocorrelation, resulting in significant overestimation of their “predictive power”. Moreover, the predictive power of these models depend heavily upon scientists' career age, producing least accurate estimates for young researchers. Our results place in doubt the suitability of such models, and indicate further investigation is required before they can be used in recruiting decisions.
  • References (24)
  • Citations (63)
📖 Papers frequently viewed together
5,164 Citations
129 Citations
339 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Alexander Michael Petersen (IMT Institute for Advanced Studies Lucca)H-Index: 21
#2Santo Fortunato (Aalto University)H-Index: 44
Last. Nicola Carmine Salerno (BU: Boston University)H-Index: 32
view all 9 authors...
Reputation is an important social construct in science, which enables informed quality assessments of both publications and careers of scientists in the absence of complete systemic information. However, the relation between reputation and career growth of an individual remains poorly understood, despite recent proliferation of quantitative research evaluation methods. Here, we develop an original framework for measuring how a publication’s citation rate Δc depends on the reputation of its centr...
107 CitationsSource
#1Orion PennerH-Index: 11
Last. Santo FortunatoH-Index: 44
view all 4 authors...
18 CitationsSource
#1Michael Schreiber (Chemnitz University of Technology)H-Index: 34
The h-index has been shown to have predictive power. Here I report results of an empirical study showing that the increase of the h-index with time often depends for a long time on citations to rather old publications. This inert behavior of the h-index means that it is difficult to use it as a measure for predicting future scientific output.
31 CitationsSource
#1Orion PennerH-Index: 11
#2Raj Kumar PanH-Index: 23
Last. Santo FortunatoH-Index: 4
view all 4 authors...
Laboratory for the Analysis of Complex Economic Systems,IMT Lucca Institute for Advanced Studies, 55100 Lucca, ItalyWe stress-test the career predictability model proposed by Acuna et al. [Nature 489, 201-2 2012] by applyingtheir model to a longitudinal career data set of 100 Assistant professors in physics, two from each of the top 50physics departments in the US. The Acuna model claims to predict h(t+ t), a scientist’s h-index t into thefuture, using a linear combination of 5 cumulative career...
15 CitationsSource
#1Jordi Duch (NU: Northwestern University)H-Index: 14
#2Xiao Han T. Zeng (NU: Northwestern University)H-Index: 6
Last. Luís A. Nunes Amaral (NU: Northwestern University)H-Index: 59
view all 7 authors...
Many studies demonstrate that there is still a significant gender bias, especially at higher career levels, in many areas including science, technology, engineering, and mathematics (STEM). We investigated field-dependent, gender-specific effects of the selective pressures individuals experience as they pursue a career in academia within seven STEM disciplines. We built a unique database that comprises 437,787 publications authored by 4,292 faculty members at top United States research universit...
94 CitationsSource
We tested the underlying assumption that citation counts are reliable predictors of future success, analyzing complete citation data on the careers of scientists. Our results show that i) among all citation indicators, the annual citations at the time of prediction is the best predictor of future citations, ii) future citations of a scientist's published papers can be predicted accurately ( for a 1-year prediction, ) but iii) future citations of future work are hardly predictable.
48 CitationsSource
Despite efforts to recruit and retain more women, a stark gender disparity persists within academic science. Abundant research has demonstrated gender bias in many demographic groups, but has yet to experimentally investigate whether science faculty exhibit a bias against female students that could contribute to the gender disparity in academic science. In a randomized double-blind study (n = 127), science faculty from research-intensive universities rated the application materials of a student—...
1,025 CitationsSource
#1Daniel E. Acuna (NU: Northwestern University)H-Index: 10
#2Stefano Allesina (UIC: University of Illinois at Chicago)H-Index: 34
Last. Konrad Paul Kording (Rehabilitation Institute of Chicago)H-Index: 44
view all 3 authors...
Daniel E. Acuna, Stefano Allesina and Konrad P. Kording present a formula to estimate the future h-index of life scientists.
129 CitationsSource
#1Alexander Michael Petersen (UCM: University of California, Merced)H-Index: 21
#2Massimo Riccaboni (Katholieke Universiteit Leuven)H-Index: 28
Last. Nicola Carmine SalernoH-Index: 32
view all 4 authors...
Understanding how institutional changes within academia may affect the overall potential of science requires a better quantitative representation of how careers evolve over time. Because knowledge spillovers, cumulative advantage, competition, and collaboration are distinctive features of the academic profession, both the employment relationship and the procedures for assigning recognition and allocating funding should be designed to account for these factors. We study the annual production ni(t...
72 CitationsSource
#1Paula E. StephanH-Index: 36
The beauty of science may be pure and eternal, but the practice of science costs money. And scientists, being human, respond to incentives and costs, in money and glory. Choosing a research topic, deciding what papers to write and where to publish them, sticking with a familiar area or going into something new--the payoff may be tenure or a job at a highly ranked university or a prestigious award or a bump in salary. The risk may be not getting any of that. At a time when science is seen as an e...
386 Citations
Cited By63
The most commonly used publication metrics for individual researchers are the the total number of publications, the total number of citations, and Hirsch’s h-index. Each of these is cumulative, and...
#1Juan Manuel Durán (TU Delft: Delft University of Technology)H-Index: 3
#2Zachary Pirtle (NASA Headquarters)H-Index: 4
When one wants to use citizen input to inform policy, what should the standards of informedness on the part of the citizens be? While there are moral reasons to allow every citizen to participate and have a voice on every issue, regardless of education and involvement, designers of participatory assessments have to make decisions about how to structure deliberations as well as how much background information and deliberation time to provide to participants. After assessing different frameworks f...
Last. Mikko KiveläH-Index: 13
view all 4 authors...
#1Matteo Chinazzi (NU: Northeastern University)H-Index: 6
#2Bruno GonçalvesH-Index: 2
Last. Alessandro Vespignani (NU: Northeastern University)H-Index: 88
view all 4 authors...
Scientific discoveries do not occur in vacuum but rather by connecting existing pieces of knowledge in new and creative ways. Mapping the relation and structure of scientific knowledge is therefore central to our understanding of the dynamics of scientific production. Here we introduce a new approach to generate scientific knowledge maps based on a machine learning approach that, starting from the observed publication patterns of authors, generates an N -dimensional space where it is possible to...
2 CitationsSource
#1Danielle H. Lee (Sangmyung University)H-Index: 1
This paper examines how early career-related factors can predict the future research performance of computer and information scientists. Although a few bibliometric studies have previously investigated multiple factors relating to early career scientists that significantly predict their future research performance, there have been limited studies on early career-related factors affecting scientists in the fields of information science and computer science. This study analyzes 4102 scientists who...
#1John PanaretosH-Index: 13
Last. Chrisovalantis MalesiosH-Index: 6
view all 2 authors...
#1Alexander Michael Petersen (UCM: University of California, Merced)H-Index: 21
#2Raj Kumar Pan (Aalto University)H-Index: 23
Last. Santo Fortunato (IU: Indiana University Bloomington)H-Index: 44
view all 4 authors...
Abstract Quantitative research evaluation requires measures that are transparent, relatively simple, and free of disciplinary and temporal bias. We document and provide a solution to a hitherto unaddressed temporal bias – citation inflation – which arises from the basic fact that scientific publication is steadily growing at roughly 4% per year. Moreover, because the total production of citations grows by a factor of 2 every 12 years, this means that the real value of a citation depends on when ...
3 CitationsSource
In all of science, the authors of publications depend on the knowledge presented by the previous publications. Thus they "stand on the shoulders of giants" and there is a flow of knowledge from previous publications to more recent ones. The dominating paradigm for tracking this flow of knowledge is to count the number of direct citations, but this neglects the fact that beneath the first layer of citations there is a full body of literature. In this study, we go underneath the "shoulders" by inv...
#1Diego F. M. Oliveira (RPI: Rensselaer Polytechnic Institute)H-Index: 10
#2Kevin Chan (ARL: United States Army Research Laboratory)H-Index: 13
Abstract In this work, we employ a minimal agent-based model to explore the mechanisms that regulate competition between memes that spread online. We investigate the case in which each piece of information has a quality, and the higher is the quality the higher are the chances of being transmitted. The model allows us to study the impact of influential nodes on the spreading behavior. We show that meme’s quality does not guarantee virility, but there is a strong correlation between the meme’s su...
1 CitationsSource
#1Oliver E. Williams (QMUL: Queen Mary University of London)H-Index: 1
#2Lucas Lacasa (QMUL: Queen Mary University of London)H-Index: 18
Last. Vito LatoraH-Index: 58
view all 3 authors...
In certain artistic endeavours—such as acting in films and TV, where unemployment rates hover at around 90%—sustained productivity (simply making a living) is probably a better proxy for quantifying success than high impact. Drawing on a worldwide database, here we study the temporal profiles of activity of actors and actresses. We show that the dynamics of job assignment is well described by a “rich-get-richer” mechanism and we find that, while the percentage of a career spent active is unpredi...
1 CitationsSource