Bias against Novelty in Science: A Cautionary Tale for Users of Bibliometric Indicators

Published on Oct 1, 2017in Research Policy
· DOI :10.1016/J.RESPOL.2017.06.006
Jian Wang15
Estimated H-index: 15
(Katholieke Universiteit Leuven),
Reinhilde Veugelers39
Estimated H-index: 39
Paula E. Stephan37
Estimated H-index: 37
(GSU: Georgia State University)
Research which explores unchartered waters has a high potential for major impact but also carries a high uncertainty of having minimal impact. Such explorative research is often described as taking a novel approach. This study examines the complex relationship between pursuing a novel approach and impact. We measure novelty by examining the extent to which a published paper makes first time ever combinations of referenced journals, taking into account the difficulty of making such combinations. We apply this newly developed measure of novelty to a set of one million research articles across all scientific disciplines. We find that highly novel papers, defined to be those that make more (distinct) new combinations, have more than a triple probability of being a top 1% highly cited paper when using a sufficiently long citation time window to assess impact. Moreover, follow-on papers that cite highly novel research are themselves more likely to be highly cited. However, novel research is also risky as it has a higher variance in the citation performance. These findings are consistent with the “high risk/high gain” characteristic of novel research. We also find that novel papers are typically published in journals with a lower than expected Impact Factor and are less cited when using a short time window. Our findings suggest that science policy, in particular funding decisions which are over reliant on traditional bibliometric indicators based on short-term direct citation counts and Journal Impact Factors, may be biased against novelty.
  • References (54)
  • Citations (92)
📖 Papers frequently viewed together
4 Authors (Brian Uzzi, ..., Benjamin F. Jones)
358 Citations
1 Author (Thomas S. Kuhn)
34.8k Citations
473 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Ruizhi Zhang (Georgia Institute of Technology)H-Index: 2
#2Jian Wang (Katholieke Universiteit Leuven)H-Index: 15
Last. Yajun Mei (Georgia Institute of Technology)H-Index: 13
view all 3 authors...
Evergreens in science are papers that display a continual rise in annual citations without decline, at least within a sufficiently long time period. Aiming to better understand evergreens in particular and patterns of citation trajectory in general, this paper develops a functional data analysis method to cluster citation trajectories of a sample of 1699 research papers published in 1980 in the American Physical Society (APS) journals. We propose a functional Poisson regression model for individ...
5 CitationsSource
#1Kevin J. Boudreau (Harvard University)H-Index: 15
#2Eva C. Guinan (Harvard University)H-Index: 44
Last. Christoph Riedl (NU: Northeastern University)H-Index: 18
view all 4 authors...
Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the "intellectual distance" between the knowledge embodied in research proposals and an evaluator's own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assign...
81 CitationsSource
#1Dennis Verhoeven (Katholieke Universiteit Leuven)H-Index: 2
#2Jurriën Bakker (Katholieke Universiteit Leuven)H-Index: 3
Last. Reinhilde VeugelersH-Index: 5
view all 3 authors...
This study provides a new, more comprehensive measurement of technological novelty. Integrating insights from the existing economics and management literature, we characterize inventions ex ante along two dimensions of technological novelty: Novelty in Recombination and Novelty in Knowledge Origins. For the latter dimension we distinguish between Novel Technological and Novel Scientific Origins. For each dimension we propose an operationalization using patent classification and citation informat...
62 CitationsSource
#1Ben R. Martin (University of Sussex)H-Index: 41
Abstract This extended editorial explores the growing range of stratagems devised by journal editors to boost their Journal Impact Factor (JIF) and the consequences for the credibility of this indicator as well as for the academic community more broadly. Over recent years, JIF has become the most prominent indicator of a journal's standing, bringing intense pressure on journal editors to do what they can to increase it. After explaining the curious way in which JIF is calculated and the technica...
48 CitationsSource
#1Andrey Rzhetsky (U of C: University of Chicago)H-Index: 48
#2Jacob G. Foster (UCLA: University of California, Los Angeles)H-Index: 10
Last. James A. Evans (U of C: University of Chicago)H-Index: 23
view all 4 authors...
A scientist’s choice of research problem affects his or her personal career trajectory. Scientists’ combined choices affect the direction and efficiency of scientific discovery as a whole. In this paper, we infer preferences that shape problem selection from patterns of published findings and then quantify their efficiency. We represent research problems as links between scientific entities in a knowledge network. We then build a generative model of discovery informed by qualitative research on ...
61 CitationsSource
#1Feng Shi (U of C: University of Chicago)H-Index: 7
#2Jacob G. Foster (UCLA: University of California, Los Angeles)H-Index: 10
Last. James A. Evans (U of C: University of Chicago)H-Index: 23
view all 3 authors...
Abstract Science is a complex system. Building on Latour's actor network theory, we model published science as a dynamic hypergraph and explore how this fabric provides a substrate for future scientific discovery. Using millions of abstracts from MEDLINE, we show that the network distance between biomedical things (i.e., people, methods, diseases, chemicals) is surprisingly small. We then show how science moves from questions answered in one year to problems investigated in the next through a we...
44 CitationsSource
#1Jacob G. FosterH-Index: 10
#2Andrey RzhetskyH-Index: 48
Last. James A. EvansH-Index: 23
view all 3 authors...
119 CitationsSource
#1Alfredo Yegros-Yegros (LEI: Leiden University)H-Index: 7
#2Ismael Rafols (Polytechnic University of Valencia)H-Index: 30
Last. Pablo D'Este (Polytechnic University of Valencia)H-Index: 20
view all 3 authors...
This article analyses the effect of degree of interdisciplinarity on the citation impact of individual publications for four different scientific fields. We operationalise interdisciplinarity as disciplinary diversity in the references of a publication, and rather than treating interdisciplinarity as a monodimensional property, we investigate the separate effect of different aspects of diversity on citation impact: i.e. variety, balance and disparity. We use a Tobit regression model to examine t...
95 CitationsSource
#1Jian Wang (Katholieke Universiteit Leuven)H-Index: 15
#2Bart Thijs (Katholieke Universiteit Leuven)H-Index: 23
Last. Wolfgang Glänzel (Katholieke Universiteit Leuven)H-Index: 59
view all 3 authors...
Interdisciplinary research is increasingly recognized as the solution to today’s challenging scientific and societal problems, but the relationship between interdisciplinary research and scientific impact is still unclear. This paper studies the association between the degree of interdisciplinarity and the number of citations at the paper level. Different from previous studies compositing various aspects of interdisciplinarity into a single indicator, we use factor analysis to uncover distinct d...
57 CitationsSource
#1Diana Hicks (Georgia Institute of Technology)H-Index: 29
#2Paul Wouters (LEI: Leiden University)H-Index: 27
Last. Ismael Rafols (Polytechnic University of Valencia)H-Index: 30
view all 5 authors...
Use these ten principles to guide research evaluation, urge Diana Hicks, Paul Wouters and colleagues.
591 CitationsSource
Cited By92
#1Magda Fontana (UNITO: University of Turin)H-Index: 6
#2Martina Iori (UNITO: University of Turin)H-Index: 1
Last. Roberta Sinatra (ITU: IT University of Copenhagen)H-Index: 17
view all 4 authors...
Abstract Novelty indicators are increasingly important for science policy. This paper challenges the indicators of novelty as an atypical combination of knowledge (Uzzi et al., 2013) and as the first appearance of a knowledge combination (Wang et al., 2017). We exploit a sample of 230,854 articles (1985 - 2005), published on 8 journals of the American Physical Society (APS) and 2.4 million citations to test the indicators using (i) a Configuration Null Model, (ii) an external validation set of a...
#1Qing Ke (NU: Northeastern University)H-Index: 6
Abstract An ongoing interest in innovation studies is to understand how knowledge generated from scientific research can be used in the development of technologies. While previous inquiries have devoted to studying the scientific capacity of technologies and institutional factors facilitating technology transfer, little is known about the intrinsic characteristics of scientific publications that gain direct technological impact. Here we focus on two features, namely basicness and novelty. Using ...
#1Yue Qian (DUT: Dalian University of Technology)
#2Yu Liu (DUT: Dalian University of Technology)H-Index: 58
Last. Quan Z. Sheng (Macquarie University)
view all 3 authors...
Abstract Detecting what type of knowledge constitutes a discipline, tracking how the knowledge changes, and understanding why the changes are triggered are the key issues in analyzing scientific development from a macro perspective, which is usually analyzed by the topic of evolution. However, traditional methods assume that the disciplinary structure is flat with only one-layer topics, rather than a tree-like structure with hierarchical topics, which leads to the inability of existing methods t...
#1Guoqiang Liang (DUT: Dalian University of Technology)H-Index: 1
#2Haiyan Hou (DUT: Dalian University of Technology)H-Index: 5
Last. Zhigang Hu (DUT: Dalian University of Technology)H-Index: 8
view all 4 authors...
Abstract Scientific ideas seldom come entirely from nothing; they often originate from existing knowledge. In this study, we aim to identify the relationship between the knowledge recency and Nobel Prize-winning articles (NPs) from the perspectives of gender, career stage, and country of the originators. Scientific literatures published by Nobel laureates in the Physiology or Medicine field are treated as NPs, and the age of references cited in those literatures are considered a measurement of k...
#1Rachel Miles (VT: Virginia Tech)H-Index: 2
#2Virginia Pannabecker (VT: Virginia Tech)H-Index: 3
Last. Jim A. Kuypers (VT: Virginia Tech)H-Index: 2
view all 3 authors...
In the spring of 2019, survey research was conducted at Virginia Polytechnic Institute and State University (Virginia Tech), a large, public, Carnegie-classified R1 institution in southwest Virginia, to determine faculty perceptions of research assessment as well as how and why they use researcher profiles and research impact indicators. The Faculty Senate Research Assessment Committee (FSRAC) reported the quantitative and qualitative results to the Virginia Tech Board of Visitors to demonstrate...
#1Guoqiang LiangH-Index: 1
#2Haiyan HouH-Index: 5
Last. Zhigang HuH-Index: 8
view all 4 authors...
A Sleeping Beauty is a publication that is apparently unrecognized for some period of time before experiencing sudden recognition by citation. Various reasons, including resistance to new ideas, have been attributed to such delayed recognition. We examine this phenomenon in the special case of co-citations, which represent new ideas generated through the combination of existing ones. Using relatively stringent selection criteria derived from the work of others, we analyze a very large dataset of...
#1Manolis Antonoyiannakis (Columbia University)H-Index: 5
We study how a single paper affects the Impact Factor (IF) by analyzing data from 3,088,511 papers published in 11639 journals in the 2017 Journal Citation Reports of Clarivate Analytics. We find t...
#1Marcos FerassoH-Index: 4
#2Tatiana Beliaeva (KEDGE Business School)H-Index: 4
Last. Domingo Ribeiro-Soriano (University of Valencia)H-Index: 17
view all 5 authors...
#2James R. Bradley (W&M: College of William & Mary)H-Index: 15
Last. George ChackoH-Index: 3
view all 5 authors...
Co-citation measurements can reveal the extent to which a concept representing a novel combination of existing ideas evolves towards a specialty. The strength of co-citation is represented by its f...