How to avoid borrowed plumes in academia

Published on Feb 1, 2020in Research Policy
· DOI :10.1016/J.RESPOL.2019.103831
Margit Osterloh27
Estimated H-index: 27
(UZH: University of Zurich),
Bruno S. Frey92
Estimated H-index: 92
(UZH: University of Zurich)
Abstract Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles. We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem. Three extant suggestions for reforming performance management are critically discussed. We advance a new proposal based on the insight that fundamental uncertainty is symptomatic for scholarly work. It suggests focal randomization using a rationally founded and well-orchestrated procedure.
  • References (105)
  • Citations (8)
📖 Papers frequently viewed together
93 Citations
159 Citations
15 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
This paper examines the relationship between placement of publications in Top Five (T5) journals and receipt of tenure in academic economics departments. Analyzing the job histories of tenure-track economists hired by the top 35 U.S. economics departments, we find that T5 publications have a powerful influence on tenure decisions and rates of transition to tenure. A survey of the perceptions of young economists supports the formal statistical analysis. Pursuit of T5 publications has become the o...
22 CitationsSource
I describe and compare sources of data on citations in economics and the statistics that can be constructed from them. Constructing data sets of the post-publication citation histories of articles published in the "Top 5" journals in the 1970s and the 2000s, I examine distributions and life cycles of citations, compare citation histories of articles in different sub-specialties in economics and present evidence on the history and heterogeneity of those journals' impacts and the marginal citation...
25 CitationsSource
#1Rick Vogel (UHH: University of Hamburg)H-Index: 11
#2Fabian Hattke (UHH: University of Hamburg)H-Index: 6
Last. Jessica Petersen (UHH: University of Hamburg)H-Index: 3
view all 3 authors...
Given the growing importance of journal rankings in academic performance management, it is relevant to researchers and managers alike whether there are certain characteristics of publications that are more prevalent the higher a journal is ranked. This paper examines how tangible and adaptable characteristics of papers vary between different rating categories of journals and what the drivers of publication in journals at the top of rankings are. We build on a bibliometric analysis of more than 8...
12 CitationsSource
#1Jian Wang (Katholieke Universiteit Leuven)H-Index: 14
#2Reinhilde Veugelers (Bruegel)H-Index: 39
Last. Paula E. Stephan (GSU: Georgia State University)H-Index: 37
view all 3 authors...
Research which explores unchartered waters has a high potential for major impact but also carries a high uncertainty of having minimal impact. Such explorative research is often described as taking a novel approach. This study examines the complex relationship between pursuing a novel approach and impact. We measure novelty by examining the extent to which a published paper makes first time ever combinations of referenced journals, taking into account the difficulty of making such combinations. ...
54 CitationsSource
#1Katja Rost (UZH: University of Zurich)H-Index: 18
#2Thorsten Teichert (UHH: University of Hamburg)H-Index: 13
Last. Alan Pilkington (University of Westminster)H-Index: 17
view all 3 authors...
Impact factors are commonly used to assess journals relevance. This implies a simplified view on science as a single-stage linear process. Therefore, few top-tier journals are one-sidedly favored as outlets, such that submissions to top-tier journals explode whereas others are short of submissions. Consequently, the often claimed gap between research and practical application in application-oriented disciplines as business administration is not narrowing but becoming entrenched. A more complete ...
4 CitationsSource
#1Yarden Katz (Harvard University)H-Index: 13
#1Yarden KatzH-Index: 2
Last. Ulrich Matter (HSG: University of St. Gallen)H-Index: 3
view all 2 authors...
Researchers and research institutes are increasingly being evaluated using metrics (from bibliometrics to patent counts), which are core instruments of a longstanding effort to quantify scientific productivity and worth. Here, we examine the relationship between commonly used metrics and funding levels for investigators funded by the National Institutes of Health, the largest public funder of biomedical research in the United States, in the years 1985-2015. We find that funding inequality has be...
11 CitationsSource
#1Vincent Larivière (UdeM: Université de Montréal)H-Index: 44
Although the Journal Impact Factor (JIF) is widely acknowledged to be a poor indicator of the quality of individual papers, it is used routinely to evaluate research and researchers. Here, we present a simple method for generating the citation distributions that underlie JIFs. Application of this straightforward protocol reveals the full extent of the skew of distributions and variation in citations received by published papers that is characteristic of all scientific journals. Although there ar...
80 CitationsSource
72 CitationsSource
#1Ludo Waltman (LEI: Leiden University)H-Index: 41
Citation impact indicators nowadays play an important role in research evaluation, and consequently these indicators have received a lot of attention in the bibliometric and scientometric literature. This paper provides an in-depth review of the literature on citation impact indicators. First, an overview is given of the literature on bibliographic databases that can be used to calculate citation impact indicators (Web of Science, Scopus, and Google Scholar). Next, selected topics in the literat...
226 CitationsSource
#1Ferric C. Fang (UW: University of Washington)H-Index: 70
#2Arturo Casadevall (Johns Hopkins University)H-Index: 103
Few science policy issues are more important than the allocation of research funding. Although a 2015 Report suggested that peer review has some ability to prioritize applications ([ 1 ][1]), it is not clear that the best science is being funded ([ 2 ][2]). There is evidence of poor precision and
4 CitationsSource
Cited By8
#1Russell Craig (Durham University)H-Index: 34
#1Russell Craig (Durham University)
view all 4 authors...
Abstract This paper explores the nature and impact of research misconduct in psychology by analyzing 160 articles that were retracted from prominent scholarly journals between 1998 and 2017. We compare findings with recent studies of retracted papers in economics, and business and management, to profile practices that are likely to be problematic in cognate social science disciplines. In psychology, the principal reason for retraction was data fabrication. Retractions took longer to make, and ge...
#1Kyriakos Drivas (UniPi: University of Piraeus)H-Index: 6
#2Dimitris Kremmydas (AUA: Agricultural University of Athens)H-Index: 3
Abstract We examine the impact of a journal's ranking on the citations its papers garner. Our testbed is journals included in the early annual publications of the Academic Journal Guide, published by the Association of Business Schools from 2007 to 2010. By focusing on a small subset of these journals, we provide causal evidence that an increase in a journal's ranking will increase citations to its papers. We argue that this increase can be attributed to (i) more authors learning about and viewi...
#1Robert Rose (HPI: Hasso Plattner Institute)
#2Katharina Hölzle (HPI: Hasso Plattner Institute)H-Index: 10
Last. Jennie Björk (KTH: Royal Institute of Technology)H-Index: 8
view all 3 authors...
#2Hoang Khanh Linh (Duy Tan University)
Last. Quan-Hoang VuongH-Index: 13
view all 5 authors...
#1Steven Wooding (University of Cambridge)H-Index: 12
1 CitationsSource
#1Margit Osterloh (UZH: University of Zurich)H-Index: 27
#2Bruno S. Frey (UZH: University of Zurich)H-Index: 92
#1Andrew J. Oswald (Warw.: University of Warwick)H-Index: 64
Abstract Should journal editors and grant-giving bodies ever make use of random-draw mechanisms to make their final decisions? This short Note argues that the answer is yes. It describes a mathematical rationale for such randomization. Put intuitively, random draw should be used when the gains from the acceptance of unorthodox path-breaking papers outweigh the costs of publishing the unorthodox poor papers. The mathematical case for this rests on an averaging argument and requires ‘convexity’ (o...
2 CitationsSource
A growing number of research agencies are assigning money randomly. A growing number of research agencies are assigning money randomly.
2 CitationsSource