Scientific peer review

Published on Jan 1, 2011in The Artist and Journal of Home Culture
· DOI :10.1002/aris.2011.1440450112
Lutz Bornmann48
Estimated H-index: 48
  • References (220)
  • Citations (240)
📖 Papers frequently viewed together
287 Citations
677 Citations
5,164 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Lutz Bornmann (MPG: Max Planck Society)H-Index: 48
#2Rüdiger Mutz (ETH Zurich)H-Index: 23
Last. Hans-Dieter Daniel (ETH Zurich)H-Index: 32
view all 3 authors...
Background This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings Altogether, 70 reliability coefficients (Cohen's Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manus...
88 CitationsSource
#1Lutz Bornmann (ETH Zurich)H-Index: 48
#2Hans-Dieter Daniel (UZH: University of Zurich)H-Index: 32
Abstract In peer review research, no study has been performed to date that has opened the “black box” of manuscript reviewing and dealt with the internal mechanisms of the process. Using as an example the peer review system of Angewandte Chemie International Edition (AC-IE), this study investigates which review requests are assigned by the editors to external reviewers, which sequences of review steps typically occur, and which rules are used by the editors to decide whether to accept or reject ...
41 CitationsSource
#1Herbert W. MarshH-Index: 126
#2Lutz BornmannH-Index: 48
Last. Alison J O'MaraH-Index: 9
view all 5 authors...
Peer review is valued in higher education, but also widely criticized in terms of potential biases, particularly gender. We evaluate gender differences in peer reviews of grant applications, extending Bornmann, Mutz, and Daniel’s meta-analyses that reported small gender differences in favor of men (d = .04), but a substantial heterogeneity in effect sizes that compromised the robustness of their results. We contrast these findings with the most comprehensive single primary study (Marsh, Jayasing...
94 CitationsSource
Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, but neglected how epistemological differences affect perception of fairness in decision making. The normative literature suggests that evaluators should overcome their epistemological differences by ‘‘translating’’ their preferred standards i...
64 CitationsSource
18 CitationsSource
#1Herbert W. Marsh (University of Oxford)H-Index: 126
#2Lutz BornmannH-Index: 48
An extensive collaborative analysis concludes that the perception is unwarranted, say Herbert Marsh and Lutz Bornmann.
10 CitationsSource
#1Lutz Bornmann (UZH: University of Zurich)H-Index: 48
#2Hans-Dieter Daniel (UZH: University of Zurich)H-Index: 32
In journal peer review, editorial decisions on submitted manuscripts are informed by referees' expert recommendations; however, the choice of referees may affect these decisions. Using data from Angewandte Chemie International Edition (AC-IE), this study tested what would have happened if referee reports had been received in a different order. In AC-IE's peer-review process, a manuscript is generally published only if two referees rate the results of the study as important and also recommend pub...
40 CitationsSource
In this brief paper we explore the Hirsch-index together with a couple of other bibliometric parameters for the assessment of the scientific output of 29 Dutch professors in clinical cardiology. It appears that even within such a homogeneous group there is large interindividual variability. Although the differences are quite remarkable, it remains undetermined what they mean; at least it is premature to interpret them as differences in scientific quality. It goes without saying that even more pr...
32 CitationsSource
#1Stefan HornbostelH-Index: 10
#2Susan BöhmerH-Index: 4
Last. Markus von InsH-Index: 7
view all 5 authors...
The German Research Foundation’s (DFG) Emmy Noether Programme aims to fund excellent young researchers in the postdoctoral phase and, in particular, to open up an alternative to the traditional route to professorial qualification via the Habilitation (venia legendi).
38 CitationsSource
Research assessment in the UK has evolved over a quarter of a century from a loosely structured, peer-review based process to one with a well understood data portfolio and assessment methodology. After 2008, the assessment process will shift again, to the use of indicators based largely on publication and citation data. These indicators will in part follow the format introduced in 2008, with a profiling of assessment outcomes at national and international levels. However, the shift from peer ass...
71 CitationsSource
Cited By240
Quantifying the creative quality of scholarly work is a difficult challenge, and, unsurprisingly, empirical research in this area is scarce. This investigation builds on the theoretical distinction between impact (e.g., citation counts) and creative quality (e.g., originality) and extends recent work on using objective measures to assess the originality of scientific publications. Following extensive evidence from creativity research and theoretical deliberations, we operationalized multiple ind...
#1Jose A. GarcíaH-Index: 32
Last. Joaquin Fdez-ValdiviaH-Index: 13
view all 3 authors...
#1Lutz Bornmann (MPG: Max Planck Society)H-Index: 48
Last. George ChackoH-Index: 3
view all 4 authors...
Recently, Wu, Wang, and Evans (2019) proposed a new family of indicators, which measure whether a scientific publication is disruptive to a field or tradition of research. Such disruptive influence...
#1Ludo Waltman (LEI: Leiden University)H-Index: 39
#2Vincent A. Traag (LEI: Leiden University)H-Index: 12
Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoreti...
Recently, high media visibility was reached by an experiment that involved “hoaxlike deception” of journals within humanities and social sciences. Its aim was to provide evidence of “inadequate” qu...
#1Jüri Allik (UT: University of Tartu)H-Index: 49
#2Kalmer Lauk (UT: University of Tartu)H-Index: 1
Last. Anu Realo (Warw.: University of Warwick)H-Index: 38
view all 3 authors...
It has been repeatedly demonstrated that economic affluence is one of the main predictors of the scientific wealth of nations. Yet, the link is not as straightforward as is often presented. First, ...
#1Mehmet Pinar (Edge Hill University)H-Index: 7
#2Emre Unlu (Edge Hill University)
#1Tom Kwanya (Technical University of Kenya)H-Index: 4
This study investigated the publishing patterns of information science academics in Kenya. Using a bibliometric approach, the author conducted an analysis of the quantity, quality and visibility of...
2 CitationsSource
#1Margit Osterloh (UZH: University of Zurich)H-Index: 28
#2Bruno S. Frey (UZH: University of Zurich)H-Index: 91
Abstract Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles. We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem. Three extant suggestions for reforming perfo...
8 CitationsSource