The use and misuse of journal metrics and other citation indicators

Published on Feb 1, 2009in Archivum Immunologiae Et Therapiae Experimentalis2.878
· DOI :10.1007/s00005-009-0008-y
David A. Pendlebury2
Estimated H-index: 2
(Thomson Reuters)
This article reviews the nature and use of the journal impact factor and other common bibliometric measures for assessing research in the sciences and social sciences based on data compiled by Thomson Reuters. Journal impact factors are frequently misused to assess the influence of individual papers and authors, but such uses were never intended. Thomson Reuters also employs other measures of journal influence, which are contrasted with the impact factor. Finally, the author comments on the proper use of citation data in general, often as a supplement to peer review. This review may help government policymakers, university administrators, and individual researchers become better acquainted with the potential benefits and limitations of bibliometrics in the evaluation of research.
  • References (68)
  • Citations (139)
📖 Papers frequently viewed together
5,164 Citations
1 Author (Eugene Garfield)
1,507 Citations
1 Author (Per Ottar Seglen)
1,759 Citations
78% of Scinapse members use related papers. After signing in, all features are FREE.
#1Michel Zitt (INRA: Institut national de la recherche agronomique)H-Index: 19
#2Henry Small (Thomson Reuters)H-Index: 31
A new approach to the field normalization of the classical journal impact factor is introduced. This approach, called the audience factor, takes into consideration the citing propensity of journals for a given cited journal, specifically, the mean number of references of each citing journal, and fractionally weights the citations from those citing journals. Hence, the audience factor is a variant of a fractional citation-counting scheme, but computed on the citing journal rather than the citing ...
173 CitationsSource
#1Marcus A. Banks Mlis (UCSF: University of California, San Francisco)H-Index: 5
#2Robert P. Dellavalle (VA: United States Department of Veterans Affairs)H-Index: 45
Purpose – This paper aims to document the proliferating range of alternatives to the impact factor that have arisen within the past five years, coincident with the increased prominence of open access publishing.Design/methodology/approach – This paper offers an overview of the history of the impact factor as: a measure for scholarly merit; a summary of frequent criticisms of the impact factor's calculation and usage; and a framework for understanding some of the leading alternatives to the impac...
20 CitationsSource
#1Matthew E. Falagas (Tufts University)H-Index: 93
#2Vasilios D. Kouranos (American Institute of Biological Sciences)H-Index: 3
Last. Drosos E. Karageorgopoulos (American Institute of Biological Sciences)H-Index: 38
view all 4 authors...
The application of currently available sophisticated algorithms of citation analysis allows for the incorporation of the “quality” of citations in the evaluation of scientific journals. We sought to compare the newly introduced SCImago journal rank (SJR) indicator with the journal impact factor (IF). We retrieved relevant information from the official Web sites hosting the above indices and their source databases. The SJR indicator is an open-access resource, while the journal IF requires paid s...
257 CitationsSource
A rich and diverse set of potential bibliometric and scientometric predictors of research performance quality and importance are emerging today, from the classic metrics (publication counts, journal impact factors and individual article/author citation counts) to promising new online metrics such as download counts, hub/authority scores and growth/decay chronometrics. In and of themselves, however, metrics are circular: They need to be jointly tested and validated against what it is that they pu...
81 CitationsSource
#1Linda Butler (ANU: Australian National University)H-Index: 18
Australia is about to move to a new system of distributing government block grants for research among universities, with the introduction of a process similar to Britain's Research Assess- ment Exercise (RAE). One of the most significant departures from the current RAE model is that, in the Australian Research Quality Framework, peer judgements will be informed by quantitative per- formance measures, including bibliometrics. The data will not be used in any formulaic way, but will sit alongside ...
75 CitationsSource
#1Michel Zitt (INRA: Institut national de la recherche agronomique)H-Index: 19
#2Elise BassecoulardH-Index: 13
Scientometrics and bibliometrics are being forced to respond to a strong increase in demand (e.g. research assessment practices, economics of science & technology, and innovation) and new forms of supply (e.g. availability of publication sources and statistics, Internet developments and online tools). This situation results in contrasting perspectives: on the one hand, it can favour spectac- ular 'hit-parades' and some veneration of numbers; on the other hand, it paves the way for more cau- tiou...
35 CitationsSource
Traditionally, the most commonly used source of bibliometric data is Thomson ISI Web of Knowledge, in particular the Web of Science and the Journal Citation Reports (JCR), which provide the yearly Journal Impact Factors (JIF). This paper presents an alternative source of data (Google Scholar, GS) as well as 3 alternatives to the JIF to assess journal impact (h-index, g-index and the number of citations per paper). Because of its broader range of data sources, the use of GS gen- erally results in...
421 CitationsSource
#1Lutz Bornmann (ETH Zurich)H-Index: 48
#2Rüdiger Mutz (ETH Zurich)H-Index: 23
Last. Hans-Dieter Daniel (ETH Zurich)H-Index: 32
view all 4 authors...
With the ready accessibility of bibliometric data and the availability of ready-to-use tools for generating bibliometric indicators for evaluation purposes, there is the danger of inappropri- ate use. Here we present standards of good practice for analyzing bibliometric data and presenting and interpreting the results. Comparisons drawn between research groups as to research perfor- mance are valid only if (1) the scientific impact of the research groups or their publications are looked at by us...
236 CitationsSource
#1Teerasak Markpin (King Mongkut's University of Technology Thonburi)H-Index: 12
#2Boonrat Boonradsamee (King Mongkut's University of Technology Thonburi)H-Index: 2
Last. Narongrit Sombatsompop (King Mongkut's University of Technology Thonburi)H-Index: 29
view all 7 authors...
This article proposed a new index, so-called “Article-Count Impact Factor” (ACIF) for evaluating journal quality in light of citation behaviour in comparison with the ISI journal impact factors. The ACIF index was the ratio of the number of articles that were cited in the current year to the source items published in that journal during the previous two years. In this work, we used 171 journal titles in materials categories published in the years of 2001–2004 in international journals indexed in...
10 CitationsSource
#1Michael J. Stringer (NU: Northwestern University)H-Index: 5
#2Marta Sales-Pardo (NU: Northwestern University)H-Index: 20
Last. Luís A. Nunes Amaral (NU: Northwestern University)H-Index: 59
view all 3 authors...
Background The rise of electronic publishing [1], preprint archives, blogs, and wikis is raising concerns among publishers, editors, and scientists about the present day relevance of academic journals and traditional peer review [2]. These concerns are especially fuelled by the ability of search engines to automatically identify and sort information [1]. It appears that academic journals can only remain relevant if acceptance of research for publication within a journal allows readers to infer i...
114 CitationsSource
Cited By139
#1Paul A. Phillips (UKC: University of Kent)H-Index: 19
#2Stephen J. Page (University of Hertfordshire)H-Index: 41
Last. Joshua Sebu (University of Cape Coast)H-Index: 1
view all 3 authors...
Abstract The marketisation of higher education has emerged as a global trend with a focus on using metrics to assess performance. This has led to the closer scrutiny by government in assessing value for money and effectiveness of research outcomes in national allocations of research funding. This paper focuses on one controversial strand of assessing research outcomes - the area of research impact. The paper examines the experiences of the UKs Research Assessment Exercise in 2014 and the tourism...
#1Thomas M. Krueger (A&M: Texas A&M University)H-Index: 2
#2Anne-Marie T. Lelkes (A&M: Texas A&M University)
Last. Jack D. Shorter (Texas A&M University–Kingsville)
view all 3 authors...
Research quality is often based on the perceived quality of journals in which the research appears. Journal quality is subjectively based on a wide array of objective journal characteristics, inclu...
#1Ludo WaltmanH-Index: 39
#2Vincent A. Traag (LEI: Leiden University)H-Index: 12
Most scientometricians reject the use of the journal impact factor for assessing individual articles and their authors. The well-known San Francisco Declaration on Research Assessment also strongly objects against this way of using the impact factor. Arguments against the use of the impact factor at the level of individual articles are often based on statistical considerations. The skewness of journal citation distributions typically plays a central role in these arguments. We present a theoreti...
#1Nathaniel S. Marshall (USYD: University of Sydney)H-Index: 28
#2Camilla M. Hoyos (USYD: University of Sydney)H-Index: 13
Impact factors and ranking lists of research journals are very often used to judge our career achievements and progression by employers and granting bodies. However, a comprehensive list for the interdisciplinary field of sleep research does not currently exist because our journals tend to be placed into discipline-specific lists that do not cope well with our historic interdisciplinarity, which embraces many core disciplines. We aimed to build a ranking list specifically for sleep research jour...
In this paper, Stamatoula Pylarinou with her supervisor Prof. Sarantos Kapidakis reports on an analysis of bibliographic data of the publications of Greek hospital personnel, conducted as part of Stamatoula's doctoral research in the Department of Archive, Library and Museum Sciences at the Ionian University in Corfu, Greece. Using freely available data, they demonstrate the questions posed and the insights gained from the analysis of the scientific publications of personnel of public hospitals ...
#1D. KoelblingerH-Index: 2
#2Georg Zimmermann (University of Salzburg)H-Index: 4
Last. Tobias KiesslichH-Index: 24
view all 4 authors...
Analyses of the Journal Impact Factor (JIF) have grown to be a major topic in scientometric literature. Despite widespread and justified critique concerning the JIF and its application, the size of a journal as a predictor for its longitudinal variability–or stability–on a long-term level has not yet comprehensively been analyzed. This study aims to provide robust evidence for an association between JIF variability and the size of journals, expressed by the number of published articles (citable ...
#1Giovanni Abramo (National Research Council)H-Index: 28
#2Ciriaco Andrea D’Angelo (University of Rome Tor Vergata)H-Index: 28
Last. Emanuela Reale (National Research Council)H-Index: 13
view all 3 authors...
In this work, we try to answer the question of which method, peer review versus bibliometrics, better predicts the future overall scholarly impact of scientific publications. We measure the agreement between peer review evaluations of Web of Science indexed publications submitted to the first Italian research assessment exercise and long-term citations of the same publications. We do the same for an early citation-based indicator. We find that the latter shows stronger predictive power, i.e. it ...
1 CitationsSource
Abstract In this article it is presented a bibliometric analysis of e-government research in the Ibero-American (IA) Community. Data from the Scopus® database relating to 1129 research documents published between 2003 and 2017 was used. Presented analyzes include the most productive and impacting researchers, institutions and countries; determinants for country results; most relevant subject areas and specific research themes; and international cooperation patterns, namely within the IA Communit...
Eugene Garfield advanced the theory and practice of information science and envisioned information systems that made the discovery of scientific information much more efficient. The Institute for Scientific Information (ISI), which he founded in Philadelphia in 1960, developed innovative information products that have revolutionized science. ISI provided current scientific information to researchers all over the world by publishing the table of contents of key scientific journals in the journal ...
#1Rabishankar Giri (Presidency University, Kolkata)H-Index: 3
Purpose The purpose of this paper is to explore the influence of selected factors in journal citations. Various factors can affect citations distribution of journals. Among them, skewness of citations distribution, author self-citation, journal self-citation and recitations (RCs) have been studied. Design/methodology/approach The present study based on 16 systematically selected journals indexed in Scopus under the subject category “Library and Information Science.” The study was confined to ori...
1 CitationsSource