Match!

Mapping the evaluation results between quantitative metrics and meta-synthesis from experts’ judgements: evidence from the Supply Chain Management and Logistics journals ranking

Published on Feb 14, 2019
· DOI :10.1007/s00500-019-03837-3
Lili Yuan2
Estimated H-index: 2
(CAS: Chinese Academy of Sciences),
Jianping LiXiaolei20
Estimated H-index: 20
(CAS: Chinese Academy of Sciences)
+ 2 AuthorsDengsheng Wu12
Estimated H-index: 12
(CAS: Chinese Academy of Sciences)
Abstract
Meta-syntheses from experts’ judgements and quantitative metrics are two main forms of evaluation. But they both have limitations. This paper constructs a framework for mapping the evaluation results between quantitative metrics and experts’ judgements such that they may be solved. In this way, the weights of metrics in quantitative evaluation are objectively obtained, and the validity of the results can be testified. Weighted average percentile (WAP) is employed to aggregate different experts’ judgements into standard WAP scores. The Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) method is used to map quantitative results into experts’ judgements, while WAP scores are equal to the final closeness coefficients generated by the TOPSIS method. However, the closeness coefficients of TOPSIS rely on the weights of quantitative metrics. In this way, the mapping procedure is transformed into an optimization problem, and a genetic algorithm is introduced to search for the best weights. An academic journal ranking in the field of Supply Chain Management and Logistics (SCML) is used to test the validity obtained by mapping results. Four prominent ranking lists from Association of Business Schools, Australian Business Deans Council, German Academic Association for Business Research, and Comite National de la Recherche Scientifique were selected to represent different experts’ judgements. Twelve indices including IF, Eigenfactor Score (ES), H-index, Scimago Journal Ranking, and Source Normalized Impact per Paper (SNIP) were chosen for quantitative evaluation. The results reveal that the mapping results possess high validity for the relative error of experts’ judgements, the quantitative metrics are 43.4%, and the corresponding best weights are determined in the meantime. Thus, some interesting findings are concluded. First, H-index, Impact Per Publication (IPP), and SNIP play dominant roles in the SCML journal’s quality evaluation. Second, all the metrics are positively correlated, although the correlation varies among metrics. For example, ES and NE are perfectly, positively correlated with each other, yet they have the lowest correlation with the other metrics. Metrics such as IF, IFWJ, 5-year IF, and IPP are highly correlated. Third, some highly correlated metrics may perform differently in quality evaluation, such as IPP and 5-year IF. Therefore, when mapping the quantitative metrics and experts’ judgements, academic fields should be treated distinctively.
  • References (42)
  • Citations (1)
References42
Newest
#1Y. V. Srinivasa Murthy (KREC: National Institute of Technology, Karnataka)H-Index: 3
#2Shashidhar G. Koolagudi (KREC: National Institute of Technology, Karnataka)H-Index: 16
6 CitationsSource
#1Soumen Atta (KGEC: Kalyani Government Engineering College)H-Index: 2
#2Priya Ranjan Sinha Mahapatra (KGEC: Kalyani Government Engineering College)H-Index: 6
Last.Anirban Mukhopadhyay (KGEC: Kalyani Government Engineering College)H-Index: 23
view all 3 authors...
7 CitationsSource
#1Dengsheng Wu (CAS: Chinese Academy of Sciences)H-Index: 12
#2Jing Li (Anda: Anhui University)H-Index: 2
Last.Jianping LiXiaolei (CAS: Chinese Academy of Sciences)H-Index: 20
view all 4 authors...
6 CitationsSource
#1Jianping LiXiaolei (CAS: Chinese Academy of Sciences)H-Index: 20
#2Xiaoyang Yao (CAS: Chinese Academy of Sciences)H-Index: 3
Last.Dengsheng Wu (CAS: Chinese Academy of Sciences)H-Index: 12
view all 4 authors...
17 CitationsSource
2 CitationsSource
#1Jing Li (CAS: Chinese Academy of Sciences)H-Index: 2
#2Dengsheng Wu (CAS: Chinese Academy of Sciences)H-Index: 12
Last.Minglu Li (National Natural Science Foundation of China)H-Index: 7
view all 4 authors...
12 CitationsSource
8 CitationsSource
#1Edward C. Rosenthal (TU: Temple University)H-Index: 11
#2Howard J. Weiss (TU: Temple University)H-Index: 12
4 CitationsSource
#1Khalid Mahmood (University of the Punjab)H-Index: 20
4 CitationsSource
#1Dejian Yu (Zhejiang University of Finance and Economics)H-Index: 19
#2Wanru Wang (Zhejiang University of Finance and Economics)H-Index: 2
Last.Rongyu Liu (Zhejiang University of Finance and Economics)H-Index: 3
view all 5 authors...
16 CitationsSource
Cited By1
Newest
#1Jing Li (SYSU: Sun Yat-sen University)
#2Xiaoli LuH-Index: 1
Last.Dengsheng Wu (CAS: Chinese Academy of Sciences)H-Index: 12
view all 4 authors...
Source
View next paperQuality metrics for information systems