Drawbacks of Normalization by Percentile Ranks in Citation Impact Studies

引用影響力研究中以百分等級正規化之缺點

10.6182/jlis.202212_20(2).075

Paul Donner

Citation Normalization ； Field Normalization ； Percentile Ranks ； Ordinal Data ； 引用正規化 ； 領域正規化 ； 百分等級 ； 次序尺度

圖書資訊學刊

20卷2期（2022 / 12 / 01）

75 - 93

英文

This paper discusses drawbacks of the percentile rank method for citation impact normalization which have hitherto been neglected in the bibliometrics literature. The transformation of citation counts to percentile ranks changes ratio scale data into ordinal scale data, for which the notions of the ratio between two values and of the magnitude of a difference between two values are not defined - a substantial loss of information. This distorts citation data particularly severely because the differences between citation counts adjacent in order in publication sets are greater for more highly cited publications and because highly cited publications are more scarce than non-highly cited ones. Moreover, arithmetic operations on ordinal scale data are not meaningful, which rules out arithmetic aggregations such as sums or averages for percentile rank data which are sometimes recommended in the literature. Distortion of citation data by aggregating percentile ranks for average impact indicators is demonstrated with several examples.

本文探討書目計量文獻中常被忽略之百分等級方法在引用影響力正規化上的缺點。此方法將引用次數轉換為百分等級，使數據由比率尺度轉變成次序尺度。然而，未定義兩值間的比率及兩值間的差異大小易導致重要資訊遺漏。由於在文獻集合中，以引用次數排序時，高被引文獻與其排序相鄰的文獻引用次數落差極大，且高被引文獻相較於非高被引文獻數量更為稀少，因而嚴重地扭曲了引用數據。此外，算術運算在次序尺度資料中是沒有意義的，這也排除了某些文獻所推薦的運算方式，如：用百分等級數據計算總和或是平均。本文以數個案例說明百分等級運算用於影響力指標將扭曲引用數據。

1. Adams, J.,Gurney, K.,Marshall, S.(2007).Profiling citation impact: A new methodology.Scientometrics,72(2),325-344.
2. Aksnes, D. W.,Sivertsen, G.(2004).The effect of highly cited papers on national citation indicators.Scientometrics,59(2),213-224.
3. Antonoyiannakis, M.(2018).Impact factors and the central limit theorem: Why citation averages are scale dependent.Journal of Informetrics,12(4),1072-1088.
4. Antonoyiannakis, M.(2020).Impact factor volatility due to a single paper: A comprehensive analysis.Quantitative Science Studies,1(2),639-663.
5. Bornmann, L.(2013).How to analyze percentile citation impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes, and top-cited papers.Journal of the American Society for Information Science & Technology,64(3),587-595.
6. Bornmann, L.,Williams, R.(2020).An evaluation of percentile measures of citation impact, and a proposal for making them better.Scientometrics,124,1457-1478.
7. D’Agostino, M.,Dardanoni, V.,Ricci, R. G.(2017).How to standardize (if you must).Scientometrics,113(2),825-843.
8. Hicks, D.,Wouters, P.,Waltman, L.,de Rijcke, S.,Rafols, I.(2015).Bibliometrics: The Leiden Manifesto for research metrics.Nature,520(7548),429-431.
9. Ioannidis, J. P.,Boyack, K.,Wouters, P. F.(2016).Citation metrics: A primer on how (not) to normalize.PLoS Biology,14(9),Article e1002542.
10. Kotz, S.(Ed.),Balakrishnan, N.(Ed.),Read, C. B.(Ed.),Vidakovic, B.(Ed.)(2006).Encyclopedia of statistical sciences.John Wiley & Sons.
11. Leydesdorff, L.,Bornmann, L.(2011).Integrated impact indicators compared with impact factors: An alternative research design with policy implications.Journal of the American Society for Information Science & Technology,62(11),2133-2146.
12. Leydesdorff, L.,Bornmann, L.,Adams, J.(2019).The integrated impact indicator revisited (I3*): A non-parametric alternative to the journal impact factor.Scientometrics,119(3),1669-1694.
13. Leydesdorff, L.,Bornmann, L.,Mutz, R.,Opthof, T.(2011).Turning the tables on citation analysis one more time: Principles for comparing sets of documents.Journal of the American Society for Information Science & Technology,62(7),1370-1381.
14. Lundberg, J.(2007).Lifting the crown—Citation z-score.Journal of Informetrics,1(2),145-154.
15. Mcallister, P. R.,Narin, F.,Corrigan, J. G.(1983).Programmatic evaluation and comparison based on standardized citation scores.IEEE Transactions on Engineering Management,EM–30(4),205-211.
16. Mutz, R.,Daniel, H.-D.(2012).Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor.Journal of Informetrics,6(2),169-176.
17. Schubert, A.,Braun, T.(1986).Relative indicators and relational charts for comparative assessment of publication output and citation impact.Scientometrics,9(5/6),281-291.
18. Seglen, P. O.(1992).The skewness of science.Journal of the American Society for Information Science,43(9),628-638.
19. Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677-680. https://doi.org/10.1126/science.103.2684.677
20. Thelwall, M.(2016).The discretised lognormal and hooked power law distributions for complete citation data: Best options for modelling and regression.Journal of Informetrics,10(2),336-346.
21. Waltman, L.(2016).A review of the literature on citation impact indicators.Journal of Informetrics,10(2),365-391.
22. Waltman, L.,Calero-Medina, C.,Kosten, J.,Noyons, E. C. M.,Tijssen, R. J. W.,van Eck, N. J.,van Leeuwen, T. N.,van Raan, A. F. J.,Visser, M. S.,Wouters, P.(2012).The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation.Journal of the American Society for Information Science & Technology,63(12),2419-2432.
23. Waltman, L.,van Eck, N. J.(2019).Field normalization of scientometric indicators.Springer handbook of science and technology indicators
24. Waltman, L.,van Eck, N. J.,van Leeuwen, T. N.,Visser, M. S.,van Raan, A. F. J.(2011).Towards a new crown indicator: Some theoretical considerations.Journal of Informetrics,5(1),37-47.
25. Zhang, Z.,Cheng, Y.,Liu, N. C.(2015).Improving the normalization effect of mean-based method from the perspective of optimization: Optimization-based linear methods and their performance.Scientometrics,102(1),587-607.
26. Zhou, P.,Zhong, Y.(2012).The citation-based indicator and combined impact indicator—New options for measuring impact.Journal of Informetrics,6(4),631-638.