The Citations Chase

The importance attached to citations as an indicator of academic excellence, prominence, impact, influence and the like is on the rise. This urgency can be termed “the citations chase.” Bibliometrics is the use of statistical methods to analyze publications or the science of citations, and activity in this area has grown considerably over the years. Citations statistics are produced, analyzed and discussed, sometimes ad nauseam. The dictum “publish or perish” has transcended into “be cited or perish.” Being published is simply not enough.

Why are Citations so Important?

Citations are important in hiring, promotion and tenure decisions, not to mention salary negotiations, funding proposals, etc. They are often considered a proxy for academic excellence, prominence, impact and influence. Certainly, they are not the only proxy, but they are becoming increasingly important.

The main trigger for this article was a number of recent bibliometric studies in which I was lucky to find my name included. Among them was a database known as the “100,000 highly cited researchers,” as described in Ioannidis et al. [1]. The actual number of scientists listed is 105,000. A colleague told me I was in that database, which to me was a pleasant surprise.

Some 6,880,389 scientists who have published five or more papers form the content of the database. Of those, the top 105,000 are selected and ranked using a composite index. Roughly 40 indices are collected for each author. The database is Scopus-based (the largest abstract and citation database of peer-reviewed literature) and breaks down scientific disciplines into 22 scientific fields and 176 subfields. Fields include broad areas such as engineering, economics and business, biology and more, and the subfields go into further detail, i.e., optoelectronics and photonics, urban and regional planning, operations research, logistics and transportation and many others. In that sense, a scientist can be ranked not only within the entire database, but also within the field and subfields in which they are associated.

Looking at the database and the accompanying paper, I was impressed that researchers in some areas get widely differing citation numbers than other areas. For instance, the 99 percentile of citations in “Nuclear and Particle Physics” is 32,708. However, the citation number for “Astronomy and Astrophysics” is 16,244, “Ecology” is 8,302 and “Drama & Theater” is only 288. For some subfields there are zero people in the top 100,000+ list. This means that it is difficult, or even futile, to compare scientists across different disciplines.

Still, digging deeper, some interesting “higher order” statistics can be calculated. Suppose a scientist has X citations. Then we may define the following:

(a) C_s, the percentage of X within the population of that person’s citations, that is ascribed to papers in which he or she is sole author.

(b) P_s, the percentage of X within the population of that person’s papers, that is ascribed to papers in which he or she is sole author.

These percentages are second-order citations statistics, being ratios of first-order statistics. They can also be defined in a broader sense as C_sf (or C_sfl), that is, percentages of X within the population of that person’s citations, that are ascribed to papers in which he or she is sole or first author (or sole, first or last author, respectively). In some disciplines the last author position is considered honorary. Similar definitions pertain to P_sf (or P_sfl).

Note that the values of C and P, as defined above, are not necessarily the same, because they are drawn from different populations. In fact, we can also define what we call the “claim to fame ratio” (for lack of a better name) as the ratio R = C/P (for each of their three variants). R is a third-order citations statistic, being the ratio of two second-order statistics. A value of R higher than 1 means that an author’s prominence is more visible within the spectrum of their citations than within the range of their papers. Note that R_s is undefined if an author has zero sole-author papers, as in this case both C_s and P_s are zero. To my surprise, there are about 895 authors in the database who have never authored a paper by themselves.

Breaking it Down

To experiment with this, I calculated the three variants of C, P and R for those scientists in the database whose primary subfields are either: (a) Logistics & Transportation or (b) Operations Research. I chose these subfields because Logistics & Transportation is my own primary subfield and Operations Research is my secondary subfield.

Logistics & Transportation has a 99-percentile citation of 1,997. According to this criterion, I was disheartened to see it ranking lower than entomology, ornithology, nursing, veterinary sciences and applied ethics, among others. But it ranks higher than civil engineering, aeronautics and astronautics and general mathematics, among others. Of the 6,880,389 researchers, 15,386 list Logistics & Transportation as their primary subfield, and 84 of them are in the top 100,000+ list. Table 1 presents the top 50 scientists on this list, together with their three R variants. (C and P statistics and secondary subfields are available in the unabridged online version of the paper.)

Table 1: Top 50 scientists who list Logistics & Transportation as their primary subfield, together with their three R variants.

The database also lists 319 scientists with operations research as their primary subfield, drawn from a group of 20,758 scientists among the 6,880,389. Operations research has a 99-percentile citation of 3,435. According to this criterion, it ranks higher than Logistics & Transportation but lower than economics, gerontology, sports sciences, optics and dentistry, among others. Table 2 lists the top 50 operations research scientists, together with the three variants of their R values. (Note. You will see many INFORMS members on either table.)

Table 2: Top 50 scientists who list operations research as their primary subfield, together with their three R variants.

I could find no discernible pattern in either table, for instance, regarding a possible correlation between the R values vis-à-vis the rank of the scientists in the database. Both tables confirm the wide diversity in citations statistics, even across classes of scientists who seem to have similar profiles. (Incidentally, I can name at least one person in either table who has passed away.)

Conclusion

In my opinion, even though bibliometrics is an interesting sport, its importance is overblown. What I think is more important is whether a paper’s content is sound, whether it has improved upon the state of the art, or whether its results are useful to science, industry or society. In addition, spending time with industry may not be reflected at all in any citation metric, even though this may be just as important professionally. (I spent 5.5 years as CEO of the Port of Piraeus, during which time I wrote zero papers.)

Editor’s note. An unabridged version of this article with more detailed considerations on this subject and with expanded tables (including all C and P values and secondary subfields) is available here.

Reference

1. Ioannidis, J.P.A., Baas, J., Klavans, R., Boyack, K.W., 2019, “A standardized citation metrics author database annotated for scientific field,” PLoS Biology, Vol. 17, No. 8, Article no. e3000384, https://doi.org/10.1371/journal. pbio.3000384.