Editorial: The 13th Rothkopf Rankings: All-Time Interfaces/INFORMS Journal on Applied Analytics 50-Year Anniversary Special Edition

Introduction: 50th Anniversary Edition

In 1996, Michael Rothkopf developed an index of which universities contribute most to the practice literature. This ranking is now known as the Rothkopf Rankings in his honor, and this article is the 13th in the series in 25 years. The purpose of these rankings is to identify the universities that contribute to the practice literature. Michael Rothkopf conducted seven applied research rankings between 1997 and 2007 (Rothkopf 1996, 1997, 1999, 2002, 2004, 2005, 2007), Ron Fricker provided three rankings from 2009 to 2012 (Fricker 2009, 2011, 2013), and I provided the most recent Rothkopf Rankings in Gorman (2016, 2019). Table 1 summarizes the date range evaluated for each ranking effort. Generally, these rankings represent a rolling measure of such contributions, with a seven-year history for each ranking conducted approximately every two years, with five overlapping years. More recently, they have been conducted every three years, with four overlapping years.

Table 1. The Rothkopf Rankings

This ranking is the first to consider “all-time” contributions to Interfaces (INT) and INFORMS Journal on Applied Analytics (IJAA) since its inception in 1970, a time span of 50 years (1970–2019). As such, it measures not what an institution has contributed recently, but the institutional contribution to the journal over a much longer period, the life of INT/IJAA. This all-in ranking offers a number of advantages.

• First, the original ranking (Rothkopf 1996) had a starting year of 1988; the major contributing organizations of the first 18 years of the journal had never been measured. This analysis considers every submission for the full 50-year history of the journal, from 1970 to 2019.

• Second, because the yield measure (Fricker 2009) was created with the 8th ranking and the academic yield measure (Gorman 2016) was created with the 11th ranking, they did not exist in earlier rankings. In this analysis, all three measures of contribution defined below (visibility, yield, and academic yield, as well as a blended ranking across the measures) are included for the life of the journal.

• Third, the prior time series analysis of the visibility, yield, and academic yield presented in Gorman (2019) was severely hampered by the lack of availability of data from some prior analyses as well as overlapping time periods. For example, only the top n universities had a reported rank in any single report; Gorman (2019) could only report an average rank for those institutions whose rank was reported in prior analyses. Thus, this report provides a more complete time series, eliminating those missing values.

• It has been noted in prior Rothkopf Rankings that, with a seven-year horizon, considerable shifts in the rankings can occur, which may be due largely to noise and the fortunate or unfortunate start and end points for the study. The full 50-year data set gives a measure of a long-lasting commitment to applied research over time.

• Finally, because data on the other measures are more complete, the newly proposed blended measure (Gorman 2019) can now be calculated for the entire history of the journal given the presence of a complete set of data and calculations of the other measures, which allows for consistent time series analysis.

Sample

The starting point for this analysis is every digital object identifier (DOI) ever produced in Interfaces or the INFORMS Journal on Applied Analytics (Freeman et al. 2020) in its full 50-year history since its inception in 1970 through the last issue of 2019. Over that time frame, INT/IJAA has produced 3,416 unique DOIs penned by 7,437 (not necessarily unique) authors. Table 2 identifies the breakdown of the frequency of article types in INT/IJAA. Consistent with prior rankings, this analysis will consider full-length, refereed articles and case studies, shorter contributions (such as Art and Science of Practice, Practice Abstracts, and Practice Summaries), and columns from regular contributing editors (such as 20/30 Hindsight, The Ombudsman, The Fifth Column, People Skills, All Around the Model, Creativity in MS/OR, Model World, The Easy Chair, and The Occasional Observer, among others). This subset represents 76% (2,587/3,416) of all the DOIs in the journal. Because full-length articles tend to have more authors than the various types of excluded submissions, the included DOIs constitute 85% (6,300/7,437) of the authors. Consistent with prior Rothkopf Rankings, excluded contributions from INT/IJAA include book and application reviews, editorials, announcements, contributor and editorial board lists, calendars, calls for papers, issue introductions, and the like.

Table 2. This Study Includes 2,587 (76%) Articles, Case Studies, Practice Abstracts and Summaries, and Columns Published in INT/IJAA Since Its Inception

In some cases, the academic or professional affiliation or nation could not be discerned for some authors. These authors are included in the counts above but are excluded from Rothkopf Rankings below. For example, among the included works in the rankings, 14 authors’ nationalities could not be discerned, and 4 author types (academic/practitioner) could not be determined.

Exclusions from This Study

Prior Rothkopf Rankings included articles from the OR Practice section of Operations Research and articles in Manufacturing and Services Operations Management (MSOM) and Decision Analysis (DA), but they are omitted from this analysis. Historically, the frequency of such papers is relatively low relative to the volume of papers in INT/IJAA that they are generally not material to the final rankings. Typically, Interfaces articles make up 95% of the articles in each ranking (Gorman 2019); MSOM and DA generally have few applied articles. This is more true in early years for INT/IJAA, when MSOM and OR Practice did not exist. Thus, there is little loss in comparability with prior rankings. Consistent with the “anniversary” nature of this article and given that the vast majority of applied research is in INT/IJAA, the focus of this report is on contributions to practice solely based solely on contributions to INT/IJAA.

Measures of Contribution

For all measures described below, consistent with the scoring proposed by Mike Rothkopf (Rothkopf 1996), full-length articles and case studies count as a full point, and articles and columns count as one half-point.

As described in Gorman (2019), the three scoring methods are simple but varied. Four different methods of apportioning these points have been developed over the years: visibility, yield, and academic yield, and a blended ranking based on the other three. The relative merits of these measures are discussed in detail in Gorman (2016) and are described in brief below.

1. Visibility (Rothkopf 1996): Visibility (so dubbed by Fricker 2009) is the original measure created by Rothkopf (1996). Visibility is a metric based on author participation; each author on an article gets a full point (half-point for columns and practice summaries). The sum of the yield measure is based on the total academic authors contributing to applied articles in the date range. The “visibility” is a function of the number of academics participating in the project.

2. Yield (Fricker 2009): Yield splits a single point for an article (half-point for a column) proportionally among all of the authors contributing to an article. Yield is a metric based on proportional author contribution to a work. The sum is the proportion of all the articles contributed by all authors, approximately equal to the number of articles plus one half of the smaller contributions of summaries and columns.

3. Academic Yield (Gorman 2016): Academic yield is a contribution metric based solely on the academically produced portion of the article. Academic yield follows a similar formula as in the yield measure; however, nonacademic authors do not count against the measured academic contribution to the article. The total academic points for each article are equal to one (0.5 for each column). The sum of the academic yield is thus equal to the total applied articles plus one half the number of columns in the date range. Because the Rothkopf Rankings were based on academic support of applied research, the measure captures the portion of the contribution to an article by academics without penalizing the participation of a nonacademic author.

4. Blended Measure (Gorman 2019): The blended ranking takes the average of the ranks achieved by each measure. It captures the scoring contribution of all three measures through an equal weighting of the ranks from each measure: the awareness (visibility), productivity (yield), and academic contribution (academic yield) to each article. In this ranking, complete data for INT/IJAA allow for more complete ranking data and time series analysis.

Table 3 illustrates the differences of each measure for a few illustrative publications with contributions from “University A.” University A receives one point per author with the visibility score and a proportion of a point in the yield score based on the fraction of its authors’ contribution to the article. The sum of the visibility score is the total number of academic authors, the sum of the yield score is the total proportion of the authors on a paper that are from academia, and the sum of the academic yield score is one for any article with academic authors.

Table 3. Example Scoring for “University A” Shows the Difference in the Three Measures of Contribution

The 13th Rothkopf Ranking—50th Anniversary Results

As is the norm, the Rothkopf Ratings are presented by U.S. and non-U.S. categories. Because we are including nonacademic contributions in this ranking, these, too, are split by U.S. and non-U.S. categories. Table 4 summarizes the total authors included, and the total of each score, by academic and nonacademic affiliation and U.S. and non-U.S. location. Count is the total authors contributing to this set of articles. Visibility is the author-based total point score, adjusted for the lower point values of practice abstracts and column contributions, and thus is slightly lower than the total count of authors. Yield is the article-based measure, after adjusting downward for practice and columns, and thus is slightly lower than the total number of articles in the sample.

Table 4. U.S. Academic and Nonacademic Institutions Contribute the Most to INT/IJAA

In the journal’s 50-year history, 335 unique, identifiable U.S.-based academic institutions have had authors who contributed to the journal. Over that same history, 314 unique, identifiable non-U.S. academic institutions have contributed to the journal. Table 5 shows the top 50 U.S. academic institutions, sorted by their average rank across the three measures. Table 6 shows the top 50 non-U.S. academic institutions, sorted by their average rank across the three measures.

Table 5. The Top 50 U.S. Academic Institutions for Contributions to INT/IJAA over Its History

Table 6. The Top 50 Non-U.S. Academic Institutions for Contributions to INT/IJAA over Its History

In the United States, the Colorado School of Mines is the leading contributor to the journal over its history, followed closely by Massachusetts Institute of Technology and the University of Pennsylvania. Those three institutions are ranked in the top three in all three measures of contribution. The remainder of the top 10 generally attain a ranking in the top 10 in all three rankings: Naval Postgraduate School, University of Texas at Austin, Pennsylvania State University, Stanford University, Georgia Institute of Technology, University of Southern California, and Northwestern University.

Internationally, the University of Chile dominates in all three measures over the life of the journal, and Erasmus University holds the second rank in all three measures. The University of Waterloo, the University of British Columbia, the Indian Institute of Management, and Lancaster University generally hold ranks 3 through 6 in the measures. The University of Toronto, the University of Ottawa, the University of Auckland, and Technion - Israel Institute of Technology round out the top 10.

It is noteworthy that a top 10 finish in the United States requires significantly higher absolute scores; a top 10 finish in the United States requires something on the order of 20 articles, whereas the same rank internationally requires something more like 6. Put another way, the University of Chile would have ranked 16th worldwide if U.S. schools were included; Erasmus University would have ranked 18th.

Time Series Analysis

Although a primary benefit of this study is to take the long view over the life of the journal, it is interesting to look at behaviors over time. This study allows a first-time look at consistently developed measures over the life of the journal. Figure 1 shows the top 20 U.S. schools in each of the five decades of INT/IJAA’s existence. In total, 54 different schools have been in the top 20 in any decade. Figure 2 shows the rankings of the 26 schools who have made the top 20 in two or more decades. They are sorted by frequency in the top 20. Three U.S. institutions have been in the top 20 in all five decades of the existence of INT/IJAA: the Colorado School of Mines, Massachusetts Institute of Technology, and the University of Texas at Arlington. The Colorado School of Mines was never ranked lower than sixth in any decade.

The University of Pennsylvania and the University of Southern California were in the top 20 in three decades, but the contributions have been lighter in recent decades. Seven institutions have appeared in the top 20 in three decades, with Carnegie-Mellon University and the Naval Postgraduate School making strong contributions in the last 30 years on a consistent basis.

Of the schools that have appeared in the top 20 twice, Cornell University, the Georgia Institute of Technology, the University of Dayton, and the University of Tennessee, Knoxville have shown a recent high level of contribution.

Figure 3 shows the ranking for the top 10 international schools by decade. (Ten was chosen because there are fewer international schools and generally make lower contributions; the top 20 in any decade might be 2/3 of the total contributors.) Internationally, 35 different schools made the top 10 over the five decades. Only four schools appear in the top 10 in three of the five decades: Waterloo in the 1970s through the 1990s; University of Chile, Erasmus University, and University of Auckland in the 1990s through the 2010s; and University of British Columbia in the 1970s, 1990s, and 2000s.

Measures Applied to Nonacademic Institutions

Fricker (2012) is the only attempt to measure the contribution of nonacademic institutions to the practice literature. That work focused on the contribution of nonacademic institutions to the applied literature for the years 2005–2011. As he did, I follow the methodologies described above for each institution; however, in this case, as with the academic analysis, I do so solely for INT/IJAA and for the entire 50-year life of the journal. Unlike Fricker, I restrict the articles solely to INT/IJAA articles over its 50-year history; however, as noted above, INT/IJAA makes up the vast majority of the applied analytics literature (over 95% of the scoring in Fricker (2012) came from INT/IJAA articles). Among practitioners, INT/IJAA is undoubtedly a more likely outlet for applied research.

Ron considered “parent” organizations (e.g., all of IBM, regardless of division; all of the U.S. government, regardless of which organization (Centers for Disease Control and Prevention, Environmental Protection Agency, etc.)); I do differentiate the U.S. military divisions, but as Ron did, do not differentiate the other governmental organizations. As with Fricker, I do not differentiate by U.S. and non-U.S. companies but rather provide a worldwide ranking, as many of the leading companies are in fact multinational. Consistency in naming was attempted to some degree. For example, over this large time frame, “International Business Machines” came to be known as IBM; these names were aggregated into a single company. However, merged companies, such as Santa Fe and BNSF or Chessie System and CSX, were counted separately, as were companies that were divested or spun off, such as Sabre from American Airlines. Finally, I drop the “academic yield” measure in this ranking, as it does not apply to these organizations.

In total, 1,050 nonacademic organizations have contributed to INT/IJAA, compared with 649 academic institutions. This large number is not surprising, as often there is author participation from the organization in which a project took place; many such organizations are a one-time participant in a research project in the journal. Also, many of the leading contributors are consulting and technology firms, of which there are many.

As with academic rankings, Table 7 sorts contributing organizations based on the average rank of the two remaining measures that were applied. By any measure and by a considerable margin, IBM is the top nonacademic contributor to INT/IJAA. IBM would rank ahead of any international academic institution and in the top five of U.S. academic institutions for their contributions.

Table 7. The Top Contributing Nonacademic Institutions in the World Since the Journal’s Inception

The close second biggest contributor is the U.S. government, which benefits in this method of scoring from the aggregation of its many departments and agencies. Of the 16 different U.S. governmental agencies included in that category, major contributing organizations include the Centers for Disease Control and Prevention, U.S. Environmental Protection Agency, Energy Information Administration, U.S. Postal Service, and Department of Energy, among others. Many of these would have been in the top 50 based on their own individual contributions.

General Electric (GE) rounds out the top three. GE and the U.S. government share second and third rankings, depending on the measure. AT&T, the U.S. Army, Hewlett Packard, and American Airlines fall approximately into the fourth through seventh rankings. Merrill Lynch, Intel, General Motors, and Xerox round out the top 10.

Discussion

This anniversary edition of the Rothkopf Rankings is the first to produce all three measures of research production developed over the last 30 years (visibility, yield, and academic yield), with a blended ranking between them, for the entire history of the journal. A number of observations can be made. Although we measure institutional contributions to the journal as a way to measure that support, it seems clear that such contributions are ephemeral and highly variable. Of course, what we are truly measuring is the interest and the ability of faculty at those schools (or employees in those organizations, in the case of nonacademic institutions) to pursue applied research, and only secondarily are we suggesting what the institutional support for that research must be in order for it to be sustained. Of course, the level of applied research can come and go over time as faculty are hired, retire, or change positions.

Nonetheless, there are some longtime institutional supporters of applied research identified. The Colorado School of Mines must be commended above all others for the consistency and quality of its support for applied research. Massachusetts Institute of Technology and the University of Texas at Austin similarly so. The University of Pennsylvania has been nearly as supportive. Recently, the Georgia Institute of Technology, Cornell, and the Naval Postgraduate School have been strong supporters.

Internationally, there are generally lower total contributions to INT/IJAA and, therefore, some additional variability to the flow of applied research from around the world. The University of Chile, Erasmus University, and Auckland University should be commended for outstanding applied research in recent decades. Longstanding support for applied research can be seen from schools such as the University of Waterloo, the University of British Columbia, the Indian Institute of Management, Lancaster University, the University of Toronto, and the University of Ottawa.

It is a pleasure to see the high level of contributions from nonacademic institutions in the journal. Although the Rothkopf Rankings were originally conceived to measure academic focus on applied research, the measures can also be used to understand the extent of analytics in practitioner organizations that are willing to share them. IBM, the U.S. government and armed forces, GE, and many other organizations have shown the willingness and ability to conduct advanced analytical projects and then share them with the world.

For the INFORMS Journal on Applied Analytics and for analytics research as a whole, the need to demonstrate our tools in practice is essential to prove the value of what we do and continue to evolve the discipline to more advanced and useful levels.