The 14th Rothkopf Rankings: Continuing the Tradition of Measuring Academic Contributions to Operations Research Practice Literature

Introduction

The ranking of universities’ contributions to the practice literature began in 1996 when Michael Rothkopf developed an index of which universities contribute most to the practice literature. This ranking is now known as the Rothkopf Rankings in his honor, and this article represents the 14th in the series spanning 28 years. The purpose of these rankings is to identify the universities that contribute to the practice literature through applied operations research and analytics, as measured by contributions to the journal Informs Journal on Applied Analytics (formerly known as Interfaces (INT/IJAA)). There may be other journals that have practice-oriented articles, but they typically have not gone through the vetting process of INT/IJAA to ensure that the work was actually put into practice.

As Table 1 describes, Rothkopf conducted seven applied research rankings between 1996 and 2007 (Rothkopf 1996, 1997, 1999, 2002, 2004, 2005, 2007). Rothkopf’s efforts were followed by three rankings conducted by Ronald Fricker from 2009 to 2013 (Fricker 2009, 2011, 2013) and, most recently, by three rankings that I have conducted from 2016 to 2020 (Gorman 2016, 2019, 2020). Table 1 summarizes the date range evaluated for each ranking effort and the measures of research output calculated in each.

Table 1. The Rothkopf Rankings Tradition: Evolution of Methodology and Coverage

Methodological Evolution

The rankings have undergone some methodological development since their inception. Rothkopf’s original approach (Rothkopf 1996) employed a single “visibility” measure that counted institutional participation in practice-oriented publications based on a per-author basis. Fricker’s contributions (2009–2013) introduced the “yield” measure, which provided an assessment of institutional contribution on a per-article basis to address some limitations of the visibility approach. More recent contributions to the series have included the “academic yield” measure (Gorman 2016), which removes the penalty imposed by the yield measure for collaborating with practitioners. Finally, Gorman (2019) proposed a “blended ranking” approach to provide a more comprehensive assessment of institutional contributions.

In all cases, the only articles that are considered are peer-reviewed articles. Peer-reviewed columns (shorter submissions with different acceptance criteria, such as Art and Science of Practice and Practice Summaries) are included as well at one-half the value of a full-length paper. Editorials, book reviews, and other similar articles are excluded.

The original measure, created by Mike Rothkopf (1996), was later termed “visibility” by Fricker (2009) to differentiate it from his yield measure. This measure gives a single point to every author of a paper, or a half-point to every author on a column (e.g., Practice Summary, Art and Science of Practice, etc.). The total academic points for an article equal the number of academic authors on the paper. Nonacademic authors are not included in the measure and are not counted in any way. There is some inflation to this score; one article with many authors generates multiple points for various institutions, and, accordingly, if multiple authors are from the same institution, that institution gets multiple points.

Rothkopf’s idea was to capture the breadth of interest in academic research across researchers in an institution and thus provides an author-weighted measure representing the visibility of applied research at an institution. His logic was that if multiple authors participate in applied research, the research has broader awareness (i.e., “visibility”) and support in the institution.

The yield measure was created by Ron Fricker (2009) to eliminate the multiple-author inflation of the visibility measure. This measure splits a single point for an article (half-point for a column) proportionally among all the authors contributing to an article, thereby measuring the applied research yield of an institution at the research project level. Thus, if three authors from an institution produce an article, the institution is granted a single point; however, if the authors are from three institutions, each institution gets a third of a point. If one of the authors is from a nonacademic institution, the article produces only two-thirds of a yield point in total; one-third of the article is nonacademic.

Fricker’s intent with this measure was to create an article-centric count of total research productivity, eliminating the double-counting problem of the visibility measure. He provided a proportional assessment of institutional contribution to the total body of applied research.

Finally, Gorman (2016) suggested an “academic yield” measure, which follows a similar formula as Fricker’s yield but counts only academic authors in the denominator of the measure, thereby not penalizing participation of practitioners in the contribution of academics to applied research. If, as above, a third author is nonacademic, each academic author gets a half-point; each author produced one-half of the academic output.

Gorman’s logic was to remove the penalty created by the yield measure that academic authors face when collaborating with practitioners to encourage academic-practitioner collaboration by not reducing academic credit when nonacademic authors participate.

An example calculation from a fictious article is given in Table 2. For the four-author article in Table 2, University A gets visibility of 2, yield of 0.5, and academic yield of 0.67; and University B gets visibility of 1, yield of 0.25, and academic yield of 0.33.

Table 2. Example Calculations of Visibility, Yield, and Academic Yield

Of course, the sum of all the points over an interval reveals an institutional ranking. The rankings, then, vary by measure. Gorman (2019) suggested a “blended ranking,” which simply took the average rankings of each measure (visibility, yield, and academic yield) of an institution to create an overall ranking. The idea was to provide a single, unified measure that combines awareness (visibility), productivity (yield), and academic contribution (academic yield) while mitigating the individual flaws inherent in each separate measure. The comparisons of the advantages and disadvantages of each measure are discussed in Table 3.

Table 3. Rothkopf Calculation Methodology Comparison

The 13th ranking (Gorman 2020) represented a departure from the traditional seven-year rolling window to examine the complete 50-year history of Interfaces and the INFORMS Journal on Applied Analytics, providing unprecedented insight into long-term institutional commitments to and trends in applied research. As such, it is somewhat of an outlier in the Rothkopf Ranking history, but it represents a unique historical perspective on the evolution of applied research across academia.

This 14th ranking returns to the established tradition of examining a seven-year rolling window. In order to maintain comparability to the approximate three-year spacing of prior Rothkopf studies (and, frankly, to atone for dropping the ball three years ago when this analysis should have been completed!), the 14th rankings are split into two seven-year periods, 2015–2021 and 2018–2024. This approach allows for assessment of recent institutional performance, while maintaining comparability with the historical series. The ranking employs all four established measures—visibility, yield, academic yield, and blended rankings—to provide a comprehensive view of institutional contributions to the practice literature.

This analysis is particularly significant, as it captures the period following the widespread adoption of business analytics and data science programs (Gorman and Klimberg 2014) across academic institutions, allowing us to assess whether these developments have translated into any significant shifts in contributions to the practice literature.

Results

As in prior Rothkopf Ranking studies, the results of the different measures are presented individually, separating out U.S. schools and international schools. The date ranges 2015–2021 and 2018–2024 are shown side-by-side. In the interest of space and table consistency, rankings were cut off at a rank of 20, but each interval’s school list is only approximately equal in length due to schools that were tied in rank. As has been done since the advent of the Rothkopf Rankings, the data collection was manual. All articles from INT/IJAA were downloaded, and their key attributes were collected: volume, issue, page, title, author(s), contribution type (article, column, editorial, others), institution(s)/organization(s), country, and continent. Every effort to collect information accurately was made, but due to manual collection of much of the detailed data, some recording errors may have been made. Examples include authors reporting school name instead of university, authors with an unclear primary affiliation, and authors with unclear location. Of course, all errors are exclusively the responsibility of the author.

Visibility Ratings

The U.S. visibility ratings are shown in Table 4. The Georgia Institute of Technology (Georgia Tech) and the Massachusetts Institute of Technology (MIT) take the top spots in the two intervals and continue their long-time leadership roles in applied research, with the University of Alabama, the Colorado School of Mines, and Rensselaer Polytechnic Institute close behind rounding out the top 4 in the most recent ranking. The University of Tennessee, the University of Michigan, and the U.S. Air Force Academy continue to consistently rank in the top 10 in visibility.

Table 4. The U.S. Visibility Rankings

The international visibility results are shown in Table 5. There is considerably more “noise” in the international visibility rankings, with institutions moving many slots or disappearing altogether from period to period. The Eindhoven University of Technology, a long-time applied research institution, has shown a recent spike in applied research. However, the Indian Institute of Technology–Bombay has shown a surge to take the top spot in the most recent ranking. The University of Chile, a long-time leader in international applied research and the prior period number-one-ranked school, has seen its output fall off in recent years.

Table 5. The International Visibility Rankings

Yield Rankings

Of course, the number of participating authors on a single work can inflate the visibility score, which, as noted by Gorman et al. (2020), has been happening on an increasing basis on major works such as Edelman Award articles. Illustrations of the inflation effect are demonstrated by the University of Alabama, which jumped from 16th to 2nd in visibility, and Rensselaer Polytechnic, which is ranked third and previously not ranked, both due exclusively to a single, many-authored article in the most recent period. The yield rankings avoid the noise accompanying articles with a large number of authors.

The North American yield rankings are given in Table 6. Yield scores paint a slightly different picture, with the University of Dayton ranking number one from 2015–2021 and the Colorado School of Mines ranking number one more recently. Generally, these smaller schools have far fewer authors per article, creating a higher yield per article. Or, stated another way, some articles out of MIT had 9 or 10 authors across a number of academic and nonacademic institutions, where none out of the University of Dayton had more than three. Still, Georgia Tech and MIT rank in the top 4 in both periods with this measure. The University of South Carolina, the University of Michigan, and the University of Tennessee all rank in the top 10 by this measure in both periods.

Table 6. The U.S. Yield Rankings

The international yield scores are presented in Table 7. Consistent with the visibility ratings, the Indian Institute of Technology–Bombay (IIT Bombay) and the Eindhoven Institute of Technology rank in the top spot in the two periods. The National University of Singapore was the only other institution to rank in the top 10 in both periods.

Table 7. The International Yield Rankings

Academic Yield Rankings

As discussed in the introduction, the yield rankings are “penalized” by the presence of nonacademics on the author list; the academic authors only get credit for the academic portion of the article. As noted in the last ranking, yield scores have fallen in each year they have been measured, which may speak to a falloff in applied academic research, as noted in Gorman (2021), or it could be due to the participation of practitioners in the research process—a desirable situation. The academic yield measure seeks to erase that penalty; in a loose sense, academic yield scores are yield scores inflated by the rate at which an institution tends to work with practitioners.

The U.S. academic yield results are given in Table 8. Georgia Tech and MIT rise to the top ranks again, owing to the fact that in those papers in which there may have been many authors, often, there is strong practitioner representation, which hindered the yield score for both institutions. The other schools that rank in the top 10 are unchanged from the yield score, except one notable exception: Cornell University ranks in the top 10 in both periods when adjusted for practitioner participation, implying that the Cornell work had a high degree of practitioner involvement.

Table 8. The U.S. Academic Yield Rankings

The international academic yield scores are given in Table 9. The top rankings are very much the same as the yield measure, with Eindhoven ranking first in the early periods and IIT Bombay first in the second period. Tsinghua University ranked fourth in both periods, which is a different result from the yield measure in which the institution did not make the top 10.

Table 9. The International Academic Yield Rankings

Blended Rankings

The blended rankings summarize the three rankings, giving an overall look at how institutions compare across all three measures. The North American rankings are given in Table 10. MIT reestablishes its consistent contribution to applied research, unseating Georgia Tech as the most prolific institution. The Colorado School of Mines and the University of Tennessee both move up in the rankings to spots 2 and 3, respectively, with the Colorado School of Mines moving up over four spots, on average, from the last ranking. Cornell and Columbia University have the most impressive improvements in rankings, with improvements of 7.0 and 12.67 in average rank, respectively. The Naval Postgraduate School and the University of Cincinnati fall 6.67 and 5 spots in average rank, respectively.

Table 10. The U.S. Blended Rankings

The international blended rankings are in Table 11. At rank 1, and then 2, IIT Bombay has proven to be a consistent contributor to applied research for the past decade. Eindhoven jumped nearly 14 spots in average rank in the most recent period to earn the top blended ranking most recently. In this highly volatile list, less than half of the schools are on both lists, lending to the inconsistent presence of applied research at these institutions.

Table 11. The International Blended Rankings

Discussion

This set of two Rothkopf Rankings reveals some very familiar patterns, demonstrating both common threads and extreme volatility. It is clear to me after a decade of conducting these rankings and serving as Editor-in-Chief of the journal that applied research is often both random and opportunistic—based on nearly random opportunities that come before an individual researcher—and systemic—a part of the infrastructure and culture of an organization. Although policy changes, funding opportunities, and turnover may contribute to an institution’s ongoing contribution to applied research, it seems that some have cultivated and supported a continued interest, and others are sporadic at best.

Let me make my point empirically. There are 29 U.S. institutions in the most recent seven-year period listed in visibility rankings from Table 4 (the ranking metric with the longest history). By my inspection, over the 13 prior Rothkopf rating periods (including 2015–2022 of this study), about 14 institutions have previously appeared only one or zero times in the rankings. These are the institutions whose researchers likely came across a research project by happenstance and, driven by curiosity, professional pride, ego, or a desire to see their applied research in print, pursued its publication, whatever their home institution deemed its value.

On the other hand, of those same institutions, approximately the other half have appeared 50% or more of the time in prior rankings. Table 12 lists those institutions, the rank they achieved in each ranking in which they appeared, the number of times they have appeared, and their average rank when they appeared. Clearly, appearing is preferred, so lexicographically, consider the number of appearances first (Ranking Count) and the average rank second (Avg. Rank). The schools achieving a number one rank in each period are in bold. These schools should be lauded for placing an emphasis on seeing research being put into practice in a sustained and systematic way.

Table 12. Rank in Visibility Rankings Since its Inception (Seven or Greater Rankings)

For applied research to thrive consistently, it is incumbent upon the institutions themselves to create an environment in which applied research can thrive, not entrepreneurial researchers who serendipitously chance upon an interesting application. Sustained records like those exhibited in Table 12 do not come by chance.

Looking at the performance of those schools with no gaps in the rankings, the Naval Postgraduate School holds the highest average rank of all schools, followed closely by the Massachusetts Institute of Technology, the Georgia Institute of Technology, and the Colorado School of Mines. Tennessee rounds out the schools to appear in every ranking. Worthy of note: the Georgia Institute of Technology, with its recent surge in contributions to practice, has ranked first five times in total, whereas the Naval Postgraduate School and the Massachusetts Institute of Technology have each ranked first three times in this measure. The University of Tennessee and has shown dramatic improvement in rank the last decade. Despite not appearing in the first few rankings, the University of Michigan and Cornell have appeared in all rankings since the late 1990s, and South Carolina and Dayton have appeared in all rankings since the early 2000s, showing consistency in applied research in more recent years.

There is a counter-story to be told by what is missing—there are many, many schools that were regularly ranked in the early rankings but no longer appear. Whether there was a change in faculty, institutional focus, budgets, policy, or incentives is a story to be told by the individual institutions. However, such a shift of focus away from application is something I believe our profession can ill afford, yet I sense is the current trend.

Internationally, there is far less consistency in applied research, but an even more impressive, yet concentrated, commitment to it. Three schools—the University of Chile 11 times ranked with an average rank of 1.4, Erasmus University Rotterdam 10 times ranked (average rank: 5.5), and the University of British Columbia 9 times ranked (average rank: 4.55)—have appeared with any regularity. Unfortunately, all are appearing now well below their normal rank, and the University of Chile, with its seven consecutive first-place rankings leading up to this ranking, including 2022, faces its first ranking below 3 in history. The University of British Columbia has three first-place rankings, and Erasmus has two; thus, the three intuitions combine for 12 of the 14 all-time first-place rankings. However, despite their long and proud history in this area, all show a decided downward trend in applied research in recent history.

Conclusions

The Rothkopf Rankings provide important information and an important service to our discipline. The application of our science is, at its core, what give us purpose. Although institutions and business school deans perhaps value journal rankings and those that appear on “certain lists,” which invariably reduces faculty contribution to practice, it is the opinion of the author that those who put our discipline’s methods to work in practice prove our value more than appearing.

It is my hope that prospective graduate students will consider these rankings when choosing a graduate program, as will graduate programs when hiring faculty. Certainly, firms hiring employees to apply the craft would want to hire from institutions that have emphasized the art of putting theory into practice. Finally, I hope some of those schools that have been leaders in applied research in the past and have led the way will notice the absence of their intuitions on this ranking and do rejoin the fray in applied research.