Op-ed: Paper killers are made, not born

Journal publications are essential for the INFORMS community to create and disseminate knowledge for society at large.

In recent years, INFORMS has made great strides in the publications department under the former and current leadership. Walls and Smith (2021) reported that the 17 INFORMS journals received nearly 10,000 submissions and accepted more than 1,200 papers in 2020 [1]. It is gratifying to learn from Matt Walls (director of publications) and Cole Smith (current VP of publications) that INFORMS is making great efforts to cut down on article review and production time. It is also important to note that all INFORMS journals continue to publish excellent research articles.  

These are great steps forward, but the INFORMS community needs to rethink the review process if we are committed to encouraging our colleagues to tackle “big problems” [2], increasing the impact of INFORMS publications on researchers (including in other fields beyond the INFORMS community), practitioners and policymakers, and improving the impact factors of our journals. (I believe that a journal’s impact factor should never be its goal, but it is a highly visible metric that we cannot ignore.)

My Motivation

To provide some context, several of my younger colleagues asked me to write an article based on my personal journey. I joined ORSA and TIMS (the predecessors of INFORMS) almost 40 years ago, and innumerable INFORMS members helped me publish articles in various INFORMS journals, including Interfaces (now INFORMS Journal on Applied Analytics), Management Science, Manufacturing & Service Operations Management, Marketing Science, Operations Research and Service Science. Because I can never repay them, I am writing this article with the hope to facilitate further discussion that can shape the future of INFORMS’ publications.  

Since 1990, I have had the privilege to serve on various editorial boards for numerous INFORMS journals and process thousands of papers submitted to them. For more than three decades I have played different roles (author, reviewer, associate editor, department editor and editor-in-chief), and I have observed a gradual change in the reviewer reports that I think our community needs to address. This issue is not new, as Brian Denton has asked more reviewers to become “coaches” [3].

Transition from Manual to Online Submission

Until 2000, paper submissions and reviewer reports were sent through snail mail and the turnaround time was approximately 4-6 months. Despite long review cycles, each reviewer’s report was usually two pages long, with 5-6 major suggestions. In my experience, most reviews were constructive; the reviewers put themselves in the author’s shoes when making reasonable suggestions. Authors could then write to the reviewers (through the department editor) to clarify some of the suggestions. With the help of many anonymous “coaches” – my thesis advisor (Eric Denardo), my senior colleague (Arthur Geoffrion) and numerous co-authors – I managed to publish some articles in various INFORMS journals.

When INFORMS adopted Manuscript Central for manuscript support in early 2000 (the predecessor of the current ScholarOne software), everyone was excited because authors could track the progress of their submissions, reviewers could see all reports, and the editor could check for compliance with submission guidelines all in one place and without use of the postal service. 

So, has the online submission system helped INFORMS reduce review cycle time and improve review quality?

The outcome, in my opinion, is mixed. Yes, the online system helped reduce the average review cycle time from 4-6 months to roughly 80 days for some INFORMS journals in 2020. However, many other INFORMS journals continued to exhibit long review cycle times that exceeded 100 days. 

At the same time, the number of cycles has increased, especially as “reject and resubmit” decisions became more common. Consequently, like everyone else, some of my joint papers were accepted only after multiple rounds of resubmission. At the same time, other papers were rejected after multiple rounds (i.e., after 4-5 years).

Not only was the number of cycles increasing, the number of major concerns raised by the reviewers was also increasing. (I received a record-breaking report with 21 major concerns; 8-10 major concerns are common these days.) Authors normally receive five reports (three from reviewers, one each from the associate editor and department editor) that are usually 15 pages long in total. In many instances, the department editor asks the authors to address all major concerns, even though some of the suggestions made by different reviewers are contradictory. In some cases, original reviewers drop off and new reviewers are added, resulting in inconsistent suggestions at different cycles.

Junior colleagues of mine are eager to continue with revisions, but it saddens me to see that their efforts are futile when their papers are eventually rejected after many rounds. (I had a joint paper that was 85 pages long rejected after five rounds, with nine different extensions requested by different reviewers, and our response letter was more than 80 pages long.) Even when some papers eventually get accepted, these papers often become “Frankenstein papers” that rarely get noticed.

Based on comments I’ve received from authors, this kind of review process is not uncommon. In fact, many authors lamented that the review process for emerging areas (e.g., COVID-19 pandemic-related research, fintech operations, innovative operations, etc.) takes a long time to converge because reviewers usually have diverging views about the research questions, methodologies and scope. By the time some of these papers get published a few years later, they will have limited impact because the context may no longer be relevant. Consequently, the lengthy review process may have hampered the visibility and impact of those papers that examine timely topics.

Worse, this kind of exhausting review process can make some young scholars become paper killers themselves when they either ultimately pull their paper or in turn become paper-killing reviewers instead of the preferred coaches.

Paper Killers Are Made

Over the last three decades, I have discussed the journal review process with hundreds of students and junior colleagues and learned about the “dark side” of our relatively transparent online submission system. First, some reviewers told me that after seeing those lengthy reports and recommendations from other reviewers, they feel a subtle pressure to expand their reports and make tougher recommendations to avoid being seen by the editors as “too soft.”

Second, some reviewers believe that they were mistreated by others during the review process when they were authors. Hence, it is justifiable for them as reviewers to treat authors the same way they were treated by reviewers, especially when so many OR/OM researchers compete for publishing in so few top journals [4]. Consequently, “checklist reviewers” are born (reviewers who make complaints or meaningless suggestions without considering the research context).

Some checklist reviewers criticize things that the authors have not done in the paper. At the same time, these reviewers downplay the value of what the authors have done. Other checklist reviewers do not give the authors the benefit of the doubt, and they often demand that authors provide real evidence to justify every single assumption. 

As an illustrative example, consider the case when U.K. Prime Minister Boris Johnson announced in early January 2021 that, due to limited supply of Oxford-AstraZeneca vaccine, the time between the doses would be “stretched” from three weeks to 12 weeks. Motivated by this controversial stretching policy in the U.K. and by the fact that the United States was facing a limited supply of COVID-19 vaccines at that time, some researchers developed an analytical inventory model supported by an epidemiological simulation model to examine the effectiveness of the U.K.’s stretching policy.

To calibrate the model parameters for the simulation study, the authors used publicly available Israeli data that was based on the use of the Pfizer-BioNTech COVID-19 vaccine to curb the spread of the original SARS-CoV-2 and alpha variant of the coronavirus. However, after a lengthy review process, some checklist reviewers may request that the authors consider the delta variant of the virus that has become more prevalent even though the delta variant was not even known when the paper was submitted. Other checklist reviewers may argue that the data from Israel was limited to the Pfizer-BioNTech vaccine. As such, the authors should conduct their study by considering the Sputnik V vaccine even though verifiable data associated with this vaccine is not easily accessible. With this kind of checklist review process, how many researchers would dare tackle big and time-sensitive problems?

These circumstances have created a vicious cycle. To ensure the future success of INFORMS journals, it is time to rethink our review process. 

A Rethink: From Paper Killers to Coaches

To encourage researchers to work on big problems so that INFORMS has a greater impact on society at large, I believe we need to rethink the review process to develop more coaches. 

One basic belief I have is that all papers have holes. Instead of having reviewers to develop a checklist of major concerns, it would be more constructive if reviewers would prepare their reports by answering three simple questions:

1. Is the paper addressing relevant questions in the given context?
2. Are there critical missing elements that the paper must contain in the given context?
3. Is the analysis correct?

Based on these basic ideas, I launched the following initiatives during my term as editor-in-chief of M&SOM from 2015 to 2020, and the results appear to be promising.

First, I personally agree with Spiegel (2012) that no paper is perfect [5]. Therefore, making authors jump through endless hoops can be discouraging and may not help to advance knowledge at a faster pace. To shorten the review process, M&SOM has had a general policy since 2015 that the final decision is made after two review cycles, with each cycle reviewed by two reviewers.

For example, recognizing Cachon et al. (2017) was one of the first operations management (OM) research articles that examines the implications of Uber’s surge pricing strategy [6], I accepted the paper after two major revisions even though some reviewers had additional suggestions. I personally believe that a paper that examines a novel issue should leave room for others to explore further. This paper was well recognized by the OM community as well as transportation researchers and economists. 

Second, I also believe that, in addition to publishing research articles based on rigorous analysis, a journal should quickly publish some short thought pieces that deal with timely issues. In doing so, these short pieces can shape research agenda. To reduce checklist reviews so that the review process can be shortened further, M&SOM developed a reviewer form in 2020 by asking each reviewer to suggest up to three major comments [7].

For example, during the early stage of the COVID-19 pandemic, the exact transmission mechanism and transmission rate were unknown, and yet universities and supermarkets needed to develop crowd-control measures and hospitals needed to plan for intensive care unit capacity. With very limited data, Ed Kaplan submitted an OM Forum paper based on a scratch model along with some sensitivity analysis [8]. I recognized the time urgency and the fact that such a paper could spark discussion for administrators to think through the capacity planning process during the pandemic. As such, I enlisted two reviewers to review the paper in just two days, and the paper was accepted in just five days. This paper was well received by healthcare and OM researchers.

Clearly, by shortening the review process and reducing checklist reviews, M&SOM may have accepted some low-quality papers. However, at the same time, it may have enticed more authors to consider M&SOM as their preferred outlet to quickly share their researching findings. Based on feedback received from authors, reviewers, associate editors and department editors, these initiatives forced them to focus on issues that truly matter so that authors could focus on addressing important issues without creating more Frankenstein papers.

If more journals could make these positive changes to reduce the number of review cycles, as well as reduce the usage of “reject and resubmit” decisions, number of reviewers and length of reviewer reports, we could nurture more coaches. Also, because reviewers are volunteers, journal editors should encourage associate editors to provide feedback to the reviewers regarding the length, tone and substance of their reports and thank the reviewers when appropriate.

It would be wonderful if journal editors could organize in-person and online sessions to discuss expectations of department editors, associate editors and reviewers. In these sessions, it would be equally important for journal editors to clarify review criteria for their respective journals, discuss the review process, and exchange ideas for improving the quality of reviewer reports. Ultimately, communication is key for us to build a mutual learning community. 

We can break the vicious cycle so that more authors can embark on big problems, ensuring that INFORMS can make an even bigger impact. Everyone wins.

Acknowledgments

The author is grateful to Gerard Cachon, Tinglong Dai, Brian Denton, Edward Kaplan, Ho-Yin Mak, Stefan Scholtes and Mohan Sodhi for their constructive comments on an earlier version of this article.

References

1. Walls, M. and J. C. Smith, 2021, “Challenges Facing INFORMS Journals: The Way Forward,” OR/MS Today, https://doi.org/10.1287/orms.2021.04.13.
2. Kavadias, S., C. Loch and S. Scholtes, 2016, “A Future for Operations,” OR/MS Today, https://doi.org/10.1287/orms.2016.01.09.
3. Denton, B., 2017, “Publications: The Good, the Bad and the Ugly,” OR/MS Today, https://doi.org/10.1287/orms.2017.04.02.
4. Staats, B. and C. S. Tang, 2016, “The Psychology of a Reviewer in Operations Management,” From the Journal Editor Blog, Manufacturing & Service Operations Management, https://www.informs.org/Blogs/M-SOM-Blogs/From-M-SOM-Journal-Editor/The-Psychology-of-a-Reviewer-in-Operations-Management.
5. Spiegel, M., 2012, “Reviewing Less – Progressing More,” Review of Financial Studies, Vol. 25, No. 5, pp. 1331-1338.
6. Cachon, G. P., K. M. Daniels and R. Lobel, 2017, “The Role of Surge Pricing on a Service Platform with Self-Scheduling Capacity,” Manufacturing & Service Operations Management, Vol. 19, No. 3, pp. 368-384.
7. Tang, C. S., 2020, “‘Review Less, Progress More,’ Revisited: 2-2-2 + 3-3-4,” From the Journal Editor Blog, Manufacturing & Service Operations Management, https://www.informs.org/Blogs/M-SOM-Blogs/From-M-SOM-Journal-Editor/Review-Less-Progress-More-Revisited-2-2-2-3-3-4.
8. Kaplan, E., 2020, “COVID-19 Scratch Models to Support Local Decisions,” Manufacturing & Service Operations Management, Vol. 22, No. 4, pp. 645-655.