Improving the Peer Review Process: How to Write a Good Referee Report

The Peer Review Process

The basic “currency” in academia are published papers in well-regarded peer-reviewed journals. The use of peer-reviewed publications ranges from (1) a mark that the scientific finding has passed through certain “quality control” (relying on non-peer-reviewed research is generally frowned upon), (2) evaluation of the reputational standing of a researcher and (3) key inputs for hiring, tenure and promotion decisions. It is almost impossible to imagine the operations research and management science (OR/MS) academic field (or most others) without the institution of peer review.

Yet, this institution is neither as old (only becoming ubiquitous since the 1970s; see [1]) or as universally established as one may imagine. In fact, recently, peer review has received serious criticism. Numerous authors have commented on its failures to ensure quality, filter out errors or malpractice, identify impactful publications and ensure fairness by evaluating a submitted paper on its merits (without regard for reputation or author connections). examples of failures of the peer review process and some suggestions for alternative models, see [2-7].

In OR/MS and operations management, there has also been recent discussion of whether our publication model is detrimental to the future of the field [8]. Every actively publishing researcher can attest to often unreasonable requests by referees and editors, with paper acceptance being more a matter of luck (which associate editor and referees are assigned to the paper) than of the paper’s strength. Very long publication timelines are another irritation (with multiple rounds of revisions, most top journals take more than two years to decide on acceptance after the initial submission).  

How to improve our peer review system, explore alternative publication models and seek the future of academic publishing is a worthwhile discussion. However, pragmatically speaking, peer review as the centerpiece of academic publishing is here to stay for the foreseeable future. Because academics will continue to function within the system in the interchangeable roles of authors, reviewers and editors, our efforts should be on making the current model function as well as possible.

Here, we focus on the referee role: how to ensure that our reports are objective, constructive and helpful to both editors and authors. also briefly discuss the responsibilities of the editorial board’s junior member – i.e., the associate editor (AE), who decides whether the paper should be sent out to referees, appoints the referees and handles the paper through each round of the refereeing process.

Preparing a Referee Report

Teaching how to write a well-structured, fair, constructive referee report is crucial in training Ph.D. students and postdocs. Discussing the refereeing process is especially important as our community grows, in both membership and tools used, and as more authors and referees from different foundational education join the ranks. This growth results in a less unified refereeing process. Our guiding principles are nicely summarized by the American Chemical Society [10]:

“A thoughtful, well-presented evaluation of a manuscript, with tangible suggestions for improvement and a recommendation that is supported by the comments, is the most valuable contribution that you can make as a reviewer, and such a review is greatly appreciated by both the authors of the manuscript and the editors of the journal.”

To be concrete, this article follows the typical structure of a good referee report for a first-round review. Although the notes below are based on our own experience, they may be applicable to many other fields of research.

1. Summary. An informative report should start with a summary section (300-500 words [11]). This section should summarize the paper and the problem under study, modeling framework and main results. The goal is to show the AE the paper’s highlights and your overall assessment.  often, summaries imply the paper made many solid contributions, yet we find a negative recommendation such as Reject or Reject and Resubmit. The key components of an informative summary should include: 
   1. Problem under study: What is it? Is it important? Is it within the journal’s scope? 
   2. Modeling framework: Describe the key parts of the framework (one supplier, one retailer, supplier sets the wholesale price, retailer responds with an order quantity and then sets the retail price, etc.). Are there major issues with this framework? Are some parts inconsistent/problematic? For example, assuming the retailer is myopic and the supplier is strategic may not be a reasonable setup. A good summary highlights standard and unique aspects. The latter are more crucial and subject to more critical evaluation. 
   3. Key results: What are they claimed to be? How significant are they (assuming they are correct)? Evaluate the significance of the claimed contributions in view of the journal’s theme and quality. Are there any issues with the correctness of the results? If so, indicate which specific comments in part 3 below address these issues.
2. Recommendation. The most common recommendations in the first round of peer review are:  
   1. Accept “As Is”: Neither author has seen this outcome on the first round, but legends are that it exists.
   2. Minor Revision: The reviewer raises a few easily addressable issues. The crucial distinction from (c) and (d) is that there is no need for another round of reviews – it should be easy for the AE to ensure that the minor issues raised have been addressed.
   3. Major Revision: The reviewer can see a clear path to publication in the journal, meaning that the issues identified are fixable (at least in the referee’s mind). Outline a clear path to publication – what it would take (additional results, computational runs, justifications, etc.) to address the issues raised and how this path is doable. 
   4. Reject and Resubmit: This outcome is similar to the Major Revision (c), except the reviewer is less sure that the issues raised are addressable (i.e., there is more risk involved in preparing an acceptable revision). The reviewer must still indicate what an acceptable version would need. This recommendation should generally be avoided if you think the current version of the paper already makes a substantial contribution but that, with some work, the paper could be improved (by generalizing the framework, analyzing more data, etc.). No paper is ever perfect or “complete,” and demanding your version of perfection can delay publication by many months or years, making the results less timely and beneficial to the field.
   5. Reject: If the issues you raise are fundamental – e.g., you do not believe that the basic model assumptions and/or the objective function are reasonable, if there are technical errors not easily fixable, or the paper does not fit the journal. This recommendation signifies that there is no way to respond to the issues identified in your report within the paper’s framework. 
3. Major comments. This part of the report should elaborate on the main criticisms of the paper and justify these criticisms. Here are some do’s and don’ts. 
   1. Philosophical objections. Some reviews raise philosophical objections specific to the paper. For example, “This paper adopts a monopolistic competition model. Such models are of limited interest.” A more specific objection could be: “This paper assumes that customers’ valuation of service includes only travel cost, but not the cost of waiting.” A referee should not, as a rule, object to widely accepted frameworks or modeling approaches; such objections are more a matter of taste. A given framework not being your favorite does not make it illegitimate and should not be a basis for rejection. Moreover, philosophical objectives are nearly impossible to address. If they are to be a basis for rejection, it should be the editorial team’s decision. The referee’s primary job is to identify potential issues specific to the paper. Examples include: 
      1. The paper assumed an infinite horizon objective for the supplier and established an asymptotic optimality result. Both are extremely common frameworks in our community. Although one can question whether these apply in some practical situations, rejecting a paper because it adopts such standard frameworks is inappropriate, unless this framework is inadequate for the application under study (in which case, ask the authors for a better justification for using this framework). 
      2. The paper uses the framework of cooperative games. Both cooperative and noncooperative games are legitimate approaches. In some settings, you can ask the authors to extend their results to the noncooperative solution if it is an important benchmark, but rejecting the paper because you prefer the noncooperative framework is inappropriate.
      3. The paper assumes an infinite horizon objective for the supplier but a finite horizon objective for the retailer. This is an inconsistency within the adopted framework and a legitimate criticism of the paper. Unless the authors can provide a strong justification for such dichotomous treatment of different players, this can be the basis for a major issue/rejection.
   2. Assess the significance and validity of the claimed contributions. The referee should refrain from subjective judgment and highlight exactly how the paper contributes to this area’s current state of the art. Judging whether this contribution is sufficiently significant can be left to the editorial team.  
      1. Are the key claimed contributions indeed established? How do these contribute to the current state of the field? Are there some extensions needed before publication? If so, what are they? Are they doable? For example:
         1. The paper considers a linear cost structure consisting of fixed plus variable costs. Previous publications only considered variable costs. Adding the fixed cost component is significant because [describe the technical difficulties and new insights that this generates].
         2. The paper assumes customer demand is linear. However, most results should also hold for convex demand functions. This extension should not create major technical difficulties and will significantly generalize the current framework.  
   3. Outline concrete steps to bring the paper to a publishable level. These should generally be divided into “must haves” (“minimal improvements” required to reach the publishable level) and “nice to haves” (“desirable extensions” that would strengthen the paper). Specific questions include:
      1. How doable are the “must haves?” This will largely determine your recommendation.
      2. How doable are “nice to haves?”
      3. Are computational results sufficient? If not, what should be added?
         
         Be as reasonable as possible at this level. Requesting steps that take considerable effort and time to complete (e.g., collecting new data, deriving new fundamental results, asking for substantial generalization of the current setting) should be avoided (assuming you recommend a Major Revision). If you believe copious effort is required to bring the paper to a publishable status (rather than to some ideal form), you should recommend Reject or Reject and Resubmit.
         
         
   4. Assess the technical correctness of the results. This is arguably the referee’s main job and often the most time-consuming part of the review process. The editors can decide on the legitimacy of basic assumptions and significance of contributions. However, they do not have the time to examine the technical details and proofs. Referees should clearly state that all technical results seem correct (if not, specify the issues found). 
   5. Some referees recommend rejection citing that the paper’s key contributions are insufficient and do not verify the correctness of the analysis (e.g., does the study address potential endogeneity?). Except in the most egregious cases, we recommend against this approach. Reviewing the results’ correctness is important because:
      1. It often provides the authors with specific and constructive feedback.
      2. Your recommendation to the editor is easier to dismiss if it did not assess technical correctness.
      3. Not assessing technical correctness may give referees the wrong incentives to reject papers on more subjective grounds just to save time.
4. Minor comments. List typos and grammatical mistakes plus other relevant literature that authors might have missed. The expected author response to most minor comments is “done.”

To summarize: Referees should review papers as if we wrote them so that the feedback we provide each other improves our papers and professional communities. The referee should also provide objective feedback, such as technical correctness of the results or placement within the existing literature. Subjective assessments (sufficiency of the level of contribution, fit to the journal, etc.) should generally be left to the editors.

Responsibilities of AEs

For first-round reviews, we use the following process and suggest others follow and improve upon it.

1. Browse through the paper and decide whether – assuming the claims made by the authors are correct – the paper merits publication in the journal. If not, it is better to quickly “desk reject” (i.e., reject the paper without sending it to referees). If yes, the paper should be sent to referees likely to appreciate the paper’s contribution and familiar with the papers’ methodologies.
   1. If the decision is to send the paper to referees, it would be unfair to later reject the paper based on AE opinion and override referees’ positive comments.
2. Monitor the timing of the referee reports. Once the reports are in:
   1. Carefully review all key points made by referees and decide:
      1. whether some reports should be ignored in whole or in part because they are sloppy, unfair or biased; or
      2. to recommend a decision based on the remaining comments and objections.
   2. The letter to the authors should be as constructive as possible. Remember that an important objective of the review process is to shorten the publication clock. Thus, it is preferable to publish a paper that clearly makes important contributions in a timely manner, even if the current version is not perfect and could be refined/generalized/expanded.
      1. When recommending rejection:
         1. What were the paper’s main shortcomings that led to this decision?
         2. What, if anything, can the authors do to make the paper publishable?
      2. When recommending a revision, the AE should:
         1. Specify which of the superset of comments by the referees are (1) “must have” – the authors must address these in the next round of review; (2) “nice to have” – the authors should address these if possible and if space allows, but they may choose to remand them to the “Next Steps/Open Problems” section; (3) or “ignorable” – the authors should ignore these comments. This is arguably the most important contribution of the AE, particularly when referees’ comments/recommendations are contradictory. For example, one referee likes the current setting, while the other thinks it is too limiting and should be generalized. Here, the AE must provide guidance on which comments should be addressed or ignored.
   3. If the decision is a Minor Revision, AEs should expect to handle the next round of review on their own, without recourse to referees. This should be the preferred option if the list of serious “must have” comments is short or null.
   4. If the decision is a Major Revision, provide clear guidance to the authors regarding the “must have” and “nice to have” comments. This recommendation implies that the authors complete the required revision within three months at most. Keep the paper length in mind and do not require revisions that increase the length above the acceptable limit (this also applies to additional references/literature review). If the required changes might push the paper above the page limit, clearly indicate which sections to cut.

Importance of Peer Review

Although the peer review process has received some serious, and likely well-deserved, criticism in recent years, we see editorial and refereeing work as an integral part of being an academic researcher and serving our profession. It allows us to maintain the standards of our field, and provide and receive invaluable feedback on important research directions and methodologies advancing research in our areas of discipline.

We hope this article encourages constructive discussion of the refereeing process, particularly in OR/MS. It is based, first and foremost, on our experience as authors, referees, and editorial and advisory board members. It is also based on discussions with many colleagues at conferences, seminars and editorial board meetings. Furthermore, we are reflecting on recent efforts by editors of journals in the field [11] to create a more coherent and constructive review process.

References

1. Hosking, R., “Peer review: A historical perspective,” Broad Research Communication Lab. https://mitcommlab.mit.edu/broad/commkit/peer-review-a-historical-perspective.
2. Heesen, R. & Bright, L. K., 2021, “Is peer review a good idea?,” The British Journal for the Philosophy of Science, Vol. 72, No. 3.
3. Ragone, A., Mirylenka, K., Casati, F. & Marchese, M., 2013, “On peer review in computer science: Analysis of its effectiveness and suggestions for improvement,” Scientometrics, Vol. 97, pp. 317-356.
4. Siler, K., Lee, K. & Bero, L., 2015, “Measuring the effectiveness of scientific gatekeeping,” Proceedings of the National Academy of Sciences, 112, No. 2, pp. 360-365.
5. Huber, J., Inoua, S., Kerschbamer, R., König-Kersting, C., Palan, S. & Smith, V. L., 2022, “Nobel and novice: Author prominence affects peer review,” Proceedings of the National Academy of Sciences, Vol. 119, No. 41, Art. no. e2205779119.
6. Brembs, B., 2018, “Prestigious science journals struggle to reach even average reliability,” Frontiers in Human Neuroscience, Vol. 12, Art. no. 327726.
7. Gowers, T., 2017, “The end of an error?,” The Times Literary Supplement, the-tls.co.uk/articles/public/the-end-of-an-error-peer-review.
8. Nelson, B., 2024, “Is our publication model an existential threat to our field?,” INFORMS Open Forum, https://connect.informs.org/discussion/is-our-publication-model-an-existential-threat-to-our-field-1.
9. American Chemical Society, Reviewer Lab, https://institute.acs.org/courses/acs-reviewer-lab.html
10. Passalacqua, A., 2019, “Guidelines to Write a Referee Report,” https://scholar.harvard.edu/files/apassalacqua/files/refguidelines_tepe.pdf.
11. “MSOM Referee Review Framework,” Manufacturing & Service Operations Management, https://pubsonline.informs.org/page/msom/submission-guidelines.