You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism

References

• Aitchison J, Silvey S (1958) Maximum-likelihood estimation of parameters subject to restraints. Ann. Math. Statist. 29(3):813–828.Crossref, Google Scholar
• Amador M, Bagwell K (2013) The theory of optimal delegation with an application to tariff caps. Econometrica 81(4):1541–1599.Crossref, Google Scholar
• Arnold BC (1987) Majorization and the Lorenz Order: A Brief Introduction, Springer-Verlag Lecture Notes in Statistics, vol. 43 (Springer, New York).Crossref, Google Scholar
• Arous I, Yang J, Khayati M, Cudré-Mauroux P (2021) Peer grading the peer reviews: A dual-role approach for lightening the scholarly paper review process. Proc. Web Conf. 2021 (Ljubljana, Slovenia), 1916–1927.Google Scholar
• Aziz H, Lev O, Mattei N, Rosenschein JS, Walsh T (2019) Strategyproof peer selection using randomization, partitioning, and apportionment. Artificial Intelligence 275(1):295–309. Crossref, Google Scholar
• Barlow RE, Bartholomew DJ, Bremner JM, Brunk H (1972) Statistical Inference Under Order Restrictions: The Theory and Application of Isotonic Regression (Wiley, New York).Google Scholar
• Battaglini M (2002) Multiple referrals and multidimensional cheap talk. Econometrica 70(4):1379–1401.Crossref, Google Scholar
• Bogdan M, Van Den Berg E, Sabatti C, Su W, Candès EJ (2015) Slope—Adaptive variable selection via convex optimization. Ann. Appl. Statist. 9(3):1103–1140.Crossref, Google Scholar
• Carlini N, Feldman V, Nasr M (2022) No free lunch in “privacy for free: How does dataset condensation help privacy.” Preprint, submitted September 29, https://arxiv.org/abs/2209.14987.Google Scholar
• Chakraborty A, Harbaugh R (2007) Comparative cheap talk. J. Econom. Theory 132(1):70–94.Crossref, Google Scholar
• Chakraborty A, Harbaugh R (2010) Persuasion by cheap talk. Amer. Econom. Rev. 100(5):2361–2382.Crossref, Google Scholar
• Chatterjee S, Guntuboyina A, Sen B (2015) On risk bounds in isotonic and other shape restricted regression problems. Ann. Statist. 43(4):1774–1800.Crossref, Google Scholar
• Cortes C, Lawrence ND (2021) Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. Preprint, submitted September 20, https://arxiv.org/abs/2109.09774.Google Scholar
• Crawford VP, Sobel J (1982) Strategic information transmission. Econometrica 50(6):1431–1451.Crossref, Google Scholar
• Frankel A (2014) Aligned delegation. Amer. Econom. Rev. 104(1):66–83.Crossref, Google Scholar
• Frankel A (2016) Delegating multiple decisions. Amer. Econom. J. Microeconom. 8(4):16–53.Crossref, Google Scholar
• Gneiting T (2011) Making and evaluating point forecasts. J. Amer. Statist. Assoc. 106(494):746–762.Crossref, Google Scholar
• Holmström BR (1978) On Incentives and Control in Organizations (Stanford University, Stanford, CA).Google Scholar
• Jecmen S, Zhang H, Liu R, Shah N, Conitzer V, Fang F (2020) Mitigating manipulation in peer review via randomized reviewer assignments. Larochelle H, Ranzato M, Hadsell R, Balcan MF, Lin H, eds. Advances in Neural Information Processing Systems, vol. 33 (Curran Associates, Inc., Red Hook, NY), 12533–12545.Google Scholar
• Johnstone IM (2002) Function Estimation and Gaussian Sequence Models (Stanford University, Stanford, CA), 497.Google Scholar
• Kobren A, Saha B, McCallum A (2019) Paper matching with local fairness constraints. Proc. 25th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 1247–1257.Google Scholar
• Kreps DM (1990) A Course in Microeconomic Theory (Princeton University Press, Princeton, NJ).Crossref, Google Scholar
• Krishna V, Maenner E (2001) Convex potentials with an application to mechanism design. Econometrica 69(4):1113–1119.Crossref, Google Scholar
• Kruskal JB (1964) Nonmetric multidimensional scaling: A numerical method. Psychometrika 29(2):115–129.Crossref, Google Scholar
• Levy G, Razin R (2007) On the limits of communication in multidimensional cheap talk: A comment. Econometrica 75(3):885–893.Crossref, Google Scholar
• Leyton-Brown K, Nandwani Y, Zarkoob H, Cameron C, Newman N, Raghu D (2024) Matching papers and reviewers at large conferences. Artificial Intelligence 331:104119.Google Scholar
• Liang W, Zhang Y, Cao H, Wang B, Ding DY, Yang X, Vodrahalli K, et al. (2024) Can large language models provide useful feedback on research papers? A large-scale empirical analysis. New England J. Medicine AI 1(8):AIoa2400196.Google Scholar
• Marshall AW, Olkin I, Arnold BC (1979) Inequalities: Theory of Majorization and Its Applications, vol. 143 (Academic Press, New York).Google Scholar
• Martimort D, Semenov A (2006) Continuity in mechanism design without transfers. Econom. Lett. 93(2):182–189.Crossref, Google Scholar
• Mattei N, Turrini P, Zhydkov S (2020) Peernomination: Relaxing exactness for increased accuracy in peer selection. Preprint, submitted April 30, https://arxiv.org/abs/2004.14939.Google Scholar
• Melumad ND, Shibano T (1991) Communication in settings with no transfers. RAND J. Econom. 22(2):173–198.Crossref, Google Scholar
• Noothigattu R, Shah N, Procaccia A (2021) Loss functions, axioms, and peer review. J. Artificial Intelligence Res. 70:1481–1515.Crossref, Google Scholar
• Ouyang L, Wu J, Jiang X, Almeida D, Wainwright C, Mishkin P, Zhang C, et al. (2022) Training language models to follow instructions with human feedback. Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, eds. Advances in Neural Information Processing Systems, vol. 35 (Curran Associates, Inc., Red Hook, NY), 27730–27744.Google Scholar
• PaperCopilot (2025a) NeurIPS statistics. Accessed February 9, 2025, https://papercopilot.com/statistics/neurips-statistics.Google Scholar
• PaperCopilot (2025b) ICML statistics. Accessed February 9, 2025, https://papercopilot.com/statistics/icml-statistics.Google Scholar
• Reddit discussants (2021) [D] Peer Review is still BROKEN! The NeurIPS 2021 Review Experiment (Yannic Kilcher). Retrieved February 9, https://www.reddit.com/r/MachineLearning/comments/r24rp7/d_peer_review_is_still_broken_the_neurips_2021//.Google Scholar
• Robertson T, Wright FT, Dykstra RL (1988) Order Restricted Statistical Inference (John Wiley, Hoboken, NJ).Google Scholar
• Rogers A, Augenstein I (2020) What can we do to improve peer review in NLP? Preprint, submitted October 8, https://arxiv.org/abs/2010.03863.Google Scholar
• Sculley D, Snoek J, Wiltschko A (2018) Avoiding a tragedy of the commons in the peer review process. Preprint, submitted December 18, https://arxiv.org/abs/1901.06246.Google Scholar
• Shah NB, Tabibian B, Muandet K, Guyon I, Von Luxburg U (2018) Design and analysis of the NIPS 2016 review process. J. Machine Learn. Res. 19(49):1–34.Google Scholar
• Stelmakh I, Shah NB, Singh A, Daumé H III (2020) A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. Preprint, submitted November 30, https://arxiv.org/abs/2011.15050.Google Scholar
• Su WJ (2021) You are the best reviewer of your own papers: An owner-assisted scoring mechanism. Ranzato M, Beygelzimer A, Dauphin Y, Liang PS, Wortman Vaughan J, eds. Advances in Neural Information Processing Systems, vol. 34 (Curran Associates, Inc., Red Hook, NY), 27929–27939. Google Scholar
• Su B, Collina N, Wen G, Li D, Cho K, Fan J, Zhao B, et al. (2025a) How to find fantastic papers: Self-rankings as a powerful predictor of scientific impact beyond peer review. Preprint, submitted October 2, https://arxiv.org/abs/2510.02143.Google Scholar
• Su B, Zhang J, Collina N, Yan Y, Li D, Cho K, Fan J, et al. (2025b) The ICML 2023 ranking experiment: Examining author self-assessment in ML/AI peer review. J. Amer. Statist. Assoc. 1–16.Crossref, Google Scholar
• Sun SH (2020) ICLR2020-openreviewdata. https://github.com/shaohua0116/ICLR2020-OpenReviewData.Google Scholar
• Ugarov A (2023) Peer prediction for peer review: Designing a marketplace for ideas. Preprint, submitted March 29, https://arxiv.org/abs/2303.16855.Google Scholar
• Van Rooyen S, Godlee F, Evans S, Black N, Smith R (1999) Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. BMJ 318(7175):23–27.Crossref, Google Scholar
• Wang J, Shah NB (2018) Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. Preprint, submitted September 13, https://arxiv.org/abs/1806.05085.Google Scholar
• Wang J, Stelmakh I, Wei Y, Shah NB (2020) Debiasing evaluations that are biased by evaluations. Preprint, submitted December 1, https://arxiv.org/abs/2012.00714.Google Scholar
• Wu J, Xu H, Guo Y, Su W (2023) An isotonic mechanism for overlapping ownership. Preprint, submitted June 19, https://arxiv.org/abs/2306.11154.Google Scholar
• Xu Y, Jecmen S, Song Z, Fang F (2023) A one-size-fits-all approach to improving randomness in paper assignment. Oh A, Naumann T, Globerson A, Saenko K, Hardt M, Levine S, eds. Advances in Neural Information Processing Systems, vol. 36 (Curran Associates, Inc., Red Hook, NJ), 14445–14468.Google Scholar
• Yan Y, Su WJ, Fan J (2025) Isotonic mechanism for exponential family estimation in machine learning peer review. J. Roy. Statist. Soc. Ser. B Statist. Methodology 87(5):1422–1456.Crossref, Google Scholar
• Zhang CH (2002) Risk bounds in isotonic regression. Ann. Statist. 30(2):528–555.Crossref, Google Scholar