Blinded versus Unblinded Review: A Field Study on the Equity of Peer-Review Processes

References

• Aigner DJ, Cain GG (1977) Statistical theories of discrimination in labor markets. ILR Rev. 30(2):175–187.Crossref, Google Scholar
• Allen-Ramdial SAA, Campbell AG (2014) Reimagining the pipeline: Advancing stem diversity, persistence, and success. BioScience 64(7):612–618.Crossref, Google Scholar
• American Educational Research Association (1999) Standards for Educational and Psychological Testing (American Educational Research Association, Washington, DC).Google Scholar
• Arkes HR, Shaffer VA, Dawes RM (2006) Comparing holistic and disaggregated ratings in the evaluation of scientific presentations. J. Behav. Decision Making 19(5):429–439.Crossref, Google Scholar
• Arrow KJ (1973) The theory of discrimination. Ashenfelter O, Rees A, eds. Discrimination in Labor Markets (Princeton University Press, Princeton, NJ), 3–33.Google Scholar
• Åslund O, Skans ON (2012) Do anonymous job application procedures level the playing field? ILR Rev. 65(1):82–107.Crossref, Google Scholar
• Becker GS (1957) The Economics of Discrimination (University of Chicago Press, Chicago, IL).Google Scholar
• Behaghel L, Crépon B, Le Barbanchon T (2015) Unintended effects of anonymous resumes. Amer. Econom. J. Appl. Econom. 7(3):1–27.Crossref, Google Scholar
• Bertrand M, Duflo E (2017) Field experiments on discrimination. Banerjee AV, Duflo E, eds. Handbook of Economic Field Experiments, vol. 1 (Elsevier, Amsterdam), 309–393.Crossref, Google Scholar
• Biernat M, Manis M, Nelson TE (1991) Stereotypes and standards of judgment. J. Personality Soc. Psych. 60(4):485.Crossref, Google Scholar
• Blank RM (1991) The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review. Amer. Econom. Rev. 81(5):1041–1067. Google Scholar
• Bohren JA, Imas A, Rosenberg M (2019) The dynamics of discrimination: Theory and evidence. Amer. Econom. Rev. 109(10):3395–3436.Crossref, Google Scholar
• Bordalo P, Coffman K, Gennaioli N, Shleifer A (2016) Stereotypes. Quart. J. Econom. 131(4):1753–1794.Crossref, Google Scholar
• Bornmann L, Mutz R, Daniel HD (2010) A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One 5(12):e14331.Crossref, Google Scholar
• Branscombe NR, Schmitt MT, Harvey RD (1999) Perceiving pervasive discrimination among African Americans: Implications for group identification and well-being. J. Personality Soc. Psych. 77(1):135–149.Crossref, Google Scholar
• Brown BA, Henderson JB, Gray S, Donovan B, Sullivan S, Patterson A, Waggstaff W (2016) From description to explanation: An empirical exploration of the African-American pipeline problem in stem. J. Res. Sci. Teaching 53(1):146–177.Crossref, Google Scholar
• Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R, Lortie CJ (2008) Double-blind review favours increased representation of female authors. Trends Ecology Evolution 23(1):4–6.Crossref, Google Scholar
• Camilli G (2006) Test fairness. Ed. Measurement: Issues and Practice 25(2):9–16. Google Scholar
• Ceci SJ, Ginther DK, Kahn S, Williams WM (2014) Women in academic science: A changing landscape. Psych. Sci. Public Interest 15(3):75–141.Crossref, Google Scholar
• Chapman DS, Uggerslev KL, Carroll SA, Piasentin KA, Jones DA (2005) Applicant attraction to organizations and job choice: A meta-analytic review of the correlates of recruiting outcomes. J. Appl. Psych. 90(5):928–944.Crossref, Google Scholar
• Cheryan S, Markus HR (2020) Masculine defaults: Identifying and mitigating hidden cultural biases. Psych. Rev. 127(6):1022–1052.Crossref, Google Scholar
• Cortes C, Lawerence ND (2021) Inconsistency in conference peer review: Revisiting the 2014 Neurips experiment. Preprint, submitted September 20, https://arxiv.org/abs/2109.09774.Google Scholar
• Cox AR, Montgomerie R (2019) The cases for and against double-blind reviews. PeerJ 7:e6702.Crossref, Google Scholar
• Crane D (1967) The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. Amer. Sociologist 2(4):195–201. Google Scholar
• Dawes RM, Faust D, Meehl PE (1989) Clinical versus actuarial judgment. Science 243(4899):1668–1674.Crossref, Google Scholar
• de Leon FLL, McQuillin B (2020) The role of conferences on the pathway to academic impact evidence from a natural experiment. J. Human Resources 55(1):164–193.Crossref, Google Scholar
• Fang FC, Casadevall A (2016) Research funding: The case for a modified lottery. mBio 7(2):e00422-16.Google Scholar
• Fang H, Moro A (2011) Theories of statistical discrimination and affirmative action: A survey. Ashenfelter O, Card D, eds. Handbook of Labor Economics, vol. 4B (Elsevier, Amsterdam), 133–200.Google Scholar
• Fiske DW, Fogg L (1992) But the reviewers are making different criticisms of my paper! Diversity and uniqueness in reviewer comments. Amer. Psychologist 47(5):591–598.Google Scholar
• Fryer R, Jackson MO (2008) A categorical model of cognition and biased decision-making. J. Theoretical Econom. 8(1):1–42.Google Scholar
• Gelman A, Hill J, Yajima M (2012) Why we (usually) don’t have to worry about multiple comparisons. J. Res. Ed. Effectiveness 5(2):189–211.Crossref, Google Scholar
• Gibbs KD Jr, McGready J, Bennett JC, Griffin K (2014) Biomedical science PhD career interest patterns by race/ethnicity and gender. PLoS One 9(12):e114736.Crossref, Google Scholar
• Ginther DK, Schaffer WT, Schnell J, Masimore B, Liu F, Haak LL, Kington R (2011) Race, ethnicity, and NIH research awards. Science 333(6045):1015–1019.Crossref, Google Scholar
• Goldin C, Rouse C (2000) Orchestrating impartiality: The impact of “blind” auditions on female musicians. Amer. Econom. Rev. 90(4):715–741.Crossref, Google Scholar
• Gorodnichenko Y, Pham T, Talavera O (2021) Conference presentations and academic publishing. Econom. Modeling 95:228–254.Crossref, Google Scholar
• Goswami I, Urminsky O (2020) No substitute for the real thing: The importance of in-context field experiments in fundraising. Marketing Sci. 39(6):1052–1070.Link, Google Scholar
• Hammerschmidt K, Reinhardt K, Rolff J (2008) Does double-blind review favor female authors? Frontiers Ecology Environment 6(7):354–354.Crossref, Google Scholar
• Hertwig R, Pleskac TJ, Pachur T, Center for Adaptive Rationality (2019) Taming Uncertainty (MIT Press, Cambridge, MA).Crossref, Google Scholar
• Hull DL (1988) Science as a Process: An Evolutionary Account of the Social and Conceptual Development of Science (University of Chicago Press, Chicago).Crossref, Google Scholar
• Jowell R, Prescott-Clarke P (1970) Racial discrimination and white-collar workers in Britain. Race 11(4):397–417.Crossref, Google Scholar
• Judd CM, Park B (1993) Definition and assessment of accuracy in social stereotypes. Psych. Rev. 100(1):109–128.Crossref, Google Scholar
• Jussim L, Crawford JT, Anglin SM, Chambers JR, Stevens ST, Cohen F (2016) Stereotype accuracy: One of the largest and most replicable effects in all of social psychology. Handbook of Prejudice, Stereotyping, and Discrimination, vol. 2, 31–63.Google Scholar
• Kahneman D, Sibony O, Sunstein CR (2021) Noise: A Flaw in Human Judgment (Little Brown, New York).Google Scholar
• Keren G, Wu G (2015) The Wiley-Blackwell Handbook of Judgment and Decision Making (John Wiley & Sons, Chichester, UK).Crossref, Google Scholar
• Kolev J, Fuentes-Medel Y, Murray F (2019) Is blinded review enough? How gendered outcomes arise even under anonymous evaluation. Technical report, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Konovsky MA (2000) Understanding procedural justice and its impact on business organizations. J. Management 26(3):489–511.Crossref, Google Scholar
• Krause A, Rinne U, Zimmermann KF (2012) Anonymous job applications in Europe. IZA J. Eur. Labor Stud. 1(1):1–20.Crossref, Google Scholar
• Kruschke JK, Liddell TM (2018) The Bayesian new statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective. Psychonomic Bull. Rev. 25(1):178–206.Crossref, Google Scholar
• Lawrence N (2015) Nips experiment analysis. Inverse Probability (January 16), https://inverseprobability.com/2015/01/16/blogs-on-the-nips-experiment.Google Scholar
• MacCoun R (2020) Blinding to remove biases in science and society. Hertwig R, Engel C, eds. Deliberate Ignorance: Choosing Not to Know (MIT Press, Cambridge, MA).Google Scholar
• Mendoza JL, Mumford M (1987) Corrections for attenuation and range restriction on the predictor. J. Ed. Statist. 12(3):282–293.Crossref, Google Scholar
• Merton RK (1968) The Matthew effect in science: The reward and communication systems of science are considered. Science 159(3810):56–63.Crossref, Google Scholar
• Milkman KL, Akinola M, Chugh D (2012) Temporal distance and discrimination: An audit study in academia. Psych. Sci. 23(7):710–717.Crossref, Google Scholar
• Okike K, Hug KT, Kocher MS, Leopold SS (2016) Single-blind vs double-blind peer review in the setting of author prestige. JAMA 316(12):1315–1316.Crossref, Google Scholar
• Phelps ES (1972) The statistical theory of racism and sexism. Amer. Econom. Rev. 62(4):659–661.Google Scholar
• Pier EL, Brauer M, Filut A, Kaatz A, Raclaw J, Nathan MJ, Ford CE, et al. (2018) Low agreement among reviewers evaluating the same NIH grant applications. Proc. Natl. Acad. Sci. USA 115(12):2952–2957.Crossref, Google Scholar
• Rabinovitch H, Bereby-Meyer Y, Budescu DV (2020) Achieving more with less: Intuitive correction in selection. Psych. Sci. 31(4):437–448.Crossref, Google Scholar
• Robertson CT, Kesselheim AS (2016) Blinding as a Solution to Bias: Strengthening Biomedical Science, Forensic Science, and Law (Academic Press, Cambridge, MA).Google Scholar
• Ryan AM, Ployhart RE (2000) Applicants’ perceptions of selection procedures and decisions: A critical review and agenda for the future. J. Management 26(3):565–606.Google Scholar
• Thorngate W, Dawes RM, Foddy M (2010) Judging Merit (Psychology Press, New York).Crossref, Google Scholar
• Tomkins A, Zhang M, Heavlin WD (2017) Reviewer bias in single-versus double-blind peer review. Proc. Natl. Acad. Sci. USA 114(48):12708–12713.Crossref, Google Scholar
• Vehtari A, Gelman A, Gabry J (2017) Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat. Comput. 27(5):1413–1432.Crossref, Google Scholar
• Whittaker RJ (2008) Journal review and gender equality: A critical comment on Budden et al. Trends Ecology Evolution 23(9):478–479.Crossref, Google Scholar
• Williams WM, Ceci SJ (2015) National hiring experiments reveal 2: 1 faculty preference for women on stem tenure track. Proc. Natl. Acad. Sci. USA 112(17):5360–5365.Crossref, Google Scholar