p-Hacking and Publication Bias in Design-Based Causal Studies in Information Systems

Design-based causal inference is now central to empirical information systems (IS) research, yet its credibility in terms of p-hacking and publication bias depends on research practices that the broader publication system can quietly violate. This commentary audits 7,516 hypothesis tests from 558 papers published 2000–2023 across seven leading IS journals, spanning randomized controlled trials (RCT), difference-in-differences (DID), instrumental variables (IV), regression discontinuity (RD), and synthetic control (SC). We find statistically significant excess mass of test statistics just above z = 1.96 for DID, IV, and RCT studies, with no comparable excess at z = 2.58. This indicates that the conventional 5% threshold is the focal point of distortion. Z-curve analysis reveals an observed-to-expected discovery gap of 0.38; 96.65% of IS IV papers are just-identified, and applying the Lee et al. (2022) tF correction (an F-adjusted t-ratio) reverses 25% of significant findings. We deliver an IS-calibrated local false discovery rate (LFDR) that converts reported z-statistics into posterior probabilities of being null (e.g., 0.31 for DID at z = 1.96 and 0.47 for IV at the same threshold). We translate these patterns into method-specific reporting checklists and editorial reforms tailored to IS. The intent is to sound an evidence-based alarm and equip IS authors, reviewers, and editors with practical, field-calibrated tools for evaluating causal claims.