Design and Analysis of Switchback Experiments

References

• Abadie A, Athey S, Imbens GW, Wooldridge JM (2020) Sampling-based vs. design-based uncertainty in regression analysis. Econometrica 88(1):265–296.Crossref, Google Scholar
• Aronow PM, Samii C (2017) Estimating average causal effects under general interference, with application to a social network experiment. Ann. Appl. Statist. 11(4):1912–1947.Crossref, Google Scholar
• Athey S, Imbens GW (2018) Design-based analysis in difference-in-differences settings with staggered adoption. Technical report, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Athey S, Eckles D, Imbens GW (2018) Exact p-values for network interference. J. Amer. Statist. Assoc. 113(521):230–240.Crossref, Google Scholar
• Azevedo EM, Deng A, Montiel Olea J, Rao JM, Weyl EG (2019) A/b testing with fat tails. J. Political Econom. 128(12):4614–4650Google Scholar
• Bai Y (2019) Optimality of matched-pair designs in randomized control trials. Preprint, submitted December 15, https://dx.doi.org/10.2139/ssrn.3483834.Google Scholar
• Bakshy E, Eckles D, Bernstein MS (2014) Designing and deploying online field experiments. Proc. 23rd Internat. Conf. World Wide Web (ACM, New York), 283–292.Google Scholar
• Basse G, Ding Y, Toulis P (2019b) Minimax crossover designs. Preprint, submitted August 9, https://arxiv.org/abs/1908.03531v1.Google Scholar
• Basse G, Ding P, Feller A, Toulis P (2019a) Randomization tests for peer effects in group formation experiments. Preprint, submitted April 4, https://arxiv.org/abs/1904.02308.Google Scholar
• Berger JO (2013) Statistical Decision Theory and Bayesian Analysis (Springer Science & Business Media, Berlin).Google Scholar
• Bickel PJ, Doksum KA (2015) Mathematical Statistics: Basic Ideas and Selected Topics, vol. I (CRC Press, Boca Raton, FL).Google Scholar
• Bojinov I, Shephard N (2019) Time series experiments and causal estimands: Exact randomization tests and trading. J. Amer. Statist. Assoc. 114(528):1665–1682.Crossref, Google Scholar
• Bojinov I, Rambachan A, Shephard N (2021) Panel experiments and dynamic causal effects: A finite population perspective. Quant. Econom. 12(4):1171–1196.Crossref, Google Scholar
• Bojinov I, Saint-Jacques G, Tingley M (2020) Avoid the pitfalls of a/b testing: Make sure your experiments recognize customers’ varying needs. Harvard Bus. Rev. 98(2):48–53.Google Scholar
• Boruvka A, Almirall D, Witkiewitz K, Murphy SA (2018) Assessing time-varying causal effect moderation in mobile health. J. Amer. Statist. Assoc. 113(523):1112–1121.Crossref, Google Scholar
• Caro F, Gallien J (2012) Clearance pricing optimization for a fast-fashion retailer. Oper. Res. 60(6):1404–1422.Link, Google Scholar
• Chamandy N (2016) Experimentation in a ridesharing marketplace—Lyft engineering. Accessed October 1, 2022, https://eng.lyft.com/experimentation-in-a-ridesharing-marketplace-b39db027a66e.Google Scholar
• Chamberlain G (1982) Multivariate regression models for panel data. J. Econometrics 18(1):5–46.Crossref, Google Scholar
• Chin A (2018) Central limit theorems via Stein’s method for randomized experiments under interference. Preprint, submitted April 9, https://arxiv.org/abs/1804.03105.Google Scholar
• Cui R, Li J, Zhang D (2020) Reducing discrimination with reviews in the sharing economy: Evidence from field experiments on Airbnb. Management Sci. 66(3):1071–1094.Google Scholar
• Cui R, Zhang DJ, Bassamboo A (2019) Learning from inventory availability information: Evidence from field experiments on Amazon. Management Sci. 65(3):1216–1235.Link, Google Scholar
• Deshpande Y, Mackey L, Syrgkanis V, Taddy M (2018) Accurate inference for adaptive linear models. Dy J, Andreas K, eds. Proc. Internat. Conf. Machine Learn., vol. 80 (PMLR), 1194–1203.Google Scholar
• Eckles D, Karrer B, Ugander J (2017) Design and analysis of experiments in networks: Reducing bias from interference. J. Causal Inference 5(1):20150021.Crossref, Google Scholar
• Farronato C, MacCormack A, Mehta S (2018) Innovation at Uber: The launch of express pool. Harvard Business School Case No. 620(062), Harvard Business School, Boston.Google Scholar
• Ferreira KJ, Lee BHA, Simchi-Levi D (2016) Analytics for an online retailer: Demand forecasting and price optimization. Manufacturing Service Oper. Management 18(1):69–88.Link, Google Scholar
• Fisher RA (1937) The Design of Experiments, 2nd ed. https://www.amazon.com/Design-Experiments-Ronald-Fisher/dp/0028446909.Google Scholar
• Garg N, Nazerzadeh H (2019) Driver surge pricing. Preprint, submitted May 18, https://arxiv.org/abs/1905.07544.Google Scholar
• Glynn P, Johari R, Rasouli M (2020) Adaptive experimental design with temporal interference: A maximum likelihood approach. Preprint, submitted June 10, https://arxiv.org/abs/2006.05591.Google Scholar
• Gupta S, Kohavi R, Tang D, Xu Y, Andersen R, Bakshy E, Cardin N, et al. (2019) Top challenges from the first practical online controlled experiments summit. SIGKDD Explorations 21(1):20–35.Crossref, Google Scholar
• Hadad V, Hirshberg DA, Zhan R, Wager S, Athey S (2019) Confidence intervals for policy evaluation in adaptive experiments. Preprint, submitted November 7, https://arxiv.org/abs/1911.02768v1.Google Scholar
• Han KW, Bojinov I, Basse G (2021) Population interference in panel experiments. Preprint, submitted February 28, https://arxiv.org/abs/2103.00553.Google Scholar
• Harshaw C, Sävje F, Spielman D, Zhang P (2019) Balancing covariates in randomized experiments using the Gram-Schmidt walk. Preprint, submitted November 8, https://arxiv.org/abs/1911.03071.Google Scholar
• Hedayat A, Afsarinejad K (1978) Repeated measurements designs, ii. Ann. Statist. 6(3):619–628.Crossref, Google Scholar
• Holtz D, Lobel R, Liskovich I, Aral S (2020) Reducing interference bias in online marketplace pricing experiments. Preprint, submitted April 26, https://arxiv.org/abs/2004.12489.Google Scholar
• Imai K, Kim IS (2019) When should we use unit fixed effects regression models for causal inference with longitudinal data? Amer. J. Political Sci. 63(2):467–490.Crossref, Google Scholar
• Imbens GW, Rubin DB (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Johari R, Li H, Weintraub G (2020) Experimental design in two-sided platforms: An analysis of bias. Preprint, submitted February 13, https://arxiv.org/abs/2002.05670.Google Scholar
• Johari R, Pekelis L, Walsh DJ (2015) Always valid inference: Bringing sequential analysis to a/b testing. Preprint, submitted December 15, https://arxiv.org/abs/1512.04922.Google Scholar
• Jones B, Kenward MG (2014) Design and Analysis of Cross-over Trials (CRC Press, Boca Raton, FL).Crossref, Google Scholar
• Kastelman D, Ramesh R (2018) Switchback tests and randomized experimentation under network effects at DoorDash. Accessed October 1, 2022, https://medium.com/@DoorDash/switchback-tests-and-randomized-experimentation-under-network-effects-at-doordash-f1d938ab7c2a.Google Scholar
• Kempthorne O (1955) The randomization theory of experimental inference. J. Amer. Statist. Assoc. 50(271):946–967.Google Scholar
• Kohavi R, Thomke S (2017) The surprising power of online experiments. Harvard Bus. Rev. 95:74–82.Google Scholar
• Kohavi R, Henne RM, Sommerfield D (2007) Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. Proc. 13th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 959–967.Google Scholar
• Kohavi R, Tang D, Xu Y (2020) Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Kohavi R, Crook T, Longbotham R, Frasca B, Henne R, Ferres JL, Melamed T (2009) Online experimentation at Microsoft. Data Mining Case Stud. 11:39.Google Scholar
• Koning R, Hasan S, Chatterji A (2019) Experimentation and startup performance: Evidence from a/b testing. Technical report, National Bureau of Economic Research, Cambridge, MA.Google Scholar
• Laird NM, Skinner J, Kenward M (1992) An analysis of two-period crossover designs with carry-over effects. Statist. Medicine 11(14-15):1967–1979.Crossref, Google Scholar
• Li H, Zhao G, Johari R, Weintraub GY (2021) Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. Preprint, submitted April 25, https://arxiv.org/abs/2104.12222.Google Scholar
• Li JQ, Rusmevichientong P, Simester D, Tsitsiklis JN, Zoumpoulis SI (2015) The value of field experiments. Management Sci. 61(7):1722–1740.Link, Google Scholar
• Li KC (1983) Minimaxity for randomized designs: Some general results. Ann. Statist. 11(1):225–239.Crossref, Google Scholar
• Li X, Ding P (2017) General forms of finite population central limit theorems with applications to causal inference. J. Amer. Statist. Assoc. 112(520):1759–1769.Crossref, Google Scholar
• Li X, Ding P, Rubin DB (2020) Rerandomization in 2k factorial experiments. Ann. Statist. 48(1):43–63.Crossref, Google Scholar
• Lillie EO, Patay B, Diamant J, Issell B, Topol EJ, Schork NJ (2011) The n-of-1 clinical trial: The ultimate strategy for individualizing medicine? Personalized Medicine 8(2):161–173.Crossref, Google Scholar
• Ma W, Simchi-Levi D, Zhao J (2021) Dynamic pricing (and assortment) under a static calendar. Management Sci. 67(4):2292–2313.Link, Google Scholar
• March JG (1991) Exploration and exploitation in organizational learning. Organ. Sci. 2(1):71–87.Link, Google Scholar
• McFowland III E, Somanchi S, Neill DB (2018) Efficient discovery of heterogeneous treatment effects in randomized experiments via anomalous pattern detection. Preprint, submitted March 24, https://arxiv.org/abs/1803.09159.Google Scholar
• Neyman J, Dabrowska DM, Speed TP (1990) On the application of probability theory to agricultural experiments: Essay on principles, section 9. Statist. Sci. 5(4):465–472.Crossref, Google Scholar
• Nie X, Tian X, Taylor J, Zou J (2018) Why adaptively collected data have negative bias and how to correct for it. Storkey A, Perez-Cruz F, eds. Proc. 21st Internat. Conf. Artificial Intelligence Statist., vol. 84 (PMLR), 1261–1269.Google Scholar
• Oman SD, Seiden E (1988) Switch-back designs. Biometrika 75(1):81–89.Crossref, Google Scholar
• Puelz D, Basse G, Feller A, Toulis P (2019) A graph-theoretic approach to randomization tests of causal effects under general interference. Preprint, submitted October 24, https://arxiv.org/abs/1910.10862.Google Scholar
• Rambachan A, Shephard N (2019) Econometric analysis of potential outcomes time series: Instruments, shocks, linearity and the causal response function. Preprint, submitted March 5, https://arxiv.org/abs/1903.01637.Google Scholar
• Robins J (1986) A new approach to causal inference in mortality studies with a sustained exposure period—Application to control of the healthy worker survivor effect. Math. Model. 7(9-12):1393–1512.Crossref, Google Scholar
• Rubin DB (1980) Randomization analysis of experimental data: The Fisher randomization test comment. J. Amer. Statist. Assoc. 75(371):591–593.Google Scholar
• Sarasvathy SD (2001) Causation and effectuation: Toward a theoretical shift from economic inevitability to entrepreneurial contingency. Acad. Management Rev. 26(2):243–263.Crossref, Google Scholar
• Senn S, Lambrou D (1998) Robust and realistic approaches to carry-over. Statist. Medicine 17(24):2849–2864.Crossref, Google Scholar
• Sitkin SB (1992) Learning through failure: The strategy of small losses. Res. Organ. Behav. 14:231–266.Google Scholar
• Sobel ME (2012) Does marriage boost men’s wages?: Identification of treatment effects in fixed effects regression models for panel data. J. Amer. Statist. Assoc. 107(498):521–529.Google Scholar
• Sun T, Viswanathan S, Huang N, Zheleva E (2018) Designing promotional incentive to embrace social sharing: Evidence from field and laboratory experiments. Preprint, submitted January 5, https://dx.doi.org/10.2139/ssrn.3095094.Google Scholar
• Sussman DL, Airoldi EM (2017) Elements of estimation theory for causal effects in the presence of network interference. Preprint, submitted February 12, https://arxiv.org/abs/1702.03578.Google Scholar
• Thomke S (2001) Enlightened experimentation: The new imperative for innovation. Harvard Bus. Rev. 79(2):66–75.Google Scholar
• Thomke SH (2020) Experimentation Works: The Surprising Power of Business Experiments (Harvard Business Review Press, Boston).Google Scholar
• Wager S, Xu K (2019) Experimenting in equilibrium. Preprint, submitted March 6, https://arxiv.org/abs/1903.02124.Google Scholar
• Wu CF (1981) On the robustness and efficiency of some randomized designs. Ann. Statist. 9(6):1168–1177.Crossref, Google Scholar
• Xiong R, Athey S, Bayati M, Imbens GW (2019) Optimal experimental design for staggered rollouts. Preprint, submitted November 9, https://arxiv.org/abs/1911.03764v1.Google Scholar