Test & Roll: Profit-Maximizing A/B Tests

References

• Agrawal S, Goyal N (2013) Further optimal regret bounds for Thompson sampling. Carvalho CM, Ravikumar P, eds. Proceedings of the 16th International Conference on Artificial Intelligence and Statistics (AISTATS) (PMLR), 99–107.Google Scholar
• Arora N, Huber J (2001) Improving parameter estimates and model prediction by aggregate customization in choice experiments. J. Consumer Res. 28(2):273–283.Crossref, Google Scholar
• Azevedo EM, Alex D, Montiel Olea J, Rao JM, Weyl EG (2019) A/B testing with fat tails. Working paper, Wharton School, University of Pennsylvania, Philadelphia.Google Scholar
• Bart Y, Stephen A, Sarvary M (2014) Which products are best suited to mobile advertising? A field study of mobile display advertising effects on consumer attitudes and intentions. J. Marketing Res. 51(3):270–285.Crossref, Google Scholar
• Berman R, Pekelis L, Scott A, Van den Bulte C (2018) p-Hacking and false discovery in A/B testing. Working paper, Wharton School, University of Pennsylvania, Philadelphia.Google Scholar
• Berry DA, Wolff MC, Sack D (1994) Decision making during a phase III randomized controlled trial. Controlled Clinical Trials 15(5):360–378.Crossref, Google Scholar
• Bertsimas D, Mersereau AJ (2007) A learning approach for interactive marketing to a customer segment. Oper. Res. 55(6):1120–1135.Link, Google Scholar
• Bitran GR, Mondschein SV (1996) Mailing decisions in the catalog sales industry. Management Sci. 42(9):1364–1381.Link, Google Scholar
• Bonfrer A, Drèze X (2009) Real-time evaluation of email campaign performance. Marketing Sci. 28(2):251–263.Link, Google Scholar
• Cheng Y, Su F, Berry DA (2003) Choosing sample size for a clinical trial using decision analysis. Biometrika 90(4):923–936.Crossref, Google Scholar
• Chick SE, Frazier P (2012) Sequential sampling with economics of selection procedures. Management Sci. 58(3):550–569.Link, Google Scholar
• Chick SE, Inoue K (2001) New two-stage and sequential procedures for selecting the best simulated system. Oper. Res. 49(5):732–743.Link, Google Scholar
• DeGroot MH (1970) Optimal Statistical Decisions (McGraw-Hill, New York).Google Scholar
• Gershoff M (2017) Do no harm A/B testing without P-values. Conductrics Blog (March 30), https://conductrics.com/do-no-harm-or-ab-testing-without-p-values/.Google Scholar
• Hitsch GJ, Misra S (2018) Heterogeneous treatment effects and optimal targeting policy evaluation. Working paper, University of Chicago, Chicago.Google Scholar
• Johnson GA, Lewis R, Nubbemeyer E (2017) The online display ad effectiveness funnel & carryover: A meta-study of ghost ad experiments. Working paper, University of Rochester, Rochester, NY.Google Scholar
• Lewis RA, Rao JM (2015) The unfavorable economics of measuring the returns to advertising. Quart. J. Econom. 130(4):1941–1973.Crossref, Google Scholar
• Luh W-M, Guo J-H (2007) Approximate sample size formulas for the two-sample trimmed mean test with unequal variances. British J. Math. Statist. Psych. 60(1):137–146.Crossref, Google Scholar
• Misra K, Schwartz EM, Abernethy J (2019) Dynamic online pricing with incomplete information using multiarmed bandit experiments. Marketing Sci. 38(2):225–252.Link, Google Scholar
• Pekelis L, Walsh D, Johari R (2015) The new Stats Engine. Technical report, Optimizely, San Francisco.Google Scholar
• Schwartz EM, Bradlow ET, Fader PS (2017) Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Sci. 36(4):500–522.Link, Google Scholar
• Scott SL (2010) A modern Bayesian look at the multi-armed bandit. Appl. Stochastic Models Bus. Indust. 26(6):639–658.Crossref, Google Scholar
• Simester D, Timoshenko A, Zoumpoulis SI (2019) Efficiently evaluating targeting policies: Improving upon champion vs. challenger experiments. Management Sci. Forthcoming.Google Scholar
• Stallard N, Miller F, Day S, Hee SW, Madan J, Zohar S, Posch M (2017) Determination of the optimal sample size for a clinical trial accounting for the population size. Biometrical J. 59(4):609–625.Crossref, Google Scholar
• Stan Development Team (2018) RStan: The R interface to Stan. R package version 2.17.3. Accessed May 1, 2019, http://mc-stan.org.Google Scholar
• Thompson WR (1933) On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25(3/4):285–294.Crossref, Google Scholar
• Wortham K (2018) Sample size calculation—Myth buster edition. Search Discovery (blog) (May 20), https://www.searchdiscovery.com/blog/sample-size-calculation-myth-buster-edition.Google Scholar
• Zantedeschi D, Feit EM, Bradlow ET (2016) Measuring multichannel advertising response. Management Sci. 63(8):2706–2728.Link, Google Scholar