Fast Selection From Multiple Treatments: A Sequential Method for Principled Digital Experimentation

In the current era of digital business, firms continuously experiment to enhance the online experience of individuals visiting their websites and platforms. The possible changes range from minor tweaks to large product or user experience updates, which are tested before the full rollout to estimate performance and minimize unintended negative outcomes. Because of the clear benefits of digital experimentation, increases in the number of experiments have strained the resource of online participants/customers. In tension with this scarcity, many digital experiments are not carefully powered for their objectives, leading to either over-sampling that wastes resources or under-sampling that weakens inference. These issues warrant methods that can help experimenters balance power and efficient resource use — that is, to sample enough for the proper power without over-sampling. To address this issue, we propose a sequential hypothesis testing method for selecting the best treatment among multiple alternatives for experimenter-specified levels of statistical power and false positive rate. Critically, the method not only samples for no more than the necessary level of statistical power, it also has low sample size variance relative to other methods, meaning that the resulting sample size is a precise estimate of the required sample size, and low bias in the treatment effect estimates, reducing a common problem for adaptive sampling methods. We also demonstrate our method on a dataset from a real multi-armed online experiment which demonstrates the method's efficacy in a realistic scenario. Our method can be implemented in an experimentation pipeline, facilitated with an R package we provide.