Optimal Experimental Design for Staggered Rollouts

References

• Abadie A , Zhao J (2021) Synthetic controls for experimental design. Preprint, submitted August 4, https://arxiv.org/abs/2108.02196.Google Scholar
• Abadie A , Diamond A , Hainmueller J (2010) Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. J. Amer. Statist. Assoc. 105(490):493–505.Crossref, Google Scholar
• Abaluck J , Kwong LH , Styczynski A , Haque A , Kabir MA , Bates-Jefferys E , Crawford E , et al. (2021) Impact of community masking on covid-19: A cluster-randomized trial in Bangladesh. Sci. 375(6577):eabi9069.Google Scholar
• Angrist JD , Krueger AB (1995) Split-sample instrumental variables estimates of the return to schooling. J. Bus. Econom. Statist. 13(2):225–235.Crossref, Google Scholar
• Angrist JD , Pischke JS (2008) Mostly Harmless Econometrics: An Empiricist’s Companion (Princeton University Press, Princeton, NJ).Google Scholar
• Angrist JD , Imbens GW , Krueger AB (1999) Jackknife instrumental variables estimation. J. Appl. Econometrics 14(1):57–67.Crossref, Google Scholar
• Athey S , Imbens G (2016) Recursive partitioning for heterogeneous causal effects. Proc. Natl. Acad. Sci. USA 113(27):7353–7360.Crossref, Google Scholar
• Athey S , Bayati M , Doudchenko N , Imbens G , Khosravi K (2021) Matrix completion methods for causal panel data models. J. Amer. Statist. Assoc. 116(536):1716–1730.Crossref, Google Scholar
• Atkinson A , Fedorov V (1975a) The design of experiments for discriminating between two rival models. Biometrika 62(1):57–70.Crossref, Google Scholar
• Atkinson A , Donev A , Tobias R (2007) Optimum Experimental Designs, with SAS, vol. 34 (Oxford University Press, Oxford, UK).Google Scholar
• Atkinson AC , Fedorov VV (1975b) Optimal design: Experiments for discriminating between several models. Biometrika 62(2):289–303.Google Scholar
• Auer P (2003) Using confidence bounds for exploitation-exploration trade-offs. J. Machine Learn. Res. 3:397–422.Google Scholar
• Bai J (2003) Inferential theory for factor models of large dimensions. Econometrica 71(1):135–171.Crossref, Google Scholar
• Bai J , Ng S (2002) Determining the number of factors in approximate factor models. Econometrica 70(1):191–221.Crossref, Google Scholar
• Bajari P , Burdick B , Imbens GW , Masoero L , McQueen J , Richardson T , Rosen IM (2023) Multiple randomization designs. Statistical Sci. 1(1):1–19.Google Scholar
• Basse G , Ding Y , Toulis P (2023) Minimax designs for causal effects in temporal experiments with treatment habituation. Biometrika 110(1):155–168.Google Scholar
• Bastani H , Bayati M (2020) Online decision making with high-dimensional covariates. Oper. Res. 68(1):276–294.Link, Google Scholar
• Bertsekas D (2012) Dynamic Programming and Optimal Control: Volume I (Athena Scientific, Nashua, NH).Google Scholar
• Bertsimas D , Johnson M , Kallus N (2015) The power of optimization over randomization in designing experiments involving small samples. Oper. Res. 63(4):868–876.Link, Google Scholar
• Bertsimas D , Korolko N , Weinstein AM (2019) Covariate-adaptive optimization in online clinical trials. Oper. Res. 67(4):1150–1161.Abstract, Google Scholar
• Bhat N , Farias VF , Moallemi CC , Sinha D (2019) Near optimal A-B testing. Management Sci. 66(10):4477–4495.Link, Google Scholar
• Bojinov I , Simchi-Levi D , Zhao J (2023) Design and analysis of switchback experiments. Management Sci. 69(7):3759–3777.Google Scholar
• Brown CA , Lilford RJ (2006) The stepped wedge trial design: A systematic review. BMC Medical Res. Methodology 6(1):54.Crossref, Google Scholar
• Bubeck S , Cesa-Bianchi N (2012) Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations Trends® Machine Learn. 5(1):1–122.Google Scholar
• Cachon GP , Gallino S , Olivares M (2019) Does adding inventory increase sales? Evidence of a scarcity effect in US automobile dealerships. Management Sci. 65(4):1469–1485.Link, Google Scholar
• Card D , Krueger AB (1994) Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. Amer. Econom. Rev. 84(4):772–793.Google Scholar
• Chernozhukov V , Chetverikov D , Demirer M , Duflo E , Hansen C , Newey W , Robins J (2018) Double/debiased machine learning for treatment and structural parameters: Double/debiased machine learning. Econometrics J. 21(1):C1–C68.Google Scholar
• Chow YS , Robbins H (1965) On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Ann. Math. Statist. 36(2):457–462.Crossref, Google Scholar
• Cui R , Zhang DJ , Bassamboo A (2019) Learning from inventory availability information: Evidence from field experiments on Amazon. Management Sci. 65(3):1216–1235.Link, Google Scholar
• De Stavola B , Cox D (2008) On the consequences of overstratification. Biometrika 95(4):992–996.Crossref, Google Scholar
• Deshpande Y , Javanmard A , Mehrabi M (2023) Online debiasing for adaptively collected high-dimensional data with applications to time series analysis. J. Amer. Statist. Assoc. 118(542):1126–1139.Google Scholar
• Deshpande Y , Mackey L , Syrgkanis V , Taddy M (2018) Accurate inference for adaptive linear models. Internat. Conf. Machine Learn. (PMLR, New York), 1194–1203.Google Scholar
• Dette H , Melas VB , Guchenko R (2015) Bayesian t-optimal discriminating designs. Ann. Statist. 43(5):1959–1985.Crossref, Google Scholar
• Dette H , Melas VB , Shpilev P (2012) T-optimal designs for discrimination between two polynomial models. Ann. Statist. 40(1):188–205.Crossref, Google Scholar
• Dette H , Melas VB , Shpilev P (2013) Robust t-optimal discriminating designs. Ann. Statist. 41(4):1693–1715.Crossref, Google Scholar
• Doudchenko N , Gilinson D , Taylor S , Wernerfelt N (2019) Designing experiments with synthetic controls. Technical report, Working paper.Google Scholar
• Doudchenko N , Khosravi K , Pouget-Abadie J , Lahaie S , Lubin M , Mirrokni V , Spiess J , et al. (2021) Synthetic design: An optimization approach to experimental design with synthetic controls. Adv. Neural Inform. Processing Systems 34:8691–8701.Google Scholar
• Efron B (1971) Forcing a sequential experiment to be balanced. Biometrika 58(3):403–417.Crossref, Google Scholar
• Fox BL (2000) Separability in optimal allocation. Oper. Res. 48(1):173–176.Link, Google Scholar
• Girling AJ , Hemming K (2016) Statistical efficiency and optimal design for stepped cluster studies under linear mixed effects models. Statist. Medicine 35(13):2149–2166.Crossref, Google Scholar
• Glynn PW , Johari R , Rasouli M (2020) Adaptive experimental design with temporal interference: A maximum likelihood approach. Adv. Neural Inform. Processing Systems 33:15054–15064.Google Scholar
• Glynn PW , Whitt W (1992) The asymptotic validity of sequential stopping rules for stochastic simulations. Ann. Appl. Probab. 2(1):180–198.Crossref, Google Scholar
• Goldenshluger A , Zeevi A (2013) A linear response bandit problem. Stochastic Systems 3(1):230–261.Link, Google Scholar
• Gupta S , Kohavi R , Tang D , Xu Y , Andersen R , Bakshy E , Cardin N , et al. (2019) Top challenges from the first practical online controlled experiments summit. SIGKDD Explorations Newsletter 21(1):20–35.Google Scholar
• Hamidi N , Bayati M , Gupta K (2019) Personalizing many decisions with high-dimensional covariates. Wallach H , Larochelle H , Beygelzimer A , d’Alché-Buc F , Fox E , Garnett R , eds. Adv. Neural Inform. Processing Systems, vol. 32 (Curran Associates, Inc., Red Hook, NY).Google Scholar
• Hayes B (2002) Computing science: The easiest hard problem. Amer. Sci. 90(2):113–117.Crossref, Google Scholar
• Hayes RJ , Moulton LH (2017) Cluster Randomised Trials (Chapman and Hall/CRC, Boca Raton, FL).Google Scholar
• Hemming K , Haines TP , Chilton PJ , Girling AJ , Lilford RJ (2015) The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ 350:h391.Crossref, Google Scholar
• Hussey MA , Hughes JP (2007) Design and analysis of stepped wedge cluster randomized trials. Contemporary Clinical Trials 28(2):182–191.Crossref, Google Scholar
• Imbens GW , Rubin DB (2015) Causal Inference in Statistics, Social, and Biomedical Sciences (Cambridge University Press, Cambridge, UK).Google Scholar
• Johari R , Koomen P , Pekelis L , Walsh D (2017) Peeking at a/b tests: Why it matters, and what to do about it. Proc. 23rd ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining, 1517–1525.Google Scholar
• Johari R , Li H , Liskovich I , Weintraub G (2022) Experimental design in two-sided platforms: An analysis of bias. Management Sci. 68(10):7069–7089.Link, Google Scholar
• Ju N , Hu D , Henderson A , Hong L (2019) A sequential test for selecting the better variant: Online a/b testing, adaptive allocation, and continuous monitoring. Proc. 12th ACM Internat. Conf. Web Search Data Mining (ACM, New York), 492–500.Google Scholar
• Kernan WN , Viscoli CM , Makuch RW , Brass LM , Horwitz RI (1999) Stratified randomization for clinical trials. J. Clinical Epidemiology 52(1):19–26.Crossref, Google Scholar
• Lai TL , Wei CZ (1982) Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. Ann. Statist. 10(1):154–166.Google Scholar
• Lattimore T , Szepesvári C (2020) Bandit Algorithms (Cambridge University Press, Cambridge, UK).Google Scholar
• Lawrie J , Carlin JB , Forbes AB (2015) Optimal stepped wedge designs. Statist. Probab. Lett. 99:210–214.Crossref, Google Scholar
• Li F , Turner EL , Preisser JS (2018) Optimal allocation of clusters in cohort stepped wedge designs. Statist. Probab. Lett. 137:257–263.Crossref, Google Scholar
• Mertens S (2006) The easiest hard problem: Number partitioning. Comput. Complexity Statist. Phys. 125(2):125–139.Google Scholar
• Mulvey JM (1983) Multivariate stratified sampling by optimization. Management Sci. 29(6):715–724.Link, Google Scholar
• Nikolaev AG , Jacobson SH , Cho WKT , Sauppe JJ , Sewell EC (2013) Balance optimization subset selection (BOSS): An alternative approach for causal inference with observational data. Oper. Res. 61(2):398–412.Link, Google Scholar
• Robinson PM (1988) Root-n-consistent semiparametric regression. Econometrica 56(4):931–954.Crossref, Google Scholar
• Siegmund D (1985) Sequential Analysis: Tests and Confidence Intervals (Springer Science & Business Media, Berlin).Google Scholar
• Singham DI , Schruben LW (2012) Finite-sample performance of absolute precision stopping rules. INFORMS J. Comput. 24(4):624–635.Link, Google Scholar
• Uciński D , Bogacka B (2005) T-optimum designs for discrimination between two multiresponse dynamic models. J. Roy. Statist. Soc. Ser. B Statist. Methodology 67(1):3–18.Crossref, Google Scholar
• Wager S , Athey S (2018) Estimation and inference of heterogeneous treatment effects using random forests. J. Amer. Statist. Assoc. 113(523):1228–1242.Crossref, Google Scholar
• Wager S , Xu K (2021) Experimenting in equilibrium. Management Sci. 67(11):6694–6715.Link, Google Scholar
• Wald A (2004) Sequential Analysis (Courier Corporation, Chelmsford, MA).Google Scholar
• Wallace TD , Hussain A (1969) The use of error components models in combining cross section with time series data. Econometrica 37(1):55–72.Crossref, Google Scholar
• Wiens DP (2009) Robust discrimination designs. J. Roy. Statist. Soc. Ser. B Statist. Methodology 71(4):805–829.Crossref, Google Scholar
• Woertman W , de Hoop E , Moerbeek M , Zuidema SU , Gerritsen DL , Teerenstra S (2013) Stepped wedge designs could reduce the required sample size in cluster randomized trials. J. Clinical Epidemiology 66(7):752–758.Crossref, Google Scholar
• Xiong R , Chin A , Taylor S (2023) Bias-variance tradeoffs for designing simultaneous temporal experiments. The KDD'23 Workshop on Causal Discovery, Prediction and Decision (PMLR, New York), 115–131.Google Scholar