A Two-Part Machine Learning Approach to Characterizing Network Interference in A/B Testing

References

• Aral S, Van Alstyne M (2011) The diversity-bandwidth trade-off. Amer. J. Sociol. 117(1):90–171.Crossref, Google Scholar
• Aral S, Walker D (2012) Identifying influential and susceptible members of social networks. Science 337(6092):337–341.Crossref, Google Scholar
• Aral S, Walker D (2014) Tie strength, embeddedness, and social influence: A large-scale networked experiment. Management Sci. 60(6):1352–1370.Link, Google Scholar
• Aronow PM, Samii C (2017) Estimating average causal effects under general interference, with application to a social network experiment. Ann. Appl. Statist. 11(4):1912–1947.Crossref, Google Scholar
• Aronow PM, Eckles D, Samii C, Zonszein S (2021) Spillover effects in experimental data. Adv. Experiment. Political Sci. (Cambridge University Press, Cambridge, UK), 289–319.Google Scholar
• Athey S, Imbens G (2016) Recursive partitioning for heterogeneous causal effects. Proc. Natl. Acad. Sci. USA. 113(27):7353–7360.Crossref, Google Scholar
• Athey S, Eckles D, Imbens GW (2018) Exact p-values for network interference. J. Amer. Statist. Assoc. 113(521):230–240.Crossref, Google Scholar
• Awan U, Morucci M, Orlandi V, Roy S, Rudin C, Volfovsky A (2020) Almost-matching-exactly for treatment effect estimation under network interference. Proc. Internat. Conf. Artificial Intelligence Statist. (PMLR, New York), 3252–3262.Google Scholar
• Bajari P, Burdick B, Imbens GW, Masoero L, McQueen J, Richardson T, Rosen IM (2021) Multiple randomization designs. Preprint, submitted December 27, https://arxiv.org/abs/2112.13495.Google Scholar
• Bakshy E, Eckles D, Bernstein MS (2014) Designing and deploying online field experiments. Proc. 23rd Internat. Conf. World Wide Web (ACM, New York), 283–292.Google Scholar
• Basse GW, Airoldi EM (2018) Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika 105(4):849–858.Crossref, Google Scholar
• Basse GW, Feller A, Toulis P (2019) Randomization tests of causal effects under interference. Biometrika 106(2):487–494.Crossref, Google Scholar
• Basu D (1980) Randomization analysis of experimental data: The fisher randomization test. J. Amer. Statist. Assoc. 75(371):575–582.Crossref, Google Scholar
• Belloni A, Fang F, Volfovsky A (2022) Neighborhood adaptive estimators for causal inference under network interference. Preprint, submitted December 7, https://arxiv.org/abs/2212.03683.Google Scholar
• Bland JM (2004) Cluster randomised trials in the medical literature: Two bibliometric surveys. BMC Medical Res. Methodology 4(1):1–6.Crossref, Google Scholar
• Bojinov I, Gupta S (2022) Online experimentation: Benefits, operational and methodological challenges, and scaling guide. Harvard Data Sci. Rev. 4(3).Google Scholar
• Bojinov I, Simchi-Levi D, Zhao J (2022) Design and analysis of switchback experiments. Management Sci. 69(7):3759–3777.Link, Google Scholar
• Bowers J, Fredrickson MM, Panagopoulos C (2013) Reasoning about interference between units: A general framework. Political Anal. (Oxford) 21(1):97–124.Crossref, Google Scholar
• Boyarsky A, Namkoong H, Pouget-Abadie J (2023) Modeling interference using experiment roll-out. Preprint, submitted May 18, https://arxiv.org/abs/2305.10728.Google Scholar
• Brennan J, Mirrokni V, Pouget-Abadie J (2022) Cluster randomized designs for one-sided bipartite experiments. Adv. Neural Inform. Processing Systems 35:37962–37974.Google Scholar
• Candogan O, Chen C, Niazadeh R (2023) Correlated cluster-based randomized experiments: Robust variance minimization. Management Sci. 70(6):4069–4086.Link, Google Scholar
• Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W (2017) Double/debiased/Neyman machine learning of treatment effects. Amer. Econom. Rev. 107(5):261–265.Crossref, Google Scholar
• Chin A (2019) Regression adjustments for estimating the global treatment effect in experiments with interference. J. Causal Inference 7(2):20180026.Crossref, Google Scholar
• Cortez M, Eichhorn M, Yu CL (2022a) Exploiting neighborhood interference with low order interactions under unit randomized design. Preprint, submitted August 10, https://arxiv.org/abs/2208.05553.Google Scholar
• Cortez M, Eichhorn M, Yu CL (2022b) Graph agnostic estimators with staggered rollout designs under network interference. Preprint, submitted May 29, https://arxiv.org/abs/2205.14552.Google Scholar
• Cox DR (1958) Planning of Experiments (Wiley, New York).Google Scholar
• Eckles D, Karrer B, Ugander J (2016) Design and analysis of experiments in networks: Reducing bias from interference. J. Causal Inference 5(1)20150021.Crossref, Google Scholar
• Forastiere L, Airoldi EM, Mealli F (2021) Identification and estimation of treatment and interference effects in observational studies on networks. J. Amer. Statist. Assoc. 116(534):901–918.Crossref, Google Scholar
• Gallagher B, Eliassi-Rad T (2008) Leveraging label-independent features for classification in sparsely labeled networks: An empirical study. Proc. Workshop Social Network Mining Analysis (Springer, New York), 1–19.Google Scholar
• Granovetter MS (1973) The strength of weak ties. Amer. J. Sociol. 78(6):1360–1380.Crossref, Google Scholar
• Hagiu A, Wright J (2015) Marketplace or reseller? Management Sci. 61(1):184–203.Link, Google Scholar
• Harshaw C, Sävje F, Eisenstat D, Mirrokni V, Pouget-Abadie J (2023) Design and analysis of bipartite experiments under a linear exposure-response model. Electronic J. Statist. 17(1):464–518.Crossref, Google Scholar
• Holtz D, Lobel R, Liskovich I, Aral S (2020) Reducing interference bias in online marketplace pricing experiments. Preprint, submitted April 26, https://arxiv.org/abs/2004.12489.Google Scholar
• Hu Y, Li S, Wager S (2022) Average direct and indirect causal effects under interference. Biometrika 109(4):1165–1172.Crossref, Google Scholar
• Hudgens MG, Halloran ME (2008) Toward causal inference with interference. J. Amer. Statist. Assoc. 103(482):832–842.Crossref, Google Scholar
• Imai K, Ratkovic M (2013) Estimating treatment effect heterogeneity in randomized program evaluation. Ann. Appl. Statist. 7(1):443–470.Crossref, Google Scholar
• Imbens GW, Rubin DB (2010) Rubin causal model. Microeconometrics (Palgrave Macmillan, London), 229–241. Google Scholar
• Johari R, Li H, Liskovich I, Weintraub GY (2022) Experimental design in two-sided platforms: An analysis of bias. Management Sci. 68(10):7069–7089.Google Scholar
• Kernighan BW, Lin S (1970) An efficient heuristic procedure for partitioning graphs. Bell System Tech. J. 49(2):291–307.Crossref, Google Scholar
• Khan S, Ugander J (2021) Adaptive normalization for IPW estimation. Preprint, submitted June 14, https://arxiv.org/abs/2106.07695.Google Scholar
• Kim M, Fernandez RM (2017) Strength matters: Tie strength as a causal driver of networks’ information benefits. Soc. Sci. Res. 65:268–281.Crossref, Google Scholar
• Kohavi R, Tang D, Xu Y (2020) Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Kohavi R, Deng A, Frasca B, Walker T, Xu Y, Pohlmann N (2013) Online controlled experiments at large scale. Proc. 19th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 1168–1176.Google Scholar
• Koning R, Hasan S, Chatterji A (2022) Experimentation and start-up performance: Evidence from A/B testing. Management Sci. 68(9):6434–6453.Link, Google Scholar
• Leskovec J, Lang KJ, Dasgupta A, Mahoney MW (2009) Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters. Internet Math. 6(1):29–123.Crossref, Google Scholar
• Leung MP (2022) Causal inference under approximate neighborhood interference. Econometrica 90(1):267–293.Crossref, Google Scholar
• Leung MP, Loupos P (2022) Unconfoundedness with network interference. Preprint, submitted November 15, https://arxiv.org/abs/2211.07823.Google Scholar
• Li H, Zhao G, Johari R, Weintraub GY (2022) Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. Proc. ACM Web Conf. (ACM, New York), 182–192.Google Scholar
• Manski CF (2013) Identification of treatment response with social interactions. Econom. J. 16(1):S1–S23.Crossref, Google Scholar
• Milo R, Shen-Orr S, Itzkovitz S, Kashtan N, Chklovskii D, Alon U (2002) Network motifs: Simple building blocks of complex networks. Science 298(5594):824–827.Crossref, Google Scholar
• Munro E, Wager S, Xu K (2021) Treatment effects in market equilibrium. Preprint, submitted September 23, https://arxiv.org/abs/2109.11647.Google Scholar
• Pollmann M (2020) Causal inference for spatial treatments. Preprint, submitted October 31, https://arxiv.org/abs/2011.00373.Google Scholar
• Pouget-Abadie J, Mirrokni V, Parkes DC, Airoldi EM (2018) Optimizing cluster-based randomized experiments under monotonicity. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 2090–2099.Google Scholar
• Pouget-Abadie J, Saint-Jacques G, Saveski M, Duan W, Ghosh S, Xu Y, Airoldi EM (2019) Testing for arbitrary interference on experimentation platforms. Biometrika 106(4):929–940.Crossref, Google Scholar
• Qu Z, Xiong R, Liu J, Imbens G (2021) Efficient treatment effect estimation in observational studies under heterogeneous partial interference. Preprint, submitted July 26, https://arxiv.org/abs/2107.12420.Google Scholar
• Rubin DB (2005) Causal inference using potential outcomes: Design, modeling, decisions. J. Amer. Statist. Assoc. 100(469):322–331.Crossref, Google Scholar
• Saveski M, Pouget-Abadie J, Saint-Jacques G, Duan W, Ghosh S, Xu Y, Airoldi EM (2017) Detecting network effects: Randomizing over randomized experiments. Proc. 23rd ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining, 1027–1035.Google Scholar
• Sävje F (2024) Causal inference with misspecified exposure mappings: Separating definitions and assumptions. Biometrika 111(1):1–15.Crossref, Google Scholar
• Shen-Orr SS, Milo R, Mangan S, Alon U (2002) Network motifs in the transcriptional regulation network of Escherichia coli. Nature Genetics 31(1):64–68.Crossref, Google Scholar
• Sussman DL, Airoldi EM (2017) Elements of estimation theory for causal effects in the presence of network interference. Preprint, submitted February 12, https://arxiv.org/abs/1702.03578.Google Scholar
• Toulis P, Kao E (2013) Estimation of causal peer influence effects. Proc. Internat. Conf. Machine Learning (PMLR, New York), 1489–1497.Google Scholar
• Ugander J, Yin H (2020) Randomized graph cluster randomization. Preprint, submitted September 4, https://arxiv.org/abs/2009.02297.Google Scholar
• Ugander J, Backstrom L, Marlow C, Kleinberg J (2012) Structural diversity in social contagion. Proc. Natl. Acad. Sci. USA 109(16):5962–5966.Crossref, Google Scholar
• Ugander J, Karrer B, Backstrom L, Kleinberg J (2013) Graph cluster randomization: Network exposure to multiple universes. Proc. 19th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 329–337.Google Scholar
• Viviano D (2020) Experimental design under network interference. Preprint, submitted March 18, https://arxiv.org/abs/2003.08421.Google Scholar
• Wang Y (2021) Causal inference under temporal and spatial interference. Preprint, submitted June 29, https://arxiv.org/abs/2106.15074.Google Scholar
• Wang Y, Samii C, Chang H, Aronow P (2020) Design-based inference for spatial experiments with interference. Preprint, submitted October 26, https://arxiv.org/abs/2010.13599.Google Scholar
• Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small-world’ networks. Nature 393(6684):440–442.Crossref, Google Scholar
• Weisburd D, Telep CW (2014) Hot spots policing: What we know and what we need to know. J. Contemporary Criminal Justice 30(2):200–220.Crossref, Google Scholar
• Xu Y, Chen N, Fernandez A, Sinno O, Bhasin A (2015) From infrastructure to culture: A/B testing challenges in large scale social networks. Proc. 19th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (ACM, New York), 2227–2236.Google Scholar
• Yan Y, Zhao R, Liu Z (2018) Strategic introduction of the marketplace channel under spillovers from online to offline sales. Eur. J. Oper. Res. 267(1):65–77.Crossref, Google Scholar
• Yu CL, Airoldi EM, Borgs C, Chayes JT (2022) Estimating the total treatment effect in randomized experiments with unknown network structure. Proc. Natl. Acad. Sci. USA 119(44):e2208975119.Crossref, Google Scholar
• Yuan Y, Altenburger KM, Kooti F (2021) Causal network motifs: Identifying heterogeneous spillover effects in A/B tests. Proc. Web Conf., 3359–3370.Google Scholar
• Zigler CM, Papadogeorgou G (2021) Bipartite causal inference with interference. Statist. Sci. 36(1):109.Crossref, Google Scholar