Open Access

Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

David Holtz
Corresponding Author
David Holtz
[email protected]
https://orcid.org/0000-0002-0896-8628
Management of Organizations and Entrepreneurship and Innovation, Haas School of Business, University of California, Berkeley, California 94720;MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142;
Search for more papers by this author
,
Felipe Lobel
Felipe Lobel
[email protected]
https://orcid.org/0000-0002-4603-4594
Department of Economics, University of California, Berkeley, California 94720;
Search for more papers by this author
,
Ruben Lobel
Ruben Lobel
[email protected]
https://orcid.org/0000-0002-5824-8442
Airbnb, San Francisco, California 94103;
Search for more papers by this author
,
Inessa Liskovich
Inessa Liskovich
[email protected]
Airbnb, San Francisco, California 94103;
Search for more papers by this author
,
Sinan Aral
Sinan Aral
[email protected]
https://orcid.org/0000-0002-2762-058X
MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142;Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02142
Search for more papers by this author

Corresponding Author

David Holtz

Management of Organizations and Entrepreneurship and Innovation, Haas School of Business, University of California, Berkeley, California 94720;MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142;

Search for more papers by this author

Felipe Lobel

[email protected]

https://orcid.org/0000-0002-4603-4594

Department of Economics, University of California, Berkeley, California 94720;

Search for more papers by this author

Ruben Lobel

[email protected]

https://orcid.org/0000-0002-5824-8442

Airbnb, San Francisco, California 94103;

Search for more papers by this author

Inessa Liskovich

[email protected]

Airbnb, San Francisco, California 94103;

Search for more papers by this author

Sinan Aral

[email protected]

https://orcid.org/0000-0002-2762-058X

MIT Initiative on the Digital Economy, MIT Sloan School of Management, Cambridge, Massachusetts 02142;Sloan School of Management, Massachusetts Institute of Technology, Cambridge, Massachusetts 02142

Search for more papers by this author

Published Online:5 Apr 2024https://doi.org/10.1287/mnsc.2020.01157

References

Airbnb (2019) Airbnb news: About us. Accessed September 1, 2019, https://news.airbnb.com/about-us/.Google Scholar
Aral S, Nicolaides C (2017) Exercise contagion in a global social network. Nat. Comm. 8(1):14753.Crossref, Google Scholar
Ariel B, Sutherland A, Sherman LW (2019) Preventing treatment spillover contamination in criminological field experiments: The case of body-worn police cameras. J. Experiment. Criminology 15(4):569–591.Crossref, Google Scholar
Aronow PM (2012) A general method for detecting interference between units in randomized experiments. Sociol. Methods Res. 41(1):3–16.Crossref, Google Scholar
Aronow PM, Samii C (2017) Estimating average causal effects under general interference, with application to a social network experiment. Ann. Appl. Statist. 11(4):1912–1947.Crossref, Google Scholar
Athey S, Eckles D, Imbens GW (2018) Exact p-values for network interference. J. Amer. Statist. Assoc. 113(521):230–240.Crossref, Google Scholar
Azevedo EM, Deng A, Montiel Olea JL, Rao J, Weyl EG (2020) A/B testing with fat tails. J. Political Econom. 128(12):4614.Crossref, Google Scholar
Baird S, Bohren JA, McIntosh C, Özler B (2018) Optimal design of experiments in the presence of interference. Rev. Econom. Statist. 100(5):844–860.Crossref, Google Scholar
Basse G, Feller A (2018) Analyzing two-stage experiments in the presence of interference. J. Amer. Statist. Assoc. 113(521):41–55.Crossref, Google Scholar
Blake T, Coey D (2014) Why marketplace experimentation is harder than it seems: The role of test-control interference. Proc. Fifteenth ACM Conf. Econom. Comput. (Association for Computing Machinery, New York), 567–582.Google Scholar
Blondel VD, Guillaume JL, Lambiotte R, Lefebvre E (2008) Fast unfolding of communities in large networks. J. Statist. Mech. Theory Experiment. 2008(10):P10008.Crossref, Google Scholar
Bojinov I, Simchi-Levi D, Zhao J (2022) Design and analysis of switchback experiments. Management Sci. 69(7):3759–3777.Link, Google Scholar
Bond RM, Fariss CJ, Jones JJ, Kramer AD, Marlow C, Settle JE, Fowler JH (2012) A 61-million-person experiment in social influence and political mobilization. Nature 489(7415):295–298.Crossref, Google Scholar
Bowers J, Fredrickson MM, Panagopoulos C (2013) Reasoning about interference between units: A general framework. Political Anal. 21(1):97–124.Crossref, Google Scholar
Bright I, Delarue A, Lobel I (2023) Reducing marketplace interference bias via shadow prices. Proc. 24th ACM Conf. Econom. Comput. (Association for Computing Machinery, New York), 300.Google Scholar
Chin A (2019) Regression adjustments for estimating the global treatment effect in experiments with interference. J. Causal Inference 7(2):20180026Google Scholar
Choi H, Mela CF (2019) Monetizing online marketplaces. Marketing Sci. 38(6):948–972.Abstract, Google Scholar
Cox DR (1958) Planning of Experiments (Wiley, New York).Google Scholar
Dubé JP, Misra S (2023) Personalized pricing and consumer welfare. J. Political Economy 131(1):131–189.Google Scholar
Eckles D, Karrer B, Ugander J (2017) Design and analysis of experiments in networks: Reducing bias from interference. J. Causal Inference 5(1):20150021.Crossref, Google Scholar
Feit EM, Berman R (2019) Test & roll: Profit-maximizing A/B tests. Marketing Sci. 38(6):1038–1058.Link, Google Scholar
Filippas A, Jagabathula S, Sundararajan A (2023) The limits of centralized pricing in online marketplaces and the value of user control. Management Sci. 69(12):7202–7216.Google Scholar
Fradkin A (2015) Search frictions and the design of online marketplaces. Working paper, Massachusetts Institute of Technology, Cambridge.Google Scholar
Gerber AS, Green DP (2012) Field Experiments: Design, Analysis, and Interpretation (W. W. Norton, New York).Google Scholar
Grbovic M, Cheng H (2018) Real-time personalization using embeddings for search ranking at Airbnb. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 311–320.Google Scholar
Holtz DM (2018) Limiting bias from test-control interference in online marketplace experiments. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA.Google Scholar
Holtz D, Zhao M, Benzell SG, Cao CY, Rahimian MA, Yang J, Allen J, et al. (2020) Interdependence and the cost of uncoordinated responses to COVID-19. Proc. Natl. Acad. Sci. USA 117(33):19837–19843.Crossref, Google Scholar
Hudgens MG, Halloran ME (2008) Toward causal inference with interference. J. Amer. Statist. Assoc. 103(482):832–842.Crossref, Google Scholar
Ifrach B, Holtz DM, Yee YH, Zhang L (2016) Demand prediction for time-expiring inventory. U.S. Patent Application No. 14/952,576.Google Scholar
Imai K, Tingley D, Yamamoto T (2013) Experimental designs for identifying causal mechanisms. J. Roy. Statist. Soc. Ser. A 176(1):5–51.Crossref, Google Scholar
Johari R, Li H, Liskovich I, Weintraub GY (2022) Experimental design in two-sided platforms: An analysis of bias. Management Sci. 68(10):7069–7089.Link, Google Scholar
Kang JH, Park CH, Kim SB (2016) Recursive partitioning clustering tree algorithm. PAA Pattern Anal. Appl. 19(2):355–367.Crossref, Google Scholar
Li H, Zhao G, Johari R, Weintraub GY (2022) Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. Proc. ACM Web Conf. 2022 (Association for Computing Machinery, New York), 182–192.Google Scholar
Liu L, Hudgens MG (2014) Large sample randomization inference of causal effects in the presence of interference. J. Amer. Statist. Assoc. 109(505):288–301.Crossref, Google Scholar
Liu M, Mao J, Kang K (2021) Trustworthy and powerful online marketplace experimentation with budget-split design. Proc. 27th ACM SIGKDD Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 3319–3329.Google Scholar
Mahalanobis PC (1936) On the generalized distance in statistics. Proc. Natl. Inst. Sci. (Calcutta) 2:49–55.Google Scholar
Manski CF (2000) Economic analysis of social interactions. J. Econom. Perspect. 14(3):115–136.Crossref, Google Scholar
Mikolov T, Chen K, Corrado G, Dean J (2013a) Efficient estimation of word representations in vector space. Preprint, submitted January 16, https://arxiv.org/abs/1301.3781.Google Scholar
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013b) Distributed representations of words and phrases and their compositionality. Burges CJ, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, eds. Advances in Neural Information Processing Systems, vol. 26 (Curran Associates Inc. Red Hook, NY), 3111–3119.Google Scholar
Miratrix LW, Sekhon JS, Yu B (2013) Adjusting treatment effect estimates by post-stratification in randomized experiments. J. Roy. Statist. Soc. Ser. B Statist. Methodology 75(2):369–396.Crossref, Google Scholar
Moffitt RA (2001) Policy interventions, low-level equilibria, and social interactions. Durlauf SN, Young HP, eds. Social Dynamics (MIT Press, Cambridge, MA), 6–17.Crossref, Google Scholar
Moore RT (2012) Multivariate continuous blocking to improve political science experiments. Political Anal. 20(4):460–479.Crossref, Google Scholar
Pouget-Abadie J, Mirrokni V, Parkes DC, Airoldi EM (2018) Optimizing cluster-based randomized experiments under monotonicity. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 2090–2099.Google Scholar
Rosenbaum PR (2007) Interference between units in randomized experiments. J. Amer. Statist. Assoc. 102(477):191–200.Crossref, Google Scholar
Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J. Ed. Psych. 66(5):688–701.Crossref, Google Scholar
Saveski M, Pouget-Abadie J, Saint-Jacques G, Duan W, Ghosh S, Xu Y, Airoldi EM (2017) Detecting network effects: Randomizing over randomized experiments. Proc. 23rd ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 1027–1035.Google Scholar
Sävje F, Aronow P, Hudgens M (2021) Average treatment effects in the presence of unknown interference. Ann. Statist. 49(2):673–701.Crossref, Google Scholar
Sinclair B, McConnell M, Green DP (2012) Detecting spillover effects: Design and analysis of multilevel experiments. Amer. J. Political Sci. 56(4):1055–1069.Crossref, Google Scholar
Srinivasan S (2018) Learning market dynamics for optimal pricing. Medium (August 10), https://medium.com/airbnb-engineering/learning-market-dynamics-for-optimal-pricing-97cffbcc53e3.Google Scholar
Tchetgen EJT, VanderWeele TJ (2012) On causal inference in the presence of interference. Statist. Methods Medical Res. 21(1):55–75.Crossref, Google Scholar
Ugander J, Karrer B, Backstrom L, Kleinberg J (2013) Graph cluster randomization: Network exposure to multiple universes. Proc. 19th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 329–337.Google Scholar
Ye P, Qian J, Chen J, Wu CH, Zhou Y, De Mars S, Yang F, Zhang L (2018) Customized regression model for Airbnb dynamic pricing. Proc. 24th ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 932–940.Google Scholar

Volume 71, Issue 1

January 2025

Pages iv-vi, 1-953, vii-vii

Article Information

Supplemental Material

Metrics

Information

Received:April 23, 2020
Accepted:July 11, 2023
Published Online:April 05, 2024

Cite as

David Holtz, Felipe Lobel, Ruben Lobel, Inessa Liskovich, Sinan Aral (2024) Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb. Management Science 71(1):390-406.

https://doi.org/10.1287/mnsc.2020.01157

Keywords

Acknowledgments

The authors are grateful to Lanbo Zhang, Minyong Lee, and Sharan Srinivasan for their assistance with the design and analysis of the experiments in this paper. The authors thank numerous other Airbnb employees who have assisted this project and are grateful to Iris Fung, Jonathan Niles-Weed, and Peter Coles for their assistance. The authors also appreciate the helpful feedback they received from Dean Eckles, Andrey Fradkin, Hannah Li, Alex Moehring, Hong Yi Tu Ye, the Berkeley Haas Management of Organizations (MORS) Macro Research Lunch, the Massachusetts Institute of Technology (MIT) Sloan Social Analytics Laboratory, and attendees of the 2019 Winter Conference on Business Analytics and the Harvard Business School Digital Doctoral Workshop. This experiment was classified as exempt by the MIT Committee on the Use of Humans as Experimental Subjects under Protocol 1807452488. David Holtz was previously an employee of Airbnb between 2014 and 2016 and was an unpaid contractor while working on this paper. He currently holds a material financial interest in Airbnb, Inc. Ruben Lobel and Inessa Liskovich are currently full-time employees of Airbnb. Felipe Lobel was previously a paid intern at Airbnb and was an unpaid contractor while working on this paper. Airbnb had the right to review this paper subject to the following stipulations: (1) delaying disclosure or publication until any intellectual property rights have been adequately protected; (2) removing Airbnb confidential information from the work product; and (3) adding Airbnb as an author, as appropriate, of the work product and/or adding an appropriate disclosure concerning Airbnb’s support of the project.

PDF download

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Available Issues

Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

References

Volume 71, Issue 1

Article Information

Supplemental Material

Metrics

Information

Cite as

Keywords

Sign Up for INFORMS Publications Updates and News