Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement

References

• Angrist JD, Imbens GW, Rubin DB (1996) Identification and estimation of causal effects using instrumental variables. J. Amer. Statist. Assoc. 91(434):444–455.Crossref, Google Scholar
• Athey S, Imbens GW (2019) Machine learning methods that economists should know about. Annual Rev. Econom. 11:685–725.Crossref, Google Scholar
• Athey S, Tibshirani R, Wager S (2019) Generalized random forests. Ann. Statist. 47(2):1148–1178.Crossref, Google Scholar
• Blake T, Nosko C, Tadelis S (2015) Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica 83(1):155–174.Crossref, Google Scholar
• Chernozhukov V, Chetverikov D, Demirer M, Duflo E, Hansen C, Newey W, Robins J (2018) Double/debiased machine learning for treatment and structural parameters. Econom. J. 21:C1–C68.Crossref, Google Scholar
• Coey D, Bailey M (2016) People and cookies: Imperfect treatment assignment in online experiments. WWW ’16: Proceedings of the 25th International Conference on World Wide Web, 1103–1111.Google Scholar
• Cohen J (1977) Statistical Power Analysis for the Behavioral Sciences (Lawrence Erlbaum Associates, Inc., Mahwah, NJ).Google Scholar
• Dehejia RH, Wahba S (2002) Propensity score matching methods for non-experimental causal studies. Rev. Econom. Stat. 84(1):151–161.Crossref, Google Scholar
• Du R, Zhong Y, Nair H, Cui B, Shou R (2019) Causally driven incremental multi touch attribution using a recurrent neural network. Preprint, submitted February 1, https://arxiv.org/abs/1902.00215.Google Scholar
• Fernández-Loría C, Provost F (2022) Causal decision making and causal effect estimation are not the same…and why it matters. INFORMS J. Data Sci. 1(1):4–16.Link, Google Scholar
• Gelman A, Hill J (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models, 1st ed. (Cambridge University Press, Cambridge, United Kingdom).Crossref, Google Scholar
• Gordon BR, Jerath K, Katona Z, Narayanan S, Shin J, Wilbur KC (2021) Inefficiencies in digital advertising markets. J. Marketing 85(1):7–25.Crossref, Google Scholar
• Gordon BR, Moakler R, Zettelmeyer F (2022) Predicting incrementality by experimentation for ad measurement. Working paper, Kellogg School of Management, Evanston, IL.Google Scholar
• Gordon BR, Zettelmeyer F, Bhargava N, Chapsky D (2019) A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Sci. 38(2):193–225.Link, Google Scholar
• Greenwell BM (2017) pdp: An R package for constructing partial dependence plots. R J. 9(1):421–436.Crossref, Google Scholar
• Gui G, Nair H, Niu F (2021) Auction throttling and causal inference of online advertising effects. Preprint, submitted December 30, https://arxiv.org/abs/2112.15155.Google Scholar
• Hartford J, Lewis G, Leyton-Brown K, Taddy M (2017) Deep IV: A flexible approach for counterfactual prediction. Precup D, Teh YW, eds. Proc. 34th Internat. Conf. Machine Learning, vol. 70 of Proc. Machine Learning Res., 1414–1423.Google Scholar
• Heckman JJ, Ichimura H, Todd PE (1997) Matching as an econometric evaluation estimator: Evidence from evaluating a job training programme. Rev. Econom. Stud. 64(4):605–654.Crossref, Google Scholar
• Hoban P, Arora N (2018) Measuring display advertising response using observational data: The impact of selection biases. Preprint, submitted October 1, http://dx.doi.org/10.2139/ssrn.3264871.Google Scholar
• Imbens G, Rubin DB (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. 1st ed. (Cambridge University Press, Cambridge, United Kingdom).Crossref, Google Scholar
• Imbens GW (2004) Nonparametric estimation of average treatment effects under exogeneity: A review. Rev. Econom. Stat. 86(1):4–29.Crossref, Google Scholar
• Imbens GW, Angrist JD (1994) Identification and estimation of local average treatment effects. Econometrica 62(2):467–475.Crossref, Google Scholar
• Imbens GW, Wooldridge JM (2009) Recent developments in the econometrics of program evaluation. J. Econom. Lit. 47(1):5–86.Crossref, Google Scholar
• Johnson GA (2022) Inferno: A guide to field experiments in online display advertising. Preprint, submitted July 18, http://dx.doi.org/10.2139/ssrn.3581396.Google Scholar
• Johnson GA, Lewis RA, Nubbemeyer EI (2017a) Ghost ads: Improving the economics of measuring ad effectiveness. J. Marketing Res. 54(6):867–884.Crossref, Google Scholar
• Johnson GA, Lewis RA, Nubbemeyer EI (2017b) The online display ad effectiveness funnel & carryover: Lessons from 432 field experiments. Preprint, submitted October 1, http://dx.doi.org/10.2139/ssrn.2701578.Google Scholar
• Kohavi R, Tang D, Xu Y (2020) Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, 1st ed. (Cambridge University Press, Cambridge, United Kingdom).Crossref, Google Scholar
• LaLonde RJ (1986) Evaluating the econometric evaluations of training programs with experimental data. Amer. Econom. Rev. 76(4):604–620.Google Scholar
• Lewis R, Rao J, Reiley D (2011) Here, there, and everywhere: Correlated online behaviors can lead to overestimates of the effects of advertising. Proc. 20th Internat. Conf. World Wide Web (Association for Computing Machines), 157–166.Google Scholar
• Lewis R, Rao J, Reiley D (2015) Measuring the effects of advertising: The digital frontier. Goldfarb A, Greenstein S, Tucker C, eds. Economic Analysis of the Digital Economy, Chapter 7 (University of Chicago Press, Chicago), 191–218.Crossref, Google Scholar
• Lewis RA, Wong J (2018) Incrementality bidding and attribution. Preprint, submitted February 27, https://ssrn.com/abstract=3129350.Google Scholar
• Lin T, Misra S (2022) Frontiers: The identity fragmentation bias. Marketing Sci. 41(3):433–440.Link, Google Scholar
• Naumov M, Mudigere D, Shi HM, Huang J, Sundaraman N, Park J, Wang X, et al.. (2019) Deep learning recommendation model for personalization and recommendation systems. Preprint, submitted May 31, https://arxiv.org/abs/1906.00091.Google Scholar
• Rosenbaum P, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrica 70:41–55.Crossref, Google Scholar
• Rubin DB (1978) Bayesian inference for causal effects: The role of randomization. Ann. Statist. 6:34–58.Crossref, Google Scholar
• Runge J, Geinitz S, Ejdemyr S (2020) Experimentation and performance in advertising: An observational survey of firm practices on Facebook. Expert Syst. Appl. 158:113554.Crossref, Google Scholar
• Runge J, Nair H (2021) The role of randomized control trials in online demand generation: Exploratory evidence from Facebook. Proceedings of the International Conference on Information Systems (ICIS). Preprint, submitted September 27, http://dx.doi.org/10.2139/ssrn.3794028.Google Scholar
• Schiff A (2020) Facebook conversion lift measurement issue goes undetected for 12 months. https://www.adexchanger.com/platforms/facebook-conversion-lift-measurement-issue-goes-undetected-for-12-months/Google Scholar
• Shapiro B, Hitsch G, Tuchman A (2021) TV advertising effectiveness and profitability: Generalizable results from 288 brands. Econometrica 89(4):1855–1879.Crossref, Google Scholar
• Stuart EA (2010) Matching methods for causal inference: A review and a look forward. Statist. Sci. 25(1):1–21.Crossref, Google Scholar
• Syrgkanis V, Lewis G, Oprescu M, Hei M, Battocchi K, Dillon E, Pan J, et al.. (2021) Causal inference and machine learning in practice with econml and causalml: Industrial use cases at Microsoft, TripAdvisor, Uber. KDD ’21: Proc. 27th ACM SIGKDD Conf. Knowledge Discovery Data Mining, 4072–4073.Google Scholar
• Tunuguntla S (2022) Display ad measurement using observational data: A reinforcement learning approach. Working paper, Duke University, Durham, NC.Google Scholar
• Waisman C, Nair H, Carrion C (2022) Online causal inference for advertising in real-time bidding auctions. Preprint, submitted August 22, https://arxiv.org/abs/1908.08600.Google Scholar