A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook

References

• Abadie A, Imbens GW (2008) On the failure of the bootstrap for matching estimators. Econometrica 76(6):1537–1557.Crossref, Google Scholar
• Abraham M (2008) The off-line impact of online ads. Harvard Bus. Rev. 86(4):28.Google Scholar
• Altonji JG, Elder TE, Taber CR (2005) Selection on observed and unobserved variables: Assessing the effectiveness of catholic schools. J. Political Econom. 113(1):151–184.Crossref, Google Scholar
• Arceneaux K, Gerber AS, Green DP (2010) A cautionary note on the use of matching to estimate causal effects: An empirical example comparing matching estimates to an experimental benchmark. Sociol. Methods Res. 39(2):256–282.Crossref, Google Scholar
• Athey S, Imbens GW, Wager S (2018) Approximate residual balancing: De-biased inference of average treatment effects in high dimensions. J. Royal Statist. Soc. Ser. B 80(4):597–623.Crossref, Google Scholar
• Belloni A, Chen D, Chernozhukov V, Hansen C (2012) Sparse models and methods for optimal instruments with an application to eminent domain. Econometrica 80(6):2369–2429.Crossref, Google Scholar
• Benady D (2016) Can advertising support a free internet? The Guardian (November 7), https://www.theguardian.com/media-network/2016/nov/07/can-advertising-support-free-internet.Google Scholar
• Berkovich P, Wood L (2016) Using single-source data to measure advertising effectiveness. Nielsen J. Measurement 1(2), http://www.nielsen.com/us/en/insights/reports/2016/using-single-source-data-to-measure-advertising-effectiveness.html.Google Scholar
• Bickel P, Götze F, van Zwet WR (1997) Resampling fewer than n observations: Gains, losses, and remedies for losses. Statistica Sinicia 7(1):1–31.Google Scholar
• Blake T, Nosko C, Tadelis S (2015) Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica 83(1):155–174.Crossref, Google Scholar
• Bond D (2017) Advertising agencies squeezed by tech giants. The Financial Times (June 25), https://www.ft.com/content/9a9ac60a-575a-11e7-9fed-c19e2700005f.Google Scholar
• Bronnenberg BJ, Dubé J-P, Mela CF (2010) Do digital video recorders influence sales? J. Marketing Res. 47(6):998–1010.Crossref, Google Scholar
• Caliendo M, Kopeinig S (2008) Some practical guidance for the implementation of propensity score matching. J. Econom. Surveys 22(1):31–72.Crossref, Google Scholar
• Cochran WG (1968) The effectiveness of adjustment by subclassifcation in removing bias in observational studies. Biometrics 24(2):295–314.Crossref, Google Scholar
• comScore (2010) comScore announces introduction of AdEffx smart controlTM ground-breaking methodology for measuring digital advertising effectiveness. Press release (May 25), comScore, Inc., Reston, VA, https://www.comscore.com/Insights/Press-Releases/2010/5/comScore-Announces-Introduction-of-AdEffx-Smart-Control-Ground-Breaking-Methodology-for-Measuring-Digital-Advertising-Effectiveness?cs_edgescape_cc=US.Google Scholar
• Cooper MJ, Gulen H, Rau PR (2005) Changing names with style: Mutual fund name changes and their effects on fund flows. J. Finance 60(6):2825–2858.Crossref, Google Scholar
• Dehejia RH, Wahba S (2002) Propensity score matching methods for non-experimental causal studies. Rev. Econom. Statist. 84(1): 151–161.Crossref, Google Scholar
• Draganska M, Hartmann WR, Stanglein G (2014) Internet versus television advertising: A brand-building comparison. J. Marketing Res. 51(5):578–590.Crossref, Google Scholar
• Eckles D, Bakshy E (2017) Bias and high-dimensional adjustment in observational studies of Peer effects. Working paper, MIT Sloan School of Management, Cambridge, MA.Google Scholar
• Ellickson P, Collins RL, Hambarsoomians K, McCaffrey DF (2005) Does alcohol advertising promote adolescent drinking? Results from a longitudinal assessment Addiction 100(2):235–246.Crossref, Google Scholar
• Ferraro PJ, Miranda JJ (2014) The performance of non-experimental designs in the evaluation of environmental programs: A design-replication study using a large-scale randomized experiment as a benchmark. J. Econom. Behav. Organ. 107:344–365.Crossref, Google Scholar
• Ferraro PJ, Miranda JJ (2017) Panel data designs and estimators as substitutes for randomized controlled trials in the evaluation of public programs. J. Assoc. Environ. Resource Econom. 4(1): 281–317.Google Scholar
• Gelman A, Hill J (2007) Data Analysis Using Regression and Multilevel/Hierarchical Models, 1st ed. (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Gluck M (2011) Best practices for conducting online ad effectiveness research. Report, Interactive Advertising Bureau, New York.Google Scholar
• Goldfarb A, Tucker C (2011) Online display advertising: Targeting and obtrusiveness. Marketing Sci. 30(3):389–404.Link, Google Scholar
• Gu X, Rosenbaum P (1993) Comparison of multivariate matching methods: Structures, distances, and algorithms. J. Comput. Graphical Statist. 2(4):405–420.Google Scholar
• Hirano K, Imbens GW, Ridder G (2003) Efficient estimation of average treatment effects using the estimated propensity score. Econometrica 71(4):1161–1189.Crossref, Google Scholar
• Ichino A, Fabrizia M, Nannicini T (2008) From temporary help jobs to permanent employment: What can we learn from matching estimators and their sensitivity? J. Appl. Econom. 23(3):305–327.Crossref, Google Scholar
• Imai K, Ratkovic M (2014) Covariate balancing propensity score. J. Royal Statist. Soc. Ser. B 76(1):243–263.Crossref, Google Scholar
• Imbens GW (2003) Sensitivity to exogeneity assumptions in program evaluation. Amer. Econom. Rev. 93(2):126–132.Crossref, Google Scholar
• Imbens GW (2004) Nonparametric estimation of average treatment effects under exogeneity: A review. Rev. Econom. Statist. 86(1):4–29.Crossref, Google Scholar
• Imbens GW (2015) Matching methods in practice: Three examples. J. Human Resources 50(2):373–419.Crossref, Google Scholar
• Imbens GW, Angrist JD (1994) Identification and estimation of local average treatment effects. Econometrica 62(2):467–475.Crossref, Google Scholar
• Imbens GW, Rubin DB (2015) Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, 1st ed. (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
• Imbens GW, Wooldridge JM (2009) Recent developments in the econometrics of program evaluation. J. Econom. Lit. 47(1):5–86.Crossref, Google Scholar
• Johnson GA, Lewis RA, Nubbemeyer EI (2017a) Ghost ads: Improving the economics of measuring ad effectiveness. J. Marketing Res. 54(6):867–884.Crossref, Google Scholar
• Johnson GA, Lewis RA, Nubbemeyer E (2017b) The online display ad effectiveness funnel & carryover: Lessons from 432 field experiments. Working paper, Questrom School of Business, Boston University, Boston.Google Scholar
• Johnson GA, Lewis RA, Reiley DH (2016) Location, location, location: Repetition and proximity increase advertising effectiveness. Working paper, University of Rochester, Rochester, NY.Google Scholar
• Johnson GA, Lewis RA, Reiley DH (2017c) When less is more: Data and power in advertising experiments. Marketing Sci. 36(1):43–53.Link, Google Scholar
• Kalyanam K, McAteer J, Marek J, Hodges J, Lin L (2018) Cross channel effects of search engine advertising on brick & mortar retail sales: Meta analysis of large scale field experiments on Google.com. Quant. Marketing Econom. 16(1):1–42.Crossref, Google Scholar
• Klein C, Wood L (2013) Cross platform sales impact: Cracking the code on single source. Report, Nielsen Catalina Solutions and Time Inc., New York.Google Scholar
• Kumar A, Bezawada R, Rishika R, Janakiraman R, Kannan P (2016) From social to sale: The effects of firm-generated content in social media on customer behavior. J. Marketing 80(1):7–25.Crossref, Google Scholar
• LaLonde RJ (1986) Evaluating the econometric evaluations of training programs with experimental data. Amer. Econom. Rev. 76(4):604–620.Google Scholar
• Lavrakas P (2010) An evaluation of methods used to assess the effectiveness of advertising on the internet. Report, Interactive Advertising Bureau, New York.Google Scholar
• Lewis R, Rao J (2015) The unfavorable economics of measuring the eturns to advertising. Quart. J. Econom. 130(4):1941–1973.Crossref, Google Scholar
• Lewis R, Reiley D (2014) Online ads and offline sales: Measuring the effect of retail advertising via a controlled experiment on Yahoo! Quant. Marketing Econom. 12(3):235–266.Crossref, Google Scholar
• Lewis R, Rao J, Reiley D (2011) Here, there, and everywhere: Correlated online behaviors can lead to overestimtes of the effects of advertising. Proc. 20th Internat. Conf. World Wide Web (Association for Computing Machines, New York), 157–166.Crossref, Google Scholar
• McCaffrey DF, Ridgeway G, Morral AR (2004) Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psych. Methods 9(4):403–425.Crossref, Google Scholar
• Murphy K, Topel R (1990) Eciency wages reconsidered: Theory and evidence. Weiss Y, Fishelson G, eds. Advances in the Theory and Measurement of Unemployment, vol. 2 (Palgrave Macmillan, London), 204–240.Crossref, Google Scholar
• Oster E (2017) Unobservable selection and coefficient stability: Theory and evidence. J. Bus. Econom. Statist., ePub ahead of print June 1, https://doi.org/10.1080/07350015.2016.1227711.Google Scholar
• Pham TT, Shen Y (2017) A deep causal inference approach to measuring the effects of forming group loans in online non-profit microfinance platform. Working paper, Stanford Graduate School of Business, Stanford, CA. Available at https://arxiv.org/pdf/1706.02795.pdf.Google Scholar
• Politis DN, Romano JP (1994) Large sample confidence regions based on subsamples under minimal assumptions. Ann. Statist. 22(4):2031–2050.Crossref, Google Scholar
• Robins J, Ritov Y (1997) Towards a curse of dimensionality appropriate (CODA) asymptotic theory for semi-parametric models. Statist. Medicine 16(3):285–319.Crossref, Google Scholar
• Rosenbaum P, Rubin DB (1983a) Assessing sensitivity to an unobserved binary covariate in an observational study with binary outcome. J. Royal Statist. Soc. Ser. B 45(2):212–218.Google Scholar
• Rosenbaum PR, Rubin DB (1983b) The central role of the propensity score in observational studies for causal effects. Biometrica 70(1):41–55.Crossref, Google Scholar
• Rosenbaum PR, Rubin DB (1985) Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. Amer. Statist. 39(1):33–38.Crossref, Google Scholar
• Rubin DB (1978) Bayesian inference for causal effects: The role of randomization. Ann. Statist. 6(1):34–58.Crossref, Google Scholar
• Rubin DB (2001) Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services Outcomes Res. Methodology 2(3/4):169–188.Crossref, Google Scholar
• Rubin DB, Waterman RP (2007) Estimating the causal effects of marketing interventions using propensity score methodology. Statist. Sci. 21(2):206–222.Crossref, Google Scholar
• Sahni N (2015) Effect of temporal spacing between advertising exposures: Evidence from online field experiments. Quant. Marketing Econom. 13(3):203–247.Crossref, Google Scholar
• Sahni N, Nair H (2018) Sponsorship disclosure and consumer deception: Experimental evidence from native advertising in mobile search. Working paper, Stanford Graduate School of Business, Stanford, CA.Google Scholar
• Stuart EA (2010) Matching methods for causal inference: A review and a look forward. Statist. Sci. 25(1):1–21.Crossref, Google Scholar
• Stuart EA, Rubin DB (2007) Matching methods for causal inference: Designing observational studieschap. Best Practices in Quantitative Methods (Sage Publishing, Thousand Oaks, CA).Google Scholar
• Tibshirani R (1996) Regression shrinkage and selection via the Lasso. J. Royal Statist. Soc. 58(1): 267–288.Google Scholar
• Westreich D, Lessler J, Funk MJ (2010) Propensity score estimation: Machine learning and classification methods as alternatives to logistic regression. J. Clin. Epidemiol. 63(8):826–833.Crossref, Google Scholar
• Wooldridge JM (2007) Inverse probability weighted estimation for general missing data problems. J. Econom. 141(2)1281–1301.Crossref, Google Scholar
• Zubizarreta JR (2012) Using mixed integer programming for matching in an observational study of kidney failure after surgery. J. Amer. Statist. Assoc. 107(500):1360–1371.Crossref, Google Scholar
• Zubizarreta JR (2015) Stable weights that balance covariates for estimation with incomplete outcome data. J. Amer. Statist. Assoc. 110(511):910–922.Crossref, Google Scholar