Overall Evaluation Criteria for A/B Tests: A Meta-Analytic Approach

Published Online:https://doi.org/10.1287/mksc.2025.0164

References

  • Ansari A, Mela CF (2003) E-customization. J. Marketing Res. 40(2):131–145.Crossref, Google Scholar
  • Ascarza E (2018) Retention futility: Targeting high-risk customers might be ineffective. J. Marketing Res. 55(1):80–98.Crossref, Google Scholar
  • Athey S, Chetty R, Imbens GW, Kang H (2026) The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Rev. Econom. Stud. 93(4):2284–2312.Crossref, Google Scholar
  • Azevedo EM, Deng A, Montiel Olea JL, Weyl EG (2019) Empirical Bayes estimation of treatment effects with many A/B tests: An overview. AEA Papers Proc. 109:43–47.Crossref, Google Scholar
  • Azevedo EM, Deng A, Montiel Olea JL, Rao J, Weyl EG (2020) A/B testing with fat tails. J. Political Econom. 128(12):4614–4672.Crossref, Google Scholar
  • Berger JO (1985) Statistical Decision Theory and Bayesian Analysis, Springer Series in Statistics, 2nd ed. (Springer-Verlag, New York).Crossref, Google Scholar
  • Berman R (2018) Beyond the last touch: Attribution in online advertising. Marketing Sci. 37(5):771–792.Link, Google Scholar
  • Blei DM, Jordan MI (2006) Variational inference for Dirichlet process mixtures. Bayesian Anal. 1(1):121–143.Crossref, Google Scholar
  • Brooks S, Gelman A, Jones G, Meng XL (2011) Handbook of Markov Chain Monte Carlo (CRC Press, Boca Raton, FL).Crossref, Google Scholar
  • Brown LD, Purves R (1973) Measurable selections of extrema. Ann. Statist. 1(5):902–912.Crossref, Google Scholar
  • Bumbaca F, Misra S, Rossi PE (2020) Scalable target marketing: Distributed Markov chain Monte Carlo for Bayesian hierarchical models. J. Marketing Res. 57(6):999–1018.Crossref, Google Scholar
  • Burr D, Doss H (2005) A Bayesian semiparametric model for random-effects meta-analysis. J. Amer. Statist. Assoc. 100(469):242–251.Crossref, Google Scholar
  • Buyse M, Molenberghs G, Burzykowski T, Renard D, Geys H (2000) The validation of surrogate endpoints in meta-analyses of randomized experiments. Biostatistics 1(1):49–67.Crossref, Google Scholar
  • Conlon A, Taylor J, Li Y, Diaz-Ordaz K, Elliott M (2017) Links between causal effects and causal association for surrogacy evaluation in a Gaussian setting. Statist. Medicine 36(27):4243–4265.Crossref, Google Scholar
  • Cornish EA (1954) The multivariate t-distribution associated with a set of normal sample deviates. Australian J. Phys. 7(4):531–542.Crossref, Google Scholar
  • Deng A (2015) Objective Bayesian two sample hypothesis testing for online controlled experiments. Gangemi A, Leonardi S, Panconesi A (General Chairs), eds. Proc. 24th Internat. Conf. World Wide Web (Association for Computing Machinery, New York), 923–928.Google Scholar
  • Deng A, Shi X (2016) Data-driven metric development for online controlled experiments: Seven lessons learned. Krishnapuram B, Shah M (General Chairs), Smola A, Aggarwal C, Shen D, Rastogi R (Program Chairs), eds. Proc. 22nd ACM SIGKDD Internat. Conf. Knowledge Discovery Data Mining (Association for Computing Machinery, New York), 77–86.Google Scholar
  • Deng A, Xu Y, Kohavi R, Walker T (2013) Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. Leonardi S, Panconesi A (General Chairs), Ferragina P, Gionis A (Program Chairs), eds. Proc. Sixth ACM Internat. Conf. Web Search Data Mining (Association for Computing Machinery, New York), 123–132.Google Scholar
  • Deng A, Hagar L, Stevens N, Xifara T, Yuan LH, Gandhi A (2023) From augmentation to decomposition: A new look at CUPED in 2023. Preprint, submitted December 5, https://doi.org/10.48550/arXiv.2312.02935.Google Scholar
  • DerSimonian R, Laird N (1986) Meta-analysis in clinical trials. Controlled Clinical Trials 7(3):177–188.Crossref, Google Scholar
  • DerSimonian R, Laird N (2015) Meta-analysis in clinical trials revisited. Contemporary Clinical Trials 45(Pt A):139–145.Crossref, Google Scholar
  • Dew R, Ansari A (2018) Bayesian nonparametric customer base analysis with model-based visualizations. Marketing Sci. 37(2):216–235.Link, Google Scholar
  • Ding P (2016) On the conditional distribution of the multivariate t distribution. Amer. Statistician 70(3):293–295.Crossref, Google Scholar
  • Efron B (2014) Two modeling strategies for empirical Bayes estimation. Statist. Sci. 29(2):285–301.Crossref, Google Scholar
  • Elliott MR (2023) Surrogate endpoints in clinical trials. Annual Rev. Statist. Appl. 10(1):75–96.Crossref, Google Scholar
  • Elliott MR, Conlon AS, Li Y, Kaciroti N, Taylor JM (2015) Surrogacy marker paradox measures in meta-analytic settings. Biostatistics 16(2):400–412.Crossref, Google Scholar
  • Escobar MD, West M (1995) Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90(430):577–588.Crossref, Google Scholar
  • Feuerriegel S, Frauen D, Melnychuk V, Schweisthal J, Hess K, Curth A, Bauer S, Kilbertus N, Kohane IS, van der Schaar M (2024) Causal machine learning for predicting treatment outcomes. Nature Medicine 30(4):958–968.Crossref, Google Scholar
  • Gelman A, Meng XL, Stern H (1996) Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica 6(4):733–760.Google Scholar
  • Hair JF Jr, Sarstedt M (2021) Data, measurement, and causal inferences in machine learning: Opportunities and challenges for marketing. J. Marketing Theory Practice 29(1):65–77.Crossref, Google Scholar
  • Henderson CR (1950) Estimation of genetic parameters (abstract). Ann. Math. Statist. 21(2):309–310.Google Scholar
  • Higgins JP, Thompson SG, Spiegelhalter DJ (2009) A re-evaluation of random-effects meta-analysis. J. Roy Statist. Soc. Ser. A Statist. Soc. 172(1):137–159.Crossref, Google Scholar
  • Huang S, Wang C, Yuan Y, Zhao J, Zhang BJ (2026) Estimating effects of long-term treatments. Management Sci., ePub ahead of print April 9, https://doi.org/10.1287/mnsc.2023.02575.Google Scholar
  • Imbens G, Kallus N, Mao X, Wang Y (2024) Long-term causal inference under persistent confounding via data combination. J. Roy Statist. Soc. Ser. B Statist. Methodology 87(2):362–388.Crossref, Google Scholar
  • Jeunen O, Ustimenko A (2024) Learning metrics that maximise power for accelerated A/B-tests. Baeza-Yates R, Bonchi F (General Chairs), eds. Proc. 30th ACM SIGKDD Conf. Knowledge Discovery Data Mining KDD 24 (Association for Computing Machinery, New York), 5183–5193.Google Scholar
  • Jin Y, Ba S (2023) Toward optimal variance reduction in online controlled experiments. Technometrics 65(2):231–242.Crossref, Google Scholar
  • Joffe MM, Greene T (2009) Related causal frameworks for surrogate outcomes. Biometrics 65(2):530–538.Crossref, Google Scholar
  • Kim JG, Menzefricke U, Feinberg FM (2004) Assessing heterogeneity in discrete choice models using a Dirichlet process prior. Rev. Marketing Sci. 2(1):1–41.Crossref, Google Scholar
  • Koenker R, Mizera I (2014) Convex optimization, shape constraints, compound decisions, and empirical Bayes rules. J. Amer. Statist. Assoc. 109(506):674–685.Crossref, Google Scholar
  • Kohavi R, Tang D, Xu Y (2020) Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, 1st ed. (Cambridge University Press, Cambridge, UK).Crossref, Google Scholar
  • Lewis RA, Rao JM (2015) The unfavorable economics of measuring the returns to advertising. Quart. J. Econom. 130(4):1941–1973.Crossref, Google Scholar
  • Liu JS, Wong WH, Kong A (1994) Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes. Biometrika 81(1):27–40.Crossref, Google Scholar
  • Meng XL (1994) Posterior predictive p-values. Ann. Statist. 22(3):1142–1160.Crossref, Google Scholar
  • Murtagh F, Contreras P (2012) Algorithms for hierarchical clustering: An overview. WIREs Data Mining Knowledge Discovery 2(1):86–97.Crossref, Google Scholar
  • Neal RM (2000) Markov chain sampling methods for Dirichlet process mixture models. J. Comput. Graph. Statist. 9(2):249–265.Crossref, Google Scholar
  • Neal RM (2011) MCMC using Hamiltonian dynamics. Brooks S, Gelman A, Jones GL, Meng XL, eds. Handbook of Markov Chain Monte Carlo (CRC Press, Boca Raton, FL), 113–162.Crossref, Google Scholar
  • Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans. Knowledge Data Engrg. 22(10):1345–1359.Crossref, Google Scholar
  • Rafieian O (2023) A matrix completion solution to the problem of ignoring the ignorability assumption. Preprint, submitted September 6, https://dx.doi.org/10.2139/ssrn.4546996.Google Scholar
  • Ray K, van der Vaart A (2021) On the Bernstein–von Mises theorem for the Dirichlet process. Electronic J. Statist. 15(1):2224–2246.Crossref, Google Scholar
  • Robinson GK (1991) That BLUP is a good thing: The estimation of random effects. Statist. Sci. 6(1):15–32.Crossref, Google Scholar
  • Rodriguez A, Müller P (2013) Nonparametric Bayesian inference. NSF–CBMS Regional Conf. Ser. Probab. Statist., vol. 9, i–110.Google Scholar
  • Rossi PE, Allenby GM, Misra S (2024) Bayesian Statistics and Marketing, Wiley Series in Probability and Statistics, 2nd ed. (John Wiley & Sons, Newark, NJ).Crossref, Google Scholar
  • Rousseau J (2016) On the frequentist properties of Bayesian nonparametric methods. Annual Rev. Statist. Appl. 3(1):211–231.Crossref, Google Scholar
  • Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J. Ed. Psych. 66(5):688–701.Crossref, Google Scholar
  • Rubin DB (1984) Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann. Statist. 12(4):1151–1172.Crossref, Google Scholar
  • Sethuraman J, Ghosh M (2024) Some properties of GEM(α) distributions. Sankhya A 86(Suppl 1):288–300.Crossref, Google Scholar
  • Spiegelhalter DJ, Best NG, Carlin BP, Van Der Linde A (2002) Bayesian measures of model complexity and fit. J. Roy Statist. Soc. Ser. B Statist. Methodology 64(4):583–639.Crossref, Google Scholar
  • Srinivasan S, Vanhuele M, Pauwels K (2010) Mind-set metrics in market response models: An integrative approach. J. Marketing Res. 47(4):672–684.Crossref, Google Scholar
  • Tanner MA, Wong WH (1987) The calculation of posterior distributions by data augmentation. J. Amer. Statist. Assoc. 82(398):528–540.Crossref, Google Scholar
  • Tripuraneni N, Richardson L, D’Amour A, Soriano J, Yadlowsky S (2024) Choosing a proxy metric from past experiments. Baeza-Yates R, Bonchi F (General Chairs), eds. Proc. 30th ACM SIGKDD Conf. Knowledge Discovery Data Mining KDD 24 (Association for Computing Machinery, New York), 5803–5812.Google Scholar
  • van Dyk DA, Meng X-L (2001) The art of data augmentation (with discussion). J. Comput. Graph. Statist. 10(1):1–50.Crossref, Google Scholar
  • van Houwelingen HC, Arends LR, Stijnen T (2002) Advanced methods in meta-analysis: Multivariate approach and meta-regression. Statist. Medicine 21(4):589–624.Crossref, Google Scholar
  • Watanabe S (2010) Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. J. Machine Learn. Res. 11(116):3571–3594.Google Scholar
  • Yang J, Eckles D, Dhillon P, Aral S (2024) Targeting for long-term outcomes. Management Sci. 70(6):3841–3855.Link, Google Scholar
INFORMS site uses cookies to store information on your computer. Some are essential to make our site work; Others help us improve the user experience. By using this site, you consent to the placement of these cookies. Please read our Privacy Statement to learn more.