Overall Evaluation Criteria for A/B Tests: A Meta-Analytic Approach

1. Introduction

A/B testing has become an essential tool in digital marketing, providing a systematic approach to assess interventions—web-page layouts, promotional inducements, and product recommendations among others—based on short-term observable outcomes, like click-through or conversion rates. These metrics offer a convenient, rapid way to compare treatments and enact decisions. However, reliance on such measures can misalign with broader downstream business objectives, like sustained revenue growth, customer retention, and brand loyalty, especially in industries with long consideration cycles, such as real estate. This incongruity highlights a methodological and practical gap; even unbiased, low-variance measures of short-term success may not correspond to those indicating sustained impact, so prioritizing them may lead firms astray.

In practice, product and marketing teams must act before downstream outcomes fully accrue, and short-run proxies seldom satisfy the conditions needed to stand in for long-run objectives. Practitioners typically track multiple upstream metrics whose signals can conflict—some improve, whereas others worsen—producing indeterminate patterns that complicate decision making. For example, when historically similar tests indicate that initial gains on a short-run metric (e.g., click-through) tend to coincide with downstream losses (e.g., six-month retention), treating the short-run uplift as a “win” can erode value months later. A related situation can arise with loyalty programs; sign-ups may surge initially, yet long-run engagement and spending can diverge from these early signals. In view of these challenges, industry professionals emphasize the importance of developing an overall evaluation criterion (OEC): a high-precision metric based on short-term indicators that produces accurate estimates of long-term business value (Kohavi et al. 2020, chapter 7). Commonly, the OEC is specified as a single metric or a weighted combination of metrics—often “by committee” rather than through analytical means—and consequently, may not produce reliable forecasts of long-term value.

To address this gap, we propose a meta-analytic framework that learns a mapping from short-term effect estimates to long-term business value. In our framework, analysts specify a target OEC as a function of the true (but unknown) long-term causal effects for a given test. Then, using estimates of both short-term and long-term causal effects from previous experiments, the framework learns the cross-experiment relationship between them and produces OEC estimates for the current test based on its available short-term effect estimates. Methodologically, this is accomplished by (i) specifying a shared prior distribution for the full causal-effect vector across a database of historical A/B tests and (ii) applying Bayesian meta-analytic methods to learn the parameters of this distribution, including the associations between short-term and long-term effects.

We show that this approach produces early, uncertainty-quantified inference for the OEC without imposing strong causal assumptions, such as the existence of a surrogate outcome that fully mediates effects of the treatment on the primary outcome. In effect, our approach replaces these causal assumptions with the statistical assumption of a shared experiment-level effect distribution. This assumption is especially credible in settings with many distinct and closely related experiments, such as experiments within a given platform, application, or website. However, it is less plausible when experiments overlap in time or derive from different domains or customer segments. When this assumption holds, we can learn the empirical strength of surrogacy relationships among causal effects in the form of cross-experiment correlations.

Because the approach is fully Bayesian and operates on experiment-level summaries (effect estimates and covariance matrices), it can integrate naturally with complementary measurement systems (e.g., multitouch attribution) without requiring customer-level data (Berman 2018, Yang et al. 2024). Unlike surrogate index approaches that rely on individual-level surrogacy and comparability assumptions, information transfer in our framework is governed by a shared distribution for experiment-level causal effects. Under our assumption of a shared effect distribution, we develop empirical surrogacy diagnostics (variance-ratio decompositions and cross-trial $R^2$) that quantify when short-term proxy metrics are informative for long-term metrics and conversely, signal when transfer should be limited. When historically similar tests indicate that initial gains on a short-run metric tend to coincide with downstream losses, our framework regularizes the current experiment’s estimates toward that empirically learned relationship. Conversely, when the diagnostics indicate weak or unstable associations, the framework tempers transfer, protecting against overconfident extrapolation.

In short, our primary contribution is to demonstrate how meta-analysis can be used to construct an OEC that links short-term proxies to long-term marketing outcomes using experiment-level summaries. Secondary contributions include empirical surrogacy diagnostics, an empirical assessment across a large archive of industry experiments, and open-source implementation materials to support reproducibility and extension. We illustrate the framework with three random-effects specifications—multivariate Gaussian (G), multivariate Student-t (T), and a Dirichlet process mixture (DPM)—to assess robustness to heavy tails and latent clustering. In simulation and in an application to 190 A/B tests from Zillow, the meta-analytic methods yield materially better inference than single-test benchmarks; the best sample-split method reduces log absolute prediction error by 0.29 points, corresponding to about 25% lower absolute prediction error on the original scale, and posterior standard deviations are about 60% smaller without compromising nominal coverage of prediction intervals. The framework also reduces overstatement when tests are reported or implemented after selecting variants with large positive estimated effects and produces actionable insights related to cross-experiment effect heterogeneity (e.g., posterior clustering under the DPM).

The remainder of the paper proceeds as follows. Section 2 reviews closely related work on long-term inference, surrogate modeling, and information transfer across experiments. Section 3 introduces the OEC meta-analytic framework, and Section 4 details the three model instantiations. Section 5 reports a simulation study comparing the methods on a range of accuracy measures (e.g., credible interval coverage). Section 6 applies the methods to 190 A/B tests from Zillow, including surrogacy diagnostics and an assessment of cross-experiment transfer. Section 7 concludes with a summary, managerial implications, and potential extensions of the framework.

2. Literature Review

The landscape of marketing experimentation has evolved substantially over the past decade, with growing reliance on A/B testing and related designs to guide product and marketing decisions. Despite these advances, existing approaches often fall short when the goal is to aggregate evidence across experiments to learn how short-run proxies map to long-run business outcomes. In what follows, we focus on literatures that bear directly on this problem: (i) research on short-term and long-term outcomes in marketing, (ii) statistical methods for meta-analysis and cross-experiment learning, and (iii) Bayesian nonparametric methods for robust inference in marketing applications. For brevity, we eschew a full review of broader topics in A/B practice, adaptive experimentation, and general Bayesian nonparametrics, except where they are directly pertinent to these aims.

2.1. Short-Term Metrics in Marketing

Many marketing decisions are evaluated on the basis of near-term quantities—clicks, sign-ups, and flow completions—that are timely and scalable but may be imperfect surrogates for long-run objectives, such as revenue, retention, and customer equity. This tension appears across common experimentation settings, where immediate responses can obscure downstream effects that accrue over months or longer. In advertising, for example, success is frequently gauged by short-run responses (impressions, clicks, and conversions), yet these signals need not reflect durable acquisition or customer lifetime value; evaluating return on investment, therefore, requires attention to longer-horizon outcomes beyond immediate response (Srinivasan et al. 2010, Lewis and Rao 2015). Likewise, in customer management, interventions that lift early engagement (e.g., loyalty program sign-ups) can diverge from long-run participation and spending patterns unless sustained incentives are in place (Ascarza 2018). These cases illustrate a general challenge; short-term proxies are informative but incomplete, and their link to long-run outcomes is ultimately empirical rather than guaranteed. Because such patterns recur in digital product and marketing contexts, methods that formally link short-run metrics to long-run objectives are particularly valuable.

2.2. Cross-Experiment Learning

The basic meta-analysis paradigm that we follow has been applied extensively in the analysis of clinical trials, enabling the use of “surrogate end points,” such as blood pressure, to stand in as indicators of long-term health outcomes, such as five-year mortality, thereby accelerating the decision-making process. In addition to the meta-analysis paradigm, this literature includes the causal effectiveness (CE) and causal association (CA) paradigms, entailing mediation analysis and principal stratification, respectively; both require restrictive causal assumptions for identification (Joffe and Greene 2009, Conlon et al. 2017). In contrast, meta-analytic methods do not require such assumptions, relying instead on the ability to learn effect distributions across historical experiments (Elliott 2023). Whereas clinical applications are often constrained by the scarcity of trials, A/B testing routinely yields extensive archives, rendering the meta-analytic paradigm a natural fit for OEC construction.

The standard approach for statistical meta-analysis is to specify a random-effects model with a shared fixed effect and random effects representing the true but unknown causal effects for each experiment (DerSimonian and Laird 2015). DerSimonian and Laird (1986) originally proposed this approach in a univariate setting, and it was subsequently extended to multivariate random-effects meta-analysis (MREMA), enabling its use in modeling surrogate end points (Buyse et al. 2000). To improve the robustness of the meta-analysis approach, researchers have proposed flexible models for the effect distribution, including the DPM approach that we adopt in Section 4.3 (Burr and Doss 2005). MREMA models can be used to target a wide variety of estimands, including trial-specific effects, average cross-trial parameters, and measures of causal surrogacy (Higgins et al. 2009). They have also been shown to avoid “surrogacy paradoxes,” in which other methods (such as the CA and CE frameworks) may produce incorrect inferences (Elliott et al. 2015). In summary, this literature suggests that MREMA methods provide a well-suited framework for long-term inference in A/B testing, especially when coupled with flexible, nonparametric modeling and surrogacy measures.

Recently, there has been increased interest in data combination methods and inference for long-term outcomes in other areas of science and business. One representative approach is that of “surrogate indices” in economics (Imbens et al. 2024, Athey et al. 2026). Yang et al. (2024) adapt this approach to learn targeting policies, enabling them to learn optimal individual-level treatment rules under surrogacy¹ and comparability assumptions. These approaches are best suited to policy-learning settings with individual-level data and identifying assumptions linking short-term and long-term potential outcomes across data sets. In contrast, our focus is experiment-level OEC estimation with calibrated uncertainty from summary data (effect estimates and within-experiment covariances), where information transfer is governed by exchangeability across experiments and learned correlations.

A different approach exploits cross-study structure using matrix-completion ideas to recover missing outcomes across studies (Rafieian 2023), and it is conceptually aligned with the broader transfer-learning perspective that leverages stable structure across related tasks to enable information transfer (Pan and Yang 2009). These methods are conceptually similar to our MREMA approach in that they borrow information across data sets, but they generally require individual-level data, introduce more restrictive assumptions, and/or target a different problem, such as policy learning.

2.3. Bayesian Nonparametrics and OEC Construction

Within marketing, Bayesian nonparametric models offer flexible, data-adaptive ways to capture heterogeneity and clustering without prespecifying segment structure. Representative applications include DPMs for random-coefficient heterogeneity in discrete choice (Kim et al. 2004), semiparametric Dirichlet process (DP)-based e-customization that accommodates complex multisource variation (Ansari and Mela 2003), and Gaussian processes for customer-base dynamics that yield interpretable and decision-oriented summaries (Dew and Ansari 2018). Collectively, these approaches suggest the power and generality of using flexible priors to discover latent structure and provide robustness in aggregation, principles especially relevant when pooling effects across experiments and summarizing multiple short-run signals into an OEC.

In summary, recent work has begun to address the business challenges of complex customer heterogeneity and long-term inference/targeting. Meanwhile, previous work in biostatistics has developed the MREMA framework for addressing closely related challenges in clinical trials, enabling robust inference for effects on long-term, noisy outcomes via short-term surrogate metrics. However, the MREMA approach has not yet been applied in A/B testing to (i) aggregate evidence across many experiments and metrics; (ii) deliver early, uncertainty-calibrated OEC forecasts; (iii) quantify surrogacy relationships among related business metrics; and (iv) explicitly account for test-level heterogeneity. The aim of this work is to improve A/B test decision-making practices by filling this gap.

3. Proposed Framework

This section outlines our proposed meta-analysis framework for OEC estimation.

3.1. Basic Setup

We assume access to $T\in \mathbb{N}$ historical A/B tests (experiments). To simplify the exposition, we assume that each test has two arms. The extension to three or more arms is conceptually straightforward; see Appendix A. In each test indexed by $t\in [T]≔\{1,\dots ,T\}$, we observe a vector, ${\mathbf{z}}_t\in {\mathbb{R}}^p$, of estimated treatment effects for p business metrics. The estimate ${\mathbf{z}}_t$ is formed from a sample of $n_t\in \mathbb{N}$ observations. Typically, each observation represents a single customer. Our methods are agnostic to the specific estimator defining ${\mathbf{z}}_t$ provided that the following assumption holds.

Assumption 1

(Asymptotic Normality). As $n_t\to \infty$, the limiting distribution of ${\mathbf{z}}_t$ is multivariate Gaussian with mean ${\mathit{\mu }}_t\in {\mathbb{R}}^p$ and positive definite covariance matrix ${\mathbf{\Sigma }}_t\in {\mathbb{R}}^{p\times p}$.

Assumption 1 ensures that the distribution of ${\mathbf{z}}_t$ is well approximated by a multivariate Gaussian distribution for large $n_t$. In practice, $n_t$ must be large enough that the multivariate Gaussian distribution can serve as an accurate approximate likelihood function for ${\mathit{\mu }}_t$ (van Houwelingen et al. 2002). As usual, precise estimators (those with small ${\mathbf{\Sigma }}_t$) are preferred, and efficiency gains in ${\mathbf{z}}_t$ directly translate into efficiency gains within the meta-analysis framework. In particular, analysts can often reduce variances by 50% or more by leveraging pretreatment covariates for individual customers (Deng et al. 2013, Jin and Ba 2023).

In practice, we often have access only to an estimator, ${\widehat{\mathbf{\Sigma }}}_t$, of ${\mathbf{\Sigma }}_t$ that satisfies the following assumption.

Assumption 2

(Consistency). The estimation error $n_t({\widehat{\mathbf{\Sigma }}}_t-{\mathbf{\Sigma }}_t)$ converges in probability to the zero matrix ${\mathbf{0}}_{p,p}$ as $n_t\to \infty$.

In our notation, the dependence on $n_t$ is already built into ${\widehat{\mathbf{\Sigma }}}_t$ and ${\mathbf{\Sigma }}_t$; thus, the $n_t$ in Assumption 2 serves only to place these matrices on a constant scale. By Slutsky’s theorem, the estimation error is asymptotically negligible, so we take the common meta-analytic approach of treating ${\mathbf{\Sigma }}_t$ as known (DerSimonian and Laird 1986). Consequently, in the remainder of the paper, we remove the hat, denoting ${\widehat{\mathbf{\Sigma }}}_t$ as ${\mathbf{\Sigma }}_t$.

In common practice, firms analyze one metric at a time, which requires only the diagonal elements of ${\mathbf{\Sigma }}_t$. Thus, our framework entails the additional practical requirements of (i) computing metric covariances and (ii) maintaining an archive of comparable experimental results. In particular, the latter requires shared metric definitions across experiments. Crucially, however, our framework requires only these summary statistics, ${({\mathbf{z}}_t,{\mathbf{\Sigma }}_t)}_{t\in [T]}$, removing the need for customer-level data access and processing. In our empirical study, this requirement reduces the size of the data set from billions to hundreds of rows.

We partition ${\mathbf{z}}_t$ into two sets of metrics: ${\mathbf{z}}_t={({\mathbf{y}}_t^{⊺},\hspace{0.17em}{\mathbf{x}}_t^{⊺})}^{⊺}$ with ${\mathbf{y}}_t\in {\mathbb{R}}^r$ and ${\mathbf{x}}_t\in {\mathbb{R}}^q$. The true latent effects, ${\mathit{\mu }}_t$, can then be partitioned in a corresponding fashion as ${\mathit{\mu }}_t={({\mathit{\mu }}_{\mathbf{y}t}^{⊺},\hspace{0.17em}{\mathit{\mu }}_{\mathbf{x}t}^{⊺})}^{⊺}$. The vectors ${\mathbf{y}}_t$ and ${\mathit{\mu }}_{\mathbf{y}t}$ represent key performance indicators (KPIs) for the business units affected by the experiments. The OEC is defined as some analyst-specified function $f:{\mathbb{R}}^r\to \mathbb{R}$ of the true effects for these KPIs: ${\text{OEC}}_t≔f({\mathit{\mu }}_{\mathbf{y}t})$. Analysts should specify ${\mathbf{y}}_t$ and f such that the OEC represents long-term business value in the most objective manner possible. Typically, the OEC will attempt to quantify customer-level profit or revenue subject to technical constraints and the requirement that ${\mathbf{y}}_t$ is observed for a large subset of previous experiments.

The remaining metrics comprising ${\mathbf{x}}_t$ are surrogates intended to improve our estimates of ${\mathit{\mu }}_{\mathbf{y}t}$ and often, are not of direct interest. The surrogates may be measured over a shorter time frame than the KPIs, but this is not a requirement. Both ${\mathbf{x}}_t$ and ${\mathbf{y}}_t$ represent estimated effects on post-treatment outcomes; neither is a covariate vector. The definitions of ${\mathbf{x}}_t$ and ${\mathbf{y}}_t$ should be standardized across all T experiments. Ideally, they should also include a specific time frame to improve comparability across experiments. For concreteness, we provide several example specifications below.

Example 1

(Total Revenue). We set $r=1$ and specify ${\mu }_{yt}$ as the average treatment effect (ATE) on revenue over a period of six months (i.e., ${\mu }_{yt}≔\mathbb{E}[m_t(1)-m_t(0)]$, where $m_t(a)$ is the potential outcome (Rubin 1974) of the business metric, m, under treatment $a\in \{0,1\}$ in the test of interest).² In this case, f is simply the identity function: $f({\mu }_{yt})={\mu }_{yt}$. We would then specify ${\mathbf{x}}_t$ as a vector of treatment-effect estimates that act as (imperfect) surrogates or proxies for ${\mu }_{yt}$, such as click-through rates, average order sizes, or measures of customer engagement.

Example 2

(Segmented Revenue). We allow $r>1$ and specify ${\mathit{\mu }}_{\mathbf{y}t}$ as a vector of treatment effects with each entry corresponding to an ATE for revenue within a specific product line. In this case, we specify f as $f({\mathit{\mu }}_{\mathbf{y}t})={\mathbf{1}}_r^{⊺}{\mathit{\mu }}_{\mathbf{y}t}$, where ${\mathbf{1}}_r$ is an r-dimensional vector of ones (i.e., we sum over product lines). We would again specify ${\mathbf{x}}_t$ as a vector of proxy treatment-effect estimates.

Example 3

(Conversion Funnel). We assume that the metrics comprise a funnel with r layers and specify each component of ${\mathit{\mu }}_{\mathbf{y}t}$ as a percentage change (lift) at a particular point in the funnel; the jth element of ${\mathit{\mu }}_{\mathbf{y}t}$ is

$${\mu }_{\mathit{\text{ytj}}}≔\frac{\mathbb{E}[m_{tj}(1)|m_{t,j-1}(1)=1]}{\mathbb{E}[m_{tj}(0)|m_{t,j-1}(0)=1]}-1,$$

where $m_{tj}(a)\in \{0,1\}$ is the potential outcome for conversion in the jth funnel layer and $m_{t0}(a)=1$ for $a\in \{0,1\}$. We would then specify ${\mathbf{x}}_t$ as a vector of proxies for these conversion rates and specify f as the bottom-of-funnel lift:

$$f({\mathit{\mu }}_{\mathbf{y}t})≔{\displaystyle \prod _{j=1}^r(1+{\mu }_{\mathit{\text{ytj}}})}-1.$$

Example 3 reveals that the framework is not restricted to ATEs. In practice, analysts may also transform ${\mathit{\mu }}_t$ and ${\mathbf{z}}_t$; after applying the multivariate delta method to obtain the corresponding ${\mathbf{\Sigma }}_t$, Assumptions 1 and 2 apply to these transformed quantities. This approach allows us to satisfy Assumptions 1 and 2 while maintaining the natural business scale of the OEC. In the case of Example 3, one such transformation is

The ${\text{OEC}}_t$ can then be expressed as $f({\mathit{\mu }}_{\mathbf{y}t})≔\text{exp}({\mathbf{1}}_r^{⊺}{\mathit{\mu }}_{\mathbf{y}t})-1$. Example 3 also provides an example in which the framework may provide benefit even when metrics are measured over the same time period. Although the bottom-of-funnel events provide the most direct business value, they are typically quite rare, resulting in low-precision estimates relative to the upper-funnel metrics. In this case, the meta-analysis framework enables us to define success in terms of bottom-of-funnel metrics but reap the statistical benefits of using sensitive top-of-funnel metrics.

3.2. The Naive Method

Given the setup of Section 3.1, the problem reduces to that of obtaining inferences for ${\text{OEC}}_t=f({\mathit{\mu }}_{\mathbf{y}t})$ given ${\mathbf{z}}_t$ and ${\mathbf{\Sigma }}_t$. As a starting point, we consider the following plug-in estimator:

We refer to ${\widehat{\text{OEC}}}_t$ as the naive (N) method because it ignores ${\mathbf{x}}_t$ and data from other experiments. Below, we let ${\mathbf{\Sigma }}_{\mathbf{y}\mathbf{y}t}$ be the submatrix of ${\mathbf{\Sigma }}_t$ corresponding to ${\mathbf{y}}_t$. Under the following assumption, ${\widehat{\text{OEC}}}_t$ has a simple asymptotic distribution.

Assumption 3

(Regularity). The function f has continuous partial derivatives at the point ${\mathit{\mu }}_{\mathbf{y}t}$ and $f^{\prime }{({\mathit{\mu }}_{\mathbf{y}t})}^{⊺}{\mathbf{\Sigma }}_{\mathbf{y}\mathbf{y}t}{f}^{\prime }({\mathit{\mu }}_{\mathbf{y}t})>0$.

Under Assumption 3, the multivariate delta method gives

where $\mathcal{N}$ denotes the Gaussian distribution and $f^{\prime }({\mathit{\mu }}_{\mathbf{y}t})$ denotes the gradient of f. In particular, when $f({\mathit{\mu }}_{\mathbf{y}t})={\mathbf{1}}_r^{⊺}{\mathit{\mu }}_{\mathbf{y}t}$ (Example 2), the asymptotic variance is ${\mathbf{1}}_r^{⊺}{\mathbf{\Sigma }}_{\mathbf{y}\mathbf{y}t}{\mathbf{1}}_r$, where ${\mathbf{1}}_r$ is an r-vector of ones. See Deng and Shi (2016) for a related usage of the multivariate delta method in online experimentation.

In Sections 5 and 6, we employ the naive method as a benchmark because it mimics the common industry practice of forming plug-in estimators using data from a single A/B test. Commonly, practitioners specify ${\mathbf{z}}_t$ as the difference-in-means estimator: ${\mathbf{z}}_t={\overline{\mathbf{m}}}_{t1}-{\overline{\mathbf{m}}}_{t0}$, where ${\overline{\mathbf{m}}}_{t1}$ and ${\overline{\mathbf{m}}}_{t0}$ denote the sample means of the metrics within the treatment and control groups, respectively. The covariance matrix, ${\mathbf{\Sigma }}_t$, can then be estimated as ${\mathbf{S}}_{t1}/n_{t1}+{\mathbf{S}}_{t0}/n_{t0}$, where ${\mathbf{S}}_{t1},\hspace{0.17em}{\mathbf{S}}_{t0}$ and $n_{t1},\hspace{0.17em}{n}_{t0}$ are the sample covariance matrices and sample sizes in the treatment and control groups, respectively.³

In our analysis, we employ these standard estimators in our specification of ${\mathbf{z}}_t$ and ${\mathbf{\Sigma }}_t$. We reiterate, however, that any estimator satisfying Assumptions 1 and 2 is permitted. If observation-level covariate information is available, practitioners may instead apply advanced causal inference methods, such as causal machine learning, to improve the statistical efficiency of ${\mathbf{z}}_t$ (Deng et al. 2013, Hair and Sarstedt 2021, Feuerriegel et al. 2024). Although these and other “supervised learning” methods can (and should) be used for generating ${\mathbf{z}}_t$ where possible, they focus only on within-experiment learning. In contrast, our proposed framework targets the largely orthogonal problem of between-experiment learning; given a set of estimates ${({\mathbf{z}}_t)}_{t\in [T]}$ and accompanying measurement-error covariances ${({\mathbf{\Sigma }}_t)}_{t\in [T]}$, we aim to improve these estimates by borrowing information across experiments. In other words, we aim to develop an estimator ${\widetilde{\text{OEC}}}_t$—dependent on ${({\mathbf{z}}_s,{\mathbf{\Sigma }}_s)}_{s\in [T]}$—that in some meaningful sense, outperforms ${\widehat{\text{OEC}}}_t$ as an estimator of ${\text{OEC}}_t$. The next section provides the basic framework for producing such an estimator.

3.3. The Bayesian Meta-Analysis Framework

To motivate our proposed framework, let $\mathcal{\ell }:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ be a loss function with $\mathcal{\ell }({\widetilde{\text{OEC}}}_t,{\text{OEC}}_t)$ denoting the realized loss for ${\widetilde{\text{OEC}}}_t$ as an estimate of ${\text{OEC}}_t$, such as the squared-error loss, $\mathcal{\ell }({\widetilde{\text{OEC}}}_t,{\text{OEC}}_t)={({\widetilde{\text{OEC}}}_t-{\text{OEC}}_t)}^2$. To define a suitably averaged loss, we introduce the following assumption.

Assumption 4

Let $\{P_{\mathit{\theta }}:\mathit{\theta }\in \mathbf{\Theta }\}$ be a family of distributions with parameter space $\mathbf{\Theta }$. For some $\mathit{\theta }\in \mathbf{\Theta }$, the true effects ${({\mathit{\mu }}_t)}_{t\in [T]}$ are independent and identically distributed (i.i.d.) according to $P_{\mathit{\theta }}$: ${\mathit{\mu }}_t\stackrel{\mathit{\text{iid}}}{\sim }{P}_{\mathit{\theta }}$.

Assumption 4 is a standard assumption in the meta-analysis literature (Elliott et al. 2015, Elliott 2023), and univariate versions of Assumption 4 have been employed in previous work on online experimentation (Deng 2015; Deng and Shi 2016; Azevedo et al. 2019, 2020; Tripuraneni et al. 2024). Assumption 4 requires that the effects for all tests arise from a common distribution, $P_{\mathit{\theta }}$, given ${({\mathbf{\Sigma }}_t)}_{t\in [T]}$ and that these effects are independent, ruling out (for example) cross-test interference. In practice, the latter can be enforced through thoughtful experimental design, such as independently randomizing arm assignments and sequencing A/B tests that may interfere with each other.

Assumption 4 is the key assumption within our meta-analytic framework. Relative to the CA, CE, and surrogate index approaches, the key requirement is different; instead of imposing additional causal surrogacy assumptions linking short-term and long-term potential outcomes, we assume that the vector of experiment-level causal effects arises from a shared distribution across tests. This assumption will be violated when the assumed model family $P_{\mathit{\theta }}$ is not large enough to contain the true effect distribution, but subtler violations are possible even with a nonparametric model class, such as violations of the requirement that the experiments arise from the same distribution. As such, Assumption 4 is most appropriate in settings with homogeneous experiments, arising from a single domain and population. Assumption 4 represents somewhat of an idealized setting; however, we show below that it leads to estimators with an approximate optimality property.

Assumption 4 imposes observable consequences, so we may attempt to falsify it using posterior predictive model checks (Rubin 1984, Meng 1994, Gelman et al. 1996). In particular, we recommend checking for dependence between ${\mathit{\mu }}_t$ and known experiment-level metadata, including ${\mathbf{\Sigma }}_t$, as this dependence would signal departures from the assumption of identical distributions. If analysts do find significant dependence, we recommend adopting the metaregression (MR) approach discussed in Section 3.4. This approach allows us to weaken Assumption 4, requiring only conditional independence given ${\mathbf{\Sigma }}_t$ and potentially other experiment-level metadata.

Because ${\mathit{\mu }}_t$ is vector valued, $P_{\mathit{\theta }}$ encodes information regarding the metrics’ location, scale, and dependence structure. Under departures from independence, the latter enables the metrics to act as surrogates for one another. Moreover, Assumption 4 allows us to define a notion of optimality in a setting that acknowledges the randomness of ${({\mathit{\mu }}_t)}_{t\in [T]}$ as opposed to treating all ${({\mathit{\mu }}_t)}_{t\in [T]}$ as unrelated. Under Assumption 4, the Bayes risk of the estimator ${\widetilde{\text{OEC}}}_t$ is defined as ${\mathbb{E}}_{\mathit{\theta }}\hspace{0.17em}\mathcal{\ell }({\widetilde{\text{OEC}}}_t,{\text{OEC}}_t)$ with the expectation taken over both ${\mathbf{z}}_t$ and ${\mathit{\mu }}_t$, the latter being with respect to $P_{\mathit{\theta }}$. Under weak regularity conditions (see Brown and Purves 1973 and Berger 1985, section 4.4.1), a minimizer of the Bayes risk can be computed by minimizing the posterior risk given ${\mathbf{z}}_t$:

In particular, under squared-error loss, it is well known that the minimizer of (3) is the posterior mean of $f({\mathit{\mu }}_{\mathbf{y}t})$. Equation (3) omits ${\mathbf{\Sigma }}_t$ from the conditioning set because we treat it as a fixed, nonrandom quantity throughout.

In practice, ${\widetilde{\text{OEC}}}_{\mathit{\theta }t}$ is not a feasible estimator because $\mathit{\theta }$ is unknown, but it motivates estimators that match its logic of minimizing posterior risk given an estimate of $\mathit{\theta }$. We adopt a fully Bayesian approach, placing a hyperprior⁴ on $\mathit{\theta }$ with hyperparameter ${\mathit{\theta }}_0$. As with $\mathit{\theta }$, we do not impose any specific parametric structure or dimension on ${\mathit{\theta }}_0$ at this stage. For test t, our analysis involves summarizing the posterior distribution with density $p\{{\text{OEC}}_t|{({\mathbf{z}}_s)}_{s\in [T]}\}$ proportional to

where $p(·|·)$ denotes the conditional density of its arguments, $\delta (·)$ is the Dirac delta function, and $\mathit{\zeta }≔\{\mathit{\theta },{({\mathit{\mu }}_s)}_{s\in [T]}\}$. The integral in (4) marginalizes over $\mathit{\theta }$ and the components of ${({\mathit{\mu }}_s)}_{s\in [T]}$ not equal to ${\mathit{\mu }}_{\mathbf{y}t}$. As above, we omit ${({\mathbf{\Sigma }}_s)}_{s\in [T]}$ from the conditioning sets because we treat it as a fixed quantity.

In practice, the integral in (4) can be approximated using posterior draws from a Markov chain Monte Carlo (MCMC) algorithm. Concretely, this is accomplished by applying f to posterior draws of ${\mathit{\mu }}_{\mathbf{y}t}$. Given a particular loss function, we can produce estimators minimizing the posterior loss in an analogous fashion to (3), replacing the unknown posterior with (4). Although the resulting estimators are not Bayes-risk optimal in finite samples, they are optimal under an enlarged model that includes ${\mathit{\theta }}_0$ (i.e., they are optimal if $\mathit{\theta }$ is sampled according to the hyperprior density, $p(\mathit{\theta }\mid {\mathit{\theta }}_0)$). Under misspecification of the hyperprior, Bayesian methods—including nonparametric models, like the DPM—can still achieve posterior consistency, meaning that the posterior distribution concentrates around the correct value of $\mathit{\theta }$ (Rousseau 2016, Ray and van der Vaart 2021). Consequently, we expect our proposed meta-analysis methods to closely mimic the optimal estimator, ${\widetilde{\text{OEC}}}_{\mathit{\theta }t}$, in large samples provided that the relevant regularity conditions are satisfied.

Figure 1 summarizes the generic meta-analysis model. Panel (a) of Figure 1 visualizes the model in the form of a directed acyclic graph (DAG). Panel (b) of Figure 1 provides an example of how Bayes’ rule adjusts the naive point estimate, ${\mathbf{z}}_t$, shrinking it toward a region of high prior probability. In this example in panel (b) of Figure 1, the sampling-based (blue) and posterior (red) confidence/credible regions have identical coverage levels, and both are centered around an unbiased estimate; however, the posterior credible region is much smaller, and the accompanying point estimate is Bayes-risk optimal under squared-error loss. This improvement is possible because the likelihood (blue) region in panel (b) of Figure 1 is highly unlikely a priori, so on average, the Bayes estimator will perform much better by adjusting the estimate and accompanying credible region toward the region of high prior probability. Also, notice that ${\mu }_{xt}$ and ${\mu }_{yt}$ exhibit strong positive correlation. Consequently, the model’s relatively high confidence that ${\mu }_{xt}$ is large (given the estimate $x_t$ and the sampling variability in panel (b) of Figure 1) causes the model to shift its less certain estimate of ${\mu }_{yt}$ substantially upward.

3.4. Extension to Metaregression

Given experiment-level metadata—target platform, intervention type, test duration, etc.—Assumption 4 can be relaxed to allow the distribution of ${\mathit{\mu }}_t$ to vary by test, resulting in a metaregression model (van Houwelingen et al. 2002). Mathematically, this information can be encoded in a covariate vector ${\mathbf{w}}_t\in {\mathbb{R}}^d$ for some $d\in \mathbb{N}$. Assumption 4 can then be relaxed to require only conditional independence across tests given ${\mathbf{w}}_t$ and ${\mathbf{\Sigma }}_t$; in the assumption below, $\mathbf{w}$ represents a generic value of ${\mathbf{w}}_t$ and $\stackrel{\mathit{\text{ind}}}{\sim }$ means “independently distributed as.”

Assumption 5

(Conditional Independence). Let $\{P_{\mathit{\theta },\mathbf{w}}:\mathit{\theta }\in \mathbf{\Theta },\hspace{0.17em}\mathbf{w}\in {\mathbb{R}}^d\}$ be a family of conditional distributions given $\mathbf{w}$ with parameter space $\mathbf{\Theta }$. Then, ${\mathit{\mu }}_t|{({\mathbf{w}}_s)}_{s\in [T]}\stackrel{\mathit{\text{ind}}}{\sim }{P}_{\mathit{\theta },{\mathbf{w}}_t}$ for some $\mathit{\theta }\in \mathbf{\Theta }$.

The vector ${\mathbf{w}}_t$ can also include summaries of ${\mathbf{\Sigma }}_t$; this may be required if especially high- or low-performing tests are stopped quickly, resulting in large variances. Analysts may also include time variables in ${\mathbf{w}}_t$ to capture changes in the effect distribution over time. In general, we recommend using model checking to determine whether a metaregression model is needed and if so, which variables to include in ${\mathbf{w}}_t$.

The data in our empirical study do not contain rich experiment-level metadata, and we find limited evidence of dependence between ${\mathit{\mu }}_t$ and ${\mathbf{\Sigma }}_{\mathit{t}}$. Consequently, with the exception of the simulation study in Section 5, the remainder of the paper focuses on meta-analytic models under Assumption 4. The next section presents three representative models in this context.⁵

4. Meta-Analysis Models

Having outlined the basic meta-analysis framework, we now provide three specific meta-analysis models that we compare in Sections 5 and 6.

4.1. Multivariate Gaussian Model

We first consider a multivariate Gaussian model for $P_{\mathit{\theta }}$: ${\mathit{\mu }}_t|\mathit{\theta }\stackrel{\mathit{\text{iid}}}{\sim }\mathcal{N}(\mathit{\eta },\mathbf{\Omega })$, where $\mathit{\eta }\in {\mathbb{R}}^p$ and $\mathbf{\Omega }\in {\mathbb{R}}^{p\times p}$ with $\mathit{\theta }=(\mathit{\eta },\hspace{0.17em}\mathbf{\Omega })$. Given $\mathit{\eta }$ and $\mathbf{\Omega }$ (or posterior samples thereof), it is well known that the posterior distribution of ${\mathit{\mu }}_t$ is available in closed form:

where the ${\{{\mathit{\mu }}_t\}}_{t\in [T]}$ are independent across t conditional on $(\mathit{\theta },{\{{\mathbf{z}}_s,{\mathbf{\Sigma }}_s\}}_{s\in [T]})$. For additional details, see Rossi et al. (2024, section 2.9.5). In the context of A/B testing, Deng et al. (2023) briefly discuss a special case of this model and show that the addition of the prior leads to a statistical efficiency gain over the naive method. In fact, the posterior mean ${\mathit{\eta }}_t$ is the best linear unbiased predictor (BLUP) of ${\mathit{\mu }}_t$ given ${\mathbf{z}}_t$, ${\mathbf{\Sigma }}_t$, and $\mathit{\theta }$ (Henderson 1950, Robinson 1991). Recent OEC approaches proposed by Google and ShareChat also propose estimators that are linear in ${\mathbf{z}}_t$ but are not equal to BLUPs (Jeunen and Ustimenko 2024, Tripuraneni et al. 2024). Under the Gaussian random-effects model, the posterior mean is the BLUP and therefore, the optimal linear predictor of ${\mathit{\mu }}_t$, providing a natural benchmark for other linear OEC estimators.

A particularly desirable feature of the BLUPs is that they represent a linear combination of $\mathit{\eta }$ and ${\mathbf{z}}_t$, which can aid in interpretation. Further, in a standard asymptotic regime with $\sqrt{n_t}{\mathbf{\Sigma }}_t\stackrel{p}{\to }{\mathbf{0}}_{p,p}$, we have $n_t({\mathbf{\Omega }}_t-{\mathbf{\Sigma }}_t)\stackrel{p}{\to }{\mathbf{0}}_{p,p}$ and $\sqrt{n_t}({\mathit{\eta }}_t-{\mathbf{z}}_t)\stackrel{p}{\to }{\mathbf{0}}_p$ as $n_t\to \infty$. Thus, the BLUP ${\mathit{\eta }}_t$ is asymptotically equivalent to the naive estimator, ${\mathbf{z}}_t$, and produces the same asymptotic inferences. In finite samples, however, the BLUP can substantially outperform the naive estimator, especially when $P_{\mathit{\theta }}$ is highly concentrated—a common scenario in online experimentation.

BLUPs generalize naturally to the setting in which ${\mathbf{z}}_t$ is only partially observed provided that the missingness is not systematic.⁶ In this case, the observed effect estimates can be written in the form $\mathbf{A}{\mathbf{z}}_t$ for some matrix $\mathbf{A}$; for example, if the first $b\in \mathbb{N}$ elements of ${\mathbf{z}}_t$ are not observed, we set $\mathbf{A}=[{\mathbf{0}}_{p-b,b}\hspace{0.17em}{\mathbf{I}}_{p-b}]$, where ${\mathbf{I}}_{p-b}$ is an identity matrix of dimension $p-b$. The analysis is then modified by replacing the term $p({\mathbf{z}}_s\mid {\mathit{\mu }}_s)$ in (4) with $p(\mathbf{A}{\mathbf{z}}_s\mid {\mathit{\mu }}_s)$ and modifying the estimation algorithm accordingly. For the Gaussian model, this modification results in the following posterior parameters in place of those given in (5):

If ${\mathbf{y}}_t$ is missing, for example, the term $\mathbf{A}{\mathbf{\Sigma }}_t{\mathbf{A}}^{⊺}$ is equal to ${\mathbf{\Sigma }}_{\mathbf{x}\mathbf{x}t}$: the (invertible) submatrix of ${\mathbf{\Sigma }}_t$ corresponding to ${\mathbf{x}}_t$. The implication for the OEC meta-analysis framework is that we can obtain inferences for unobserved metrics—including long-term metrics before their time frame has elapsed—provided that we have access to short-term proxies. Notably, this approach does not require restrictive causal assumptions beyond what is needed for standard effect estimation. Rather, it requires that ${\mathit{\mu }}_{\mathbf{x}t}$ and ${\mathit{\mu }}_{\mathbf{y}t}$ are correlated; these correlations are encoded in $\mathbf{\Omega }$, and stronger correlations result in more precise inference for ${\mathit{\mu }}_{\mathbf{y}t}$. The Student-t and DPM models can be modified in a similar fashion to allow for missing data.

In practice, $\mathit{\theta }$ is unknown and must be estimated, so posterior summaries for ${\mathit{\mu }}_t$ and ${\text{OEC}}_t$ effectively average over the BLUPs defined in (5) according to the posterior density associated with different values of $\mathit{\theta }$. Depending on the choice of computational approach, this posterior averaging may occur implicitly. All three models that we consider admit efficient “collapsed” Gibbs samplers (Liu et al. 1994), so we employ this computational approach throughout, although other techniques, such as Hamiltonian Monte Carlo (HMC), could also be applied (Brooks et al. 2011, Neal 2011). The Gibbs sampling methods that we employ are well known (Neal 2000); however, to assist practitioners in applying our methods, we include them in Appendix B with specific suggestions on how to specify the hyperprior. In general, we recommend specifying dispersed, weakly informative prior distributions where possible, allowing the data set to “speak for itself” in terms of specifying plausible values for $\mathit{\theta }$. For small T, this specification has the benefit of producing estimates that closely align with those of the naive method; as T increases, the estimates improve as the model develops a better estimate of $\mathit{\theta }$.

Buyse et al. (2000) were the first to propose a (bivariate) version of this model in the surrogate end-points literature. Buyse et al. (2000) note that the strength of a surrogate can be assessed by the coefficient of determination from a regression of ${\mu }_{yt}$ on ${\mu }_{xt}$: ${\mathit{\text{R}}}^2≔{\mathrm{\Omega }}_{yx}^2/({\mathrm{\Omega }}_{xx}{\mathrm{\Omega }}_{yy})$, where the indices denote elements of $\mathbf{\Omega }$ corresponding to either ${\mu }_{xt}$ or ${\mu }_{yt}$. In the case of vector-valued ${\mathit{\mu }}_{\mathbf{x}t}$, this measure generalizes naturally to ${\mathit{\text{R}}}^2≔{\mathbf{\Omega }}_{y\mathbf{x}}{\mathbf{\Omega }}_{\mathbf{x}\mathbf{x}}^{-1}{\mathbf{\Omega }}_{\mathbf{x}y}/{\mathrm{\Omega }}_{yy}$. When ${\mathit{\mu }}_{\mathbf{y}t}$ is also vector valued and we employ the summed OEC of Example 2, we can further generalize this surrogacy measure as

which assesses the proportion of variance in ${\text{OEC}}_t$ explained by ${\mathit{\mu }}_{\mathbf{x}t}$.

For test t, we define the total variance ratio (${\text{TVR}}_t$) as the ratio of the meta-analysis posterior variance to the naive variance: ${\text{TVR}}_t≔v_{\mathbf{z}t}/v_{0t}$, where $v_{\mathbf{z}t}≔{\mathbf{u}}^{⊺}{\mathbf{\Omega }}_t\mathbf{u}$ and $v_{0t}≔{\mathbf{u}}^{⊺}{\mathbf{\Sigma }}_t\mathbf{u}$ with $\mathbf{u}={({\mathbf{1}}_r^{⊺},{\mathbf{0}}_q^{⊺})}^{⊺}$. This ratio can be interpreted in terms of cost savings as follows; achieving a similar posterior variance (or standard error) using the naive method would require increasing the sample size by a factor of $1/{\text{TVR}}_t$. To further understand the source of the variance reduction, we define the intermediate quantity $v_{\mathbf{y}t}≔{\mathbf{1}}_r^{⊺}{({\mathbf{\Omega }}_{\mathbf{y}\mathbf{y}}^{-1}+{\mathbf{\Sigma }}_{\mathbf{y}\mathbf{y}t}^{-1}\hspace{0.17em})}^{-1}{\mathbf{1}}_r$: the posterior variance from a hypothetical meta-analysis using only ${\mathbf{y}}_t$ (discarding ${\mathbf{x}}_t$). We can then define the prior variance ratio (PVR), ${\text{PVR}}_t≔v_{\mathbf{y}t}/v_{0t}$, and surrogacy variance ratio (SVR), ${\text{SVR}}_t≔v_{\mathbf{z}t}/v_{\mathbf{y}t}$, representing the multiplicative effects on the posterior variance because of (i) the prior distribution of ${\mathit{\mu }}_{\mathbf{y}t}$ and (ii) surrogacy of ${\mathit{\mu }}_{\mathbf{x}t}$ for ${\mathit{\mu }}_{\mathbf{y}t}$, respectively. Because the elements of ${\mathit{\mu }}_{\mathbf{y}t}$ may, in general, correlate with one another, ${\text{PVR}}_t$ includes reduction in the variance attributable to internal surrogacy: surrogacy of the elements of ${\mathit{\mu }}_{\mathbf{y}t}$ for one another.⁷

Having defined these quantities, we can then decompose ${\text{TVR}}_t$ as ${\text{TVR}}_t={\text{PVR}}_t·{\text{SVR}}_t$ to assess the contribution of each component to the efficiency gain. These metrics can also be computed for the individual components of ${\mathit{\mu }}_{\mathbf{y}t}$, providing measures of surrogacy for one outcome given all of the others—a strategy that we employ in our analysis in Section 6. In practice, we recommend using these surrogacy metrics to inform which metrics to include in ${\mathbf{x}}_t$ so as to maximize the strength of surrogacy subject to model complexity constraints.

4.2. Multivariate Student-t Model

Previous work has noted that large outliers are empirically much more common in A/B testing than the Gaussian assumption of Section 4.1 would allow (Azevedo et al. 2020). Perhaps the simplest extension of Section 4.1 to capture this phenomenon is to allow $P_{\mathit{\theta }}$ to take the form of a multivariate Student-t distribution:

where $\mathit{\eta }$ and $\mathbf{\Omega }$ maintain their interpretation as location and scale parameters, $\gamma >0$ is the degrees of freedom parameter, and $\mathit{\theta }=(\mathit{\eta },\mathbf{\Omega },\gamma )$. The expression for ${\mathit{\text{R}}}^2$ in (7) remains valid provided that $\gamma >2$ (Cornish 1954, Ding 2016). Because this model is not fully conjugate, closed-form analogs of TVR, PVR, and SVR are not available, and the posterior mean does not possess a simple linear form as in (5b).

4.3. DPM Model

The Gaussian and Student-t models serve as a useful starting point, especially when A/B tests are believed to be fairly homogeneous. This section introduces the DPM method, a flexible nonparametric model that avoids placing restrictive parametric assumptions on $P_{\mathit{\theta }}$. The DPM method assumes that ${({\mathit{\mu }}_t)}_{t\in [T]}$ are drawn from a mixture distribution, $P_{\mathit{\theta }}$, where $P_{\mathit{\theta }}$ itself is a random draw from a Dirichlet process with base distribution $G_0$ and concentration parameter ${\alpha }_0>0$. Formally, the DPM model can be represented as

where $c_t$ denotes the cluster assignment for experiment t; $\mathit{\pi }$ is the vector of cluster probabilities; and GEM denotes the Griffiths, Engen, and McCloskey distribution (Sethuraman and Ghosh 2024). In our assumed model, $G_0$ is a Gaussian-inverse-Wishart distribution parameterized by $\mathit{\xi }=(\mathbf{\Psi },\nu ,{\mathit{\eta }}_0,\varphi )$. Figure 2 summarizes our assumed DPM model in the form of a DAG. The DP prior on $P_{\mathit{\theta }}$ informs a clustering structure among the experiments, whereas the concentration parameter, ${\alpha }_0$, dictates the tendency to form new clusters; larger values of ${\alpha }_0$ encourage the formation of more clusters.

The surrogacy measures ${\mathit{\text{R}}}^2$, TVR, PVR, and SVR generalize in a straightforward manner to the DPM model by replacing $\mathbf{\Omega }$ with the cluster-specific covariance matrices, ${\{{\mathbf{\Omega }}_k\}}_{k\in [K]}$, where $K$ is the estimated number of clusters. These measures can be used in a cluster-specific or test-specific manner or aggregated across tests and clusters to obtain integrated measures of surrogacy.

5. Simulation Study

We performed a simulation study to compare the performance of the methods developed in Section 4. The simulations comprise independent replicates with known ground-truth ${\text{OEC}}_t$ values, enabling a richer comparison than is possible in our empirical study in Section 6. In particular, the synthetic nature of the data allows us to compute a larger set of performance metrics and apply a metaregression model based on a synthetic cluster membership variable.

5.1. Simulation Setup

To enforce fidelity between the simulation study and data analysis, we draw 100 independent data sets from the DPM model’s posterior predictive distribution after fitting it to the observed data set as described in Section 6.1. However, we apply transformations, including translations, rescalings, and rotations, to ensure that the simulated data set does not reveal any material nonpublic information regarding Zillow’s business metrics or experimentation processes. We have made these simulated data sets available on the Open Science Framework to allow researchers to reproduce and build on our results. The simulation analyses use the same methods, prior distributions, and computational algorithms as the empirical analysis described in Section 6.

In addition to testing the three meta-analysis approaches introduced in Section 4, we also implement the naive plug-in method described in (1), which mimics the common industry practice of analyzing A/B tests individually with no shared prior distribution. All methods use the same values of ${\mathbf{z}}_t$ and ${\mathbf{\Sigma }}_t$ as input. We also include two additional variations of the methods presented in Section 4: Gaussian-0 (G-0) and metaregression. The Gaussian-0 model fixes $\mathit{\eta }=\mathbf{0}$, allowing us to quantify the benefit of estimating the prior metric mean compared with fixing it to zero. MR is a metaregression variant of the DPM model in which the cluster labels are known. From a practical standpoint, this represents a setting in which experiment type is known a priori—for example, mobile versus browser tests—so that the effect distribution may differ across known but related groups. For brevity, the method names are sometimes abbreviated N, G-0, G, T, DPM, and MR.

5.2. Simulation Results

We first compare the methods on six performance metrics.

1. Correct decision rate. The percentage of tests that correctly identify the test arm (A or B) that will maximize ${\text{OEC}}_t$: B if ${\text{OEC}}_t>0$ and else, A.

2. Regret. The difference in ${\text{OEC}}_t$ between the selected arm and the optimal arm.

3. Overestimation. The average estimation error (predicted − true) for ${\text{OEC}}_t$ among selected arms. This metric quantifies the overstatement that arises when variants are selected for implementation or reporting because their estimated OEC is positive.

4. Mean absolute error (MAE). The average (across tests) absolute error in the estimation of ${\text{OEC}}_t$.

5. Average 95% confidence interval (95% CI) length. The average (across tests) length of 95% credible (confidence) intervals for ${\text{OEC}}_t$.

6. 95% CI coverage. The coverage rate for the above 95% credible (confidence) intervals.

In the results reported below, we first compute these metrics within each replication, and then, we compute averages and standard errors across replications. Figure 3 displays the average values of the six metrics introduced above with 95% confidence intervals. To enhance the precision of the comparison, we also report the underlying metric values in Table 1.

Table 1. Simulation Results

In the upper left panel of Figure 3, we see that the meta-analysis methods correctly identify the winning variant more frequently than the naive method, resulting in lower regret in the upper center panel of Figure 3. In both comparisons, the Gaussian and Student-t methods perform similarly, correctly identifying the winning arm in 87% of tests compared with 81% for the naive method, 84% for the Gaussian-0 method, 91% for the DPM method, and 93% for the metaregression method. Thus, compared with the naive method, the DPM and metaregression methods reduce the number of incorrect decisions by more than 50%. The upper right panel of Figure 3 shows that the naive method substantially overestimates the effects of implemented variants, where implementation is based on positive estimated effects, with an average overestimation of 0.48 compared with negligible values in the range of [−0.01 to 0.02] for the meta-analysis methods.

The lower left panel of Figure 3 shows that the MAE follows a pattern similar to that of the above metrics; the metric improves (decreases) from left to right, with the DPM and MR methods performing best. The remaining lower center and lower right panels of Figure 3 compare the 95% credible (confidence) intervals. The meta-analysis methods produce intervals less than one fifth the length of the naive intervals (on average), yet they also cover at a higher rate. As expected, the naive method has near-nominal coverage of 95.2%. In contrast, the meta-analysis methods exhibit mild overcoverage—a consequence of employing misspecified (hyper-)prior distributions.⁸ The DPM and MR methods produce the narrowest intervals, presumably because their assumed prior distribution matches the data-generating process, enabling these methods to accurately adjust credible regions based on cluster membership.

Table 2 displays pair-wise comparisons across five⁹ of the six metrics listed above and two additional metrics: the Win Rate and the Mean log Error Quotient (MLEQ). The Win Rate is defined as the percentage of A/B tests in which the method listed in the row outperformed the method listed in the column in terms of the magnitude of estimation error. The MLEQ is defined as $\text{log}(|a-c|/|b-c|)$, where a is the row prediction, b is the column prediction, and c is the ground-truth value of ${\text{OEC}}_t$; more negative $\mathrm{\Delta }$MLEQ indicates that the row method has smaller error. The meta-analysis methods outperform the naive method across both the Win Rate and MLEQ metrics, and the performance ordering is MR, DPM, T, Gaussian, Gaussian-0, and naive from best to worst. The MR and DPM methods outperform each of the other four methods at least 67% of the time (in separate comparisons). The performance order is consistent across the remaining performance metrics (excluding the 95% CI coverage, which is at or above the nominal 95% level for all methods). In total, 98 of 105 pair-wise comparisons are statistically significant, with all 7 nonsignificant comparisons involving the Gaussian-0, Gaussian, and T methods.

Table 2. Simulation Pair-Wise Method Comparisons

In summary, the meta-analysis methods substantially outperform the naive method. The Gaussian and T methods perform comparably with each other and slightly better than the Gaussian-0 method. The DPM and MR methods substantially outperform the other methods. Of these two methods, the MR method performs best, presumably because it has access to the cluster membership variables, which the DPM method must estimate.

6. Application to Real Estate Technology A/B Tests

In this section, we present results from applying the Bayesian meta-analysis methods to 190 A/B tests from Zillow. These tests represent all A/B tests during a subset of 2022–2024 that meet certain data quality standards, such as involving a sufficiently large number of users and not exhibiting unexpected imbalances in the assignment proportions. The A/B tests involve a variety of interventions ranging from new or altered email campaigns to website design changes, new product features, and updated personalization models. For each A/B test, we use the largest available sample (i.e., the most mature cohort). In most cases, these samples correspond with the data that were used to make implementation decisions.

6.1. Analysis Setup

Zillow receives revenue through various interrelated lines of business: new construction, rentals, sales leads for real estate agents, mortgages, products for sellers (e.g., promoted homes), and more. We might reasonably expect some lines of business to operate synergistically and others to result in cannibalization. To maintain confidentiality, we restrict attention to only two (undisclosed) lines of business, specifying ${\mathbf{y}}_t$ as estimates of the (rescaled) financial impact of the tested interventions on each individual line of business in terms of dollars per customer. If, for example, the two lines of business were new construction and rentals, ${\mathbf{y}}_t$ would consist of two components, $y_{t,nc}$ and $y_{t,r}$, representing the estimated per-customer effect of experiment t on revenue for new construction and rentals, respectively. Consequently, the natural definition for ${\text{OEC}}_t$ in this case is ${\text{OEC}}_t={\mathbf{1}}_2^{⊺}{\mathit{\mu }}_{\mathbf{y}t}$—the sum over the two lines of business as in Example 2.

The vector ${\mathbf{x}}_t$ consists of three short-term proxies for ${\mathbf{y}}_t$. Although we cannot disclose the specific definitions of these proxies, typical proxy metrics in this industry include measures of customer engagement, form-fill rates, and shorter-horizon measurements of ${\mathbf{y}}_t$. The primary purpose of including these proxies is to increase the model’s posterior precision by including causal effect estimates that are predictive of ${\text{OEC}}_t$ as quantified by the surrogacy measures given in Section 4.1.

We implemented all meta-analysis methods in Python using the collapsed Gibbs samplers described in Appendix B. For the priors, we set ${\mathit{\eta }}_0={\mathbf{0}}_p$, $\mathbf{\Phi }={\mathbf{I}}_p$, $\nu =8$, and $\mathbf{\Psi }=8{\mathbf{I}}_p$ for the Gaussian and T methods. For the T method, we assumed the prior $\text{log}(\gamma )\sim \mathcal{N}(2,1)$ with a prior median of $e^2\approx 7.4$ degrees of freedom. For the DPM, we set hyperpriors as follows: ${\mathit{\eta }}_0\sim \mathcal{N}({\mathbf{0}}_p,{\mathbf{I}}_p)$, $\mathbf{\Psi }\sim {\mathcal{W}}^{-1}(8{\mathbf{I}}_p,8)$, $\varphi \sim \mathcal{G}(1,1)$, ${\alpha }_0\sim \mathcal{G}(4,4)$, and $\nu =10$. Appendix C details a sensitivity analysis showing that the results in Section 6.2 are quite robust to specification of the prior distribution. We drew 30,000 samples for each method, with the first 10,000 samples discarded as warm-up iterations. The computation times were 1.35, 2.45, 4.73, and 142.53 minutes for the Gaussian-0, Gaussian, T, and DPM methods, respectively, on a cloud server with eight central processing units (CPUs) and 32 GB of memory. Appendix B.4 provides additional details on the computational scalability of our methods.

6.2. Sample-Split Predictive Analysis

To evaluate the methods’ performance, we implement a sample-splitting approach, in which each A/B test is randomly divided at the customer level into two equal-sized subsets: a training set and a test set. We fit the methods on the training set, obtaining posterior-based estimates of ${\text{OEC}}_t$. We then compare these estimates with independent estimates generated from the test set. We generate these estimates using ${\widehat{\text{OEC}}}_t$, the naive estimator, because the naive method is asymptotically unbiased under only Assumptions 1 and 3; in particular, it does not require a correctly specified parametric model or even i.i.d. effects (Assumption 4). Consequently, it serves as an independent holdout benchmark for predictive evaluation in the empirical setting. In addition to point estimates, we produce 95% posterior predictive intervals for the naive test-set estimates of ${\text{OEC}}_t$ for each method. These intervals allow us to assess whether the inferences from the meta-analysis methods are appropriately calibrated.

Table 3 summarizes the methods’ predictive performance in terms of two performance metrics defined in Section 5.2: the Win Rate and MLEQ. The definitions of these metrics are modified slightly to accommodate the fact that we no longer have access to the ground-truth values of ${\text{OEC}}_t$; we replace these ground-truth values with the naive method’s test-set estimates. We see that the four meta-analysis methods substantially outperform the naive approach in terms of the Win Rate, achieving lower prediction error in 69%–73% of A/B tests. The MLEQ metric produces the same conclusion as the Win Rate—that the meta-analysis methods substantially outperform the naive method. Among the meta-analysis methods, DPM and T perform best overall, with Gaussian and Gaussian-0 performing similarly but clearly better than the naive method. The DPM-naive comparison in the MLEQ block equals $-0.29$, meaning that the DPM’s log absolute prediction error is 0.29 points lower than that of the naive method. Because this comparison is on the log absolute-error scale, exponentiating gives $\text{exp}(-0.29)=0.75$; equivalently, the DPM method reduces absolute prediction error by about 25% on the original scale. Quantifying the uncertainty in these performance metrics is challenging because the meta-analysis predictions are statistically dependent. The standard errors shown in parentheses in Table 3 are calculated assuming independence of the predictions, so they should be interpreted as rough approximations. We address the uncertainty quantification more rigorously in our simulation study in Section 5 using independent replications. Compared with the results in Table 2, we see that the predictive nature of this comparison dilutes the differences between methods because (i) the “true value” in this analysis is a noisy estimate of ${\text{OEC}}_t$ and (ii) the estimates are formed using only 50% of the data.

Table 3. Pair-Wise Method Comparisons for Predictive Performance

Figure 4 visually compares the methods’ absolute prediction errors in a pair-wise fashion. Below the diagonal in Figure 4, methods listed in each row can visually be seen to outperform the method listed in the column when the points fall below the ${45}^{°}$ line. These plots in Figure 4 provide visual confirmation of the relative performances shown in Table 3. In particular, we see that the meta-analysis methods typically produce smaller absolute prediction errors than the naive method. The large differences in scale can be observed by noting that the absolute prediction errors shown are plotted on the log scale, with most values falling between 0.01 and 100.

Figure 5 plots the ranks of the predictions against the ranks of the prediction errors. Ideally, we would see no linear relationship between these ranks, indicating that the inferences are unbiased in a conditional sense (i.e., given a subset of tests having estimated ${\text{OEC}}_t$ equal to $u\in \mathbb{R}$, the conditional expectation of ${\text{OEC}}_t$ for these tests is, in fact, equal to the estimate, u). Although this seems to be a natural methodological requirement, the naive approach does not satisfy it because of regression to the mean. This shortcoming is visually apparent in panel (a) of Figure 5 with the strong negative correlation between the predictions and errors ($\rho =-0.59$). In common practice, A/B tests with large, statistically significant ${\text{OEC}}_t$ estimates are implemented and internally reported. The unfortunate implication is that the impact of these “winning” tests is typically overstated. Bayesian methods provide a natural fix for this problem by performing conditional inference; under correct model specification, the posterior mean possesses a conditional unbiasedness property by design. This property is evident in panels (b)–(e) of Figure 5, which show much lower correlation between prediction ranks and error ranks compared with the naive approach ($|\rho |<0.3$ in all four cases). The DPM model achieves the greatest reduction as evidenced by the relatively flat line in panel (e) of Figure 5 and a small Spearman rank correlation of −0.14. This finding suggests that the DPM method is likely modeling the true prior distribution, $P_{\mathit{\theta }}$, more accurately than the other methods. It effectively debiases the naive estimates (${\mathbf{z}}_t$) by accurately modeling the heterogeneity in the true effects (${\mathit{\mu }}_t$) via its flexible representation of the prior distribution as a mixture of inferred clusters. It regularizes each estimate just enough that its expected error is near zero regardless of the value of the naive estimate. This property can also be seen in the overestimation metric in Section 5.2, which is almost exactly zero for the DPM method.

Table 4 compares the performance of the methods’ prediction intervals. All methods produce near-nominal coverage. However, the meta-analysis methods produce shorter intervals. This is especially true of the DPM method, which produces median interval lengths that are about 25% shorter than those of the naive method, indicating substantial benefit from including a prior distribution for the ${\mathit{\mu }}_t$ values. Again, the predictive nature of this comparison dilutes the difference in interval width.

Table 4. Performance of 95% Prediction Intervals

6.3. Full-Data Analysis

In addition to the sample-split predictive analysis, we also applied the DPM method to the full data set. To evaluate Assumption 4, we used the DPM model to compute posterior predictive p-values, checking for correlation between the entries of ${\mathit{\mu }}_t$ (and their absolute values) and the corresponding variance entries in ${\mathbf{\Sigma }}_t$. The smallest posterior predictive p-value was 0.05. With 10 comparisons, this represents only minimal evidence against Assumption 4. The rest of the analysis provides insights into the inner workings of the DPM model and the latent structure of the causal effects across tests.

Table 5 displays estimates of the surrogacy measures introduced in Section 4.1. Because the DPM model produces test-specific estimates of these measures, we average their values across tests to obtain estimates of the overall strength of surrogacy. The estimated values of ${\mathit{\text{R}}}^2$ fall in the range from 0.46 to 0.78, indicating moderately strong surrogacy. Effects on the surrogates (${\mathbf{x}}_t$) do, in fact, predict effects on the KPIs (${\mathbf{y}}_t$); however, they do so imperfectly. The ${\mathit{\text{R}}}^2$ value for ${\text{OEC}}_t$ is smaller than that of both ${\mu }_1$ and ${\mu }_2$, indicating that ${\mu }_1$ and ${\mu }_2$ show some degree of internal surrogacy. The estimates of TVR are in the range from 0.10 to 0.16, indicating a reduction in the posterior standard deviation of over 60% compared with the naive approach. Achieving a comparable reduction using the naive method alone would require increasing the sample size by a factor of $1/\text{TVR}$; here, this corresponds with a factor of 6–10, which is not achievable because these tests are continuously run in sequence across different teams. These metrics can also be used to expand and refine the list of included surrogates. For instance, Table 5 indicates that the surrogates for ${\mu }_2$ have larger ${\mathit{\text{R}}}^2$ values than those for ${\mu }_1$, suggesting that the analysis may benefit relatively more from adding a strong surrogate for ${\mu }_1$ compared with adding a strong surrogate for ${\mu }_2$.

Table 5. Aggregate Surrogacy Measures

Figure 6 displays heat maps that visualize the inner workings of the DPM model. The posterior mean for the number of clusters is 7.2 with a 95% credible interval of 4–12. Panel (a) of Figure 6 plots posterior probabilities that pairs of tests are clustered together. These probabilities represent the model’s posterior belief that the tests arise from the same distribution. We used hierarchical clustering methods (Murtagh and Contreras 2012) to group and sort the tests.¹⁰ Doing so reveals the existence of several small, distinct clusters that are internally homogeneous but distinct from the other clusters. We applied the DPM method to an enlarged set of business metrics and obtained similar results, although with more distinct clusters. Upon inspecting these clusters, we found many cases where the DPM method had clustered tests involving a similar intervention or platform. For instance, one cluster of tests primarily affected the search experience for rental properties, and another involved only tests implemented on mobile devices. In practice, these model-based clusterings could be used to identify experiment-level metadata that could be used within a metaregression model.

The values of ${\mathit{\eta }}_k$ and ${\mathbf{\Omega }}_k$ provide valuable strategic insights, such as the average effect of these interventions and how these effects correlate across metrics. An example of the latter is that the surrogacy relationships may differ across these clusters relative to the main cluster. The fact that the DPM model has access only to the values of ${\mathbf{z}}_t$ and ${\mathbf{\Sigma }}_t$ indicates that these groupings are apparent from the test results themselves. In addition to improving our estimates of ${\mathit{\mu }}_t$, these latent groupings could be used to learn the “profile” of certain types of tests, which could be used to ideate and prioritize future interventions. In analyzing the tth test, the latent groupings could also inform decision making by comparing results across similar tests.

Panel (b) of Figure 6 displays estimated correlations among the components of ${\mathit{\mu }}_t$. To further preserve confidentiality, the order of the metrics has been permuted. The plot in panel (b) of Figure 6 reveals that two pairs of metrics show strong positive correlation within pairs but are nearly uncorrelated between pairs. Additionally, the last metric shown is positively correlated with all four of the other metrics. The fact that all five metrics are, to varying degrees, positively correlated suggests that these metrics tend to have synergistic effects—improvements in one metric are associated with improvements in the others. Thus, in addition to offering improved effect estimates, the meta-analysis methods also uncover interpretable insights regarding the distribution of causal effects. As an example, suppose we learn that the treatment effects on a form-fill metric and a conversion metric are highly correlated across experiments. This finding implies that interventions improving the form-fill rate also tend to improve downstream conversions, which may influence the firm’s allocation of experimentation resources.

In the biostatistics literature on clinical trials, correlations like these are sometimes used to justify using one surrogate metric in place of another that is noisier or more difficult to measure. Although this strategy could be applied in A/B testing as well, we generally recommend selecting a target estimand (${\text{OEC}}_t$) that closely aligns with long-term business goals and applying our meta-analysis methods to frame all subsequent inferences in terms of this estimand; the resulting strategy is analogous to the procedures proposed in Elliott et al. (2015) to avoid the so-called “surrogacy paradox,” in which a strong surrogate produces incorrect inferences for a particular intervention. A key benefit of the experimental meta-analysis approach over purely observational (e.g., data-mining) methods is that the effects are causal by design—not a misleading artifact of noncausal associations. Thus, in addition to improving A/B test decision making, the meta-analysis framework can provide crucial business insights that may be missed or badly biased in an observational analysis.

7. Discussion

This section provides a brief summary of our proposed framework, the relevant managerial considerations, and possible extensions.

7.1. Methodological and Results Summary

The primary contribution of this work is a meta-analytic framework for estimating the long-term value of A/B tests that (i) replaces additional causal surrogacy assumptions with an assumption that comparable experiments have exchangeable causal-effect vectors, (ii) leverages cross-metric relationships for increased efficiency, and (iii) is computationally scalable because of its reliance on summary statistics. Additionally, we demonstrate via simulation and an empirical application to 190 A/B tests in real estate technology that the proposed framework provides a large improvement over single-test analyses. Although our application is in real estate technology specifically, the framework is generally applicable to settings with many experiments that (i) share metric definitions, (ii) present little or no cross-experiment interference, and (iii) arise from a common domain so that the shared-effect distribution assumption in Assumption 4 or its conditional version in Assumption 5 is credible. Conversely, the framework is less suitable when experiments interfere significantly or differ sharply in domain, customer population, or metric definition unless these differences can be modeled explicitly through metaregression or careful filtering of the experiment archive.

We have presented three basic meta-analytic models: Gaussian, T, and DPM. We recommend informing the choice of the meta-analytic model via modeling diagnostics, including posterior model checking and information criteria (Spiegelhalter et al. 2002, Watanabe 2010). Because of its flexibility, we generally expect the DPM to outperform the Gaussian and T models; however, the latter two models can be fit much more quickly. In practice, analysts also have the flexibility to generalize well beyond these models, specifying metaregression models that allow the effect distribution to vary based on experiment attributes. When relevant sources of heterogeneity are omitted (e.g., differences in customer segments, product types, or experimental contexts), the model may be well calibrated only on average across experiments and can perform poorly for specific subsets of tests. In general, we expect that more heterogeneous experiments will require more flexible models to satisfy Assumption 4 or Assumption 5. Appendix A discusses additional practical considerations for applying our framework.

7.2. Managerial Implications

This work carries two primary managerial implications. First, Sections 5 and 6 uncovered a large statistical benefit in adopting meta-analytic methods. Our proposed framework offers a satisfying solution to the dilemma of choosing which business metrics or portion of the business funnel (top versus middle versus bottom) to prioritize. Our framework enables managers to define success in terms of bottom-of-funnel results (purchases, customer retention, etc.) and simultaneously reap the statistical benefits of top-of-funnel proxies (click-through rates, form-fill rates, etc.). These proxy metrics are relevant to decision making to the extent that causal effects on the proxies correlate with causal effects on strategic bottom-of-funnel outcomes. As a result, our meta-analysis methods provide a data-driven alternative to the industry practice of “OEC by committee,” in which business leaders manually select a set of metrics to prioritize. The meta-analysis methods also provide an elegant solution to the challenges of simpler OEC methods, including (i) the need to customize the OEC to different lines of business or software platforms, (ii) the inability to measure certain metrics for some tests because of technical constraints, and (iii) the sensitivity of the OEC definition to the time frame over which metrics are measured.

Second, our framework enables rigorous tracking of cumulative long-term experimental impacts, even when tests are selected for implementation or reporting based on large positive estimated effects. Because of the statistical phenomenon of regression to the mean, this common decision-making practice can substantially overstate cumulative impacts; see panel (a) of Figure 5 and our results on the overestimation metric in Section 5. Our meta-analytic framework addresses this problem by “shrinking” estimates toward regions of high prior probability, substantially reducing regression to the mean effects and the overstatement induced by selecting tests with large positive naive estimates ${\mathbf{z}}_t$. In practice, this feature of our framework can facilitate decision making surrounding organizational resource allocation and strategic priorities by producing reliable estimates of the financial value of implemented A/B tests.

7.3. Extensions

We present three common meta-analysis methods ranging from simple parametric models to flexible nonparametric alternatives. However, the statistical literature on meta-analysis and random-effects models is vast and includes many alternatives that could be applied within our OEC framework, including parametric and nonparametric empirical Bayes methods (Higgins et al. 2009, Efron 2014, Koenker and Mizera 2014). The Bayesian solutions presented here are notable for their extensibility and ability to provide actionable inferences, including surrogacy measures, explicit estimates of $P_{\mathit{\theta }}$, and estimated clusterings.

Another promising direction is to develop methods that extrapolate effects beyond a short-term treatment window. Our framework can estimate the long-term impact of short-term treatments, averaging over any trends or business decisions between the test end date and the end of the measurement period. However, in some business settings, the true goal may be to estimate the long-term impact of long-term treatments, such as the effect on 12-month revenue of adding (versus not adding) a new feature over a 6-month period. This extension requires a different set of methods that effectively forecast the long-term effects using short-term proxies (Huang et al. 2026). Future work could consider how to combine these methods with the meta-analysis perspective presented here.