Estimating Effects of Long-Term Treatments

1. Introduction

Online controlled experiments, often referred to as A/B tests, have become the gold standard for evaluating the impact of product updates for technology companies. These updates can include the introduction of new product functions, user interface designs, and recommendation algorithms (Kohavi et al. 2013, Bakshy et al. 2014, Xu et al. 2015, Bojinov and Gupta 2022, Ye et al. 2022, Larsen et al. 2024). By randomly assigning experimental units (e.g., users) to different groups and exposing them to different product versions, A/B tests can measure the effects of the product update and guide business decisions. Modern technology companies deploy thousands of experiments daily to enable rapid iterations in their product development (Kohavi et al. 2013, Hohnhold et al. 2015, Leng and Dimmery 2021, Ye et al. 2023).

Estimating the effects of product updates presents a challenge in A/B testing (Gupta et al. 2019, Kohavi et al. 2020). When companies deploy a product update, it is usually intended to remain in the system for a long duration, typically spanning several months or over a year. Ideally, companies need to conduct long-term experiments to ensure that these updates have a lasting positive impact on user satisfaction and improve key product metrics. However, in practice, A/B tests are often short-term, typically lasting only several days or weeks. This is due to the considerable costs associated with long-term experiments, such as occupying substantial user traffic for an extended period and causing potential delays in the product iteration process (Kohavi et al. 2020, Bojinov et al. 2023). Short-term A/B tests offer the benefits of rapid feedback and lower costs, allowing companies to economize resources and maintain their agility in a competitive market.

The treatment effects derived from these short-term experiments can substantially differ from the actual effects of long-term product updates (Kohavi et al. 2012, Hohnhold et al. 2015, Munro et al. 2021). A notable phenomenon here is the “novelty effect”: users may show higher levels of interest or response to a new or unfamiliar feature, resulting in stronger short-term outcomes in the treatment group. However, as users become more acquainted with this feature, this effect often diminishes over time (Xu et al. 2015). Similarly, the “primacy effect” arises when the benefits of a new feature only become evident after users have had sufficient time to become familiar with it, leading to a gradual increase in treatment effects over time (Kohavi et al. 2020). Moreover, the introduction of new product changes in online marketplaces can cause disturbances in the product ecosystem, which could take a long duration to stabilize (Glynn et al. 2020, Wager and Xu 2021, Bright et al. 2022, Farias et al. 2022, Hu and Wager 2022, Johari et al. 2022). Although practitioners often rely on the treatment effects in short-term experiments to represent the impact of long-term product changes in decision making, the above scenarios underscore that this practice can mislead their decisions.

To address the above challenge, we introduce the “longitudinal surrogate framework” in this paper. Our theoretical results and empirical evidence suggest the feasibility of making trustworthy estimation of the effects of long-term treatments using data collected from short-term experiments. Our framework proposes to use “longitudinal surrogates,” which are the intermediate outcomes that saturate the causal links between historical treatments and future outcomes. We iteratively make use of these longitudinal surrogates and define the “longitudinal surrogate index” and “pivot index” functions. These index functions enable us to extrapolate the longitudinal surrogates from the short-term experimental periods to the long-term future periods. Within this framework, we explain the underlying identification assumptions, the estimation strategies, inferential techniques, and strategies for validating our assumptions.

Empirically, we collaborated with WeChat, one of the world’s largest social networking platforms, to validate the effectiveness of our framework through two large-scale, long-term experiments, each involving over a million users. To leverage the long-term nature of these experiments, we partition the horizon into an “experimental period” and a “future period.” At the end of the experimental period, we apply our approach to estimate the treatment effects in the future period and compare our estimates with the true treatment effects observed in those periods. We show that our approach consistently outperforms two baseline approaches—the Constant Extrapolation Baseline (CEB) and the Vector Autoregressive (VAR) model (Stock and Watson 2001)—as well as several related existing solutions. Compared with the baseline approaches, our approach reduces the estimation bias across different experimental periods by $59.8\%$, averaged across both experiments in our study, without increasing mean squared errors (MSEs). Additionally, we conduct synthetic experiments to supplement our real-world experiments. We also conduct tests for the assumptions made under our framework, and discuss the practical guidelines to facilitate the applications of our methods in real-world settings.

Our longitudinal surrogate framework builds on the literature on proxies and surrogates (Prentice 1989, Weir and Walley 2006, Joffe and Greene 2009, Anderer et al. 2022, Imbens et al. 2022, Yang et al. 2023, Athey et al. 2025). Yet our work differs from these previous studies in both the problem it addresses and the solutions it offers. Previous studies often employ surrogates to estimate the “long-term effects of short-term treatments,” as seen in applications such as job training programs (Athey et al. 2025) and marketing campaigns (Yang et al. 2023). In contrast, our framework is designed to estimate the “long-term effects of long-term treatments,” where subjects receive continuous treatments over extended periods. This context necessitates the estimation of the combined effects of both past and ongoing treatments, requiring a novel approach. For a comprehensive comparison of our work with that of Athey et al. (2025), please refer to Online Appendix E.4. Similarly, Battocchi et al. (2021) address treatment effect estimation in long-term time series using surrogates. Their research focuses on a dynamic treatment setting, where treatment decisions in each period are influenced by previous treatments and outcomes, differing from the question in our study where the same treatment is employed over a long-term period.

Prior works, such as Hohnhold et al. (2015) and Munro et al. (2023) from online advertising applications, take a different approach when estimating the long-term effects. They model user learning behavior over time using parametric models with stronger assumptions, and combine such parametric models with nontrivial (i.e., Cookie-Cookie-Day) experiments. In contrast, our approach focuses on traditional randomized experiments, and conducts nontrivial postexperiment analysis. Our approach is designed to integrate with the conventional A/B testing pipelines at modern technology companies, avoiding the additional conceptual or implementation cost associated with executing nontrivial experiments.

More broadly, our work is also related to panel data experiments. In panel data experiments, not only are subjects repeatedly measured over time, but the treatment itself is also flexibly introduced, modified, or removed at different points in time (e.g., Basse et al. 2019; Doudchenko et al. 2019, 2021; Xiong et al. 2019, 2023; Abadie and Zhao 2021; Athey et al. 2021; Chen and Bayati 2021; Ni et al. 2023). The major difference is that our approach only uses data collected from short-term experiments with standard A/B testing procedures, instead of using the entire panel.

2. The Longitudinal Surrogate Framework

2.1. Problem Setup

Consider an A/B testing problem that an experimenter faces on an online platform. The platform conducts an A/B test to evaluate the effects of introducing a new product update. To do so, the platform includes a total of N experimental subjects, denoted by set $[N]=\{1,2,\dots ,N\}$. Each subject is typically an active user. Each subject $i\in [N]$ is endowed with some R-dimensional covariates ${\mathit{X}}_i\in \mathbb{X}\subseteq {\mathbb{R}}^R$, which we refer to as the pretreatment variables. For example, the pretreatment variables ${\mathit{X}}_i$ are typically user demographics at online platforms. In this paper, we only consider the setting where the pretreatment variables are low-dimensional; that is, the dimension of ${\mathit{X}}_i$ is much smaller than the number of experimental subjects N.

The experimenter is interested in understanding the effects of a long-term treatment, yet they can only run the experiment for a shorter duration. We explain the horizon as follows. Let there be a discrete, finite time horizon consisting of $T=T_E+T_F$ time periods in chronological order. Out of these T time periods, the first $T_E$ time periods are referred to as the experimental periods, and the last $T_F$ time periods are referred to as the future periods. After conducting the experiment until the end of the experimental periods $T_E$, the experimenter has access to data collected from periods 1 to $T_E$, and is interested in some causal effects that will not be directly observed until the end of period T. In our running example, the experimenter could run the experiment for a few weeks, and then use the experimental data to estimate what would happen if the intervention continues to last for additional weeks. See Figure 1 for an illustration.

We consider two versions of treatments although our approach can easily extend to multiple treatments. One version is the control condition (or, simply, “control”), which represents the status quo of the product; the other version is the active treatment (or, simply, “treatment”), which represents the product with the new feature. Let $W_{i,t}$ be the random treatment assignment that subject $i\in [N]$ receives in time period $t\in [T]$. $W_{i,t}$ takes values from $\{0,1\}$, where 0 stands for control and 1 stands for treatment. For each subject, we use ${\mathit{W}}_{i,1:t}$ to stand for the treatment assignments that subject $i\in [N]$ receives during periods 1 to t. Following convention, we use ${\mathit{W}}_{i,1:t}$ to stand for a random treatment assignment and ${\mathit{w}}_{i,1:t}$ to stand for one realization. When the subscript i is clear from the context, we sometimes drop it for brevity, and write ${\mathit{W}}_{1:t}$ instead.

We conduct a randomized experiment wherein once a subject is assigned into either the treatment or control group, it stays in that group during the entire horizon. If subject i is assigned into the treatment group, then ${\mathit{W}}_{i,1:T}={\mathbf{1}}_T$; if subject i is assigned into the control group, then ${\mathit{W}}_{i,1:T}={\mathbf{0}}_T$, where we use ${\mathbf{1}}_t$ and ${\mathbf{0}}_t$ to stand for a length-t vector of ones and zeros, respectively. As we stand at the end of period $T_E$, we have only conducted the experiment during the first $T_E$ experimental periods, and not yet in the last $T_F$ future periods.

We do not consider other types of treatment patterns that change the treatment assignment in the middle of the horizon, such as a step-wedge design (i.e., a staggered adoption pattern; Brown and Lilford 2006, Hussey and Hughes 2007, Hemming et al. 2015, Li et al. 2018, Xiong et al. 2019) or a switchback design (Cochran et al. 1941, Glynn et al. 2020, Hu and Wager 2022, Bojinov et al. 2023, Xiong et al. 2023). This implies that, for simplicity, we could just use a single binary variable to indicate if a subject is assigned to the treatment or control group. But for clarity, we would rather carry the treatment assignment vector. Although the treatment assignments remain the same over time, the treatment probabilities across different subjects can be different. Our framework allows treatment assignments to be dependent on ${\mathit{X}}_i$ (i.e., stratified randomization), although we have only conducted complete randomization in our empirical execution.

During the $T_E$ experimental periods, the experimenter observes several quantities of interest. For each subject $i\in [N]$ and at each time period $t\in [T_E]$, the experimenter observes a primary outcome $Y_{it}$ that takes values from $\mathbb{Y}\subseteq \mathbb{R}$ and D intermediate outcomes ${\mathit{S}}_{it}$ that take values from $\mathbb{S}\subseteq {\mathbb{R}}^D$. In our running example, the primary outcome could be the click-through rate and the intermediate outcomes could include a number of user activity metrics such as log-in frequency, average usage duration, number of total searches, and the numbers of searches in each category.

Following the potential outcomes framework (Neyman 1923) and under the Stable Unit Treatment Value Assumption (Rubin 1974, Holland 1986, Imbens and Rubin 2015), each subject $i\in [N]$ at each time period $t\in [T_E]$ has a set of potential outcomes $Y_{it}({\mathit{W}}_{i,1:t})$ and ${\mathit{S}}_{it}({\mathit{W}}_{i,1:t})$. Each observed outcome, either the primary outcome or the intermediate outcome, is related to its respective potential outcomes as follows,

During the future periods $\{T_E+1,\dots ,T\}$, we could also define the same quantities as above, although the observed outcomes have not been observed by the experimenter. See Table 1 for an illustration of our problem setup and summary of notations.

Table 1. Illustration of Our Problem Setup and Summary of Notations

In addition, let ${\mathit{S}}_{i0}$ be some pretreatment intermediate outcomes at time 0, which may reflect subject-level heterogeneity before the experiment. For notational convenience, we collect ${\mathit{Y}}_i={\{Y_{it}({\mathit{w}}_{1:t})\}}_{t\in [T],{\mathit{w}}_{1:t}}$ and ${\mathit{S}}_i={\{{\mathit{S}}_{i0},{\mathit{S}}_{it}({\mathit{w}}_{1:t})\}}_{t\in [T],{\mathit{w}}_{1:t}}$ to be all the potential outcomes. Further, we introduce a short-hand notation to emphasize the most recent treatment assignments. For any $i\in [N]$ and any $t<t^{\prime }\in [T]$, if ${\mathit{W}}_{i,1:t}={\mathbf{0}}_{1:t}$, then we write $Y_{i{t}^{\prime }}({\mathit{W}}_{i,t+1:t^{\prime }})≔Y_{i{t}^{\prime }}({\mathit{W}}_{i,1:t^{\prime }})$. Note that this is only a short-hand notation, and does not impose any assumptions.

In this paper, we postulate a superpopulation that each subject is sampled from with replacement, so that each subject $i\in [N]$ is identically and independently distributed. For each $i\in [N]$, let $\mathcal{F}$ be the joint probability distribution that $({\mathit{X}}_i,{\mathit{Y}}_i,{\mathit{S}}_i)$ is sampled from. There are two sources of randomness in our experiment: one comes from the randomized experiment, that is, the treatment assignments are random; the other comes from the sampling from a superpopulation, that is, the pretreatment variables and all the potential outcomes are random.

The experimenter is interested in understanding the average effect of long-term treatments on the primary outcome,

Such causal effects often emerge when experimenters aim to permanently launch a new product. In our running example, this relates to click-through rates over weeks or months.

2.2. Conventional Wisdom and New Challenges

In this paper, the duration of treatments spans the entire horizon, which we refer to as long-term treatments. To estimate the effects of long-term treatments, the ideal approach is to conduct experiments for an extended duration of time in the future periods $\{T_E+1,\dots ,T\}$ and directly estimate ${\tau }_T$ from such an ideal experiment. However, as discussed in Section 1, the experimenter is often unable to assign treatments for a long-term duration, and there is no observation from the future periods at the moment of estimation. The fundamental challenges associated with this problem are two-fold:

1. (Missing treatments) At the moment of estimation, the experimenter has not conducted any treatment in the future periods.

2. (Missing observations) At the moment of estimation, the experimenter has not observed any outcome in the future periods.

The presence of the above two challenges requires a new method that explicitly considers the longitudinal nature of the treatments, where the existing surrogate approach (Prentice 1989, Weir and Walley 2006, Joffe and Greene 2009, Yang et al. 2023, Athey et al. 2025) does not directly apply. For example, Athey et al. (2025) and Yang et al. (2023) examine the treatment effects where the duration of treatments is relatively short compared with the length of future periods and the treatments never occurred during the future periods. We thus refer to the effect they studied as the long-term effects of short-term treatments; in other words, they focus on estimating the long-term “carryover effects,” that is,

Therefore, the existing surrogate approach addresses the second challenge only and establishes a surrogate predictor using the historical data, which are used to extrapolate from the short-term observations. Unless the treatments in the future periods have no direct effects, that is, ${\mathbb{E}}_{\mathcal{F}}[Y_{iT}({\mathbf{1}}_T)]={\mathbb{E}}_{\mathcal{F}}[Y_{iT}({\mathbf{1}}_{T_E},{\mathbf{0}}_{T_F})]$, the existing surrogate approach will lead to biased estimation of ${\tau }_T$, the average effect of long-term treatments.

To address the above two challenges, we propose a framework to extend the existing surrogate approaches to the longitudinal setting discussed above. Below, we introduce a few identification assumptions that we make in the longitudinal surrogate framework.

2.3. Identification Assumptions

Below, we first introduce the longitudinal surrogate model and the two required identification assumptions. These two identification assumptions are what we refer to as the first level of assumptions. Because the longitudinal surrogate model may suffer from the potentially limited sample size (see Section 3.1 for details), we introduce an additional assumption to the first level of assumptions, leading to the linear surrogate model.¹

2.3.1. Longitudinal Surrogate Model.

We start with the basic assumptions that lay out the foundations of estimating the causal effect. There are two such basic assumptions.

Assumption 1

(Longitudinal Surrogacy). The treatment assignment at an earlier period is independent of the primary and intermediate outcomes at a later period, conditional on the intermediate outcomes at a middle period, that is, there exists a subset of time indices $\mathbb{T}=\{t_1,t_2,\dots ,t_K\}\subseteq [T]$, such that for any $i\in [N]$, any $t\in \mathbb{T}$, and any $t^{\prime }>t$,

$$\left(Y_{i{t}^{\prime }},{\mathit{S}}_{i{t}^{\prime }}\right)⫫{\mathit{W}}_{i,1:t}|{\mathit{S}}_{it},{\mathit{X}}_i.$$

Moreover, we refer to the intermediate outcomes at the time periods $t\in \mathbb{T}$ as surrogate outcomes, or, simply, surrogates.

Assumption 1 is the longitudinal extension of the surrogacy assumption in the literature (Prentice 1989, Weir and Walley 2006, Joffe and Greene 2009, Yang et al. 2023, Athey et al. 2025). Intuitively, Assumption 1 implies that the surrogate outcomes at a middle period fully saturate the causal link between the treatment assignment at an earlier period and the primary and intermediate outcomes at a later period. In other words, there is no effect of the treatment assignment at an earlier period on the primary and intermediate outcomes at a later period that does not pass through the surrogate outcomes at the middle period. See Figure 2 for an illustration using the directed acyclic graph representation (Pearl 1995). We discuss practical guidelines for choosing surrogates in Online Appendix Section D.

There are two direct implications of Assumption 1. The first implication is that, if Assumption 1 holds for some $\mathbb{T}$, it also holds for any subset of $\mathbb{T}$; that is, for any ${\mathbb{T}}^{\prime }\subseteq \mathbb{T}$, Assumption 1 also holds for ${\mathbb{T}}^{\prime }$. The second implication is that, for any $i\in [N]$, any $t\in \mathbb{T}$, and any $t^{\prime }>t\ge t^{″}$,

This is because, if $(Y_{i{t}^{\prime }},{\mathit{S}}_{i{t}^{\prime }})$ and ${\mathit{S}}_{i{t}^{″}}$ are not independent, then ${\mathit{W}}_{i,1:t^{″}}$ and ${\mathit{S}}_{i{t}^{″}}$ will not be independent, violating Assumption 1.

In the longitudinal surrogate model, the surrogate outcomes serve as critical links in the causal diagram in two ways. First, conditional on the surrogate outcomes, we extrapolate to the primary outcomes in the future periods using what we refer to as the longitudinal surrogate index, which we define in Definition 1. Second, conditional on the surrogate outcomes at an earlier period, we build our understanding of the future surrogate outcomes using what we refer to as the pivot index, which we define in Definition 2.

Definition 1

(Longitudinal Surrogate Index). For any $t\in [T],\mathit{s}\in \mathbb{S},\mathit{x}\in \mathbb{X},{\mathit{w}}_{1:t}\in \{{\mathbf{0}}_t,{\mathbf{1}}_t\}$, the surrogate index is the conditional expectation of the primary outcome at time t, given the surrogate outcomes at time 0, the pretreatment variables, and the treatment assignments, that is,

$$h_t(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})={\mathbb{E}}_{\mathcal{F}}\left[Y_{it}|{\mathit{S}}_{i0}=\mathit{s},{\mathit{X}}_i=\mathit{x},{\mathit{W}}_{i,1:t}={\mathit{w}}_{1:t}\right],$$

where the expectation is taken over $Y_{it}$.

Intuitively, the longitudinal surrogate index serves as a prediction of future primary outcomes using the current intermediate outcomes, the pretreatment variables, and the treatment assignments. This index has a time-dependent subscript, which reflects the longitudinal nature of our setup, and is different from the surrogate index as originally defined in Athey et al. (2025).

In addition to the longitudinal surrogate index, we introduce the pivot index as defined below.²

Definition 2

(Pivot Index). For any $t\in [T]$, $\mathit{s}\in \mathbb{S}$, $\mathit{x}\in \mathbb{X}$, ${\mathit{w}}_{1:t}\in \{{\mathbf{0}}_t,{\mathbf{1}}_t\}$, the pivot index is a vector of the conditional expectations of the surrogate outcomes at time t, given the surrogate outcomes at time 0, the pretreatment variables, and the treatment assignments, that is,

$${\mathit{g}}_t(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})={\mathbb{E}}_{\mathcal{F}}\left[{\mathit{S}}_{it}|{\mathit{S}}_{i0}=\mathit{s},{\mathit{X}}_i=\mathit{x},{\mathit{W}}_{i,1:t}={\mathit{w}}_{1:t}\right],$$

where the expectation is taken over ${\mathit{S}}_{it}$. Moreover, we denote the conditional surrogate outcomes at time t, given the surrogate outcomes at time 0, the pretreatment variables, and the treatment assignments, to be

$${\mathit{G}}_t(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})\sim {\mathit{S}}_{it}|{\mathit{S}}_{i0}=\mathit{s},{\mathit{X}}_i=\mathit{x},{\mathit{W}}_{i,1:t}={\mathit{w}}_{1:t}.$$

The pivot indices (or the conditional surrogate outcomes, depending on which identification strategy to use) are the key idea behind our longitudinal surrogate framework. “$\sim$” indicates following the same distributions. Intuitively, they bridge the surrogates at the earlier periods and the surrogates at the later periods. The use of pivot indices is necessary in our model because the experimental duration is short, and what we learn from the experimental data needs the pivot indices (or the conditional surrogate outcomes) to iterate and extrapolate to the future periods. Note that the definition of pivot indices replaces the primary outcomes as defined in Definition 1 by the surrogate outcomes.

Assumption 2

(Comparability). The primary and intermediate outcomes across different periods share the same support. The distribution of the primary and intermediate outcomes at a later period, conditional on the intermediate outcomes at an earlier period, on the treatment assignments during the earlier and later periods, and on the pretreatment variables, is the same across different time periods; that is, for any $t,t^{\prime }\in [T]$, and any positive integer $\delta \in {\mathbb{N}}^{+}$,

$$(Y_{it},{\mathit{S}}_{it})|{\mathit{S}}_{i(t-\delta )},{\mathit{W}}_{i,t-\delta +1:t},{\mathit{X}}_i\hspace{0.17em}\sim \hspace{0.17em}(Y_{i{t}^{\prime }},{\mathit{S}}_{i{t}^{\prime }})|{\mathit{S}}_{i(t^{\prime }-\delta )},{\mathit{W}}_{i,t^{\prime }-\delta +1:t^{\prime }},{\mathit{X}}_i.$$

Intuitively, Assumption 2 implies that the relationship between the primary and intermediate outcomes at a later period and the intermediate outcomes at an earlier period is the same at other time periods. So, we could use data collected from the experimental periods to learn the relationship and apply it to future periods. Note that Assumption 2 does not necessarily assume the primary outcomes or the surrogate outcomes are time-homogeneous; instead, Assumption 2 assumes the functions of the surrogate index and the pivot indices to be time-homogeneous.

Assumptions 1 and 2 are the most basic level of assumptions. Under Assumptions 1 and 2, and using the succinct notations from Definitions 1 and 2, we present the first identification result as follows.

We first introduce a special case to illustrate the key idea behind our main theorem.

Lemma 1.

Consider the special case when $T_E=T_F$. Under Assumptions 1 and 2, where Assumption 1 holds for $\mathbb{T}=\{T_E\}$, the average effect of long-term treatments on the primary outcome is equal to the following expression:

$${\tau }_T={\mathbb{E}}_{\mathcal{F}}\left[h_{T_E}({\mathit{G}}_{T_E}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{1}}_{T_E}),{\mathit{X}}_i,{\mathbf{1}}_{T_E})\right]-{\mathbb{E}}_{\mathcal{F}}\left[h_{T_E}({\mathit{G}}_{T_E}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{0}}_{T_E}),{\mathit{X}}_i,{\mathbf{0}}_{T_E})\right].$$

Lemma 1 consists of two components: the surrogate index component $h_{T_E}(·,·,·)$ that predicts the primary outcomes using the pivots and a conditional surrogate outcomes component ${\mathit{G}}_{T_E}(·,·,·)$ that reweighs the distributions of the random surrogate outcomes using the pretreatment surrogate outcomes. Lemma 1 illustrates how the surrogate outcomes at $T_E$ as the outputs of the inner loop reweighting are used as the input of the outer loop surrogate index. The surrogate outcomes at $T_E$ effectively serve as the link between the two components.

In the more general setting when $T_F>T_E$, we need to have more surrogate outcomes to serve as the links. We split the horizon of T periods into several intervals, each length of which is no larger than the length of the experimental periods. Mathematically, denote $\mathrm{\Delta }{t}_k≔t_k-t_{k-1}$. The above condition suggests that $T_E\ge {\max }_{k\in [K+1]}\mathrm{\Delta }{t}_k$. We write $t_{K+1}=T$ and $t_0=0$ as the end and start of all periods. Then, we apply the same method as in Lemma 1 on each interval and update the surrogate outcomes iteratively. We formalize the above intuition as follows.

Theorem 1

(Longitudinal Surrogate Model). Under Assumptions 1 and 2, where Assumption 1 holds for $\mathbb{T}=\{t_1,t_2,\dots ,t_K\}$, the average effect of long-term treatments on the primary outcome is equal to the following expression:

$$\begin{array}{l}{\tau }_T={\mathbb{E}}_{\mathcal{F}}\left[h_{\mathrm{\Delta }{t}_{K+1}}({\mathit{G}}_{\mathrm{\Delta }{t}_K}(\dots {\mathit{G}}_{\mathrm{\Delta }{t}_1}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_1})\dots ,{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_K}),{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_{K+1}})\right]\hfill \\ -{\mathbb{E}}_{\mathcal{F}}\left[h_{\mathrm{\Delta }{t}_{K+1}}({\mathit{G}}_{\mathrm{\Delta }{t}_K}(\dots {\mathit{G}}_{\mathrm{\Delta }{t}_1}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_1})\dots ,{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_K}),{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_{K+1}})\right],\hfill \end{array}$$

where the expectation is taken over ${\mathit{S}}_{i0},{\mathit{X}}_i$, as well as the conditional surrogate outcomes ${\mathit{G}}_{\mathrm{\Delta }{t}_1},\dots ,{\mathit{G}}_{\mathrm{\Delta }{t}_K}$.

Theorem 1 consists of a sequence of iterative components. There is one surrogate index component $h_{\mathrm{\Delta }{t}_{K+1}}(·,·,·)$ that predicts the primary outcomes during the last interval, using the conditional surrogate outcomes reweighted from the second last interval. There is a sequence of conditional surrogate outcomes ${\mathit{G}}_{·}(·,·,·)$ that reweighs the distributions using the conditional surrogate outcomes reweighted from the previous interval. Both components (i.e., the surrogate index and the conditional surrogate outcomes) can be estimated from the data during the experimental periods.

2.3.2. Linear Surrogate Model.

Although general, the first identification strategy as suggested by Lemma 1 and Theorem 1 suffers from a major challenge resulting from the random nature of conditional surrogate outcomes and potentially limited sample sizes. We will revisit this challenge in greater detail in Section 3.1. To address this, we introduce an additional assumption to the two basic assumptions. This set of three assumptions is the second level of assumptions.

Assumption 3

(Linearity of Surrogates).

1. The surrogate index function is linear with respect to the surrogates; that is, there exists ${\alpha }_d(\mathit{x},{\mathit{w}}_{1:t})$, $\forall d\in \{0,1,\dots ,D\},\mathit{x}\in \mathbb{X},{\mathit{w}}_{1:t}\in \{{\mathbf{0}}_t,{\mathbf{1}}_t\}$, such that

   $$h_t(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})={\alpha }_0(\mathit{x},{\mathit{w}}_{1:t})+{\displaystyle \sum _{d=1}^D{s}_d}·{\alpha }_d(\mathit{x},{\mathit{w}}_{1:t}).$$(2)

2. The pivot index function is linear with respect to the surrogates; that is, there exists ${\beta }_{d,d^{\prime }}(\mathit{x},{\mathit{w}}_{1:t})$, $\forall d\in [D],d^{\prime }\in \{0,1,\dots ,D\},\mathit{x}\in \mathbb{X},{\mathit{w}}_{1:t}\in \{{\mathbf{0}}_t,{\mathbf{1}}_t\}$, such that for each $d\in [D]$,

   $$g_{t,d}(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})={\beta }_{d,0}(\mathit{x},{\mathit{w}}_{1:t})+{\displaystyle \sum _{d^{\prime }=1}^D{s}_d}·{\beta }_{d,d^{\prime }}(\mathit{x},{\mathit{w}}_{1:t}),$$(3)

   where $g_{t,d}(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})$ stands for the d-th component of ${\mathit{g}}_t(\mathit{s},\mathit{x},{\mathit{w}}_{1:t})$, the pivot index.

Assumption 3 specifies a linear functional form to the surrogate index and the pivot index. It is worth mentioning that Assumption 3 assumes both the surrogate index and the pivot index to be linear with respect to the surrogates, but not necessarily with respect to the pretreatment variables. Under this additional Assumption 3, we simplify Theorem 1 and introduce the second identification result as follows.

Theorem 2

(Linear Surrogate Model). Under Assumptions 1, 2, and 3, where Assumption 1 holds for $\mathbb{T}=\{t_1,t_2,\dots ,t_K\}$, the average effect of long-term treatments on the primary outcome is equal to the following expression:

$$\begin{array}{l}{\tau }_T={\mathbb{E}}_{\mathcal{F}}[h_{\mathrm{\Delta }{t}_{K+1}}({\mathit{g}}_{\mathrm{\Delta }{t}_K}(\dots {\mathit{g}}_{\mathrm{\Delta }{t}_1}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_1})\dots ,{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_K}),{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_{K+1}})]\hfill \\ -{\mathbb{E}}_{\mathcal{F}}[h_{\mathrm{\Delta }{t}_{K+1}}({\mathit{g}}_{\mathrm{\Delta }{t}_K}(\dots {\mathit{g}}_{\mathrm{\Delta }{t}_1}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_1})\dots ,{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_K}),{\mathit{X}}_i,{\mathbf{0}}_{\mathrm{\Delta }{t}_{K+1}})],\hfill \end{array}$$

where the expectation is taken over ${\mathit{S}}_{i0},{\mathit{X}}_i$.

Theorem 2 involves both the surrogate index and the pivot index. The input of an outer iteration is the output of an inner iteration, which, under the linearity assumption, is simply the pivot index in the inner iteration. With this linear model, the identification strategy as suggested by Theorem 2 properly mitigates the issues of large sample sizes as required by the longitudinal surrogate model, and thus estimates the future treatment effects with reasonable sample sizes.

3. Estimation and Inference

In this section, we discuss the estimation strategies, inference strategies, and model validation strategies for the models discussed above. We focus on conventional randomized experiments where subjects are randomly assigned into the treatment or the control groups under (covariate-independent) complete randomization. Let $N_1$ and $N_0$ be the number of users in the treatment and the control group, respectively, which are fixed quantities under complete randomization. Our approach readily applies to more general randomization schemes, which we omit in this paper.

3.1. Estimation Strategies

Recall that in Section 2.3 we introduce two levels of identification assumptions. Below, we introduce two estimation strategies, each requiring one level of assumptions discussed in Section 2.3.

3.1.1. Estimators for the Longitudinal Surrogate Model.

Given estimators of the surrogate index and estimators of the conditional surrogate outcomes, we follow Theorem 1 and obtain the following plug-in estimator:

We explain how to estimate the surrogate index functions in (4). For any $t\in [T_E]$, $\mathit{x}\in \mathbb{X}$, $\mathit{s}\in \mathbb{S}$, one naive estimator of the surrogate index under consecutive controls is given by

Under complete randomization, such an estimator is unbiased for the surrogate index function. Similarly, for any $t\in [T_E]$, $\mathit{x}\in \mathbb{X}$, $\mathit{s}\in \mathbb{S}$, one naive estimator of the surrogate index under consecutive treatments is given by

Under complete randomization, such an estimator is unbiased for the surrogate index function. Yet given the oftentimes multidimensional nature of $\mathit{s}$ and $\mathit{x}$, and the limited number of treatment subjects in the experimental periods, the above two estimators are not always well-behaved. For each combination of $\mathit{s}$ and $\mathit{x}$, we need a sufficiently large number of samples in the experimental periods to have reasonably accurate estimation, which is often challenging in practice.

3.1.2. Estimators for the Linear Surrogate Model.

Because of the limitations of the longitudinal surrogate model, we introduce the linear surrogate model, which requires the additional Assumption 3. Given the surrogate and pivot index estimators, we follow Theorem 2 and obtain the following plug-in estimator:

Note that because $S_{i\mathrm{\Delta }{t}_1}$ is directly observable, we can use the observed $S_{i\mathrm{\Delta }{t}_1}$ to replace ${\widehat{\mathit{g}}}_{\mathrm{\Delta }{t}_1}({\mathit{S}}_{i0},{\mathit{X}}_i,{\mathbf{1}}_{\mathrm{\Delta }{t}_1})$ in the first (inner) plug-in. We use the following plug-in estimator in empirical estimation:

We explain how to estimate the surrogate and pivot index functions in (6). We first consider a proper discretization of the pretreatment variables $\mathit{x}$. Then, for each $\mathit{x}$ and under homoscedasticity, a naive estimator of the coefficients of the surrogate index function is given by

and for each $d\in [D]$, the pivot index function is given bywhere $S_{i0,d}$ and $S_{it,d}$ stand for the d-th dimension of surrogate outcomes ${\mathit{S}}_{i0}$ and ${\mathit{S}}_{it}$, respectively. The estimators of the surrogate and pivot index functions are obtained by replacing the coefficients in (2) and (3) with their estimated counterparts. Under complete randomization, both estimators are unbiased for the linear coefficients in (2) and (3). The second part in (6) can be estimated similarly. See Lemma 2 in Online Appendix C.4.

The above two least squares estimators find the coefficients for any $\mathit{x}\in \mathbb{X}$. This is suitable when the pretreatment variables are low-dimensional and discrete. Given the multidimensional nature of $\mathit{x}$, and especially when $\mathit{x}$ is continuous, the least squares estimators are not always well-behaved. To address the above concern, we could include the pretreatment variables ${\mathit{X}}_i$ in the least square term instead of conditioning on them. Instead of estimating ${\widehat{\alpha }}_d(\mathit{x},{\mathbf{1}}_t)$ and ${\widehat{\beta }}_{d,d^{\prime }}(\mathit{x},{\mathbf{1}}_t)$, we pool the data and run the following linear regression to estimate ${\widehat{\alpha }}_d({\mathbf{1}}_t)$ and ${\widehat{\beta }}_{d,d^{\prime }}({\mathbf{1}}_t)$, as well as ${\widehat{\varphi }}_r({\mathbf{1}}_t)$ and ${\widehat{\psi }}_{d,r}({\mathbf{1}}_t)$:

and for each $d\in [D]$,

The second part in (6) can be estimated similarly. The above expressions find the best linear unbiased estimator for the coefficients of the pretreatment variables. They mitigate the issue of requiring a large sample size in the longitudinal surrogate model.

3.2. Inference and Testing

Our estimator leverages an additional layer of randomness from the random treatment assignments. Here, we propose Fisher’s exact test to draw inference from the collected data. We consider the following sharp null hypothesis of no treatment effect at any time period for any subject:

We can conduct exact tests by leveraging the completely randomized experiment to simulate new treatment assignments; see Algorithm 1 in the Online Appendix. To obtain a confidence interval, we propose inverting a sequence of exact hypothesis tests to identify the region outside of which (7) is violated at the prespecified nominal level (Imbens and Rubin 2015, chapter 5). Alternatively, one could also use bootstrap to obtain a confidence interval. The source of randomness comes from our random treatment assignments; see Algorithm 2 in the Online Appendix. In later empirical sections, we mainly report the results using the bootstrap method.

Our work is also related to forecasting methods in the time series analysis and the macroeconometrics literature, such as autoregressive models, the VAR model, and Autoregressive Integrated Moving Average (ARIMA) (Stock and Watson 2001, 2020; Andersen et al. 2003; Fuller 2009; Hamilton 2020). The macroeconometrics literature has also provided ways to construct confidence intervals by leveraging the randomness of the joint probability distribution that $({\mathit{X}}_i,{\mathit{Y}}_i,{\mathit{S}}_i)$ is sampled from. Such confidence intervals are generally recognized to have more power than Fisher’s exact test, which relies on the randomness of the random treatment assignments. For simplicity, we adopted the simpler approach of Fisher’s exact test and the bootstrap method.

3.3. Validation of Assumptions

As the longitudinal surrogacy (assumption Assumption 1) and the comparability (assumption Assumption 2) play a critical role in determining the validity of our method in practice, we explore approaches to validate whether these assumptions are satisfied in this section.³

3.3.1. Validation of Assumption 1.

Similar to the tests on the validity of instrumental variables, Assumption 1 cannot be directly tested. Instead, we propose conducting a sensitivity analysis to determine how sensitive the treatment effect estimation is when Assumption 1 is violated. Our approach is inspired by the literature on sensitivity analysis of instrumental variables (Baiocchi et al. 2014). Arguably, the most common violation of Assumption 1 occurs when there are omitted surrogates. Figure G.12 in the Online Appendix illustrates such a scenario: Assumption 1 is violated because the treatment assignment during the experimental periods $1:T_E$ affects the primary outcome through both variables ${\mathit{S}}_{T_E}$ and ${\mathit{U}}_{T_E}$. Here, only ${\mathit{S}}_{T_E}$ are considered as the surrogate variables, whereas ${\mathit{U}}_{T_E}$ represent the omitted surrogates that remain unidentified or uncollected.

First, a straightforward approach for sensitivity analysis on Assumption 1 is to assess the fluctuation in estimation given that only a subset of surrogate outcomes are applied as surrogates. This analysis reveals how the estimation is impacted by the exclusion of certain already-collected surrogates. We demonstrate that as more surrogates are removed, the estimation performance deteriorates, aligning with our intuition. Overall, our estimation approach is relatively robust across different subsets of surrogates. Detailed analysis of this approach is provided in Online Appendix G.1.

Second, we design an approach to test the sensitivity of omitted surrogates, focusing on assessing the model’s sensitivity to surrogates that were never observed. This approach can be particularly valuable in real-world experiments where some of the surrogates can be potentially unobservable and missing from our estimation. Our method can be seen as an adaptation of the sensitivity analysis for assessing the Exclusion Restriction assumption for instrumental variables (Baiocchi et al. 2014).

Suppose, for any $i\in [N]$, $t\in \mathbb{T}$, and ${\mathit{w}}_{1:t}\in \{{\mathbf{0}}_t,{\mathbf{1}}_t\}$, the treatment assignment affects the primary outcome not only through the identified surrogates but also via a missing variable ${\zeta }_{it}$. We create this variable ${\zeta }_{it}$ following a normal distribution with mean zero, and variance equal to the average variance of the Y during the experimental periods. We manually introduce an additional causal path between the treatment assignment and the primary outcome through variable ${\zeta }_{it}$:

where $\theta$ is a parameter that we generate to vary the degree of omitted surrogates and $\mathrm{𝟙}[·]$ is the indicator function. In this sensitivity analysis, we treat $\widetilde{Y_{it}}$ instead of $Y_{it}$ as the primary outcome and consider only the observed surrogates $S_{it}$, as if the omitted surrogate ${\zeta }_{it}$ were neither observed nor collected. Clearly, Assumption 1 is violated because of the omitted surrogate ${\zeta }_{it}$, and a larger $\theta$ indicates a greater violation of Assumption 1. We then follow the same procedure to estimate the average effect of long-term treatments. Finally, we compare these estimates with the ones obtained using $Y_{it}$ as the true primary outcome variable, where Assumption 1 is not violated. This approach allows us to examine the sensitivity of our estimation results to varying degrees of violation of the surrogacy assumption. A detailed demonstration of this sensitivity analysis, along with empirical experiments, is provided in Online Appendix G.2. The results show that the bias and root mean squared error (RMSE) remain stable when $\theta$ is relatively small, demonstrating the robustness of the estimation.

3.3.2. Validation of Assumption 2.

We begin by introducing a straightforward test directly for Assumption 2 (the comparability assumption). Moreover, we discuss that even when Assumption 2 does not hold, we can still apply our longitudinal surrogate framework, by leveraging a relaxation of Assumption 2, which we refer to as the Parallel Trends assumption (Assumption 2′). We also provide a test for this parallel trends assumption.

3.3.2.1. Direct Test for Assumption 2.

The objective of this test is to identify matched observations across two distinct time periods, t and $t^{\prime }$, based on exact matching criteria involving the surrogates ${\mathit{S}}_i$, the treatment assignments ${\mathit{W}}_i$, and the pretreatment variables ${\mathit{X}}_i$. More specifically, we begin by specifying the two time periods of interest, t and $t^{\prime }$, and the lag parameter $\delta$. For each unit i at time t, we collect the following information: ${\mathit{S}}_{i,t-\delta },{\mathit{W}}_{i,t-\delta +1:t}$, and ${\mathit{X}}_i$. Next, we search for any unit $i^{\prime }$ at time $t^{\prime }$ that satisfies the following conditions:

All pairs of observations $(i,i^{\prime })$ that meet the above conditions are included in the analysis pool, which results in two groups of observations from each of the two time periods t and $t^{\prime }$, with the corresponding outcomes $(Y_{it},Y_{i^{\prime }{t}^{\prime }})$. If no observations meet the requirement at time t, the test for that specific condition is excluded from further analysis. For each possible combination of $\mathit{s},\mathit{w},\mathit{x}$, we perform statistical tests to examine the difference between $Y_{it}$ and $Y_{i^{\prime }{t}^{\prime }}$ and report p-values.

3.3.2.2. Parallel Trends Test.

To make our longitudinal surrogate framework more useful to practitioners, we relax Assumption 2 to Assumption 2′, which we call the Parallel Trends assumption. When combined with the linearity assumption and under certain conditions, this new assumption still guarantees that Theorem 2 holds. The detailed theory of Assumption 2′ is presented in Online Appendix F. Assumption 2′ can be more robust to real-world settings.

Below, we introduce a statistical test to evaluate whether the parallel trends assumption holds by focusing on two distinct time periods, denoted as t and $t^{\prime }$, along with a specified positive integer $\delta$. The first step is a matching procedure. For each unit i in the treatment group characterized by the preperiod surrogates ${\mathit{S}}_{i,t-\delta }$ and pretreatment covariates ${\mathit{X}}_i$ at time t, where the treatment assignment satisfies ${\mathit{W}}_{i,t-\delta +1:t}={\mathbf{1}}_{\delta }$, we identify an exact match in the time period $t^{\prime }$. The matching criteria require that the matched unit $i^{\prime }$ satisfies

Upon locating an exact match, one observation from period $t^{\prime }$ is randomly selected to form a matched pair $(Y_{it},Y_{i^{\prime }{t}^{\prime }})$ within the treatment group. Observations without an exact match are excluded from the evaluation. This matching process is similarly applied to the control group, where the treatment assignment condition is ${\mathit{W}}_{i,t-\delta +1:t}={\mathbf{0}}_{\delta }$, resulting in matched pairs $(Y_{it},Y_{i^{\prime }{t}^{\prime }})$ within the control group.

The second step is a regression analysis. This exact matching ensures that the paired observations in both the treatment and control groups are conditioned on identical distributions of preperiod surrogates and pretreatment covariates. The regression model is specified as follows for the matched pairs only:

We estimate the parameters of this regression model and conduct a t-test for the null hypothesis $H_0:{\beta }_3=0.$ Failure to reject $H_0$ suggests that the parallel trends assumption may not be violated. Note that a comprehensive discussion on the validation of the comparability assumption and parallel trends assumption, including theorem, related proof, and the statistical testing results derived from empirical experiments, is provided in Online Appendix F.

4. Empirical Validation

We collaborated with WeChat and analyzed two real-world long-term experiments on WeChat Search to validate the effectiveness of our proposed approach.⁴ WeChat Search serves as a function within WeChat, enabling users to search for information both internally and externally to the WeChat platform.⁵ These experiments offer valuable data, enabling us to observe the ground truth of treatment effects in the future periods and compare them with our estimates made at the end of the experimental period.⁶ Sections 4.1 and 4.2 offer detailed descriptions of the experiment background and our empirical strategy and results.

After analyzing the experimental results, we further validate the effectiveness of our approach using multiple synthetic experiments, detailed in Section 4.3. These synthetic experiments discuss scenarios not necessarily represented in the two real-world experiments, offering a thorough examination of our proposed method. In Section 4.4, we provide additional robustness analyses of our real-world experiments.

4.1. Experiment 1: Mini-programs in Search History

4.1.1. Experiment Background.

Similar to many other social media platforms, WeChat provides a search box that allows users to search for a variety of embedded WeChat features, such as chat history, news articles, and mini-programs (embedded third-party apps). In Experiment 1, practitioners aimed to test whether displaying recently searched mini-programs as part of the search query history in the search box would affect user activity on WeChat Search.

As presented in Figure 3, the “search history” panel provides a shortcut for users to quickly access the search results of keywords they previously searched. In the treatment condition, the experiment extended the functionality of the “search history” panel by providing additional shortcuts to access mini-programs that users had recently used. The control condition did not show this new function and remained as the status quo. The experimenters hypothesized that with this new feature, users would be more likely to visit their frequently used mini-programs through the shortcuts provided by WeChat Search, rather than swiping down on WeChat and scrolling to find the target mini-programs. The business objective of this treatment was to encourage users to engage more with WeChat Search, thereby increasing its user engagement. Figure 3 illustrates the user interfaces for both the treatment and control groups.

In the experiment, about 1.3 million users were randomly assigned to treatment or control groups. The treatment group consists of 667,206 users, whereas the control group consists of 665,830 users. The primary outcome of interest is weekly search_uv, the average number of days that a user has searched in a week.⁷ During this seven-week experiment, the results showed a positive treatment effect with a sharp increasing trend in the short term (the first two weeks), setting high initial expectations for the new feature’s potential. However, the positive treatment effect becomes stable, albeit slightly diminished, in the long term (see the trends for the ground truth in Figure 4(a)). As a result, the treatment was launched to all users after the experiment.

For randomization checks, we performed two tests. First, we conducted the sample ratio mismatch (SRM) test (Fabijan et al. 2019), which uses chi-squared tests to examine whether the sample sizes of the two groups are not significantly different, as 50% of the number of experiment participants were assigned to each group. The experiment passed the chi-squared test, indicating no sample ratio mismatch problem. Second, we observed that there were no significant differences in the pretreatment variables between the two groups before the experiment. We performed t-tests for mean comparisons, where all the p-values are larger than 0.1, suggesting the insignificant differences and the validity of our randomization process. See details in Online Appendix E.2.

4.1.2. Empirical Strategy.

In our analysis, we divided the seven-week experimental period into two phases: the experimental periods 1 to $T_E$ and the future periods $T_E+1$ to T. During the experimental periods, we collected data and observed the effects of the treatment. After the experimental periods end, our goal is to predict the treatment effects for each week in the future periods, starting from week $T_E+1$ and continuing through the last period. While making these predictions, we do not use data from the future periods, as they have not been observed yet at time $T_E$. We use our model to estimate the treatment effects during the future periods. Finally, we compare these estimated effects with the actual treatment effects observed during the future period. These observed effects in the long-term experiment serve as the “ground truth” to evaluate the accuracy of our approach.

We consider variables that capture various aspects of user behavior during the search process as our surrogates. Detailed descriptions of all surrogate and primary outcomes are provided in Table E.1 in the Online Appendix. These surrogates not only are responsive to the treatments but also reflect the diverse aspects of user behavior that lead to variations in primary outcomes over time (Deng et al. 2013, Duan et al. 2021). Note that we include past primary outcomes as a subset of the surrogate variables, as they are shown to be useful in modeling the future primary outcomes (Deng et al. 2013). This is a little different from the causal diagram shown in Figure 2, yet this still satisfies Assumption 1. To see this, consider the following simplest example with only two periods $t_k$ and $t_{k-1}$ for any $k\in \{2,\dots ,K\}$. Let the surrogates consist of two parts ${\mathit{S}}_{i,t_k}=(Y_{i,t_k},Y_{i,t_{k-1}},{\tilde{\mathit{S}}}_{i,t_k})$, where $Y_{i,t_k}$ and $Y_{i,t_{k-1}}$ are primary outcomes and ${\tilde{\mathit{S}}}_{i,t_k}$ are the other surrogate outcomes. We still have

because $Y_{i,t_{k-1}}$ is a constant when conditional on $Y_{i,t_{k-1}}$, and it is conditionally independent of ${\mathit{W}}_{i,1:t_{k-1}}$. This could enlarge the surrogate space and potentially better satisfy the longitudinal surrogacy assumption. We provide detailed practical guidelines for choosing surrogates in Section E.6 of the Online Appendix.

Note that we not only use surrogates from the immediate preceding time period $t-1$ but also incorporate surrogates including primary outcomes from earlier periods—$t-2,\hspace{0.17em}t-3,\cdots$ up to $t-T_E+1$—into our model. For example, search_uv in period $t-2$ can be seen as the “search_uv two weeks ago,” which is then used as a surrogate in period t. Therefore, to establish the models, we use the surrogates (including the primary outcomes) from week 1 to week $T_E-1$ in total to be our training features, and the surrogates and primary outcome of the week $T_E$ serve as the training outcomes. As we have five surrogate variables, our prediction model has $5\times (T_E-1)$ training features.⁸ By employing this approach, we effectively broaden the surrogate space, thereby enhancing the precision of our predictions.

After establishing the models for the primary outcome and surrogates, we iteratively use each model to estimate surrogate and primary outcome values for each week during the future period, that is, $T_E+1,T_E+2,\dots ,T$. Note that the prediction model is not supposed to have access to the actual values of any surrogates or primary outcomes post-$T_E$. Consequently, the input features for each model are based on both the observed surrogate and primary outcome values during the $T_E$ experimental periods and their predicted values post-$T_E$. For example, we employ observed surrogates from weeks 2 to $T_E$ to project those in $T_E+1$, and then we utilize the surrogates observed from weeks 3 to $T_E$ as well as the surrogates previously predicted for time $T_E+1$ to estimate those in $T_E+2$ (and so on). With this iteration, we are able to predict both primary outcomes and surrogates until time T.

We focus on presenting the results from our main model, the linear surrogate model.⁹ We construct confidence intervals using the bootstrapping technique (Efron 1987, Efron and Tibshirani 1994). We use a bootstrapping approach to estimate the confidence intervals for the long-term treatment effects. We resample 50% of the users with replacement to create each replica, selecting half of the original sample to form a new subsample.¹⁰ For each replica, we build a separate prediction model using only this subsample. Based on this model, we then estimate the long-term treatment effects for each replica. This process is repeated 100 times to determine a 95% confidence interval for the true treatment effect. This method allows us to account for variability from both the random assignment of subjects and the model itself.

4.1.3. Baselines.

We employ two different baselines with confidence intervals using the same bootstrap technique described above.

• CEB: We use the average treatment effects observed during the first $T_E$ weeks of the experiment to predict the treatment effects for the future period. Although obviously this approach cannot capture any increasing or decreasing trends in the treatment effects, this serves as a common industry practice.

• VAR model: We employ a VAR model with lag order $p=T_E-2$ on the initial $T_E$ weeks of the multivariate time series, using the average values of four surrogates and one primary outcome variable as input candidates. This allows the VAR model to forecast future outcomes based on past values of all included variables, though VAR is traditionally used for forecasting rather than causal inference (Stock and Watson 2001).¹¹

4.1.4. Results.

We present the estimates of the linear surrogate model, the baselines, and true effects in Figure 4(a). We vary the value of $T_E$ from two to four to ensure that $T_E$ is meaningfully short compared with the entire duration, constituting around half or less of the entire horizon. We observe that the CEB consistently underestimates the effects of long-term treatments. The vector autoregressive models perform slightly better than CEB, especially when $T_E$ is larger. However, these baseline models cannot predict the long-term increasing trend of the treatment effect.

By contrast, our estimation, indicated by red curves in Figure 4, can successfully capture an increasing trend in the treatment effect regardless of the choice of $T_E$. For instance, our estimation successfully predicts both a long-term increasing trend at $T_E=2$ and a stable trend at $T_E=4$, which other baseline models fail to do. In practice, successfully predicting the trend of treatment effects over time is critical for making product decisions. In addition, using the first two weeks only, our estimation of the effect of long-term treatment in week 7 is 1.347, which is less biased compared with the true effect (1.278) compared with baselines.

Further, we compare the bias and MSE between our model and the baselines. Specifically, we present their averages over all weeks during future periods and present the results in Table 2. Overall, considering the three choices of $T_E$, our model consistently outperforms the baseline model in terms of bias, and outperforms the baseline model in terms of MSE in the majority of cases. These results further underscore the effectiveness of our approach.¹²

Table 2. Comparison Result Between Different Methods in Terms of Bias and MSE for Experiment 1

As $T_E$ increases, an estimation model is generally expected to have higher bias and MSE, as predictions are made further into the future and are less anchored by current observations. However, temporal fluctuations—such as seasonal effects, holidays, and rare events—can introduce additional complexity that disrupts this trend. Such events can make certain short- or medium-term periods more challenging to predict accurately than other periods further in the future. As a result, although a general increase in bias and MSE with forecast horizon length may hold, this trend is not strictly monotonic and can vary based on the occurrence of these less predictable events.

We also examine whether the surrogacy and comparability assumptions hold for the experiment using the methods proposed in Section 3.3. First, we conduct a sensitivity analysis for the surrogacy assumptions, presenting the results in Online Appendix G.1 and Online Appendix G.2, to demonstrate the robustness of our estimation to the potential of omitted surrogates. Moreover, we perform both the tests for comparability and parallel trends assumptions. Detailed results are presented in Online Appendix F.1 and Online Appendix F.5.

4.2. Experiment 2: Search Discovery

4.2.1. Experiment Background.

Similar to Experiment 1, Experiment 2 also involves a change in WeChat Search. Instead of adding shortcuts to mini-programs in the “search history,” practitioners aimed to test whether displaying hot topics as part of the search discovery in the search box would affect user activity on WeChat Search. The experimenters hypothesized that with this new feature, users would be more likely to read and engage with these new shortcuts to trendy topics. The business objective of this treatment was to encourage users to engage more with WeChat Search, thereby increasing its user engagement. Figure 3 illustrates the user interfaces for both the treatment and control groups. In the treatment condition, the users were offered this new feature, whereas in the control condition the users were not. However, the long-term effect of this treatment remains uncertain and critical because including this new panel of hot topics might also crowd out users’ intention to search. Different from Experiment 1, where the new feature mainly assists in searching for the mini-programs based on individuals’ search history, the new feature for Experiment 2 is to provide shortcuts that help users to explore hot topics, which might affect their initial search intention. Thus, WeChat launched this experiment for a total of 20 weeks.

This 20-week experiment involves 3.6 million WeChat users. Among them, 1,807,335 users were randomly assigned to the treatment group, whereas 1,803,675 users were randomly assigned to the control group. Again, the primary outcome of interest is search_uv. Because both experiments focus on WeChat Search and share the same primary outcome, the same set of surrogates described in Table E.1 in the Online Appendix is used for Experiment 2. In this experiment, the goal is to predict the treatment effects over a long period until period T (the 20th week) using the data available at the end of period $T_E$.

To examine the validity of randomization, we also performed the SRM test and mean comparisons on the pretreatment variables between the treatment and control groups, similar to Experiment 1. The results confirm the validity of our randomization process, showing that there is no statistically significant difference in the sample sizes and no statistically significant differences in the pretreatment variables between the two groups. More details are discussed in Section E.2 of the Online Appendix. In addition, we present the summary statistics in Table E.2 of the Online Appendix.

The average treatment effect shows continuous fluctuations without an apparent downward trend signal in the first seven weeks, whereas these are followed by a continuous decline after eight weeks of treatment over time. It is suspected that there is a long-lasting novelty effect for this treatment, and the effect is likely to decay over time. As a result, this new product change (treatment) was not eventually adopted or launched to all users. Nevertheless, the valuable insights gained from this experiment have inspired the development of other significant product strategies.

4.2.2. Empirical Strategy, Baselines, and Results.

Both experiments conducted on WeChat Search have the same primary outcomes and surrogates, so we use the same empirical strategy as in Experiment 1. The potential consistency of surrogates among different experiments can enable easy scalability of our approach in practice. Because this experiment is longer (20 weeks), we employ the linear surrogate model and showcase results for ($T_E=8,9,10$) in the main text, while presenting results with different choices of ($T_E$) in Online Appendix E.3. Additionally, we use the same baselines for validation in Experiment 2 as those in Experiment 1 for consistency.

The estimation results are presented in Figure 4(b). We observe that our approach effectively captures the decreasing trend of the average treatment effect in the long run. By contrast, the CEB model consistently overestimates the treatment effects during the future period $T_F$, as it fails to capture the decreasing trend of the treatment effect. The VAR model exhibits fluctuating estimates over time and unstable prediction trends across different experimental periods $T_E$, because of the volatility of the primary outcome (Y), search_uv, over time in both the treatment and control groups. The VAR model appears to be unable to handle this scenario well.

Table 3 reports the average bias and MSE over the $T_F$ future periods for each $T_E$. Consistent with the results from Experiment 1, our method outperforms both baseline models (CEB and VAR) in terms of bias across all values of $T_E$. As the forecast horizon extends beyond the experimental period, our model tends to exhibit increased estimation variance for more distant future periods. This phenomenon occurs because errors in near-term predictions can propagate and amplify when used as inputs for subsequent, longer-term forecasts. A similar issue arises with the VAR baseline model, which also relies on near-future periods’ information for extended predictions. Despite both our method and the VAR model exhibiting higher variance compared with the trivial constant extrapolation, we consider this an acceptable bias-variance tradeoff. Similar to Experiment 1, we conduct analyses proposed in Section 3.3 to examine surrogacy and comparability assumptions. We demonstrate our estimation’s robustness to both assumptions and present the results in Online Appendices G.1, G.2, F.1, and F.5.

Table 3. Comparison Result Between Different Methods in Terms of Bias and MSE for Experiment 2

4.3. Simulations Using Synthetic Data

In addition, to encompass scenarios not necessarily represented in the real-world experiments, we undertake synthetic experiments for a more thorough evaluation of our approach.

4.3.1. Stabilized Treatment Effect.

In our synthetic experiments, the first scenario we investigate is when the effects of long-term treatments plateau or stabilize over time. To illustrate this, we set up the following synthetic experiment: The simulation presupposes four surrogates, ${\mathit{S}}_{it}$, for each time period t and unit i. For each dimension d, each of its corresponding surrogates draws from a normal distribution, $S_{it,d}\sim \mathcal{N}({\mu }_d,{\sigma }_d^2)$. Surrogates in different dimensions are independent from each other. In this synthetic experiment, subjects assigned to the control group experience no deviation from the status quo; as a result, the surrogates’ distribution remains unchanged. In contrast, for those in the treatment group, there is a time-dependent decay in the four surrogates, governed by decay factors $\mathit{\gamma }=(0.8,0.6,0.4,0.2)$, respectively (e.g., $S_{it,d+1}={\gamma }_d·S_{it,d}$). In order to comply with both the surrogacy and linearity assumptions, the primary outcome, $Y_{it}$, is designed as a linear combination of these four surrogates.

In the first synthetic experiment, we set the parameter ${\mu }_d\sim \mathcal{N}(2,1)$ and ${\sigma }_d\sim \mathcal{N}(2,1)$, and the primary outcome Y in period t is formulated as $Y_{it}=-(0.1S_{it,1}+0.1S_{it,2}+0.4S_{it,3}+0.4S_{it,4})$. In this setup, the effect of long-term treatments on $Y_{it}$ initially increases and then stabilizes, showcasing a characteristic “leveling-off” pattern. Using experimental data spanning $T_E=2,\hspace{0.17em}3,\hspace{0.17em}\text{and} 4$ periods, we compare our approach’s estimates with the true future effects. The first row of Figure H.18 in the Online Appendix demonstrates a precise estimation of the effects of long-term treatments.

The second simulation shares the settings with the first one except for the parameter ${\mu }_d\sim \mathcal{N}(1.5,1)$ and ${\sigma }_d\sim \mathcal{N}(1,1)$, and the primary outcome being formulated as $Y_{i,t+1}=0.1S_{it,1}+0.1S_{it,2}+0.4S_{it,3}+0.4S_{it,4}$.¹³ This configuration leads to the effect of the long-term treatment on Y initially declining and then stabilizing, exemplifying another typical “leveling-off” trend. The first row of Figure H.19 in the Online Appendix presents the estimation results, demonstrating that our approach can accurately capture the future treatment effects.

In both synthetic experiments, our estimates closely align with the true effects of long-term treatment, demonstrating our approach’s capability to account for scenarios where treatment effects stabilize over time. Figures H.18 and H.19 in the Online Appendix showcase the graphical comparison between our approach and all the other baseline models, including the CEB model and the VAR model in two synthetic experiments. Moreover, numerical comparison between our approach and multiple baselines in terms of bias and MSE is provided in Tables H.10 and H.11 in the Online Appendix. Our approach surpasses all of the baseline models in both synthetic experiments regarding bias and MSE. Collectively, these analyses further show the validity and generalization of our approach to various empirical settings.

Further, we performed a sensitivity analysis for the surrogacy (assumption Assumption 1) on both of the two synthetic experiments to demonstrate the relationship between the degree of violation of Assumption 1 and estimation accuracy. The results in Online Appendix G.1 show that performance worsens with more severe violations of Assumption 1, but a longer observational experimental period can mitigate this deterioration to some extent.

4.3.2. Additional Synthetic Experiments.

To complement our real-world experiments, we conduct additional synthetic experiments that challenge certain assumptions or alter the behavior of long-term treatment effects. We explore two scenarios.

4.3.2.1. Violation of Comparability.

In this experiment, we create synthetic contexts where the comparability assumption may not hold and test whether our framework can detect these violations, as well as observe how its performance changes accordingly. We simulate scenarios with varying degrees of comparability assumption violations. In this simulation, the primary outcome for users in the treatment group is defined as $Y_{it}=-\gamma \times (0.1S_{it,1}+0.4S_{it,2})$. When $t=2$ and i is in the treatment group, we vary $\gamma$ over the values [1, 1.5, 2, 2.5, 3] to control the extent of the comparability violation. For all other time periods for the treatment group and for all time periods in the control group (including $t=2$), we set $\gamma =1$. We demonstrate that both the comparability and parallel trends assumption tests we proposed can effectively detect this violation. Moreover, as the degree of violation increases (i.e., as $\gamma$ becomes larger), estimation bias increases accordingly. Please refer to Online Appendix H.2 for more details.

4.3.2.2. Nonlinear Outcome Function.

Although our main results rely on the linearity assumption in the linear surrogate model, we also create synthetic contexts where this assumption may not hold and test how our method’s performance may be affected. We evaluate our method under a nonlinear outcome function $Y_{it}$ by introducing two surrogates $S_{it,1}$ and $S_{it,2}$. The primary outcome is $Y_{i,t+1}=-(S_{it,1}+\theta e^{S_{it,2}})$, where $\theta$ adjusts the degree of nonlinearity. Note that to create the treatment effect, we allow the surrogates in the treatment group to decay over t, whereas the surrogates for users in the control group do not exhibit this decay; their difference is the treatment effect. Our method yields accurate long-term estimates when linearity is not severely violated, demonstrating the robustness of our approach to linearity to some extent. The detailed setups and results are provided in Online Appendix H.3.

4.3.2.3. No Long-Term Treatment Effect.

Here, the long-term treatment effect diminishes over time, with surrogates following the same distribution across the treatment and control groups. The outcome for the treatment group includes a diminishing term $\frac{{(-1)}^t}{{(t+2)}^3}$. That is, $Y_{i,t+1}=-(0.1S_{it,1}+0.1S_{it,2}+0.4S_{it,3}+0.4S_{it,4})+\frac{{(-1)}^t}{{(t+2)}^3}$. By contrast, the outcome for the control group is the same but excludes the term $\frac{{(-1)}^t}{{(t+2)}^3}$. The difference between the treatment and control groups reveals a treatment effect that gradually fades, converging to zero as t increases. Our empirical results show that our method effectively predicts this decline using short-term data, showing its capability with transient effects. The detailed setups, results, and analyses are provided in Online Appendix H.4.

Overall, the findings from these experiments demonstrate that our approach remains effective even when some assumptions are moderately relaxed or when the treatment effects exhibit different temporal patterns, demonstrating its applicability in a variety of real-world settings.

4.4. Robustness Checks

The following analyses demonstrate the robustness of our methods further. First, instead of using the full sample, we focused on each heterogeneous user group in the two WeChat experiments. Detailed implementations are illustrated in Online Appendix E.5.1. Figures E.5 and E.6 in the Online Appendix illustrate the estimated long-term treatment effects for each group in the two separate experiments. We also present the biases and MSEs for each subgroup in Tables E.4 and E.5 in the Online Appendix. The results show a close alignment of our estimation with the true effects across various heterogeneous groups.

Second, to address the challenge of the curse of dimensionality in surrogates, we implemented a linear surrogate model with elastic net regularization to mitigate potential overfitting issues. The details of the methodology and the empirical validations are presented in Online Appendix E.5.2. The effectiveness of this approach is confirmed by the consistency in long-term effect estimation shown in Figures E.7 and E.8 in the Online Appendix, compared with prior models, underscoring the robustness and predictive accuracy of our linear surrogate model with regularization.

5. Conclusions and Future Research

In this paper, we propose a longitudinal surrogate framework to estimate the long-term effects of long-term treatments using data collected from short-term experiments, which has remained an open challenge in the existing literature. We used two real-world long-term experiments conducted on WeChat to validate the effectiveness of our proposed framework. Our framework emphasizes the practical relevance of applying our method in real-world A/B testing scenarios, allowing practitioners to evaluate the effects of long-term product updates without incurring high costs and an extended waiting period. We discuss the limitations of our model in Section H.5 of the Online Appendix, by providing examples when our modeling assumptions do not hold. This serves as a cautionary note on when to apply our method in practice.

We outline several future research directions. One such direction is the integration of our concept of estimating future experimental effects with the existing literature on optimal stopping in A/B testing (Deng et al. 2016, Xiong et al. 2019, Berman and Van den Bulte 2022). Specifically, a valuable direction would be developing a method to optimally determine the parameter $T_E$, the experimental period duration. This approach would allow practitioners to conclude the experiment earlier, thereby directing toward the most beneficial treatment arm more efficiently. Second, it would be interesting to combine structural information, such as user behavior modeling, with estimating the effects of long-term treatments. In our current empirical study, we recognize that certain outcome variables, such as retention rates and subscription fees, may not show significant changes in the short term, because of factors such as data scarcity. Leveraging structural information may potentially improve the performance when the data sample is limited.