Strategy Experiments in Nonexperimental Settings: Challenges of Theory, Inference, and Persuasion in Business Strategy

1. Introduction

Theory building and experimentation have long been understood as central to the scientific process (Bacon 1889, Cook and Campbell 1977). Increasingly, these two engines of understanding have been proposed as being central to the development of effective business strategies as well (Felin and Zenger 2017, Camuffo et al. 2020). On the face of it, there is little to argue with such efforts. But scientific methods developed for laboratory investigations, and near-laboratory-like investigations such as field experiments, may not directly translate when experiments are conducted in the nonlaboratory settings that typify many strategy-making contexts.

We highlight two factors that distinguish experimenting with business strategies from other experimental interventions: nonrepeatability and joint action. First, unlike a laboratory experiment, “experimenting” with strategies often changes how the world—consumers, competitors, ecosystem partners—reacts to future actions. This potential impact on the broader business context creates an endogeneity that not only muddles the informational value of the experiment—will the same action–outcome relationship hold in the future—but, because a strategic experiment may not be neatly confined to an isolated treatment group, may change the cost of future experimentation as well. A failure of an experiment may not just update a company’s beliefs about the merit of an initiative; it can change latent customers, government regulators, and possible ecosystem partners’ beliefs about the efficacy of the broader enterprise and its management. In this latter sense, nonrepeatability is not merely the property that the same action may not generate the same result, but that other actions, including future experiments and even prior strategies, will yield different results than would have occurred in the absence of the experiment. Nonrepeatability has implications of irreversibility with respect to a strategy experiment and should therefore inform the way in which experiments with strategic initiatives are conceptualized, conducted, and interpreted.

Second, although some actions can be mandated by an individual decision maker, most strategic efforts require a coordinated joint effort, and hence some level of collective buy-in. A wide set of actors, both within the focal organization (the internal ecosystem) and across partner organizations (the external ecosystem), must collaborate in some fashion in order for the action to be realized (e.g., Adner 2017, 2021; Jacobides et al. 2018; Kapoor 2018). Thus, the pursuit of a strategic conjecture is not merely a matter of a focal individual’s interest or conviction, but requires joint agreement regarding the merit of carrying out the experiment, as well as its ex post interpretation. This need for joint action necessitates some degree of persuasion of a broader coalition in order for the experiment to be undertaken in the first place, and for effort to be continued in the face of the outcomes that are observed.

These two properties of strategic action, its nonrepeatability and its collective nature, make causal theories not only important in guiding action and learning from prior actions, but also critical to persuading others and coordinating joint action. To the extent that “strategic” initiatives are strategic because they have greater impact on the firm’s competitive environment and require greater degrees of collaboration, it is important to be careful as to how we conceptualize theory and experimentation for strategy making.

In the following section, we examine the alternative contexts for theorizing and experimentation that are characterized by the two dimensions of repeatability and individual versus joint action. In Section 3, we consider the interaction between these dimensions and the nature of strategic initiatives in each of the four quadrants characterized by these two dimensions, as well as the dynamics across these different contexts as initiatives and organizations develop. In Section 4, we examine the distinct challenges of inference and persuasion, and the role of theorizing, posed by the different contexts. Section 5 concludes with a discussion of what this perspective suggests both for strategists and for researchers of the strategy-making process.

2. Alternative Contexts for Theorizing

We characterize the context of theorizing along two different dimensions: (i) whether an experiment is repeatable or not and (ii) whether the experiment is conducted by, or at the behest of, an individual, or requires the taking of joint action by multiple independent actors (see Figure 1).

2.1. Repeatable vs. Nonrepeatable Experiments

The first key dimension is the degree to which experimental action has a repeatable quality or not. Repeatability means that the context that generates the relationship between actions and consequences is not impacted by prior actions. This property implies not only that can a given action be taken again, but that taking that same action would yield the same result or, in a more stochastic setting, that the result would be drawn from the same distribution of outcomes.¹ Further, it implies that the consequences of other possible actions will not be impacted by the experimental action.

The classic illustration of a setting of repeatable experiments is a scientific laboratory. A critical property of a laboratory is that it is decontextualized from the broader environment. Actions—experiments—are taken and their consequences are realized in the laboratory, and that realization is not contaminated by external considerations, factors that would make inferring the implications of the experiment less clear. One can consider such efforts as offline as opposed to online actions (Gavetti and Levinthal 2000). Although laboratory experiments are not acts of pure cognitive reasoning—physical experiments are run, chemical reagents are activated—the consequences have no direct implications outside of the laboratory. Such experiments have the important property that they are repeatable. If the experiment is run again, the same precise result may not ensue, but the “generating function” remains the same—the test environment remains the same, and a given act will correspond to the same distribution of outcome realizations. Further, the outcomes of other possible experiments will not be impacted by a prior experiment. The laboratory corresponds to the standard assumption of our statistical models of inference that the world is “iid”—independent events that are identically distributed—and comprise a setting for which standard techniques of inference and updating apply. Although knowledge of the world develops across experimental trials, the world itself is unchanged by these experiments.

Companies can also run near-laboratory experiments outside a laboratory setting. The A/B testing approach is premised on running a wide variety of experiments that explore the market reaction to a firm’s products without lasting impact on the “experimental subjects” or the broader user community. If one A/B trial was not promising, another one can be tried or the initial one can be tried again—the iid property holds and standard rules of inference and updating apply. Further, the experiment with B, does not impact the response of the world to some future experiments with C or D.

In contrast, nonrepeatable experiments have a different character in that the world becomes different as a consequence of the action. Coca-Cola can try an “experiment” and revise its formula for its flagship soda, and Facebook can change its name to Meta; however, these experiments impact their environments such that the outcomes of future experiments will be changed by the undertaking of the present experiment. As a result, the given experiment can never be rerun under the original set of conditions. Such experiments are nonrepeatable, and, indeed, even efforts toward reverting to the prior status quo, the original formula for the soda or corporate name, will yield different results. Similarly, a firm can “unacquire” a company it purchased, but not only is the divestiture itself not costless in a direct sense, the legacy business of the acquirer and the acquired are now different in meaningful ways (Feldman 2014). An important part of what makes strategic actions “strategic” is that they entail a degree of irreversibility, both as regards the firm itself as well as regarding the broader context in which the firm operates.² Further, the consequences of other actions, whether future possible initiatives or even the firm’s prior strategic actions, may now be different as a result of the experimental initiative as, for instance, credibility and reputation both within and external to the firm may be impacted.

Even initiatives that appear to have a more purely experimental quality may, depending on what enterprise is doing the experimenting, have very different implications. A fledging software company may introduce a new update to its small vanguard customer set that has some problematic bugs. Because of the firm’s modest status, those customers, and the vast latent set of potential customers that are not yet aware of the firm and its products, may be tolerant of these issues, and the firm would face no reputational consequences. However, if a major enterprise provider such as Microsoft or SAP engaged in a similar problematic update, the reputational consequences could be substantial.

This contingent perspective on whether the business context is impacted by a firm’s experimental initiatives has important implications for what might constitute a minimally viable product (MVP). The literature on lean startup has highlighted the role of minimally viable products as an efficient mechanism to ascertain product–market fit (Ries 2011, Blank 2020). The MVP is characterized as an offering that is sufficiently developed so as to elicit meaningful feedback as to the underlying merit of the initiative. However, if the introduction of a minimally viable product possibly taints the perception of latent customers and ecosystem partners as to the merit of the broader initiative and the enterprise behind that initiative, then future variants of product offerings will experience a different fate than if the MVP had not been introduced. This, in turn, suggests that the firm may wish to impose a higher bar than simply a product that can elicit feedback. Not only should an offering be sufficiently developed so as to elicit meaningful feedback as to its merit, it should be of sufficient quality so as not to damage the prospects of future offerings.

We can think of the endogeneity of the firm’s context with respect to the firm’s actions, even if those actions are intended as experiments, to be akin to the “observer effect” in physics whereby the act of observation (measurement) affects the entity that is being measured.³ As Shelef et al. (2024) note, in some instances, these observers are a firm’s competitors, who may be learning from the firm’s experimental efforts. But in contrast to the observer effect in physics, where the observations from a particular experiment may be impacted by the act of observation, in the strategy context, an experiment may impact the context in which all future observations will be made.

2.2. Individual vs. Joint Experiments

The second key dimension is the contrast between individual and joint experiments. In the former case, the challenge a given actor faces is persuading just themselves of what constitutes a more or less favorable initiative, whereas in the case of joint experiments, the challenge is one of collective persuasion. Individual learning may have a tacit quality (Polanyi 1962). Individual actors may develop a set of beliefs regarding preferred actions, as characterized in behavioral learning processes (Thorndike 1932). However, in a joint effort that involves convincing others as well as oneself, the exchange of beliefs and arguments must be explicit and compelling. Just as blueprints can be an important shared artifact to facilitate communication and coordination among actors with disparate bases of knowledge (Bechky 2003, Adner 2013), explicit theories can be an important basis for facilitating shared conversation and understanding around possible courses of strategic action (Adner 2021, Sørensen and Carroll 2021).

The need for joint action expands the role of theory from anchoring prediction to facilitating persuasion. In these settings, a coalition needs to be activated in order for the experiment to be undertaken in the first place, and for effort to be continued in the face of the outcomes that are observed. When experiments depend on collaborative agreement among multiple actors, the ability to undertake follow-on steps is dependent on the collective’s interpretation of the outcomes of the prior attempt. This highlights that a key factor influencing the value of learning in the context of joint action is the ability of actors to take the follow-on steps that are informed by the new knowledge acquired through the initial effort.

We note that the character of repeatability and joint action is, in some instances, a function of how the experiment is structured. In some settings, such as the protocol for testing new drugs, there is an up-front commitment to collect a certain number of data points prior to examining the data. Thus, the updating is not sequential, but rather a batch process. By structuring the commitment to continue data collection in this manner before attempting to draw a statistical inference, prior data in the “batch” do not contaminate subsequent data within the batch. For instance, there is no learning among clinicians administering the drug during a given phase of a clinical trial as to the efficacy of the treatment within the trial. As a result, there is no motivation to deviate from the initial protocol of the testing program, and potential experimental subjects are not influenced regarding their decision to participate. As a consequence, the data gathering within a batch is repeatable—providing treatment to the n + 1st subject is not impacted by the treatment of the nth subject. In contrast, when continued experimentation requires a joint renewal of commitment among participants, negative observations can serve to shut down further experiments, truncating further observations from which inferences could be drawn.

3. Quadrants of Strategizing

Given the dimensions of actors (individual or joint) and context (repeatable or not repeatable), there remains the question of how theorizing and experimentation might operate in the different regimes demarcated by their interaction. We first examine the four archetypes below. We then consider some of the natural dynamics as the act of strategy theorizing moves among these quadrants.

3.1. Individual Undertaking of Repeatable Experiments: Bench Scientists and Entrepreneurs

The upper left quadrant of Figure 1 (Quadrant 1) corresponds to the stylized imagery of what it means to engage in acts of theorizing and experimentation. The innovative scientist or creative entrepreneur ideates and comes up with a novel set of propositions. Further, the scientist or entrepreneur, as an empiricist, puts these ideas to the test. For instance, Edison had a belief about the existence of a filament that offered the basis for an improved electric light bulb. That belief guided extended experimental efforts to identify a filament with the appropriate qualities. Indeed, Edison’s theoretical belief was sufficiently strong that the extended series of failed experiments did not persuade him that his fundamental theoretical proposition was mistaken. Edison’s famous remark of ‘not having failed even once, but rather of finding 10,000 ways that didn’t work,’ is a statement not just of perseverance, but also of independent experiments—the failure of one effort impacted his beliefs about the properties of a particular candidate filament but did not change his beliefs as to the underlying behavior of electrons, nor did it change the outcomes that would be observed in any subsequent experiment. As a result of the iid property, the success of an experimental trial with a new filament was independent of the prior history. Moreover, as head of the laboratory, Edison wielded authority and did not face the challenge of persuading others to continue the experimental effort.

Many efforts at product development fit within this upper left cell. A novel product, whether a next generation of a product family, such as a new iPhone, or a new to the world products, often experience a substantial developmental journey well before being released to the market. Features are added and their technical feasibility assessed, speculations are made regarding the likely market reaction to those features, opinions are formed, perhaps informed by focus groups brought in to help provide insight as to these possible market reactions, but none of these efforts impact the true test environment of the marketplace. User interfaces can be experimented with, functions added or deleted, and the opportunity to change more features or to revert to a prior choice remains. Furthermore, none of those premarket additions and deletions impact the market’s reaction to the product that is ultimately released.

3.2. Joint Actions and Repeatable Experiments: Strategic Debate

Although Edison needed to mobilize a substantial team of scientists and engineers in pursuit of a viable light bulb, their participation on Edison’s terms was assured by their relative positions within the organization. In contrast, Edison held no such hierarchical authority in launching Pearl Street Station, the first commercial power plant in the United States. Running this first commercial experiment required permission, collaboration, and support from various government entities, suppliers, and financial investors over whom Edison had no control, but whose participation was essential for the effort. Persuading the coalition was essential first to execute the initial experiment, and then to interpret the results to enable the commitment of new resources to the next effort later that year—the Holborn Viaduct Station in London.

Although a firm can offer contracts and licensing arrangements in an effort to align interests among actors in an ecosystem, the beliefs of these other actors regarding the payoffs to those arrangements will influence their level of investment and effort in support of the proposed ecosystem. Thus, persuasion regarding the merit of the proposed strategy is a critical aspect of the alignment of interests.

Such considerations shift the context of theorizing from the upper left quadrant to the upper right quadrant (Quadrant 2) of Figure 1. As one moves from the stylized imagery of a lone bench scientist, the issue of persuasion and collective belief formation becomes important.⁴ Persuasion at the collective level can be challenging, even when evidence accumulates in controlled laboratory contexts. A cynical expression of this challenge is the aphorism that “science advances one funeral at a time.”

Even in the nominally hierarchical context of organizations, crossing boundaries in an internal ecosystem to drive collaboration requires active persuasion and management (Adner 2021). Although a superior can require that a certain strategic approach be carried out, whether that request results in some form of nominal, performative compliance or captures the hearts and minds of a managerial and technical team can have a significant impact on the ultimate fate of a given initiative. Karp (2023) points out that the need to persuade and motivate internal actors within an organization may even influence the nature of the experiment carried out with actors wishing to generate “experimental” results that are favorable to a particular interest within the organization.

Quadrant 2 of Figure 1 corresponds to much of the literature on strategy process. The literature on strategy process has many strands (Hutzschenreuter and Kleindienst 2006), including work that emphasizes organizational politics (Pettigrew 1985, Kaplan 2008), the hierarchical nature of organizations (Bower 1970, Burgelman 1983), and the role of uncertainty and speed of decision making (Eisenhardt and Bourgeois 1988), but central to this line of work is the collective nature of the strategic decision-making process. We note, however, that an implicit assumption in this stream is that the decision-making process largely takes place in the absence of strategic actions and is therefore reversible. The deliberation and testing of ideas take place in the boardroom and debates among managers, but the contestation is a contestation of rhetoric and persuasion. The primary focus is on the ex ante choice of strategy. There may be evidence marshalled as part of this process, but that evidence was not generated as a result of the strategic action under consideration.

Whereas the challenge of persuasion within the organization corresponds to work on strategy process, the challenge of persuasion beyond the boundaries of the organization links to work on ecosystems. When success requires an alignment of partners’ actions, alignment of partners’ perspectives as to the merit of the strategy of a focal firm among prospective ecosystem partners is needed. Although contractual arrangements set business terms and indicate agreements on payoff structures, they neither create nor signal an alignment of perspective. Offering a potential partner a large share of a joint payoff is compelling only to the extent that the potential partner is aligned in their perspective regarding the expectations of the payoff.

3.3. Individual Undertaking of Nonrepeatable “Experiments”

The shift from repeatability to a setting in which actions are nonrepeatable (from the top row to the bottom row of Figure 1) presents a critical distinction in the process of strategy theorizing and experimentation. Nonrepeatability has implications for the decision calculus regarding what action to take, as well as for inferences from the realized outcomes of those actions.

Exploration when actions engender some degree of nonrepeatability thus presents an alternative version of path dependence. Rather than viewing the learning processes as being akin to a multiarm bandit problem in which an actor may learn over the course of a series of trials the merit of the various opportunities that are present, a more apt image in many contexts is that of a set of branching trees (Levinthal 2021a). Once a strategy is taken, a new pathway opens up as a result of those actions. Furthermore, a strategic opportunity not seized at one point is often no longer available at a subsequent point: the world does not stand still. Other actors may fill a temporary void. Needs change. And affirmatively, a strategic action does not just provide information that validates or invalidates the merit of that action, it potentially positions the firm for a new pathway of opportunities—a new branch on an evolving, endogenously emerging set of opportunities, not merely new information about a fixed set of alternatives.

In a unitary actor setting (Quadrant 3), theorizing about the merit of a strategic initiative when the action itself influences the business context raises a number of key questions.⁵ For instance, one remedy for drawing stronger inference from experience is the randomization of the “treatment” with a “test” or “control” subsample. But as one moves from the realm of tactics (i.e., localized experiments in pricing and promotion) to the realm of strategy (i.e., nontrivial commitments to courses of action with some degree of nonrepeatability), the opportunity for distinct test and control groups diminishes. When a salient firm such as Apple introduces a new product, the world is its test group, and it is hard to imagine what might constitute a control. Along these lines, recent work (Johari et al. 2022, Bajari et al. 2023) has pointed to the “interference” effect when firms run A/B testing on platforms such as Airbnb, showing that promotional intervention on a test group can impact the available options for the set of individuals who are to constitute the control group, thereby undermining both experimental design and inference.

In this way, endogeneity is not just a feature of the experiment; it is also a feature of the experimenter. Large firms can afford to conduct a greater number of experiments because of their budgets, but they are much more constrained in their capacity to take reversible public acts, precisely because of their visibility and the saliency of the actions that they take. If the firm leaves its “footprint” on its competitive setting as a by-product of any “experiment,” and in that sense the business environment is endogenous to the firm’s actions, then firms need to be careful as to what is construed as the testing of a hypothesis of that experiment. What is being tested is not a prediction about the world as it is, for instance, the rate of descent of an object in a vacuum, but rather a speculation as to what the world will be as a result of a given strategic act.

Thus, in nonrepeatable settings, the hypothesis is not a prediction about some fixed, underlying truth; rather, it is a prediction about an idiosyncratic result subsequent to a specific action. Further insight gained from that experience may be suggestive of the implications of additional strategic actions, but those strategic actions will be distinct, and they will be enacted in a world that is now different in some material fashion. In such settings, the actors’ theories of the business situation become a critical connecting tissue between experience and inference. This makes initiatives in such settings markedly different from settings that are repeatable, in which actors can engage in model-free learning as in standard images of reinforcement, experienced-based learning. In settings of nonrepeatability, which effectively reduces one’s experience base, theorizing becomes even more central as a guide to action because past experience is an increasingly imperfect guide (March et al. 1991, Ehrig and Schmidt 2022). Further, these same properties make the testing of those theories and drawing inferences from observations of the consequences of actions even more challenging and problematic.

Not only is the world different for the focal firm, it is also different for other firms that might have considered a similar action (Shelef et al. 2024). The literature on vicarious learning (Miner and Haunschild 1995) points to how beliefs may diffuse among actors. However, that discussion tends to treat the world as if it is a fixed object over which there is some joint updating. But the success or failure of a focal firm’s actions may not only change the business environment in which the focal firm is operating, they may change the environment that others face.

3.4. Collective Undertakings of Nonrepeatable Acts: Joint Strategic Action

Learning from experience in a world that is endogenous with respect to the firm’s actions, and unstable because of the fragility of coalitions required for joint action, is not a process of possible convergence to some stable “truth.” Rather, updating is of the form that the prediction of the merit of the strategic action is sufficiently valid so as to provide a basis for continuing on the current trajectory, or, alternatively, that the prediction of merit is sufficiently wide of its mark that the strategic trajectory should be changed and possibly abandoned. Updating addresses the enterprise’s decision to continue, shift, or abandon a given strategic trajectory. This calculus is complex and is not simply a forecast about the external world.

Moreover, in a resource-constrained setting when joint action is required, it may be that even if repetition were possible in theory, it is not supportable in practice, as other initiatives may emerge that will argue for their own claims on resources. The coalition challenge shifts from building a coalition to support a new initiative to maintaining a coalition in support of an on-going, though possibly changing, initiative (Adner 2021). The competition for resources and psychological commitment between the hopes and aspirations for a new initiative and an ongoing initiative is complex (Levinthal and Wu 2024). The momentum around the latter might favor continuation, but the appeal of shifting attention and resources to a “new shiny object,” particularly when the ongoing initiative has hit some bumps in the road, can be very strong.

When collective support is required for an effort (Quadrant 4), the need for persuasion applies not only to the garnering of support inside the firm, but also external actors, such as investors, potential ecosystem partners, and potential consumers. To some degree, this collective learning can be viewed as rational updating with regard to a common business proposition, but it also is the case that the business proposition itself is now different because of the failure. The challenge of persuasion is now more difficult, independent of the “truth” of the merit of the initial underlying “theory.”

In this light, the notion of a pivot (e.g., Kirtley and O’Mahony 2023) does not just impose an opportunity cost of foregoing a prior course of action; it requires retaining and possibly rebuilding a shared sense of purpose. This is not an agency problem of incentive alignment, but rather one of motivation: Does the collective of necessary actors within and across the collaborating organizations commit their time, energy, and ideally heart and soul, to this initiative? A pivot is not simply a midcourse correction in some individual strategic journey, but a collective commitment to this new course of action. Observing an executed pivot is thus observing a collective that has overcome a crisis of confidence, as the initial prediction of what makes success has proven false, and a collective set of actors who have converged on a new and sufficiently compelling prediction of success along a different path to catalyze a renewed commitment of effort and resources. Framed this way, a pivot is a new effort derived by modifying an initial theory, with the modification itself suggested by the failure of the initial experiment to deliver expected results. In the context of experimenting with strategies, however, the new experiment will be conducted in a world that was itself changed by the initial effort.

Indeed, a prominent failure may even diminish resources available for other actors who might have tried a variant of the same initiative. Thus, for example, Pfizer’s failure with inhalable insulin, Project Betterplace’s failed battery-switching model, or Michelin’s failed effort at repairable run flat tires made any efforts at creating ecosystems around similar business models far more challenging (Adner 2013).

Of course, there can be positive demonstration effects as well that are, again, both updating events about the world and also actions that change the business context. Just as an unsuccessful initiative can cause latent investors, ecosystem partners, and customers to shy away from similar initiatives, a successful initiative can act as a business model attractor. At the level of the focal enterprise, we observe positive fly wheels (Collins 2001) whereby positive feedback makes the enterprise a stronger competitor. We can also observe positive feedback at the population level (Arthur 1989). Indeed, central to population ecology arguments on industry dynamics is how the legitimacy for the new industry can develop subsequent to initial entry (Hannan and Freeman 1989). Regulatory structures that may develop, customer awareness, and the development of supply chains all make the business context more hospitable for “second movers.” Alternatively, later movers can face negative externalities that may be generated by early movers through preemption effects and, more generally, the diminishing of the attractiveness of subsequent entry via the emergence of entry and mobility barriers (Porter 1980). In the context of these competitive effects, we again face the challenge of generalizing from the experience of, and in particular the performance of, early movers for latter movers.

3.5. Development of the Enterprise and the Shifting Landscape of Experimentation and Theory Building

Although we have treated the four quadrants (Figure 1) as demarcating independent elements of a “landscape” of experimentation, enterprises’ strategic challenges often transverse this landscape over time. An initial incipient entrepreneurial act may constitute an individual initiative. However, even this initiative is typically not ultimately an ex situ laboratory exercise. Schumpeterian recombinations are not laboratory experiments with a high degree of repeatability, but rather a shift to the world of commercial activity of, what are often, nonrepeatable actions. At the same time, efforts at prototyping and establishing feasibility may more proximate a laboratory effort and have the quality of repeatability.

Few businesses are unitary, solo enterprises. Other actors need to be persuaded to join if the firm is to realize its ambitions, and even more so if the effort is to achieve scale. Firms’ effectiveness often hinges on the engagement and alignment of efforts of various ecosystem partners. Thus, there is a tendency to shift from initial, possibly individual, ideation efforts downward to the row of joint action. Persuasion and some degree of shared understanding become critical.

An implication of this dynamic is that the opportunities for a firm to engage in stealth-like behavior and to be able to more closely approximate highly reversible laboratory-like conditions diminish with the firm’s growth and salience. Thus, there is a natural drift down and right across the quadrants in Figure 1 to the cell corresponding to the challenge of collective strategizing with strategic consequence—a degree of nonrepeatability. Although the imagery of theorizing and experimenting is animated by classic images of bench scientists and stylized models of learning such as multiarmed bandits, the more established enterprise is squarely in the setting of collective theorizing and experimenting with irreversible consequences.

4. Inference in a Shifting Landscape: Experimentation and Theory Building

The problem of inference varies markedly across the different settings of individual versus joint action and when actions are repeatable versus nonrepeatable and have irreversible consequences. Learning when actions are repeatable is the classical experimental setting of the laboratory. The experiment is run, the test enzyme achieved a certain biochemical reaction or not, a randomized set of human subjects reacted to test stimuli, and so on. The inference problem is a straightforward one of statistical method (Fisher 1925). Further, in nonlaboratory settings, one might take a Bayesian approach to assessing the merit of a business plan; for instance, sales efforts could be made with respect to an initial set of customers to test the hypothesis regarding the appeal of the product or service offering. Critically, the “engine” of Bayesian updating of prior beliefs assumes that the likelihood function—the probabilistic relationship between actions and outcomes—is stable. This property corresponds to quadrant 1 of Figure 1 of the “lab bench experimentation.”

However, when actions are not repeatable—when an initial action changes the likelihood function associated with future actions—then the Bayesian apparatus is no longer directly applicable. What exactly is one learning when the outcome of an action in period t is observed for the prospects of that action in period t + 1 when the world in t + 1 is now different as a result of that action? One might, for instance, have a higher-order belief that positive outcomes will be amplified. If the initial incarnation of the strategic effort was somewhat successful, then subsequent efforts will be even more successful because of a positive cascade of the willingness of ecosystem partners to participate, greater consumer acceptance, and the like. One might also have beliefs about the power and possibilities of “corrective learning”, a moderate failure can be transformed to a success by drawing on the lessons of that failure. More generally, the inference problem shifts from being one of statistical inference about a fixed world to speculation about “equations of motion” and dynamics. Clearly, these predictions about dynamics based on observations of prior outcomes are a much greater challenge than refining one’s prediction about a fixed environment.

The extent to which inference made in a prior context carries over to a new distinct context is an active area of research within the artificial intelligence community, when considering problem environments in which the “training” environment is distinct from the test environment (Belkin et al. 2019). In the artificial intelligence setting, the change in the environment is typically treated as exogenous—the computer had been playing one board game and now is facing another, or a medical imaging algorithm trained on one demographic of patients and now is faced with another. In the context of the machine learning literature, this issue of the relationship between the training data set and the test data in which the algorithm will operate has led to discussions regarding the dangers of possibly over-fitting the algorithm to the training data set and the robustness of more parsimonious algorithms (Hastie et al. 2001). In contrast, in the strategy context, this robustness may come from more general theories of the “training” data set (the experiment and prior history), rather than the development of specific hypotheses regarding this training data.

The strategist’s problem of generalizing across contexts requires an assessment of whether lessons learned from a prior cycle of action and outcomes will carry forward to another iteration of that cycle or how that response function might differ going forward (Choi and Levinthal 2023). The market may have responded favorably to a new product introduction or the firm’s existing products, will the market continue to respond favorably in view of competitor’s reactions, technological change, and possible new entry?

When the new context is endogenous, strategizing about appropriate actions should include not only the informational value the actions might generate, but also their implications for the “state” of the firm and its environment. This is a vastly more complex problem. As Bellman (1961) noted in reflecting on the inherent limits of dynamic programming, such future oriented conjectures suffer from the “curse of dimensionality,” characterized by the explosive growth of the state space of action–outcome relationships as one considers such problems in settings with an array of alternative actions over moderate to long-run time horizons.

4.1. The Role of Theory in Linking Prediction and Persuasion

When actions are not repeatable, the role of prediction and theorizing becomes more important precisely because the process becomes less amendable to iterative experimentation. Theories of strategies and strategic consequence provide a basis for reflection prior to action. And this ex ante reflection, guided by one’s theory of the business context, allows one to “test” one’s ideas with one’s mental model of the world, rather than by the outcome of actions taken in the world (Gavetti and Levinthal 2000). In contrast, when actions or experiments are repeatable, one can accumulate a large experience base with a given action and rely on the tools of statistical inference to judge the merit of a given action and update beliefs accordingly. Indeed, even model-free processes of reinforcement learning can prove effective in such settings.

The nature of the prediction challenge also speaks to the ultimate goal of the “experimental” process. Whereas scientists, both natural and social, are often motivated to ascertain the validity of a particular theory or hypothesis, managers have a more pragmatic motivation to ascertain what actions move them forward most effectively in the pursuit of profitability (Zellweger and Zenger 2023). The pursuit of scientific truth versus profits suggests different decision criteria. Being highly exploratory will lead to greater knowledge of one’s setting, particularly if that setting is exogenous to those exploratory actions. Exploration in the pursuit of profits, however, is likely to be much more circumscribed as the investment required for taking each step presents direct and opportunity costs.

When joint action is required, theory plays a different critical role in its generation of predictions, which links to the process of collective persuasion. Here, the issue is not simply a focal actor’s belief about a landscape of possibilities, but what promising actions can the focal actor persuade others of. There may be an extraordinary peak in a fitness landscape, but if that peak is only in the mind’s eye of the focal actor and not of critical partners, then it is not obtainable. Organizations face a dual problem of search and joint action (Kocak et al. 2023).

When the setting shifts to a joint activity, even if repeatable, the inference problem becomes more complex. For instance, did the product development effort fail to achieve a working prototype because of the inherent engineering challenge, or did it fail to achieve its desired goals because of a lack of effort and insight on the part of the participants. This resulting attribution problem is similar to that posed by Adner and Levinthal (2004a, b) in the context of real options when the outcome of pursuing these options is endogenous with respect to the effort and insights of the individuals involved. The resolution of uncertainty in the context of real options or experimenting does not occur as the result of passive observation, but rather through action. This endogeneity of the outcome, in turn, makes inference as to the underlying merit of the initiative problematic.

Theories and the discourse around them are critical in the process of collective persuasion. Furthermore, persuasion is critical to effective action. If the outcome of an action is exogenous—individuals vote on a preferred action and the distribution of outcomes from that action is only contingent on what choice is made—then persuasion plays an important role in guiding the collective to a possibly preferred action. However, if the outcome is endogenous to the energy, creativity, and commitment of those involved in executing it, then persuasion becomes even more critical. Does the “theory of the case” sufficiently convince actors of its merit that they give their hearts and minds and not just performative compliance?

To take a salient recent example of this contrast between inference informing the knowledge of focal actors and collective persuasion, consider the development of a vaccine for the COVID virus. Bench scientists tested the new vaccines and convinced themselves and regulators, such as the U.S. Food and Drug Administration, of its merit. However, the speed of the discovery effort, and its use of novel processes of mRNA vaccines, ended up as fodder for different groups to feed skepticism about the value and merit of the vaccine initiative on the part of vast swaths of the potential patient population. The intensity of the subsequent reactions changed the public health landscape—both feeding conspiracy theories and (hopefully) changing future launch and communications strategies for all future efforts along these lines. Clearly the emergence of skepticism did not itself change the true biological merit of the vaccines; but, that skepticism did change the environment into which the effort was launched and, in so doing, changed the impact of the vaccine and reshaped (for the worse) its contribution to social welfare (Adner 2024). Initiatives in Quadrant 4 of Figure 1 highlight the simultaneous need to manage the challenge of persuasion in addition to the problem of scientific testing of merit.

5. Discussion

Experimenting in the strategy context has many of the surface features of our classic notions of the scientific method regarding theorizing and learning about the merit of theories. Conjectures are made, and those conjectures, and their underlying bases, are revised in light of new evidence. However, it is important to look beyond the surface similarities and consider some of the key, distinct properties of theorizing in the strategy context. We have highlighted two of those properties: the typically endogenous, nonrepeatable, nature of the contexts in which strategic action is taken and the fact that strategic initiatives are typically joint acts and therefore require some degree of persuasion as to the merit of those acts.

The strategy theorist makes reasoned conjectures and works to convince others of those conjectures. Further, the strategy theorist makes commitments not just conjectures. The initial theories and actions may change course subject to new evidence, but that initial course of action is not typically an experiment if we take the construct of an “experiment” as corresponding to situations that lend themselves to testing and retesting cases and controls. Experience should not be conflated with the much more specific construct of experiments. The challenge for management scholars and practicing managers is to consider how these collective experiences, not experiments in the sense of potentially repeatable actions, may inform the revision of strategic action (Levinthal 2017, 2021b). An analytic deconstruction of the challenge of theorizing in the context of strategic initiatives can provide some contribution to that effort. Theorizing and experiential learning about such theories in the strategy context, is, we suggest, both more important and more challenging than previously recognized.

When the structure of experience shifts from being independent and identically distributed to being strongly path dependent, our standard engines of statistical inference no longer readily serve to answer the question as to what should constitute forward looking conjectures and beliefs. Robust conceptual arguments regarding the desirability of alternative strategic actions are necessary to guide such actions. Further, the firm’s business models, cognitive schemas, or theories need to be subject to change, both because the initial conception may not be apt and because the world itself is changing. Thinking about possible worlds that actions may engender, and revising that thinking in lieu of experience, becomes critical to strategic decision making. We are not well served by treating that process as if it is a strong analogue to the exercise of a classic laboratory experiment. Rather than being guided by the analogue to the scientific method in the laboratory context, strategy theorists need to engage in the difficult challenge of individuals and broader collectives learning in settings that are shaped in part by their actions and where the “arrow of time” offers no opportunity for replications or controls.