Algorithm Supported Induction for Building Theory: How Can We Use Prediction Models to Theorize?

Introduction

Building theory from specific data involves proposing general processes, features, and relationships between means and ends around a particular phenomenon. Such theory building occupies a central role in the organizational sciences, as it does in any science that gives the explanation of phenomena at least as much importance as the testing of deductively derived implications from axioms (Glaser and Strauss 1967, Lave and March 1993, Deetz 1996, Walsh et al. 2015, Bamberger 2018). However, explicit theory building from data in management and organization research has traditionally been reserved for researchers working with small numbers of cases rather than for those working with large N data. Moreover, the norms for presenting quantitative papers often involve proposing theoretically derived hypotheses and then testing them, and this may sometimes obscure, de-emphasize, and delegitimize inductive reasoning, even if it plays an important role in generating the results (Gelman and Loken 2014, Goldfarb and King 2016).

In this paper, we argue that machine learning (ML) represents a useful new methodology to enable theory building from data for organization scholars working with large samples of data. ML can facilitate algorithm-supported induction, yielding interpretable conclusions about patterns in data that are likely to be robustly replicable by other analysts and in other samples from the same population. This can be accomplished, as we demonstrate, with well-established ML algorithms that are neither new to the world, nor new to the field of ML. What we consider new, and more importantly, valuable to the field of organization science, is our approach to using ML for the specific purpose of algorithm supported induction as a step in the process of building theory from data.

To fully understand the role of algorithm-supported induction in theory building, it is useful to first review different forms of reasoning. Deduction of theoretical implications from known axioms is at the heart of hypothesis testing research (Popper 1959). In contrast, induction of a pattern from the data and abduction of an explanation for the pattern are core to the theory-building process (Peirce 1878, Bamberger 2018, Behfar and Okhuysen 2018). Although ML plays a key role in induction by revealing robust patterns, it does not (yet) offer a solution to conducting abduction. As of today, what Henry Mintzberg stated in 1979 still holds true: “The data do not generate the theory—only researchers do that—any more than the theory can be proved true in terms of the data” (Mintzberg 1979).

What algorithm-supported induction indeed offers is the detection of complex but interpretable patterns in data in a robust and replicable manner. By interpretable, we mean ML algorithms can be tuned to detect complexity in patterns that would not always be intuitive for humans to identify but intuitive enough to understand once found. By robust and replicable, we mean that ML algorithms (a) incorporate procedures that avoid overfitting (i.e., avoid producing results that are highly idiosyncratic to the observed sample) and (b) applies procedures—including inevitable judgement calls—that are codifiable, thereby enabling replication. However, other fundamental steps in the theory-building process that precede and follow the generation of these robust patterns or stylized facts (Helfat 2007), such as conceptualization (defining the constructs of interest), measurement (selecting or developing measures for the constructs), and explanation (theorizing about the relationships among the observed patterns), remain largely human prerogatives.

To be sure, the applications of ML in management and organization studies are broader than algorithm-supported induction (Tonidandel et al. 2018, von Krogh 2018, Shrestha et al. 2019). For instance, by using a small amount of hand-coded data as the training set, algorithms can learn the patterns implicit in this coding and predict the coding for a much larger data set (Medlock and Briscoe 2007; Crowston et al. 2010, 2012; Yan et al. 2014; Christensen et al. 2017). ML-based data text analytic approaches such as Latent Dirichlet Allocation (LDA) are becoming popular to discover themes and trends in a large collection of documents (Blei 2012, Bao and Datta 2014, Puranam et al. 2017, Huang et al. 2018, Hannigan et al. 2019). In economics, researchers have introduced ML techniques in conjunction with instrumental variable analysis (which requires prediction accuracy in stage-one models) to improve causal inferences (Belloni et al. 2013). ML techniques can also be used as an alternative to propensity score matching (Varian 2016). Another application in economics pertains to estimating heterogeneity in causal effects (Athey and Imbens 2016). Mullainathan and Spiess (2017) provide an overview of ML applications in economics (Varian 2014, Kleinberg et al. 2015).

Although the applications noted previously are all worthy of consideration for more widespread use in our field, our perspective focuses on the application of ML techniques as a tool for building theory from data, which plays a central and perhaps unique role in management and organizations research (Leonard-Barton 1990, March et al. 1991, Sutton 1997, Eastman and Bailey 1998, Burton and Obel 2011).

In the rest of this paper, we propose and illustrate a procedure to conduct algorithm-supported induction. First, we provide a concise and accessible introduction to the core logic of ML principles, with details available in Appendix A for the interested reader. Second, we offer a perspective on how management and organization researchers can conduct algorithm-supported induction by using ML’s core analytical property—detecting robust and replicable patterns of tunable complexity (i.e., researchers can adjust the degree of acceptable complexity in the patterns uncovered). Third, we illustrate (with a simple example using simulated data) how a researcher can apply the technique to build and test theory in the same data, without running the risks of overfitting and low replicability of results. Our illustration unveils the tradeoff between comprehensibility and predictive accuracy that theory building from data inevitably entails and suggests how one might approach this tradeoff in the research process.

What Do ML Algorithms Do?

In this section, we give a brief overview of the basic features of ML methods. Readers familiar with the techniques can skip directly to the next section that describes how we propose to apply it to theory building. ML is a subdomain within the field of artificial intelligence (AI). It endows computers with “the ability to learn without being explicitly programmed” (Samuel 1959, p. 120). Mitchell (1997, p. 2) provided the classic statement about the components of a learning problem that ML algorithms can tackle: “A computer program is said to learn [effectively; author’s note] from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.” Suppose that task T is that we want to predict the profits of a firm in the current year and future years. We want to do this based on the data from past years’ experience (E). Experience may comprise data on past profits with a set of predictors such as price and sales changes, chief executive officer (CEO) characteristics, organization design, planned expenditures, demand trends, liquidity position, or any of the factors that researchers believe are associated with firm profitability. The performance of the algorithm (P) is considered satisfactory if it can predict a firm’s profits accurately. Hence, with increasing data from experience, the algorithm should behave in such a way that the difference between the actual and predicted profit is minimized.

Broadly, two classes of algorithms have been developed to tackle such problems involving learning from existing data: supervised and unsupervised learning.¹ Supervised learning involves learning what associates with a given outcome. In the terms of the learning problem discussed above, the goal of a supervised learning algorithm is to make good predictions about future profits (future Ys) based on knowing only the future Xs. This is accomplished by fitting a model that does a good job of predicting past Ys based on past Xs and then assuming that the same relationship between Xs and Ys will hold in the future. The fitted model captures this insight in the form of a function that maps predictors to profit.

Of course, we could also obtain such a prediction using familiar ordinary least squares (OLS) regression. Through techniques such as stepwise regression, we can also find the best fitting model (i.e., linear weighted combination of Xs) that predicts Y in the past data and then use this model to make predictions about future Ys based on future Xs. However, ML algorithms combine two features in a useful manner to improve on such techniques: tunable complexity of functional form and improved protection against overfitting (Abu-Mostafa et al. 2012).

First, with ML, complex functional forms (e.g., higher-order and interaction terms among the Xs) can be incorporated without necessarily having to be specified in detail in advance. For instance, random forest models allow for fitting a hierarchical interaction structure, and neural networks can help fit arbitrarily complex polynomials (Breiman 2001, LeCun et al. 2015). The resulting models can achieve higher levels of fit in the data, which hopefully leads to better predictions in the future. This complexity in functional form may also make the models harder to interpret. For this reason, a feature of ML algorithms that may prove extremely useful to the inductive and abductive theorist is that we can, with many model families, tune the extent of complexity we are willing to tolerate: second-order but not third-order polynomials, or 4 versus 40 coefficients, for instance. We can start with a model family that can accommodate considerable complexity but then use the data to fit a model that can be of considerably lower actual level of complexity. This tuning is done through model hyperparameters, and we can tune these both algorithmically (for maximizing predictive accuracy given the model family) and manually (for interpretability). (Technical details can be found in Appendix A).

Second, ML algorithms enlarge the set of procedures that we commonly use in traditional statistical models to guard against overfitting. For example, if we were to build a well-fitting OLS model by selectively adding or dropping variables to find significant effects, we would run two related risks of overfitting: (a) excessive model complexity—the realized R² may be high simply because we have too many parameters in the model, and the model must be penalized for this to enable comparison with other models; and (b) excessive sample dependence—including cherry-picked variables can produce a model that may fit the particular sample of data but may not be generalized beyond the data at hand. Both pose fundamental challenges to the validity of the results for understanding future samples. ML algorithms combine an automated model building process (which allows us to find acceptably complex functional forms) with sophisticated procedures for mitigating both types of overfitting. The details of these procedures, namely regularization and cross-validation, can also be found in Appendix A.

Unsupervised learning algorithms, as the name suggests, operate in the absence of a supervisor variable. The data (E) lack any specific target outputs (i.e., Y) associated with each input. These algorithms are generally tasked with detecting patterns of association between groups of X variables, without any particular variable being selected as the dependent variable. Statistical clustering is a canonical example of unsupervised learning with which most management scholars are already familiar. Its purpose is to partition cases into subsets such that similar cases are in the same cluster and dissimilar cases are in different clusters. In our profit prediction example, unsupervised learning can help users find a cluster of firms that are similar to one another on observed dimensions such as CEO attributes, demand trends, and profitability. Strategic group analysis (Harrigan 1985) is a well-established methodology in strategy where studies have frequently used statistical cluster analysis. Unsupervised ML techniques use the same basic logic but provide more flexibility in terms of choosing different types of algorithms to perform the clustering, followed again by procedures such as regularization and cross-validation to prevent overfitting. The relevant algorithms include K-means, hierarchical, and spectral clustering, all of which largely share a similar intuition. LDA is a powerful suite of unsupervised learning algorithms that detect clusters of topics in a corpus of text. It is already extensively in use by organizations researchers (Hannigan et al. 2018).

It is important to highlight that often the same models can be used either for traditional hypothesis testing (i.e., inferential statistics) or for ML (i.e., statistical learning). For instance, OLS regression (Bishop 2006, p. 140; Robert 2014, p. 217; Shalev-Shwartz and Ben-David 2014, p. 123) and logistic regression (Bishop 2006, p. 205; Robert 2014, p. 245; Shalev-Shwartz and Ben-David 2014, p. 126) can both be used for supervised ML, and principal component analysis is a standard tool for unsupervised ML (as well as a standard tool in conventional multivariate analysis) (Shalev-Shwartz and Ben-David 2014, p. 324). The difference between ML and traditional inferential statistics is not rooted in the models per se (although ML in general has many additional model families that allow for more nonlinearity), but in (1) objectives for using them and (2) their accompanying assumptions.

First, inferential statistics are primarily used for testing hypotheses derived from a priori theories, whereas statistical learning, as the name suggests, is used to learn the patterns in data. Second, in inferential statistics it is necessary to assume the distribution of partial density functions of data in order to draw conclusions about statistical significance and potential confidence intervals. In ML, that assumption is not necessary because the goal is predictive accuracy and not inference (Bzdok 2017).

In sum, it is crude but accurate to think of ML algorithms (both supervised and unsupervised) as enabling prediction through searching in data for complex, robust, and replicable associations—complex associations between variables that are unlikely to be the result of sample idiosyncrasy and can be rediscovered by anybody using the same procedures. The complexity and robustness in associations produced by ML algorithms result from procedures that allow (tunably) complex models to fit the data (reducing bias in prediction) while also mitigating against overfitting (reducing variance in predictions). A fundamental theorem in machine learning research pertains to this tradeoff between bias and variance (see Appendix A), and most ML algorithms explicitly aim to optimize this tradeoff to maximize predictive accuracy given the constraints inherent in a model family.² Replicability, on the other hand, is a property of the algorithmic nature of the process.

However, before we explain how ML techniques can aid theory building from data, three caveats must be stated. First, ML methods are not by themselves a substitute for randomization to obtain causal inference. All ML methods are associative, although they can play an important role in enabling causal inference under the assumption that there are no omitted variables that can create spurious relationships (Pearl 2000; Davis and Heller 2017, p. 548; Athey et al. 2019, p. 20). Second, underlying all ML methods is an assumption that the future can be predicted from the past (more technically, an assumption that probability distributions for the relevant variables remain stationary over time). Hence, current ML algorithms work best for relatively stable phenomena (e.g., in the example above, firm membership in strategic groups and within-group collective behavior is likely to be stable over some extended period of time given strategic and organizational inertia, strategic group mobility barriers, etc.) Third, ML techniques are not geared toward testing an explanation through inferences about the relationships between variables; instead, they focus primarily on prediction (y hat not beta, to use econometric terminology, as noted by Mullainathan and Spiess 2017).

How ML Can Aid Theorizing from Data

We propose that ML algorithms can play a powerful role in aiding theorizing from data by providing a key ingredient necessary for it—robust patterns in data. Such patterns form the basis for prediction by these algorithms, but at the same time they can also be treated as a robust stylized fact to be explained through theorizing and in turn replicated in additional data (Helfat 2007). This stylized fact can then become the target of explanation by the theorist. We see three advantages to separating the process of theorizing into pattern detection (primarily algorithmic and aided by ML techniques) and pattern explanation (primarily nonalgorithmic and driven by human researchers).

First, algorithmic pattern detection has high intersubject reliability. An algorithm used by different individuals will still yield (highly) similar results. This does not preclude the need to make judgement calls in applying algorithms (e.g., which learning algorithm to apply to the data to obtain high predictive accuracy while keeping results interpretable). Our point, however, is that these can be made in a structured, traceable manner, which enables replicability and assessment of robustness. Even for algorithms that involve random components, reproducibility of results can be guaranteed when one keeps track of underlying random seeds used to start off a random number generator. Furthermore, in conventional inductive research methods, where pattern detection and interpretation may occur simultaneously, it is recognized that researchers’ confirmatory biases and motivated information processing could potentially lead to false positives or false negatives (e.g., seeing patterns that are explainable in an intuitive or interesting manner but not seeing others that are less intuitive).³ Analytically separating these steps and performing the first algorithmically enhances replicability. This is not necessarily an advantage if the goal is to enhance creative variation of interpretation and gain novel insights, but it is an advantage if we seek to enhance the reproducibility of an inductive inference.

Second, pattern detection through ML algorithms need not have default human comprehension constraints. For instance, we doubt that any management theorist would hold an entrenched view about the key relationships in their models being linear, although linear models tend to be our workhorse for summarizing multivariate relationships. The advantage of assuming linearity is interpretability, and we willingly give up (some) predictive accuracy for it. If interpretability were not important, we would simply optimize predictive accuracy, but if the goal is to build human comprehensible explanations (i.e., theory), interpretability is key. An advantage with ML algorithms is that the researcher can tune the complexity of patterns the algorithms will detect, depending on his or her goals, taking a more nuanced approach to the tradeoff between predictive accuracy and interpretability rather than just maximizing one or the other. Suppose that a researcher’s appetite for complexity is limited to no more than the second-order interactions with concave functions and no more than a few coefficients overall, ML algorithms can then be set up to find a predictive model that is least likely to over or underfit the data subject to these constraints.

Third, ML algorithms offer protection against results that are highly idiosyncratic to a sample (overfitting). If the central problem in deductive theory testing is spuriousness (i.e., omitted variables that provide alternative explanations), the central problem in theorizing from data are overfitting (i.e., patterns that do not generalize to other samples). However, it has not been given nearly as much attention as the problem of spuriousness, for which we have available a suite of statistical techniques (e.g., instrumental variables, matching, regression discontinuity designs, and ideally, of course, randomization). ML algorithms come equipped with procedures such as regularization and cross-validation that help mitigate the overfitting problem and aid the researcher in detecting (if they exist) reliable associations that replicate across subsamples of data. This is an advantage if we wish to build generalizable theory that can predict out of sample from our inductive and abductive efforts. It is irrelevant if we only seek to generalize to theory based on one or a few cases (Yin 2009). However, to the extent researchers want their theories to be applicable in other samples, overfitting their original sample is always an important concern.

In sum, our central argument is that ML algorithms selected with interpretability in mind can play a useful role in generating robust patterns that are an input to theory building from data. We develop this argument in detail by discussing the elements of algorithm supported induction to aid theory building.

A Procedure to Conduct Algorithm-Supported Induction

We propose a procedure for algorithm-supported induction in Table 1. This procedure we propose can be helpful in creating theory from data by establishing robust stylized facts as an input to abductive theory creation and creating a separate hold out sample for testing the theory without running the risk of overfitting. Although our current description relies on ML algorithms⁴ in stage 2, the entire procedure itself can be thought of as a meta-algorithm for inductive theorizing and testing of the constructed theory with large sample data.

Table 1. Algorithm-Supported Induction: Overview

In stage 1, we split the data randomly into samples I and II. Sample I is reserved for pattern detection, and Sample II is the hold-out sample (Goldfarb and King 2016). We use sample I to search for interpretable robust patterns in the data (Locke 2015), which we explain by constructing a theory and use sample II to test the hypotheses derived from the theory. If the original full sample is representative of the population, a random subsample of it also will be representative of the population. If the original sample is not representative of the population, any inference procedure (including conventional hypothesis testing) is futile anyway. The procedure we advocate does not therefore raises any new constraints.

Beyond the idea of a hold-out sample, however, algorithm-supported induction also adds some key ingredients. The search for patterns when aided by ML can allow for tunable complexity. Our goal is finding robust and comprehensible associations, not necessarily maximizing predictive accuracy (which is the case in more typical ML applications). This is why stage 2 has two parts. In effect we do two-stage feature selection to generate interpretable patterns. In stage 2.1, we find important features in a predictive model. Using an algorithm that can capture complex functional form without the need to be interpretable, a small set of most important features that contribute the most to predictive accuracy is identified. From the identified features, in stage 2.2, we construct a predictive model from a model family that is relatively easy to interpret.

For instance, in a supervised learning exercise, we might use a random forest or a neural network model in stage 2.1 to give us the most important features that predict an outcome of interest. In stage 2.2, a low degree polynomial (with the degree decided by the researcher, typically of degree 1, linear, or degree 2, two-way interactions and quadratics) constructed from the important variables identified in stage 2.1 can then be put through a second stage of feature selection using an easy to interpret model, such as LASSO (least absolute shrinkage and selection operator) or RIDGE regression.⁵ Critically, in both stages 2.1 and 2.2, we use cross-validation for hyperparameter tuning (a process that produces a model that is optimized for predictive accuracy given constraints of the model family). In addition, in stage 2.2, we conduct subsample (or bootstrap sample) replication to ensure that the patterns (i.e., the specific associations between variables) we end with are robust to sampling error.

This two-stage feature selection process aims to balance predictive accuracy of the stylized patterns obtained in the data, with interpretability, which is critical if these patterns are to be useful as inputs to abductive theorizing. As shown in Figure 1 (left), the optimal complexity of a ML model estimated from the data and to be used for prediction will optimize the bias-variance tradeoff subject to the constraints of the model family. A particular model (i.e., the set of nonzero coefficients in an OLS model if the model family used is OLS) achieves the lowest distance between training set error (shown by small dotted lines) and test set error (shown by longer dotted lines), indicating that the model does not either overfit or underfit the data (Bishop 2006, Abu-Mostafa et al. 2012, Alpaydin 2014). Such an optimized model is not necessarily a perfect representation of the underlying data generation process (because we have assumed a model family, OLS with its linearity in parameters assumption, which may or may not correspond to the underlying data generation process of a phenomenon) but rather one that fits the data as best as possible (i.e., neither over- nor underfits) given the constraints of the model family (e.g., the general functional form of an OLS regression in this case).

In organization science, validity may be conceptualized in terms of correspondence to the underlying data generation process (Shadish et al. 2002). A valid model in this sense is one that not only optimizes predictive accuracy (optimizes error in training and test data) for a given model family but also has the lowest absolute values of prediction error possible. Minimizing prediction error through optimizing the bias-variance tradeoff is therefore a precondition to having the lowest absolute prediction error. This implies that if the prediction error is not even optimized within a model family, then we know it definitely cannot be at its global minimum (i.e., the model cannot be a valid model).

This leads to the critical interpretability/prediction error tradeoff: as we impose tighter restrictions on interpretability, prediction accuracy will necessarily decline. For our purposes, we think of interpretability simply as researcher’s ability to understand and explain the results of a model to each other.⁶ To illustrate why interpretability may not come for free, we can superimpose a difficulty of interpretation function over the bias-variance tradeoff reported in Figure 1 (right). Assuming that the interpretation difficulty function is increasing more rapidly in complexity than the variance in predictions it follows that (a) the best interpretable model likely underfits the data—it is therefore likely to have lower prediction accuracy than a model that optimizes the bias-variance tradeoff—and (b) models optimizing the bias-variance tradeoff may have high interpretation difficulty.

It is therefore inevitable that as we select model families that allow for greater complexity of functional form, we will at some point also sacrifice interpretability. As long as the world is complex (i.e., the underlying data generating processes can be of arbitrary complexity) but humans are boundedly rational (i.e., limited in their ability to comprehend and explain complexity to each other), researchers will have to make their own tradeoffs between predictive accuracy and interpretability but can do so in an explicit and transparent manner by showing the differences in these for varying levels of model complexity.⁷

Besides tunable complexity, there are two other features of algorithm-supported induction to note. First, we can check the robustness of patterns within sample I by recursively applying the hold-out principle within it (i.e., through subsample and/or bootstrap sample replication). This means that the patterns we obtain at the end of stage 2.2 are not only tuned to a complexity we can interpret, but they are also robust within subsamples of sample I, making it less likely they are spurious and more likely they will be replicated in the hold-out sample II.

Second, the procedure entails making not only out of sample predictions, but also out of pattern predictions in stage 3, which allows for a test of the abductively generated theory. Out-of-pattern predictions require the theory that we devise to explain patterns found in sample I to also be able to make predictions about new and yet to be observed associations (Lave and March 1993). Such out-of-pattern predictions can arise because the patterns detected algorithmically in sample I have high predictive power; but the additional associations that are predicted by the theory used to explain the pattern may be valid but not have high explanatory power or were not checked with the particular target variable in mind.

For instance, suppose ML produces a robust pattern in sample I, showing that A always seems to associate with C when B is high. A first step would be to abductively generate theory of why this might be the case (Lave and March 1993). Examining this pattern as a hypothesis test in sample II (i.e., out-of-sample test) is useful, but it would be even more valuable if our theory predicts that we should also observe that A should be correlated with D when B is low. This is not part of the original pattern found through ML in sample I (because these relationships may not be the strongest or involve a different target variable) but are an implication of the theory we constructed that accounts for the original pattern. This is an out-of-pattern test.

In fact, one may even begin with using sample I to test theoretical priors in a traditional hypothesis testing manner. If the hypotheses are supported, a replication in sample II concludes the research process. If they are not, sample I can be explicitly analyzed by algorithm-supported induction, as noted previously, to offer input to generating abductive hypotheses, which are then tested in sample II. The smaller sample used in sample II may imply lower power unless the initial unpartitioned sample was large enough. However, (a) samples I and II need not be of identical size and (b) low power has asymmetric effects—if no effect is detected, it may still exist in the population. Therefore, if an effect is detected and replicated within subsamples, it is very likely present in the population. In Table 2, we show how our procedure differs from exploratory regression.

Table 2. Comparing Algorithm-Supported Induction to Simple Exploratory Regressions

Algorithm-Supported Induction in Action: An Illustration

To illustrate how algorithm-supported induction can help in constructing theory, we build a simulated data set, so that we know what the true underlying process that generates the data—also known as the data-generating process (DGP) is. Using the logic of algorithm-supported induction, we show how a theorist might approach the task of finding interpretable approximations to the underlying DGP even when she starts out with no knowledge about it. The advantage of such a synthetic exercise over application to a real data set is that we have full knowledge of the underlying DGP and can therefore assess how well algorithm-supported induction performs in terms of finding interpretable approximations to this process. This would not be possible in the case of real data. Precisely because of this knowable ground truth, it has become standard practice in ML and statistics literature to demonstrate the advantage of any new data analysis procedure over existing ones using simulated data (see, for a recent example, Boughorbel et al. 2017). Such an approach is also increasingly used in management research (Zelner 2009, Kalnins 2018, Shaver 2019). We follow this practice by exploring how our procedure fares with simulated data.

Let us suppose that a researcher has collected firm level cross-sectional data (n = 1,000), partly based on survey responses and partly from secondary sources on innovation outcomes (Y) and a vector of 11 attributes that might be expected to be associated with it (X variables). The names of these 11 variables are set out in Table 3. All the variables in X except x₁₁ are randomly sampled from a normal distribution with mean 0 and standard deviation 1, and x₁₁ is a binary variable randomly sampled from the set {0,1}.

Table 3. Descriptive Statistics of Variables in the Illustrative Simulation

Unknown to the theorist at this point, let us assume the true underlying DGP conforms to

such that all $b_i\hspace{0.17em}$coefficients are zero, except $b_1=0.4,b_2=0.2,b_3=0.5,\alpha =-0.3,\beta =-0.7$, and $\gamma =-\mathrm{0.3.}$ The Gaussian error term has a mean 0 and standard deviation of 0.7. In addition, there exists a small negative correlation (−0.3) between x₁ and x₆, and all the other correlations among xs are zero. This particular set of six parameters was selected purely arbitrarily to illustrate our arguments and to make the interpretation of these coefficients reasonably intuitive: innovation outcomes for a firm in this sample are greater when it inhabits a weak intellectual property (IP) regime (x₁) and its advertising (x₂) and research and development (R&D) intensities (x₃) are high. The weakness of the IP regime and R&D intensity are substitutes, as are R&D intensity and advertising intensity. The benefits of doing business in a weak IP regime subside if the firm has a history of aggressive litigation (x₆). We assume the theorist is unaware of these relationships when they collect the sample. The summary statistics and the correlation among variables in the sample are presented in Tables 3 and 4, respectively.

Table 4. Correlation Table of Variables in the Illustrative Simulation

Stage 1: Splitting the Sample

The first step is to divide the data randomly into a sample for algorithmically supported induction (i.e., for pattern detection) and hold-out sample (i.e., for hypothesis testing) of equal size (sample I = 500, sample II = 500). We will set aside sample II for the next few steps in stage 2 and focus on sample I.

Stage 2: Identifying Comprehensible and Robust Associations Within Sample I

Within sample I, we have a number of possible approaches to finding a small (to make it easy to interpret) and robust (i.e., unlikely to be the result of sampling errors) set of predictors of Y. The resulting function linking the predictors and Y is a stylized fact about this sample. To allow for possible nonlinearities in functional form, but at the same time keep the resulting patterns interpretable (because our objective is abductive theory generation from data), we rely on a two-stage sequence of decision trees (specifically their ensemble version, random forests) to identify key predictors and then use LASSO to narrow down the subset of all second-order polynomials of these key predictors. Although we pick these two algorithms because they allow tunable complexity in functional form while still being interpretable, they are not necessarily unique in these properties. Appendix D replicates our results using (1) gradient boosted regression trees and (2) neural network-based feature selection in stage 2.1 and ridge regression in stage 2.2.

Stage 2.1

We use the random forest algorithm to reduce the initial set of variables from 11 to 5. We configured the random forest to ensemble exactly 35 trees, because increasing beyond this did not improved the performance of the model with respect to mean squared error (Figure 2). A plot of the reduction in model error as we reduce the number of variables (Figure 3) shows that most of the accuracy improvement occurs with the first three variables. To be conservative (i.e., allow some false positives to be weeded out in later stages), we take up to the first five variables. This number (i.e., 3 + 2 = 5) is arbitrary and just for illustration here. It represents the appetite of an inductive theorist for complexity in theorizing. The five variables extracted from random forest analysis are the ones that have the greatest explanatory power (reduction in entropy) across all the decision trees constructed in subsamples (within sample I). This procedure extracts the five variables {x₁, x₂, x₃, x₆, x₇}. These five variables are a superset of the variables that form a part of the DGP; x₇ is spurious.

Stage 2.2

Let us say we are willing to theorize up to second-order polynomial effects (i.e., two-way interactions and quadratic terms) in terms of interpretability. Again, this is a subjective choice by the theorist, based on her/his appetite for complexity in theorizing. In this case, the DGP is also of degree 2. Based on the five variables extracted in step 2.1, we created a model with all possible degree 2 terms of the five variables extracted in stage 2.1 (20 in total: 5 linear, 5 quadratic, and 10 two-way interactions).

We feed this model to the LASSO algorithm in order to reduce the number of terms in the polynomial. Just as we use random forest algorithms to estimate models on many subsets of sample I to tune its hyperparameter (i.e., tree depth), we also obtain results from LASSO across multiple subsets of sample I using k-fold cross-validation (Figure 4; Appendix A).⁸ This procedure aids in tuning the hyperparameter for regularization. LASSO identifies seven terms {x₁, x₂, x₃, x₁ × x₃, x₁ × x₆, x₂ × x₃, and x₆ × x₇} as the most important and robust predictors (out of 20). These seven are also robust predictors across subsamples (Figure 5) and bootstrap samples (Figure 6) in the data. The terms identified by LASSO are a superset of terms that form the DGP and the term x₆ × x₇ is again a false positive; it does not exist in the DGP. The last step within sample I is to run an OLS with these seven variables. The results of this OLS model are presented in Table 5 model 1. As one can observe, the results recover the DGP quite precisely and drop the spurious x₆ × x₇ interaction. Another OLS model is then run after dropping this spurious term and is presented in Table 5 model 2. Because our dependent variable Innovation is continuous between 0 and 1, we are dealing with a regression task. The accuracy of regression models is commonly measured by mean squared errors (MSEs). The prediction by random forest in sample I had an MSE of 0.96, and the combination of random forest and LASSO reduced the MSE to 0.43.

Table 5. OLS Models on Sample I

Stage 3: Building a Theoretical Explanation

At this point, the theorist must ask herself if these robust associations make theoretical sense. This is the abductive reasoning highlighted in stage 3. In our example, hopefully the theorist will spot that R&D intensity and advertising intensity could indeed be substitutes and that R&D intensity can be less useful if a firm operates in weak IP regimes. Less intuitively, weak IP regimes are positively associated weak innovation outcomes, and this effect is diminished when a firm engages in aggressive litigation. Should the theorist be well versed in the innovation literature, this pattern might suggest a logic of mutual benefit from spillovers among the firms within a sector. The theorist’s engagement with the context, in the form of literature reviews, interviews, or case analysis, may be helpful to stimulate the abductive process that produces a theory to account for the patterns in the data uncovered by the previous steps.

The abductive reasoning and the iteration between theory and data necessary at this stage require human expertise and judgment and cannot be left to an ML algorithm. The first output of this abductive reasoning is therefore a set of predictions to be tested in sample II by simply replicating out of sample. The six terms identified in OLS at stage 2.2, if replicated in the hold-out sample, instantiate this.

However, a more ambitious theoretical development is to construct out-of-pattern tests; these are patterns that were not necessarily found in sample I but are implied based on the explanations constructed to account for the patterns found in sample I. Thus, given the theory used to explain the patterns in sample I (e.g., R&D intensity and advertising intensity are substitutes and a weak IP regime lowers the value of R&D investment but may benefit the firm as long as it is not too litigious), what additional hypotheses could we make? If firms recognize some of the benefits of a weak IP regime, then an additional hypothesis might be that there should be a negative correlation between litigation intensity and weak IP regimes through some form of oligopolistic coordination. The specific theoretical explanation here is not relevant and we do not require the reader to endorse it; rather, the point is that such an additional out-of-pattern hypothesis test can be made in the first place.

In sum, the resultant stage 3 is a set of hypotheses, some of which are out of sample, whereas other are out of pattern. These hypotheses can now be tested in sample II (hold-out), with usual statistical significance reported.

Stage 4: Testing Hypotheses in Sample II

We test the hypotheses with a simple OLS regression in sample II, and the results are presented in Table 6. We add two control variables (age and size of the firm), which the theorist might feel are critical in light of prior studies. As one can observe in Table 6, the model is able to correctly identify the significant linear and interaction terms. In addition, as a test of out of pattern prediction, a hypothesis about correlation between IP regime strength and litigation intensity (controlling for other variables) also receives support (Table 7).

Table 6. OLS Model on Sample II for Testing Out-of-Sample Predictions

Table 7. OLS Model on sample II for Testing Out-of-Pattern Prediction

These results are still correlational and do not support a causal interpretation. However, the theorist could of course use the standard methodologies that enable closer-to-causal inference in nonrandomized data, such as matching instrumental variables and fixed effects in both the theory-generating and testing phases wherever feasible. Because those concerns are distinct from ours (and well understood), we do not discuss those methods here. In addition to the results from the hold-out sample, we recommend full and transparent reporting of the analysis within sample I (used for pattern detection) and explicit recognition of the assumption that the two samples can be treated as independent samples drawn on the same DGP.

As a contrast, it is interesting to see what a simple exploratory regression approach would produce and how far it would get with the same data in terms of recovering the DGP. Clearly, an ad hoc search for significant (i.e., with p < 0.05) effects using only a simple linear model or by trying pairwise interactions or one quadratic term at a time is likely to be both time consuming and unlikely to find the DGP. In contrast, a brute-force approach may be to put all 11 variables, their 55 two-way interactions, and 11 quadratic terms (77 terms in total) into a kitchen-sink regression. In our example, such a procedure identified all six correct terms as being statistically significant but also identified two other terms as significant, although their true coefficients in the DGP are zero (i.e., false positives). Finally, one could run a stepwise regression on the 77 terms. In our example, this identified all six terms in the DGP correctly but also produced four additional terms (false positives) as significant, which did not exist in the DGP. This is because the naïve exploratory approach lacks the protection against overfitting that our procedure provides.

For this illustration, we consciously made two important simplifications: First, we assumed a particular set of coefficients in the DGP. However, independent of the specific parameters assumed previously, our arguments can be made with any arbitrary set of coefficients for this underlying DGP. In Appendix B, we show a generalization of this analysis to a larger variety of data generation processes (all of degree 2, but with randomly drawn coefficients).

We also illustrate a central tradeoff in algorithmically supported induction between interpretability and predictive accuracy. In this exercise, we evaluate the performance of our procedure (measured by the difference between the estimated and true coefficients in the DGP) while varying the number of features selected in step 2.1 and the amount of noise present in DGP. As expected from our reasoning about the tradeoff between interpretability and prediction accuracy, the procedure performs better as the number of features selected in step 2.1 (a measure of appetite for complexity in theorizing) increases and the amount of noise in the DGP (a measure of randomness versus pattern) decreases. Because our procedure begins without any knowledge of the underlying DGP, a useful benchmark is to compare it with random selection of hypotheses to be tested. The results show that our procedure performs significantly better (Appendix B, Figure B.1).

Second, in this illustration, the DGP is designed to be a degree 2 polynomial. In management research, surveys of empirical papers (Hulland 1999, Haans et al. 2016) indicate that, in our field, most hypotheses feature a complexity up to quadratics and two-way interaction terms. In other words, the typical theory testing paper assumes (quite reasonably, given the importance of interpretability) that the DGP can be approximated by a linear function with one or two interaction terms at most. Our assumption about the DGP mirrors this. However, this might raise a legitimate concern that, when we approximate the underlying DGP with a degree 2 polynomial in stage 2.2, we (unsurprisingly) find a good fit.

In Appendix C, we therefore show what happens when the underlying DGP is of (a) higher degree, (b) lower degree, and (c) contains a variable not considered in the analysis while keeping the choice of a degree 2 polynomial in stage 2.2 of the algorithm-supported induction procedure. These illustrations cover the instances of oversimplification, undersimplification, and omitted variable bias that could occur while working with real data sets. In each of these cases, we evaluate our method in comparison with that of an analyst who conducts exploratory analysis without adopting procedures such as regularization and cross-validation that we propose in our procedure. We consider two types of exploratory analysts, namely, one who has either oversimplified or undersimplified the true degree of the underlying DGP or omitted a variable and does not use procedures to avoid overfitting or who has correctly identified the true degree of the underlying DGP but does not use procedures to avoid overfitting. We show that our method performs significantly better at identifying the underlying patterns compared with both these versions of exploratory analysis that differs from our procedure.

Discussion

In management and organization research, the use of large sample data for inductive and abductive theorizing rather than the test of hypotheses is rare. We suspect this may be partly the result of an incorrect (but, in our experience, widely held) premise that theorizing from data is necessarily restricted to qualitative data (Shah and Corley 2006, Locke 2015). As also noted by Glaser (2008), who helped lay the foundations for qualitative induction along with Strauss (Glaser and Strauss 1967), quantitative analysis can serve as a powerful stimulus to theory building. For instance, case control designs, which are popular in medical research, represent quantitative induction. In this method, a sample of cases that vary in their outcome of interest are statistically (i.e., algorithmically) analyzed to detect the correlates of the outcome in the data (Shadish et al. 2002, p. 128).

Building theory from data, whether with large or small samples, requires pattern detection and pattern explanation. The procedure for conducting algorithm supported induction that we have outlined in this paper can help scholars identify robust patterns in data, which can then become inputs to abductive theory construction (He et al. 2020). There are four key stages in this approach (Table 1): a split of the data into samples for pattern detection and hypothesis testing, the use of algorithms with tunable complexity to strike a well-considered balance between comprehension and prediction, the use of subsample replication to avoid overfitting even within the sample used for induction, and the imperative to create out-of-pattern tests in the hold-out sample. These four stages together constitute a new perspective on theory building from large-scale data, a meta-algorithm, none of whose components are necessarily new, but the integrated approach, we believe, is novel.

The insights that algorithm-supported induction produces may have long-term consequences for the way we craft theories in organization science; in the future, we may come to think of honest and sophisticated data mining as a sign of high-quality work (Bamberger 2018). At the very least, by complementing traditional inductive inference by humans, ML algorithms are likely to lend increasing prominence to theorizing from large sample data, as a manner of knowledge creation in our field. If there are concerns about the balkanization of theory, where a patchwork of sample specific theories emerge, it is useful to consider what their emergence implies: perhaps the existing more general theories are invalid or at least incomplete. Aggregation of results through meta-analysis of robust patterns is still the best-known methodology for building general insights from empirical research, and algorithm-supported induction helps, in our view, to produce the robust patterns (Hunter et al. 1982).

An instructive question to ask is whether the benefits of algorithm-supported induction require ML algorithms per se or even simpler and more familiar can suffice. First, as we already noted, familiar techniques such as OLS and logistic regression can be used for ML if the purpose is to fit data in a manner that optimizes the bias-variance tradeoff (Abu-Mostafa et al. 2012). Second, how our recommended procedure differs from the practice of a sophisticated user of exploratory regressions would depend on what that sophistication entails. If the user of exploratory regressions does all four stages of our procedure, then they are effectively replicating our procedure; depending on which step they diverge on. Table 2 shows what is lost.

As noted, most theory building from data in our field has been based on a limited number of cases, which may even provoke debates on replicability of results and transparency (Aguinis and Solarino 2019, Pratt et al. 2019). Thus, at this point, the theorist will naturally ask about the sample size required to be able to apply algorithm-supported induction. In principle, it is possible to detect robust patterns through ML with relatively small sample sizes. As we noted, in contrast to hypothesis testing, the key concern with induction through ML is overfitting and not statistical inference. For instance, one of the most widely known data sets for teaching ML, known as iris, contains only 150 observations of data with five variables. This is a data set for three species of iris flowers and has been used extensively to test and validate diverse ML algorithms and models in more than 100 academic papers (Dua and Graff 2017). In medicine, where generating cases is costly, ML techniques to work with small data sets have been developed through medical research (Shaikhina and Khovanova (2017) (where n = 56). A study by Jiang et al. (2009) is an instance of such an approach; they proposed modifications in existing ML algorithms for learning from samples as small as 24 cases (see also Zhou and Jiang 2003). Specific methods for small data sets include aggregation of regularized classifiers (Lu et al. 2010), robust sparse representation (Sami Ul Haq et al. 2012), and discriminant analysis (Chen et al. 2000).

The implication is that many things we have argued regarding algorithm-supported induction are in principle relevant even for smaller N methods of inductive and abductive theory building, such as the comparative case method (Eisenhardt 1989) or qualitative comparative analysis (Ragin 1987, 2000). The risks arising from neglecting to separate pattern detection and explanation in the multiple case method, and the risk of overfitting inherent in fitting multiple interactions to small samples in Boolean qualitative comparative analysis (QCA) are well known to its sophisticated practitioners, and they recommend careful measures to mitigate these (i.e., see a thoughtful discussion on these matters in Eisenhardt 1989, p. 545 and Fiss 2011). Our point in this paper simply is that where a sufficient number of cases exist relative to the parameters in the theory to be developed, algorithm-supported induction using ML may offer at least an alternate path to be explored.

Challenges and Opportunities

Having illustrated the promise of ML algorithms for inductive theorizing in organization science, it is equally important for us to point out the caveats. First, as we stressed at multiple points in the paper, ML algorithms today can support but cannot replace human judgment entailed in theory building through inductive and abductive reasoning. What is measured and how, and, critically, the explanation we propose for the observed pattern is still very dependent on human intuition and creativity, at least at the current state of development in the field of AI.

Second, ML itself is no magic bullet and involves many tradeoffs in choosing among suitable model families and algorithms. These choices are not straightforward given that no one algorithm fits all prediction tasks (there is no free lunch for optimizing loss functions, as Wolpert and Macready 1997 famously noted). Balancing the tradeoffs between ML algorithms requires the researcher’s ingenuity and domain knowledge to carefully evaluate the performance of the algorithm and compare this with others that are also potentially suitable. In general, we stress the importance of selecting algorithms that offer a high degree of interpretability (given that our purpose ultimately is the creation of theory) in stage 2.2 and procedures (such as regularization and cross-validation) that help mitigate overfitting in sample I. Within these parameters, many different algorithms may be useful, including familiar OLS and logistic regressions, and not only those we have illustrated in this paper (random forest and LASSO).

Third, algorithm-supported induction is not guaranteed to find interpretable robust patterns in the data, and falsification of a hypothesis constructed most carefully with algorithm-assisted induction can (of course) occur; this indicates that despite our best efforts, we stand defeated by sampling error or undetected flaws in measuring the data used for induction. That is also valuable learning for the theorist.⁹

Fourth, although ML methods rely on detecting associations, causal interpretations and conclusions require careful scrutiny. Although this is true for hypothesis testing in general (stage 4), the risk of biases in data leading to biased conclusions in the pattern detection phase is distinctly worth highlighting. Studies have demonstrated that, although ML algorithms are not inherently biased, they could under some conditions magnify biases already present in the training data (Kamishima et al. 2011, Zemel et al. 2013, Shrestha and Yang 2019). Such biases may constitute a more serious problem in ML than in traditional statistics because many ML models remain difficult to interpret and thus may have difficult-to-spot potential biases. For example, while training a neural network to aid a decision to award home loans, it is thus far not entirely possible to interpret how the weights of the network edges capture underlying lack of fairness in the training data with respect to the predicted decision. This is one more reason we strongly recommend against using black-box algorithms like deep learning for the purposes of algorithm-supported induction in stage 2.2, as well as reiterating that a causal theory that we develop in stage 3 to explain patterns obtained in stage 2 might nonetheless gain no support in stage 4 once properly tested with the techniques of causal inference.

Fifth, the use of ML techniques is fairly easy, but understanding the technical foundations for the advantages and disadvantages of the algorithms used is not, nor is keeping pace with rapid advances. Like any new tool, individual researchers should also be careful when importing methodologies from other fields. Organization and management researchers, like most social scientists, typically aspire to be sophisticated users rather than producers of statistical methodology. Identical to the adoption of other techniques developed by statisticians, the adoption of ML techniques in organization and management research requires not only familiarity and access to software that embeds these procedures but also a solid conceptual understanding of what the algorithms do and what they assume. This is made particularly challenging in the case of ML because of the rapid pace of developments in the field.

For instance, in recent years, the ML community has debated and made considerable progress in developing both fairer (Yao and Huang 2017) and more interpretable (Rudin 2014) algorithms. The subdomain of ML today referred to as interpretable AI is being developed for the sole propose of interpreting parts of very complex but effective ML models, such as deep neural networks (Guidotti et al. 2018, Samek et al. 2019). Interpretability is achieved in these models by (1) explaining the prediction based on local marginal effects—changes are made to particular inputs in either direction to observe the change in the prediction (LIME by Murphy et al. 2006); (2) abstract programming where learning is combined with query (Fischer and Krauss 2018); and (3) graphical plots and visualizations (Samek et al. 2019). Most of these methods to date focus on interpreting components of a complex and largely uninterpretable model and not necessarily explainability in intuitive human terms, and therefore, we do not think they are ready yet for use in theory generation by organizations researchers. In our field, we generally prefer imperfectly predicting but wholly comprehensible theories over perfectly predicting but only partially comprehensible theories. Nevertheless, given the rapid advances in this area, we encourage researchers to monitor progress on interpretable AI, because it may produce the next level of sophistication in algorithm-assisted theorizing.

Conclusion: Taming Pavlov’s Dog

ML algorithms such as supervised and unsupervised learning can be considered the descendants of Pavlov’s dogs: they are trained to develop associations between variables (e.g., establish the copresence of bell ringing and food) and then tested in their ability to predict the rest when presented with only some of the variables (e.g., will the bell ringing predict the presence of food?).¹⁰ However, these rudimentary learning mechanisms found (even) in our pets have enormous power and are at the heart of the current explosion of interest in ML. Admittedly, researchers in our field have already begun to exploit this power for data coding, data reduction, and support of traditional hypothesis testing by aiding causal inference. We offered an alternative perspective: treating a validated prediction model as a stylized fact opens up the path to theory development from data followed by theory testing in a mutually complementary manner, even possibly within the same study. As management and organization researchers, we have much to gain from understanding these opportunities.