Intellectual Diversity in IS Research: Discipline-Based Conceptualization and an Illustration from Information Systems Research

“Our backgrounds, training and interests are very different. We must make that as a strength, not a cause of argument.” Peter G. W. Keen (1987)

“Because information systems began fairly early to become an international discipline, a variety of views about the field and its research were encouraged. The field has a richer set of views than other fields because the positivist philosophy that dominated the American research and the phenomenology philosophy that tended to dominate in Europe were both supported by the worldwide community.” Gordon B. Davis (2000)

1. Motivation and Objective

Information systems (IS) is a pluralistic scholarly discipline. IS scholars study many topics (Robey 1996, Benbasat and Zmud 2003, Hirschheim and Klein 2012, Rai 2016, Gupta 2018, Tarafdar and Davison 2018), use a variety of methods (Mingers 2001, Burgess et al. 2017), and draw on theoretical insights from several disciplines (Goes 2013). Moreover, the topics they study morph continually in response to information technology (IT) innovations, changing IT-based business models and consumer preferences, IT-based products, and new contexts of use in technology-mediated personal, societal, organizational, and government processes. Consequently, they not only investigate continually emerging novel phenomena but also develop new insights into established phenomena (Gupta 2017, Burton-Jones et al. 2021).

It would seem that the IS discipline values intellectual diversity. A quick glance at the mission statements of the top IS scholarly journals indicates that most ascribe to the operating principles of publishing IS scholarship that is inclusive regarding topics and methods and of being fluid with respect to the IS phenomena, the units of analysis, and the theories studied (Straub 2010). For example, this field-based value is embodied in the editorial statement of Information Systems Research (ISR): “The journal is receptive to a wide variety of phenomena and topics related to the design, management, use, valuation, and impacts of information technologies at different levels of analysis (e.g., individuals, groups, firms, networks, societies, and nations).” The editorial statements of other leading IS journals express similar principles.

Yet, enacting intellectual diversity is not a simple matter and can be complicated by how the field constructs research traditions and scholarly understanding. It is easy to overlook, when in the thick of the technical action (e.g., literature review, data collection, data analysis, and writing), that academic socialization significantly shapes editorial judgement, which in turn shapes what appears in the pages of journals. We are shaped not only by our own training but also by our interactions with colleagues and by research in adjacent fields. As Gupta (2017) noted, “the vast majority of research that ISR and other journals publish [also] follow [reflects] other researchers in the field—typically senior, successful researchers.” As we mirror the work of our esteemed colleagues, we coconstruct our shared, cumulative IS traditions.

This coconstruction can be ad hoc. Publication decisions are associated with people (in the roles of editors and reviewers) and thus are shaped by their interest in topics, methodological specializations, and training within IS (Straub 2010). Researchers in different IS subdisciplines may not embrace one another’s methods and approaches because they are not trained to do so, because bridging subdisciplines may be too difficult, or because they simply do not view it as a worthwhile endeavor, particularly if their universities do not provide incentives for doing it, creating what Goes (2013) termed an absence of intradisciplinarity. One consequence of this organically generated cacophony of topics and methods is the sometimes lively, sometimes contentious, always interesting, and constantly recurring discussion of “what IS research is and what it is not” (Gupta 2017). A second consequence is that it can lead some authors to feel disaffected because they perceive that some journals prefer certain topics, theories, or methods over others (Straub 2010), even when these perceptions are not supported by the journals’ editorials, their published mission statements, or the breadth of views found in their pages. The perceived exclusion of methods or topics can result in the perception that a journal is biased toward or against a set of topics and methods. For example, Gupta (2017, p. 1) wrote, “one question (sometimes framed as an accusation) that arises often is regarding the nature of research that ISR would consider publishing. Countless times have I heard the opinion of potential authors that ‘their’ research is not welcome at ISR.” Although this statement concerns a particular journal, the perception it describes could be said of any leading IS journal (Straub 2010).

Such perceived biases, once they take hold, are likely to be reproduced (even if unintentionally) through repeated socialization and authors’ choices, undermining intellectual diversity in the IS discipline. If there is bias in favor of specific topics or methods, then perceptions of field relevance and the possibilities of the kinds of research the field will support will themselves be biased. Taken to an extreme, this may result in the field missing important phenomena from the scholarly point of view, as well as becoming irrelevant to the audience of practitioners it seeks to inform. Furthermore, if authors and readers perceive such an exclusion bias, they may simply be turned off and not submit papers to journals in the IS discipline, decreasing the heterogeneity of ideas necessary to support a vibrant discipline. If perceived biases change authors’ submission patterns, leading IS journals may miss insights offered by IS authors who are from all over the world (Saunders 2006) and who investigate a diverse range of topics relevant to their lived experiences using different methods, and not speak to the interests of our global community (Leidner 2021). Most importantly, the discipline would risk losing ideas and skills necessary to the study of emerging IS phenomena that are complex, have a broad scope, and transcend the perspectives of the separate IS subdisciplines (Rai 2018).

To tackle the previous dilemmas, it is necessary for the discipline to move away from its disjointed and fragmented approach to supporting intellectual diversity (Taylor et al. 2010) and toward a more purposeful approach to encouraging the study of diverse IS phenomena using diverse methods and philosophies. To encourage a focused and positive conversation about how to coconstruct intellectual diversity and inclusion in the IS discipline, this commentary develops a relatively simple and lean conceptualization of intellectual diversity and applies it to the past 10 years of articles published in ISR. Although there have been attempts to identify clusters of keywords representing various topics (Culnan 1987, Barki et al. 1988, Sidorova et al. 2008) and methodological commentaries (Walsham 2006, Myers and Klein 2011) from time to time, these approaches have tended to focus primarily on specific topics and methods. Hence, our objective is twofold: (1) to develop a multidimensional conceptualization of IS intellectual diversity and (2) to provide an empirical illustration and analysis of the IS intellectual diversity of research published in ISR. Through this systematic study, we offer insights into the evolving nature of topics, methods, contributions, and phenomena that appear in research published in ISR and identify implications for intellectual diversity and inclusion in the broader IS discipline.

2. Defining IS Intellectual Diversity

This section reviews the literature from which we derive the four dimensions of IS intellectual diversity, namely (1) topic, (2) level of phenomenon, (3) type of contribution, and (4) method.

Diversity is the attribute or quality of “having or being composed of differing elements” (Merriam-Webster Dictionary). Based on this definition, we suggest that IS intellectual diversity manifests in a dissimilitude of concepts and ideas that constitute the IS body of knowledge. Because IS is a sociotechnical discipline (Mumford 2006), broadly speaking, IS scholars examine a variety of phenomena at the intersection of technical topics (e.g., IS applications such as Enterprise Resource Planning (ERP), social media, autonomous robots, and fitness trackers) and social topics (e.g., how and why such applications are developed and used).¹ Specifically, IS scholars investigate how IT artifacts (1) are conceived, constructed, and implemented; (2) are used, are supported, and evolve; and (3) impact (and are impacted by) the contexts in which they are embedded. Thus, topics essential to the IS discipline include the sociotechnical capabilities, practices, and behaviors involved in planning, designing, constructing, implementing, and using IT artifacts (Benbasat and Zmud 2003). Examining these topics from different perspectives—behavioral, design science, economic, and organizational—leads to the creation of intradisciplinary IS knowledge that integrates different IS topics and research traditions (Goes 2013). Furthermore, IS phenomena intersect with those studied in other disciplines (e.g., computer science, economics, marketing, psychology, and sociology). Thus, IS scholars engage with these other disciplines (Baskerville and Myers 2002, Grover and Lyytinen 2015) and create interdisciplinary knowledge at the intersection of IS and other disciplines (Rai 2016, Tarafdar and Davison 2018).

Researchers have developed some general notions of what might constitute IS intellectual diversity. For example, there is broad agreement on two important high-level aspects of IS research: research topics and research methods (Landry and Banville 1992, Taylor et al. 2010). In further categorizations, Benbasat and Weber (1996) separated research diversity into three areas: problems, theoretical foundations and reference disciplines, and methods. Rai (2016) stated that the range of articles published in MIS Quarterly spans diverse IS problem domains and research traditions, interstitial topics between IS with other disciplines, theory generation, and methodological advancement. Thus, although we did not find an overarching framework that includes all aspects of intellectual diversity, we did find instances of a number of different dimensions.

Based on the background outlined previously, we conceptualize four dimensions of IS intellectual diversity: (1) domain topic, (2) level of phenomenon, (3) type of contribution, and (4) method. We develop these dimensions next.

2.1. Domain Topic

IS scholars generate novel theoretical understandings of emerging and established IS phenomena by investigating a variety of topics. A number of studies (Culnan 1987, Baskerville and Myers 2002, Taylor et al. 2010) have examined broad streams of such topics. Referred to as intradisciplinarity (Goes 2013, Tarafdar and Davison 2018), the study of diverse topics results in different approaches to IS research questions, such as behavioral, economic, and design approaches (Goes 2013). Markus and Rowe (2021) suggested that multifocal thinking is needed to understand interlinkages among topics. Furthermore, these topics are dynamic; they evolve as IS researchers organically respond to new waves of digitization by investigating new phenomena and contexts (Taylor et al. 2010). This notion of change is termed the plasticity (Lyytinen and King 2004) of the IS discipline. Thus, topics studied in the IS discipline emerge and change over time, requiring conceptualizations of intellectual diversity to consider their evolution.

2.2. Level of Phenomenon

Given that IS are used individually and collectively, the sociotechnical phenomena spawned by IS use are enacted by and affect individuals, groups, organizations, and societies. The topics thus span different levels. Sidorova et al. (2008) identified four levels of analysis: groups, markets, organizations, and individuals. Recent editorials have commented on the need to examine phenomena at the societal level as well (Majchrzak et al. 2016, Aanestad et al. 2021). Additionally, a robust stream of studies has introspectively reflected on the state and preoccupations of IS research, concentrating on the level of the IS discipline (Orlikowski and Iacono 2001, Benbasat and Zmud 2003, Grover and Lyytinen 2015, Sarker et al. 2019).

2.3. Type of Contribution

The third dimension is the type of contribution. Gupta (2018) presented an analysis of different types of contributions. For example, a contribution can be theory based in that it tests or generates theoretical principles, frameworks, or models. Alternatively, it can be empirically focused in that insights are generated from data (Grover and Lyytinen 2015, Dennis 2019). These two types of contribution generate understandings of concepts and propositional relationships underlying particular IS phenomena. Another type of contribution advances the methodological foundations of IS. Additionally, research on societal IS phenomena, such as public health and cyberinfrastructures, presents opportunities for influencing policy makers. Finally, some IS research has focused on the application or artifact itself to generate design principles. The sections of leading IS journals partly mirror these types, with sections dedicated to contributions vis-à-vis theory, IS discipline (e.g., research perspectives, issues, and opinions), and research notes (which often concern methods and empirical contributions).²

2.4. Method

The dimension of method represents how IS researchers conduct their research. The primacy of this dimension is evident in the different methods IS scholars adopt and in the numerous special issues of IS journals dedicated to IS research methods. Editorials in leading IS journals have emphasized the importance of “articles that are forward-looking and open up new ways for the IS research community to conduct research” (Rai 2016, p. viii). Over the last two decades, we have seen an expansion of methods from positivist and hypothetical–deductive forms of inquiry to approaches that afford opportunities to glean insight from qualitative data (Sarker et al. 2018). Recent and ongoing innovations in IS methods include advances in computational approaches to analyzing large qualitative and quantitative data sets (Miranda et al. 2022) and analyzing novel forms of structured and unstructured digital traces of IS-enabled actions and communication (Vaast and Walsham 2013). All of this suggests that evaluating intellectual diversity in the IS discipline requires considering the plurality of methods applied in IS research.

In the next section, we first develop a schema for evaluating these four dimensions of IS intellectual diversity. We then illustrate its application by evaluating the IS intellectual diversity of all the papers published in ISR from 2012 to 2021.

3. Analyzing the Intellectual Diversity of Papers Published in Information Systems Research from 2012 to 2021

We present this section in four steps. In Steps 1–4, we operationalize the dimensions of IS intellectual diversity, develop a schema for evaluating papers, and describe how we used the schema to code 575 research papers published in ISR between January 2012 and December 2021. In Step 5, we present and interpret the research patterns revealed by our analysis.

3.1. Operationalization and Measurement of IS Intellectual Diversity

3.1.1. Step 1: Initial Operationalization.

Based on the literature review, IS intellectual diversity was operationalized as having four dimensions: (1) domain topic, (2) level of phenomenon, (3) type of contribution, and (4) method.

3.1.1.1. Domain Topic.

Studies from 1987 to 2008 have identified broad clusters of IS research in leading IS journals. Early studies suggested that eight broad topics consistently appear in the literature. (1) External environment includes topics relating to the social and political environment and ethical issues. (2) IS development and design includes topics such as systems and requirements analysis, IS project management, IS development methods, and social and organizational aspects of IS design. (3) IS usage includes topics such as end-user support, acceptance, and resistance; these studies have often used psychological and individual approaches to IS. (4) IS management includes the topics of strategic IS planning, IS evaluation, IS implementation, IS sourcing, and IS staffing. (5) IS education and research includes topics concerning IS research methods and IS curriculum. (6) Organizational environment topics include task characteristics and organizational communication (Ives et al. 1980; Culnan 1987; Barki et al. 1988, 1993; Baskerville and Myers 2002). (7) Representations in IS includes topics such as databases, coding, storage, and system structure. (8) Specific application systems topics include knowledge management systems, enterprise systems and decision support systems (DSSs) (Culnan 1987, Baskerville and Myers 2002). A later classification (Sidorova et al. 2008) identified two emergent clusters. (9) The IT and markets cluster includes the topics of interorganizational relationships, electronic data interchange, and markets. (10) The IT and groups cluster includes topics related to group DSS, group dynamics, and team-level phenomena. Later studies have focused on more granular topics, such as IS use, e-commerce, outsourcing, software development, the business value of IT, IS capability, and IS security, suggesting that IS scholars are applying increasing specialization and analytical depth to certain topics (Tarafdar and Davison 2018). Based on these topic clusters, we generated our initial list of 14 topics, which is presented in Appendix A.

3.1.1.2. Level of Phenomenon.

We focused on four broad levels of analysis: (1) microlevel analysis, which focuses on individual phenomena; (2) macro-level analysis, which investigates phenomena at the firm, platform, and industry levels; (3) societal analysis, which examines phenomena at the societal, community, and family levels; and (4) IS-discipline-level analysis, which focuses on introspective studies of the IS field (see Appendix A).

3.1.1.3. Type of Contribution:

We drew on editorials and special issues (Majchrzak et al. 2016, Gupta 2017) and guidelines for submissions to leading IS journals to develop a schema for coding the type of contribution. As shown in Appendix A, we identified five types of intellectual contribution: theoretical, policy related, empirical, methodological, and artifact related.

3.1.1.4. Method.

We drew on existing work to categorize the methods (Gupta 2017, Rai 2018), as shown in Appendix A.

3.1.2. Step 2: Test Coding and Validation.

Two of the authors applied the schema to the coding of all the articles in the first two issues of 2021. They identified a few papers whose content did not clearly map to the schema’s categories and that displayed categories not captured by our schema. These discrepancies were discussed among the authors, and the discussion resulted in additions to the topic dimension and in a sharpening of the definitions of some of the method dimension’s categories, as shown in the updated schema presented in Appendix B.

3.1.3. Step 3: Coding of Papers.

In this step, a third author, who was not involved in Steps 1 or 2, coded all the papers published from 2012 to 2021, 575 papers in all. During this process, the author met regularly with the other authors to discuss coding progress, tackle discrepancies that arose, and make slight adjustments to refine the categories further. The culmination of this step yielded the coding of 575 articles according to the four dimensions of IS intellectual diversity. The final coding scheme is presented in Appendix C.

3.1.4. Step 4: Interrater Reliability Test.

Two of the authors who did not code in Step 3 independently coded two papers randomly selected from each of the 10 years from 2012 to 2021, for a total of 20 papers. The coding results for this set were then compared with one another and with the coding results from Step 3. This exercise yielded 90% interrater reliability. No new categories emerged. At the end of this step, we concluded that our coding schema for evaluating IS intellectual diversity was reasonably robust vis-à-vis the empirical context of our study.

3.2. Analysis and Findings

Our analysis focused not only on the frequency and content of each dimension but also on how they occurred together and over time. From this content-based, comparative, and temporal approach to the articles, we were able to glean insights about each dimension of IS intellectual diversity for all the papers collectively and of its temporal variation. We were also able to identify the intersections between dimensions. We provide a narrative review of the results for (1) each dimension, (2) intersections between certain dimensions, and (3) longitudinal analysis of each dimension.

3.2.1. Domain Topic.

We found that 10 topics comprised 76% of papers. Figure 1 shows that topics related to e-commerce (such as online reviews and multichannel retailing) were the most prevalent, followed by those related to digital products and services, online communities, and the business value of IT.

3.2.2. Level of Phenomenon.

To examine the level of the phenomenon, we examined the frequency distribution of papers that focused on phenomena at the micro-, macro-, societal, and IS-discipline level. Microlevel studies focused on individual phenomena. Macro-level research addressed phenomena at the firm, industry, and platform level. Societal and IS-discipline studies focused on phenomena related to the broader society and the IS discipline, respectively. Figure 2 illustrates that most of the papers (64.14% = 369/575) focused on macro-level phenomena. Most of the remaining papers (29.91% = 172/575) focused on microlevel phenomena. Papers that examined societal phenomena (1.91% = 11/575) and phenomena related to the IS discipline (4% = 23/575) were published less frequently.

3.2.3. Type of Contribution.

We examined paper frequency with respect to five types of contribution, namely, theoretical, policy related, empirical, methodological, and artifact related; a single paper can make multiple types of contribution. Papers that included theoretically derived and explicitly stated hypotheses, models, or propositions and discussed related theoretical contributions were coded as theoretical. Papers explicitly stating policy implications for industry or government were coded as policy related. Papers that lacked formal hypotheses and propositions and drew inferences from data analysis were coded as empirical. Papers focused on research methods and the design of artifacts were coded as methodological and artifact related, respectively. Figure 3 shows that most of the papers (67.13% = 386/575) focused on theoretical contributions, followed by empirical (34.09% = 196/575) and policy-related (15.65% = 90/575) contributions. Papers that made methodological and artifact-related contributions were less frequent, accounting for 2.78% (16/575) and 3.48% (20/575) of the papers, respectively. We also found that most papers that made empirical contributions used econometric methods. It should be noted that the total number of papers shown in Figure 3 is greater than 575 because we counted the distinct contributions of single papers separately. Policy-related contributions appeared only in combination with other types of contribution.

Figure 4 visualizes a more granular analysis of contributions (e.g., theoretical only, empirical only, theoretical and empirical both). We found that most papers made only theoretical or only empirical contributions. Among the contribution combinations, theoretical combined with policy related and empirical combined with policy related were the two most common combinations. The artifact-related contributions were fewer in comparison.

3.2.4. Method.

Finally, we describe the distribution of methods. We counted a particular method each time it appeared in a paper. The results are shown in Figure 5, where n > 575 because a single paper could use multiple methods, each of which were counted separately. We found that 11 methods appeared in the papers. Econometrics was the most frequently used method, appearing in 38.96% (224/575) of the papers, followed by game theory (20% = 115/575); experimental methods (laboratory experiments (including online laboratory experiments), field experiments, and A/B testing) were used in 15.13% (87/575) of the papers. To differentiate between related yet distinct methods, we separated analytic modeling papers into three categories: econometrics, game theory, and optimization.

Next, we count each paper once, which means we report papers that include multiple methods only in the “Multiple Methods” category. Thus, n = 575, with papers having multiple methods are 105. The results are shown in Figure 6, and the relative distributions of different methods are similar to those illustrated in Figure 5. Among the papers that used one method, econometrics was the dominant method, found in 38.30% (180/470) of the papers, followed by game theory (18.51% = 87/470); experimental methods (laboratory experiments (including online laboratory experiments), field experiments, and A/B testing) were used in 13.40% (63/470) of the papers. We included the 12th column to indicate papers that applied more than one method.

We ran a separate analysis concerning different types of experiments to gain a more granular understanding of the experimental methods used in the papers. This analysis considered both papers using only one method and those using multiple methods. Figure 7 shows that 113 papers used experimental methods, comprising nearly 20% of the total coded papers (575). Our coding placed econometric papers that used natural or quasi-experimental designs, such as difference-in-difference and regression discontinuity design, in the same category, because these designs are used to establish the exogeneity of the treatment variable. Laboratory experiments, natural or quasi-experimental designs, and field experiments were the most frequently used, followed by online experiments and A/B testing methods.

To further illustrate the frequency of combinations of methods, we examined the pairing of the four most frequently used methods (econometrics, game theory, experiments, and use of quantitative data) with another method. Figure 8 shows that econometric methods were frequently combined with machine learning, experiments, and game theory. The last column (“Other Combinations”) includes all the combinations that appeared only once, as shown in Appendix D. Appendix D also includes more detailed information about all the combinations of methods.

3.2.5. Cross-Analyses.

We conducted a series of cross-analyses of the top-10 topics (n = 575; 76% of papers that studied the top-10 topics) with the following: type of contribution (Table 1), level of phenomenon (Table 2), and method (Table 3). Tables 1–3 indicate that the same topic has been studied at different levels of analysis, has been associated with theoretical, policy-related, artifact-related, and empirical contributions, and has been investigated through multiple research methods.

Table 1. Cross-Analysis of Level of Phenomenon and Top-10 Topics

Table 2. Cross-Analysis of Type of Contribution and Top-10 Topics

Table 3. Cross-Analysis of Method and Top-10 Topics

3.2.6. Longitudinal Analysis of Three-Year Clusters.

To understand how the different dimensions (domain topic, level of phenomenon, type of contribution, and method) evolved over the last 10 years, we graph them within three-year increments: 2012–2014, 2015–2017, and 2018–2020. We also provide a graph for 2021, noting that because 2021 is only one year, the corresponding numbers are lower than for the three-year periods.

Regarding domain topic, Figures 9–12 show that the 10 most studied topics varied from one range to the next. E-commerce, the business value of IT, online communities, and IT sourcing appeared during all the year ranges, which is consistent the status of macro-level phenomena as the most widely studied and the status of econometric analysis as the method most widely applied to the topic of e-commerce (Tables 1–3). Interest in software development waned after the first period; interest in IS use waned after the second period. Interest in IS security emerged in the second period and persisted through 2021. Three completely new topics appeared in 2021: Fintech, social media–driven societal phenomena, and IS innovation.

For the level of phenomenon, we provide two types of visualization. Figure 13 shows that across all the time periods, macro-level research was the most common, followed by micro-level, IS-discipline, and societal research. Figure 14 shows that microlevel analysis increased, whereas IS-discipline research decreased, over the first three time periods. Macro-level and societal research appeared to be relatively stable.

Regarding contribution type, Figure 15 shows that theoretical contributions were the leading type, followed by empirical contributions and policy-related contributions, for all four time periods. Figure 16 shows that policy-related and empirical contributions increased from the second to the third period.

For the method dimension, we visualized the papers (n = 575) using 1 of the 11 different methods and multiple methods. Figure 17 shows that for all four time periods, econometrics was the most popular method, followed by game theory. Over the first three time periods, experiments (including field experiments, laboratory and online experiments, and A/B testing) increased and methods based on quantitative data decreased. As Figure 18 shows, conceptual papers and methods based on quantitative data decreased over the first three time periods. Machine learning (e.g., natural language processing) increased over the first three time periods, computational modeling remained relatively stable over the same time periods, and econometrics increased from time period 2 to time period 3. Action research did not appear after the first time period.

4. Contributions and Implications

We set out to understand intellectual diversity in IS research and provide an illustration from ISR. We lay out the implications of our study here.

4.1. Applying IS Intellectual Diversity

IS research has been referred to as a broad church (Robey 2003). Consistent with this view, we take a “big tent” approach to conceptualizing and operationalizing IS intellectual diversity. Our approach draws on the introspective IS literature and an empirical analysis of a substantial corpus of papers published in one journal. From these two vantage points, we identify the intellectual components with which IS scholars engage, and we highlight differences along these components. The literature has primarily commented on diversity of IS topics (Benbasat and Zmud 2003); accordingly, analyses of IS topics have been undertaken from time to time (Culnan 1987, Sidorova et al. 2008, Taylor et al. 2010). There has also been concerns about fragmentation arising from the study of too many different topics, referring to the IS discipline as a “fragmented adhocracy” (Taylor et al. 2010). Based on our conceptualization of IS intellectual diversity, we systematically consider a combination of research dimensions—topic, level of phenomena, type of contribution, and method—and thus both broaden the scope of and provide a structure for understanding IS intellectual contributions. Furthermore, we show that it is not just the topic in isolation but the cohesiveness of a combination of different dimensions that represents an IS intellectual contribution. Thus, the IS discipline can leverage multiple dimensions when examining its intellectual diversity, not just the topics studied.

Moreover, we provide a framework for evaluating the ebb and flow of the topics, contributions, and methods of IS scholarship over time. In the empirical context of this paper, the analysis of the top-10 topics (Figures 9–12) reveals that authors have focused mainly on topics that accompany digitalization, such as online communities, e-commerce, and digital products and services. Interestingly, papers that make a societal contribution have been more common over the last four years, as have papers with a policy focus. These findings mirror the increasing reach of digitalization, from organizational to individual and societal processes, and support the view that the field is actively evolving. There has been a rise in experimental (laboratory, natural, and quasi-experimental) methods, machine learning analysis, and mixed methods, which is indicative of opportunities provided by the application of different methods to varied kinds of data to glean understandings of complex IS phenomena in multimodal ways. In a more general sense, the capacity of a discipline to respond to needs and opportunities by evolving its theoretical, topical, and methodological approaches is essential to its legitimacy and staying power (Kuhn 2012) and relevance (Keen 1987). Seen in that light, a longitudinal analysis of the IS intellectual diversity of a unit (e.g., an IS journal) can serve as an assessment of that unit’s resilience and of what Robey (2003) referred to as adaptative instability.

Although we examine IS intellectual diversity regarding a journal, this concept could be applied to other substantive bodies of IS research, such as a community of scholars (e.g., the members of the editorial board of a journal or the faculty of an IS group or department). Considering the intellectual diversity of editorial boards of IS journals is important for enhancing the intellectual inclusiveness of their published research.

4.2. Implications for ISR

Editorially, ISR has maintained a philosophy of publishing what the scholars in the IS field want to research (Gupta 2018). Our key finding is that while the journal exhibits a diversity of contribution types (theoretical, empirical, and, increasingly, policy related) and a reasonable diversity of topics (macro and micro), it exhibits less diversity in terms of methods, with a preponderance of analytical methods, especially econometrics. Perhaps this is not surprising given that ISR is published by INFORMS—an organization devoted to professionals and academics who focus on analytical methodologies.

However, that ISR reflects the orientation of its sponsoring organization makes the journal susceptible to the criticism that it is biased against other methods. Although ISR is an INFORMS journal, its editorial board includes individuals representing all the methodological traditions that inform the IS discipline. Moreover, as an editorial policy, ISR does not express a leaning for or against a particular method; papers submitted to the journal are handled by editors in line with this policy. Is the problem, then, that authors who use other methods are not submitting enough papers, are there are not enough such submissions of a quality high enough to be accepted at ISR, or are such papers not being reviewed using criteria appropriate to their chosen methods? The chief editor of ISR will share his views on this matter in a forthcoming editorial.

It is encouraging to see that is the same method is being used to investigate substantially different questions and that similar questions are being investigated using different methods (Table 3). In addition, the same topics are being studied at different levels of analysis (Table 1). Notably, theory, policy-related, and empirical contributions can be found across all 10 topics (Table 2). Irrespective of a paper’s topic, level, and method, it is important that authors highlight their work’s theoretical contributions so that they are clear to scholars from the field’s many theoretical and methodological traditions.

Overall, our analysis underscores that ISR supports an intellectually diverse community open to investigating many topics and applying different methods to make different contributions. Such a diverse intellectual community is necessary if ISR is to continue to realize its goal of publishing research that provides novel contributions to theory and/or has practical implications for IS design and use. The prospective indications of our analysis are that although new topics and methods will emerge in the journal, this core goal should not change. As our review of the last 10 years suggests, emerging phenomena will be investigated with new methods, reflecting the journal’s traditional focus on multimethod- and multidiscipline-informed investigations at the interstices of information technology, individuals, organizations, and societies.

Ultimately, we believe that ISR, as a premier journal in the IS field, should create opportunities for IS scholars to publish intellectually diverse research. Doing so requires sustaining editorial-board diversity particularly with respect to methodological and topical perspectives so that editors have the skills and perspective to perform the necessary stewardship. We believe that the current board significantly reflects such intellectual diversity. We hope this diversity is maintained to encourage submissions of articles in topics that emerged as less represented in our analysis, such as societal, artifact and IS discipline focused (type of contribution); IS-discipline level (level of phenomenon); and methods other than analytical and experimental.

4.3. Implications for IS Intellectual Diversity and Inclusion in the IS discipline

Should the IS discipline be intellectually diverse? We believe yes, it should be representative along the different dimensions of IS intellectual diversity. For the IS scholarly discourse to speak to the broader scholarly and practice-based communities, we must encourage intellectual diversity to engage with emerging phenomena and keep up with evolving methods. The empirical observations presented in this paper bode well for the IS field’s ability to begin developing an intellectual foundation with shared topics and methods. The ensuing hope is that we will see a natural evolution in which IS scholars leverage research insights from the different traditions in our field to begin building intradisciplinary and multiperspective explorations of complex IS research questions that afford opportunities for triangulating insights across levels of analysis. Strong IS intradisciplinary linkage is considered an indication of disciplinary resilience (Banville and Landry 1989, Taylor et al. 2010).

For specific units such as editorial boards of IS journals and IS departments/academic groups, the extent and quality of IS intellectual diversity is likely to be specific to the needs and goals of the unit; the four dimensions of IS intellectual diversity are indicative criteria that enable the unit to make choices. For example, an IS department may strategically choose to focus on specific topics and methods and, accordingly, recruit members whose work exhibits the required emphasis. By shining a light on IS intellectual diversity, we highlight the need to recognize that in practice, such a focus on specific topics, methods, or types of contribution often means that others are excluded. Although we do not view this as a problem, acknowledging IS intellectual diversity enables us to recognize implicit and unconscious biases we may have for familiar or preferred topics and methods and against not-so-familiar ones and, by extension, to take steps to mitigate the undesirable potential exclusion effects of such biases, particularly as editors and reviewers.

It is important to point out that we observe many ways in which contributions are being made by combining different dimensions of IS intellectual diversity. For example, a paper that makes a primarily empirical contribution using, for example, exploratory thematic coding or natural language processing, can also make a theoretical contribution by analyzing additional data based on theory-driven predictive modeling or structural equation modeling. Alternatively, a paper addressing a phenomenon at the micro level can be augmented with data on macro-level aspects of the same phenomenon, affording opportunities for triangulating insight across levels of analysis. The cross-tabulation analysis presented in Tables 2 and 3 illustrates that many topics are investigated at different levels of analysis and through different methods. We are not suggesting that all IS research should use multiple methods or that all topics should necessarily be researched to the same extent, but all this is to say that editors and reviewers should be intellectually open and should seek to identify interesting and novel ways of developing a paper’s contribution.

Finally, our inquiry into intellectual diversity has implications for the progression of doctoral students and early-career researchers; it suggests the need to promote the development of scholars who, irrespective of their own (narrow or broad) specializations within IS, are collegially respectful of different intellectual dimensions of IS research and have a positive orientation toward a pluralistic sociotechnical IS discipline.

Appendix A

A.1. Topic Domain

1. IS use

2. E-commerce

3. Outsourcing

4. Software development

5. Business value of IT

6. IS capability

7. IS security

8. Social networks

9. Digital products and services

10. Healthcare

11. IS innovation

12. Online communities (crowdsourcing, open source etc)

13. Digital business models

14. Privacy

A.2. Level of IS Phenomenon

1. Micro

   1. Individual

   2. Group (workplace)

2. Macro

   1. Firm

   2. Market/platform

   3. Industry

3. Societal

   1. Family/community

4. IS discipline (e.g., commentaries/papers on IS research, IS theory, and IS methods)

A.3. Focus of Contribution

1. Theoretical (testing hypotheses from theory or proposing a model prior to empirical analysis)

2. Policy

3. Empirical (no hypothesis or model, but empirical exploration of data)

4. Methods related

5. Artifact related

A.4. Method

1. Use of qualitative data (case study: positivist/interpretivist/exploratory, ethnography, hermeneutics, grounded theory)

2. Experiment (field/laboratory/quasi)

3. Use of quantitative data (survey or meta-analytic: regression, QCA)

4. Machine learning analysis (secondary data)

5. Analytical modeling (Monte Carlo, optimization, and discrete simulation)

6. Econometric modeling (econometric, game theory, etc.)

7. Algorithm and artifact development (design science)

8. Action research

Appendix B

Text in bold italic shows additions over Appendix A.

B.1. Topic Domain

• a. IS use (including adoption, diffusion)

• b. E-commerce (multi-channel, online reviews)

• c. IT sourcing

• d. Software development

• e. Business value of IT

• f. IS capability

• g. IS security

• h. Social networks

• i. Digital products and services (including online auctions)

• j. Healthcare

• k. IS innovation

• l. Online communities (crowdsourcing, open source, crowdfunding, etc.)

• m. Digital business models

• n. Privacy

• o. Fintech (investors, market announcements, etc.)

• p. Social media–driven societal phenomenon (social movements, social protest, election campaigns, etc.)

B.2. Level of the IS Phenomenon

1. Micro

   1. Individual

   2. Group (workplace)

2. Macro

   1. Firm

   2. Market/platform

   3. Industry

3. Societal

   1. Family/community

4. IS discipline (e.g., commentaries/papers on IS research, IS theory, and IS methods)

B.3. Focus of contribution

1. Theoretical (testing hypotheses from theory or proposing a model prior to empirical analysis)

2. Policy

3. Empirical (no hypothesis or model, but empirical exploration of data)

4. Methods related

5. Artifact related

B.4. Method

• a. Use of qualitative data (case study: positivist/interpretivist/exploratory, ethnography, hermeneutics, grounded theory)

• b. Experiment (field/laboratory/quasi)

• c. Use of quantitative data (survey or meta-analytic: regression, QCA)

• d. Machine learning analysis (secondary data)

• e. Computational modeling (Monte Carlo and discrete simulation)

• f. Analytical modeling (econometric, optimization, game theory, etc.)

• g. Algorithm and artifact development (design science)

• h. Action research: Methods used

• i. Multiple methods: Methods used

Appendix C

Text in bold underline shows additions over Appendix B.

C.1. Topic Domain

• a. IS use (including adoption, diffusion)

• b. E-commerce (multichannel, online reviews)

• c. IT sourcing

• d. Software development

• e. Business value of IT

• f. IS capability

• g. IS security

• h. Social networks

• i. Digital products and services (including online auctions and piracy)

• j. Healthcare

• k. IS innovation

• l. Online communities (crowdsourcing, open source, crowdfunding, etc.)

• m. Digital business models

• n. Privacy

• o. Fintech (investors, market announcements, etc.)

• p. Social media–driven societal phenomenon (social movements, social protest, election campaigns, etc.)

• q.IS discipline (state of IS research, etc.)

• r.Human–AI interaction

• s.Internet-driven negative phenomena (cyberbullying, trolling, gambling, etc.)

• t.IT professionals

• u.IT leadership (CIO)

• v.Knowledge management (knowledge sharing, knowledge transfer)

• w.IT governance (IT control, audit, etc.)

• x.IS implementation (IT project management, IT change management)

• y.IS methods (specific methods topics such as experiments, surveys, sample size, what is design science, PLS, etc.)

• z.IS theory (what is IS theory, how to build IS theory, what are various theoretical contributions, how to make various theoretical contributions)

C.2. Level of the IS Phenomenon

1. Micro

   • i. Individual

   • ii. Group (workplace)

2. Macro

   • iii. Firm

   • iv. Interorganizational

   • v. Market/platform

   • vi. Industry

3. Societal

   •  vii. Family/community

4. IS discipline (e.g., commentaries/papers on IS research, IS theory, and IS methods)

C.3. Focus of Contribution

1. Theoretical (review and theory papers, research commentaries on all topics, testing hypotheses from theory or proposing a model prior to empirical analysis)

2. Policy

3. Empirical (no hypothesis or model, but empirical exploration of data)

4. Methods related

5. Artifact related

C.4. Method

• a. Use of qualitative data (Case study: positivist/interpretivist/exploratory, ethnography, hermeneutics, grounded theory)

• b. Experiment (field/laboratory/quasi)

• c. Use of quantitative data (survey or meta-analytic: regression, QCA)

• d. Machine learning analysis (secondary data)

• e. Computational modeling (Monte Carlo and discrete simulation)

• f. Analytical modeling (econometric, optimization, game theory, etc.)

• g. Algorithm and artifact development (design science)

• h. Action research: Methods used

• i. Multiple methods: Methods used

• j. Conceptual (review and theory papers, research commentaries on all topics including theory, and papers on methods if they do not test anything empirically but propose solutions after reviewing different methods)

Appendix D

The frequency distribution of the detailed method combination is shown in Figure D.1.